Patent Description:
A containerized environment may be used to efficiently run applications on a distributed or cloud computing system. For instance, various services of an application may be packaged into containers. The containers may be grouped logically into pods, which may then be deployed on a cloud computing system, such as on a cluster of nodes that are virtual machines ("VM"). The cluster may include one or more worker nodes that run the containers, and one or more master nodes that manage the workloads and resources of the worker nodes according to various cloud and user defined configurations and policies. A cluster control plane is a logical service that runs on the master nodes of a cluster, which may include multiple software processes and a database storing current states of the cluster. To increase availability, master nodes in the cluster may be replicated, in which case a quorum of master node replicas must agree for the cluster to modify any state of the cluster. Clusters may be operated by a cloud provider or self-managed by an end user. For example, the cloud provider may have a cloud control plane that set rules and policies for all the clusters on the cloud, or provides easy ways for users to perform management tasks on the clusters.

When a cloud provider or an end user makes changes to an environment of a cluster, the changes may carry risks to the cluster. Example environment changes may include software upgrades, which may be upgrades for the nodes, for the cluster control plane, or for the cloud control plane. Another example environment change may include movement of a cluster's resources between locations, such as between datacenters at different physical locations, or between different logical locations, such as regions or zones within the same datacenter. Additionally, a user may wish to migrate from a self-managed cluster - where the user is operating as the cloud provider - to a cluster managed by a cloud provider, or generally between two clusters managed by different cloud providers. Such a migration carries risks because it involves transitioning the cluster's control plane to the control of the new cloud provider. As still another example, a user may wish to change clouds for a cluster without stopping the cluster, which may be risky to the processes that are currently running in the cluster.

<FIG> and <FIG> illustrate a current process to change an environment of a cluster, in particular a software upgrade for the cluster control plane. For instance, the cloud control plane may introduce a software upgrade, such as a new version of configurations and policies for VMs hosted by the cloud provider. As shown in <FIG>, to switch a cluster from the old version "v1. <NUM>" to the new version "v1. <NUM>," the cloud control plane deletes an old master node in the cluster and creates in its place a new master node. During this replacement process as shown in <FIG>, the new master node may be blocked from being attached to a persistent disk ("PD") until the old master node is detached from the PD and the old master node is deleted.

Document "<NPL>" refers to Blue/green deployment being a technique for releasing applications by shifting traffic between two identical environments running different versions of the application. Blue/green deployments try to mitigate common risks associated with deploying software, such as downtime and rollback capability. This paper provides an overview of the blue/green deployment methodology and describes techniques customers can implement using Amazon Web Services (AWS) services and tools. This paper also addresses considerations around the data tier, which is an important component of most applications.

Document <CIT> refers to a method for migrating a virtual machine (VM) in a computing environment. The method comprises receiving a request to migrate a VM executing on a source host to a destination host; defining a recovery point to which the VM is restored during recovery from a fault; and iteratively copying a memory of the source host associated with the VM to the destination host. During the copying, the original state of each page in the memory is preserved. At some point, the VM suspends executing on the source host, copies state information associated with the VM to the destination host, and resumes executing on the destination host. If a fault is detected on the source host, the VM is restored to the recovery point using preserved information.

Document "<NPL>" refers to application migration and the overall restructuring of an IT application landscape, wherein decisions have to be made regarding (i) the portion of the application stack to be migrated and (ii) the process to follow during the migration in order to guarantee an acceptable service level to application users.

The present disclosure provides for a method for migrating from a first cluster to a second cluster as claimed in independent claim <NUM>.

The received requests include requests from a workload running in the first cluster, wherein the requests from the workload are intercepted by a sidecar container injected in the first cluster and routed to cluster bridging aggregators of the second cluster, wherein the first cluster and the second cluster are operated on different clouds.

The allocation of the received requests may be performed in a plurality of predetermined stages, wherein the requests are directed to either the first cluster or the second cluster based on one or more of: user-agent, user account, user group, object type, resource type, a location of the object, or a location of a sender of the request.

The method may further comprise synchronizing, by the one or more processors, one or more databases in the control plane of the second cluster with one or more databases in the control plane of the first cluster, wherein the first cluster and the second cluster are operated on different clouds.

The method may further comprise allocating, by the one or more processors, a predetermined fraction of object locks to one or more controllers of the second cluster, and a remaining fraction of object locks to one or more controllers of the first cluster; actuating, by the one or more processors, objects locked by the one or more controllers of the second cluster; detecting, by the one or more processors, whether there are failures in the second cluster while actuating the objects locked; increasing, by the one or more processors based on not detecting failures in the second cluster, the predetermined fraction of object locks allocated to the one or more controllers of the second cluster.

The method may further comprise determining, by the one or more processors, that all received requests are allocated to the control plane of the second cluster; deleting, by the one or more processors based on the determination, the control plane of the first cluster, wherein the first cluster and the second cluster are operated on the same cloud. The method may further comprise stopping, by the one or more processors based on detecting one or more failures in the second cluster, allocation of the received requests to the control plane of the second cluster. The method may further comprise generating, by the one or more processors based on detecting one or more failures in the second cluster, output including information on the detected failures. The method may further comprise decreasing, by the one or more processors based on detecting failures in the second cluster, the predetermined fraction of requests allocated to the control plane of the second cluster until all received requests are allocated to the control plane of the first cluster. The method may further comprise determining, by the one or more processors, that all received requests are allocated to the control plane of the first cluster; deleting, by the one or more processors based on the determination, the second cluster.

The method may further comprise scheduling, by the one or more processors, a pod in the second cluster; recording, by the one or more processors, states of a pod in the first cluster; transmitting, by the one or more processors, the recorded states of the pod in the first cluster to the pod in the second cluster. The method may further comprise pausing, by the one or more processors, execution of workloads by the pod in the first cluster; copying, by the one or more processors, changes in states of the pod in the first cluster since recording the states of the pod in the first cluster; transmitting, by the one or more processors, the copied changes in states to the pod in the second cluster; resuming, by the one or more processors, execution of workloads by the pod in the second cluster; forwarding, by the one or more processors, traffic directed to the pod in the first cluster to the pod in the second cluster; deleting, by the one or more processors, the pod in the first cluster.

The method may further comprise determining, by the one or more processors, that a first worker node in the first cluster has one or more pods to be moved to the second cluster; creating, by the one or more processors, a second worker node in the second cluster; preventing, by the one or more processors, the first worker node in the first cluster from adding new pods; moving, by the one or more processors, the one or more pods in the first worker node to the second worker node in the second cluster; determining, by the one or more processors, that the first worker node in the first cluster no longer has pods to be moved to the second cluster; deleting, by the one or more processors, the first worker node in the first cluster.

The method may further comprise receiving, by the one or more processors, requests to one or more workloads, wherein the one or more workloads include workloads running in the first cluster and workloads running in the second cluster; allocating, by the one or more processors using at least one global load balancer, the received requests to the one or more workloads between the workloads running in the first cluster and the workloads running in the second cluster.

The method may further comprise determining, by the one or more processors, that a pod running in the second cluster references a storage of the first cluster; creating, by the one or more processors, a storage in the second cluster, wherein the storage of the first cluster and the storage of the second cluster are located at different locations; reading, by the one or more processors using a storage driver, the storage of the second cluster for data related to the pod in the second cluster; reading, by the one or more processors using the storage driver, the storage of the first cluster for data related to the pod in the second cluster. The method may further comprise writing, by the one or more processors, changes made by the pod in the second cluster to the storage of the second cluster; copying, by the one or more processors, data unchanged by the pod from the storage of the first cluster to the storage of the second cluster.

The present disclosure further provides for a system for migrating from a first cluster to a second cluster, according to independent claim <NUM>.

The first cluster and the second cluster may be at least one of: operating different software versions, operating at different locations, operating on different clouds provided by different cloud providers, operating on different clouds where at least one is a user's on-premise datacenter, or connected to different networks.

The technology relates generally to modifying an environment of a cluster of nodes in a distributed computing environment. To reduce the risks and downtime for environment changes involved in software upgrades, or moving between locations, networks, or clouds, a system is configured to modify the environment of a cluster via a live migration in a staged rollout. In this regard, while a first, source cluster is still running, a second, destination cluster may be created.

During the live migration, operations are handled by both the source cluster and the destination cluster. In this regard, various operations and/or components may be gradually shifted from being handled by the source cluster to being handled by the destination cluster. The shift may be a staged rollout, where in each stage, a different set of operations and/or components may be shifted from the source cluster to the destination cluster. Further, to mitigate damage in case of failure, within each stage, shifting operations or components from the source cluster to the destination cluster may be gradual or "canaried. " The live migration may be performed for the control planes of the clusters, as well as the workloads of the clusters.

For instance, during live migration of the cluster control plane, traffic may be allocated between the cluster control plane of the source cluster and the cluster control plane of the destination cluster. In this regard, where the source cluster and the destination cluster are operated on the same cloud, cluster bridging aggregators may be configured to route incoming requests, such as API calls from user applications and/or from workloads, to cluster control planes of both the source cluster and the destination cluster. Where the source cluster and the destination cluster are operated on different clouds, in particular where one of the clouds may not support cluster migration, one or more sidecar containers may be injected in the cluster that does not have cluster bridging aggregators. These sidecar containers may intercept and route API calls to the cluster having cluster bridging aggregators for further routing/re-routing.

Allocation of request traffic for the cluster control plane may be canaried during the live migration. For instance, initially a predetermined fraction of requests may be allocated to the cluster control plane of the destination cluster, while the remaining fraction of requests may be allocated to the cluster control plane of the source cluster. The destination cluster may be monitored while its cluster control plane is handling the predetermined fraction of requests. If no failures are detected, then allocation of requests to the cluster control plane of the destination cluster may be gradually increased, until all requests are eventually allocated to cluster control plane of the destination cluster.

Allocation of requests between the cluster control planes of the source cluster and the destination cluster may be based on predetermined rules. For example, the requests may be allocated based on resource type, object type, or location. Further, the requests may be allocated in predetermined stages.

As another example, during the live migration of the cluster control plane, object actuation may be allocated between the cluster control plane of the source cluster and the cluster control plane of the destination cluster. To further mitigate damage in case of failure, allocation of object actuation may also be canaried. For instance, at first, a predetermined fraction of object locks may be allocated to controllers of the destination cluster, while the remaining fraction of object locks may be allocated to controllers of the source cluster. The destination cluster may be monitored while actuating the objects locked by the predetermined fraction of object locks. If no failures are detected, or at least no additional failures that were not already occurring in the source cluster prior to the migration, then allocation of object locks to controllers of the destination cluster may be increased, until all objects are eventually actuated by controllers of the destination cluster.

Further, consistent data storage for the cluster control plane is to be maintained during the live migration. In this regard, if the source cluster and the destination cluster are in the same datacenter and thus share the same storage backend, databases of the source cluster and the destination cluster may be bridged, for example by joining a same quorum. On the other hand, if the source cluster and the destination cluster are operated on different locations or clouds such that they do not have access to each other's storage backend, databases of the source cluster and the destination cluster may be synchronized.

Still further, a migration may also be performed for workloads running in the cluster. In this regard, migration of the workloads may also be live. For example, as new nodes are created in the destination cluster, pods may be created in the destination cluster. Rather than immediately deleting the pods in the source cluster, execution of pods in the source cluster may be paused. States of the pods in the source cluster may be transmitted into the pods in the destination cluster, and execution may resume in the pods in the destination cluster. Additionally, a global load balancer may be configured to route requests to workloads running in both the source cluster and the destination cluster. Where the workload migration is between different locations or clouds, live storage migration may be performed for workloads to change the location of the storage for the workloads.

Once all components of the cluster control plane and/or all components of the workloads are shifted to the destination cluster, and that there is no additional failures that were not already occurring in the source cluster prior to the migration, the source cluster may's components may be deallocated or deleted. However, if failures are detected during or after the live migration, the live migration may be stopped. Additionally, a rollback may be initiated from the destination cluster back to the source cluster, and the destination cluster's components may be deallocated and deleted.

The technology is advantageous because it provides a gradual and monitored rollout process for modifying cluster infrastructure. The staged and canaried rollout process provides more opportunity to stop the upgrade in case issues arise, therefore preventing large scale damage. Traffic allocation, such as for requests to cluster control plane and/or requests to workloads, between the simultaneously running source and destination clusters may reduce or eliminate downtime during upgrade. Further, due to the traffic allocation, from the perspective of the client it may appear as if only one cluster existed during the live migration. In case of a failed upgrade, the system also provides rollback options since the source cluster is not deleted unless a successful upgrade is completed. The technology further provides features to enable live migration between clusters located in different locations, as well as between clusters operated on different clouds where one of the clouds does not support live migration.

<FIG> is a functional diagram showing an example distributed system <NUM> on which clusters may be operated. As shown, the system <NUM> may include a number of computing devices, such as server computers <NUM>, <NUM>, <NUM>, <NUM> coupled to a network <NUM>. For instance, the server computers <NUM>, <NUM>, <NUM>, <NUM> may be part of a cloud computing system operated by a cloud provider. The cloud provider may further maintain one or more storages, such as storage <NUM> and storage <NUM>. Further as shown, the system <NUM> may include one or more client computing devices, such as client computer <NUM> capable of communication with the server computers <NUM>, <NUM>, <NUM>, <NUM> over the network <NUM>.

The server computers <NUM>, <NUM>, <NUM>, <NUM> and storages <NUM>, <NUM> may be maintained by the cloud provider in one or more datacenters. For example as shown, server computers <NUM>, <NUM> and storage <NUM> may be located in datacenter <NUM>, while server computers <NUM>, <NUM> and storage <NUM> may be located in another datacenter <NUM>. The datacenters <NUM>, <NUM> and/or server computers <NUM>, <NUM>, <NUM>, <NUM> may be positioned at a considerable distance from one another, such as in different cities, states, countries, continents, etc. Further, within the datacenters <NUM>, <NUM>, there may be one or more regions or zones. For example, the regions or zones may be logically divided based on any appropriate attribute.

Clusters may be operated on the distributed system <NUM>. For example, a cluster may be implemented by one or more processors in a datacenter, such as by processors <NUM> of server computers <NUM>, or by processors <NUM> and <NUM> of server computers <NUM> and <NUM>. Further, storage systems for maintaining persistent and consistent records of states of the clusters, such as persistent disks ("PD"), may be implemented on the cloud computing system, such as in storages <NUM>, <NUM>, or in data <NUM>, <NUM>, <NUM>, <NUM> of server computers <NUM>, <NUM>, <NUM>, <NUM>.

Server computers <NUM>, <NUM>, <NUM>, <NUM> may be configured similarly. For example as shown, the server computer <NUM> may contain one or more processor <NUM>, memory <NUM>, and other components typically present in general purpose computers. The memory <NUM> can store information accessible by the processors <NUM>, including instructions <NUM> that can be executed by the processors <NUM>. Memory can also include data <NUM> that can be retrieved, manipulated or stored by the processors <NUM>. The memory <NUM> may be a type of non-transitory computer readable medium capable of storing information accessible by the processors <NUM>, such as a hard-drive, solid state drive, tape drive, optical storage, memory card, ROM, RAM, DVD, CD-ROM, write-capable, and read-only memories. The processors <NUM> can be a well-known processor or other lesser-known types of processors. Alternatively, the processor <NUM> can be a dedicated controller such as a GPU or an ASIC, for example, a TPU.

The instructions <NUM> can be a set of instructions executed directly, such as computing device code, or indirectly, such as scripts, by the processors <NUM>. In this regard, the terms "instructions," "steps" and "programs" can be used interchangeably herein. The instructions <NUM> can be stored in object code format for direct processing by the processors <NUM>, or other types of computer language including scripts or collections of independent source code modules that are interpreted on demand or compiled in advance. Functions, methods, and routines of the instructions are explained in more detail in the foregoing examples and the example methods below. The instructions <NUM> may include any of the example features described herein.

The data <NUM> can be retrieved, stored or modified by the processors <NUM> in accordance with the instructions <NUM>. For instance, although the system and method is not limited by a particular data structure, the data <NUM> can be stored in computer registers, in a relational or non-relational database as a table having a plurality of different fields and records, or as JSON, YAML, proto, or XML documents. The data <NUM> can also be formatted in a computer-readable format such as, but not limited to, binary values, ASCII or Unicode. Moreover, the data <NUM> can include information sufficient to identify relevant information, such as numbers, descriptive text, proprietary codes, pointers, references to data stored in other memories, including other network locations, or information that is used by a function to calculate relevant data.

Although <FIG> functionally illustrates the processors <NUM> and memory <NUM> as being within the same block, the processors <NUM> and memory <NUM> may actually include multiple processors and memories that may or may not be stored within the same physical housing. For example, some of the instructions <NUM> and data <NUM> can be stored on a removable CD-ROM and others within a read-only computer chip. Some or all of the instructions and data can be stored in a location physically remote from, yet still accessible by, the processors <NUM>. Similarly, the processors <NUM> can include a collection of processors that may or may not operate in parallel. The server computers <NUM>, <NUM>, <NUM>, <NUM> may each include one or more internal clocks providing timing information, which can be used for time measurement for operations and programs run by the server computers <NUM>, <NUM>, <NUM>, <NUM>.

The server computers <NUM>, <NUM>, <NUM>, <NUM> may implement any of a number of architectures and technologies, including, but not limited to, direct attached storage (DAS), network attached storage (NAS), storage area networks (SANs), fibre channel (FC), fibre channel over Ethernet (FCoE), mixed architecture networks, or the like. In some instances, the server computers <NUM>, <NUM>, <NUM>, <NUM> may be virtualized environments.

Server computers <NUM>, <NUM>, <NUM>, <NUM>, and client computer <NUM> may each be at one node of network <NUM> and capable of directly and indirectly communicating with other nodes of the network <NUM>. For example, the server computers <NUM>, <NUM>, <NUM>, <NUM> can include a web server that may be capable of communicating with client computer <NUM> via network <NUM> such that it uses the network <NUM> to transmit information to an application running on the client computer <NUM>. Server computers <NUM>, <NUM>, <NUM>, <NUM> may also be computers in one or more load balanced server farms, which may exchange information with different nodes of the network <NUM> for the purpose of receiving, processing and transmitting data to client computer <NUM>. Although only a few server computers <NUM>, <NUM>, <NUM>, <NUM>, storages <NUM>, <NUM>, and datacenters <NUM>, <NUM> are depicted in <FIG>, it should be appreciated that a typical system can include a large number of connected server computers, a large number of storages, and/or a large number of datacenters with each being at a different node of the network <NUM>.

The client computer <NUM> may also be configured similarly to server computers <NUM>, <NUM>, <NUM>, <NUM>, with processors <NUM>, memories <NUM>, instructions <NUM>, and data <NUM>. The client computer <NUM> may have all of the components normally used in connection with a personal computing device such as a central processing unit (CPU), memory (e.g., RAM and internal hard drives) storing data and instructions, input and/or output devices, sensors, clock, etc. Client computer <NUM> may comprise a full-sized personal computing device, they may alternatively comprise mobile computing devices capable of wirelessly exchanging data with a server over a network such as the Internet. For instance, client computer <NUM> may be a desktop or a laptop computer, or a mobile phone or a device such as a wireless-enabled PDA, a tablet PC, or a netbook that is capable of obtaining information via the Internet, or a wearable computing device, etc..

The client computer <NUM> may include an application interface module <NUM>. The application interface module <NUM> may be used to access a service made available by one or more server computers, such as server computers <NUM>, <NUM>, <NUM>, <NUM>. The application interface module <NUM> may include sub-routines, data structures, object classes and other type of software components used to allow servers and clients to communicate with each other. In one aspect, the application interface module <NUM> may be a software module operable in conjunction with several types of operating systems known in the arts. Memory <NUM> may store data <NUM> accessed by the application interface module <NUM>. The data <NUM> can also be stored on a removable medium such as a disk, tape, SD Card or CD-ROM, which can be connected to client computer <NUM>.

Further as shown in <FIG>, client computer <NUM> may include one or more user inputs <NUM>, such as keyboard, mouse, mechanical actuators, soft actuators, touchscreens, microphones, sensors, and/or other components. The client computer <NUM> may include one or more output devices <NUM>, such as a user display, a touchscreen, one or more speakers, transducers or other audio outputs, a haptic interface or other tactile feedback that provides non-visual and non-audible information to the user. Further, although only one client computer <NUM> is depicted in <FIG>, it should be appreciated that a typical system can serve a large number of client computers being at a different node of the network <NUM>. For example, the server computers in the system <NUM> may run workloads for applications on a large number of client computers.

As with memory <NUM>, storage <NUM>, <NUM> can be of any type of computerized storage capable of storing information accessible by one or more of the server computers <NUM>, <NUM>, <NUM>, <NUM>, and client computer <NUM>, such as a hard-drive, memory card, ROM, RAM, DVD, CD-ROM, write-capable, and read-only memories. In some instances, the storage <NUM>, <NUM> may include one or more persistent disk ("PD"). In addition, storage <NUM>, <NUM> may include a distributed storage system where data is stored on a plurality of different storage devices which may be physically located at the same or different geographic locations. Storage <NUM>, <NUM> may be connected to computing devices via the network <NUM> as shown in <FIG> and/or may be directly connected to any of the server computers <NUM>, <NUM>, <NUM>, <NUM>, and client computer <NUM>.

Server computers <NUM>, <NUM>, <NUM>, <NUM>, and client computer <NUM> can be capable of direct and indirect communication such as over network <NUM>. For example, using an Internet socket, the client computer <NUM> can connect to a service operating on remote server computers <NUM>, <NUM>, <NUM>, <NUM> through an Internet protocol suite. Server computers <NUM>, <NUM>, <NUM>, <NUM> can set up listening sockets that may accept an initiating connection for sending and receiving information. The network <NUM>, and intervening nodes, may include various configurations and protocols including the Internet, World Wide Web, intranets, virtual private networks, wide area networks, local networks, private networks using communication protocols proprietary to one or more companies, Ethernet, WiFi (for instance, <NUM>, <NUM>. 81b, g, n, or other such standards), and HTTP, and various combinations of the foregoing. Such communication may be facilitated by a device capable of transmitting data to and from other computers, such as modems (for instance, dial-up, cable or fiber optic) and wireless interfaces.

<FIG> is a functional diagram showing an example distributed system <NUM> on which live cluster migration may occur. Distributed system <NUM> includes a first cloud <NUM> and a second cloud <NUM>. As shown, cloud <NUM> may include server computers <NUM>, <NUM>, <NUM>, <NUM> in datacenters <NUM>, <NUM>, and storages <NUM>, <NUM> connected to network <NUM>. One or more client computers, such as client computer <NUM> may be connected to the network <NUM> and using the services provided by cloud <NUM>. Further as shown, cloud <NUM> may similarly include computing devices, such as server computers <NUM>, <NUM> organized in one or more datacenters such as datacenter <NUM>, and one or more storages such as storage <NUM>, connected to a network <NUM>. One or more client computers, such as client computer <NUM> may be connected to the network <NUM> and using the services provided by cloud <NUM>. Although only a few server computers, datacenters, storage, and client computer are depicted in <FIG>, it should be appreciated that a typical system can include a large number of connected server computers, a large number of datacenters, a large number of storages, and/or a large number of client computers, with each being at a different node of the network.

Cloud <NUM> and cloud <NUM> may be operated by different cloud providers. As such, cloud <NUM> and cloud <NUM> may have different configurations such that clusters operated on cloud <NUM> and cloud <NUM> are running in different software environments. Further, clusters hosted by cloud <NUM> and cloud <NUM> may or may not share any storage backend, be connected to the same network, or be in the same physical locations. As such, clusters on cloud <NUM> and cloud <NUM> may not be able to modify or even access resources, software components, and/or configurations in each other. In some instances, one or both of cloud <NUM> and cloud <NUM> may be self-managed by a user.

Live cluster migration in the distributed system <NUM> may occur in any of a number of ways. For instance, while a cluster is running in datacenter <NUM>, the cloud provider for cloud <NUM> may introduce a software upgrade for the cloud control plane, the cluster control plane running on the master nodes, or the worker nodes. As such, a migration may be performed for objects in the cluster to a destination cluster created in datacenter <NUM> that conforms with the software upgrade. In such instances, the migration is within the same datacenter <NUM>, on the same network <NUM>, and in the same cloud <NUM>.

As another example, live cluster migration may include moving between physical locations. For instance, a cloud provider for cloud <NUM> may be relocating resources, or a developer of the application running on the cluster may want to move to a different location, etc. As such, a migration may be performed for objects in the cluster in datacenter <NUM> to a destination cluster created in datacenter <NUM>. In such cases the migration may still be within the same network <NUM> and the same cloud <NUM>.

Sometimes, however, a user may want to switch from using one cloud, which may be self-managed or operated by one cloud operator, to another cloud operated by a different cloud operator. For example, a live migration may be performed for objects in a cluster on cloud <NUM> to a destination cluster created in cloud <NUM>. In addition to changing clouds, such a migration may in some cases involve a change in network and/or a change in region.

As further explained in examples below, for migration between clouds, one or both of cloud <NUM> and cloud <NUM> may be configured with features for performing live cluster migrations. For example, in instances where cloud <NUM> and cloud <NUM> both include features for performing live cluster migrations, these features may together facilitate the live cluster migration. In instances where cloud <NUM> includes features for performing live cluster migrations, while cloud <NUM> does not include features for performing live cluster migrations, cloud <NUM> and the migrating cluster on cloud <NUM> may use additional tools and methods to facilitate the migration, while such are not available to the cloud <NUM> and the migrating cluster on cloud <NUM>.

<FIG> is a functional diagram illustrating an example cluster <NUM>. For instance, a user, such as a developer, may design an application, and provide configuration data for the application using a client computer, such as client computer <NUM> of <FIG>. The container orchestration architecture provided by a cloud, such as cloud <NUM> of <FIG>, may be configured to package various services of the application into containers. The container orchestration architecture may be configured to allocate resources for the containers, load balance services provided by the containers, and scale the containers (such as by replication and deletion).

As shown in <FIG>, the container orchestration architecture may be configured as a cluster <NUM> including one or more master nodes, such as master node <NUM> and a plurality of worker nodes, such as worker node <NUM> and worker node <NUM>. Each node of the cluster <NUM> may be running on a physical machine or a virtual machine. The cluster <NUM> may be running on a distributed system such as system <NUM>. For example, nodes of the cluster <NUM> may be running on one or more processors in datacenter <NUM> shown in <FIG>. The master node <NUM> may control the worker nodes <NUM>, <NUM>. The worker nodes <NUM>, <NUM> may include containers of computer code and program runtimes that form part of a user application.

Further as shown, in some instances, the containers may be further organized into one or more pods. For example as shown in <FIG>, the worker node <NUM> may include containers <NUM>, <NUM>, <NUM>, where containers <NUM> and <NUM> are organized into a pod <NUM>, while the worker node <NUM> may include containers <NUM>, <NUM>, <NUM>, where containers <NUM> and <NUM> are organized into a pod <NUM>. The containers and pods of the worker nodes may have various workloads running on them, for example the workloads may serve content for a website or processes of an application. The pods may belong to "services," which expose the pod to network traffic from users of the workloads, such as users of an application or visitors of a website. One or more load balancers may be configured to distribute traffic, for example requests from the services, to the workloads running on the cluster <NUM>. For example the traffic may be distributed between the pods in the worker nodes of the cluster <NUM>.

Still further, some of the nodes, such as worker node <NUM>, may be logically organized as part of a node pool, such as node pool <NUM>. For example, a node pool may be a group of nodes sharing one or more attributes, such as memory size, CPU/GPU attached, etc. In some instances, all nodes of a node pool may be located in the same location of a cloud, which may be the same datacenter, same region/zone within a datacenter, etc..

The master node <NUM> may be configured to manage workloads and resources of the worker nodes <NUM>, <NUM>. In this regard, the master node <NUM> may include various software components or processes that form part of a cluster's control plane. For instance, as shown, the master node <NUM> may include an API server <NUM>, a database <NUM>, a controller manager <NUM>, and a scheduler <NUM> in communication with one another.

Although only one master node <NUM> is shown, the cluster <NUM> may additionally include a plurality of master nodes. For instance, the master node <NUM> may be replicated to generate a plurality of master nodes. The cluster <NUM> may include a plurality of cluster control plane processes. For example, the cluster <NUM> may include a plurality of API servers, a plurality of databases, etc. In such cases, a quorum of replica master nodes, such as a majority of the replica master nodes, must agree for the cluster <NUM> to modify any state of the cluster <NUM>. Further, one or more load balancers may be provided on the cloud on which the cluster <NUM> is running for allocating requests, such as API calls, between the multiple API servers. The plurality of master nodes may improve performance of the cluster <NUM> by continuing to manage the cluster <NUM> even when one or more master nodes may fail. In some instances, the plurality of master nodes may be distributed onto different physical and/or virtual machines.

The API server <NUM> may be configured to receive requests, such as incoming API calls from a user application or from workloads running on the worker nodes, and manage the worker nodes <NUM>, <NUM> to run workloads for handling these API calls. As shown, the API server <NUM> may include multiple servers, such as a built-in resource server <NUM> and an extensions server <NUM>. Further as shown, the API server <NUM> may include an aggregator <NUM> configured to route the incoming requests to the appropriate server of the API server <NUM>. For instance, when an API call comes in from a user application, the aggregator <NUM> may determine whether the API call is to be handled by a built-in resource of the cloud, or to be handled by a resource that is an extension. Based on this determination, the aggregator <NUM> may route the API call to either the built-in resource server <NUM> or the extension server <NUM>.

The API server <NUM> may configure and/or update objects stored in the database <NUM>. The API server <NUM> may do so according to a schema, which may include format that API objects in the cluster must conform to in order to be understood, served, and/or stored by other components of the cluster, including other API servers in the cluster. The objects may include information on containers, container groups, replication components, etc. For instance, the API server <NUM> may be configured to be notified of changes in states of various items in the cluster <NUM>, and update objects stored in the database <NUM> based on the changes. As such, the database <NUM> may be configured to store configuration data for the cluster <NUM>, which may be an indication of the overall state of the cluster <NUM>. For instance, the database <NUM> may include a number of objects, the objects may include one or more states, such as intents and statuses. For example, the user may provide the configuration data, such as desired state(s) for the cluster <NUM>.

The API server <NUM> may be configured to provide intents and statuses of the cluster <NUM> to a controller manager <NUM>. The controller manager <NUM> may be configured to run control loops to drive the cluster <NUM> towards the desired state(s). In this regard, the controller manager <NUM> may watch state(s) shared by nodes of the cluster <NUM> through the API server <NUM> and make changes attempting to move the current state towards the desired state(s). The controller manager <NUM> may be configured to perform any of a number of functions, including managing nodes (such as initializing nodes, obtain information on nodes, checking on unresponsive nodes, etc.), managing replications of containers and container groups, etc..

The API server <NUM> may be configured to provide the intents and statuses of the cluster <NUM> to the scheduler <NUM>. For instance, the scheduler <NUM> may be configured to track resource use on each worker node to ensure that workload is not scheduled in excess of available resources. For this purpose, the scheduler <NUM> may be provided with the resource requirements, resource availability, and other user-provided constraints and policy directives such as quality-of-service, affinity/anti-affinity requirements, data locality, and so on. As such, the role of the scheduler <NUM> may be to match resource supply to workload demand.

The API server <NUM> may be configured to communicate with the worker nodes <NUM>, <NUM>. For instance, the API server <NUM> may be configured to ensure that the configuration data in the database <NUM> matches that of containers in the worker nodes <NUM>, <NUM>, such as containers <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>. For example as shown, the API server <NUM> may be configured to communicate with container managers of the worker nodes, such as container managers <NUM>, <NUM>. The container managers <NUM>, <NUM> may be configured to start, stop, and/or maintain the containers based on the instructions from the master node <NUM>. For another example, the API server <NUM> may also be configured to communicate with proxies of the worker nodes, such as proxies <NUM>, <NUM>. The proxies <NUM>, <NUM> may be configured to manage routing and streaming (such as TCP, UDP, SCTP), such as via a network or other communication channels. For example, the proxies <NUM>, <NUM> may manage streaming of data between worker nodes <NUM>, <NUM>.

<FIG> shows some example components of two clusters involved in live migration. <FIG> shows a first cluster <NUM> as a source cluster from which objects are to be migrated, and a second cluster <NUM> as a destination cluster to which objects are to be migrated. <FIG> further shows both cluster <NUM> and cluster <NUM> with replicated master nodes, hence cluster <NUM> and cluster <NUM> are both shown with multiple API servers <NUM>, <NUM>, <NUM>, <NUM> and corresponding aggregators <NUM>, <NUM>, <NUM>, <NUM>. Although only two replicas are shown in <FIG> for ease of illustration, it should be appreciated that any of a number of replicas may be generated.

Destination cluster <NUM> runs in a different environment as source cluster <NUM>. As described above in relation to <FIG>, the different environments may be different software versions, different physical locations of datacenters, different networks, different cloud control planes on different clouds, etc. Instead of deleting a source cluster and creating a destination cluster to change the environment such as shown in <FIG>, the change of environment can be performed by a live migration of various objects from the source cluster <NUM> to the destination cluster <NUM>, while both clusters <NUM> and <NUM> are still running.

During the live migration, requests to the cluster control plane may be allocated between the source cluster <NUM> and the destination cluster <NUM>. For example, traffic such as API calls may be allocated between API servers <NUM>, <NUM> of the source cluster <NUM> and API servers <NUM>, <NUM> of the destination cluster <NUM>. As described in detail below, this may be accomplished by modifications to the aggregators <NUM>, <NUM>, <NUM>, <NUM> (see <FIG>), or by adding a component that intercepts API traffic (see <FIG>). Further, to handle the API calls routed to cluster <NUM>, cluster <NUM> may run controllers <NUM> to manage resources in cluster <NUM>, such as managing replication of worker nodes and objects. Likewise, to handle API calls routed to cluster <NUM>, cluster <NUM> may run controllers <NUM> to manage resources in cluster <NUM>.

Further as described in detail below, live migration between clusters <NUM> and <NUM> may include handling objects stored for the cluster control plane in database <NUM> and database <NUM>. For example, if clusters <NUM> and <NUM> are in the same datacenter and thus share the same storage backend, database <NUM> and database <NUM> may be bridged. On the other hand, if cluster <NUM> and cluster <NUM> are on different locations or clouds such that they do not have access to each other's storage backend, database <NUM> and database <NUM> may need to be synchronized (see <FIG>).

In addition to migration for the cluster control plane, a live migration may be performed for workloads running in the clusters, such as workloads <NUM> running on the source cluster <NUM> and workloads <NUM> running on the destination cluster. Requests to workloads, such as API calls to workloads, may also be routed between the source cluster <NUM> and the destination cluster <NUM>, for example by using a global load balancer (see <FIG>). Further, the location of the storage for workloads may need to be changed for a migration across different locations or different clouds (see <FIG>).

Further as shown in <FIG>, a coordinator <NUM> may be provided, for example by the cloud provider for cloud <NUM>, which includes various rules for implementing the live migration. In this regard, if the migration is within the same cloud, such as cloud <NUM>, both the source cluster <NUM> and the destination cluster <NUM> may perform the migration based on the rules set in the coordinator <NUM>. On the other hand, if the migration is between two different clouds, such as cloud <NUM> and cloud <NUM>, in some instances only the cluster in the same cloud as the coordinator <NUM> might be able to follow the rules set in the coordinator <NUM>. For example, the destination cluster <NUM> may be on cloud <NUM> and able to perform live migration based on the rules set in the coordinator <NUM>; while the source cluster <NUM> may be on cloud <NUM> that is self-managed or managed by a different cloud, and may not have necessary features for following the rules set in the coordinator <NUM>. As such, cloud <NUM> may include additional features to facilitate a migration from or to cloud <NUM>.

With respect to live migration of a cluster control plane, <FIG> illustrates example cluster bridging aggregators configured to route requests, such as API calls, between control planes of two clusters during a live migration within the same cloud. <FIG> shows a first cluster <NUM> as a source cluster from which objects are to be migrated, and a second cluster <NUM> as a destination cluster into which objects are to be migrated. In this example, both source cluster <NUM> and destination cluster <NUM> are hosted on the same cloud, such as cloud <NUM>. <FIG> further shows both cluster <NUM> and cluster <NUM> with replicated master nodes, hence cluster <NUM> and cluster <NUM> are both shown with multiple API servers <NUM>, <NUM>, <NUM>, <NUM> and corresponding cluster bridging aggregators <NUM>, <NUM>, <NUM>, <NUM>.

One or more load balancers may be configured to allocate incoming requests, such as API calls, between the various API servers based on traffic volume. For instance, a load balancer may be associated with all the API servers of a cluster, such as by network addresses of the API servers. However, the load balancer may be configured to provide client(s) of the cluster, such as application(s) run by the cluster, a single network address for sending all API calls. For example, the single network address may be a network address assigned to the load balancer. As the load balancer receives incoming API calls, the load balancer may then route the API calls based on traffic volume. For example, the load balancer may divide the API calls among the API servers of the cluster, and send the API calls based on the network addresses of the API servers.

Further as shown, the aggregators in the source cluster <NUM> and destination cluster <NUM> are both modified into cluster bridging aggregators <NUM>, <NUM>, <NUM>, <NUM>. The cluster bridging aggregators <NUM>, <NUM>, <NUM>, <NUM> are configured to receive the incoming requests, such as API calls, from the load balancer <NUM>, and further route requests to the API servers <NUM>, <NUM>, <NUM>, <NUM>. For example, control plane of the cloud <NUM>, for example through coordinator <NUM>, may notify the cluster bridging aggregators <NUM>, <NUM>, <NUM>, <NUM> when migration is initiated. Once the cluster bridging aggregators <NUM>, <NUM>, <NUM>, <NUM> become aware of the migration, the cluster bridging aggregators <NUM>, <NUM>, <NUM>, <NUM> may determine whether the incoming API calls should be handled by the source cluster <NUM> or the destination cluster <NUM>. Based on this determination, the cluster bridging aggregators <NUM>, <NUM>, <NUM>, <NUM> may route the API calls to the appropriate API servers.

For instance, if an API call arrives at cluster bridging aggregator <NUM> of the source cluster <NUM>, the cluster bridging aggregator <NUM> may determine whether the API call should be handled by the API servers of the source cluster <NUM>, or the API servers of the destination cluster <NUM>. If the cluster bridging aggregator <NUM> determines that the API call is to be handled by the API servers of the source cluster <NUM>, cluster bridging aggregator <NUM> may route the API call to the corresponding API server <NUM>. Otherwise, the cluster bridging aggregator <NUM> may re-route the API call to the API servers of the destination cluster <NUM>. Likewise, if an API call arrives at cluster bridging aggregator <NUM> of the destination cluster <NUM>, the cluster bridging aggregator <NUM> may determine whether the API call should be handled by the destination cluster <NUM>, or the source cluster <NUM>. If the cluster bridging aggregator <NUM> determines that the API call is to be handled by the destination cluster <NUM>, cluster bridging aggregator <NUM> may route the API call to the corresponding API server <NUM>. Otherwise, the cluster bridging aggregator <NUM> may route the API call to the API servers of the source cluster <NUM>. Because the API servers of the source cluster <NUM> and the API servers of the destination cluster <NUM> may implement different schema for objects they handle, changes in API traffic allocation may effectively change the portion of objects conforming to the schema of the destination cluster <NUM>.

The cluster bridging aggregators <NUM>, <NUM>, <NUM>, <NUM> may route or re-route API calls based on any of a number of factors. For example, the routing may be based on a resource type, such as pods, services, etc. For instance, the cluster bridging aggregators <NUM>, <NUM> may route API calls for all pods to the API servers <NUM>, <NUM> in the source cluster <NUM>, and re-route API calls for all services to the destination cluster <NUM>. The routing may alternatively be based on object type. For instance, cluster bridging aggregators <NUM>, <NUM> may route <NUM>% of API calls for pod objects to the API server <NUM>, <NUM> in the source cluster <NUM>, and re-route the rest to the destination cluster <NUM>. As another alternative, routing may be based on physical location of a resource. For example, cluster bridging aggregators <NUM>, <NUM> may route <NUM>% of API calls for pods in a particular datacenter, and re-route the rest to the destination cluster <NUM>. Other example factors may include user-agent, user account, user group, location of a sender of the request, etc. The factors for API call routing may be set in the coordinator <NUM> by the cloud provider for cloud <NUM>.

The cluster bridging aggregators <NUM>, <NUM>, <NUM>, <NUM> may route or re-route API calls in a staged manner. For example, cluster bridging aggregators <NUM>, <NUM> may start routing API calls for one resource type to API servers <NUM>, <NUM> of the destination cluster <NUM> in one stage, and then changes to include API calls for another resource type to the API servers <NUM>, <NUM> of the destination cluster <NUM> in a next stage, and so on. Alternatively, cluster bridging aggregators <NUM>, <NUM> may start routing API calls for one physical location to API servers <NUM>, <NUM> of destination cluster <NUM> in one stage, and then changes to include routing API calls for another physical location to API servers <NUM>, <NUM> of destination cluster <NUM> in a next stage, and so on. As another example, cluster bridging aggregators <NUM>, <NUM> may route API calls to the API servers <NUM>, <NUM> in increasing proportions, such as routing API calls for <NUM>% of pod objects to API servers <NUM>, <NUM> of the destination cluster <NUM> in one stage, and routing API calls for <NUM>% of pod objects to API servers <NUM>, <NUM> of the destination cluster <NUM> in a next stage, and so on. The stages of API call routing may be set in the coordinator <NUM> by the cloud provider for cloud <NUM>.

To determine whether to route or re-route a request, the cluster bridging aggregators <NUM>, <NUM>, <NUM>, <NUM> may be provided with information on the allocations to be made. For instance, the cluster bridging aggregators <NUM>, <NUM>, <NUM>, <NUM> may be configured to access one or more databases, such as database <NUM> of the destination cluster <NUM>, for the fraction of traffic to be allocated to the source cluster <NUM> and to the destination cluster <NUM>. As such, when an API call arrives for example at cluster bridging aggregator <NUM>, the cluster bridging aggregator <NUM> may compute a hash value for the API call based on the faction (<NUM><F<<NUM>) of API calls to be allocated to the destination cluster <NUM>. The hash value may be further computed based on other information of the API call, such as IP address of the source of the API call and metadata of the API call. Such information may be used to determine resource type, object type, physical location, etc., that are relevant in the staged rollout process described above. In some examples, the hash value may also be interpreted as a numeric value p that is a fraction between <NUM> and <NUM>. If p < F, then the cluster bridging aggregator <NUM> may route the API call to the destination cluster <NUM>, otherwise, the cluster bridging aggregator <NUM> may route the API call to the source cluster <NUM>. Decisions made based on the hash values may be defined deterministically so that no matter which cluster bridging aggregator involved in the migration receives the API call, it will make the same decision as the other cluster bridging aggregators. As such, there will not be a need to re-route an API call more than once. In some instances, during transitions in the staged rollout described above, different fractions F may be set, for example different resources, different physical locations, etc..

Additionally, the cluster bridging aggregators may further be configured to allocate other resources between the two clusters. For example, the destination cluster <NUM> may use different controllers to run control loops as compared to controllers used by the source cluster <NUM>. As such, switching between the controllers of the source cluster and controllers of the destination cluster may also be performed in a staged rollout. For instance, to ensure that inconsistent changes are not made to objects, controllers may acquire locks before manipulating the objects. As such, the cluster bridging aggregators <NUM>, <NUM>, <NUM>, <NUM> may be configured to allocate controller locks between the controllers of the source cluster <NUM> and the controllers of the destination cluster <NUM>. The allocation may also be performed in predetermined stages, which may also be canaried.

Together, the API servers <NUM>, <NUM>, <NUM>, <NUM>, and cluster bridging aggregators <NUM>, <NUM>, <NUM>, <NUM> in <FIG> essentially form a logical API service. Clients of this logical API service may thus send requests to this logical API service, and the requests will be routed by the various cluster bridging aggregators and handled by the various API servers. To the clients, there may be no observable difference other than possible latency.

However, if the first, source cluster <NUM> and the second, destination cluster <NUM> are hosted on different clouds, one of the source cluster <NUM> or the destination cluster <NUM> may not be provided with cluster bridging aggregators, <FIG> illustrates an additional component intercepting requests, such as API calls, to the cluster control plane when performing a live cluster migration between two different clouds. In this example shown, destination cluster <NUM> is on cloud <NUM> configured to perform live migration, while source cluster <NUM> is on cloud <NUM> that is self-managed or managed by a different cloud provider that is not configured to perform live migration. As such, the destination cluster <NUM> on cloud <NUM> is provided with cluster bridging aggregators <NUM>, <NUM> as described above, while the source cluster <NUM> on cloud <NUM> is provided with aggregators <NUM>, <NUM> that cannot route and re-route API calls between clusters.

Since the two clusters here are on different clouds, requests, such as API calls, will not be received through the same load balancer <NUM> as shown in <FIG>. Rather, API calls will be routed to the cluster bridging aggregators in the source cluster <NUM> and the destination cluster <NUM>, based on their different network addresses, such as IP addresses.

Further as shown in <FIG>, since cluster <NUM> does not include cluster bridging aggregators, sidecar containers may be injected into pods on cloud <NUM> for intercepting requests, such as API calls directed to the API servers locally in the cluster <NUM>, and re-routing them to the cluster bridging aggregators <NUM>, <NUM> in the destination cluster <NUM>. For example, the sidecar containers may be injected by an extension the user installs on the cloud control plane of cloud <NUM>. The sidecar containers may be injected into every workload pod running in the source cluster <NUM>. For example as shown, sidecar container <NUM> is injected into pod <NUM> in cluster <NUM>. The sidecar container <NUM> may be configured to intercept API calls from the workloads <NUM> running in pod <NUM>, which are directed to API server <NUM> or <NUM>, and simulate the cluster bridging aggregator which is absent from source cluster <NUM>. It does this simulation simply by redirecting these API calls to the cluster bridging aggregators <NUM>, <NUM> in the destination cluster <NUM>. The cluster bridging aggregators <NUM>, <NUM> may then determine whether these API calls shall be handled locally by API server <NUM>, <NUM>, or if it should be sent back to the source cluster's API servers <NUM>, <NUM>. The cluster bridging aggregators <NUM>, <NUM> may make determinations as discussed above in relation to <FIG>, and route the API calls accordingly.

Together, the API servers <NUM>, <NUM>, <NUM>, <NUM>, aggregators <NUM>, <NUM>, sidecar container <NUM>, cluster bridging aggregators <NUM>, <NUM> in <FIG> essentially form a logical API service. Clients of this logical API service may thus send requests to this logical API service, and the requests may be intercepted by the sidecar container <NUM>, and/or routed by the various cluster bridging aggregators, and handled by the various API servers. To the clients, there may be no observable difference other than possible latency.

As alternatives to injecting a sidecar container as described above, other components or processes may be used to intercept and re-route requests. For example, domain name service (DNS) entries may be injected into the nodes for re-routing to the cluster bridging aggregators of the destination cluster.

Returning to <FIG>, with respect to storage for the cluster control plane, in instances where the source cluster <NUM> and destination cluster <NUM> are on the same cloud and within the same datacenter, database <NUM> may join the same quorum as database <NUM>. As such, the quorum of databases including the database <NUM> or database <NUM> must reach an agreement before objects are to be modified or written into any of the quorum of databases. For example, an agreement may be reached when a majority of the database replicas agree to the change. This ensures that database <NUM> and database <NUM>, and their replicas, reflect consistent changes. In some examples, database <NUM> may join at first as non-voting member of the database quorum, and later becomes a voting member of the quorum.

However, if the source cluster <NUM> and the destination cluster <NUM> are not on the same cloud or same datacenter, database <NUM> may not be able to join the quorum of database <NUM>. As such, <FIG> illustrates example cluster control plane storage synchronization during live migration for clusters on different clouds and/or regions. For example, a first, source cluster <NUM> may be on cloud <NUM> and a second, destination cluster <NUM> may be on cloud <NUM>. As another example, destination cluster <NUM> may be in datacenter <NUM> and source cluster <NUM> may be on datacenter <NUM>.

In a containerized environment, some fields of an object can only be modified by an API server and are otherwise immutable. Thus, once immutable fields of an object are written or modified by an API server of the source cluster <NUM>, such as API server <NUM> or <NUM>, API servers of the destination cluster <NUM>, such as API server <NUM> or <NUM>, may not be able to modify these fields as stored in the database <NUM> of the source cluster <NUM>. Thus as shown, for example when an API call comes in at the cluster bridging aggregator <NUM> requesting a new object be created or immutable fields modified, the API call may be modified by the cluster bridging aggregator <NUM> and sent first to the source cluster <NUM>, such as to aggregator <NUM>. The API server <NUM> may create or modify object <NUM> stored in database <NUM> according to the modified API call.

The cluster bridging aggregator <NUM> may then use its local API server <NUM> to create its own copy of the object <NUM> in database <NUM>, shown as object <NUM> in database <NUM>. For instance, the cluster bridging aggregator <NUM> may read the immutable fields having the values chosen by the API server <NUM> of the source cluster <NUM>, and write these values into object <NUM>.

In some instances, the cluster bridging aggregator <NUM>, <NUM> may block read-only operations for an object while write operations are in progress for that object to ensure that API callers see a consistent view of the world. Otherwise, API callers may observe only part of the changes performed, since as described above, making a write in this migrating environment may be a multi-step process. Additionally, API callers have expectations around the concurrency model of API server which need to be upheld for the process to be transparent to these callers.

In another aspect, a migration may also be performed for workloads running in the clusters. <FIG> shows example features involved in performing workload migration. For instance, a first, source cluster <NUM> is shown with node pool <NUM>, which includes nodes <NUM>, <NUM>, <NUM>. One or more pods may be running in the nodes of cluster <NUM>, such as pod <NUM> and pod <NUM> shown. Cluster <NUM> may further include a local load balancer <NUM> for allocating traffic to workloads in the cluster <NUM>. For instance, requests from websites or applications served by the workloads may be received by the local load balancer <NUM>, and the local load balancer <NUM> may allocate these requests to the various pods and nodes in node pool <NUM>. For example, the websites or application served by the workloads of cluster <NUM> may be configured with domain name service (DNS) records associating the website or application to a network address of the local load balancer <NUM>.

Further as shown, workloads within cluster <NUM> are to be migrated to a second, destination cluster <NUM>. The cluster <NUM> may be initialized with a node pool <NUM> that does not have any node, and a local balancer <NUM> for allocating incoming requests to workloads once pods and nodes are created in the cluster <NUM>. A migration may be performed for the node pool <NUM> from cluster <NUM> to cluster <NUM> within the same location, such as within the same datacenter or within the same region/zone of a datacenter, or it may be between different locations. The migration may also be performed within the same cloud or between different clouds. Although clusters <NUM> and <NUM> are shown with only one node pool, in practical examples the clusters <NUM> and <NUM> may include a plurality of node pools. In instances where a cluster does not already group nodes into node pools, during the migration each node may be treated as its own node pool, or nodes with similar sizes may be grouped together, etc..

Once the destination cluster <NUM> is initialized, the node pool <NUM> may gradually increase in size. For example, a new node <NUM> may be allocated in node pool <NUM>. The new node <NUM> initially may not include any pods. In response to the increase in size of the node pool <NUM>, the old node pool <NUM> may decrease in size. For example, old node <NUM> may be deleted. The allocation of new nodes and removal of old nodes may be performed by a cloud provider as instructed by the coordinator.

The cluster control plane of the source cluster <NUM> and/or the destination cluster <NUM> may be notified that node <NUM> is now missing, and register all the pods previously existing in node <NUM>, such as pods <NUM> and <NUM> shown, as lost. As such, cluster control plane of the destination cluster <NUM> may create replacement pods in the new node pool <NUM>. For instance, controllers of the destination cluster <NUM> may determine that new node <NUM> in node pool <NUM> has capacity, and may create replacement pods, such as replacement pods <NUM> and <NUM> shown, in the new node <NUM>. Thus, effectively, the pods <NUM>, <NUM> are moved into the second cluster as pods <NUM>, <NUM>. This may be repeated for other nodes in node pool <NUM>, such as creating new nodes <NUM> and <NUM> in node pool <NUM> corresponding to nodes <NUM>, <NUM> as shown, and replacing any missing pods, until node pool <NUM> no longer has any nodes and/or pods.

As an alternative to deleting node <NUM> and adding node <NUM> before moving any pods, a live migration may be performed. For instance, once new node <NUM> is created, node <NUM> may be "cordoned" such that new pods are prevented from being scheduled on node <NUM>. Then, new pod <NUM> is created in node <NUM>. The states of the pod <NUM> may be recorded and transmitted to pod <NUM>. Then, executions of processes in pod <NUM> may be paused. If there had been any changes to pod <NUM> since recording the states, these changes may also be copied into pod <NUM>. The paused executions may then resume in pod <NUM>. Pod <NUM> may then be deleted. During this live migration, traffic directed to pod <NUM>, such as requests to workloads, may be forwarded to pod <NUM>, until pod <NUM> is deleted. For example, a load balancer may have directed requests to pod <NUM>, before being aware of newly created pod <NUM>. This may be repeated for each pod in the various nodes and node pools of source cluster <NUM>, until there is no pod left.

Further, migration of the workloads may include, in addition to migration of the pods, also migration of the services to which the pods belong. Migration of the services may overlap with migration of the pods. For instance, once one or more pods are created in the destination cluster <NUM>, services previously handled by pods of the source cluster <NUM> may be migrated to be handled by the pods in the destination cluster <NUM>. Further, migration of the services may need to be completed before there is no more pods in the source cluster <NUM> to handle the services.

In this regard, one or more global load balancers may be created. For instance, once the workload node and pod migration is initiated but before any node is moved, the source cluster <NUM> and the destination cluster <NUM> may each be associated with one or more load balancers configured to route requests to workloads running in both the source cluster <NUM> and the destination cluster <NUM>. For example as shown, both the local load balancer <NUM> and the local load balancer <NUM> may be associated with global load balancer <NUM>. Thus, if the source cluster <NUM> and the destination cluster <NUM> are in different locations or clouds, the global load balancer <NUM> may be configured to route requests to these different locations or clouds. The websites or application previously served by the workloads of cluster <NUM> may be configured with DNS records associating the website or application to a network address of the global load balancer <NUM>, instead of previously to the local load balancer <NUM>. As such, once workload node and pod migration starts, requests from the website or application may be routed through the global load balancer <NUM> to both local load balancers <NUM> and <NUM>.

Once workload node and pod migration is complete, association between the local load balancer <NUM> and the global load balancer <NUM> may be removed. Further, the websites or application previously served by both cluster <NUM> and cluster <NUM> may be configured with DNS records associating the website or application to a network address of the local load balancer <NUM>. Thus, from this point on, local load balancer <NUM> may be configured to route requests from the website or application to only the workloads running in the destination cluster <NUM>.

Still further, where migration of workloads as shown in <FIG> is between different locations or between different clouds, live migration of workload storage may need to be performed. <FIG> shows live workload storage migration between different locations or clouds. For instance, the live workload storage migration may occur simultaneously as the migration of pods as shown in <FIG>. A storage system for a containerized environment may include various objects storing data. For example, the storage system may include persistent disks provided by a cloud provider, and metadata objects containing references. For instance, the metadata objects may be used to set up or "mount" persistent disk(s) for pods or containers. As some examples, the metadata objects may include persistent volumes that refer to data on the persistent disks, and persistent volume claims that refer to the persistent volumes and store information on usage of such data by containers or pods.

When the migration is between different locations or clouds, the metadata objects may be copied to a destination environment, but the persistent disk may not be copied to the destination environment. Thus, a live migration of the storage system for workloads may be performed by tracking locations of each persistent disk, duplicating the metadata objects in a destination environment, and using a copy-on-write system to copy over data.

For example as shown, while running in a first, source cluster <NUM>, a pod <NUM> may have an already existing metadata object <NUM>, which may refer to a persistent disk <NUM>. To make effective copies of these storage objects, a helper pod <NUM> may be created in the source cluster <NUM> and attached to the metadata object <NUM>. This helper pod <NUM> may be configured to read from the persistent disk <NUM> after the pod <NUM> migrates to a second, destination cluster <NUM> as pod <NUM>.

The migrated pod <NUM> is then attached to a node in the destination cluster <NUM> and to a newly created metadata object <NUM>, which may be a duplicate of metadata object <NUM>. It may be determined that the metadata object <NUM> of the migrated pod <NUM> includes references to the persistent disk <NUM>. To set up storage for the migrated pod <NUM>, a storage driver <NUM> may determine that the persistent disk <NUM> is in a different cluster. As such, a new persistent disk <NUM> may be created in the destination cluster <NUM>.

However, instead of being directly attached to the new persistent disk <NUM>, the pod <NUM> may initially perform reads and/or writes through the storage driver <NUM>, which may determine that the pod <NUM> and the metadata object <NUM> are referring to persistent disks at two different locations. For example, the storage driver <NUM> may be run as a plugin on the node <NUM> of <FIG>. The storage driver <NUM> may be configured to access both the old persistent disk <NUM>, for example, via network access to helper pod <NUM>, and the new persistent disk <NUM>.

For instance, to read, the pod <NUM> may use storage driver <NUM> to read from the new persistent disk <NUM>. Additionally, the storage driver <NUM> may also call the helper pod <NUM>, which may read from the persistent disk <NUM>.

In order to write, the pod <NUM> may also do so through the storage driver <NUM>. The storage driver <NUM> may be configured to direct all writes to the persistent disk <NUM>. This way, any new changes are written into the new persistent disk <NUM>. Writing may be performed by copy-on-write, where changes are directly written into the new persistent disk <NUM>, while unchanged data are copied over from the old persistent disk <NUM>.

Further, a migration may be performed in the background to gradually move all data from storage objects in the source cluster <NUM> to the destination cluster <NUM>. For example when the network is not busy, the storage driver <NUM> may continue to read data from persistent disk <NUM>, and then write this data into persistent disk <NUM>. Once all the data are copied over, the persistent disk <NUM> will contain the complete file system, and the pod <NUM> may be directly attached to the persistent disk <NUM> without the storage driver <NUM>. The old persistent disk <NUM> may be deleted. During this process, from the perspective of the pod <NUM>, there is no difference other than possible latency.

Although <FIG> shows one metadata object between a pod and a persistent disk, in some examples there may be multiple metadata objects referring to one another forming a chain of references. For example, a pod may refer to a persistent volume claim, which may refer to a persistent volume, which may then refer to a persistent disk.

Further to example systems described above, example methods are now described. Such methods may be performed using the systems described above, modifications thereof, or any of a variety of systems having different configurations. It should be understood that the operations involved in the following methods need not be performed in the precise order described. Rather, various operations may be handled in a different order or simultaneously, and operations may be added or omitted.

For instance, <FIG> are timing diagrams illustrating an example live cluster migration for the cluster control plane. <FIG> shows various actions occurring at a source master node <NUM> in a first, source cluster, a destination master node <NUM> in a second, destination cluster, a logical API service <NUM>, and a coordinator <NUM>. The source master node <NUM> and destination master node <NUM> may be configured as shown in any of <FIG>. Although only one source master node <NUM> and only one destination master node <NUM> are shown, there may be any number of master nodes in either or both of the source cluster and the destination cluster, such as shown in <FIG>. The logical API service <NUM> may be a quorum of API servers for one or more clusters, which include aggregators and/or cluster bridging aggregators as shown in <FIG>, and/or sidecar containers as shown in <FIG>. The timing diagram may be performed on a system, such as by one or more processors shown in <FIG> or <FIG>.

Referring to <FIG>, initially, a source master node <NUM> of a source cluster may already be running on a cloud. As such, the source master node <NUM> is already attached to a PD, and API server(s) of the source master node <NUM> may already be member(s) of the logical API service <NUM>.

At some point, a cloud provider of the cloud or a user may initiate an environment change, such as introducing a software upgrade, moving to a different datacenter, moving to/from a different cloud, etc. The cloud provider may further define rules for a live migration to implement the environment change in the coordinator <NUM>, and the coordinator <NUM> may instruct the logical API service <NUM> to implement the rules. For example, the rules may include factors for workload traffic allocation and stages of migration.

Once the environment change is initiated, a destination master node <NUM> may be created and attached to a PD. To maintain consistent changes as the source master node <NUM>, one or more databases of the destination master node <NUM> may be bridged or synchronized with the one or more database(s) of the source master node <NUM>. For example, in instances where the source master node <NUM> and the destination master node <NUM> are in the same cloud and location, database(s) of the destination master node <NUM> may join the same quorum as the database(s) of the source master node <NUM>. In instances where the source master node <NUM> and the destination master node <NUM> are in different clouds or locations, database(s) of the destination master node <NUM> may be synchronized to the database(s) of the source master node <NUM> as shown in <FIG>.

At this point the destination master node <NUM> may begin running, while the source master node <NUM> continues to run. As such, downtime is reduced or eliminated as compared to the process shown in <FIG> and <FIG>. To simultaneously handle requests to the cluster control plane, such as API calls, API server(s) of the destination master node <NUM> may join the logical API service <NUM>. For instance, the API server(s) of the destination master node <NUM> may join the logical API service <NUM> via cluster bridging aggregator(s) as shown in <FIG>, or sidecar pod(s) may be created as shown in <FIG>.

Once the coordinator <NUM> observes the API server(s) of the destination master node <NUM>, the coordinator <NUM> may begin a staged rollout to change the environment. Continuing to <FIG>, the timing diagram illustrates an example staged rollout of API traffic from the source cluster to the destination cluster. As shown, the coordinator <NUM> may instruct the logical API service <NUM> to implement a staged traffic allocation between API server(s) of the source master node <NUM> and API server(s) of the destination master node <NUM>. The API traffic allocation may be implemented using cluster bridging aggregator(s) as shown in <FIG>, and/or using one or more sidecar containers as shown in <FIG>. Since API servers of the source cluster and the destination cluster may handle objects based on different schemas, the destination schema for objects in the destination environment is gradually rolled out as API traffic is increasingly routed to API server(s) of the destination master node <NUM>.

As shown in <FIG>, during the rollout stage, incoming API calls may be routed to API server(s) of the destination master node <NUM> and the API server(s) of the source master node <NUM> via the logical API service <NUM>. The coordinator <NUM> may set predetermined proportions of API traffic allocation. In the particular example shown, initially <NUM>% of the received API calls may be handled by API server(s) of the destination master node <NUM> and remaining <NUM>% of the received API calls may be handled by API server(s) of the source master node <NUM>. In other words, initially only <NUM>% of API calls are handled by API server(s) of the destination master node <NUM> according to the schema of the destination environment, the rest are handled by API server(s) of the source master node <NUM> according to the schema of the source environment. In addition to or as alternative to allocating the API traffic by predetermined proportions, API traffic may be further allocated according to other criteria, such as by resource type, by user, by namespace, by object type, etc..

During the rollout process, activities in the API server(s) of the destination master node <NUM> may be monitored. For instance, the coordinator <NUM> may monitor activities of cluster control plane components, such as API servers, controller managers, etc. The coordinator <NUM> may further monitor the workloads, such as comparing workloads handled by the source and destination clusters for problematic differences. As such, if no failure is detected with one proportion of API calls handled by the API server(s) of the destination master node <NUM>, or at least no additional failures that were not already occurring in the source cluster <NUM> prior to the migration, then API traffic to the API server(s) of the destination master node <NUM> may be increased to a higher proportion, and so on. For example as shown, the API calls routed to the API server(s) of the destination master node <NUM> may increase from <NUM>% to <NUM>%, <NUM>%, <NUM>%, etc. However, if one or more failures are detected in the proportion of API calls handled by the API server(s) of the destination master node <NUM>, the failure may act as a warning that more failures may result if a greater proportion of API calls are handled by the API server(s) of the destination master node <NUM>. Appropriate actions may be taken based on the warning, such as reverting all API traffic to the source API server as shown in <FIG>.

Further as shown, in some instances a discovery document including information on the destination environment, such as the exact schema to be followed by objects, may be made available to a user only once the API server(s) of the destination master node <NUM> handle all the incoming API calls. For example, as each type of object becomes fully handled by the destination cluster, a section in the discovery document for the corresponding type of object may be updated with destination schema for that type of object. In other words, end users may not be able to observe any environment change up until this point, when all objects are being handled by API server(s) of the destination master node <NUM> based on the destination schema. At this point, there is no more API traffic received by the source master node <NUM>, and thus no object is being handled by the API server(s) of the source master node <NUM> based on the old schema. Control plane of the source master node <NUM> may also observe the new discovery document, and is notified that the schema migration is complete.

Once the coordinator <NUM> observes the completed schema migration, the coordinator <NUM> may optionally begin a staged rollout for one or more other aspects of the clusters. For example, continuing to <FIG>, the timing diagram illustrates an example staged rollout for controllers. In some instances, an environment change may involve change in controllers that actuate objects of a cluster. For example, the destination master node <NUM> in the destination environment may use different controllers to run control loops as compared to the controllers used by the source master node <NUM>. As such, switching between the controllers of the source master node <NUM> and the controllers of the destination master node may also be performed in a staged rollout. For instance, to ensure that inconsistent changes are not made to objects, controllers may acquire locks before manipulating the objects. As such, the coordinator <NUM> may instruct the logical API service <NUM> to implement a staged controller lock allocation between controllers of the source cluster and controllers of the destination cluster.

Thus in the particular example shown in <FIG>, initially only <NUM>% of controller locks are given to the controllers of the destination master node <NUM>, the rest of the controller locks are given to the controllers of the source master node <NUM>. As with rollout of API servers, the coordinator <NUM> may monitor activities of cluster control plane components, such as API servers, controller managers, and/or workloads for any failure due to switching to the controllers of the destination master node <NUM>. If no failure is detected, or at least no additional failures that were not already occurring in the source cluster <NUM> prior to the migration, the proportion of controller locks given to the controllers of the destination master node <NUM> may be gradually increased. Further, to ensure no object is manipulated by two controllers while adjustments are made to the controller lock allocation, such as going from <NUM>% lock to <NUM>% lock allocation, the controllers may be configured to maintain the locks on the objects they already control in the previous stage. Eventually, all controller locks may be given to the controllers of the destination master node <NUM>, and at that point, there is no more controller activity at the source master node <NUM>.

At this point, optionally the coordinator <NUM> may switch any other remaining add-ons. For example, objects may be handled by add-on components of the destination master node <NUM>, instead of add-on components of the source master node <NUM>. Example add-on components may include a user interface, such as a dashboard, a Domain Name System (DNS) server, etc. Optionally, the add-on components may be switched in the staged rollout as described above for API servers and controllers.

Once the rollout from the source environment to the destination environment is completed, a shutdown process may begin for the source master node <NUM>. For instance, any bridging, synchronization, or migration of databases between the source master node <NUM> and the destination master node <NUM> may be stopped. Further, PD may be detached from the source master node <NUM>, and the source master node <NUM> may then be deleted. Once the source master node <NUM> is destroyed, the coordinator <NUM> may report the successfully completed migration to the cloud.

In addition to migration of cluster control plane, a live migration may be performed for workloads. <FIG> is a timing diagram illustrating an example live migration for workloads in a cluster from one environment to another environment. <FIG> shows various actions occurring at an old pod <NUM> on a node of a first, source cluster, a new pod <NUM> created on a node of a second, destination cluster, and the cluster control planes <NUM> of the two clusters. The pods may be configured on worker nodes as shown in any of <FIG> or <FIG>, for example old pod <NUM> may be configured on node <NUM> of source cluster <NUM> and new pod <NUM> may be configured on node <NUM> of cluster <NUM>. Although example operations involving only one old pod <NUM> and only one new pod <NUM> are shown, such operations may be performed for any number of pairs of pods in the source cluster and the destination cluster. The control planes <NUM> may include components from the control planes of both the destination cluster and the source cluster, such as those shown in <FIG>. The timing diagram may be performed on a system, such as by one or more processors shown in <FIG> or <FIG>.

Referring to <FIG>, while an old pod <NUM> is still running on a node of a source cluster, cluster control planes <NUM> may schedule a new pod <NUM>. For example, new pod <NUM> may be scheduled by controllers of destination cluster <NUM>. The cluster control planes <NUM> may record the states of the old pod <NUM>, and then transmit these states to the new pod <NUM>. The cluster control planes <NUM> may pause execution of old pod <NUM>. The cluster control planes <NUM> may then copy any changes in states of old pod <NUM>, and transmit these changes to new pod <NUM>. The cluster control planes <NUM> may then resume execution of pod <NUM>.

Once the pod <NUM> starts execution, network traffic, such as requests from applications or websites directed to old pod <NUM>, may be forwarded by the cluster control planes <NUM> to the new pod <NUM>. For example, the allocation may be performed by global load balancers as described with relation to <FIG>. Once workload migration is complete, connection to old pod <NUM> may be closed. The old pod <NUM> may then be deleted. Still further, during the live workload migration, a live migration of workload storage may be performed as shown in <FIG>. For example, the live migration of workload storage may be performed during the live migration of requests to workloads.

As mentioned above, the destination cluster may be monitored during and/or after the live migration for failures. As such, <FIG> shows example further actions that may be taken based on whether a live migration succeeds or fails. As shown, a change from a source environment to a destination environment may be initiated by a cloud platform <NUM> that instructs the coordinator <NUM>. The cloud platform <NUM> may then instruct a cloud control plane <NUM> to start one or more new destination VMs for the migration. If the coordinator <NUM> reports failures during or after migration to the cloud platform <NUM>, the cloud platform <NUM> may instruct the coordinator <NUM> to stop or pause the migration. Additionally, output including information on the detected failures may be generated. For example the information may be displayed to cloud administrators, users, etc..

Alternatively or additionally, the cloud platform <NUM> may instruct the coordinator <NUM> to initiate a change from the destination environment back to the source environment. Once the rollback is complete, cloud platform <NUM> may instruct the cloud control plane <NUM> to delete the destination VMs created for the migration. Error reporting, diagnostics, and fixing may then be performed, for example by administrators of the cloud platform <NUM>. Once the errors are fixed, the cloud platform <NUM> may instruct the coordinator <NUM> to re-initiate the change from the source environment to the destination environment. Importantly, the workloads running on the clusters never experiences more than a very minor interruption even if the migration fails and is rolled back.

Further as shown, in some instances the coordinator <NUM> may report a successful migration. In such cases, if the source VM(s) are on the same cloud as the cloud platform <NUM>, the cloud platform <NUM> may instruct the cloud control plane <NUM> to delete the source VM(s). If the source VM(s) are on a different cloud as the cloud platform <NUM>, the cloud platform <NUM> may not be able to do anything to the source VM(s). In that case, a user may need to instruct the other cloud to delete these source VM(s).

Although <FIG> shows a number of example actions, not all of the actions may need to be performed, and the order may be different. For example, whether to start a complete rollback or merely pause the migration to fix some failures may be based on a determination of the severity of the failure, or whether the failures already existed prior to the migration. Further in that regard, the reporting, diagnosing, and fixing of failures may occur additionally or alternatively after the migration is paused, and the destination VM(s) may not be deleted, but instead remain so that the migration may be resumed once the errors are fixed.

<FIG> is a flow diagram <NUM> that is performed by one or more processors, such as one or more processors <NUM>, <NUM>. For example, processors <NUM>, <NUM> receive data and makes various determinations as shown in the flow diagram. <FIG> shows an example live migration from the control plane of a first cluster to the control plane of a second cluster. Referring to <FIG>, at block <NUM>, requests to one or more cluster control planes are received, wherein the one or more cluster control planes include a control plane of a first cluster and a control plane of a second cluster. At block <NUM>, a predetermined fraction of the received requests are allocated to the control plane of the second cluster, and a remaining fraction of the received requests are allocated to the control plane of the first cluster. At block <NUM>, the predetermined fraction of requests are handled using the control plane of the second cluster. At block <NUM>, while handling the predetermined fraction of requests, it is detected whether there are failures in the second cluster. At block <NUM>, based on not detecting failures in the second cluster, the predetermined fraction of requests allocated to the control plane of the second cluster is increased in predetermined stages until all received requests are allocated to the control plane of the second cluster.

In summary, the technology according to an implementation provides for live migration from a first cluster to a second cluster. For instance, when requests to one or more cluster control planes are received, a predetermined fraction of the received requests may be allocated to a control plane of the second cluster, while a remaining fraction of the received requests may be allocated to a control plane of the first cluster. The predetermined fraction of requests are handled using the control plane of the second cluster. While handling the predetermined fraction of requests, it is detected whether there are failures in the second cluster. Based on not detecting failures in the second cluster, the predetermined fraction of requests allocated to the control plane of the second cluster may be increased in predetermined stages until all requests are allocated to the control plane of the second cluster.

The technology is advantageous because it provides a gradual and monitored rollout process for upgrading clusters, or modifying other aspects of a cluster's environment. The staged and canaried rollout process provides more opportunity to stop the upgrade in case issues arise, therefore preventing large scale damage. Workload traffic allocation between the simultaneously running source and destination clusters may reduce or eliminate downtime during upgrade. Further, due to the workload traffic allocation, from the perspective of the client it may appear as if only one cluster existed during the live migration. In case of a failed upgrade, the system also provides rollback options since the source cluster is not deleted unless a successful upgrade is completed. The technology further provides features to enable live migration between clusters located in different physical locations, as well as between clusters operated on different clouds where one of the clouds does not support live migration.

Claim 1:
A method (<NUM>) for migrating from a first cluster to a second cluster, comprising:
receiving (<NUM>), by one or more processors, requests to two or more cluster control planes, wherein the two or more cluster control planes include a control plane of the first cluster and a control plane of the second cluster, wherein the received requests include requests from a workload running in the first cluster, wherein the requests from the workload are intercepted by a sidecar container injected in the first cluster and routed to cluster bridging aggregators of the second cluster, wherein the sidecar container is configured to intercept application programming interface, API, calls from the workload and is configured to simulate a cluster bridging aggregator being absent in the first cluster, and wherein the cluster bridging aggregators are configured to route incoming requests, such as API calls from user applications and/or from workloads, to cluster control planes of both the first cluster and the second cluster, wherein the first cluster and the second cluster are operated on different clouds;
allocating (<NUM>), by the one or more processors, a predetermined fraction of the received requests to the control plane of the second cluster, and a remaining fraction of the received requests to the control plane of the first cluster;
handling (<NUM>), by the one or more processors, the predetermined fraction of requests using the control plane of the second cluster;
detecting (<NUM>), by the one or more processors, whether there are failures in the second cluster while handling the predetermined fraction of requests; and
increasing (<NUM>), by the one or more processors, based on not detecting failures in the second cluster, the predetermined fraction of requests allocated to the control plane of the second cluster in predetermined stages until all received requests are allocated to the control plane of the second cluster.