Patent Description:
Multi-single-tenant (MST) services run individual service instances in an isolated environment, such as in a separate virtual machine (VM). Cloud SQL is an example of such a service. Since individual service instances run on a single VM, the failure of the VM will result in unavailability of the service instance. Failures may be planned or unplanned, but either requires the VM to be taken down. An example of a planned failure is VM updates, such as critical security patches to the kernel. A control plane of the service may be able to detect VM failures, and failover to a newly created VM. For planned failovers, the unavailability time for the VM may be reduced by re-creating the new VM before taking down the old one. However, both approaches are prone to VM stockouts, where users are unable to create VMs due to lack of capacity of the requested type of machine in the target zone. This is unacceptable as it renders customer's existing service instances unavailable for an indefinite period of time. Correspondingly, <CIT> discloses a system and a method for reserving resources for a virtual machine to use during failure of an underlying node. Said system and said method include a scheduling module that is configured to reserve resources for the virtual machine on at least one candidate node. To reserve the resources, the scheduling module is configured to identify a list of candidate nodes based upon a qualifier function, rank the candidate nodes based upon a priority function, and determine the candidate nodes that satisfy a high availability threshold. The scheduling module is also configured to select a highest-ranked candidate node that satisfies the high availability threshold and reserve the resources on the highest-ranked candidate node.

Some MST services, such as Cloud SQL, provide High Availability (HA) solutions where a hot-standby secondary VM is kept around next to a primary VM all the time to avoid stockouts in case the primary fails, and to be able to do a quick failover in case the primary VM fails or it needs to be updated. Primary and secondary VMs are typically put behind a load balancer endpoint so that the IP address of the instance remains stable throughout failovers. For services that require this active-passive model, the secondary VM is used exclusively during the failover and sits idle otherwise. This is expensive, as it doubles the compute resources associated with a single instance of the service and can increase the service cost as much as nearly double. For this added cost, the secondary VM is typically only used approximately once every <NUM>-<NUM> years for unplanned failures and once every quarter for planned updates.

The present disclosure provides a solution which reduces the cost of maintaining a standby VM in an HA solution without sacrificing availability characteristics, such as uptime, failover time, etc. The secondary pool of VMs, or more generally computing resources, is used to run secondary services or jobs. Upon detection of a failure of a given VM, the secondary services or jobs are evicted from secondary pool resources, so that those secondary pool resources can be automatically allocated to a primary job or service on the failed VM. In this regard, a secondary job may be thought of as a preemptible job and comprises services or jobs that are deserving of a lower priority than the failed service or job. By using computing resources in the secondary pool to run secondary or preemptible jobs, this technology makes use of what would be otherwise idle resources. This beneficially avoids having to allocate additional and separate computing resources for secondary jobs. This leads to more efficient use of network resources and potentially mitigates having to otherwise grow potentially under-utilized networks, which results in reductions in high availability ("HA") costs.

The disclosure provides a method, a system and a non-transitory computer readable medium for managing pooled computing resources used for providing services in a distributed system.

A secondary pool of VMs is used to run secondary services or jobs, which may be evicted upon failure of a corresponding primary VM. For example, upon detection of a failure of a primary resource, the secondary services or jobs are evicted from secondary pool resources, and the secondary pool resources can be automatically allocated to the jobs of the failed primary resource. In this regard, a secondary job may be thought of as a preemptible job and comprises services or jobs that are lower priority than the service or job on the primary resource. By using computing resources in the secondary pool to run secondary or preemptible jobs, this technology makes use of what would be otherwise idle resources. This beneficially avoids having to allocate additional and separate computing resources for secondary jobs. This leads to more efficient use of network resources and potentially mitigates having to otherwise grow potentially under-utilized networks, thereby reducing HA costs.

A sub-aspect of this technology comprises gracefully dialing down and transferring computing resources from a secondary job to the failed or failing job in response to detecting a failover condition. For example, if a preemptible or secondary job is running in a container (e.g., docker, lxc, etc.) using resources in a secondary pool, those resources can be dynamically shifted from the secondary job to the primary or main job. Dynamic shifting or allocation of resources comprises gradually reducing the resources (e.g., CPU, memory, networking, etc.) being used by the secondary job and in concert with that reduction allocate the freed-up or now unused resources to the primary job. The freed-up resources may then run the primary job on the VM, by way of example only, in another docker container. The primary job may fail over to a different VM, given that the VM it was running on was the source of the failure. The different VM is allocated from the secondary VM pool. Where the resource that is being dynamically allocated comprises a CPU, either the number of CPUs or performance of each CPU being used may be throttled to achieve a desired result. The dialing down and reallocation of computing resources may impact the performance of the secondary job, but at least provides an opportunity for it to gracefully shut down, while at the same time avoid significantly impacting the performance of the primary job.

The dialing down and reallocation of computing resources may be set to occur during a pre-determined period, e.g., a so called "grace period. " During this period, as the computing resources used by the secondary jobs are dialed down, they are allocated for use by the primary job. Once the grace period ends, the secondary job is terminated, e.g., the secondary job's container is torn down. During the grace period, the redistribution of computing resources between the secondary job and the primary job in effect comprises an inversely proportional relationship. That is, while computing resources used for a secondary job are dialed or ramped down, the primary job can begin with the resources made available to it and the resources made available to the primary job can be increased as they are freed up by the completion of the secondary job.

There may be cases where the primary job could be started at the time the VM is created and use whatever resources are available at that time (e.g., a single virtual CPU ("vCPU")) and then those resources could be increased during the grace period at failover time. As an example, various primary jobs may be run on secondary VMs in what may be referred to as idle mode, where they don't serve traffic. A modest amount of resources, such as CPU, memory, network, will be allocated for these jobs as most of the resources will typically be allocated for the secondary jobs. In this example, when a VM running a primary job X fails, that primary job can quickly failover to secondary VM and the primary job X would have already been running on the secondary VM and it therefore only needs to be allocated more resources.

As another example, this technology may also be applied in a "bare metal" environment, in which a server acts as single tenant physical server. More generally, "bare metal" relates to the removal of the virtualization technology or any form for containerization from the host. For example, if a job was running on a VM as described above, it would now run on an entire physical machine, and failures that take down the machine would be handled similarly to how failures are handled with respect to a VM. In such an environment, a slot of resources can be reserved on a particular bare metal host. A secondary job can be run in a container created on that slot. In the bare-metal example, applications do not run in VMs, but rather on the physical machine's operating system (OS). So while the secondary jobs are in VMs or containers, the primary workload is running directly on the machine's OS. Therefore, this disclosure also applies to clusters that aren't using a virtualization layer.

In another aspect of this technology, the failover or secondary VM pool may be sized to accommodate worst case failure scenarios. For example, a predetermined portion of the failover or secondary VM pool may be set aside for running secondary or pre-emptive jobs such that failovers that may occur are not impacted. In this regard, a threshold can be identified that indicates the number of failover VMs that must remain free and available for failovers. The threshold may be based on the run time for preemptible or secondary jobs and a typical number of failovers that may occur. As a particular example, if a preemptible job has a minimum and a maximum run time (e.g., must run for at least <NUM> minutes and no more than <NUM> hours), then predictions can be made of the maximum number of preemptible jobs that can be executed without impacting a typical number of failovers that need to occur. As such, preemptible job requests may be satisfied and continue to launch until the maximum threshold for computing resources is consumed and no, or insufficient, computing resources set aside for preemptive jobs are available. At this point, no failover jobs will be launched in the secondary or preemptive pool. In addition, if the number of failovers becomes large enough so that additional failover or secondary pool computing resources are needed, then some portion of the preemptible jobs within the thresholded section of the pool may then be evicted to free up resources to accommodate additional failover requests. In this way, the available thresholded resources will be reduced to some level below the threshold limit.

As an option, preemptible or secondary jobs may be spread across available VMs evenly or may be distributed in series across the available VMs so as to fully consume the resources of given VM before using another VM.

<FIG> illustrates an example system <NUM>, including a distributed system <NUM> configured to run a service <NUM>, such as a software application, in a virtual computing environment <NUM> executing on a pool of primary VM instances 350P. A user device <NUM>, such as a computer, associated with a user <NUM>, such as a customer, communicates via a network <NUM> with the distributed system <NUM> to provide commands <NUM> for deploying, removing, or modifying primary VM instances 350P running in the virtual computing environment <NUM>. The number of primary VM instances 350P in the pool of primary VM instances 350P may dynamically change based on commands <NUM> received from the user device <NUM>. In some examples, the software application <NUM> is associated with a MST service and each primary VM instance 350P is configured to execute a corresponding individual service instance <NUM> (e.g., a single tenant of the MST service) of the software application <NUM>.

The virtual environment <NUM> further includes a pool of secondary VM instances <NUM> running secondary jobs. For example, the secondary jobs may be of lower priority than the service <NUM> running on the primary VM instances 350P, such that disruptions to the secondary jobs are less problematic than disruptions to the primary instances. As discussed further below in connection with <FIG>, in the event that one or more primary VM instances 350P become unavailable, the distributed system <NUM> executes a computing device <NUM> configured to identify one or more of the secondary VM instances <NUM> to run failed over jobs from the unavailable VM instance.

In some examples, the pool of primary VM instances 350P is associated with a single user/customer <NUM> and the pool of secondary VM instances <NUM> are also for use by the single user/customer <NUM>. In other examples, the pool of primary VM instances 350P includes multiple sub-pools of primary VM instances 350P with each sub-pool associated with a different user/customer <NUM> and isolated from the other sub-pools. In these examples, the pool of secondary VM instances <NUM> is shared among the multiple different user/customers <NUM> in events that one or more primary VM instances 350P in any of the sub-pools are unavailable.

In some implementations, the virtual computing environment <NUM> is overlaid on resources <NUM>, 110a-n of the distributed system <NUM>. The resources <NUM> may include hardware resources <NUM> and software resources <NUM>. The hardware resources <NUM> may include computing devices <NUM> (also referred to as data processing devices and data processing hardware) or non-transitory memory <NUM> (also referred to as memory hardware). The software resources <NUM> may include software applications, software services, application programming interfaces (APIs) or the like. The software resources <NUM> may reside in the hardware resources <NUM>. For example, the software resources <NUM> may be stored in the memory hardware <NUM> or the hardware resources <NUM> (e.g., the computing devices <NUM>) may be executing the software resources <NUM>.

The network <NUM> may include various types of networks, such as local area network (LAN), wide area network (WAN), and/or the Internet. Although the network <NUM> may represent a long range network (e.g., Internet or WAN), in some implementations, the network <NUM> includes a shorter range network, such as a local area network (LAN). In some implementations, the network <NUM> uses standard communications technologies and/or protocols. Thus, the network <NUM> can include links using technologies, such as Ethernet, Wireless Fidelity (WiFi) (e.g., <NUM>), worldwide interoperability for microwave access (WiMAX), <NUM>, Long Term Evolution (LTE), digital subscriber line (DSL), asynchronous transfer mode (ATM), InfiniBand, PCI Express Advanced Switching, Bluetooth, Bluetooth Low Energy (BLE), etc. Similarly, the networking protocols used on the network <NUM> can include multiprotocol label switching (MPLS), the transmission control protocol/Internet protocol (TCP/IP), the User Datagram Protocol (UDP), the hypertext transport protocol (HTTP), the simple mail transfer protocol (SMTP), the file transfer protocol (FTP), etc. The data exchanged over the network <NUM> can be represented using technologies and/or formats including the hypertext markup language (HTML), the extensible markup language (XML), etc. In addition, all or some of the links can be encrypted using conventional encryption technologies, such as secure sockets layer (SSL), transport layer security (TLS), virtual private networks (VPNs), Internet Protocol security (IPsec), etc. In other examples, the network <NUM> uses custom and/or dedicated data communications technologies instead of, or in addition to, the ones described above.

As shown in <FIG>, A primary VM instance 350P may become unavailable as a result of an unplanned and unexpected failure, a delay in re-creating the primary VM instance 350P, and/or as a result of a planned maintenance time period for the primary VM instance 350P, such as updates for critical security patches to a kernel of the primary VM instance 350P. When unavailability of one or more primary VM instances 350P is planned or detected, such as during updating or during a failure, the primary instance may automatically fail over to one or more of the secondary VMs <NUM>. For example, the distributed system <NUM> executes a computing device <NUM> configured to identify one or more of the secondary VM instances <NUM> to run jobs from the unavailable VM instance. In this regard, the distributed system maintains availability of the one or more individual service instances <NUM> associated with the unavailable primary VM instances 350P. Because the identified one or more secondary VMs are already running secondary jobs, those jobs may be dialed down and removed in order to free up resources for the jobs from the failed primary VM instance. For example, the secondary services or jobs are evicted from secondary pool resources, and the secondary pool resources can be automatically allocated to the jobs of the failed primary resource. In this regard, a secondary job may be thought of as a preemptible job and comprises services or jobs that are lower priority than the service or job on the primary resource. By using computing resources in the secondary pool to run secondary or preemptible jobs, this technology makes use of what would be otherwise idle resources. This beneficially avoids having to allocate additional and separate computing resources for secondary jobs. This leads to more efficient use of network resources and potentially mitigates having to otherwise grow potentially under-utilized networks, thereby reducing HA costs.

While disclosure above refers to resources as VM instances, it should be understood that other types of resources may be used, such as containers, server slots in a bare metal environment, etc. For example, the arrangement of secondary jobs running on secondary resources may be applied in a "bare metal" environment, in which a server acts as single tenant physical server. More generally, "bare metal" relates to removal virtualization technology or any form for containerization from the host. For example, a job may be run on an entire physical machine, and failures that take down the machine would be handled similarly to how failures are handled with respect to a VM. In such an environment, a slot of resources can be reserved on a particular bare metal host. A secondary job can be run in a container created on that slot. In the bare-metal example, applications do not run in VMs, but rather on the physical machine's operating system (OS). So while the secondary jobs are in VMs or containers, the primary workload is running directly on the machine's OS. Therefore, this disclosure also applies to clusters that aren't using a virtualization layer.

In the example shown in <FIG>, the distributed system <NUM> includes a collection <NUM> of resources <NUM> (e.g., hardware resources <NUM>) executing the virtual computing environment <NUM>. The virtual computing environment <NUM> includes a virtual machine manager (VMM) <NUM> and a virtual machine (VM) layer <NUM> running one or more virtual machines (VMs) <NUM>, 350a-n configured to execute instances 362a, 362a-n of one or more software applications <NUM>. Each hardware resource <NUM> may include one or more physical central processing units (pCPU) <NUM> ("data processing hardware <NUM>") and memory hardware <NUM>. While each hardware resource <NUM> is shown having a single physical processor <NUM>, any hardware resource <NUM> may include multiple physical processors <NUM>. A host operating system (OS) <NUM> may execute on the collection <NUM> of resources <NUM>.

In some examples, the VMM <NUM> corresponds to a hypervisor <NUM> (e.g., a Compute Engine) that includes at least one of software, firmware, or hardware configured to create, instantiate/deploy, and execute the VMs <NUM>. A computer, such as data processing hardware <NUM>, associated with the VMM <NUM> that executes the one or more VMs <NUM> may be referred to as a host machine <NUM>, while each VM <NUM> may be referred to as a guest machine. Here, the VMM <NUM> or hypervisor is configured to provide each VM <NUM> a corresponding guest operating system (OS) <NUM>, 354a-n having a virtual operating platform and manage execution of the corresponding guest OS <NUM> on the VM <NUM>. As used herein, each VM <NUM> may be referred to as an "instance" or a "VM instance". In some examples, multiple instances of a variety of operating systems may share virtualized resources. For instance, a first VM <NUM> of the Linux® operating system, a second VM <NUM> of the Windows® operating system, and a third VM <NUM> of the OS X® operating system may all run on a single physical x86 machine.

The VM layer <NUM> includes one or more virtual machines <NUM>. The distributed system <NUM> enables the user <NUM> to launch VMs <NUM> on demand, i.e., by sending a command <NUM> (<FIG>) to the distributed system <NUM> via the network <NUM>. For instance, the command <NUM> may include an image or snapshot associated with the corresponding operating system <NUM> and the distributed system <NUM> may use the image or snapshot to create a root resource <NUM> for the corresponding VM <NUM>. Here, the image or snapshot within the command <NUM> may include a boot loader, the corresponding operating system <NUM>, and a root file system. In response to receiving the command <NUM>, the distributed system <NUM> may instantiate the corresponding VM <NUM> and automatically start the VM <NUM> upon instantiation. A VM <NUM> emulates a real computer system (e.g., host machine <NUM>) and operates based on the computer architecture and functions of the real computer system or a hypothetical computer system, which may involve specialized hardware, software, or a combination thereof. In some examples, the distributed system <NUM> authorizes and authenticates the user <NUM> before launching the one or more VMs <NUM>. An instance <NUM> of a software application <NUM>, or simply an instance, refers to a VM <NUM> hosted on (executing on) the data processing hardware <NUM> of the distributed system <NUM>.

The host OS <NUM> virtualizes underlying host machine hardware and manages concurrent execution of one or more VM instances <NUM>. For instance, host OS <NUM> may manage VM instances 350a-n and each VM instance <NUM> may include a simulated version of the underlying host machine hardware, or a different computer architecture. The simulated version of the hardware associated with each VM instance <NUM>, 350a-n is referred to as virtual hardware <NUM>, 352a-n. The virtual hardware <NUM> may include one or more virtual central processing units (vCPUs) ("virtual processor") emulating one or more physical processors <NUM> of a host machine <NUM> (<FIG>). The virtual processor may be interchangeably referred to a "computing resource" associated with the VM instance <NUM>. The computing resource may include a target computing resource level required for executing the corresponding individual service instance <NUM>.

The virtual hardware <NUM> may further include virtual memory in communication with the virtual processor and storing guest instructions (e.g., guest software) executable by the virtual processor for performing operations. For instance, the virtual processor may execute instructions from the virtual memory that cause the virtual processor to execute a corresponding individual service instance <NUM> of the software application <NUM>. Here, the individual service instance <NUM> may be referred to as a guest instance that cannot determine if it is being executed by the virtual hardware <NUM> or the physical data processing hardware <NUM>. If a guest service instance <NUM> executing on a corresponding VM instance <NUM>, or the VM instance <NUM> itself, malfunctions or aborts, other VM instances executing corresponding individual service instances <NUM> will not be affected. A host machine's microprocessor(s) can include processor-level mechanisms to enable virtual hardware <NUM> to execute software instances <NUM> of applications <NUM> efficiently by allowing guest software instructions to be executed directly on the host machine's microprocessor without requiring code-rewriting, recompilation, or instruction emulation. The virtual memory may be interchangeably referred to as a "memory resource" associated with the VM instance <NUM>. The memory resource may include a target memory resource level required for executing the corresponding individual service instance <NUM>.

The virtual hardware <NUM> may further include at least one virtual storage device that provides storage capacity for the service on the physical memory hardware <NUM>. The at least one virtual storage device may be referred to as a storage resource associated with the VM instance <NUM>. The storage resource may include a target storage resource level required for executing the corresponding individual service instance <NUM>. The guest software executing on each VM instance <NUM> may further assign network boundaries (e.g., allocate network addresses) through which respective guest software can communicate with other processes reachable through an internal network <NUM> (<FIG>), the external network <NUM> (<FIG>), or both. The network boundaries may be referred to as a network resource associated with the VM instance <NUM>.

The guest OS <NUM> executing on each VM <NUM> includes software that controls the execution of the corresponding individual service instance <NUM>, 362a-n of the application <NUM> by the VM instance <NUM>. The guest OS <NUM>, 354a-n executing on a VM instance <NUM>, 350a-n can be the same or different as the other guest OS <NUM> executing on the other VM instances <NUM>. In some implementations, a VM instance <NUM> does not require a guest OS <NUM> in order to execute the individual service instance <NUM>. The host OS <NUM> may further include virtual memory reserved for a kernel <NUM> of the host OS <NUM>. The kernel <NUM> may include kernel extensions and device drivers, and may perform certain privileged operations that are off limits to processes running in a user process space of the host OS <NUM>. Examples of privileged operations include access to different address spaces, access to special functional processor units in the host machine <NUM> such as memory management units, and so on. A communication process <NUM> running on the host OS <NUM> may provide a portion of VM network communication functionality and may execute in the user process space or a kernel process space associated with the kernel <NUM>.

Referring to <FIG>, in some implementations, a virtual computing environment <NUM> running on the distributed system <NUM> includes multiple host machines <NUM>, 310a-n (e.g., one or more data processing apparatus such as rack mounted servers or different computing devices) that may be located in different physical locations and can have different capabilities and computer architectures. The host machines <NUM> may communicate with each other through an internal data communications network <NUM> (internal network). The internal network <NUM> can include one or more wired (e.g., Ethernet) or wireless (e.g., Wi-Fi) networks, for example. In some implementations, the internal network <NUM> is an intranet. Optionally, the host machines <NUM> may also communicate with devices on the external network <NUM>, such as the Internet. Other types of external networks are possible.

In the example shown, each host machine <NUM> executes a corresponding host operating system (OS) <NUM>, 312a-n that virtualizes the underlying hardware (i.e., data processing hardware <NUM> and memory hardware <NUM>) of the host machine <NUM> and manages concurrent execution of multiple VM instances <NUM>. For instance, host operating systems 312a-312n-<NUM> each manage concurrent execution of multiple primary VM instances 350P to collectively provide the pool of primary VMs 350P, while host operating system 312n executing on host machine 310n manages execution of the pool of secondary VM instances <NUM>. Here, a dedicated host machine (e.g., host machine 310n) hosts the entire pool of secondary VM instances <NUM>, thereby ensuring that sufficient resources are available for use by the secondary VM instances <NUM> in the event of a failover (without requiring the failover secondary VM instances <NUM> to migrate to a different host machine <NUM> with sufficient resources). In other examples, however, one or more of the secondary VM instances <NUM> may be instantiated across multiple host machines <NUM> that may also be executing one or more primary VM instances 350P.

In some implementations, the virtual machine manager <NUM> uses a primary VM manager <NUM> to create and deploy each primary VM instance 350P in the pool of primary VM instances <NUM> for execution on a designated host machine <NUM>. The VMM <NUM> may create each primary VM instance <NUM> by allocating computing resource levels, memory resource levels, network specifications, and/or storage resource levels required for executing the corresponding individual service instance <NUM>. Thus, each primary VM instance 350P in the pool of primary VM instances 350P may include a corresponding VM type <NUM> that indicates at least one of memory resource requirements, computing resource requirements, network specification requirements, or storage resource requirements for the corresponding primary VM instance <NUM>. In the example shown, all the primary VM instances 350P in the pool of primary VM instances 350P have VM type <NUM> of Type A or of Type B. Thus, a VM type <NUM> of Type A may include at least one of computing resource levels, memory resource levels, a network specification, or storage resource levels that are different than a VM type <NUM> of Type B.

The primary VM manager <NUM> at the VMM <NUM> may maintain an active log of each VM instance 350P deployed into the pool of primary VM instances 350P, the VM type <NUM> of each VM instance 350P, and the corresponding individual service instance <NUM> executing on each primary VM instance 350P. The log may be updated as primary VM instances 350P are deployed into, or removed from, the pool of primary VM instances 350P. Additionally, the pool of primary VM instances 350P may be further divided into sub-pools based on a distribution of the primary VM instances 350P in various fault domains, such as building, zone, or region. In some implementations, the individual service instances <NUM> each execute in a corresponding container that runs on a single primary VM instance 350P with multiple other containers. Accordingly, the log may indicate a list of containers running on each primary VM instance 350P, as well as the corresponding service instance <NUM> executing in each container.

The primary VM manager <NUM> further obtains the rate of unavailability for each primary VM instance 350P. For example, each primary VM instance 350P may include a corresponding mean-time-to-failure (MTTF) indicating how long (e.g., a number of days) the primary VM instance 350P is expected to be operational before incurring a failure. The MTTF value could be <NUM> days (e.g., <NUM> year) or <NUM> days (e.g., <NUM> years). The rate of unavailability for each primary VM instance 350P may further include an expected length of time to re-create (e.g., stock-out value) the corresponding primary VM instance. For example, a VM instance <NUM> may be associated with a stock-out value while the distributed system <NUM> waits for resources (i.e., processing resources and/or memory resources) become available for re-creating the VM instance <NUM>. The MTTF and the expected length of time to re-create each primary VM instance 350P can be obtained through statistical analysis and/or machine learning techniques by observing execution of VM instances <NUM> having a same or similar VM type (i.e., processing resources, memory resources, storage resources, network configuration).

In some examples, all of the primary VM instances 350P in the pool of primary VM instances 350P include the same rate of unavailability. In other examples, the primary VM instances 350P associated with the type A VM type <NUM> include a rate of unavailability that is different than a rate of unavailability for the primary VM instances 350P associated with the type B VM type <NUM>. As set forth in the remarks above, each primary VM instance 350P may include the corresponding MTTF value, indicating how long (e.g., a number of days) the primary VM instance 350P is expected to be operational before incurring a failure, and the stock-out value, indicating an expected length of time to re-create the primary VM instance 350P. The MTTF value and the stock-out value may be derived from observed monitoring data as well as machine learning algorithms that observe execution of similar VM instances <NUM> over time.

The VMM <NUM> may further maintain a service instance repository <NUM> indicating each individual service instance <NUM> of the software application <NUM> executing on a corresponding primary VM instance 350P of the pool of primary VM instances 350P and the target resource levels required for executing the corresponding individual service instance <NUM>. The VMM <NUM> may further maintain in the service instance repository <NUM> an indication of the secondary jobs being run in each secondary VM instance <NUM>. In this regard, the VMM <NUM> may determine, upon failure of one or more of the primary VMs 350P, which jobs to evict from the secondary VMs <NUM> in order to run the instances from failed primary VM on the secondary VMs <NUM>. Such determination may be based on, for example, a priority value associated with the secondary jobs, an amount of space needed to run the primary instance failed over from the failed primary VM, an amount of time needed to evacuate particular secondary jobs, an amount of time the failed primary VM is expected to be unavailable, the VM type (e.g., A or B), or any of a variety of other factors.

In some examples, the VMM <NUM> includes a maintenance scheduler <NUM> that identifies maintenance time periods when one or more primary VM instances 350P in the pool of primary VM instances 350P will be unavailable for maintenance/updates performed off-line. For instance, the maintenance scheduler <NUM> may indicate a number of primary VM instances 350P that will be unavailable during a planned maintenance time period to perform maintenance/updates. In one example, the distributed system <NUM> periodically rolls out a kernel update at a two-percent (<NUM>%) deployment rate (or other percentage/value) such that two-percent of primary VM instances 350P in the pool of primary VM instances 350P will be unavailable during the planned maintenance time period to complete the update. A kernel update may include fixing security patches in a kernel <NUM> associated with the VM instance <NUM>. In some examples, the VMM <NUM> receives a planned failover message <NUM> from a computing device <NUM> that indicates the number (or percentage) of primary VM instances 350P that will be unavailable during a planned maintenance time period to perform maintenance/updates. The computing device <NUM> may belong to an administrator of the distributed system <NUM>. Optionally, the user device <NUM> may provide the planned failover message <NUM> via the external network <NUM> when the user <NUM> wants to update one or more primary VM instances 350P in the pool of primary VM instances 350P.

In some implementations, the pool of secondary VM instances <NUM> is per customer/user <NUM>, rather than global, when the customer/user <NUM> deploys a large number of primary VM instances 350P and has specific networking or isolation requirements that prevents sharing of the pool of secondary VM instances <NUM> with other users/customers of the distributed system <NUM>. In other implementations, the pool of secondary VM instances <NUM> is shared among all individual service instances <NUM> across all customers/users of the distributed system <NUM>.

In some examples, the VMM <NUM> includes a secondary VM manager <NUM>, which may determine how to allocate failed over instances from the primary VMs 350P to the secondary VMs <NUM>. For example, the secondary VM manager <NUM> may determine which jobs should be evicted from which secondary VMs <NUM> in order to make room for the failed over primary VM instances. As mentioned above, such determination may be based on, for example, a priority value associated with the secondary jobs, an amount of space needed to run the primary instance failed over from the failed primary VM, an amount of time needed to evacuate particular secondary jobs, an amount of time the failed primary VM is expected to be unavailable, the VM type (e.g., A or B), or any of a variety of other factors. The evicted secondary jobs may be temporarily ceased until the secondary pool resources are no longer needed by the failed over primary instances. For example, once the failed VM is back up and running the primary instance, the secondary VM may resume the previously evicted secondary jobs. In other examples, the secondary jobs may be gracefully shutdown and then killed after the grace period. For example, a job may receive a notification from the system that it is going to be shutdown in X seconds, and may choose what it means for it to do graceful shutdown. For example, some applications may choose to flush state to a stable storage.

According to some implementations, evicting the secondary jobs includes gracefully dialing down and transferring computing resources from the secondary job to the failed or failing job in response to detecting a failover condition. For example, if a preemptible or secondary job is running in one of the secondary VMs <NUM>, the resources assigned to that secondary VM can be dynamically shifted from the secondary job to the primary or main job. Dynamic shifting or allocation of resources may include gradually reducing the resources being used by the secondary job while allocating the freed-up resources to the primary job. For example, the secondary resources may be allocated as they are freed. The freed-up resources may then run the primary job. Where the resource that is being dynamically allocated comprises a CPU, either the number of CPUs or performance of each CPU being used may be throttled to achieve a desired result. The dialing down and reallocation of computing resources may impact the performance of the secondary job, but at least provides an opportunity for it to gracefully shut down, while at the same avoid significantly impacting the performance of the primary job.

There are a number of different ways in which secondary jobs can gracefully ramp down and use less resources in proportion to primary failover jobs ramping up their resources. For example, an amount of input/output (I/O) for containers can be dynamically tuned. As such, secondary jobs may be allotted a predetermined amount of time, such as <NUM>, <NUM>, <NUM> minutes, etc., to wind down before being killed. The predetermined amount of time may be the same for all secondary jobs, or it may different based on the type of job, the resources allocated to the secondary job, an urgency of the failed primary job, or any other factors. During that predetermined amount of time, the primary job fails over to the resources that are freed by the secondary job. As another example, resources assigned to the secondary job may be decreased incrementally, such as every <NUM>, <NUM>, <NUM>, or other interval of time. As these resources are incrementally freed, they may be allocated to the primary failover job that is ramping up. While these are a couple examples of how dynamic resource allocation may be performed, it should be understood that a number of other ways are possible.

The way in which dynamic resource allocation is performed may be based on, for example, the type of preemptible secondary job that is ramping down. For example, a secondary job that requires a lot of I/O can preliminarily reduce its CPU usage while maintaining a similar amount of I/O, and then later reduce its I/O. As another example, the dynamic resource allocation may be based on processing state of the secondary job at the time of failure of the primary job. For example, if the secondary job is only doing a small amount of computation, such as less than a particular predefined threshold, the job can be killed in a relatively short period of time. For example, the time allotted to the secondary job to ramp down before it is killed may be proportional to an amount of computation being performed by the secondary job.

According to some examples, rather than killing a secondary job after the graceful ramping down, the secondary job may remain running using less resources. For example, secondary jobs may specify a range of resources needed, such as a minimum number of CPUs, amount of memory, I/O, etc. on a lower end of the range, and a maximum desired number of resources on the higher end of the range. While the secondary job may be running using an amount of resources closer to the higher end of the range, when a failover of a primary instance occurs, the secondary job may ramp down. For example, some of the allocated resources may be freed for running the failover job, such that the secondary job is then running using an amount of resources closer to the lower end of the range.

There may be cases where the primary job could be started at the time the VM is created and use whatever resources are available at that time (e.g., a single virtual CPU ("vCPU")) and then those resources could be increased during the grace period at failover time. As an example, various primary jobs may be run on secondary VMs in what may be referred to as idle mode, where they don't serve traffic. A modest amount of resources, such as CPU, memory, network, may be allocated for these jobs as most of the resources will typically be allocated for the secondary jobs. In this example, when a VM running a primary job X fails, that primary job can quickly failover to the secondary VM, because it would not need to be moved. Rather, the idle mode of the primary job X would begin running.

According to some examples, the secondary VM manager <NUM> may include a plurality of separate managers that control different aspects of the secondary jobs. By way of example only, the secondary VM manager <NUM> may include a cluster resource manager that controls information on the secondary resources. Further, the secondary VM manager <NUM> may include a failover transition manager. When failure occurs, the failure transition manager may determine which node to failover to and orchestrate the transition. The secondary VM manager <NUM> may further include a preemptible job scheduler. For example, when a request to run a preemptible job is received, the preemptible job scheduler may figure out which resources are available in the secondary pool and allocated such resources to the request. While these other managers are described as possible sub-managers of the secondary VM manager <NUM>, it should be understood that some or all of these other managers may be functions that are handled by the secondary VM manager <NUM> or by other managers, such as the primary CM manager <NUM>, the maintenance scheduler <NUM>, etc. In other examples, some or all of these other managers may be additional mangers within the VM manager <NUM>.

In some scenarios, the VMM <NUM> (or a host machine <NUM>) identifies unavailability of one of the primary VM instances 350P in the pool of primary VM instances 350P. For example, each primary VM instance 350P may employ an agent to collect an operational status <NUM> indicating whether or not the primary VM instance 350P is operating or is unavailable due to a failure. The host machines <NUM> may communicate the operational status <NUM> of VM instances <NUM> to the VMM <NUM>, in addition to one another. As used herein, the term "agent" is a broad term, encompassing its plain and ordinary meaning, including, but not limited to, a portion of code that is deployed inside a VM instance <NUM> (as part of the guest OS <NUM> and/or as an application running on the guest OS <NUM>) to identify the operational status <NUM> of the VM instance <NUM>. Accordingly, the VMM <NUM> and/or the host machine <NUM> may receive the operational status <NUM> indicating unavailability of one of the primary VM instances <NUM>, and cause the unavailable primary VM instance 350P to fail over to one of the secondary VM instances <NUM> to commence executing the individual service instance <NUM> associated with the unavailable primary VM instance 350P. In the example shown in <FIG>, the operational status <NUM> indicates unavailability (e.g., due to failure) of one of the primary VM instances 350P executing on host machine 310n-<NUM> and having VM type <NUM> of Type B, thereby causing the primary VM instance 350P to fail over to a secondary VM instance <NUM> having the VM type <NUM> of Type B' to commence executing the individual service instance <NUM> associated with the unavailable primary VM instance 350P having the VM type <NUM> of Type B.

The VM manager <NUM> may select which secondary jobs to pre-empt. Such selection may be performed in any of a variety of ways, based on any of a variety of factors. According to one example, the VM manger <NUM> may identify the node, such as the VM or bare metal, in the secondary pool that has the most resources available to it. For example, it may identify the node having the most CPU, the most memory, the most I/O, or any combination of these. In some examples, machine learning techniques may be used to predict which node will have the most available resources at a given point in time. For example, failures of primary instances may be tracked over time, along with the secondary jobs that were selected for pre-emption to free resources for the failover. Such information may be used to create a model for predicting when a particular primary instance will fail. The model may further predict which secondary job to pre-empt at that time, such as by determining which secondary nodes will have the most available resources at that time.

According to other examples, the secondary jobs to pre-empt may be selected based on location. For example, a global cloud provider may have multiple datacenters in different cities or countries around the world. A node in physical proximity to the failed node may be selected for handling the failover jobs, such that the failover jobs can be sent over a short physical distance and therefore may ramp up on the selected node more quickly.

According to yet another example, selection of which secondary jobs to pre-empt may be based on a type of application for the failover job and the type of hardware resources needed for that application. For example, some applications may require graphics processing units (GPUs), accelerators, etc. Accordingly, nodes having such hardware resources required by the failover application may be selected, and the secondary jobs running on such nodes may be pre-empted.

A software application (i.e., a software resource <NUM>) may refer to computer software that causes a computing device to perform a task.

Non-transitory memory (e.g., memory hardware <NUM>) may be physical devices used to store programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by a computing device (e.g., data processing hardware <NUM>). The non-transitory memory <NUM> may be volatile and/or non-volatile addressable semiconductor memory.

The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the inventions described and/or claimed in this document.

The non-transitory memory <NUM> may be physical devices used to store programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by the computing device.

In some implementations, the storage device <NUM> is a computer- readable medium.

By way of example only, the high speed controller <NUM> manages bandwidth-intensive operations for the computing device <NUM>, while the low speed controller <NUM> manages lower bandwidth-intensive operations.

As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non- transitory computer readable medium, apparatus and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal.

A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the scope of the disclosure. Accordingly, other implementations are within the scope of the following claims.

<FIG> illustrates an example method <NUM> of running secondary jobs on secondary resources, where such resources are freed when needed to run a primary instance.

In block <NUM>, a primary pool of computing resources is designated for running one or more primary jobs. The computing resources may include, for example, virtual machine, containers, servers, etc. The primary jobs may be related to a service performed for customers, for example, such as a MST service.

In block <NUM>, a secondary pool of computing resources is designated for running one or more secondary jobs. The secondary jobs are different than the primary jobs, and lower priority than the primary jobs. For example, the secondary jobs may be assigned a lower priority value, and may relate to preemptible tasks where an interruption would be less detrimental than an interruption to the primary jobs. The secondary pool of computing resources may be the same type of resources as the primary pool, or it may be different. For example, the secondary resources may have different computing capacities, network capacities, etc. While in some examples the same number of secondary resources may be designated as a number of primary resources, in other examples the number of secondary resources may be different.

In block <NUM>, a failure condition associated with a given one of the one or more primary jobs is detected. For example, the failure condition may be a planned outage, such as for a system update, or an unexpected failure.

In block <NUM>, a given one of the one or more secondary jobs is preempted upon detection of the failure condition. For example, the secondary job may be paused and evicted from the secondary resources. Such secondary resources at this point have freed capacity for handling the one or more primary jobs.

In block <NUM>, the primary jobs associated with the failure condition are allocated to the secondary resources. In this regard, the service of the primary jobs may be continued with minimal interruption. At the same time, however, backup resources are not idle.

According to some examples, the evicted secondary job may be restored to the secondary resource when the primary resource is restored and the primary jobs are restored on the primary resource. According to other examples, the secondary job may be terminated after a period of time.

Claim 1:
A method for managing pooled computing resources (<NUM>, 110a-n) used for providing services in a distributed system (<NUM>), wherein the computing resources (<NUM>, 110a-n) are arranged to execute an virtual computing environment (<NUM>), which is arranged to run a primary pool (350P) of virtual machine, VM, instances and a secondary pool (<NUM>) of VM instances, comprising:
running an individual service instance (<NUM>) in a VM within the primary pool (350P) of VM instances;
running one or more secondary jobs different than the individual service instance on the secondary pool (<NUM>) of VM instances;
detecting, by a VM manager (<NUM>) of the distributed system, a failure condition associated with the VM within the primary pool (350P) of VM instances;
selecting, by the VM manager of the distributed system, one or more of the secondary jobs to preempt, wherein the one or more secondary jobs comprise preemptible jobs each having a lower priority than the individual service instance; and
freeing, by the VM manager of the distributed system, computing resources of the VM instances in the secondary pool by reducing computing resources allocated to the selected secondary jobs in the VM instances in the secondary pool, while allocating the freed resources to the individual service instance of the VM, which is within the primary pool (350P) of VM instances and which is associated with the failure condition, wherein the freeing is performed in proportion to the allocating and at a same time.