Patent Description:
Cloud computing refers to highly scalable networked computing systems capable of delivering elastic computing performance to numerous users. Cloud computing typically involves clusters of densely packed computing servers, called nodes, with each node potentially executing dozens of virtual machines. Typically, each node includes a hypervisor or other virtualization framework, and the entire cloud computing cluster includes one or more cloud controllers that manage instantiation of virtual machines on the particular compute nodes. OpenStack is one example of such a cloud computing framework.

In a multi-tenant cloud computing environment, different customers may control the virtual machines on a particular node. Thus, the resources of the node, such as processor, network, and storage resources, must be shared among the virtual machines and thus among different customers. When a virtual machine is created, the user selects an instance type that specifies the resource requirements of the virtual machine. Static resource requirements include a number of virtual central processing units (vCPUs), memory, disk and network.

Such requirements for a cloud service provider (CSP) has two challenges. The first challenge is that the CSP must provide the promised resources to the virtual machine instance at the same performance in any server system, regardless of the CPU and other hardware components. Thus, the CSP must define a performance metric and ensure that each virtual machine meets the performance. However, this requirement has often not been met consistently. The second challenge is that there should be a maximum use of the provisioned infrastructure. Thus, the CSP may often wish to overprovision CPU and memory to a limit that maximizes use of infrastructure and minimizes difference in performance degradation.

Existing overprovisioning solutions suggest using an optimum ratio. However when implementing such solutions performance degradation occurs when overprovisioning in instances where a virtual machine is assigned four or more vCPUs. For example if a first virtual machine (VM1) is allocated <NUM> vCPUs, one of which is assigned on a first physical CPU (pCPU1) and a second virtual machine (VM2) is assigned <NUM> vCPU which is also on pCPU1, then VM1 will have to wait for its vCPU to be free as it was shared by VM2. If VM1's <NUM> other vCPUs are shared with other VMs, VM1 would have to wait much longer for all its vCPUs to be free. This results in performance delays for VM1.

Such degradation is currently being remedied by not overprovisioning virtual vCPUs (e.g., starting with a single vCPU and scaling out when is necessary); monitoring workload usage, CPU ready and CPU utilization metrics and re-sizing virtual machines; and migrating a virtual machine to a server that has relatively free resources, or has not been overprovisioned.

However, not overprovisioning is a generic recommendation that does not guarantee a performance Service Level Agreement (SLA) that is met on every single server system every single unit of time. Further, monitoring and re-sizing virtual machines requires a virtual machine to start slow with less vCPUs and add more, thus requiring infrastructure to be free or not fully utilized always. Moreover, migrating a virtual machine may result in the same performance delay once a destination server has full allocated capacity.

In <CIT> allocation of physical resources is described by accessing consumption data for each of a plurality of application components executing in one or more virtual machines and consuming a plurality of allocated physical resources. The consumption data is indicative of consumption levels by each of the plurality of application components of each of the plurality of physical resources. Following a determination that a value for a performance metric associated with the application has crossed an associated threshold value, the consumption data is analyzed to identify a consumption level of a first of the plurality of physical resources being consumed by a first of the plurality of application components has deviated from a historical trend for that physical resource. An instruction is then communicated that when executed will cause a change in an allocation level of the first of the plurality of physical resources.

In <CIT> embodiments are described that provide for performance monitoring of virtualized environments by establishing external performance monitoring (in a primary domain) of a virtual machine manager in whose context a virtual machine operates, and simultaneously capturing information on the virtual machine execution states. The execution states may comprise any combination of a virtualized processor index, execution address, program (task) identifier, and a timestamp. A primary domain performance monitoring component may initiate time- or event-based profiling of the virtual machine, and a hypervisor may report the virtualized execution states to the performance monitoring component upon reception of each profiling interrupt. Alternatively, the time-based profiling may be initiated in the virtual machine domain. In this case, the hypervisor or virtual machine manager may enable access from within the virtual machine to performance characteristics collected in the primary domain, or communicate the execution states to the external monitoring component upon reception of each profiling interrupt or upon a change in the virtualized execution states. Performance information collected in the primary domain may then be correlated with the execution states of the virtual machine.

From <CIT> system and method and computer program product implemented for a cloud computing infrastructure are known that allows a hypervisor to optimize resource allocation in a cloud computing environment by exploiting the application-level performance, O/S system performance, and hypervisor performance information through a reliable and efficient channel.

In <CIT> a method, system and computer program product for managing resource utilization of virtual machines in a cloud computing environment are described. A cloud controller computes an index of the utilization of a resource by a virtual machine to determine its adverse impact on its neighboring virtual machine(s) that share the same resource. If the interference index is greater than a threshold, then the utilization of the resource by that virtual machine will be throttled or reduced provided that the servicing of its workload is not weighted at too high of a priority and that one or more of its neighboring virtual machines are not able to meet their service-level agreement requirements. ; In this manner, the adversely affected neighboring virtual machines may have its performance restored in meeting their service-level agreement requirements without having to add additional resources and/or be migrated to other areas of the cloud computing environment.

<CIT> relates to a method of allocating resources in a computer system. The method includes accessing information identifying service requirements between a service provider and each of a plurality of users and allocating processing resources to each of the plurality of users. In response to a change in the service requirements of the service level agreement for at least one of the users over the specified period of time, the allocation of the processing resources is modified. This may include over-provisioning based on the obtained service requirements. The service requirements are enforced accordingly.

Further embodiments are set forth in the dependent claims. Any references to inventions or embodiments not falling within the scope of the independent claims are to be interpreted as examples useful for understanding the invention.

Embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like reference numerals refer to similar elements.

A first aspect of the present invention relates to a compute node of a cloud computing cluster. The compute node comprises one or more processors configured to implement a plurality of virtual machines. Each of the one or more processors has at least one central processing unit, CPU, core. Each virtual machine is constrained to consume resources within one of a plurality of subsets of CPU cores of the one or more processors of the compute node.

The one or more processors are configured to establish an operating environment. The operating environment includes a node agent in communication with a cloud controller for controlling scheduling any one or more of the plurality of virtual machines. The node agent is configured to.

The overprovisioning process to allocate resources is performed for all virtual machines scheduled to the specified subset of CPU cores and comprises logically partitioning central processing unit, CPU, resources per CPU core included in the specified subset of CPU cores of the compute node. The operating environment established by the one or more processors further includes a VM monitor of the virtual machine to monitor performance data of the virtual machine corresponding to the mapped SLO parameters. The node agent receives performance data of the virtual machine from the VM monitor and dynamically modifies resource allocation within the specified subset of CPU cores to achieve the expected performance of the virtual machine.

A second aspect of the present invention relates to a method for managing virtual machine performance at a compute node. The method comprises:.

A third aspect of the present invention relates to a computer readable medium having instructions, which when executed by a processor, cause the processor to perform the methods of the second aspect.

The present invention may be embodied in systems, apparatuses, and methods for workload scheduling in a cloud computing environment, as described below. In the description, numerous specific details, such as component and system configurations, may be set forth in order to provide a more thorough understanding of the present invention. In other instances, well-known structures, circuits, and the like have not been shown in detail, to avoid unnecessarily obscuring the present invention.

According to one embodiment, server resource utilization is maximized, while ensuring that the SLA is always honored. In such an embodiment, a service level objective (SLO) is defined based on processor performance characteristics. In a further embodiment, virtual machine instance types are defined based on SLO parameters in order to capture the resource requirements of the virtual machine. In still a further embodiment, the virtual machine is scheduled on a node and managed according to the user-defined SLO parameters and the current server state. Still further, the SLO parameters are stored and transmitted to various server nodes.

According to one embodiment, the virtual machine SLO parameters are translated to platform resources (e.g., compute time, processor frequency, cache, memory/disk/network bandwidth, memory size). In other embodiments, the virtual machine is continuously monitored to detect SLO violations based on the translated parameters. In such embodiments, dynamic adjustments are made in the form of corrective actions that are applied to restore performance that complies with the SLO parameters.

Elements of embodiments of the invention may be implemented in hardware, software, firmware, or any combination of hardware, software, or firmware. The term hardware generally refers to an element having a physical structure such as electronic, electromagnetic, optical, electro-optical, mechanical, electro-mechanical parts, etc. The term software generally refers to a logical structure, a method, a procedure, a program, a routine, a process, an algorithm, a formula, an expression, etc. The term firmware generally refers to a logical structure, a method, a procedure, a program, a routine, a process, an algorithm, a formula, or an expression that is implemented or embodied in a hardware structure (e.g., flash memory or read only memory). Examples of firmware are microcode, writable control store, and micro-programmed structure.

<FIG> illustrates one embodiment of an information processing system <NUM>. System <NUM> includes a number of compute nodes <NUM>, a cloud controller <NUM>, and a cloud state database <NUM>, all in communication over a network <NUM>. In use, as discussed in more detail below, each compute node <NUM> instantiates one or more virtual machines to perform computational tasks. Each compute node <NUM> monitors its performance, including cache contention, and generates contention and capacity scores. The contention scores for each compute node <NUM> are stored in the cloud state database <NUM>. The cloud controller <NUM> monitors the contention scores and schedules new virtual machines based on the reported contention scores of each of the compute nodes <NUM>.

In one embodiment, each compute node <NUM> may be embodied as any type of computation or computer device capable of performing the functions described herein, including, without limitation, a computer, a multiprocessor system, a server, a rack-mounted server, a blade server, a laptop computer, a notebook computer, a network appliance, a web appliance, a distributed computing system, a processor-based system, and/or a consumer electronic device. Each compute node <NUM> illustratively includes two processors <NUM>, an input/output subsystem <NUM>, a memory <NUM>, a data storage device <NUM>, and communication circuitry <NUM>. Of course, the compute node <NUM> may include other or additional components, such as those commonly found in a server device (e.g., various input/output devices), in other embodiments. Additionally, in some embodiments, one or more of the illustrative components may be incorporated in, or otherwise form a portion of, another component. For example, memory <NUM>, or portions thereof, may be incorporated in one or more processor <NUM> in some embodiments.

Each processor <NUM> may be embodied as any type of processor capable of performing the functions described herein. Each illustrative processor <NUM> is a multi-core processor, however in other embodiments each processor <NUM> may be embodied as a single or multi-core processor(s), digital signal processor, microcontroller, or other processor or processing/controlling circuit. Each processor <NUM> illustratively includes four processor cores <NUM> and an uncore <NUM>. Each of the processor cores <NUM> is an independent processing unit capable of executing programmed instructions.

Each processor core <NUM> includes a performance monitoring unit ("PMU") <NUM>. Each PMU <NUM> may be embodied as a number of performance counters capable of recording and monitoring the flow of instructions through the respective processor core <NUM>. For example, each PMU <NUM> may be capable of counting clock cycles, instructions issued, instructions retired, cache misses, or similar events. The PMUs <NUM> may be programmed to monitor particular performance statistics using model-<NUM> specific registers of the processor core <NUM>. In one embodiment, each PMU <NUM> may include four fully programmable hardware counters and three fixed-function hardware counters. Software may access the PMUs <NUM> using a kernel interface such as the "perf" subsystem of the Linux kernel. Although each of the illustrative compute nodes <NUM> includes two processors <NUM> having four sockets of <NUM> processor cores <NUM>; each compute node <NUM> may include one, two, or more processors <NUM> having one, two, or more processor cores <NUM>.

In particular, this disclosure is also applicable to uniprocessor or single-core compute nodes <NUM>.

Each processor <NUM> also includes an uncore <NUM>. In the illustrative embodiment, each uncore <NUM> includes any part of the particular processor <NUM> not included in the processor cores <NUM> (e.g., all components of the particular processor <NUM> except for the processor cores <NUM> themselves). For example, the uncore <NUM> of each illustrative processor <NUM> includes a PMU <NUM> and cache memory <NUM>. Similar to the PMUs <NUM> of the processor cores <NUM>, the PMU <NUM> monitors performance statistics of the uncore <NUM>, and may include a number of programmable or fixed-function hardware performance counters. The cache memory <NUM> may be a last-level cache shared by the processor cores <NUM>. In some embodiments, the PMU <NUM> may monitor accesses to the cache memory <NUM>, including recording cache misses, amounts of data transferred, and other cache information. Although not illustrated, the uncore <NUM> may additionally include typical components of a processor or a system-on-a-chip. For example, each uncore <NUM> may include a memory controller, processor graphics, input/output controllers, power management circuitry, or other components of the processor <NUM>.

The memory <NUM> may be embodied as any type of volatile or non- volatile memory or data storage capable of performing the functions described herein. In operation, the memory <NUM> may store various data and software used during operation of the compute node <NUM> such as operating systems, applications, programs, libraries, and drivers. The memory <NUM> is communicatively coupled to the processor <NUM> via the I/O subsystem <NUM>, which may be embodied as circuitry and/or components to facilitate input/output operations with the processor <NUM>, the memory <NUM>, and other components of the compute node <NUM>. For example, the I/O subsystem <NUM> may be embodied as, or otherwise include, memory controller hubs, input/output control hubs, firmware devices, communication links (i.e., point-to-point links, bus links, wires, cables, light guides, printed circuit board traces, etc.) and/or other components and subsystems to facilitate the input/output operations. In some embodiments, the I/O subsystem <NUM> may form a portion of a system-on-a-chip (SoC) and be incorporated, along with the processor <NUM>, the memory <NUM>, and other components of the compute node <NUM>, on a single integrated <NUM> circuit chip.

The data storage device <NUM> may be embodied as any type of device or devices configured for short-term or long-term storage of data such as, for example, memory devices and circuits, memory cards, hard disk drives, solid-state drives, or other data storage devices. The data storage device <NUM> may store performance statistics monitored by the compute node <NUM>. The communication circuitry <NUM> of the compute node <NUM> may be embodied as any communication circuit, device, or collection thereof, capable of enabling communications between the compute node <NUM>, the cloud controller <NUM>, the cloud state database <NUM>, and/or other remote devices over the network <NUM>. The communication circuitry <NUM> may be configured to use any one or more communication technology (e.g., wired or wireless communications) and associated protocols (e.g., Ethernet, Bluetooth®, Wi-Fi®, WiMAX, etc.) to effect such communication.

In one embodiment, cloud controller <NUM> manages virtual machines or other compute instances distributed among the compute nodes <NUM> of the system <NUM>. Cloud controller <NUM> may be embodied as any type of server computing device, or collection of devices, capable of performing the functions described herein. As such, cloud controller <NUM> may be embodied as a single server computing device or a collection of servers and associated devices. For example, in some embodiments, cloud controller <NUM> may be embodied as a "virtual server" formed from multiple computing devices distributed across network <NUM> and operating in a public or private cloud. Accordingly, although cloud controller <NUM> is illustrated in <FIG> as embodied as a single server computing device, it should be appreciated that cloud controller <NUM> may be embodied as multiple devices cooperating together to facilitate the functionality described below. Illustratively, cloud controller <NUM> includes a processor <NUM>, an I/O subsystem <NUM>, a memory <NUM>, a data storage device <NUM>, communication circuitry <NUM>, and/or other components and devices commonly found in a server or similar computing device. Those individual components of cloud controller <NUM> may be similar to the corresponding components of compute nodes <NUM>, the description of which is applicable to the corresponding components of the cloud controller <NUM> and is not repeated herein so as not to obscure the present disclosure.

Cloud state database <NUM> stores information that is synchronized across system <NUM>, including performance statistics. Cloud state database <NUM> may be embodied as a dedicated database server, distributed data storage, or any other data storage system capable of maintaining consistent state for the system <NUM>. As such, copies or portions of cloud state database <NUM> may be stored in data storage <NUM> of each compute node <NUM> and/or the data storage <NUM> of cloud controller <NUM>. Updated cloud state information may be transferred between compute nodes <NUM>, cloud controller <NUM>, and/or the cloud state database <NUM> using any communication protocol. In some embodiments, cloud state information may be transferred asynchronously using a message bus, for example a message bus implementing the advanced message queuing protocol (AMQP), such as rabbitmq.

As discussed in more detail below, compute nodes <NUM>, cloud controller <NUM>, and cloud state database <NUM> may be configured to transmit and receive data with each other and/or other devices of the system <NUM> over the network <NUM>. The network <NUM> may be embodied as any number of various wired and/or wireless networks. For example, the network <NUM> may be embodied as, or otherwise include, a wired or wireless local area network (LAN), a wired or wireless wide area network (WAN), a cellular network, and/or a publicly-accessible, global network such as the Internet. As such, the network <NUM> may include any number of additional devices, such as additional computers, routers, and switches, to facilitate communications among the devices of system <NUM>.

<FIG> illustrates one embodiment of a compute node <NUM>, which establishes an environment <NUM> during operation. The illustrative environment <NUM> includes a number of virtual machines <NUM>, a data collection module <NUM>, a per virtual machine (Per VM) module <NUM>, a contention score determination module <NUM>, a node agent <NUM> and a communication module <NUM>. The various modules of the environment <NUM> may be embodied as hardware, firmware, software, or a combination thereof.

According to one embodiment, each virtual machine <NUM> performs a cloud computing workload on the compute node <NUM>. Each virtual machine <NUM> is allocated to one or more of the processor cores <NUM>. In some embodiments, each virtual machine <NUM> may specify a number of desired virtual CPUs, and the virtual machine <NUM> may be assigned to that number of processor cores <NUM>. In a multi-tenant cloud computing environment, each virtual machine <NUM> may be controlled by a different entity and therefore additionally may execute a workload having different performance characteristics. In particular, each virtual machine <NUM> may exert different pressure on the cache memory <NUM> of the compute node <NUM>.

As described below, virtual machines <NUM> are instantiated and otherwise managed by cloud controller <NUM>. Data collection module <NUM> collects performance data for the compute node <NUM>. Data collection module <NUM> may collect data indicative of cache contention on the compute node <NUM>, for example data measuring cache misses in the cache memory <NUM>. Data collection module <NUM> may include individual data collectors for each processor <NUM> and/or processor core <NUM> of the compute node <NUM>, or for each virtual machine <NUM>. Data collection module <NUM> may filter the collected data to remove noise. The data collectors of data collection module <NUM> may communicate with each other asynchronously, for example using a message bus such as the zeromq message bus.

Per VM monitor <NUM> monitors utilization and performance of virtual machines <NUM>. In one embodiment, Per VM monitor <NUM> identifies when a virtual machine is active. In such an embodiment, Per VM monitor <NUM> uses hardware events to monitor virtual machine or process activity using a per thread ID. Further, Per VM monitor <NUM> maintains a list of thread IDs for each process and samples per thread ID hardware counters at a predetermined interval (e.g., per second) using a performance analyzing tool (e.g., Performance Counters for Linux (perf)). In still a further embodiment, Per VM monitor <NUM> groups the per thread monitored data to per process performance, which is acquired by data collection module <NUM>.

Contention score determination module <NUM> calculates a contention score as a function of the performance data collected by the data collection module <NUM>. The contention score may include both a contention metric and a contention score level. The contention metric may include aggregated data describing cache misses for all processors <NUM> of the compute node <NUM>. The contention score level may be generated by a heuristic algorithm, and may be selected from levels for low contention, medium contention, and high contention (e.g., via a form of quantization). In one embodiment, the contention score measures the cache memory <NUM> contention experienced on compute node <NUM>. The contention score may be embodied as a tuple including the cache misses per some reference number of instructions (e.g., per thousand instructions), as well as a contention score level (e.g., high, medium, or low contention).

Node agent <NUM> implements a control process to perform overprovisioning at a compute node <NUM>. According to one embodiment, node agent receives the virtual machine monitoring data from Per VM monitor <NUM> and dynamically readjusts resources to maintain performance at compute node <NUM>, as will be described in more detail below.

Communication module <NUM> transmits performance data to cloud state database <NUM>. The performance data is then accessible to cloud controller <NUM> through cloud state database <NUM>. Communication module <NUM> may transmit the contention score as a tuple including the contention metric and the contention score level. Communication module <NUM> may use any communication method to interface with the other members of the system <NUM>. For example, communication module <NUM> may be embodied as a message bus.

According to one embodiment, each compute node <NUM> communicates with cloud controller <NUM>, which each may establish an environment <NUM> during operation. The illustrative environment <NUM> includes a compute service module <NUM>, a scheduler module <NUM>, and a communication module <NUM>. The various modules of the environment <NUM> may be embodied as hardware, firmware, software, or a combination thereof. Compute service module <NUM> receives requests to instantiate a new virtual machine <NUM>, and to schedule the new virtual machine <NUM> for execution on a compute node <NUM> selected by the scheduler module <NUM>. Compute service module <NUM> may receive commands from any appropriate source. For example, compute service module <NUM> may receive commands from a cloud system administrator or from a cloud computing customer through a web-based control interface. As another example, in some embodiments, compute service module <NUM> may receive a command for a new virtual machine <NUM> based on demand for computing services, for example to scale to respond to application demand. In a further embodiment, compute service module <NUM> detects SLA violations based on the contention score while monitoring application performance. In such an embodiment, compute service module <NUM> determines control actions that are to be applied to restore the application performance to the specified SLA.

Scheduler module <NUM> selects an appropriate compute node <NUM> based on received performance data. In some embodiments, scheduler module <NUM> may be embodied as a filter scheduler that selects a compute node <NUM> having the lowest reported contention score level. When two or more compute nodes <NUM> have the lowest reported contention score level, the scheduler module <NUM> may select one randomly or by using any other available selection criteria. For example, in some embodiments, the scheduler module <NUM> may further sort compute nodes <NUM> and select the compute node <NUM> having the lowest contention metric.

Communication module <NUM> receives performance data from the cloud state database <NUM> for all compute nodes <NUM> within system <NUM>. Communication module <NUM> may receive a tuple from the cloud state database <NUM> for each compute node <NUM> including the contention metric and the contention score level. Communication module <NUM> may use any communication method to interface with the other members of the system <NUM>. For example, communication module <NUM> may be embodied as a message bus.

According to one embodiment, compute service module <NUM> includes service compute unit (SCU) <NUM>. In such an embodiment, SCU <NUM> defines a service level objective (SLO) in terms of a compute consumption metric. The compute consumption metric is based on processor <NUM> performance characteristics (e.g., ability to execute instructions). In one embodiment, the compute consumption metric is measured as giga instructions per second (GIPS) rating. A processor <NUM> achieves a peak GIPS rating when executing a program that only has instructions of a type having a lowest cycles per instruction (CPI) or highest instructions per cycle (IPC). Thus, a compute performance metric is also calculated at compute service module <NUM> to qualify an actual GIPS using operational frequency.

According to one embodiment, SCU <NUM> is attached to each compute instance generated at compute service module <NUM>. In this embodiment, SCU <NUM> is multiplied with allocated virtual cores to provide an allowed total consumption to a virtual machine <NUM>. For example, Table <NUM> shows instance types and corresponding SLOs.

According to one embodiment, a total and headroom capacity of a compute node <NUM> as a virtual machine is calculated. Additionally, the calculations measure whether the IPC provided to each instance matches the SLO. The calculations include the following:.

At processing block <NUM>, instance types similar to those shown in Table <NUM> and SLAs are provided in a user interface implemented at compute service module <NUM> in response to a request to instantiate a new virtual machine <NUM>. Thus, a virtual machine is created using compute service module <NUM> that belongs to one of multiple instances types specified and supported by SCU <NUM>. At processing block <NUM>, compute service module <NUM> identifies resource requirements for the virtual machine <NUM>. At processing block <NUM>, Scheduler <NUM> identifies a compute node <NUM> on which to schedule the virtual machine <NUM>. In one embodiment, scheduler module <NUM> takes into consideration the resource requirements of the virtual machine <NUM> and the current state of the various control nodes <NUM> to determine the initial placement for the virtual machine <NUM>.

At processing block <NUM>, scheduler module <NUM> schedules the virtual machine <NUM> on the identified compute node <NUM>. At processing block <NUM>, the SLO parameters for the virtual machine <NUM> is stored within database <NUM> and transmitted to the node agent <NUM> at the identified compute node <NUM> at which the virtual machine <NUM> will execute (e.g., using the rabbitmq messaging). At processing block <NUM>, the SLO requirements are mapped to resources on the compute node <NUM>. At processing block <NUM>, the virtual machine <NUM> begins operation at the compute node <NUM>. According to one embodiment, adjustments are implemented at the compute node <NUM> according to a fair overprovisioning process described below.

At processing block <NUM>, Per VM monitor <NUM> measures utilization and performance of the virtual machine <NUM>. Because applications exhibit phase changes or experience varying load over time, resource usage also varies. Due to such variations across all of the virtual machines <NUM> executing on a compute node <NUM>, contention faced by a virtual machine <NUM> of interest also varies. Consequently, dynamic readjustment may be performed based on continuous monitoring by Per VM monitor <NUM>.

At decision block <NUM>, node agent <NUM> determines whether the virtual machine <NUM> performance is below the SLO parameters based on monitored performance results received from Per VM monitor <NUM>. According to one embodiment, node agent <NUM> uses the current virtual machine <NUM> performance and expected virtual machine <NUM> performance based on allocated the SLO parameters. If the virtual machine <NUM> performance is not below the SLO parameters, control is returned to processing block <NUM> where the virtual machine <NUM> continues to be monitored. If, however, the virtual machine <NUM> performance is below the SLO parameters, a corrective action is initiated, processing block <NUM>. In one embodiment, a magnitude of change of resource allocation is calculated to modify the resource allocation to achieve expected performance.

According to one embodiment, node agent <NUM> may perform boosting, throttling or migration actions to adjust resource allocation. Boosting increases resources when an application performance falls below expected levels. Throttling reduces resources provided to the application. In one embodiment, reduction is initiated when such a reduction of resource does not result in SLA violation. For example, streaming applications do not exhibit reduction in cache miss rate when additional cache is provided. Similarly, when cache availability is reduced, the cache miss rate does not increase significantly. Migration involves migrating a virtual machine <NUM> to either a different socket on the same compute node <NUM> or to a different compute node <NUM> in order to reduce interference from noisy neighbors. This effectively increases the resources available to the virtual machine <NUM> that was migrated.

According to one embodiment, node agent <NUM> implements resource controls to limit, account, and isolate resource usage (e.g., CPU, memory, disk I/O, etc.) to manage a CPU Controller and CPUSET Controller subsystem in order to meet the application SLAs by reducing resource contention and increasing predictability in performance, and to prevent a single or group of virtual machines from monopolizing resources, or impacting other environments.

<FIG> illustrates one embodiment of a high level functional description for performing virtual machine resource allocation.

In one embodiment, a user space daemon (control service agent or CSA) <NUM> is implemented to interact with kernel resource groups subsystems <NUM> to initialize and mount the resource group controllers. Additionally, CSA interacts with a virtualization management daemon service <NUM> to perform resource allocation and partitioning of compute node <NUM> resources and dynamic resource assignment. CPUSET provides a mechanism for assigning a set of pCPUs to a set of virtual vCPUs to enable a mapping to constrain CPU placement of the virtual machines to only the resources within a VM's current CPUSET. In one embodiment, the mapping forms a nested hierarchy visible in a virtual file system that is controlled by CSA. Resource groups subsystems <NUM> define CPU time shares that are utilized for each virtual machine. CSA <NUM> uses this group to manage CPU shares, wherein each group gets shares hierarchically. In one embodiment, CPU shares are set to <NUM> as a default. However, setting CPU shares to a higher value provides the CPU groups a higher quantum of the CPU, which is performed by CSA <NUM> boosting or throttling a virtual machine. According to one embodiment, OS scheduler <NUM> is a router provides for scheduling (e.g., round-robin or fair share). <FIG> illustrates one embodiment of virtual machine shares allocation.

According to one embodiment, a fixed resource allocation model partitions system resources between all virtual machines such that the total resources requested is equal to the resource available. In such an embodiment, the resources requested by a virtual machine is based on its maximum consumption such that: <MAT>.

However due to variation in phases, the virtual machines do not continuously consume all of the requested resources. Therefore resources may be overprovisioned, and shared between virtual machines. In an overprovisioned environment, the total system resources available will be lesser than the sum of the resources required by each VM. The resources are dynamically reassigned as shown in the resource model: <MAT>.

Resource optimizing and fair over provisioning is implemented by logically partitioning processor <NUM> resources on a per core basis. Such a logical portioning approach is suitable for scenarios where placement of virtual machines are to be implemented in such a way as to be contained exclusively in a core where no other virtual machine else competes for available bandwidth within that core. For instance, noisy neighbor scenarios can be handled with this approach.

<FIG> is a flow diagram illustrating one embodiment of a process <NUM> for performing fair overprovisioning. Method <NUM> may be performed by processing logic that may comprise hardware (e.g., circuitry, dedicated logic, programmable logic, etc.), software (such as instructions run on a processing device), or a combination thereof. The processes of method <NUM> are illustrated in linear sequences for brevity and clarity in presentation; however, it is contemplated that any number of them can be performed in parallel, asynchronously, or in different orders. For brevity, clarity, and ease of understanding, many of the details discussed with reference to <FIG> may not be discussed or repeated here.

At processing block <NUM>, a system topology is created using resource groups <NUM> CPU and CPUSET controller subsystems. At processing block <NUM>, these subsystems are mounted by a CSA at a specified path. At processing block <NUM>, a system default pool is created to exclusively reserve some CPU cores for the system daemons and other processes to run. At processing block <NUM>, additional pools are created on a per socket basis by logically creating resource group cpusets for all logical cores belonging to the particular socket.

At processing block <NUM>, scheduler module <NUM> schedules virtual machines <NUM> on a compute node <NUM> and sends an attached SLO message with each virtual machine <NUM> that includes the number of vCPUs on which the virtual machine <NUM> is to run, along with other elements (e.g., a requested compute service modules <NUM>). At processing block <NUM>, the virtual machines <NUM> are placed and the requested compute service modules <NUM> are translated into cgroup <NUM> CPU shares. As a result, minimum guarantee is ensured.

According to one embodiment, the virtual machine <NUM> are placement is performed by pinning the vCPUs corresponding to the virtual machine to the CPUSET subsystem. In addition, a new hierarchy is created under CPU controller resource group subsystem with a virtual machine universally unique identifier (UUID) as a new control group. Scheduler module <NUM> may place additional virtual machines on the compute node <NUM> as long as the total platform capacity is utilized. At processing block <NUM>, execution begins.

In one embodiment, the virtual machine <NUM> run time dynamics typically indicate that actual usage is less even though the virtual machine <NUM> SLO requested more CPU bandwidth reservation. This allows compute service module <NUM> to make a decision of densely packing the load by a predefined over overprovisioning factor. In a further embodiment, a virtual machine <NUM> migration decision is made when the platform capacity is reached in order to honor its SLO. The above-described model offers a powerful and flexible set of resource controls and a wide range of resource management policies that may be utilized and provide differentiated Quality of Service to the participating virtual machines <NUM>.

<FIG> is a flow diagram illustrating one embodiment of a process <NUM> for performing fair over provisioning. Method <NUM> may be performed by processing logic that may comprise hardware (e.g., circuitry, dedicated logic, programmable logic, etc.), software (such as instructions run on a processing device), or a combination thereof. The processes of method <NUM> are illustrated in linear sequences for brevity and clarity in presentation; however, it is contemplated that any number of them can be performed in parallel, asynchronously, or in different orders. For brevity, clarity, and ease of understanding, many of the details discussed with reference to <FIG> may not be discussed or repeated here.

At processing block <NUM>, a platform topology is created on a per socket basis. In one embodiment, a pool of CPUs are reserved per socket such that: <MAT> where a compute service module <NUM> = <NUM> SCUs.

At processing block <NUM>, a loop is run to accept incoming SLO messages for new scheduled virtual machines <NUM> by scheduler <NUM>. In one embodiment, a virtual machine SLO message includes SCU requirements for each vCPU per virtual machine <NUM>. At processing block <NUM>, the virtual machine SLO is honored by placing the virtual machine <NUM> on an appropriate socket. In one embodiment, this process is performed by pinning vCPUs, using resource groups <NUM>, to logical CPUs that belong to the particular socket. At processing block <NUM>, the virtual machines <NUM> reservation is guaranteed using proportionate resource groups <NUM> shares. Additionally, a limit equal to a reservation value included in the SLO message is applied. At processing block <NUM>, the total used SCUs per socket is calculated and database <NUM> is updated. At processing block <NUM>, additional virtual machines <NUM> are assigned until: <MAT>.

Full capacity is reached once the condition is met. Subsequently, an observer loop periodically checks to detect the total used SCUs per socket. Overprovisioning may often be performed since virtual machines <NUM> typically consume much less resources than requested. However, overprovisioning cannot be achieved when virtual machines <NUM> are using the requested resources. According to one embodiment, overprovisioning occurs for a factor of ρ when: <MAT> where <MAT>.

At decision block <NUM>, a determination is made as to whether the total number of currently used virtual machine SCUs ) ≥ <NUM>% Socket SCUs. If the condition has not met, control is returned to processing block <NUM> where additional virtual machines are assigned to the socket. However, a determination that the total number of currently used virtual machine SCUs ≥ <NUM>% Socket SCUs indicates that one or more of the virtual machines <NUM> is now trying to use its reserved SCUs. However, the virtual machine <NUM> will soon begin suffering when the total used SCUs becomes <NUM>% of the socket capacity (e.g., resources have been overprovisioned).

At processing block <NUM>, Per VM Monitor <NUM> reads data to identify which virtual machine <NUM> is attempting to claim its share. In one embodiment, the data includes telemetry data read from registers within processor <NUM> and performance events mapped to the virtual machine <NUM>. In a further embodiment, the virtual machine <NUM> is temporarily assigned to a CPU in the pool of CPUs. In a further embodiment, the observer loop monitors the system for a predetermined time interval to determine if the system attains an equilibrium state. In such an embodiment, p is readjusted if necessary to maintain a steady state. At processing block <NUM>, the identified virtual machine <NUM> is migrated based on a policy that ensures least disturbance to the cloud.

The exemplary computer system <NUM> includes a processor <NUM>, a main memory <NUM> (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM), etc., static memory such as flash memory, static random access memory (SRAM), volatile but high-data rate RAM, etc.), and a secondary memory <NUM> (e.g., a persistent storage device including hard disk drives and persistent multi-tenant data base implementations), which communicate with each other via a bus <NUM>. Main memory <NUM> includes emitted execution data <NUM> (e.g., data emitted by a logging framework) and one or more trace preferences <NUM> which operate in conjunction with processing logic <NUM> and processor <NUM> to perform the methodologies discussed herein.

Processor <NUM> represents one or more general-purpose processing devices such as a microprocessor, central processing unit, or the like. More particularly, the processor <NUM> may be a complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, processor implementing other instruction sets, or processors implementing a combination of instruction sets. Processor <NUM> may also be one or more special-purpose processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), network processor, or the like. Processor <NUM> is configured to execute the processing logic <NUM> for performing the operations and functionality of the above-described mechanism.

The computer system <NUM> may further include a network interface card <NUM>. The computer system <NUM> also may include a user interface <NUM> (such as a video display unit, a liquid crystal display (LCD), or a cathode ray tube (CRT)), an alphanumeric input device <NUM> (e.g., a keyboard), a cursor control device <NUM> (e.g., a mouse), and a signal generation device <NUM> (e.g., an integrated speaker). The computer system <NUM> may further include peripheral device <NUM> (e.g., wireless or wired communication devices, memory devices, storage devices, audio processing devices, video processing devices, etc. The computer system <NUM> may further include a Hardware based API logging framework <NUM> capable of executing incoming requests for services and emitting execution data responsive to the fulfillment of such incoming requests.

The secondary memory <NUM> may include a machine-readable storage medium (or more specifically a machine- accessible storage medium) <NUM> on which is stored one or more sets of instructions (e.g., software <NUM>) embodying any one or more of the methodologies or functions described above. The software <NUM> may also reside, completely or at least partially, within the main memory <NUM> and/or within the processor <NUM> during execution thereof by the computer system <NUM>, the main memory <NUM> and the processor <NUM> also constituting machine-readable storage media. The software <NUM> may further be transmitted or received over a network <NUM> via the network interface card <NUM>. The machine-readable storage medium <NUM> may include transitory or non-transitory machine-readable storage media.

Portions of various embodiments may be provided as a computer program product, which may include a machine-readable medium having stored thereon computer program instructions, which may be used to program a computer (or other electronic devices) to perform a process according to the embodiments. The machine-readable medium may include, but is not limited to, floppy diskettes, optical disks, compact disk read-only memory (CD-ROM), and magneto-optical disks, ROM, RAM, erasable programmable read-only memory (EPROM), electrically EPROM (EEPROM), magnet or optical cards, flash memory, or other type of media/machine-readable medium suitable for storing electronic instructions.

The techniques shown in the figures can be implemented using code and data stored and executed on one or more electronic devices (e.g., an end station, a network element). Such electronic devices store and communicate (internally and/or with other electronic devices over a network) code and data using computer-readable media, such as non-transitory computer- readable storage media (e.g., magnetic disks; optical disks; random access memory; read only memory; flash memory devices; phase-change memory) and transitory computer -readable transmission media (e.g., electrical, optical, acoustical or other form of propagated signals - such as carrier waves, infrared signals, digital signals). In addition, such electronic devices typically include a set of one or more processors coupled to one or more other components, such as one or more storage devices (non-transitory machine-readable storage media), user input/output devices (e.g., a keyboard, a touchscreen, and/or a display), and network connections. The coupling of the set of processors and other components is typically through one or more busses and bridges (also termed as bus controllers). Thus, the storage device of a given electronic device typically stores code and/or data for execution on the set of one or more processors of that electronic device. Of course, one or more parts of an embodiment may be implemented using different combinations of software, firmware, and/or hardware.

Embodiments may be implemented as any or a combination of: one or more microchips or integrated circuits interconnected using a parentboard, hardwired logic, software stored by a memory device and executed by a microprocessor, firmware, an application specific integrated circuit (ASIC), and/or a field programmable gate array (FPGA). The term "logic" may include, by way of example, software or hardware and/or combinations of software and hardware.

Moreover, embodiments may be downloaded as a computer program product, wherein the program may be transferred from a remote computer (e.g., a server) to a requesting computer (e.g., a client) by way of one or more data signals embodied in and/or modulated by a carrier wave or other propagation medium via a communication link (e.g., a modem and/or network connection).

References to "one embodiment", "an embodiment", "example embodiment", "various embodiments", etc., indicate that the embodiment(s) so described may include particular features, structures, or characteristics, but not every embodiment necessarily includes the particular features, structures, or characteristics. Further, some embodiments may have some, all, or none of the features described for other embodiments.

In the following description and claims, the term "coupled" along with its derivatives, may be used. "Coupled" is used to indicate that two or more elements co-operate or interact with each other, but they may or may not have intervening physical or electrical components between them.

Claim 1:
A compute node (<NUM>) of a cloud computing cluster comprising:
one or more processors (<NUM>) configured to implement a plurality of virtual machines (<NUM>), each of the one or more processors having at least one central processing unit, CPU, core (<NUM>), each virtual machine constrained to consume resources within one of a plurality of subsets of CPU cores of the one or more processors of the compute node;
the one or more processors configured to establish an operating environment (<NUM>), the operating environment including a node agent (<NUM>) in communication with a cloud controller (<NUM>) for controlling scheduling any one or more of the plurality of virtual machines, the node agent configured to
receive a request from the cloud controller to schedule a virtual machine to a specified one of the subsets of CPU cores, the request including service level objective, SLO, parameters representing a consumption allowance of resources for enabling the virtual machine to achieve an expected performance based on performance characteristics of the one or more processors;
map the SLO parameters to resources of the compute node, wherein mapping the SLO parameters to resources of the compute node comprises computing consumption metrics corresponding to the performance characteristics of the one or more processors; and
initiate an overprovisioning process to allocate available resources within the specified subset of CPU cores in accordance with the mapped SLO parameters, wherein the allocated available resources are less than a total of all of the consumption allowances of resources requested for all virtual machines scheduled to the specified subset of CPU cores,
wherein the overprovisioning process to allocate resources is performed for all virtual machines scheduled to the specified subset of CPU cores and comprises logically partitioning central processing unit, CPU, resources per CPU core included in the specified subset of CPU cores of the compute node;
the operating environment established by the one or more processors further including a VM monitor (<NUM>) of the virtual machine to monitor performance data of the virtual machine corresponding to the mapped SLO parameters,
wherein the node agent receives performance data of the virtual machine from the VM monitor and dynamically modifies resource allocation within the specified subset of CPU cores to achieve the expected performance of the virtual machine.