Patent Description:
A server executes a virtual machine monitor, or hypervisor, to manage any virtual machines executing therein. The hypervisor loads the operating systems of the virtual machines and allocates CPU resources, RAM, and disk storage space to each virtual machine. Each virtual machine then interacts with the hypervisor to utilize its allocated resources.

Physical resources are typically allocated to virtual machines according to a first-in, first-out protocol. Such a protocol may result in uneven usage of the physical resources of a given resource type. Moreover, conventional resource allocation does not efficiently account for degradation of individual physical resources over time. Improved systems for allocating physical resources to virtual machines are needed. <CIT> relates to techniques for creating a virtual machine and/or a container through a unified resource scheduling coordinator, a unified resource scheduling system and a computer storage medium. The technique includes: receiving an instruction to create the virtual machine and/or the container, wherein the instruction comprises parameters of the virtual machine and/or the container; selecting a first computing node from physical nodes according to the parameters; transmitting the parameters and information of the first computing node to a virtual machine management module and/or a container management module depending on whether the virtual machine or the container is to be created; receiving a resource allocation condition from the virtual machine management module and/or the container management module after creation of the virtual machine and/or the container; notifying the resource allocation condition to the container management module and/or the virtual machine management module after recording of resource change condition. In one example, a virtual machine management module finds that computing resources in a NUMA node <NUM> and NUMA node <NUM> on a computing node meet the requirements of a virtual machine, and randomly selects three CPUs.

The following description is provided to enable any person in the art to make and use the described embodiments. Various modifications, however, will remain readily-apparent to those in the art.

Some embodiments employ pseudo-random selection of physical resources to be allocated to virtual machines. For example, a request may be received to create a virtual machine including three processor cores in a server including eight processor cores. In response, embodiments may pseudo-randomly select three of eight processor cores for allocation to the virtual machine. Since each of the eight processor cores is associated with a substantially-equal likelihood of being selected, each core may be selected a substantially-equal number of times over the course of operation. In some scenarios, particularly in a case that resource allocation is optimized for each created virtual machine and the lifecycles of each created virtual machine are fairly uniform, each of the eight processor cores experiences a substantially equal amount of wear. As will be described below, embodiments are not limited to processor cores.

The pool of resources from which resources are pseudo-randomly-selected may be determined based on wear. For example, some embodiments may track utilization information associated with each resource. If the utilization information indicates that a resource is near or past the end of the lifetime over which it may be assumed to function reliably, the resource is not included in the pool of resources from which resources are randomly-selected. For example, if one of the eight processor cores of the prior example is deemed unsuitable due to its associated utilization information, the three requested processor cores are randomly-selected from the remaining seven processor cores.

The utilization information may comprise a number and/or an intensity of computations in the case of a processor core resource type and may comprise a number of reads/writes in the case of a disk or RAM resource type. In some embodiments, the utilization information comprises the total uptime of virtual machines to which a particular resource was allocated. Total uptime may be a particularly effective proxy for resource wear in a case that resource allocation is optimized for each created virtual machine. Moreover, existing landscapes for deploying virtual machines may already include mechanisms for monitoring the total uptime metric.

<FIG> is a block diagram of computer server <NUM> including physical resources <NUM> and executing hypervisor <NUM> according to some embodiments. Embodiments are not limited to computer server <NUM> or to the foregoing description thereof.

Computer server <NUM> may comprise any type of server hardware that is or becomes known. The components of computer server <NUM> may be located within a same physical chassis or multiple independent physical units located local to or remote from one another. Such multiple independent physical units may be "striped" in order to behave as a single logical monolithic unit. Computer server <NUM> may include other hardware and software required to execute the functions attributed to computer server <NUM> herein, as well as any other functions to be performed by computer server <NUM>.

Physical resources <NUM> include processing (i.e., compute) resources <NUM>, RAM storage <NUM> and disk storage <NUM>. Each of these resources will be referred to herein as a resource type. Embodiments are not limited to the foregoing resource types.

Resources <NUM> of computer server <NUM> operate to execute program code of hypervisor <NUM>. As described above, hypervisor <NUM> may allocate resources <NUM> to virtual machines executing within computer server <NUM>, instantiate the virtual machines, execute the guest operating system of each virtual machine, and allow the virtual machines to access the resources <NUM> allocated thereto.

Hypervisor <NUM> may comprise a native hypervisor or a hosted hypervisor. A native hypervisor runs directly on host system hardware and does not execute on an underlying operating system. A native hypervisor therefore directly accesses the physical resources of the host system. A hosted hypervisor is installed on the underlying operating system of a host system. A hosted hypervisor therefore issues requests to the underlying operating system to access the physical resources of the host system.

Hypervisor <NUM> allocates subsets of each resource type to each virtual machine created within server <NUM>. The individually-allocatable subsets of each resource type will be referred to herein as resource units. For example, if hypervisor <NUM> is capable of allocating individual processor cores of processing resources <NUM> to a virtual machine executing within server <NUM>, each processor core is considered a distinct resource unit. Each dual in-line memory module (DIMM) of RAM resources <NUM> may be considered a single resource unit, and/or each Gigabyte of storage of RAM resources <NUM> may be considered a single resource unit, depending on the ability of hypervisor <NUM>, computer server <NUM> and the RAM design to directly address and allocate these distinct subsets. Fixed disks <NUM> may similarly consist of resource units defined by distinct drive, platter, sector, or any other suitable subset.

Each resource type of resources <NUM> is associated with respective metadata <NUM>, <NUM> and <NUM>. Metadata <NUM>, <NUM> and <NUM> may represent only resource units of resources <NUM> which may be allocated to virtual machines, while other resource units of resources <NUM> may be specifically earmarked for implementing other operations of computer server <NUM> (e.g., execution of hypervisor <NUM>).

The metadata of metadata <NUM>, <NUM> and <NUM> identifies, for each resource unit of the respective resource type, an identifier of the resource unit, an identifier of a virtual machine to which the resource unit is currently associated, and utilization information. The utilization information may comprise any suitable indicator of a degree of wear, such as but not limited to a total number of executed operations, a percentage of expected lifetime operations utilized, a total usage time, and a percentage of expected lifetime usage elapsed. Computer server <NUM> may track, update and store metadata <NUM>, <NUM> and <NUM>, but embodiments are not limited thereto.

<FIG> also illustrates reception of request <NUM> to create a virtual machine. Request <NUM> may be received from any client or control system, specific examples of which will be discussed below. Request <NUM> may specify, among other details, a number of resource units to allocate to the virtual machine for each of several resource types. For example, request <NUM> may comprise an instruction to create a virtual machine to which 2Gb of RAM, one processor core and 512Gb of fixed disk space are allocated.

According to some embodiments, hypervisor <NUM> receives request <NUM> and instantiates a virtual machine conforming to request <NUM>. In order to do so, hypervisor <NUM> may first determine a pool of resources for each resource type specified in the request. The example request described above requests resource units of processor core, RAM and fixed disk resource types. The pool of resources for each specified resource type may therefore be determined based on metadata <NUM>, <NUM> and <NUM> associated with the resource types.

None of the resource units represented by metadata <NUM>, <NUM> and <NUM> of <FIG> are assigned to any virtual machines. Also, none of the resource units represented by metadata <NUM>, <NUM> and <NUM> are associated with any utilization, or wear. Accordingly, hypervisor <NUM> may determine that each of these resource unit is available for inclusion in a pool of resource units of its respective resource type.

The requested number of resource units for each of the resource types is then randomly selected from the pool of available resource units of the resource type. It is appreciated that true random behavior may be impossible to achieve in practice and the term random as used herein is intended to encompass pseudo-random selection and any other selection technique which substantially simulates true random behavior.

According to some embodiments, the random selection of resource units for a given resource type includes applying a randomized algorithm to the available pool of resource units for the given resource type to generate a random set of the requested number of resource units. Any suitable system for random selection may be employed in some embodiments.

Next, hypervisor <NUM> allocates the selected resource units of each of the resource types to the virtual machine. <FIG> illustrates metadata <NUM>, <NUM> and <NUM> after allocation in response to request <NUM> according to some embodiments. As shown, one processor core C1, two RAM DIMMs R1 and R4 (at 1Gb each), and two fixed disks D0 and D4 (at 256Gb each) have been allocated to virtual machine VM1. Hypervisor <NUM> may then instantiate VM1 <NUM> in accordance with the resource units allocated thereto as is known in the art.

<FIG> comprises a flow diagram of process <NUM> to allocate resource units of physical resource types to a virtual machine according to some embodiments. In some embodiments, processing units (e.g., one or more processors, processing cores, processor threads) host server execute software program code of a hypervisor to perform process <NUM>. In other embodiments, process <NUM> is executed by a cloud service provider and/or distributed resource manager. Process <NUM> and all other processes mentioned herein may be embodied in processor-executable program code read from one or more of non-transitory computer-readable media, such as a volatile or non-volatile random access memory, a fixed disk drive, a DVD-ROM, a Flash drive, and a magnetic tape, and then stored in a compressed, uncompiled and/or encrypted format. Embodiments are therefore not limited to any specific combination of hardware and software.

Initially, at S310, a request to create a virtual machine is received. The requested virtual machine is associated with a requested number of resource units of each of a plurality of resource types. The request may be received from a client system intending to use the virtual machine, from a resource manager for providing virtual machines to one or more client systems, or from any other suitable entity.

A pool of available resource units is determined for each resource type of the request at S320. The pool of available resource units may include all resource units which may be used by virtual machines on the host platform, excluding those resource units which are currently allocated to a virtual machine.

<FIG> illustrates an example in which request <NUM> is received at S310 at server <NUM> a short time after the depiction of <FIG>. It is assumed that request <NUM> requests one resource unit of each resource type of resources <NUM>. At this time, metadata <NUM>, <NUM> and <NUM> show the allocations of <FIG> and also show utilization information associated with each allocated resource unit. At S320, a pool of resources is determined for each resource type which includes each resource unit that is not currently allocated to a virtual machine (i.e., VM1). In particular, the pool for the processor core resource type includes processor cores C0, C2, C3 and C4, the pool for the RAM storage resource type includes RAM DIMMs R0, R2, R3, and the pool for the fixed disk resource type includes fixed disks D1, D2, D3.

The requested number of resource units for each of the resource types is then randomly selected from the pool of available resource units of the resource type at S330. According to the present example, processor core resource unit C4 is selected from the pool including processor cores C0, C2, C3, the pool for the RAM storage resource unit R3 is selected from the pool including RAM DIMMs R0, R2, R3, and fixed disk resource unit D2 is selected from the pool including fixed disks D1, D2, D3.

Next, and as reflected in the metadata of <FIG>, the selected resource units of each of the resource types are allocated to the requested virtual machine at S340. <FIG> further illustrates instantiation of corresponding VM2 <NUM> in accordance with the resource units allocated thereto.

<FIG> illustrates server <NUM> after a period of execution of virtual machines VM1 <NUM> and VM2 <NUM>. Metadata <NUM>, <NUM> and <NUM> have been updated to reflect utilization of the resource units associated with each virtual machine. Metadata <NUM>, <NUM> and <NUM> may be updated based on utilization information collected by hypervisor <NUM>, by an underlying operating system of server <NUM>, and/or by any other monitoring mechanism. As shown, the percentage utilizations (e.g., consumed percent of total recommended operational lifetime) associated with the processor core and RAM resource units of virtual machine VM1 <NUM> are <NUM>% and above, indicating that these resource units may be reaching the end of their recommended life. The utilizations reflected in <FIG> are for example purposes only, and would be expected to result from the instantiation, operation and de-allocation of many virtual machines over time, and not only two virtual machines as in the example.

It will now be assumed that hypervisor terminates virtual machine VM1 <NUM>, for example, in response to an instruction to terminate VM1 <NUM>. <FIG> shows server <NUM> in which VM1 <NUM> is no longer executing. Metadata <NUM>, <NUM> and <NUM> of <FIG> therefore indicates that no resource units of resources <NUM> are associated with VM1 <NUM>. Accordingly, the resource units formerly associated with VM1 <NUM> may be considered free for allocation to another virtual machine.

Continuing the present example, <FIG> shows server <NUM> upon receipt of request <NUM> to create another virtual machine. As described above, hypervisor <NUM> may respond to request <NUM> by determining a pool of resources for each requested resource type. The pool for a given resource type may include each resource unit of the resource type that is not currently allocated to a virtual machine and that is associated with a percentage utilization which is less than a threshold value. The threshold value may differ for different resource types and/or in any other suitable manner.

A threshold value of <NUM>% will be assumed for each resource type in the following description. Accordingly, the determined pool for the processor core resource type includes processor cores C0, C2, C3, the pool for the RAM storage resource type includes RAM DIMMs R0, R2, and the pool for the fixed disk resource type includes fixed disks D0, D1, D3, D4.

The requested number of resource units for each of the resource types is then randomly selected from the pool of available resource units of the resource type at S330. According to the present example, processor core resource units C2, C3 are selected from the pool including processor cores C0, C2, C3, RAM storage resource unit R0, R2 are selected from the pool including RAM DIMMs R0, R2, R3, and fixed disk resource units D0, D3, D4 are selected from the pool including fixed disks D0, D1, D3, D4.

Next, and as reflected in the metadata of <FIG>, the selected resource units of each of the resource types are allocated to the requested virtual machine. <FIG> further illustrates instantiation of corresponding VM3 <NUM> in accordance with the resource units allocated thereto.

<FIG> comprises a flow diagram of process <NUM> to allocate resource units of physical resource types to a virtual machine as described with respect to <FIG> and according to some embodiments.

A request to create a virtual machine is received at S810 as described above with respect to S310. The requested virtual machine is associated with a requested number of resource units of each of a plurality of resource types. A pool of available resource units is determined for each resource type of the request at S820. The determined pool of available resource units may include all resource units which may be used by virtual machines executing on the host platform, excluding those resource units which are currently allocated to a virtual machine.

At S830, worn resource units are determined from each pool of resource units determined at S820. Worn resource units are determined based on utilization information associated with each resource unit. The utilization information associated with a resource unit comprises number of computations, and may further comprise an intensity of computations, a device temperature, an operational time, a total operation time of virtual machines to which the resource unit was allocated, and/or any other suitable measure. With reference to the <FIG> example, resource units C1, R1 and R4 were determined to be worn at S830 and therefore removed, at S840, from their respective pools of available resource units.

The requested number of resource units for each of the resource types is then randomly selected from the pool of available resource units of the resource type at S850. The selected resource units of each of the resource types are allocated to the requested virtual machine at S860. Again referring to <FIG>, processor core resource units C2, C3, RAM storage resource units R0, R2, and fixed disk resource units D0, D3, D4 are allocated to the requested virtual machine at S860.

<FIG> illustrates an architecture in which server <NUM> executes resource manager <NUM> to allocate resource units to a virtual machine as described above. Client system <NUM> transmits request <NUM> to create a virtual machine to server <NUM>. Client system <NUM> and server <NUM> may each comprise any standalone or distributed computing system.

Server <NUM> may transmit request <NUM> to host servers <NUM>, <NUM>, <NUM>, which may then execute their respective hypervisors <NUM>, <NUM>, <NUM> as described above to allocate resource units to a virtual machine and instantiate the virtual machine. A prioritization/load-balancing algorithm may be employed to determine the host server of host servers <NUM>, <NUM>, <NUM> in which the virtual machine is instantiated.

In some embodiments, resource manager <NUM> maintains metadata such as metadata <NUM>, <NUM>, <NUM> associated with each resource unit of resources <NUM>, <NUM>, <NUM> of host servers <NUM>, <NUM>, <NUM>. Accordingly, resource manager <NUM> may be executed to identify a host server of host servers <NUM>, <NUM>, <NUM>, to determine a pool of available resource units of the identified host server for each resource type and to randomly select resource units from the pools as described herein. Resource manager <NUM> may then instruct a hypervisor associated with the identified host servers to allocate the selected resource units to a virtual machine and to instantiate the virtual machine.

<FIG> illustrates an architecture for providing utilization information to a resource manager according to some embodiments. The <FIG> architecture may be implemented as a deployment of cloud-based resources. A cloud-based resource is a hosted and manageable item that is available to an external device via the Internet protocol stack. Cloud-based resources may include but are not limited to servers, virtual machines, storage accounts, web apps, databases, virtual networks, software, analytics and intelligence. A customer typically pays only for resources used, which may reduce operating costs while providing scale to cope with increased demand.

Each of client systems <NUM>, <NUM> and <NUM> may comprise any system for communicating with resource manager <NUM> of server <NUM>. In some embodiments, each of client systems <NUM>, <NUM> and <NUM> may comprise a desktop computing system and may be operated by a customer to access an interface provided by resource manager <NUM> to create, update, and delete cloud-based resources. In some embodiments, the customer establishes a subscription with an organization hosting server <NUM>, and the cloud-based resources managed by the customer are associated with the subscription.

These cloud-based resources may be accessed by the customer, the customer's customers, or any suitable entity. For example, a customer may define compute and storage resources of virtual machines to provide a publicly-accessible online store. In another example, instantiated virtual machines execute the customer's back office functions.

According to some embodiments, resource manager <NUM> operates as described above in response to a received request to create a virtual machine. In particular, resource manager <NUM> determines a pool of resource units for each resource type from one of servers <NUM>, <NUM>, <NUM>, possibly excluding worn resource units. Resource units satisfying the request are then randomly selected from the respective pools and allocated to a virtual machine, which is then instantiated.

<FIG> also illustrates usage server <NUM> executing usage monitor <NUM>. The hosting entity may operate usage monitor <NUM> to monitor usage of the virtual machines and/or constituent resource units of servers <NUM>, <NUM>, <NUM>. Usage monitor <NUM> may track the total time during which each resource unit and/or virtual machine is being used, as well as a time spent in various states (e.g., turbo state, idle state). Accordingly, usage monitor may transmit total usage time <NUM> to resource manager <NUM> as shown. Resource manager <NUM> may use total usage time <NUM> to maintain utilization metadata associated with each resource unit of servers may be used <NUM>, <NUM>, <NUM>.

Usage monitor <NUM> may also transmit total usage time <NUM> to reporting system <NUM> of the hosting entity. Generally, reporting system <NUM> may operate to generate usage report <NUM>. Usage report <NUM> may comprise a list of resources associated with a given subscription. Report <NUM> may also associate each instantiated virtual machine (and any other cloud-based resources of the customer's subscription) with a total usage time and a cost calculated based on the total usage time (e.g., = cost_per_hour x total_usage_time_in_hours).

<FIG> is a block diagram of apparatus <NUM> according to some embodiments. Apparatus <NUM> may comprise a general-purpose computing apparatus and may execute program code to perform any of the functions described herein. Apparatus <NUM> may comprise an implementation of server <NUM> in some embodiments. Apparatus <NUM> may include other unshown elements according to some embodiments.

Apparatus <NUM> includes processing unit <NUM> operatively coupled to communication device <NUM>, data storage device <NUM>, one or more input devices <NUM>, one or more output devices <NUM> and memory <NUM>. Communication interface <NUM> may facilitate communication with a network and ultimately with client systems as described above. Input device(s) <NUM> may comprise, for example, a keyboard, a keypad, a mouse or other pointing device, a microphone, knob or a switch, an infra-red (IR) port, a docking station, and/or a touch screen. Input device(s) <NUM> may be used, for example, to enter information into apparatus <NUM>. Output device(s) <NUM> may comprise, for example, a display and a speaker.

Data storage device <NUM> may comprise any appropriate persistent storage device, including combinations of magnetic storage devices (e.g., magnetic tape, hard disk drives and flash memory), optical storage devices, Read Only Memory (ROM) devices, etc., while memory <NUM> may comprise Random Access Memory (RAM), Storage Class Memory (SCM) or any other fast-access memory. Data storage device <NUM> may be implemented using distributed storage systems.

Resource manager <NUM> may comprise program code executable by processing unit <NUM> to cause apparatus <NUM> to perform any one or more of the processes described herein. Embodiments are not limited to execution of these processes by a single apparatus. For example, the processes attributed to a resource manager, a hypervisor, a usage monitor and a reporting server herein may be executed by one or more nodes of a distributed computing system.

Resource unit metadata <NUM> may associate each resource unit managed by resource manager <NUM> with a virtual machine (if any) to which the resource unit is allocated. Usage data <NUM> may associate usage information such as usage hours with resource units and be used to determine pools of resource units as described herein. Data storage device <NUM> may also store data and other program code for providing additional functionality and/or which are necessary for operation of apparatus <NUM>, such as device drivers, operating system files, etc..

Each functional component described herein may be implemented in computer hardware (integrated and/or discrete circuit components), in program code and/or in one or more computing systems executing such program code as is known in the art. Such a computing system may include one or more processing units which execute processor-executable program code stored in a memory system.

The above-described diagrams represent logical architectures for describing processes according to some embodiments, and actual implementations may include more or different components arranged in other manners. Other topologies may be used in conjunction with other embodiments. Moreover, each component or device described herein may be implemented by any number of devices in communication via any number of other public and/or private networks. Two or more of such computing devices may be located remote from one another and may communicate with one another via any known manner of network(s) and/or a dedicated connection. Each component or device may comprise any number of hardware and/or software elements suitable to provide the functions described herein as well as any other functions.

Claim 1:
A system (<NUM>) comprising:
a memory storing processor-executable process steps;
a processing unit configured to execute the processor-executable process steps to:
receive (S810) a request to create a virtual machine associated with a requested number of resource units of each of a plurality of resource types;
determine (S820), for each of the plurality of resource types, a pool of available resource units of the system;
determine (S830) worn resource units from each pool of available resource units based on utilization information associated with each resource unit, the utilization information comprising a number of computations;
remove (S840) the worn resource units from each pool of available resource units;
for each of the plurality of resource types, randomly select (S850) the requested number of resource units from the pool of available resource units of the resource type; and
allocate (S860) the selected resource units of each of the plurality of resource types to the virtual machine.