Microprocessor with software control over allocation of shared resources among multiple virtual servers

A system and computer program product for controlling the allocation of shared resources. The system includes a next request priority module connected to a shared resource and to a plurality of requesters identified by requester identifiers. The next request priority module includes a pending request mechanism that prioritizes pending requests for the shared resource, a logging mechanism logging requester identifiers associated with previous grants to the shared resource, and next request priority logic. The next request priority logic accesses the next pending request to determine if it should be granted priority to the shared resource. The determining is responsive to logged requester identifiers and to the next requester identifier. Priority is granted to the shared resource to the next pending request in response to determining that the next pending request should be granted priority to the shared resource. The next requester identifier is logged in response to the granting.

BACKGROUND

This invention relates generally to processing within a computing environment, and more particularly to allocating shared resources in a computing environment.

Overall computer system performance is affected by each of the key elements of the computer structure, including the performance/structure of the processor(s), any memory cache(s), the input/output (I/O) subsystem(s), the efficiency of the memory control function(s), the main memory device(s), and the type and structure of the interconnect interface(s).

Extensive research and development efforts are invested by the industry, on an ongoing basis, to create improved and/or innovative solutions to maximizing overall computer system performance and density by improving the system/subsystem design and/or structure. High-availability systems present further challenges as related to overall system reliability due to customer expectations that new computer systems will markedly surpass existing systems in regard to mean-time-between-failure (MTBF), in addition to offering additional functions, increased performance, increased storage, lower operating costs, etc. Other frequent customer requirements further exacerbate the computer system design challenges, and include such items as ease of upgrade and reduced system environmental impact (such as space, power, and cooling).

Current microprocessors have many processors, each running many threads of execution. For example, a current microprocessor may have eight processors, each with four threads, with hypervisor software being utilized to manage the multiple virtual processors. The number of simultaneous threads being executed is predicted to increase and in the future and microprocessors will likely have dozens of threads running simultaneously.

Microprocessors have many hardware resources that are shared by the multiple virtual processors that are under the control of the hypervisors. These shared resources include physical processors, caches, interconnection networks that provide cache coherence among multiple caches, memory controllers, input/output (I/O) controllers etc. Current microprocessor hardware designs rely on the hypervisor software combined with hardware timers in each physical processor to insure that virtual processors receive a fair share of the physical hardware resources. In current designs, virtual processors rely on the hypervisor to dispatch virtual processes onto physical processors in a manner that will provide each processor a fair share of “down-stream” shared resources such as cache accesses and memory controller accesses.

SUMMARY

An exemplary embodiment of the present invention includes a system for controlling the allocation of shared resources. The system includes a next request priority module connected to a shared resource and to a plurality of requestors identified by requester identifiers. The next request priority module includes a pending request mechanism, a logging mechanism and next request priority logic. The pending request mechanism prioritizes one or more pending requests for the shared resource from one or more of the requesters. Each pending request includes an associated requester identifier. One of the pending requests is designated as a next pending request with an associated next requester identifier. The logging mechanism logs requestor identifiers associated with previous requests granted priority to the shared resource. The next request priority logic accesses the next pending request and the next requester identifier and determines if the next pending request should be granted priority to the shared resource. The determining is responsive to logged requester identifiers and to the next requester identifier. Priority is granted to the shared resource to the next pending request in response to determining that the next pending request should be granted priority to the shared resource. The next requester identifier is logged in response to the granting.

A further exemplary embodiment includes a computer program product for controlling the allocation of shared resources. The computer program product includes a tangible storage medium readable by a processing circuit and storing instructions for execution by the processing circuit for performing a method. The method includes receiving a request to access a shared resource. The request is received from a requester executing on a processor. The receiving is at a next request priority module connected to the processor and the shared resource. It is determined if any of a specified number of most recent priority grants to the shared resource were to the requester. The request is granted to the shared resource if none of the specified number of most recent priority grants to the shared resource were to the requester. If it is determined that any of the specified number of most recent priority grants to the shared resource were to the requester, then it is determined if one or more other requests for the shared resource are pending. The request is granted to the shared resource if there are no other requests for the shared resource pending. It is determined if one of the other requests should be granted priority to the shared resource if other requests for the shared resource are pending.

DETAILED DESCRIPTION

An exemplary embodiment of the present invention provides hardware controls at the entrances to shared resources that limit the number of requests that can be dedicated to a single virtual server or group of virtual servers.

In current microprocessor hardware designs, a single virtual server that has many virtual processors could create large demands for shared hardware resources outside the microprocessor core. The demands could be large enough to create a performance impact on other virtual servers currently running on the same hardware system. In the past, it was relatively easy for a hypervisor, or any similar software system, to monitor the progress of the virtual servers and restrict the dispatching of a virtual processor that was making progress at the expense of others. This becomes more difficult for the hypervisor to monitor when there are more threads per virtual server, and as the number of physical processor cores (also referred to herein as core processors or processors) on a single microprocessor chip grows from the current one to eight to an even larger number.

In addition, excessive demands on system resources could be made by a virtual server running a workload with very poor cache behavior or by a malicious virus program that intends to disrupt the operation of a physical server (among other possibilities).

An exemplary embodiment of the present invention allows a hypervisor or other control program to set limits on the hardware resources that can be consumed by a virtual server or by a group of virtual servers. An exemplary embodiment of the present invention provides the ability to limit the maximum share of resources that can be consumed by a single virtual server. These limits can be set by the hypervisor or another control program. In an exemplary embodiment, a virtual server identifier (VSID) is assigned to each unique virtual server under the control of the hypervisor. The VSID is sent with any requests that leave the processor and attempt to use shared resources on the microprocessor chip or on other chips or interfaces in the system. Hardware and/or software controls prevent any single virtual server or group of virtual servers from occupying more than a certain percentage of the hardware resources.

FIG. 1depicts a portion of a multiprocessor computing system that may be implemented by an exemplary embodiment. Each core processor100is connected to one or more shared resources150via specialized interfaces110140and a next request priority module120. In an exemplary embodiment of the present invention, the next request priority module120is implemented in software. In an exemplary embodiment of the present invention, the next request priority module120is part of system control logic (SCL) that is located between a core processor100and a shared resource150. The SCL provides a physical interconnection among a plurality of microprocessor cores and between those cores and other system elements such as memory, data caches, I/O controllers, specialized accelerators, data compression engines, etc. Contemporary computing systems utilizing a centralized switch structure may have the SCL implemented on one or more chips dedicated to the switch function. These switch chips are then connected to microprocessor chips and arrays of memory chips and other elements of the system. Computing systems utilizing a distributed switch structure may implement a portion of the SCL on each of the microprocessor chips. In this case, the microprocessor chips are connected directly to each other and the interfaces used for those connections are controlled by the SCL. The SCL works to control the flow of requests between the source microprocessor core and the shared system resource. It also controls the returning flow of data.

In either the centralized switch case or the distributed switch case, a computing system which uses an embodiment of the current invention may also place a portion of the SCL at a point that is closest to the shared resource, on a memory DRAM for example, in addition to typical locations for the SCL seen in current systems. Other exemplary embodiments of the current invention may place part of the SCL at either end of a shared communication interface such as the memory controller logic that drives an otherwise standard memory interface. The placement of portions of the SCL in a particular computing system design will depend on a large number of factors including, but not limited to, available logic space, power dissipation and design complexity. Since many current designs contain multithreaded processors, in the exemplary embodiments the critical shared resources may also be located on the same microprocessor chip as the core, and therefore a portion of the SCL may be located there as well. Other exemplary embodiments may place the logic at the beginning of a shared pipeline of logic that leads to a shared resource such as a SRAM or eDRAM data cache.

One core processor100bin the system depicted inFIG. 1is connected to next request priority module120avia interface110b.The same core processor100bis also connected to next request priority module120bvia another dedicated interface110c.Access to the shared resources150is controlled by the next request priority modules120.

The next request priority modules120depicted inFIG. 1include virtual server control logic121and next request priority logic122. An exemplary embodiment of the virtual server control logic121is described below with respect toFIG. 2. An exemplary embodiment of the next request priority logic122is described below with respect toFIG. 4.

An exemplary embodiment of the invention uses a number of IBM P6 microprocessors, each of which has four processing cores. Exemplary embodiments of the invention can be used on systems with any number of microprocessor chips. In addition, exemplary embodiments may be implemented with any type of microprocessor, containing any number of core processors. Further, exemplary embodiments may be used in combination with microprocessors that support any of a number of multithreading implementations.

As depicted inFIG. 1, a VSID register160is provided for each of the core processors100in the system. The VSID registers160may be written to by software executing on the core processors100. An exemplary embodiment of the present invention uses a hypervisor to perform this task. In a system where multithreading is expected to be active and where individual threads may be assigned to run software from multiple virtual servers at once, a VSID register160can be provided for each thread. In an exemplary embodiment of the present invention, the hypervisor selects a VSID for each virtual server that it controls. When a virtual server is dispatched onto a physical processor (e.g., a core processor100), the VSID is written into the VSID register160for that core processor100.

The contents of the VSID register160are included with command, address and data that are communicated over interfaces110towards the shared resources150. The interfaces110may be similar to current request interfaces such as those used to connect with contemporary memory controllers. The combination of the VSID information and the prior-art command, address and data information are used by logic downstream from the core processor100to control the access to the shared resources150. An exemplary embodiment of the invention uses the next request priority module120to control access to shared resources150that are outside of (i.e., not located on) the microprocessor chip having core processors100a-100c.Other embodiments may use similar hardware and methods to control access to resources that are located inside of the microprocessor chip such as on-chip memory controllers, on-chip I/O controllers, on-chip data caches or any other on-chip resource.

In an exemplary embodiment of the present invention, the VSID is unique for each virtual server that the hypervisor is currently hosting on the computing system. Other embodiments of the invention may use a hash of the unique VSID or a simple subset of the bits of the full VSID. One exemplary embodiment of the invention uses just the two lowest order bits of the VSID in order to reduce the hardware overhead required to add the VSID to the request information that must be communicated from the core processors100to the shared resources150. The use of a subset of the VSID reduces the level of accuracy of the distribution of request slots to virtual servers but will still be able to prevent unpredictable performance for most virtual servers running on the computing system.

As used herein, the VSID is an example of a requester identifier that may be utilized to associate a requester (e.g., a virtual server) to a request for a shared resource (e.g. a memory). Other requester identifiers may be utilized without departing from the scope of the invention.

The interconnection between the core processors100and the shared resources150is shown inFIG. 1as a collection of point-to-point busses110,140. Alternate exemplary embodiments of the invention may utilize any other type of interconnection such as a shared bus, a full crossbar switch or a series of interconnected switches among many other configurations.

FIG. 2depicts virtual server control logic121that may be utilized by an exemplary embodiment of the invention. The virtual server control logic121depicted inFIG. 2includes a group of registers220that make up a request queue for storing pending requests for a shared resource150. The registers220also contain requester identifiers (e.g., VSIDs) for each request. The requester identifiers correlate the request to the requester (e.g., a virtual server executing on a core processor100). The request queue is ordered (e.g., based on how long each request has been in the queue, a combination of time in the queue and a priority associated with the requestor, etc.) and one of the requests is designated as a next request that will be evaluated for processing when the shared resource150is available to process a new request. The registers220contain the pending requests that have arrived from a core processor100via one of the interfaces110. An exemplary embodiment of the invention allows four requests to be held pending; other embodiments may use any number of additional (or fewer) registers for this function. The number of requests held in the queue could also be designed with a programmable depth.

An exemplary embodiment of the invention also includes a previous requester queue made up of a plurality of registers230. Exemplary embodiments of the present invention implement the previous requester queue using a four position deep first-in-first-out buffer to hold the VSIDs of the last four requests to be granted priority to the shared resource150. Any other depth of the buffer could be used. The buffer could also be designed with a programmable depth. In an exemplary embodiment, the depth is set by the hypervisor or by any other suitable supervisor software or by a suitable hardware algorithm. In an exemplary embodiment, the depth of the buffer is set during the initial power-on sequence of the system and then altered at a future time by the hypervisor.

The request queue220depicted inFIG. 2is an example of a pending request mechanism that may be implemented by an exemplary embodiment of the invention. Other manners of implementing a pending request mechanism in hardware and/or software may be utilized by exemplary embodiments of the present invention. For example, the pending request mechanism could be designed to favor a particular VSID. The favored VSID could be guaranteed a specific fraction of use of the shared resource, twenty-five percent for example.

Similarly, the previous requester queue230depicted inFIG. 2is an example of a logging mechanism that may be implemented by an exemplary embodiment of the invention. Other manners of implementing a logging mechanism in hardware and/or software may be utilized by exemplary embodiments of the present invention. For example, the logs could be implemented as a set of counters that increment each time a request with a particular VSID is granted priority for the next use of the shared resource. Each counter could be assigned a range of VSIDs. It should be understood that a wide range of combinations of pending request mechanisms and logging mechanisms may be used to implement the current invention. Any suitable logic that uses the state of the logging mechanism together with the state of the pending request mechanism may be used to determine the next request to be sent to the shared resource

An exemplary embodiment of the invention uses a memory as a shared resource150. Memory system300inFIG. 3Afrom U.S. Pat. No. 7,320,047 is one example of a prior art memory system that can be used with an exemplary embodiment of the invention. The memory system300includes a memory controller310, a plurality of point-to-point links320a-320n,a plurality of memory subsystems330a-330c,and a plurality of memory subsystem ports378a-378non an interface375. A more detailed depiction of memory subsystem330ais illustrated as memory subsystem340inFIG. 3A. Memory subsystem340includes a buffer device350and a plurality of memory devices360disposed on the memory subsystem340. The buffer device350is coupled to the plurality of memory devices360via channels370.

Memory system300B depicted inFIG. 3Bdepicts an exemplary embodiment that may be implemented with the prior art memory system depicted inFIG. 3A. Similar toFIG. 3A,FIG. 3Bdepicts a memory system300B that includes a memory controller310B, a plurality of point-to-point links320aB-320nB, a plurality of memory subsystems330aB-330cB, and a plurality of memory subsystem ports378aB-378nB on an interface375B. A more detailed depiction of memory subsystem330aB is illustrated as memory subsystem340B inFIG. 3B. Memory subsystem340B includes a buffer device350B and a plurality of memory devices360B disposed on the memory subsystem340B. The buffer device350B is coupled to the plurality of memory devices360B via channels370B. The memory controller310B depicted inFIG. 3Balso includes next request priority logic120,130, shown as element390B inFIG. 3B. The depicted exemplary embodiment locates the next request priority module390B in the memory controller310B but it should be understood that it could also be located in the memory buffer device350B or in the memory devices360B. Exemplary embodiments are not limited to use in memory systems, as any other shared resource can be controlled in a similar way.

FIG. 4depicts a generalized flowchart for choosing the next request to be sent to the shared system resource150that may be implemented by an exemplary embodiment of the invention.FIG. 4illustrates the functions implemented in the next request priority logic122which manages the virtual server control logic121and the request interface140. At block410, the next request priority logic122examines the next entry in the request queue. As described before, in one exemplary embodiment of the present invention the next entry is the oldest entry in the request queue. At block420, it is determined if the requestor identifier associated with the next entry matches any of the entries in the previous requestor queue. In the exemplary embodiment depicted inFIG. 4, the previous requester queue is implemented by a four deep first-in-first-out buffer that contains the requestor identifiers of the last four requestors who were granted priority to the shared resource. In this example, four is the specified number of most recent priority grants, in other embodiments other specified numbers are implemented.

If it is determined, at block420, that the requestor identifier matches any of the requester identifiers in the previous requester queue, then processing continues at block450where it is determined if the next request is the last request in the request queue. In this case the requester should be granted priority to the shared resource150because no other requesters are waiting for the shared resource150; the requester is granted priority to the shared resource at block460. In an alternate exemplary embodiment, a different action is taken when there is only one request in the request queue and its associated requester identifier has been found in the previous requestor queue. In this alternate exemplary embodiment, no priority is given to the next request and no request is sent to the shared resource150in the next cycle. This might be desirable when the service time for the request is long in comparison to the number of requests that the shared resource150can handle at once.

If it is determined at block450that the next request is not the last request in the request queue, then another request in the queue is designated as the next request (e.g., based on it being the next oldest request, or some other priority scheme) and processing continues at block410.

It is determined, at block420, that the requestor identifier does not match any of the requester identifiers in the previous requester queue, then processing continues at block430. At block430, the next request is granted priority to the shared resource150. At block440, the requester identifier associated with the next request is added to the previous requester queue. In an exemplary embodiment of the invention, the new requester identifier replaces the oldest entry in the previous requester queue. The request that is granted priority becomes the next request to be sent to the shared resource150via interface140.

FIG. 4depicts one processing flow that may be implemented by an exemplary embodiment and other processing flows may be implemented by alternative exemplary embodiments of the present invention. For example, in another exemplary embodiment of the invention, a priority function which favors requests from one or more particular virtual servers over the requests of all others is implemented by assigning a requester identifier to one or more virtual servers and by selecting how many previous requester identifiers to save in the previous requestor queue (or other logging mechanism). In one example, a particular virtual server could be guaranteed at least ten percent of request slots for a memory controller that is shared among many CPUs.

In another exemplary embodiment of the invention a request tag that is similar in function to the VSID (an example of a requestor identifier) is used, but it is produced by a different layer of software. An exemplary embodiment uses a VSID that is produced by a hypervisor, however, any individual operating system could produce a set of process IDs (PIDs) that identify individual units of work running on that operating system. These PIDs are another type of requester identifier that could be implemented and they could be written directly to the same hardware registers as the VSIDs during dispatch.

In another exemplary embodiment, the hardware is designed to intercept an attempt by the operating system to change the VSID of the CPU and then pass control of the CPU to a hypervisor. In this case, the hypervisor would use any number of algorithms to determine what value the VSID hardware register should be set to.

Technical effects and benefits include the ability for a physical computer server to provide a reliable level of high performance when running a large number of virtual servers. The server allows a hypervisor or other control program to set limits on the hardware resources that can be consumed by a virtual server or by a group of virtual servers. This provides the ability to limit the maximum share of resources that can be consumed by a single virtual server and may result in increased performance of the processor.

As will be appreciated by one skilled in the art, the present invention may be embodied as a system, method or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module” or “system.” Furthermore, the present invention may take the form of a computer program product embodied in any tangible medium of expression having computer-usable program code embodied in the medium. Any combination of one or more computer-usable or computer-readable medium(s) may be utilized. The computer-usable or computer-readable medium may be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CDROM), an optical storage device, a transmission media such as those supporting the Internet or an intranet, or a magnetic storage device. Note that the computer-usable or computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory. In the context of this document, a computer-usable or computer-readable medium may be any medium that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-usable medium may include a propagated data signal with the computer-usable program code embodied therewith, either in baseband or as part of a carrier wave. The computer usable program code may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc.

As described above, embodiments can be embodied in the form of computer-implemented processes and apparatuses for practicing those processes. In exemplary embodiments, the invention is embodied in computer program code executed by one or more network elements. Embodiments include a computer program product500as depicted inFIG. 5on a computer usable medium502with computer program code logic504containing instructions embodied in tangible media as an article of manufacture. Exemplary articles of manufacture for computer usable medium502may include floppy diskettes, CD-ROMs, hard drives, universal serial bus (USB) flash drives, or any other computer-readable storage medium, wherein, when the computer program code logic504is loaded into and executed by a computer, the computer becomes an apparatus for practicing the invention. Embodiments include computer program code logic504, for example, whether stored in a storage medium, loaded into and/or executed by a computer, or transmitted over some transmission medium, such as over electrical wiring or cabling, through fiber optics, or via electromagnetic radiation, wherein, when the computer program code logic504is loaded into and executed by a computer, the computer becomes an apparatus for practicing the invention. When implemented on a general-purpose microprocessor, the computer program code logic504segments configure the microprocessor to create specific logic circuits.