Framework and method for monitoring performance of virtualized systems based on hardware base tool

The disclosed invention involves a framework and method based on hardware base tool to monitor the performance of virtualized systems, wherein the said framework comprises at least one master host, and each of the said master host comprises user space components, guest space components, kernel space components and hardware. The said user space components comprise policy manager, workload mediator, monitor library, and host performance monitor. The said host performance monitor is connected to workload mediator, and host performance monitor comprises user space monitor and kernel space monitor. The disclosed invention applies PMU or the similar tools to monitor the performance of virtualized systems. The performance monitoring of the disclosed invention is to monitor CPU, memory, cache, IO, network, processes, etc. of the host of virtualized systems. Meanwhile, the method based on hardware to monitor performance in this disclosed invention resolves the problem to acquire performance data for virtualized systems.

CROSS REFERENCE TO RELATED APPLICATION

This application is the national phase under 35 USC 371 of international application no. PCT/CN2012/075274, filed May 10, 2012, which claims the benefit of the priority date of Chinese application no. 201110141642.X, filed May 27, 2011. The contents of the aforementioned applications are incorporated herein in their entirety.

FIELD OF THE INVENTION

The present invention discloses a framework and method for monitoring performance of virtualized systems based on hardware base tool.

BACKGROUND

In the past, the performance of computer software system, including operating system (OS) and its applications, has been a concern of software vendors and their customers, since system performance relates directly to the service quality and success of software product sales. In general there are two angles to view the issue of performance:

The first angle views the “consequence” of application software, or the “user experience” as performance. For example, software code is inserted into applications to measure the “response time” from the return key to the response appearing on computer screen. The performance measurement code may be combined with application functions to form a complicated Performance Monitor, such as Couture's patent US20080235075 “Enterprise application performance monitors”.

The second angle views the “cause” that impacts the performance of running software. The impacting root cause is, of course, the hardware resources such as CPU, memory, IO and network. In the past there is no way to measure performance from hardware, hence the second best way can only utilize the OS performance commands to measure performance of the resources like threads, processes, etc. Lately, new hardware monitoring base tool emerges, such as x86-based hardware register capable of monitoring CPU, called Performance Monitor Unit (PMU). There are quite a few PMU patents: e.g. Davidson's U.S. Pat. No. 6,718,403 “Hierarchical selection of direct and indirect counting events in a PMU”, Mericas's U.S. Pat. No. 7,500,138 “Simplified event selection for a PMU” and Mericas's US20060167658 “Method and product of PMU for sampling all performance events generated by a processor”. There are also non-PMU patents that are hardware and performance monitoring based, such as Fowles's US20060277395 “Processor performance monitoring”; Kosche's US20080177756 “Method and apparatus for synthesizing hardware counters from performance sampling”, and Hunter's US20080294944 “Processor bus for performance monitoring with digests”.

Two kinds of registers help monitoring performance: performance control register (PMC) and performance data register (PMD). PMC monitors CPU at regular time intervals, and saves monitored data into PMD. PMC is event-based or time-based to consolidate the collected event information and report the monitoring results to higher software layers. Advantages of hardware monitoring are: (1) in the past, certain software-based monitors require to continuously change the source code of the monitored software to reach its goal, e.g. optimizing a Java virtual machine routine. On the contrary, hardware-based monitoring requires no source code. (2) The performance of monitored software is not affected by monitoring activities. (3) Very low-level kernel code can be monitored. Most importantly, (4) Capability to monitor cache is nowhere found in any other monitoring means. One example is TLB miss and hit, where TLB stands for “translation look-aside buffer”, the CPU cache hardware for memory management to improve the speed of virtual H physical address translation. TLB takes virtual address as search keyword. The search result is physical address: if the needed address can be found in TLB, it is called TLB hit. Otherwise, it is called TLB miss. Frequent TLB misses indicate the performance of resource is degraded. The details of PMU are described in the document written by Eranian, Hewlett Packard Company, “The perfmon2 interface specification”.

There are not very many PMU applications. Moreover, there is no existing method to utilize PMU and combine OS performance commands to measure the performance of “virtualized systems”. Here is the explanation of virtualized systems:

The technology of OS virtualization leads to the result of a physical machine is capable of running multiple “guest” OSs (or Virtual Machine, briefly VM, or simply “guest”). The VMs run on VMM (virtual machine monitor, or hypervisor), and Hypervisor runs directly on physical machine. In most cases, the monitored physical machines are servers (or host) in the data center. There are different OS virtualization techniques, such as para-virtualization or full virtualization. In general, a host has at least two spaces to be monitored: one is the space where VMs are running, called (1) guest space; the other is the space where a hypervisor is running, called (2) kernel space. Some virtualization technique, such as Linux KVM, keeps the use of the original (3) user space. The above explains the internals of a host. Multiple such hosts form a cluster, with a lead host called master, and several subordinate hosts called slaves. Multiple clusters become a network system, being distributed, centralized, or mixed. This explains “virtualized systems” of the disclosed invention, a network system covering large geographical areas.

Tang's patent US20110296411 “Kernel bus system to build Virtual Machine Monitor and the performance service framework & Method therefore” describes how virtualized systems analyze performance data, perform VM migration, and mediate workload. The patent however does not describe how virtualized systems acquire performance data.

SUMMARY

In order to resolve the existing technical issue, the objective of the disclosed invention is to provide a framework and method of monitoring performance of virtualized systems based on hardware base tool, to apply PMU (or similar tool) to the performance monitoring of virtualized systems. The said method monitors CPU, memory, cache, IO, network, process, etc. of multiple hosts in virtualized systems. Meanwhile, the hardware-based monitoring method of the disclosed invention resolves the issue of how to acquire performance data for virtualized systems.

One of the disclosed inventions involves the said framework and method to monitor the performance of virtualized systems based on hardware base tool, comprise at least one master host, each of the said master host includes user space components, guest space components, and kernel space components, wherein

The said user space components comprise connected policy manager, workload mediator, monitoring library, and host performance monitor. The host performance monitor is further connected with workload mediator, and the host performance monitor comprises user space monitor and kernel space monitor;

The said guest space components comprise at least one virtual machine (VM) connected with said host performance monitor via kernel serial channel;

The said kernel space components comprise performance monitor base tool application interface (API) and performance monitoring base tool core code module, connected with said monitor library, as well as task scheduler, memory management unit, network driver, file system and device driver, wherein the said network driver is connected with said workload mediator;

The said hardware supports peripheral performance monitoring base tool, wherein the hardware comprises PMU connected with said performance monitoring base tool core code module, PMD and PMC respectively connected with PMU, and CPU, memory, hard disk and network card wherein the said network card is connected with network driver.

The said framework to monitor virtualized systems based on hardware base tool comprises at least one slave host. Each of the said master host is connected with the said at least one slave host via net pipe to form a cluster. Multiple said master hosts and their respective slave hosts form multiple clusters, and said clusters communicate with each other via centralized, distributed or mixed network architecture. The said slave host comprises said host performance monitor wherein the host performance monitor of slave host is connected to the network card of said master host.

The second disclosed invention involves the said method to monitor performance of the virtualized systems based on hardware base tool, comprising:

Firstly, administrators via graphical interface, or user applications via said performance monitor APIs, make performance mediating requests to said policy manager;

Secondly, when said policy manager interprets the said performance mediating requests, it instructs the said workload mediator making request to said master host performance monitor, asking the monitor to report individual slave resource performance conditions via net pipe;

Finally, said policy manager reports overall performance condition to said administrators or said user applications.

Because of the above said technical solution, the disclosed invention is capable of monitoring performance of at least one cluster, wherein the cluster comprises at least one host and the host comprises at least one virtual machine. Therefore, the disclosed invention is also capable of monitoring the performance of hosts. The disclosed invention provides for enterprises and cloud service providers a framework and method to monitor the performance of virtualized systems. The disclosed invention is not about the PMU hardware itself. Rather, it uses a method based on hardware base tool to monitor, calls the APIs of the tool to realize the method, and builds a software performance monitor. System administrators may also interact with the interface provided by the monitor, e.g. an application to adjust virtual and physical resources may call the said monitor interface to obtain the rough ideas of idle/busy situations of the entire clusters, and proceed to adjust these resources.

DETAILED DESCRIPTION

The followings are detailed description of the preferred embodiments combining with their drawings.

One of the embodiments comprises multiple clusters, with each cluster comprising multiple hosts, and each host multiple virtual machines. Between clusters there may be centralized, distributed or mixed communications. Between hosts there is net-pipe (e.g. TCP/IP) communication amongst a leading master host, and several subordinate slave hosts. Between host and its virtual machine there is virtual serial port as connection pipe. The serial port pipe means that virtual machine takes this “pipe (which is really just a guest OS system call)” as a virtual serial port. In kernel space, the system call running via hypervisor delivers data to the serial port. The delivering speed is fast enough to avoid slow external network communication.

Refer toFIG. 1a, which shows a block diagram of an embodiment as one of the disclosed invention. The framework of this disclosed invention comprises at least one master host11and at least one slave host12, each master11connecting with at least one slave12via net pipe13to form a cluster. Multiple masters11and their respective salves12form multiple clusters which communicate to each other using centralized, distributed or mixed network architecture. In other words, although servers of this embodiment have master-slave and clustering relationship, the practical architecture may be a centralized or distributed server federation, meaning that master host11may control at least one local (distributed federation) or global (centralized federation) slave12.

Each master host11comprises user space components111, guest space components112, kernel space components113, and hardware114, wherein,

User space components111comprise connected policy manager1111, workload mediator1112, monitor library1113, and host performance monitor1114, with host performance monitor1114connecting to workload mediator1112. The host performance monitor1114comprises user space monitor11141and kernel space monitor11142running inside of kernel space. The host performance monitor1114also resides at other hosts of the virtualized systems, e.g. at slave12;

Guest space components112comprise at least one virtual machine1121which connects to host performance monitor1114via kernel serial pipe1131, meaning that there is at least one virtual machine running on top of the hypervisor. Virtual machine1121comprises VM performance agent11211which collects the performance data of VM resources;

Kernel space components113comprise a performance monitor base tool API1132connected with monitor library1113, the performance monitor base tool core code module1133, as well as other kernel components like task scheduler1134, memory management unit1135, network driver1136, file system1137, and device driver1138, wherein the network driver1136is connected with workload media1112. Some OS virtualization technique like Xen modifies the “other kernel components” as part of Hypervisor, while Linux KVM technique puts these components outside of Hypervisor;

Hardware114supports peripheral performance monitoring base tool. Hardware114comprises PMU1141connected with performance monitoring base tool core code module1133, the PMD1143and PMC1142connected respectively with PMU1141, and CPU1144, memory1145, network card1146, hard disk1147, and other devices1148wherein network card1146is connected to network driver1136;

Slave host12comprises host performance monitor1114and VM performance agent (not shown in the Fig.), etc., wherein the host performance monitor1114is connected with network card1146.

Following the descriptions of the above framework to monitor performance of virtualized system based on hardware base tool, we proceed to describe the second part of the disclosed invention, i.e. the method to monitor performance of virtualized system based on hardware base tool. The said method comprises the following steps:

Firstly, administrators via graphical interface or user applications via performance monitor API1132make request to policy manager1111to mediate performance;

Secondly, after policy manager1111interprets the request of performance mediation, it instructs workload mediator1112to make request to host performance monitor1114of master host11via net pipe13to report resource performance condition for each of its slave12;

Lastly, policy manager1111reports entire system performance conditions to user applications or administrators.

The technical details of the above said policy manager1111and workload mediator1112has been revealed in Tang's patent US20110296411 “Kernel bus system to build Virtual Machine Monitor and the performance service framework & Method therefore”. However, the disclosed invention here details the two components in the following embodiments from a viewpoint based on monitor base tool PMU.

Specifically, the requests or enquiries regarding performance come from user via policy manager1111, and enter into virtual systems. There are three types of requests:

(1) Most monitoring data are about the idle/busy conditions of resources, or monitoring data for resources like CPU, memory, IO, network, etc. The data can be acquired from “OS performance commands”. However, it is the “performance monitoring base tool” that controls when to start and end the monitoring process, and how to setup the monitoring events.

(2) Other monitoring data have to do with resource health condition, the data of cache and TLB miss, or the information regarding work-stopping of certain resources. Such data cannot be acquired from OS system performance commands, and can only be relying on performance monitoring base tool (such as PMU). In other words, in such situation performance monitoring base tool is used alone.

(3) Mix idle/busy condition and health condition. For example, (3) process monitoring relies only on performance monitoring base tool (such as PMU), but the acquired monitoring data mix idle/busy condition and health condition.

Therefore, in order to realize the above functions, the disclosed invention has two features: (1) capable of monitoring VM performance; (2) applying hardware-based performance monitoring base tool (although the embodiments uses PMU as base tool, none-PMU tool can be used as well), including sampling methods based on time or events.

The above two features may be described from an architecture viewpoint, namely the disclosed invention comprises two layers of performance monitoring architecture: top layer and bottom layer.

Here is the description of top layer architecture: top layer is a hierarchical object-oriented class diagram covering entire network topology of the “virtualized systems”. Policy manager is the “ancestor”, inherited by workload mediator, which is in turn inherited by host performance monitor. VM performance monitor and host performance monitor eventually are inherited by system internal and external resource monitors. system internal and external resource monitors use OS system performance commands to catch data, and employ performance monitoring base tool to monitor based on time and events (Notice that VM performance data are acquired by combining monitoring data from internal and external parts of virtualized systems. Therefore, it is an important feature of the disclosed invention to be capable of monitoring internal and external system resources. In other words, the virtualized systems of the disclosed invention include system internal resource monitor and system external resource monitor).

Refer toFIG. 1bwhich shows top layer architecture class diagram. In other words, it is an object-oriented engineering class diagram for policy manager1111and its inherited components. Specifically, policy1111, workload mediator1112and the host performance monitor1114in the master host form the top-layer architecture, wherein:

The classes of top layer is policy manager1111, where each policy manager1111is the set of at least one workload mediator1112(expressed by, where the end near ♦ is a set, and other end is the member of that set);

Each workload mediator1112is a set of at least one host performance monitor1114;

Each host performance monitor1114is a set comprising at least one resource monitor0106(e.g. user space monitor11141and kernel space monitor11142), and VM performance agent11211;

The most important class is host performance monitor1114. It is the base class for all user space monitor11141, kernel space monitor11142, and VM performance agent11211. It is also inherited respectively by system internal resource monitor104, and system external resource monitor105(expressed by, where the end nearis inherited class, the other end is inheriting class).

Therefore, host performance monitor1114can be a set, and comprises at least one member of the set. Moreover, host performance monitor1114is also an abstract concept of monitoring function, where all those inherited classes such as system internal resource monitor104, system external resource monitor105, more or less possess the same function as the host performance monitor. The member in the set of host performance monitor1114, also possess the function of internal and external monitoring.

Refer toFIG. 2, which shows the flowchart of policy manager1111.

In summary, policy manager is used to interpret user policy request, specifying the performance boundary value of CPU, memory, IO, network, and cache. It divides the user policy requests into three levels: high, median, and low. It then provides event configuration information, delivers policy to workload mediator1112, and returns resource address and conditions after the data of virtualized systems are collected. If the master host11of a cluster is unable to satisfy the user policy request, policy manager1111simultaneously or in the order of priority, enquires policy manager1111of the master host of other clusters according the policy of enquiring cluster in order to satisfy the user policy request. In other words, if local cluster has no resource to satisfy the policy request, policy manager1111enquires resource satisfying the request to the policy manager1111of the master host11of other clusters via net pipe13(e.g. TCP/IP). As for the order of enquiries, it can be sequential or simultaneous, depending on the rules of this kind in the policy base. The rules may involve cluster management method, e.g. the method for centralized clusters, distributed clusters, or mixed clusters.

The flow of policy manager1111is as follows:

Step202, determine if operation is to accept policy request. If yes, proceed to step203. Otherwise, proceed to step204;

Step203, interpret policy request. Specify the boundary performance value of CPU, memory, IO, network, and cache (e.g. high, median or low). Proceed to step212;

Step204, determine if the operation is to deliver policy request. If yes, proceed to step205. Otherwise, proceed to step208;

Step205, deliver policy to workload mediator1112. Proceed to step206;

Step206, workload mediator1112enquires resources satisfying the request to the host via net pipe13(e.g. tcp/ip). Proceed to step207;

Step207; execute policy to resource satisfying request. Proceed n to step212;

Step208, determine if operation is to respond to policy request. If yes, proceed to step209. Otherwise proceed to step213;

Step209, determine if any host in the cluster satisfies the request from the reports of workload mediator1112. If yes, proceed to step211. Otherwise proceed to step210;

Step210, policy manager1111enquiries to the policy manager1111of the master host11of other cluster, if there is resource satisfying request in that cluster via net pipe13(e.g. tcp/ip). Proceed to step212;

Step211, return resource address and conditions. Proceed to step212;

Refer toFIG. 3a, which shows the flowchart of workload mediator1112.

In summary, workload mediator1112accepts policy request from policy manager1111via net pipe13, collects resource performance data from the slave hosts12of the clusters in the virtualized systems, and executes the policy for the individual slave host12according to the performance of the entire clusters by consolidating the collected performance data, e.g. add/delete host, add/delete VM resource, or discover/handle anomaly.

The flow of workload mediator1112is as follows:

Step301, workload mediator1112initializes itself, including activating monitoring library1113. After monitoring library1113is activated, (1) workload mediator1112delivers in time the PMU event configuration information to monitoring library1113, e.g. PMC event definition configuration and configuration of monitoring time length, as well as the information of resources to be collected; (2) Create monitors, monitored objects, and events as needed, according to the known configuration information from monitor library to configure PMU;

Step302, determine if policy is to handle resource mediating request. If yes, proceed to step303. Otherwise, proceed to step308;

Step303, determine if all the host performance monitor1114are enquired. If yes, proceed to step304. Otherwise, proceed to step307;

Step304, enquire host performance monitor1114of the next host vianet pipe13. Proceed to step305;

Step305, determine if the resource request can be satisfied. If yes, proceed to step306. Otherwise proceed to step303;

Step306, host performance monitor1114handles the request to mediate resources. Proceed to step303;

Step307, the workload mediator1112at the master host11collects its concerned data. A data consolidation enables the understanding of the conditions of individual host and entire cluster of other clusters. Thus the satisfactory host is reported back (e.g. two most-idle hosts), or none of the host is reported. Proceed to step318;

Step308, determine if the policy is to add/delete host. If yes, proceed to step309. Otherwise, proceed to step310;

Step310, determine if policy is add/delete VM resource. If yes, proceed to step311. Otherwise, proceed to step313;

Step313, determine if policy is to discover and handle anomaly. If yes, proceed to step314. Otherwise, proceed to step319;

Step314, determine if some resource goes from busy to not-working state. If yes, proceed to step315. Otherwise repeat step314;

Step315, VM performance agent11211discovers anomaly of VM resource, the user space portion of host performance monitor1114discovers anomaly of host application, or its kernel space portion discovers anomaly of kernel state. Proceed to step316;

Step316, inform workload mediator1112the abnormal condition of that resource via host performance monitor1114. Proceed to step317;

Step317, policy manager1111handles anomaly according to the anomaly information. Proceed to step318;

Refer toFIG. 3b, which shows the state diagram of workload mediator1112.

The collected resource performance data by workload mediator1112are used to determine the idle/busy states of slave host12, while the change of states also shows the anomaly and recovery health conditions. Workload mediator1112also consolidates the performance of entire cluster according to the collected performance data of individual slave hosts12of the cluster where the master host11resides. The host resource state includes: resource idle state31, resource busy state32and resource not-working state33.

If performance data is over threshold, then state changes from resource idle to resource busy; if performance data is below threshold, then state change from resource busy to resource idle; if state changes from resources busy to resource not-working, then there is anomaly; and if state changes from resource not-working to resource busy, then the resource returns to normal.

Refer toFIG. 4, which shows the flowchart of host performance monitor1114.

In summary, host performance monitor1114is responsible monitoring user space resources (i.e. application processes), guest space resources (VMs) and host kernel space resources. It also activates, configures, and stops host kernel space monitor11142, wherein:

Monitor user space, meaning that monitoring performance information of all processes using system external resource monitor105;

Monitor kernel space, including activating, configuring and stopping kernel space monitor11142, and executing system internal resource monitor via system internal resource monitor104;

Monitor user space, meaning that firstly, acquiring performance data from VM1121, via Hypervisor-internal kernel serial port1131, and the interaction amongst VM performance agents11211running at their individual VM1121, thus VM performance agent11211monitoring VM resources via system internal resource monitor104; secondly, monitoring VM1121from hypervisor via system external resource monitor105, and acquiring process performance by taking that VM1121as a process, in order to match the acquire data from VM performance agent11211, and to determine if VM1121satisfies performance request.

The flow of host performance monitor1114is as follows:

Step401, host performance monitor1114initializes itself Proceed to step402;

Step402, if the operation is monitoring user space. If yes, proceed to step403. Otherwise, proceed to step404;

Step0403, enter user space monitor11141. Collect concerned performance information of all processes in the user space, from system external resource monitor105and at certain time interval. Proceed to step416;

Step404, determine if operation is to manage kernel space monitor11142. If yes, proceed to step405. Otherwise, proceed to step406;

Step405, activate, configure, and stop kernel space monitor11142. Proceed to step416;

Step406, determine if operation is to monitor kernel space. If yes, proceed to step407. Otherwise, proceed to step408;

Step407, execute monitoring internal resources like CPU, memory, IO, network, and cache using system internal resource monitor104. Proceed to step416;

Step408, determine if operation is to monitor resources of all VMs1121of the current host. If yes, proceed to step409. Otherwise, proceed to step417;

Step409, start entering kernel space monitor11142. Determine if all VM performance agents11211are enquired. If yes, proceed to step415. Otherwise, proceed to step410;

Step410, enquire next VM performance agent11211via kernel serial port1131. Proceed to step411;

Step412, system external resource monitor105conduct monitoring by taking VM1121as process. Proceed to step413;

Step413, consolidate internal and external monitoring information of VM1121to mediate the resource request. Proceed to step414;

Step414, report if current VM1121satisfies the resource request. Proceed to step409;

Step415, host performance monitor1114collects its concerned data. After consolidating its data, it understands the condition of each individual VM1121in the guest space as well as the entire situation of VMs1121(VM1to VMn), hence it is able to report VMs1121satisfying the request (e.g. 2 most idle VM1121), or report no VM1121satisfying the request. Proceed to step416;

Refer to5a, which shows the flowchart of system internal resource monitor104.

The embodiment uses Linux OS system performance commands as examples, but the monitored Hypervisor OS may be any other OS, such as IBM AIX, z-OS or Oracle SunOS. System internal resource monitor104provides the performance data of CPU, memory, IO, network, and cache. It monitors host kernel space, while VM performance agent11211monitors the internal of VM1121. Here is the difference a PMU makes: VM performance agent11211is unable to use kernel-level sampling and kernel-level event set for multiple events. This is because none-kernel-level monitoring consumes greater resources (e.g. more context switch), which impacts the accuracy of monitoring VM1121.

System internal resource monitor104acquires data by soliciting two relative performance data and subtracting them at certain time interval, based on performance monitor base tool; The virtual resources that is monitored by system internal resource monitor104, include: CPU resource, memory resource, IO resource, cache resource and network resource, wherein:

CPU resource comprises the total time of processes used by CPU to handle user space states, the total time of processes used by CPU to handle kernel space states, and total CPU idle time, total number of hard interrupts handled by CPU, and total number of soft interrupts handled by CPU, acquired by OS system performance commands;

Memory resource comprises information of total memory, free memory, shared memory and buffer memory acquired by OS system performance commands;

IO resource comprises the number of IO reads & writes of one or more processes, acquired by OS system performance commands;

Cache resource comprises data of cache, and TLB miss/hit acquired by said performance monitor base tool;

Network resource comprises network traffic data from network states acquired by OS system performance commands.

The flow of system internal resource monitor104is as follows:

Step501, system internal resource monitor104initializes itself Proceed to step502;

Step502, determine if operation is to monitor CPU. If yes, proceed to step503. Otherwise, proceed to step506;

Step503, set monitor begin time T1. Use cpu_usage_state to acquire the total number of processes used by CPU to handle user space states, the total time of processes used by CPU to handle kernel space states, and total CPU idle time, and total number of hard/soft interrupts handled by CPU. Proceed to step504;

Step504, determine if the monitoring end time T2is reached. If yes, proceed to step0505. Otherwise, repeat step504;

Step505, acquire the relative data of the monitor end time T2. Subtract the data at monitor begin time T1from the data at monitor end time T2to obtain the time of each processes consumed by CPU, the number of soft interrupts, and the number of hard interrupts during the time interval of monitor begin time T1→monitor end time T2. Proceed to step518;

Step506, determine if operation is to monitor memory. If yes, proceed to step507. Otherwise, proceed to step510;

Step507, set monitor begin time T1. Acquire information of totalram, freeram, sharedram, bufferram, etc. for various memory, as well as information regarding process from sysinfo. Proceed to step518;

Step508, determine if the monitor end time T2is reached. If yes, proceed to step509. Otherwise, proceed to step508;

Step509, acquire the relative data of monitor end time T2. Subtract the data at monitor begin time T1from the data at monitor end time T2to get memory usage situation during the time interval of monitor begin time T1→monitor end time T2. Proceed to step518;

Step510, determine if the operation is to monitor IO. If yes, proceed to step509. Otherwise, proceed to step513;

Step512, collect the number of IO reads/writes of one or more processes at certain interval. Consolidate the data. Proceed to step518;

Step513, determine if operation is to monitor network. If yes, proceed to step514. Otherwise, proceed to step516;

Step515, interpret the network traffic from TCPEXT of the Linux command netstat at certain time interval. Consolidate the data. Proceed to step518;

Step516, determine if operation is to monitor cache. If yes, proceed to step0517. Otherwise, proceed to step519;

Step517, acquire the result of monitoring cache via PMU tool interface. Proceed to step518;

Refer toFIG. 5b, which shows the flowchart of system external resource monitor105.

The embodiment uses Linux OS system performance commands as examples, but the monitored Hypervisor OS may be any other OS, such as IBM AIX, z-OS or Oracle SunOS. System external resource monitor105provides performance data of user space and guest space; specifically it provides:

(1) Host user space monitoring primarily monitors the processes running in the host user space. For example, the PMU base tool helps to acquire for each process, the CPU usage data, cache usage data, and TLB miss data.

(2) Monitoring of guest space means monitoring from external when taking VM as a process. For example, QEMU is an emulator for Hypervisor. QEMU uses software to emulate various CPU main boards (e.g. x86 Mainstone). Therefore, some OS virtualization technique is to run VM code on top of QEMU emulator, while QEMU is also a process running on Linux. So here are the guest-space monitoring activities: Monitor all processes of (QEMU+VM), acquire CPU usage data for processes in the user space, buffer the usage data and TLB miss data, and observe if VM CPU usage data is over said threshold so as to control CPU usage for that VM with other OS commands. This is the monitoring of the guest space.

It seems simple to monitor VM from the above description, but not so for virtualization technique such as Linux KVM, where the running of QEMU involves mode switch between user space and kernel space, hence the required accuracy of monitoring data is more complex. For PMU to monitor VM from external, kernel level sampling and kernel level event set can still be used. When performance data of VM running is over some threshold of CPU percentage, the Linux command cgroup can still be used to restrict the running of that VM, thus can remedy the deficiency that a VM agent cannot see the global condition from internal.

System external resource monitor105acquires data by soliciting two relative performance data and subtracting them at certain time interval, based on performance monitor base tool; the flow of system external resource monitor105is as follows:

Step520, system external resource monitor105initializes itself. Proceed to step521;

Step521, determine if operation is monitoring user space. If yes, proceed to step522. Otherwise, proceed to step525;

Step522, set monitoring begin time T1. Apply PMU tool API (application interface): “perf monitor-e cpu,cache,tlb_missing-pid target” to obtain CPU usage data, cache usage data, and TLB miss data for a process. Proceed to step523;

Step523, determine if the monitoring end time T2is reached. If yes, proceed to step524. Otherwise, repeat step523;

Step524, acquire the relative data for monitoring end time T2. Subtract the data at monitor begin time T1from the data at monitor end time T2to get CPU usage data, cache usage data and TLB miss data for each process during the interval monitoring begin time T1→monitoring end time T2. Proceed to step529;

Step525, determine if the operation is to monitor guest space. If yes, proceed to step0526. Otherwise proceed to step0530;

Step0526, set monitoring begin time T1. Apply PMU tool API: “perf monitor-e cpu,cache,tlb_missing-pid target” to get CPU usage data, cache usage data and TLB miss data of process (QEMU+VM). Proceed to step527;

Step527, determine if the monitor end time T2is reached. If yes, proceed to step528. Otherwise, repeat step527;

Step528, acquire the relative data at the monitor end time T2. Subtract the data at monitor begin time T1from the data at monitor end time T2to get CPU usage data, cache usage data and TLB miss data of process (QEMU+VM) during the time interval from monitor begin time T1→monitor end time T2;

Here is the description of bottom layer architecture: Bottom layer is implemented only at single host. The bottom layer is an object-oriented hierarchy where “ancestor” is “monitor library”, and its descendants inherit from it, in the order of monitor, host performance monitor, system internal/external resource monitor. Monitor is also a collection class comprising at least on monitored object, and a monitored object comprising at least one event. An event acquires resource performance data from the performance monitor base tool based on event to sample or based on time to sample. It used the following 6 method to monitor: monitor based on single event, monitor based on n types of events, monitor based on event probability, monitor based on the number of events during certain time interval, monitor based on n types of events, and monitor based on resource utilization. Therefore, the details of an event are another important feature of the disclosed invention.

Refer toFIG. 6, which shows the bottom layer class diagram, or the object-oriented engineering class diagram for monitor library and its inherited components. Specifically, monitor library1113, its inherited components and the host performance monitor1114in master host11form a bottom layer, wherein:

Monitor library1113is the top layer class, and its inherited components are monitors602;

Monitor602is the most important class. It is the base class for all other monitors, and is inherited by host performance monitor1114(expressed by, where the end nearis inherited class, and the other end is inheriting class);

Each monitor602is a set of at least one monitored object604, expressed by, where the end near ♦ is a set, and the other end is a member of the set);

Each monitored object604is a set of at least on event605, meaning it contains one or more events605;

Each event605means an atom monitoring event.

From the above description, monitor602may be an abstract concept of a monitoring function, and all inherited classes like host performance monitor1114, VM performance agent11211, etc. have more or less the same function to monitor resource performance. Monitor602is also a set, containing at least one member of the set.

Refer toFIG. 7, which shows a flowchart of monitor library1113.

In summary, monitor library1113is responsible for activating all monitor processes, creating a monitor602for all processes monitoring resources, and loading monitor602to the monitor library1113. When monitor library1113process starts, it decomposes the resources that can be monitored by the performance monitor base tool into atom resources that can be monitored for all monitor602, monitors them, and dispatches resource monitoring information to each individual monitor602. Monitor library1113is also responsible to stop all monitor processes, or delete all monitor602, or activate individual monitor602, or delete individual monitor602. When the performance monitor base tool (i.e. PMU) discovers an overflow of CPU interrupts, PMU reports the anomaly to monitor602.

The flow of monitor library1113is as follows:

Step702, determine if operation is to monitor process starts running. If yes, proceed to step703. Otherwise, proceed to step705;

Step703, create a monitor602for all processes monitoring resources and load the monitor in library. Proceed to step704;

Step704, monitor library1113process starts: all monitors602decompose resources that can be monitored by performance monitor base tool into atom resources, and monitor them. Dispatch resource monitoring information to each individual monitor. Proceed to step722;

Step705, determine if operation is to monitor process stop running. If yes, proceed to step706. Otherwise, proceed to step711;

Step706, determine if all monitors602are iterated. If yes, proceed to step707. Otherwise, proceed to step708;

Step708, enter next monitor602. Proceed to step709;

Step709, end monitoring all monitored object604by the monitor602. Proceed to step710;

Step711, determine if operation is add monitor. If yes, proceed to step712. Otherwise, proceed to step713;

Step713, determine if operation is to delete monitor602. If yes, proceed to step714. Otherwise, proceed to step719;

Step714, determine if all monitored object604are iterated. If yes, proceed to step715. Otherwise, proceed to step716;

Step716, enter next monitored object604. Proceed to step717;

Step717, stop the monitored object604. Proceed to step718;

Step718, delete the monitored object604. Proceed to step714;

Step719, determine if operation is to inform monitor602. If yes, proceed to step720. Otherwise, proceed to step723;

Step721, PMU informs monitor602the anomaly of the resource. Proceed to step722;

Refer toFIG. 8, which shows the flowchart of monitor602.

In summary, monitor602is an abstract concept, has multiple basic monitoring function, and is suited for host performance monitor1114, VM performance agent11211, user space monitor11141, and kernel space monitor11142. The basic functions comprise save configuration, load configuration, add monitored object604of monitor602, delete monitored object604of monitor602, start monitoring all monitored objects604, end monitoring all monitored objects604, save monitoring data and refresh monitoring data.

The flow of monitor602is as follows:

Step802, determine if operation is to save configuration. If yes, proceed to step803. Otherwise, proceed to step805;

Step803, collect current configuration message. Proceed to step804;

Step804, write to configuration file. Proceed to step826;

Step805, determine if operation is to load configuration. If yes, proceed to step0806. Otherwise, proceed to step807;

Step806, read configuration file and load the configurations. Proceed to step826;

Step807, determine if operation is to add monitored object604of the monitor602. If yes, proceed to step808. Otherwise, proceed to step811;

Step809, add the monitored object of the monitor602. Proceed to step826;

Step811, determine if operation is delete monitored object604of the monitor602. If yes, proceed to step812. Otherwise, proceed to step814;

Step814, determine if operation is to start monitoring all monitored objects604. If yes, proceed to step815. Otherwise, proceed to step817;

Step815, determine if all monitored objects604are iterated. If yes, proceed to step826. Otherwise, proceed to step816;

Step816, start the monitored object604. Proceed to step815;

Step817, determine if operation is to stop monitoring all monitored objects604. If yes, proceed to step818. Otherwise, proceed to step820;

Step818, determine if all monitored objects604are iterated. If yes, proceed to step826. Otherwise, proceed to step819;

Step819, stop the monitored object604. Proceed to step818;

Step820, determine if operation is to save monitoring data. If yes, proceed to step821. Otherwise, proceed to step823;

Step821, determine if the time to save data is reached. If yes, proceed to step822. Otherwise, repeat step821;

Step822, write the monitoring information to file. Proceed to step826;

Step823, determine if operation is to refresh new monitoring data. If yes, proceed to step824. Otherwise proceed to step827;

Step824, determine if refresh interval is reached. If yes, proceed to step825. Otherwise, repeat step824;

Step825, read the newest monitor information. Proceed to step826;

Refer toFIG. 9, which shows the flowchart of monitored object604.

In summary, the most important basic function of a monitored object604is to create monitored object604. Create monitored object604according to the configurations in the configuration file. The configuration file contains the objects that can be monitored by performance monitor base tool. Examples are CPU, memory and cache, etc. The other basic functions of monitored object604are to add monitored object604, delete monitored object604and release monitored object604.

The flow of monitored object604is as follows:

Step902, determine if operation is to create monitored object604. If yes, proceed to step903. Otherwise, proceed to step0907;

Step904, enter next event605. Proceed to step905;

Step906, create monitored object604according to the configurations in the configuration file. The configuration file contains objects that can be monitored by performance monitor base tool. Examples are CPU, memory, cache, etc. Proceed to step917;

Step907, determine if operation is to add event605. If yes, proceed to step908. Otherwise, proceed to step910;

Step908, obtain type and value of event605. Proceed to step909;

Step910, determine if operation is to delete event605. If yes, proceed to step911. Otherwise, proceed to step912;

Step912, determine if operation is to release monitored objects604. If yes, proceed to step913. Otherwise, proceed to step918;

Step913, determine if all events605are deleted. If yes, proceed to step916. Otherwise, proceed to step914;

Step914, enter next event605. Proceed to step915;

Step916, release monitored object604. Proceed to step917;

Refer to10, which shows the flowchart of event605.

In summary, monitor602uses 6 methods based on event605or based on time:

(1) base on a single event605: set the PMD threshold as maximum value minus 1. If the threshold is reached, trigger monitor602to report.

(2) based on n times of events605: set the PMD threshold as maximum value minus n, where n is a nature number greater than 1. If the threshold is reached, trigger monitor602to report.

(3) based on event605probability: set the PMD threshold as maximum value minus a random number of n events. If the threshold is reached, trigger monitor602to report.

(4) based on the number of events within certain time: accumulate the event number of triggering CPU until the clock reach certain specific time interval, and report the total number of events at various time intervals.

(5) based on multiple types of event605(assume there exist n types of events): accumulate the number of type 1 event in interval 1, . . . , accumulate the number of type n event in interval n, accumulate the number of type n+1 event in interval n+1, . . . , accumulate the number of type 2n event in interval n, accumulate the number of type 1 event in interval (m−1)×n+1, . . . , accumulate the number of type n event in interval m×n, until interval m×n reach the tally time. Then multiply n to the number total at individual time interval for event type 1, . . . , multiply n to the number total at individual time interval for event type n. Finally, report the resulted number for each type of events.

(6) based on resource utilization: Let N be total # of statistical instructions, report the utilization between time T1and T2=N/((T2−T1)×total number CPU is interrupted).

The flow of event605is as follows:

Step1002, determine if operation is to monitor based on single event605. If yes, proceed to step1003. Otherwise, proceed to step1004;

Step1004, determine if operation is to monitor based on n events605. If yes, proceed to step1005. Otherwise, proceed to step1006;

Step1006, determine if operation is to monitor based on the probability of event605. If yes, proceed to step1007. Otherwise, proceed to step1009;

Step1007, PMD threshold=MAX−random number of n events. Proceed to step1008;

Step1008, when PMD variable reaches PMD threshold, PMD triggers CPU to generate event605. Proceed to step1021;

Step1009, determine if operation is to monitor based on the number of events at some time interval. If yes, proceed to step1010. Otherwise, proceed to step1012;

Step1010, tally the number of events when PMD triggers CPU. Proceed to step1011;

Step1011, clock triggers CPU to generate event605, report the total number of events at individual time intervals. Proceed to step1021;

Step1012, determine if operation is to monitor based on multiple types of events605. If yes, proceed to step1013. Otherwise, proceed to step1018;

Step1013, PMD tallies the number of event605trigger CPU at the time interval n. Proceed to step1014;

Step1014, clock trigger CPU to generate event605. Proceed to step1015;

Step1016, the MUX total of each individual event605is multiplied by total number of event type. Report for each type of event605the result that is multiplied by total number of event type. Proceed to step1021;

Step1017, count MUX total of nthtype of event at each time interval. Proceed to step1015;

Step1018, determine if operation is to monitor based on utilization. If yes, proceed to step1019. Otherwise, proceed to step1022;

Step1019, Let N be total # of statistical instructions, report the utilization between time T1and T2=N/((T2−T1)×total number CPU is interrupted). Proceed to step1020;

In a practical implementation environment, the performance monitor for virtualized systems of the disclosed invention is not restricted to any commercial hypervisor, which may be TVM from Transoft (shanghai), Inc., Xen from Citrix, or ESX from VMWare, etc. The guest OS may be Windows, Linux, Android, or other cell phone OS. In addition, although the performance monitor method of this disclosed invention in many embodiments uses performance monitoring base tool of PMU running on x86 architecture, the method covers other x86 or none-x86 (e.g. IBM and Oracle) monitoring tools.

According to the above flow description, the said method of performance monitoring for virtualized systems provides techniques of resource performance monitoring for enterprise private cloud. On one hand, the technique is based on hardware to provide performance and event data such that administrators are able to understand the idle/busy conditions of entire virtualized systems and mediates workload, or the upper layer applications are able to call API of the method to realize automation of workload mediation. On the other hand, the cloud provider of public cloud may also use the sais administrator manual means and application automatic means to take advantage of the disclosed invention, with no restriction of multi-tenant or multi-data centers of large geographical network architecture (e.g. distributed or centralized federations).

Therefore, the disclosed invention combines performance monitoring base tool based on hardware and VM performance agents running of various servers as well as host performance monitors, to provide a framework and method of performance monitoring for virtualized systems. The disclosed invention is able to monitor VM performance, sample data based on events, time or mix of both, to realize a new performance monitoring method for virtualized systems.

The embodiments and their illustrative diagrams describe in details for the disclosed invention. Those skilled in the art may make various modified examples according to the description. Therefore, the details of the embodiments do not limit the invention. The disclosed invention takes the defined coverage by attached claims to be its protected scope.