Fault management in virtual computing environments

A method and system for fault management in virtual computing environments are provided. The system includes: a monitor of fault events from virtual machines and host devices in a virtual computing environment and a monitor of situation events in the virtual computing environment wherein the situation events refer to a virtual machine name and a host device name. A universal faults component determines if a fault event affects both virtual machines and host devices and a correlating component correlates fault events relating to virtual machines and host devices as relating to the same problem. Host machine fault events correlated to a problem are marked as a root event and virtual machine fault events correlated to a problem are marked as symptoms with a link to the root event.

RELATED APPLICATIONS

This application is based on and claims the benefit of priority from European Patent Application EP 09174602.4, filed Oct. 30, 2009.

BACKGROUND

This invention relates to the field of fault management. In particular, the invention relates to fault management in virtual computing environments.

It is common to run operating systems in virtual environments. These in turn are used to run applications that implement a range of services. Each Virtual Machine (VM) directly replicates a physical computer but is run under a hypervisor on a physical host machine. A host machine can host several VMs. To maximise host machine utilization and increase fault tolerance, VMs are often run on a cluster of host machines. If one host machine fails then the VMs can be moved (or migrated) to run on another host machine in the cluster.

Faults may occur on VMs in a similar way to how they occur on physical machines. Fault management systems can be used to detect and monitor these problems and report them to an operator allowing rapid resolution. For example, IBM Netcool is a service level management system that collects enterprise-wide event information from many different network data sources including fault events (IBM and Netcool are trade marks of International Business Machines Corporation).

In a virtual environment, faults may be caused by faults on the host hypervisor system that is running the VM. If many VMs are being run by a single host this can potentially result in a flood of faults being reported that are not caused by faults on the VMs themselves. This can be confusing and time consuming for an operator to work through and fix quickly. Furthermore, even if hypervisor fault monitoring is also implemented this (often less severe) root cause fault can be lost in the flood of VM fault events and overlooked by the operator.

In addition, one way of resolving some faults on VMs is to move them to a different physical host machine. This will result in an instant resolution of some problems but traditional fault monitoring systems running on these VMs can be slow to update this change in status and clear the problem.

BRIEF SUMMARY

According to one embodiment of the present invention there is provided a method for fault management in virtual computing environments, including monitoring fault events from virtual machines and host devices in a virtual computing environment, monitoring situation events in the virtual computing environment where the situation events refer to a virtual machine name and a host device name, determining if a fault event affects both virtual machines and host devices, and correlating fault events relating to virtual machines and host devices as relating to the same problem.

In an embodiment, the method may include maintaining a virtual machine status table, and maintaining a mapping of virtual machine names and host device names.

In another embodiment, the method may also include inserting fault events into a table of fault events. Host machine fault events correlated to a problem may be marked as a root event in the table of fault events. Virtual machine fault events correlated to a problem may be marked as symptoms in the table of fault events and a link to the root event added in the table of fault events.

In another embodiment, when a virtual machine is moved to a new host machine, fault events for the virtual machine may be located in the table of fault events and reduced in severity. In addition, a root link may be removed.

In another embodiment, the situation events may be monitored by a monitoring system from which a probe feeds event alerts and include an indication of relations between a virtual machine and a host device and host fault events.

In another embodiment, the fault events in the virtual machines may be fed by probes running on the virtual machines and the fault events in the host machines may be fed by probes running on the host machines.

In another embodiment, the fault events in the host machines may be monitored by a monitoring system from which a probe feeds event alerts.

According to another embodiment of the present invention there is provided a computer software product for fault management in virtual computing environments, the product including a computer-readable storage medium, storing a computer in which program comprising computer-executable instructions are stored, which instructions, when read executed by a computer, monitor fault events from virtual machines and host devices in a virtual computing environment, monitor situation events in the virtual computing environment where the situation events refer to a virtual machine name and a host device name, determine if a fault event affects both virtual machines and host devices, and correlate fault events relating to virtual machines and host devices as relating to the same problem.

According to another embodiment of the present invention there is provided a system for fault management in virtual computing environments, including a processor, a monitor of fault events from virtual machines and host devices in a virtual computing environment, a monitor of situation events in the virtual computing environment where the situation events refer to a virtual machine name and a host device name, a normalising component for determining if a fault event affects both virtual machines and host devices, and a correlating component for correlating fault events relating to virtual machines and host devices as relating to the same problem.

In an embodiment, the system may include a virtual machine status table and a mapping of virtual machine names and host device names.

In another embodiment, the system may further include a table of fault events for maintaining a list of fault events relating to virtual machines or host devices. The system may include a root event marker in the table of fault events for host machine fault events correlated to a problem. The system may also include a symptom marker in the table of fault events for virtual machine fault events correlated to a problem and a link to the root event in the table of fault events.

In another embodiment, the system may include a fault resolution component where when a virtual machine is moved to a new host machine, the resolution component locates fault events for the virtual machine in the table of fault events and reduces the events in severity.

In another embodiment, the monitor for situation events may obtain the situation events from a virtual environment cluster control centre and may feed event alerts to the fault management system.

In another embodiment, the monitor for fault events may be probes running on the virtual machines and the monitor for fault events may be probes running on the host machines.

In another embodiment, the monitor for fault events in the host machines may obtain the fault events from a virtual environment cluster control centre and may feed event alerts to the fault management system.

In another embodiment, the probes may run on a remote machine and may monitor the host machine via a network.

DETAILED DESCRIPTION

A method and system are described in which fault events are gathered from both a hypervisor and the VMs and are normalised and fed into a fault management system. Data is also gathered from the hypervisor indicating which host machine each VM is running on. This information is used to perform the following actions.

1. Fault event correlation. Fault events produced by the host machine or the hypervisor (referred to as host events) and the virtual machine (referred to as VM events) which relate to the same original problem are identified. The host event that is causing the errors on the VMs is increased in severity and marked as a root cause. The corresponding errors from the VMs are reduced in severity, marked as symptom events and a field in the event is set to point to the root cause event. This allows the operator to filter out the symptom events and identify the high severity root cause event much more quickly. The problem can be more rapidly resolved, which in turn will resolve the symptom VM events. If the operator is looking at a particular symptom event then the root cause can also be rapidly identified.

2. Fault resolution after VM migration. Hardware related faults can be resolved by moving the virtual machine to a new physical host machine. When a VM is migrated to a new physical host then this class of faults are reduced in severity. Once the monitors on the VM receive the information that the fault has been cleared then the VM events will be resolved and cleared as normal. The advantage of doing this is that the operator's display of high severity fault events will be cleared far more rapidly allowing them to concentrate on any more important problems.

Referring toFIG. 1, a system100is provided including a fault management server110for handling fault events in a virtual computing environment.

The virtual computing environment includes one or more virtual machines121-126, each of which runs on a host machine operating system131,132or computer hardware which has a layer of software which is a virtual machine monitor or hypervisor141,142that allocates hardware resources dynamically and transparently. Multiple operating systems can run concurrently on a single physical computer and share hardware resources with each other. By encapsulating a machine, including central processor, memory, operating system, and network devices, a virtual machine121-126is compatible with standard operating systems, applications, and device drivers.

To maximise host machine utilization and increase fault tolerance, VMs121-126are run on a cluster130of host machines131,132. If one host machine131,132fails then the VMs can be moved (or migrated) to run on another host machine131,132in the cluster130.

The VMs121-126run probes to provide probe feeds160to report VM faults161caused by underlying hardware failure or problems (referred to as VM hardware faults) to the fault management server110. The host machines131,132may also run probes (if these are supported by the host machines) to provide probe feeds170to report host hardware faults171to the fault management server110.

The term probes is used for programs which connect to an event source such as the VM or host machine and detect and acquire event data and forward the event data to the fault management server110as alerts. Probes may use the logic specified in a rules file to manipulate the event elements before converting them into the fields of an alert in the fault management server110alerts status table. Each probe may be designed to acquire event data from a specific source. Probes may also be referred to as monitoring devices or agents for remotely or directly monitoring network devices.

For example, VMs121-126and host machines131,132may run standard OMNIbus probes160if they are Linux based or run on OMNIbus supported platforms (OMNIbus is a trade mark of International Business Machines Corporation and Linux is a trade mark of Linus Torvalds).

In an alternative embodiment, the probes may be run on a remote machine and monitor the hypervisor machine131,132via a network protocol or remote mount.

A cluster130is controlled via a control centre133and will migrate VMs121-126between the host machines131,132as required. The control centre133can be run on a VM121-126on the cluster130.

A monitor system150is provided, which has an agent151which talks to the control centre133via an API and reports situation events181to the monitor system150. A probe or monitor is used to feed180these events to the fault management server110. Situation events181keep track of which VM121-126is on which host machine131,132and also report host faults171.

In an alternative embodiment, the agent151could optionally connect to the hypervisors141,142directly instead of through the control centre133but this is less robust if an entire host machine131,132were to fail.

Host hardware faults171on the hosts131,132are reported in the situation events181via the control centre133to the monitor system150and then to the fault management server110via the probe feeds180used to report the situation events181.

Additionally, if the host machines131,132support probes, host hardware faults171may be fed direct to the fault management system110. If the hardware faults171can be fed170via probes directly to the fault management system (the option shown in broken lines inFIG. 1), the probes can be used to gather a richer set of potential errors direct from the host machines131,132.

One common and widely used hypervisor in an industrial high availability environment is VMware ESX (VMware and ESX are trade marks of VMware, Inc.). In an example embodiment, using VMware ESX hypervisor clusters, a monitor system may be used in the form of IBM Trivoli Monitor (ITM) (Trivoli is a trade mark of International Business Machines Corporation) with an agent for VMware for virtual infrastructure (VMware VI agent). The fault management server may the object server of IBM's Netcool/OMNIbus system using Event Integration Facility (EIF) probes to feed events from the VMware VI agent.

In one embodiment, the hardware faults of the VMs are reported using OMNIbus probes. The VMware ESX hypervisors are Linux based and therefore may run standard OMNIbus probes. The hardware faults on the hosts are also reported via the ESX control centre to ITM and then to the ObjectServer via the EIF probe.

Other hypervisors may be used including IBM hHype (hHype is a trade mark of International Business Machines Corporation), Microsoft HyperV (Microsoft and HyperV are trade marks of Microsoft Corporation), Kernal-based Virtual Machine on Linux, z/VM (z/VM is a trade mark of International Business Machines Corporation), and others.

The monitor agent151connects to the control centre133via its SDK API and can produce, for example, the following situation events181. Each situation is raised when it occurs and is cleared when it no longer applies.

AvailabilityThe status of the host machine server is unreachable.

CPUThe vmkernel is not loadedThe CPU utilization is very high.The CPU utilization is low.The CPU is overloaded.

DiskThe filesystem is almost full.The disk read activity is high.The disk write activity is high.

MemoryAmount of available free memory for the console OS (COS) is low.Amount of available free memory for the host machine sever is low.

NetworkThe network transmit activity is high.The network receive activity is high.

Virtual MachinesThe virtual machine is powered offThe virtual machine is in a stuck state.The virtual machine is in an unknown state.The virtual machine is in a suspended state.

With the exception of “The status of the host machine server is unreachable” each of these situations has a VM server name and a VM name attribute. This corresponds to the hardware server name and the name of the VM configured in the control centre software. This would normally correspond to the host name of the virtual machine.

Referring toFIG. 2, a block diagram shows a fault management server110. The fault management server110includes a fault event correlation component210and a fault resolution component220. In addition, the fault management server110includes a VM status table230and a table of fault events referred to as an alerts status table240for hardware faults.

The fault event correlation component210identifies fault events produced by the host or hypervisor and the virtual machine which relate to the same original problem. The host event that is causing the errors on the VMs is increased in severity and marked as a root cause. The corresponding errors from the VMs are reduced in severity, marked as symptom events and a field in the event is set to point to the root cause event.

The fault resolution component220resolves faults after VM migration. Hardware related faults may be resolved by moving the virtual machine to a new physical host machine. When a VM is migrated to a new physical host then this class of faults are reduced in severity. Once the monitors on the VM receive the information that the fault has been cleared then the VM events will be resolved and cleared as normal.

The VM status table230includes the VMs host names and VM server host names and the VM status.

An example embodiment of the status table230of the fault management server110contains four columns.

For situation events listed above (apart from “the status of the host machine is unreachable”), the probe feed will send the VM server name and the VM name attributes to the VM status table230of the fault management server110.

A first trigger231is associated with the VM status table to update the VM status from probe feeds. The first trigger231of the VM status table230takes no action if the data in the table remains unchanged. For example, if “the virtual machine is powered off”, “the virtual machine is in a stuck state”, or “the virtual machine is in a suspended state”, become true then the active column value will be 0 otherwise it will be 1.

In another embodiment, procedures may be called. For example, if the host associated with a virtual machine changes then a VM_Host_Change procedure will be called. If the state of a VM entry changes from active to in-active then a VM_Down procedure will be called. If the state of a VM entry changes from in-active to active then a VM_Restored procedure will be called. The configuration files may use signals that invoke another set of triggers to perform these actions rather than procedures.

The VM status table230includes a second temporal trigger232associated with to delete a VM status entry. This clears out unused entries from the status table230and may run, for example, once a day. Each status entry is checked and if there is no change for a given time period (for example, two weeks) then the entry is deleted. The purpose of this trigger is to prevent the VM status table from growing if temporary virtual images are being created and destroyed on a regular basis.

The status of the VM is kept up to date by the situation events181from the monitor agent.

Information is duplicated as different situations may provide the same information for the VM status table230. This is done to provide an additional degree of fault tolerance in the unlikely occurrence that a situation event is missed.

The monitoring of a cluster (VMware agent, EIF probe and ObjectServer) may be running before any VMs are started. This ensures that the VM status table230is populated correctly. If VMs are already running on the cluster when the monitoring is started then they should be suspended and resumed in order to populate the VM status table230or VMotioned to another host if uninterrupted service is required.

A probe rules file260includes a normalizing component261which performs a mapping of the situation events into a normalized format consistent with similar error events produced by other probes and can be used to insert error events and resolutions for the error situations described above. An event generation component262will be used in the rules file260to insert events into alerts status table240as well as the VM status table230.

The fault event correlation component210correlates hardware error events between the virtual machines and the hypervisor. These hardware errors161,171are gathered by probes running on the VMs and by probes or agents running on or communicating with the hypervisor and/or the host machines.

For the example configuration for VMWare then fault events will be gathered remotely via the ITM VI VMware agent.

Some groups of hardware faults161,171will affect both the hypervisor and the VMs running on it. Typical examples are high CPU use, memory failure, or shared device failure. These will be classified and normalized by a normalizing component271which identifies universal faults within the probe rules file270. For ITM VMware events this will be done in the EIF probe rules file260.

Once the error events have been inserted into alerts status table240they will be correlated using a temporal correlation trigger241that will run at regular intervals, for example, every 20 seconds.

Once host and VM events have been correlated then they may be modified to indicate this relationship. In one embodiment, the VM events242will be marked as “Symptom” events243and the host events244will be marked as “Root” events245. The Symptom events will then point to the Root events.

If a VM fails completely then it can also be the root cause of many more error events.

In the embodiment using procedures, the VM_Down and VM_Restored procedures may be used to introduce Root links for these types of error. For example, a probe running on a VM may fail and produce an error because a heartbeat is no longer received. The details and summary for the error will be updated by the VM_Down procedure and the VM_Restored procedure will again update this information and reduce the severity of the error event and/or introduce a Root link. However, in this embodiment, the error will only be cleared when the probe is running again.

The fault resolution component220resolves hardware error events. When a VM is migrated then it will update the VM status table230. This procedure will perform a scan of hardware error events associated with the VM hostname and will reduce their severity in order to indicate that they are not longer a significant problem. If a Root link was introduced to a physical host event then this will be removed.

These hardware error events will be a superset to the situation events. The error events that affect both the hypervisor and the VM are reduced in severity and will have some form of Root link that will need removing. There may also be uncorrelated events that will need their severity reducing. Finally, some hardware error events will not be resolved by migrating the VM such as low disk space and so will be left unchanged.

The proposed solution may use a VM hostname to hypervisor hostname mapping table250. This table250will be searched within several triggers. The primary key will be the VM hostname and this will be used to search the table and an efficient hash table lookup will be used by the fault management server. The highest processing overhead will be for the VM to physical host correlation trigger241. This should keep the number of traversals of alerts status table240to a minimum. However, if a hypervisor host goes down then a potentially large number of VMs could be migrated to different hosts.

Similarly, if a hardware error occurs on a hypervisor running many VMs then a potentially large number of hardware error events could be received from the VMs.

Referring toFIG. 3, an exemplary system for implementing aspects of the invention includes a data processing system300suitable for storing and/or executing program code including at least one processor301coupled directly or indirectly to memory elements through a bus system303. The memory elements can include local memory employed during actual execution of the program code, bulk storage, and cache memories which provide temporary storage of at least some program code in order to reduce the number of times code must be retrieved from bulk storage during execution.

The memory elements may include system memory302in the form of read only memory (ROM)304and random access memory (RAM)305. A basic input/output system (BIOS)306may be stored in ROM304. System software307may be stored in RAM305including operating system software308. Software applications310may also be stored in RAM305.

The system300may also include a primary storage means311such as a magnetic hard disk drive and secondary storage means312such as a magnetic disc drive and an optical disc drive. The drives and their associated computer-readable media provide non-volatile storage of computer-executable instructions, data structures, program modules and other data for the system300. Software applications may be stored on the primary and secondary storage means311,312as well as the system memory302.

The computing system300may operate in a networked environment using logical connections to one or more remote computers via a network adapter316.

Input/output devices313can be coupled to the system either directly or through intervening I/O controllers. A user may enter commands and information into the system300through input devices such as a keyboard, pointing device, or other input devices (for example, microphone, joy stick, game pad, satellite dish, scanner, or the like). Output devices may include speakers, printers, etc. A display device314is also connected to system bus303via an interface, such as video adapter315.

Referring toFIG. 4, a flow diagram400shows a method according to an embodiment of the present invention. Situation events are received401and normalized402. In parallel, VM probe events are received403(and additionally host probe events if these are also supported) and normalized404. The normalized events are inserted405as fault events into the alerts status table.

VM and host mapping information is extracted406from the situation events and the VM status table is updated407.

It is determined408if the VM to host mapping has changed. If it has not changed, the entry time is simply updated409. If it has changed, the entry time is updated and hardware faults in the alerts status table are found410for the VM that has moved.

It is then determined411if there are any suitable hardware faults for the VM. If not, the method ends412. If there are, the severity of the VM faults is reduced413and a symptom classification is removed, if it exists. The link to the host root cause fault is broken, if it exists. The method then ends414.

Referring toFIG. 5, a flow diagram500shows a method of correlation of faults which is run at regular intervals in accordance with an embodiment of the present invention.

Firstly, faults are identified501which are VM correlation candidates. This is by finding the faults in the alerts status table that meet the following points:they have not been classified as root cause or symptoms,they have not already been resolved,they have a hostname that matches one of the VM hostnames in the VM status table, andthey are of a type that could be caused by a host problem.

It is determined502if any suitable faults are found. If there are none, the method ends503. If there are such faults, for each fault in the VM correlation candidates, look up504the host server name in the VM status table. Collect together505all the host server names as the set of host server names.

Next, faults are identified506which are host correlation candidates. This is by finding the faults in the alerts status table that meet the following points:they have not been classified as symptoms,they have not already been resolved,they have a host name in the set of host server names in step505, andthey are of a type that could cause a host problem.

It is determined507if any suitable faults are found. If there are none, the method ends508. If there are such faults, for each fault in the VM correlation candidates look up509the host fault in the host correlation candidates.

It is determined510if the error type matches. If not, the method ends511. If it does match there is a pair of correlated VM and host faults512.

The VM fault is marked513as a symptom, the “local root object” field is pointed to the host fault, and the severity of the fault is lowered. The host fault is marked514as a root cause and its severity raised.

It should be noted that more than one VM fault can point to a single host root cause fault.

A fault management system may be provided as a service to a customer over a network.

The invention can take the form of a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, a computer usable or computer readable medium can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus or device. For the purposes of this description, a computer readable storage medium can be any apparatus that can contain or store the program for use by or in connection with the instruction execution system, apparatus or device. However, a computer readable storage medium is not a transitory or propagation medium.