Method and apparatus for managing software errors in a computer system

A method for managing a system includes monitoring a plurality of applications running in the system for errors. A prediction is made as to whether errors detected would result in a failure. Fault recovery is initiated in response to a failure prediction. According to one aspect of the present invention, monitoring the plurality of applications includes reading error recorders associated with error occurrence. Other embodiments are described and claimed.

FIELD

An embodiment of the present invention relates to error management of applications run on a computer system. More specifically, an embodiment of the present invention relates to a method and apparatus for detecting errors and predicting failure of applications on a computer system.

BACKGROUND

Software applications running on computer systems may experience a variety of errors that may affect its operational state. Errors which software applications may experience include, for example, errors relating to memory allocation, memory corruption, segment violation, unexpected state transitions, interprocess communication between applications, and timer related system calls. It is important for a computer system to recognize an application's operational state in order to allow for the computer system to take recovery actions and prevent the degradation of operational services.

Some software applications are capable of logging errors internally to be reviewed by the computer system user or a system manager. Other software applications are capable of generating an error report that may be transmitted outside the computer system to be reviewed by a software developer. These logs or reports typically include information about the error, such as the time it occurred and information about the nature of the error. Although these applications are capable of logging and reporting errors, no further action is typically taken during the application run time. Eventually, if the errors reach a high enough severity level, the result may be a software application failure or worse, an operating system failure. These failures may cause valuable data to be lost from the application. In the event of an operating system failure, data may also be lost from other applications and operational services may be interrupted.

Thus, what is needed is an effective method and apparatus for managing errors in a computer system to predict failures in advance and to take appropriate recovery action.

DETAILED DESCRIPTION

In the following description, for purposes of explanation, specific nomenclature is set forth to provide a thorough understanding of embodiments of the present invention. It will be apparent to one skilled in the art that specific details in the description may not be required to practice the embodiments of the present invention. In other instances, well-known circuits, devices, and programs are shown in block diagram form to avoid obscuring embodiments of the present invention unnecessarily.

FIG. 1is a block diagram of a fault prediction module100according to an embodiment of the present invention. The fault prediction module100detects errors occurring on applications running on a computer system, predicts the failure of an application, and initiates fault recovery. According to an embodiment of the present invention, an application may be an instance of software running on a processor, processing element, or a specialized processor core. An application may be an operating system running on a processor, firmware running on an embedded microcontroller or other code executed on a machine. The fault prediction module includes a fault prediction module (FPM) manager110. The fault prediction module manager110interfaces with and transmits information between other components in fault prediction module100. The fault prediction module manager110may be used to configure rules and policies applied to various components in the fault prediction module100.

The fault prediction module100includes a fault detection unit120. The fault detection unit120monitors a plurality of applications running in a system for errors. According to an embodiment of the fault prediction module100, the fault prediction module manager110provides an application program interface (API) to applications running on the computer system to allow the applications to inform the fault prediction module manager110when an error has occurred and forward error information. The error information may include error type, severity of error and other related information. The fault detection unit120may include a plurality of error recorders121that record error information. According to an embodiment of the present invention, the error recorders121may be implemented using error counters. In one embodiment, an application may have a corresponding set of error recorders. The number of occurrence of each error type may be tracked by the error recorders121. In another embodiment, the fault detection unit120monitors an application by reading the error recorders121associated with error occurrence. For an application that does not utilize application program interfaces, the fault detection unit120may also monitor the application by utilizing error handlers. The error handlers may read one or more system log files or private log files associated with the application for error occurrence and update the error recorders121to reflect the occurrence of an error.

The fault prediction module100includes a failure prediction unit130. The failure prediction unit130analyzes the errors detected by the fault detection unit120and predicts whether an application or operating system will experience a critical failure ahead of time. The failure prediction unit130may correlate the various errors detected and isolate the root cause of the software problem.

The failure prediction unit130may utilize numerous prediction techniques. For example, the failure prediction unit130may perform prediction without a time window. In one embodiment, performing prediction without a time window involves predicting whether the errors detected will result in a failure by determining whether a number of errors of a particular type reach a threshold. The failure prediction unit130may perform adaptive error count prediction. In one embodiment, performing adaptive error count prediction involves comparing a number of faults detected against a critical threshold where the number of errors is decremented periodically to age the error information. The failure prediction unit130may perform adaptive time window prediction. In one embodiment, performing prediction with a time window involves determining whether a number of errors of a particular type reach a threshold within a time window. The failure prediction unit130may also adjust the time window (adaptive time window) to include a smaller period of time if error rate increases or a larger period of time if error rate decreases. This may also include tuning critical thresholds depending on the error rate besides time windows. The failure prediction unit130may use training mechanisms to learn appropriate values of heuristics parameters for various errors. In one embodiment, the training mechanisms may monitoring application failures and scan error records for determining data sets of sequence of errors or individual errors and its associated heuristics parameters. The training mechanisms may choose a data set having a highest probability of causing an application failure for each error (individual or a unique sequence). Heuristics parameters may include error rates, threshold counts, time windows, probabilities, and other parameters. In other embodiments, failure prediction unit130may use statistical variations to determine the probability of a critical failure.

It should be appreciated that the failure prediction unit130may apply a combination of these techniques or other techniques to a variety of applications and error types. In one embodiment, error types may be given a severity level, such as high, medium, and low. In this embodiment, when analyzing the errors, the error types may be weighted according to the severity levels. According to an embodiment of the present invention, errors relating to memory allocation, memory access violation and buffer overruns may be given a high severity level. Errors relating to interprocess communication mechanisms to send messages or information may be given a medium severity level. Errors relating to timer related systems call may be given a high severity level if the timer is critical to the proper functioning of the application. Errors related to invalid parameter or argument values being passed to the function invocations may be given a low severity level. Errors related to mismatch of the states of correlated software components may be given a high severity level. It should be appreciated that the categorization of errors into severity type may be used in the prediction techniques described earlier to determine probability of a critical failure of a given application.

The failure prediction module100includes a fault recovery unit140. The fault recovery unit140operates to initiate fault recovery of an application or an operating system in response to a failure prediction made by the failure prediction unit130. According to an embodiment of the failure prediction module100, the fault recovery unit140may initiate a diagnostic of an application or an operating system. The fault recovery unit140may also restart an application or an operating system, save data from one or more applications, or initiate failover. Saving data may include saving data onto a storage medium for use later when the system is available. The fault recovery unit140may also initiate selective restart of a sub-component of the application which is causing failure e.g. an individual thread of a multi-threaded application. The fault recovery unit140may also perform audits to check communication links with other applications with which it is interacting with if a predicted failure is due to errors in interprocess communication mechanisms.

According to an embodiment of the fault prediction module100, the fault detection unit120also monitors the sanctity of applications by checking the status of an application upon the expiration of a watch dog timer. The status of the application may be checked by reading a strobe or a watch dog counter corresponding to the application. The status may also be checked by sending messages to the application and getting acknowledgement back. The fault recovery unit140initiates recovery of the application after expiration of a recovery period.

It should be appreciated that watch dog counters and error recorders121may be implemented, for example, using any interprocess communication mechanism such as a shared memory, message queues, semaphores, sockets, or other mechanisms. Exemplary watch dog application program interface which may be implemented by the fault prediction module100are listed below.1. Watch Dog Configurationa. Input Parameters: Application type, Application User ID, WatchDog Timeout Value, Watch Dog Implementation Type, Recovery Period, Recovery action informationb. Output Parameters: Status, Identity of IPC, Unique Application ID2. Start Monitoring Watch Doga. Input Parameters: Unique Application IDb. Output Parameters: Status3. Stop Monitoring Watch Doga. Input Parameters: Unique Application IDb. Output Parameters: Status4. Increment Watch Dog Countera. Input Parameter: Unique Application IDb. Output Parameters: Status5. Get Application Statusa. Input Parameters: Unique Application IDb. Output Parameters: Watch Dog Counter Value, Active/Inactive Status

Exemplary fault detection application program interface which may be implemented by the fault prediction module100are listed below.1. Configuration Error Recordersa. Input Parameters: Application Type, Application User ID, List of Error Information (like error types, counters, severity type), Prediction parameters like Leaky Bucket thresholds, time windows, probabilities, error rates2. Update Error Recordera. Input Parameters: Unique Application ID, Error Severity Value, Error Type, Error Infob. Output Parameters: Status3. Reset Error Recordera. Input Parameters: Unique Application ID, Error Severity Value, Error Typeb. Output Parameters: Status4. Get Error Recordera. Input Parameters: Unique Application, List of Error Severity Valuesb. Output Parameters: Status, Error Recorder

The fault prediction module may be implemented on a number of different types of computer systems.FIG. 2illustrates a first computer system200in which a fault prediction module may be implemented. The computer system200includes a processor201that processes data signals. The processor201may be a complex instruction set computer microprocessor, a reduced instruction set computing microprocessor, a very long instruction word microprocessor, a processor implementing a combination of instruction sets, or other processor device. In an alternate embodiment, the processor can be a network processor having multiple processing elements or it can have multiple general purpose processing cores or combination of general purpose cores and specialized cores. The processor201is coupled to a CPU bus210that transmits data signals between processor201and other components in the computer system200.

The computer system200includes a memory213. The memory213may be a dynamic random access memory device, a static random access memory device, read-only memory, and/or other memory device. The memory213may store instructions and code represented by data signals that may be executed by the processor201. According to an embodiment of the present invention, the processor201may execute an operating system and applications (shown collectively as202) as well as implement a fault prediction module203to manage errors generated by the operating system and applications202. The fault prediction module203may be implemented by the fault prediction module100shown inFIG. 1.

A bridge memory controller211is coupled to the CPU bus210and the memory213. The bridge memory controller211directs data signals between the processor201, the memory213, and other components in the computer system200and bridges the data signals between the CPU bus210, the memory213, and a first IO bus220.

According to an embodiment of the present invention, the processor201and bridge memory controller211may support virtualization where a plurality of virtual machines may function as a self-contained platform that runs its own software stack. In this embodiment, the fault prediction module203may reside and run in a protected partition of the processor201while the operating system and applications202reside and run in an open partition of the processor. This allows the fault prediction module203to monitor errors from the operating system and applications202while being isolated and protected from the impact of the errors.

According to an embodiment of the present invention, the computer system200may also include an embedded microcontroller where the fault prediction module can execute. The embedded microcontroller gives an isolated environment independent on the host processor to monitor the applications and OS(es) for predicting failures in these components.

The first IO bus220may be a single bus or a combination of multiple buses. The IO bus may also be connected through a controller to the memory controller. The first IO bus220provides communication links between components in the computer system200. A network controller221is coupled to the first IO bus220. The network controller221may link the computer system200to a network of computers (not shown) and supports communication among the machines. A display device controller222is coupled to the first IO bus220. The display device controller222allows coupling of a display device (not shown) to the computer system200and acts as an interface between the display device and the computer system100.

A second IO bus230may be a single bus or a combination of multiple buses. The second IO bus230provides communication links between components in the computer system200. A data storage device231is coupled to the second IO bus230. The data storage device231may be a hard disk drive, a floppy disk drive, a CD-ROM device, a flash memory device or other mass storage device. An input interface232is coupled to the second IO bus230. The input interface232may be, for example, a keyboard and/or mouse controller or other input interface. The input interface232may be a dedicated device or can reside in another device such as a bus controller or other controller. The input interface232allows coupling of an input device to the computer system200and transmits data signals from an input device to the computer system200. An audio controller233is coupled to the second IO bus230. The audio controller233operates to coordinate the recording and playing of sounds and is also coupled to the10bus230. A bus bridge223couples the first IO bus220to the second IO bus230. The bus bridge223operates to buffer and bridge data signals between the first IO bus220and the second IO bus230.

FIG. 3illustrates a second computer system300in which a fault prediction module may be implemented. The computer system300includes components similar to those shown inFIG. 2. It should be appreciated that not all of the components illustrated inFIG. 3are required for implementing the computer system300. The computer system300includes a first processor201and an nth processor301, where n may be any number. The computer system300includes a control processor310. The control processor310may be a processor that is similar to the processor201. The control processor301may be used by the computer system300to offload operations that may otherwise be performed by the processor201. According to an embodiment of the present invention, a fault prediction module311may be implemented on the control processor310to manage errors from the operating system and applications202executed on processor201. In another embodiment, fault prediction module311may executed on one of the processor cores of a processor on the computer system. The fault prediction module311may be implemented by the fault prediction module100shown inFIG. 1.

It should be appreciated that although a single processor is shown inFIG. 2, that the computer system200may also include a plurality of processors. It should further be appreciated that each of the processors in computer systems200and300may include a plurality of processor cores.

FIG. 4illustrates a third computer system400in which a fault prediction module may be implemented. The computer system400is a blade server system. The computer system400includes a plurality of server blades. Block410represents a first server blade, block420represents a second server blade, and block430represents an nth server blade, where n may be any number. Each of the server blades410,420, and430may be implemented on a single circuit board. Server blades410,420, and430include blade units411,421, and431respectively. Each of the blade units411,421, and431is an inclusive computing system that may include one or more processors, memory, communication paths, and other resources. In one embodiment, the computer system described inFIG. 2can be hosted on these blade units. According to an embodiment of the present invention, processors in the blade units411,421, and431may execute operating systems and applications412,422, and432.

Server blades410,420, and430include management microcontrollers (MMs)415,425, and435respectively. The management microcontrollers415,425, and435provide monitoring and control functionality to its corresponding server blade. Each of the management microcontrollers415,425, and435may maintain event logs, manage sensor data, and support other functions for its corresponding server blade. In one embodiment, each of the management microcontrollers415,425, and435may also include a fault prediction module (FPM)417,427, and437and provide isolation from errors in the applications and host memory. The fault prediction modules417,427, and437manages the errors generated by operating systems and applications412,422, and432. In another embodiment, fault prediction modules417,427,437may execute on a virtual partition of one of processing cores or elements of the blade. Each of the fault prediction modules417,427, and437may be implemented by the fault prediction module100shown inFIG. 1.

The blade server system400includes shared resources440. The shared resources440may include a network interface, storage device, power supply, cooling/ventilation system, and/or other resources that may not be available on the server blades410,420, and430and must be shared among the server blades410,420, and430.

The blade server system400includes a chassis management module (CMM)450. The chassis management module450performs hardware management of the server blades410,420, and430. For example, the chassis management module450may manage power allocation, insertion extraction, and compatibility checks for the server blades410,420, and430.

It should be appreciated that the fault prediction module100shown inFIG. 1may be implemented in a variety of environments using various techniques or procedures. For example, the fault prediction module100may be implemented as a set of native instructions on a processor. The instructions may be specifically designed and optimized for performing failure prediction. The fault prediction module100may be implemented in an application specific integrated circuit, field programmable gate array, in one or more processor core and/or processor chipset, or on a dedicated core in a multi-core processor system.

FIG. 5is a flow chart illustrating a method for managing errors according to an embodiment of the present invention. At500, it is determined whether a timer has expired. According to an embodiment of the present invention, the timer may be a periodic timer. If the timer has not expired, control returns to500. If the timer has expired, control proceeds to501.

At501, error recorders related to operating system (OS) errors are read. According to an embodiment of the present invention, system log files may be read to determine the occurrence of errors for operating systems and applications that do not utilize application program interfaces to update error recorders. According to an embodiment of the present invention, the error recorders may be implemented with error counters.

At502, it is determined whether an error that corresponds to or impacts the operating system has been detected. If an error is detected, control proceeds to503. If an error is not detected, control proceeds to508.

At504, it is determined whether an operating system failure is predicted. According to an embodiment of the present invention, failure may be predicted by analyzing the detected errors and their severity utilizing a number of prediction techniques including prediction with or without a time window, adaptive error count prediction, adaptive time window prediction, statistical variations using conditional probabilities and/or other techniques. If an operating system failure is predicted, control proceeds to505. If an operating system failure is not predicted, control proceeds to507. According to an embodiment of the present invention where multiple operating systems are running on a processor (using virtualization), the procedures described repeat for all the operating systems before proceeding to508.

At505, a notification of the predicted operating system failure is generated. According to an embodiment of the present invention, updating the probability of the instance error set is performed.

At506, operating system recovery is initiated. Operating system recovery may include initiating failover, saving data, and restarting the operating system.

At507, the prediction techniques are tuned. According to an embodiment of the present invention, if a time window is used in the prediction technique, the time window may be adjusted based on the increase or decrease of the error rate. The tuning also includes updating probability of instance being used for prediction analysis of a given error set and determining the instance of the highest probability.

At508, error recorders related to application errors are read. According to an embodiment of the present invention, system log files may be read to determine the occurrence of errors for applications that do not utilize application program interfaces to update error recorders. According to an embodiment of the present invention, the error recorders may be implemented by error counters.

At509, it is determined whether an error that corresponds to or impacts an application has been detected. If an error is detected, control proceeds to510. If an error is not detected, control proceeds to514.

At511, it is determined whether an application failure is predicted. According to an embodiment of the present invention, failure may be predicted by analyzing the detected errors and their severity utilizing a number of prediction techniques including prediction with or without a time window, adaptive error count prediction, adaptive time window prediction, statistical variations and/or other techniques. If an application failure is predicted, control proceeds to512. If an application failure is not predicted, control proceeds to514.

At512, a notification of the predicted application failure is generated. According to an embodiment of the present invention, updating the probability of the instance error set is performed.

At513, application recovery is initiated. Application recovery may involve initiating failover, saving data, or restarting the application or other appropriate actions.

At514, the prediction techniques are tuned. According to an embodiment of the present invention, if a time window is used in the prediction technique, the time window may be adjusted may be adjusted depending on in response to an increase or decrease of an error rate. The tuning may also includes updating probability of instances being used for prediction analysis of a given error set and determining the instance of the highest probability.

At515, it is determined if the error recorders for all applications have been checked. If all of the error recorders for all the applications have been checked, control proceeds to516. If not all of the error recorders for all the applications have been checked, control returns to508. According to an embodiment of the present invention, error recorders may be implemented using error counters.

At516, the timer is started.

FIG. 6is a flow chart illustrating a method for monitoring the operational status of an application according to an embodiment of the present invention. At601, it is determined whether a watch dog timer has expired. If the watch dog timer expired, control proceeds to602. If the watch dog timer has not expired, control returns to601.

At602, the application for which the watch dog timer expired is identified.

At603, it is determined whether the application is alive. According to an embodiment of the present invention, the determination may be made by invoking an interprocess communication mechanism to read a strobe or watch dog counter for the application. If the application is not alive, control proceeds to604. If the application is alive, control proceeds to606.

At604, notification of the application failure is generated.

At605, the error is recorded and a recovery period timer is started. According to an embodiment of the present invention, recovery is initiated after the recovery timer expires and the application is determined to still not be alive. Recovery may include writing data into storage, restarting an application, and/or initiating failover.

At606, the watch dog timer is re-started. Control returns to601.

FIG. 7is a flow chart illustrating a method for learning error sets according to an embodiment of the present invention. At701, error information is recorded. According to an embodiment of the present invention error information is recorded for each type and instance of an application. Error information may include time stamps, error types, error severity, and associated system calls that failed.

At702, a fault leading to the application failure is recorded. Exemplary faults may include stack overflow or memory allocation failure. According to an embodiment of the present invention, a recorder is incremented to indicate the number of occurrence of the fault.

At703, it is determined whether a platform needs to be reset. According to an embodiment of the present invention, the platform may be reset in response to a particular type of fault occurrence. If the platform is to be reset, the platform is reset and control returns to704. If the platform is not to be reset, control proceeds to705.

At704, the fatal fault that led to the application failure is determined. According to an embodiment of the present invention, the fatal fault is determined by scanning the faults recorded at702.

At705, the relevant errors associated with the fault is identified. According to one embodiment, the relevant errors may be determined by scanning all previous error information recorded.

At706, it is determined whether all relevant error information recorded has been scanned. If not all relevant error information recorded has been scanned, control proceeds to707. If all relevant error information has been recorded, control proceeds to709.

At707, statistics are computed for the error to generate error sets. According to an embodiment of the present invention, an error set includes a combination of parameters such as a sequence or order of errors, time window, error count, and/or error rate. The conditional probability of the occurrence of an error may be determined. The conditional probability may take into account the probability of parent errors. According to an embodiment of the present invention, a time window, error count, and error rate is calculated for an error such as memory allocation failure. The number of occurrences of this instance may also be determined for a present training set to determine the conditional probability of the error set.

At708, a conditional probability of this error set is updated. Control returns to706.

At709, it is determined whether a training period is over. According to an embodiment of the present invention, the training period is timed and control checks to see whether a predetermined period of time has expired. If the training period is not over, control returns to701. If the training period is over, control proceeds to710.

At710, scan instances of heuristics parameters. According to an embodiment of the present invention, all instances of heuristic parameters determined in the training set (701-709) are scanned.

At711, designate instances with highest probability as error set to be used in heuristic analysis.

FIGS. 5-7are flow charts illustrating methods according to embodiments of the present invention. Some of the techniques illustrated in these figures may be performed sequentially, in parallel or in an order other than that which is described. It should be appreciated that not all of the techniques described are required to be performed, that additional techniques may be added, and that some of the illustrated techniques may be substituted with other techniques.

Embodiments of the present invention may be provided as a computer program product, or software, or firmware that may include an article of manufacture on a machine accessible or a machine-readable medium having instructions. The instructions on the machine accessible medium may be used to program a computer system or other electronic device. The machine accessible medium may include, but is not limited to, floppy diskettes, optical disks, CD-ROMs, magneto-optical disks, or other type of media/machine accessible medium suitable for storing or transmitting electronic instructions. The techniques described herein are not limited to any particular software configuration. They may find applicability in any computing or processing environment. The term “machine accessible medium” used herein shall include any medium that is capable of storing, encoding, or transmitting a sequence of instructions for execution by the machine and that cause the machine to perform any one of the methods described herein. Furthermore, it is common in the art to speak of software, in one form or another (e.g., program, procedure, process, application, module, unit, logic, firmware and so on) as taking an action or causing a result. Such expressions are merely a shorthand way of stating that the execution of the software by a processing system causes the processor to perform an action to produce a result.