Abstract:
One embodiment of the present invention provides a system that facilitates using a multi-variate pattern-recognition technique to trigger software rejuvenation for a computer system. The system operates by periodically measuring values of a target set of performance parameters associated with the computer system while the computer system is operating. The system then predicts values for the target set of performance parameters based upon previously determined correlations between performance parameters in the target set of performance parameters. If the predicted values deviate beyond a predetermined threshold from the measured values, thereby indicating that software aging is likely to have occurred, the system schedules a software rejuvenation operation for the computer system.

Description:
BACKGROUND  
         [0001]    1. Field of the Invention  
           [0002]    The present invention relates to the design of highly reliable computer systems. More specifically, the present invention relates to a method and an apparatus that uses pattern-recognition techniques to trigger software rejuvenation in order to enhance performance and availability in computer systems.  
           [0003]    2. Related Art  
           [0004]    As electronic commerce grows increasingly more prevalent, businesses are increasingly relying on enterprise computing systems to process ever-larger volumes of electronic transactions. A failure in one of these enterprise computing systems can be disastrous, potentially resulting in millions of dollars of lost business. More importantly, a failure can seriously undermine consumer confidence in a business, making customers less likely to purchase goods and services from the business. Hence, it is critically important to ensure reliability in such enterprise computing systems.  
           [0005]    Unfortunately, as computer systems run for longer periods of time, they are increasingly affected by a phenomenon known as “software aging.” Software aging is typically caused by resource contention problems that build up over time until the computer system eventually hangs, panics, crashes or otherwise grinds to a halt. Software aging can be caused by a multitude of factors, including memory leaks, unreleased file locks, accumulations of unterminated threads, data round-off accrual, file space fragmentation, shared memory pool latching and thread stack bloating.  
           [0006]    Many of the adverse effects of software aging can be mitigated through a technique known as “software rejuvenation.” Software rejuvenation operates by cleaning up the internal state of a computer system and/or application to prevent the occurrence of more severe crash failures in the future. For some extreme problems, software rejuvenation can involve therapeutic reboots. However, less drastic measures suffice for the vast majority of software aging problems, such as flushing stale locks, reinitializing application components, preemptively rolling back, defragmenting memory and shutting down individual applications.  
           [0007]    If the software aging is caused by parasitic resource consumption (for example, a memory leak), periodic software rejuvenation can restore the resource and can thereby avoid a system crash caused by shortage of the resource as is illustrated by FIG. 1.  
           [0008]    Unfortunately, it is very hard to determine when these software rejuvenation operations are required. Some existing systems monitor a single system parameter. For example, some systems monitor an amount of free memory, and if the amount of free memory falls below a threshold value, they perform a software rejuvenation operation in an attempt to free up some memory. Unfortunately, this technique is only effective in mitigating known types of software aging problems (such as memory leaks) that can be detected by monitoring a single system parameter.  
           [0009]    Other systems perform software rejuvenation at periodic intervals. However, this may not catch software aging problems that arise between the periodic intervals. On the other hand, if these periodic rejuvenation operations are performed too frequently, they can unnecessarily degrade system performance.  
           [0010]    What is needed is a method and an apparatus for performing software rejuvenation operations without the limitations and problems of the above-described techniques.  
         SUMMARY  
         [0011]    One embodiment of the present invention provides a system that facilitates using a multi-variate pattern-recognition technique to trigger software rejuvenation for a computer system. The system operates by periodically measuring values of a target set of performance parameters associated with the computer system while the computer system is operating. The system then predicts values for the target set of performance parameters based upon previously determined correlations between performance parameters in the target set of performance parameters. If the predicted values deviate beyond a predetermined threshold from the measured values, thereby indicating that software aging is likely to have occurred, the system schedules a software rejuvenation operation for the computer system.  
           [0012]    In a variation on this embodiment, the system additionally determines correlations between parameters in the target set of performance parameters based on measurements gathered during a training mode that takes place during normal error-free operation of the computer system.  
           [0013]    In a variation on this embodiment, the system additionally prefilters a larger set of performance parameters to identify the target set of performance parameters. This pre-filtering operation eliminates redundant and/or poorly correlated performance parameters from the larger set of performance parameters to produce the target set of performance parameters.  
           [0014]    In a further variation, the pre-filtering operation determines correlations between performance parameters based on measurements of the larger set of performance parameters gathered during a training mode.  
           [0015]    In one embodiment of the present invention, scheduling the software rejuvenation operation involves sequencing the software rejuvenation operation in a way that substantially minimizes cost of the software rejuvenation operation.  
           [0016]    In one embodiment of the present invention, the software rejuvenation operation can involve: flushing stale locks, reinitializing application components, defragmenting memory, purging database shared memory pool latches, failing over between computing nodes, shutting down individual applications, preemptively rolling back, and performing a therapeutic reboot.  
           [0017]    In one embodiment of the present invention, the target set of performance parameters can include: system throughput parameters, processor load, system queue lengths, transaction latency, and an amount of available memory.  
           [0018]    In one embodiment of the present invention, the tasks of predicting values for the target set of performance parameters and scheduling the software rejuvenation operation are performed by a service processor that is co-located with other processors in the computer system.  
           [0019]    In one embodiment of the present invention, the tasks of predicting values for the target set of performance parameters and scheduling the software rejuvenation operation are performed by a remote service center that communicates with the computer system across a network. 
       
    
    
     BRIEF DESCRIPTION OF THE FIGURES  
       [0020]    [0020]FIG. 1 illustrates effects of software rejuvenation process in accordance with an embodiment of the present invention.  
         [0021]    [0021]FIG. 2A illustrates a service processor that schedules software rejuvenation operations in accordance with an embodiment of the present invention.  
         [0022]    [0022]FIG. 2B illustrates a remote service center that schedules software rejuvenation operations in accordance with an embodiment of the present invention.  
         [0023]    [0023]FIG. 3 illustrates a system for performing software rejuvenation in accordance with an embodiment of the present invention.  
         [0024]    [0024]FIG. 4 presents a flow chart illustrating how a target set of performance parameters is identified in accordance with an embodiment of the present invention.  
         [0025]    [0025]FIG. 5 presents a flow chart illustrating how correlations between performance parameters are determined in accordance with an embodiment of the present invention.  
         [0026]    [0026]FIG. 6 presents a flow chart illustrating the software rejuvenation process in accordance with an embodiment of the present invention. 
     
    
     DETAILED DESCRIPTION  
       [0027]    The following description is presented to enable any person skilled in the art to make and use the invention, and is provided in the context of a particular application and its requirements. Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the present invention. Thus, the present invention is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein.  
         [0028]    The data structures and code described in this detailed description are typically stored on a computer readable storage medium, which may be any device or medium that can store code and/or data for use by a computer system. This includes, but is not limited to, magnetic and optical storage devices such as disk drives, magnetic tape, CDs (compact discs) and DVDs (digital versatile discs or digital video discs), and computer instruction signals embodied in a transmission medium (with or without a carrier wave upon which the signals are modulated). For example, the transmission medium may include a communications network, such as the Internet.  
         [0029]    Computer Systems  
         [0030]    [0030]FIG. 2A illustrates a service processor  204  that schedules software rejuvenation operations for a server  202  in accordance with an embodiment of the present invention. Server  202  and service processor  204  can generally include any type of computer system, including, but not limited to, a computer system based on a microprocessor, a mainframe computer, a digital signal processor, a portable computing device, a personal organizer, a device controller, and a computational engine within an appliance.  
         [0031]    During operation, service processor  204  monitors performance parameters  203  from server  202 . Service processor  204  uses pattern recognition module  206  to detect if a rejuvenation operation should be scheduled. If so, service processor  204  uses software rejuvenation module  208  to schedule one or more software rejuvenation operations. These rejuvenation operations can be initiated by sending an alarm signal  210  (or other command) to server  202 , or alternatively to a system administrator in charge of server  202 .  
         [0032]    In one embodiment of the present invention, service processor  204  monitors performance parameters from a cluster of servers, including server  202 . This cluster of servers is located in the same enclosure as service processor  204 .  
         [0033]    [0033]FIG. 2B illustrates a remote service center  212  that schedules software rejuvenation operations in accordance with an embodiment of the present invention. Remote service center  212  operates in the same way as service processor  204 , except that remote service center  212  communicates with server  202  remotely through network  205 . Network  205  can generally include any type of wire or wireless communication channel capable of coupling together computing nodes. This includes, but is not limited to, a local area network, a wide area network, or a combination of networks. In one embodiment of the present invention, network  205  includes the Internet.  
         [0034]    During operation, remote service center  212  receives performance parameters  203  from server  202  and, if necessary, initiates software rejuvenation by sending sends alarm signal  210  or other commands to server  202  or to a system administrator.  
         [0035]    Software Rejuvenation System  
         [0036]    [0036]FIG. 3 illustrates a system for performing software rejuvenation in accordance with an embodiment of the present invention. During operation, this system monitors certain dynamic system performance parameters and uses pattern recognition techniques to detect the incipience of conditions leading to software aging. The system also estimates the time remaining until the resource contention reaches a critical level, and then schedules one or more rejuvenation actions for an application, a process group, a cluster node or an entire domain, depending on the pervasiveness of the resource exhaustion.  
         [0037]    This system is made up of number of modules, including sensitivity analysis module  304 , pattern recognition module  206 , and software rejuvenation module  208 . Sensitivity analysis module  304  operates during a training mode and is typically initiated whenever the system configuration changes significantly. Referring to the flow chart illustrated in FIG. 4, sensitivity analysis module  304  gathers a large number of performance parameters  303  from a monitored server (or cluster of servers)  302  (step  402 ). These performance parameters  303  can include parameters relating to: system throughput, processor load, system queue lengths, transaction latency, I/O traffic and amount of available memory. Note that in general there can be hundreds of monitored parameters.  
         [0038]    Sensitivity analysis module  304  then selects a subset of these parameters that provide maximal information content for a subsequent pattern recognition surveillance process. This can be accomplished by eliminating parameters that are redundant, are poorly correlated with other parameters, or are otherwise unsuitable for data-driven empirical modeling. Note that the resulting target set of performance parameters  306  is significantly smaller than the initial set of performance parameters  303  (step  404 ). For example, in one embodiment of the present invention, sensitivity analysis module  304  reduces about 240 performance parameters  303  to a smaller set of about 40 target performance parameters  306 .  
         [0039]    After the target set of performance parameters  306  is selected, pattern recognition module  206  (in FIG. 3) monitors the target set of performance parameters during a training mode. Referring to the flow chart that appears in FIG. 5, pattern recognition module  206  gathers a target set of performance parameters (step  502 ) and then determines correlations between parameters in the target set of performance parameters using one or more techniques for establishing such correlations (step  504 ). Note that many multivariate pattern recognition and/or correlation techniques are well-known in the art and will not be discussed further in this specifications. Identified correlations are then archived in a database  310  for use in subsequent surveillance operations. Note that this process of determining patterns and correlations can be performed periodically, or whenever the configuration of the computer system changes (even in a small way).  
         [0040]    During a subsequent surveillance mode, pattern recognition module  206  examines the target set of parameters and uses the archived correlations to determine if software aging is taking place. This process is described in more detail below with reference to FIG. 6. If software aging is detected, the system triggers software rejuvenation module  208 , which sequences the rejuvenation operation in way that substantially minimizes the “cost” of the rejuvenation action. For example, the system may schedule a therapeutic reboot or other rejuvenation operation for a less busy time of the day.  
         [0041]    Software Rejuvenation Process  
         [0042]    [0042]FIG. 6 presents a flow chart illustrating the software rejuvenation process in accordance with an embodiment of the present invention. The system starts when pattern recognition module  206  measures values for the target set of performance parameters when the system is in surveillance mode (step  602 ). Next, pattern recognition module  206  uses the measured values for the target set of performance parameters to predict values for other parameters (step  604 ). For example, if a first performance parameter is strongly correlated with a second and a third performance parameter, the value of the first performance parameter can be predicted from the measured values of the second and third performance parameters.  
         [0043]    Next, the system produces a set of residuals by computing differences between the measured performance parameters and those estimated by the pattern recognition algorithm (step  606 ). The system then determines if the computed residuals exceed a predetermined threshold (step  608 ). This can involve computing a function of the residuals that produces a binary output indicating whether software aging is likely to have taken place. If not, the system returns to step  602  to repeat the measuring process.  
         [0044]    Otherwise, if the residuals indicate a deviation has taken place, thereby indicating that software aging is likely to have occurred, the system schedules the software rejuvenation operation using software rejuvenation module  208  (step  610 ). Note that this scheduling operation can use any known optimization technique to schedule one or more rejuvenation operations in a manner that minimizes a specific cost function. This cost function can consider both the cost of the rejuvenation operations and the cost of any potential down time.  
         [0045]    The foregoing descriptions of embodiments of the present invention have been presented for purposes of illustration and description only. They are not intended to be exhaustive or to limit the present invention to the forms disclosed. Accordingly, many modifications and variations will be apparent to practitioners skilled in the art. Additionally, the above disclosure is not intended to limit the present invention. The scope of the present invention is defined by the appended claims.