Abstract:
Method, apparatus and computer program product are configured to perform computer monitoring activities; to collect information regarding computer system status during the computer monitoring activities; to detect a problem in dependence on the information collected during the computer monitoring activities; and to determine whether to launch a diagnostic probe when the problem is detected. The monitoring activities may be performed on a periodic or event-driven basis. The determination whether to launch a diagnostic probe is based on a rule included in a hierarchy of rules. The hierarchy of rules is based on problem tickets; system logs; and computer system configuration information.

Description:
TECHNICAL FIELD 
       [0001]    The invention generally concerns monitoring of computer systems, and more particularly concerns monitoring computer systems using both periodic and event-driven probes, wherein the event-driven probes may be triggered by data gathered from periodic probes. 
       BACKGROUND 
       [0002]    Problem determination for computing systems is a complex process through which computer problems are reported, diagnosed and solved. A typical sequence is for a problem monitoring system. The process continues with basic diagnosis by first level support personnel based on documented procedures. Simple issues such as password reset or file restoration can often be resolved without progressing further. For problems needing further investigation, they are then passed on to more skilled personnel such as system administrators otherwise known as SA&#39;s. 
         [0003]    When solving computing system problems, administrators often consult monitoring tools that provide some specific system indicators as well as physically access the problematic system to collect additional detailed information using system utilities. Since there are generally few problem determination tools available on most systems, SA&#39;s rely on system commands or small scripts in order to obtain system details that are related to the problem cause. In the course of day-to-day problem management, this process is often the most time consuming and expensive task for SA&#39;s because it requires field experience and expert knowledge in diagnosing problems. 
         [0004]    In addition to the limitation of tool availability, many SA&#39;s write their own homegrown tools for monitoring system status and collecting system details. Knowledge used for determining the root cause of various problems is not shared among various SA&#39;s in a centralized database of problems and root causes. 
         [0005]    Thus there is a need in the art for a method and apparatus for rule based directed problem resolution. 
       SUMMARY OF THE INVENTION 
       [0006]    A first embodiment of the invention is a computer monitoring system comprising a memory storing a computer program, the computer program configured to perform computer system monitoring activities when executed; and a data processing apparatus configured to execute the computer program, wherein when the computer program is executed the computer monitoring system is configured to perform computer monitoring activities; to collect information regarding computer system status during the computer monitoring activities; to detect a problem in dependence on the information collected during the computer monitoring activities; and to determine whether to launch a diagnostic probe when the problem is detected. 
         [0007]    A second embodiment of the invention is a computer program product comprising a computer readable memory medium storing a computer program, the computer program configured to be executed by digital processing apparatus, wherein when executed, the computer program is configured to cause a computer system to perform periodic computer monitoring activities; to collect information regarding computer system status during the periodic computer monitoring activities; to determine whether an event-driven probe has been triggered in dependence on the information collected during the periodic computer monitoring activities; and if an event-driven probe has been triggered, to perform the event-driven probe of the computer system. 
         [0008]    A third embodiment of the invention is a computer-implemented method comprising: performing monitoring activities of a computer system; collecting information regarding computer system status during the monitoring activities; detecting a problem in dependence on the information collected during the computer monitoring activities; and determining whether to launch a diagnostic probe when the problem is detected. 
         [0009]    In conclusion, the foregoing summary of the various embodiments of the present invention is exemplary and non-limiting. For example, one or ordinary skill in the art will understand that one or more aspects or steps from one embodiment can be combined with one or more aspects or steps from another embodiment to create a new embodiment within the scope of the present invention. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0010]    The foregoing and other aspects of these teachings are made more evident in the following Detailed Description of the Invention, when read in conjunction with the attached Drawing Figures, wherein: 
           [0011]      FIG. 1  is a block diagram depicting a computer monitoring system configured in accordance with the invention; 
           [0012]      FIG. 2  is probe in XML format configured in accordance with the invention; 
           [0013]      FIG. 3  is a rule associated with the probe depicted in  FIG. 2 , the rule configured in accordance with the invention; 
           [0014]      FIG. 4  depicts a tree graph configured in accordance with the invention; and 
           [0015]      FIG. 5  depicts a sample rule in accordance with the invention. 
       
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
       [0016]    One embodiment of the invention addresses system problem determination by providing health indicators and automated problem diagnosis capabilities. In one embodiment a multi-level approach is used that provides high-level health monitoring of key subsystems, and scoped probing that collects additional details in an on-demand, rule-based fashion. This method has the advantage of performing detailed drill-down probing only when it is relevant to the problem at hand and avoids the overhead of collecting such data continuously. In addition, the rules are not determined arbitrarily; they are created based on prior knowledge including problem tickets, individual experiences and design documents. The problem determination process is captured into a decision-rule tree whose execution is triggered by high-level monitoring events and launching low-level scoped probing. 
         [0017]    The system is encapsulated in an infrastructure which allows users of the system to customize, author and share monitoring tools, items to be monitored and problem resolution rules. 
         [0018]    In one embodiment, the present invention is a method and apparatus for rule-based directed problem resolution. The method combines high-level health monitoring of key subsystems and scoped probing that collects additional system details. In a typical situation a two-step determination process is involved. The first step is to monitor a pre-defined set of sub-systems to provide a health view at either periodic intervals or based on event-triggers. The second step is to launch diagnostic probes when a problem is detected from the first step. 
         [0019]    Referring now to  FIG. 1 ,  FIG. 1  depicts a block diagram of the overall system.  101  is the probe scheduler module and  107  is shown as the probe collection module. An instance of  101  and  107  would typically reside on each managed server and run as a daemon like process. Element  101  schedules the execution of probes according to a frequency rate or a triggered by an external event. 
         [0020]    Block  103  is shown as the probe controller and  109  is the rule engine. Both  103  and  109  would typically reside on each PDA monitoring server. Most of the information exchange and processing is handled by the probe controller  103  and the rule engine  109 . Periodically the probe controller  103  receives probe results from the probe scheduler  101 . A rule will be triggered if there is a corresponding rule for the particular probe. The rule engine  109  parses rules from the rule library  113  and compares the entry level probe results between the one defined in the rule and the one reported by the probe controller  103 . The triggering condition can be a threshold violation, change in a key configuration file, or other detected problem. As the rule tree is traversed, a command is sent to the probe scheduler  101  to execute the diagnostic probe and the result is returned and evaluated for further steps of diagnostic probes. 
         [0021]    The probe collection  107  contains one or more probes usually implemented as a script such as Perl, shell etc. that either executes native commands available in the system or interfaces with other monitoring tools deployed in the environment. Each probe parses and aggregates the output of the commands and returns the results in an organized format. 
         [0022]    The probe and rule authoring module is shown as  115 . This module allows for a user to create their own probes and corresponding rules. 
         [0023]    The user interface module is shown as  105  which provides for a way of users of the system to see various aspects including alerts, probes, rules and previous results that are saved in the history database  111 . 
         [0024]      FIG. 2  shows an example output  201  from a probe in XML format. In this example the output is for a probe that monitors an Ethernet interface. The output  201  can be any data format. 
         [0025]      FIG. 3  shows an example rule  301  which is associated with the probe that monitors an Ethernet interface. The first step within the rule tests if the number of collisions is beyond a certain threshold. If the threshold is exceeded, the next probe, chk_switch, is executed to collect some information about the network switch, for example related to the firmware version. 
         [0026]      FIG. 4  shows a sample rule tree graph which can diagnose problems related to the network connectivity of a managed server. The process starts at  401 . At  403  a test is made to see if the local interface is accessible by running a utility like a ping. If the test at  403  is unsuccessful the process ends at  405  where the TCP/IP configuration should be setup. If  403  is successful the next test performed is  407  which tests if the routing table is valid. If  407  is not successful the process ends at  409 . If the test is successful the next test is performed at  411  which tests if the default gateway is reachable. If the test  411  is unsuccessful the next test is to check the network interface adapter  413 . If the test at  413  is successful the next time is to check the resolv.conf file  415 . If  415  is successful the next test is to determine if the DNS server is reachable at  417 . 
         [0027]    The rule tree graph shown in  FIG. 4  is typically represented by a binary tree with each non-leaf node ( 403 ,  407 ,  411 ,  413 ,  415 ,  417 ) having two possible outcomes; success or failure. The process ends anytime a leaf node is reached ( 405 ,  409 ). It can be seen by those skilled in the art that any type of tree or graph representation is possible with each node allowing for more than two outcomes. 
         [0028]    Clearly some diagnostic probes have dependencies and may to be executed in a certain order. For example, to check that the system file /etc/resolv.conf exists before checking that a DNS server is reachable. In the absence of dependencies, probe could be ordered differently, perhaps tailored to the likelihood of certain types of failures in a given environment. 
         [0029]      FIG. 5  shows a sample rule using a profiler to find storage capacity problems. This type of rule is needed in situations when setting up a single threshold is not sufficient. Some number of discrete samples are taken during each interval  501 ,  503 ,  505 ,  507 . When the system disk will be full depends both on the current utilization of the space and the speed at which the space is utilized. A simple linear regression model is used to predict the trending. Within interval  507  the rate at which disk space is being used  509  is high enough to raise an alert  511  at some time in the future. Within interval  501  the rate of disk usage  513  is not sufficient to raise an alert. More complex methods can be used to further suppress false alarms. 
         [0030]    Thus it is seen that the foregoing description has provided by way of exemplary and non-limiting examples a full and informative description of the best apparatus and methods presently contemplated by the inventors for implementing rule-based directed problem resolution for servers with scalable proactive monitoring. One skilled in the art will appreciate that the various embodiments described herein can be practiced individually; in combination with one or more other embodiments described herein; or in combination with methods and apparatus differing from those described herein. Further, one skilled in the art will appreciate that the present invention can be practiced by other than the described embodiments; that these described embodiments are presented for the purposes of illustration and not of limitation; and that the present invention is therefore limited only by the claims which follow.