Abstract:
Methods, apparatus and system programs are provided for holistic monitoring and troubleshooting an application where the application functionally depends upon a plurality of components on a network system and at least one of the plurality of components being selected from a group consisting of a network component, a hardware component, and a software component. The method includes collecting data from the components and analyzing data collected from the components to discover one or more issues in the components. The analyzing step includes considering domain knowledge of the components and considering the interrelationships and correlations between components working within the application. The method function includes diagnosing the issues in the components to determine an action plan.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS  
       [0001]    This application claims the benefit of U.S. Provisional Application No. 60/362,661 filed on Mar. 9, 2002. 
     
    
     
       BACKGROUND  
         [0002]    The present invention generally relates to management of applications.  
           [0003]    Enterprises can use distributed systems comprised of many applications to carry out day-to-day operations. Applications in the distributed systems can themselves be distributed and spread across many computers connected by a network. Such distributed systems have proven successful in providing enterprise users with effective processing of large amounts of data in a short duration of time.  
           [0004]    Although distributed applications provide efficient and powerful tools to an enterprise, the complexities, and intricacies of the distributed architecture make the integration and operations of such applications very difficult. Enterprises have addressed application integration issues by using Enterprise Application Integration (EAI) platforms. EAI platforms have been used for the integration of Customer Relations Management (CRM), Supply Chain Management (SCM), Enterprise Resource Planning (ERP) and many other large and important applications of an enterprise. Once integrated, such large and complex applications are equally difficult to manage. Indeed, once an application is deployed on a distributed system, it becomes extremely difficult to judge where the bottlenecks of the application may be located and how to “troubleshoot” the same.  
           [0005]    The issues of integration and management of applications have been further exacerbated due to the emergence of electronic business (often called e-business). With the emergence of the Internet and with advances in wired as well as wireless communications, there has been a revolutionary change in business dynamics, as a result of which, many operational boundaries between enterprises have virtually become non-existent. Further, emergence of e-business has facilitated business collaborations wherein enterprises exchange real-time product, services, and market information with their partners, manufacturers, suppliers, transporters, and customers. As expected, such collaboration, interoperability and integration of various applications usually leads to a much harder task of identifying, analyzing and troubleshooting issues related to components and improving enterprise&#39;s efficiency.  
           [0006]    It is therefore very important for enterprises to not only integrate large applications, but effectively manage them as well because even a small inefficiency in an application can result in tremendous economic and operational loss for the enterprise. For instance, a minor inefficiency in an ERP application can have a tremendous impact on the operation of the entire business of an enterprise. An ERP application not only impacts the way companies do business, but also impacts the productivity of a large number of employees using these applications, which in turn is largely dependent on other computational components.  
           [0007]    Enterprises have relied on a combination of skilled human administrators and a plethora of application management tools for regularly monitoring and debugging issues associated with the components that comprise their applications.  
         SUMMARY  
         [0008]    Since human administrators are costly, both in terms of time and money, a system is provided that includes tools for autonomic management, monitoring and troubleshooting of enterprise applications. Such tools provide an automated mechanism to test, track and report the availability and the condition of various enterprise applications. These tools also aid in maximizing the performance and make the enterprise system coherent.  
           [0009]    In another aspect, a system, method and computer program is provided to deliver an end-to-end performance management solution for enterprise applications and systems comprised of networks of enterprise applications. The system, method and computer program regularly monitors applications, detects faults and troubleshoots the same. The system, method and computer program goes beyond managing individual applications and takes into consideration the relationships and dependencies among all the components internal and external to the applications as well as the use or style of use of the component within the application. The system, method and computer program also takes into consideration the interrelationship of components within the application, correlation between components, and other information, knowledge, structure, logic, or behavior that is a result of using a component within the application, thus providing for holistic management of the entire system.  
           [0010]    A coordinated set of techniques for querying, probing, measuring, analyzing, baselining and troubleshooting is provided. The system is able to identify and describe a set of possible issues and is able to rectify and/or notify the user accordingly. In this way the system significantly reduces the time it takes to find and fix issues and it prevents many issues from even occurring. Such an integrated and automated approach for solving application performance issues results in significant reduction in the total cost of ownership for the enterprises. 
       
    
    
     BRIEF DESCRIPTION OF DRAWINGS  
       [0011]    The preferred embodiments of the invention will hereinafter be described in conjunction with the appended drawings provided to illustrate and not to limit the invention, wherein like designations denote like elements, and in which:  
         [0012]    [0012]FIG. 1 is a block diagram that illustrates co-operation between elements of a holistic monitoring system; and  
         [0013]    [0013]FIG. 2 is a flowchart that illustrates the steps involved in performing holistic monitoring and troubleshooting of components by the system. 
     
    
     DESCRIPTION OF PREFERRED EMBODIMENTS  
       [0014]    The present invention relates to a system, method and computer program for monitoring and troubleshooting applications in a computing environment. An issue is an event that affects the performance of any hardware or software component within the applications. If any issue is encountered, the system may take corrective actions like repairing the problem or tuning the system to make sure that desired performance level is achieved. The system, method and computer program regularly monitors the applications to automatically and preemptively identify any performance issues. The system, method and computer program takes an integrated approach to find the cause of the issues, by understanding the inter-relationships between various applications, and automatically suggests corrective measures and optimization and control strategies for such issues. In some cases, the system, method and computer program may also automatically repair the issues in applications.  
         [0015]    [0015]FIG. 1 is a block diagram that illustrates co-operation between elements of holistic monitoring system  140 . Referring to FIG. 1, the elements of system  140  are monitored components  100 , an Autonomic Engine (AE)  102 , a Persistent Store (PS)  104 , a data warehouse  101  and an AE Graphical User Interface (GUI)  106 .  
         [0016]    Monitored components  100  include network components such as routers, WAN links, gateways, hubs, subnets and/or software and hardware components such as client hosts, server hosts, Operating Systems (OS) and applications running on them, that are spread across the network  141 . All these monitored components  100  are monitored by AE  102 . Monitored components  100  can also be grouped together to be managed by a correlator  143 . Correlator  143  is a unified computational view that comprises logical topology and interrelationships between the monitored components  100  and a given correlator  143 .  
         [0017]    In one embodiment, AE  102  is a multi-threaded process and a logically centralized element that resides on a server computer  130 . The system  140  can have a set of AEs  102  that coordinate to cover all the individual monitored components  100  and group of monitored components  100  that are to be collectively managed. Upon initial use, after deployment or upon selection by a user, AE  102  performs an automatic topology and component discovery process of the overall system that is to be monitored. Such a topology discovery process includes discovering the network topology, hosts, Operating Systems (OS) as well as the applications and services running on these hosts.  
         [0018]    AE  102  can discover the network topology using protocols such as Simple Network Management Protocol (SNMP) and networking components such as routers and switches. AE  102  can discover hosts by using their Name Servers or by capturing Internet Protocol (IP) packets. The hosts are then contacted via protocols such as Telnet, and thereafter, commands are used to discover the operating system of the host. AE  102  can also read relevant files on the hosts to discover the applications that have been automatically started on the host. All the discovered data is stored in PS  104 .  
         [0019]    AE  102  is administered using GUI  106 . Multiple instances of the GUI  106  can be active at the same time. Through the GUI  106 , monitoring parameters like the choice of hosts, the services to be included within the scope of monitoring, timer values, Service Level Agreement (SLA) thresholds and method of alert notification (pager, email, or via the GUI) are configured. All these configurations can be viewed in a tabular as well as graphical form.  
         [0020]    AE  102  further includes of a set of analyzer  110  elements, a set of correlator  143  elements, an execution framework  112 , and a GUI framework  114 . Corresponding to each monitored component  100  monitored by AE  102 , is an analyzer  110  element within AE  102 . Corresponding to each relevant combination of monitored components, such as an application, is a correlator  143  element.  
         [0021]    Analyzer  110  is a software “object” that comprises various sub-elements including a set of sensors  116 , a set of monitors  118 , a set of checkups  120 , a set of diagnosers  122  and a presenter  124 .  
         [0022]    A correlator  143  is another element in AE  102  that can have the identical internal structure as an analyzer  110 . In software terms, correlator  143  can be a specialization of an analyzer  110 . correlator  143  is software code that embodies the overall application specific knowledge, i.e., how the components in an application interact with each other. For instance, in an enterprise application comprising Siebel and Oracle, correlator  143  for Siebel would have knowledge of networks, databases, web servers, and other components that operate within a Siebel execution context. Further, correlator  143  for Siebel would also have the knowledge of the frequent queries that are expected when Siebel interacts with a component like Oracle. Analyzer  110  and correlator  143  are different in that analyzer  110  relates to some software or hardware component, while correlator  143  relates to a collection of components such as an Enterprise application together with its supporting infrastructure.  
         [0023]    The sub-elements of analyzer  110  and correlator  143  are described below. Sensors  116  are the sub-elements of analyzer  110  that interact with a monitored component  100  to gather performance data, configuration data and status data. The kind of data required by sensors  116  varies from one monitored component  100  to another. For instance, routers could be queried to determine their connections to their neighbors, their queuing algorithms and their buffer size. While, databases could be queried to determine buffer pool size, segment size and concurrent number of users. Hosts could be queried to determine their memory size, processor speed, type of processor, number of processors, virtual memory size, and patches. Similarly, enterprise applications could be queried for their configuration parameters.  
         [0024]    Each monitored component  100  has a specific manner in which an external element may interact with it. Sensors  116  can be code or protocol that interacts with a monitored component  100  in the required manner. Thus, if a component  100  only provides a library callable from C in a single threaded process, then its sensor  116  accesses the component  100  in C from a single threaded process. Similarly, to access an OS like Solaris, and associated sensor  116  could run Solaris specific commands like “vmstat”, “iostat”, “mpstat” and “sar” to access the performance and other relevant data from Solaris.  
         [0025]    Sensors  116  access monitored components  100  in one of three ways: directly  132 , via a “helper” process  126 , or via an “agent” process  128 .  
         [0026]    To directly  132  access monitored components  100 , sensors  116  use various distributed computing techniques. If sensor  116  is unable to directly access a monitored component  100  due to reasons such as threading, library linking, security or connection management, sensor  116  can use a helper process  126 . In this case, sensor  116  communicates with helper process  126 , and helper process  126  in turn communicates with a monitored component  100 . Helper process  126  resides on the same server platform  130  as AE  102  and can be started by AE  102 . Alternatively, sensors  116  may also use agent process  128  to communicate with monitored components  100 . Agent process  128  reside on the same host as a given monitored component  100 , and can be configured to communicate with all monitored components  100  on their respective hosts.  
         [0027]    A special kind of agent process  128 , used only for monitoring network health, is a probe  108 . Probes  108  are software code deployed at various places, like workstations and servers, for monitoring network traffic and/or for introducing network traffic. The position of probes  108  depends on the type of data that is required to be monitored. For instance, in case Wide Area Network (WAN) link traffic is to be monitored, probes  108  are positioned near the WAN link. Similarly, if external Internet traffic is to be monitored, probes  108  are positioned near the Internet access points. A system  140  may have multiple probes  108  that work in conjunction with each other to monitor the network traffic.  
         [0028]    Probes  108  can perform network monitoring actively or passively. While actively monitoring, probes  108  receive and inject packets into the network to determine the networks&#39; performance, topology, availability and other characteristics. Probes  108  can also initiate application transactions to support activities like, diagnostics or Service Level Agreement (SLA) computation. Further, probes  108  can send out echo packets to simulate “ping” and “trace route” functions. In addition, a throughput test may also be performed by probes  108  in which a burst of traffic of known size is sent and the time this traffic takes to arrive at its destination is determined. While passively monitoring, probes  108  only receive packets from the network and report to AE  102 .  
         [0029]    Probes  108  can also be configured by AE  102  to watch for certain kinds of “flows”. When a new flow is seen that matches a predefined pattern, the flow is reported to AE  102 . One kind of flow can be the communication for a particular application. Flows have a particular structure and protocol and can be “parsed” to determine the activity being performed by the given monitored component  100 . A single probe  108  can monitor the flows of multiple monitored components  100 .  
         [0030]    After gathering the required data, probes  108  conduct primary analysis and construct higher-level semantics. The data can be summarized to include such items as the number of packets, total number of bytes, estimates of packet loss and round trip time.  
         [0031]    All the data collected by sensors  116 , either directly or through helper process  126  or agent process  128  (including probes  108 ) can be stored in PS  104 . Further, automatically or upon direction from an administrator, the data of various monitored components  100  can be saved over regular intervals of time. This is termed as baselining and these baselines can be used later for troubleshooting. Monitors  118  are the sub-elements of analyzer  110  that perform sampling and first level analysis of the gathered data. Monitors  118  are an optional element of analyzer  110  when configured as a correlator  143 , but are mandatory when the analyzer  110  is connected to a monitored component  100 .  
         [0032]    First level analysis performed by the monitors  118  typically involves testing if any threshold limits have been exceeded. Each monitor  118  runs periodically according to a set schedule and stores its own set of data in PS  104 .  
         [0033]    The data obtained by the first level analysis is further analyzed for discovering any issues in the performance of the monitored components  100 . Analysis is performed by checkups  120 . Checkups  120  are sub-elements of analyzer  110  and correlator  143  and perform second-level analysis. There are various forms of second-level analysis. One form of analysis involves computing various minimum, maximum, average values over a time period. Another form of analysis combines various probes  108  data to determine monitored components  100  response time for a user request. Yet another form of analysis compares the current state of a particular monitored component  100  against a stored baseline. Yet another form of analysis is directed towards network routing whereby the network traffic can be monitored to determine if broadcast storms, excessive retries, or excessive redirections are occurring.  
         [0034]    The analysis of the data may reveal a violation of some condition against a stored baseline or a Service Level Agreement (SLA). In such a case, an alert is generated that prompts the system  140  about an issue. These alerts act as stimuli to activate diagnoser  122 . Other stimuli that can activate diagnoser  122  include a user request whereby the AE  102  is requested to look at a particular monitored component  100 , an alert like a Simple Network Monitoring Protocol (SNMP) trap or an entry in the OS event logs that has been parsed and processed. Diagnoser  122  can also be activated periodically to check for any issues that may be building in various monitored components  100 .  
         [0035]    Upon receiving stimuli, AE  102  start troubleshooting the issue by defining the scope of the issue. Thus, if a component  100  has an issue, then AE  102  identifies all the analyzers  110  and correlators  143  that directly or indirectly contain the monitored component  100  that is associated with the issue.  
         [0036]    Diagnoser  122  can be a software code that incorporates the codified knowledge of professionals that know how to troubleshoot and repair a particular monitored component  100  in the form of knowledge modules. For instance, the knowledge of an engineer who is an expert in a packaged application like Siebel is tapped and codified into knowledge modules for that application. Similarly, the knowledge of a database administrator who specializes in databases for a particular packaged application is tapped and codified into knowledge modules for that packaged application. Diagnosers  122  contain knowledge modules that comprise many algorithms for troubleshooting and repairing, called tasks. Each task deals with a particular type of issue. Each task attempts to assess if some part of the monitored component  100  or some combination of monitored components  100  are working correctly. Each task requires performance data that may have already been acquired by earlier analysis to find and fix issues related to monitored components. Analyzer  110  for a particular monitored component  100  can have a set of tasks that pertain to the issues of that particular monitored component  100  in the corresponding diagnoser  122 . Similarly, correlator  143  also has a set of tasks where some of these tasks deal in part with a particular monitored component  100 . Diagnoser  122  tasks can be grouped into a set of categories together based on the type of issues they troubleshoot. These categories can be specific to the type of analyzer  110  or correlator  143 , but there is a great deal of commonality. A particular diagnoser  122  task might fit into a multiple categories. For instance, display units would only have issues relating to their hardware circuits or Cathode Ray Tubes (CRTs), and hence the categories of display units would be those dealing with these specific issues. Similarly, modern routers typically do not have disk Input/Output (I/O), and hence the diagnoser tasks for a modern router would not have a category for Disk I/O.  
         [0037]    Upon initiation by AE  102 , tasks are “fired” that correspond to the categories of the analyzer  110  and/or correlator(s)  143  within the scope of an issue. The execution framework  112  in the AE  102  can run the tasks concurrently. The data gathered from the tasks by diagnosers  122  is analyzed using such techniques as “Expert Systems”, “Case Based Reasoning”, rule systems, modeling, differences between systems, and baselining with predictive modeling to identify one or more root causes of the issue.  
         [0038]    Once diagnoser  122  identifies a root cause of an issue, the issue can be stored in PS  104 . Diagnoser  122  can take one of the three actions. First, diagnoser  122  may fix the issue related to the component automatically, which we refer to as corrective action. Second, diagnoser  122  may alert the user for repair by proposing a method to automatically fix the issue, which we refer to as prescriptive. If the user accepts the proposed method, diagnoser  122  repairs the issue related to the monitored component. Finally, if diagnoser  122  cannot automatically repair the issue, diagnoser  122  displays a characterization of the issue and/or a recommended solution to the user in GUI  106 . In any case, diagnoser  122  can notify the user about the issue using GUI  106 . Additionally, diagnoser  122  may also use other notification techniques such as pagers and email.  
         [0039]    To display results, diagnoser  122  uses presenter  124 . Presenter  124  is a sub-element of analyzer  110 . Presenter  124  uses GUI Framework  114  to support the display of different kinds of data, configuration, and alerts.  
         [0040]    While the above discussion has dealt with the near-real-time behavior of the system, in addition there is long-term analysis. The data within the database can be loaded into data warehouse  101  or another analysis system, and there have further computation performed.  
         [0041]    As correlator  143  is a specialization of analyzer  110 , much of the above discussion applies directly to correlators  143 . However, there are some important differences.  
         [0042]    Monitors  118  are an optional element of analyzer  110  when it is configured as correlator  143 , but are mandatory when the analyzer  110  is connected to a monitored component  100 .  
         [0043]    Correlators  143  have a similar structure as analyzers  110 , but they interact with applications or other coherent groups of components. Correlators  143  use the facilities of a group of analyzers  110  for a given application to acquire performance, configuration, and other data, either through sensors  116  or through their access to the PS  104 . Correlators  143  have their own monitors  118  (but they are usually omitted), checkups  120 , and diagnosers  122 . Correlators  143  deal with the entire application, in a holistic fashion. Correlators  143  use knowledge of the use or style of use of a monitored component  100  within the application, the interrelationship of components  100  within the application, correlation between components  100 , and other information, knowledge, structure, logic, or behavior that is a result of using the component within the application. When diagnosers  122  find issues they also can send a prescriptive message or perform a corrective action, and similarly use the GUI  106  for notification.  
         [0044]    Overall, system  140  is operative to manage and monitor the monitored components  100  individually, as well as with applications and their respective monitored components  100 . In addition, the system  140  categorizes issues, produces prescriptive messages, and/or affects repair, tuning, restart, or other changes to the applications or monitored component  100 .  
         [0045]    [0045]FIG. 2 is a flowchart that illustrates the steps involved in performing holistic monitoring and troubleshooting of components by the system. Referring to FIG. 2, data from various components is gathered at  200  by, for example, sensors  116 , and analyzed at  202 , by, for example, monitors  118  and checkups  120 . Data is then checked for any potential issues or disruption of Service Level Agreements at  204 . If an issue is discovered in any component(s), tasks (e.g., diagnoser  112  tasks) are fired that are associated with the monitored components  100  and categories of the issue. Diagnosers  112  can use the data from sensors  116 , monitors  118 , and checkups  120  to determine the cause of the issue at  206 . Diagnosers  112  can analyze the data using such techniques as “Expert Systems”, “Case Based Reasoning”, modeling, differences between systems, and baselining with predictive modeling to find the one or more causes of the issue at  208 . Once the cause of the issue is identified, it can be stored in PS  104 . The cause of the issue can be displayed at  210  to the user using, for example, the presenter at  216 .  
         [0046]    Thereafter, an appropriate action can be taken at  212 , by, for example, diagnosers  122 . Diagnosers  122  can either fix the issue related to the component automatically, or display a recommended solution to the user in GUI  106 . Diagnosers  122  may also use other notification techniques such as pagers and email.  
         [0047]    While embodiments of the invention have been illustrated and described, it will be clear that the invention is not limited to these embodiments only. Numerous modifications, changes, variations, substitutions and equivalents will be apparent to those skilled in the art without departing from the spirit and scope of the invention as described in the claims.