Abstract:
A method and apparatus that uses application errors as a predictive metric for overall measuring of applications functional health are disclosed. The automated system intercepts messages exchanged between inter-services of enterprise applications, analyzes the context of those messages, and automatically derives application errors embedded in the message. Thereafter, it is capable of showing deviations from expected behavior for the purposes of predicting failures of the monitored application. Furthermore, the invention displays the user&#39;s real-time actionable data generated using the application errors.

Description:
CROSS REFERENCE TO RELATED APPLICATIONS  
       [0001]     This application claims priority from U.S. Provisional Patent Application No. 60/592,676 filed on Jul. 30, 2004, the entire disclosure of which is incorporated herein by reference. 
     
    
     BACKGROUND OF THE INVENTION  
       [0002]     1. Technical Field  
         [0003]     The invention relates generally to automated systems for monitoring the performance and functional health of enterprise applications more particularly, the invention relates to automated systems for monitoring application errors as a metric for overall application and functional health, as well as for the purpose of early notification of failures that result from those errors.  
         [0004]     2. Discussion of the Prior Art  
         [0005]     Messaging infrastructure, integration servers, Web services, and service oriented architectures (SOA), for many reasons, are being adopted today to integrate applications in enterprise information technology (IT). Existing implementations of SOA are based on message buses, e.g. IBM MQ, or application servers, e.g. BEA WebLogic that serve as the connection medium and the glue logic between the independent applications. SOA, independently of its implementations, significantly lowers application integration costs which, in many cases, are estimated to be a third of IT budgets. Such architecture further allows the enterprises to become more agile and adaptive because application development becomes easier.  
         [0006]     SOA implementations dramatically change the way applications behave and operate within enterprise IT. These technologies break monolithic applications into a loosely-coupled application system, usually referred to as “enterprise applications” or “composite applications”. An enterprise application includes multiple services connected through messaging-based interfaces. This architecture enables cross application transactions that consist of messages that are communicated among services to perform a single business transaction.  FIG. 1  shows an exemplary diagram of a simple SOA architecture  100  representing several independent services, each operating on a different platform. The services are all connected to each other through a messaging interface which, here for simplicity, is referred to as an enterprise service bus. Communication between services is performed by interchanging messages which have a well defined structure. These messages are transferred on top of communication protocols including, for example, simple object access protocol (SOAP), hypertext transfer protocol (HTTP), extensible markup language (XML), Microsoft message queuing (MSMQ), Java message service (JMS), IBM WebSphere MQ, and the like. An example of an enterprise application is a car rental system that may include a website which allows a customer to make vehicle reservations through the Internet, a partner system, such as airlines, hotels, and travel agents; and legacy systems, such as accounting and inventory applications.  
         [0007]     Enterprises demand high-availability and performance from their enterprise applications. Hence, automated continuous monitoring of these applications is essential to ensure continuous availability and satisfactory performance. Specifically, the most critical performance factor in enterprise applications is the application availability. Traditionally application availability is determined according to the operation status of the application, i.e. whether the application is “up” or “down.” However, in many cases an application can be up, but still returns errors, and thus the application would not deliver the required service. In SOA environments, due to the dynamic nature of application usage by other applications, many of those errors are anticipated. Therefore, the application availability is the percentage of application service calls which do not return errors. For instance, Table I below shows error codes returned by a service call “GetQuote”. The returned error “19” means that the requested product is not available on location and, thus it is a legitimate usage error. However, error code “−1001” is a pure application error, which returned due to the inability of the backend service to execute. It can be easily claimed that each request that returned a “−1001” error, causes a pure revenue loss, simply due to application failures. As can be seen, the estimated revenue loss resulting from the error code “−1001” is around $2M per year.  
                                 TABLE I                           Error Codes Returned                        Estimated           Service   Number of   revenue quote       Error Code   function   loss quotes   loss/yearly               19 - product type   “GetQuote”   13,173   $2M       is not available at       location       −1001   “GetQuote”   11,370   $2M                  
 
         [0008]     In the related art, monitoring tools exist to measure resource-usage of such enterprise applications, or to drive synthetic transactions into these applications to measure their external performance and availability characteristics. These monitoring tools function to alert IT personnel within an enterprise to failures or poor performance. Specifically, these monitoring tools are mostly designed for measuring infrastructure performance and availability. However, other important metrics that are perceived as meaningless to IT personnel are not monitored, and thus the application behavior is not truly measured.  
         [0009]     As an example, application errors comprise one metric that is not monitored by the monitoring tools known in the prior art. An application error is returned by the calling service and may result from a function, e.g. a SOAP function of a Web service or a response message to request message in a MQ environment; an application, e.g. a partner system; or an infrastructure, e.g. servers. Application errors returned by an application are meaningful to software developers and are generally used for debugging purposes. However, application errors, by themselves, are not understood by IT personnel and, thus, are not used for system health monitoring. Nevertheless, application errors (or bugs) have a huge part as a cause of IT application failures and in affecting IT health in general. In many cases, errors between the services can serve as predictive indicators, if only they were monitored.  
         [0010]     It would be, therefore, advantageous to provide a solution that discovers application errors and that uses them as a health metric, as well as a predictive metric for providing early notifications of failures of the monitored enterprise system.  
       SUMMARY OF THE INVENTION  
       [0011]     A method and apparatus that uses application errors as a predictive metric for overall monitoring of applications functional health are disclosed. The automated system intercepts messages exchanged between services or applications components of enterprise applications, analyzes the context of those messages, and automatically discovers application errors embedded in the message. Thereafter, it is capable of showing deviations from expected behavior for the purposes of predicting failures of the monitored application. Furthermore, the invention provides the user with real-time actionable data and the context of the errors, thus allowing fast root cause and recovery. 
     
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0012]      FIG. 1  is an exemplary diagram of a typical system architecture for executing a composite application (prior art);  
         [0013]      FIG. 2  is a block diagram of an automated monitoring system disclosed in accordance with the invention;  
         [0014]      FIG. 3  is an exemplary screenshot of a matrix view that shows a summary baseline of applications errors in context of transactions;  
         [0015]      FIG. 4  is an exemplary screenshot of a deviation graph view;  
         [0016]      FIG. 5  is another exemplary screenshot of a deviation graph view;  
         [0017]      FIG. 6  is an exemplary screenshot of a bar graph showing the application availability; and  
         [0018]      FIG. 7  is flowchart of a method for using application errors as a predictive metric according to the invention 
     
    
     DETAILED DESCRIPTION OF THE INVENTION  
       [0019]      FIG. 2  is an exemplary block diagram of an automated monitoring system  200  according to the invention. The system  200  comprises a plurality of data collectors  210 , a correlator  220 , a context analyzer  230 , a baseline analyzer  250 , a database  260 , and a graphical user interface (GUI)  270 . The data collectors  210  are deployed on the services or applications that they monitor, or on the network between these applications as a network appliance, and are designed to capture messages that are passed between the various services. The data collectors  210  are non-intrusive, i.e. they do not to impact the behavior of the monitored services. The data collectors  210  can capture messages transmitted using communication protocols including, but not limited to, SOAP, XML, HTTP, JMS, MSMQ, and the like.  
         [0020]     The correlator  220  classifies raw objects received from the data collectors  210  into events. Each event represents a one-directional message as collected by a single collector  210 . Each event includes one or more dimension values, as generated by the collectors  210  from the original message data. The dimension values are based on the dimensions of interest, as defined by the users. For example, to extract an application error code it is necessary to define at least one error dimension and analyze each response message generated by the application.  
         [0021]     The context analyzer  230  analyzes streams of events that are provided in a canonical representation. This representation can be thought of as a set of name-value pairs. Each such pair represents dimension and dimension value and, thus, defines the context to be derived for the event. A canonical message structure can be represented as follows: 
    {&lt;DIM 1 , DV 1 &gt;, &lt;DIM 2 , DV 2 &gt;, &lt;DIM 3 , DV 3 &gt;, . . . , &lt;DlMn, DVn&gt;}   
 
         [0023]     During the system  200  setup, users may define the tuple schemas of interest for context monitoring. A tuple schema is a n-dimensional cube of dimensions. Following are examples for tuple schemas that are defined using dimensions 
    DIM 1 , DIM 2 , and DIM 3 :     TS 1 =&lt;DIM 1 &gt;    TS 2 =&lt;DIM 1 ×DIM 2 &gt;    TS 3 =&lt;DIM 1 ×DIM 2 &gt;DIM 3 &gt;,     TS 4 =&lt;DIM 3 &gt;   
 
         [0029]     The context analyzer  230  classifies each canonical message into all schemas that are defined by the dimensions present in the message. Each combination of dimension values per such tuple schema defines the specific tuple to which the event belongs. If such tuple exists, the event is added to the statistics of that tuple. Otherwise, a new tuple is created and the event is added to the new tuple. In both cases, the metrics measured on the event are added to the statistics of the tuple. For example, a tuple schema (TS 1 ) includes the dimensions function and an error type, i.e. TS 1 =&lt;function, error type&gt;. The dimension values of TS, may be: T=&lt;“getLocation,” “DB is not responding”&gt;. A collection of measured values, e.g. an error rate, an application availability, each having a numeric value that can be statistically aggregated over time, is saved in cells. The statistics are later used for determining a baseline for each of the tuples, and define the normal context of the event. The operation of the context analyzer  230  is further discussed in U.S. patent application Ser. No. 11/092,447, assigned to common assignee, and which is hereby incorporated herein for all that it contains.  
         [0030]     In accordance with the invention statistics are gathered on application errors on each tuple schema that includes an error dimension. Application errors are defined as a dimension and a tuple schema in the system  200 . For example, an error dimension is calculated from the “return code type” which includes the application errors returned by the service to its client. The measured values (or statistics) associated with an error dimension include, but are not limited to, an error rate and the total amount. The error rate defines the number of errors of an error dimension aggregated over a specified time period. Statistic measures for the error rate, such as an average, a standard deviation, a minimum value, and a maximum value, may be also computed by system  200 .  
         [0031]     The context analyzer  230  may derive errors from messages using a set of extraction expressions each corresponding to a predefined dimension and, especially, to an error dimension. In an exemplary embodiment, an extraction expression is defined using an XML X-path expression. The context analyzer  230  applies the extraction expressions to the collected messages to extract the dimension values. The context analyzer  230  may also derive errors from error fields in the messages. The error fields are selected by users, e.g. IT personnel, on the fly. Errors included in a message generally contain an error code and, description. For error dimensions, the extracted dimension values are an error code and preferably, an error description. The error description is parsed to determine the error name, e.g. “DB is not responding.” Additionally, the error rate, i.e. the measure value of an error dimension, and its statistical measures are calculated and kept together with the dimension values in a cell. Each of the statistics variables is calculated for a specified and configurable time period.  
         [0032]     The context analyzer  230  is also capable of associating errors with transaction instances. The context analyzer  230  analyzes the context of both messages and transaction instances composed of these messages. Thus, discovered errors can be associated with transaction instances, and thus transactions. By relating messages, as well as transactions to detected errors, the system  200  provides a reliable indicator of the IT health.  
         [0033]     For predicting failures in the monitored enterprise application, the baseline analyzer  250  compares the current error rate against its normal rate, hereinafter referred as “the norm.” The norm determines the behavior of the enterprise application and whether that behavior is considered correct. As an example, the norm may determine the allowable maximum number of errors returned by a calling service per a request type. The norm may be predetermined by the user as a constant threshold value, a threshold having variable value, or dynamically determined by the baseline analyzer  250 .  
         [0034]     By comparing measured values to the norm, a scoring for a tuple is calculated based on the statistical distance of the error rate from an expected normal value. The results of the scoring may be categorized as a normal, a degrading, or a failure state. If the baseline analyzer  250  detects a deviation from a norm, an alert is generated and sent to the GUI  270  for presentation. Alerts can also be sent to an external system including, but not limited to, an email server, a personal digital assistant (PDA), a mobile phone, and the like. The baseline analyzer  250  also generates a plurality of analytic reports for specified periods of time, and a plurality of views that enable the user to view the state and statistical measures calculated for each combination of error groups over time.  
         [0035]     In one embodiment of the invention, the baseline analyzer  250  may operate as a verification engine. In this embodiment, the verification engine compares the application errors, or the error rate, to a predefined set of rules. If one of the rules is triggered then an alert is generated. An example of such a rule is: generate an alert if at least one application error was detected between 10:00 am and 11:00 am.  
         [0036]     In one embodiment of the invention, the baseline analyzer  250  generates real-time actionable data for the users, e.g. IT personnel. The actionable data are generated, and presented by GUI  270 , in a format and context that allows users to perform their roles within the business process. It is important that the actions triggered by the data occur in a timely manner to have the greatest impact on the business.  
         [0037]     In accordance with an exemplary embodiment of the invention, tuples may be categorized according to the error dimensions, into error groups. An error group includes a different class of errors that identifies the error source, for example, application errors, infrastructure errors, function errors, and so on. For each error group a decisive level is assigned. The decisive level determines whether or not the errors in the group are critical for the successful operation of the monitored enterprise application. The criteria for categorizing the errors and the decisive levels are predefined by the system  200  and can be also defined by the user.  
         [0038]     The baseline analyzer  250  may automatically generate the norm, adapted to typical or seasonal behavior patterns. The baseline analyzer  250  uses historic statistics of a plurality of content characteristics to determine expected behavior in the future. The methods used by the baseline analyzer  250  to determine the norm are described in U.S. patent application Ser. No. 11/093,569, assigned to common assignee, and which is hereby incorporated herein for all that it contains.  
         [0039]     The GUI  270  presents the actionable data generated by the baseline analyzer  250 . Specifically, the GUI  270  displays to the user a constant status of the monitored services in a dashboard, alerts, analytical reports for specified periods of time, and the dependencies between monitored entities. This enables the user to locate the cause of failures in the monitored enterprise application. The GUI  270  also enables the user to view the state and statistics variables that were calculated over time. The invention provides multiple different views of the calculated metrics, and statistics variables are provided. These views include at lease a matrix view and a deviation graph view.  
         [0040]      FIG. 3  shows an exemplary and non-limiting matrix view  300 . The matrix view  300  provides a view at a glance of the scoring of a single error group that includes errors classified as application errors. The rows of the matrix view  300  list the values of a single attribute e.g. an application return error, while the column lists the values of a related transaction. Each cell shows the scoring state for the crossed values of the independent and dependent attributes. The scoring states normal, degrading, and failure are presented as a green cell, a yellow cell, and a red cell, respectively. For example, the cell  310  indicates a failure in the transaction “getLocations” with the return error code “location605.” 
         [0041]      FIG. 4  shows an exemplary and non-limiting deviation graph view  400 . The graph view  400  provides a series of graphs, each showing the error rate measured for the errors depicted in  FIG. 3 . The graph preferably displays the baseline and the range of normal and abnormal values. As shown in the graph  410 , a spike in the measured error rate of the error code “location605” is discovered in certain time period of the operation of the monitored application. This is a significant deviation from the norm determined for the type error. This behavior provides a good indication to a future failure. In fact, a deviation graph view  500 , provided in  FIG. 5 , shows a sharp fall in the application availability as detected immediately after the occurrence of the spike in the measured error rates. On the other hand, the deviation graph view  420  displays a burst of errors detected for the error code “profile804” during a certain time period of the operation of the monitored application. This represents a normal behavior of the application and, thus, a failure notification is not generated in this case. It is clearly understood from this example that the disclosed invention can use the application errors as a predictive metric.  
         [0042]      FIG. 6  shows another exemplary and non-limiting graph view  600  generated by the GUI  270 . The Graph view  600  depicts the availability of a “MakeReservation” function of an exemplary car rental system. As can be seen, the availability  610  of this critical function is often below 99% per day. In this case, each failure to respond to a reservation request is tied directly to revenue loss. In other cases, the relationship can be less direct. Still, indirectly, any application failure affects the revenue and quality of service. As opposed to prior art solutions, the invention provides a clear indication on functional availability and, by that, significantly reduces revenue loss to enterprises.  
         [0043]      FIG. 7 a  non-limiting flowchart  700  describing the method for employing application errors as a predictive metric in accordance with an exemplary embodiment of the invention. At step S 710 , the user designates, on the fly, error fields in messages exchanged between the various components of the monitored system. The configuration of these error fields is performed by application support personnel, and does not require the intervention of the software developers. When monitoring a standard protocol, for example, BPEL or FIXML the automated monitoring system  200  is pre-configured to recognize their standard return codes.  
         [0044]     At step S 720 , raw messages exchanged between the different components of the monitored enterprise application are captured and only the parameters of interest including, but not limited to, return codes are extracted from the messages for generating light weight messages. These messages may be sent to a transaction correlator. At step S 730 , independent messages collected from independent application&#39;s components may be assembled into transaction instances.  
         [0045]     At step S 740 , the context of the collected messages is analyzed for the purpose of detecting application errors in the monitored messages and transaction instances.  
         [0046]     At step S 750 , the error rate and total number for each error value is calculated. Optionally, other statistical measures of the error rate are also calculated. In an exemplary embodiment of the invention, error values, the measured error rate, and other statistical measures are kept in cells, as described in greater detail above.  
         [0047]     At step S 760 , the calculated error rate of respective error values are compared to a range band, which defines the norm of that error in the monitored message or transaction. At step S 770 , a check is made to determine if the error rate for an error value deviated from its expected value, as defined by the norm and, if so, at step S 780  an alert is generated and sent to the user. Otherwise, at step S 790 , information about failures detection, as well as application errors and performance evaluation, is displayed to the user through a series of GUI views. It should be noted that an alert is generated depending on the statistical deviation from the norm.  
         [0048]     It should be appreciated by a person skilled in the art that a key advantage of the invention is the ability to discover application error codes automatically, learn their normal distribution and determine whether the discovered errors can induce a system failure. This is achieved by comparing the error rate of errors associated with a transaction to the norm.  
         [0049]     The invention has been described with reference to a specific embodiment, in which the automated monitoring system is used as a stand alone system. Other embodiments will be apparent to those of ordinary skill in the art. For example, the invention described herein can be adapted to embed with network appliances, such as wired or wireless bridges, routers, hubs, gateways, and so on. In this embodiment, the invention can be used to detect errors in messages transferred through or generated by the network appliances.  
         [0050]     In other embodiments, the invention can be used for application messages routing and provisioning.  
         [0051]     The values in the text and figures are exemplary only and are not meant to limit the invention. Although the invention has been described herein with reference to certain preferred embodiments, one skilled in the art will readily appreciate that other applications may be substituted for those set forth herein without departing from the spirit and scope of the present invention. Accordingly, the invention should only be limited by the claims included below.