Programmatic root cause analysis for application performance management

Programmatic root cause analysis of application performance problems is provided in accordance with various embodiments. Transactions having multiple components can be monitored to determine if they are exceeding a threshold for their execution time. Monitoring the transactions can include instrumenting one or more applications to gather component level information. For transactions exceeding a threshold, the data collected for the individual components can be analyzed to automatically diagnose the potential cause of the performance problem. Time-series analytical techniques are employed to determine normal values for transaction and component execution times. The values can be dynamic or static. Deviations from these normal values can be detected and reported as a possible cause. Other filters in addition to or in place of execution times for transactions and components can also be used.

BACKGROUND OF THE INVENTION

1. Field of the Invention

Embodiments of the present disclosure are directed to application performance management.

2. Description of the Related Art

Maintaining and improving application performance is an integral part of success for many of today's institutions. Businesses and other entities progressively rely on increased numbers of software applications for day to day operations. Consider a business having a presence on the World Wide Web. Typically, such a business will provide one or more web sites that run one or more web-based applications. A disadvantage of conducting business via the Internet in this manner is the reliance on software and hardware infrastructures for handling business transactions. If a web site goes down, becomes unresponsive or otherwise fails to properly serve customers, the business may lose potential sales and/or customers. Intranets and Extranets pose similar concerns for these businesses. Thus, there exists a need to monitor web-based, and other applications, to ensure they are performing properly or according to expectation.

For many application developers, a particular area of concern in these types of environments is transaction time. Longer transaction times may correlate directly to fewer transactions and thus, lost sales, etc. It may be expected that a particular task that forms part of a type of transaction may take a fraction of a second to complete its function(s). The task may execute for longer than expected for one or more transactions due to a problem somewhere in the system. Slowly executing tasks can degrade a site's performance, degrade application performance, and consequently, cause failure of the site or application.

Accordingly, developers seek to debug software when an application or transaction is performing poorly to determine what part of the code is causing the performance problem. While it may be relatively easy to detect when an application is performing slowly because of slow response times or longer transaction times, it is often difficult to diagnose which portion of the software is responsible for the degraded performance. Typically, developers must manually diagnose portions of the code based on manual observations. Even if a developer successfully determines which method, function, routine, process, etc. is executing when an issue occurs, it is often difficult to determine whether the problem lies with the identified method, etc., or whether the problem lies with another method, function, routine, process, etc. that is called by the identified method. Furthermore, it is often not apparent what is a typical or appropriate execution time for a portion of an application or transaction. Thus, even with information regarding the time associated with a piece of code, the developer may not be able to determine whether the execution time is indicative of a performance problem or not.

SUMMARY OF THE INVENTION

Programmatic root cause analysis of application performance problems is provided in accordance with various embodiments. Transactions having multiple components can be monitored to determine if they are exceeding a threshold for their execution time. Monitoring the transactions can include instrumenting one or more applications to gather component level information. For transactions exceeding a threshold, the data collected for the individual components can be analyzed to automatically diagnose the potential cause of the performance problem. Time-series analytical techniques are employed to determine normal values for transaction and component execution times. The values can be dynamic or static. Deviations from these normal values can be detected and reported as a possible cause. Other filters in addition to or in place of execution times for transactions and components can also be used.

In one embodiment, a method of processing data is provided that includes collecting data about a set of transactions that each include a plurality of components associated with a plurality of tasks. The data includes time series data for each task based on execution times of components associated with the task during the set of transactions. The method further includes determining whether the transactions have execution times exceeding a threshold and for each transaction having an execution time exceeding the threshold, identifying one or more components based on a deviation in time series data for a task that is associated with the one or more components of each transaction, and reporting said one or more components for said each transaction.

One embodiment includes an apparatus for monitoring software that includes one or more agents and a manager in communication with the agents. The agents collect data about a set of transactions that each include a plurality of components associated with a plurality of systems. The manager performs a method including receiving the data about the set of transactions from the one or more agents and developing time series data for each of the systems based on execution times of components associated with each system during the set of transactions. For each transaction having an execution time beyond a threshold, the manager identifies one or more components based on a deviation in time series data for a system that is associated with the one or more components of each transaction, and reports the one or more components for each transaction.

Embodiments in accordance with the present disclosure can be accomplished using hardware, software or a combination of both hardware and software. The software can be stored on one or more processor readable storage devices such as hard disk drives, CD-ROMs, DVDs, optical disks, floppy disks, tape drives, RAM, ROM, flash memory or other suitable storage device(s). In alternative embodiments, some or all of the software can be replaced by dedicated hardware including custom integrated circuits, gate arrays, FPGAs, PLDs, and special purpose processors. In one embodiment, software (stored on a storage device) implementing one or more embodiments is used to program one or more processors. The one or more processors can be in communication with one or more storage devices, peripherals and/or communication interfaces.

DETAILED DESCRIPTION

Programmatic root cause analysis of performance problems for application performance management is provided in accordance with embodiments of the present disclosure. Transactions are traced and one or more components of a transaction that are executing too slowly or otherwise causing a performance problem are reported. A transaction is traced to determine whether its execution time is beyond a threshold. If a transaction has a root level execution time outside a threshold, it can be reported. Tracing the transaction includes collecting information regarding the execution times of individual components of the transaction. For reported transactions, one or more components of the transaction can be identified and reported as a potential cause of the slow execution time for the transaction. If a particular component has an execution time beyond a threshold for a task or system associated with the component, the component can be identified and reported.

Component data is collected when tracing a set of transactions of a particular type. This component data can be organized into time series data for a particular type of component. For example, time series data can be formulated using the execution time of related components of multiple transactions. Related components can include a component from each transaction that is responsible for executing a particular task. The execution time of these components can be organized into time series data for the particular task. The data can also be organized by the system associated with or on which each of the components execute. If a particular transaction is performing abnormally, each of its components can be examined. Each component's execution time can be compared to a threshold based on a normal execution time for the task or system associated with that component. If a component's execution time is outside a normal time for the task it performs or the system with which it is associated, the component can be reported as a potential cause of the transaction's performance problem.

In one embodiment, a graphical user interface is used to report transactions and components that exceed a threshold. For each reported transaction, a visualization can be provided that enables a user to immediately understand where time was spent in a traced transaction. The visualization can identify select components of the reported transaction as a potential cause of a transaction's performance problem, by virtue of having an execution time beyond a threshold. The component thresholds may take the form of threshold deviations from a normal value or threshold execution times.

In one embodiment of the present disclosure, methods, etc. in a JAVA environment are monitored. In such an embodiment, a transaction may be a method invocation in a running software system that enters the JAVA virtual machine (JVM) and exits the JVM (and all that it calls). A system in accordance with embodiments as hereinafter described can initiate transaction tracing on one, some or all transactions managed by the system. Although embodiments are principally disclosed using JAVA implementation examples, the disclosed technology is not so limited and may be used in and with other programming languages, paradigms, systems and/or environments.

In one embodiment, an application performance management tool is provided that implements the performance analysis described herein.FIG. 1provides a conceptual view of one such implementation. The tool includes an enterprise manager120, database122, workstation124, and workstation126.FIG. 1also depicts a managed application6containing probe102, probe104, and agent8. As the managed application runs, the probes relay data to agent8. Agent8collects, summarizes, and sends the data to enterprise manager120.

Enterprise manager120receives performance data from managed applications via agent8, runs requested calculations, makes performance data available to workstations124,126and optionally sends performance data to database122for later analysis. The workstations include the graphical user interface for viewing performance data. The workstations are used to create custom views of performance data which can be monitored by a human operator. In one embodiment, the workstations consist of two main windows: a console and an explorer. The console displays performance data in a set of customizable views. The explorer depicts alerts and calculators that filter performance data so that the data can be viewed in a meaningful way. The elements of the workstation that organize, manipulate, filter and display performance data include actions, alerts, calculators, dashboards, persistent collections, metric groupings, comparisons, smart triggers and SNMP collections.

In one embodiment ofFIG. 1, each component runs on a different machine. For example, workstation126is on a first computing device, workstation124is on a second computing device, enterprise manager120is on a third computing device, and managed application6is on a fourth computing device. In another embodiment, two or more (or all) of the components are operating on the same computing device. For example, managed application6and agent8may be on a first computing device, enterprise manager120on a second computing device and a workstation on a third computing device. Any or all of these computing devices can be any of various different types of computing devices, including personal computers, minicomputers, mainframes, servers, handheld computing devices, mobile computing devices, etc. Typically, these computing devices will include one or more processors in communication with one or more processor readable storage devices, communication interfaces, peripheral devices, etc. Examples of the storage devices include RAM, ROM, hard disk drives, floppy disk drives, CD ROMS, DVDs, flash memory, etc. Examples of peripherals include printers, monitors, keyboards, pointing devices, etc. Examples of communication interfaces include network cards, modems, wireless transmitters/receivers, etc. The system running the managed application can include a web server/application server. The system running the managed application may also be part of a network, including a LAN, a WAN, the Internet, etc. In some embodiments, all or part of the disclosed technology is implemented in software that is stored on one or more processor readable storage devices and is used to program one or more processors.

In one embodiment, an application performance management tool monitors performance of an application by accessing the application's source code and modifying that source code. In some instances, however, the source code may not be available to the application performance management tool. Accordingly, another embodiment monitors performance of an application without requiring access to or modification of the application's source code. Rather, the tool can instrument the application's object code (also called bytecode).

FIG. 2depicts an exemplary process for modifying an application's bytecode to create managed application6.FIG. 1includes application2, probe builder4, application6and agent8. Application6includes probes, which will be discussed in more detail below. Application2is the Java application before the probes are added. In embodiments that use a programming language other than Java, application2can be a different type of application.

Probe Builder4instruments (e.g. modifies) the bytecode for application2to add probes and additional code to application2in order to create application6. The probes measure specific pieces of information about the application without changing the application's business logic. Probe builder4also installs agent8on the same machine as application6. Once the probes have been installed in the bytecode, the Java application is referred to as a managed application. More information about instrumenting byte code can be found in the following: U.S. Pat. No. 6,260,187, entitled “System For Modifying Object Oriented Code;” U.S. patent application Ser. No. 09/795,901, entitled “Adding Functionality to Existing Code at Exits;” U.S. patent Ser. No. 10/692,250, entitled “Assessing Information at Object Creation;” and U.S. patent application Ser. No. 10/622,022, entitled “Assessing Return Values and Exceptions, all of which are incorporated by reference herein in their entirety.

In accordance with one embodiment, bytecode is instrumented by adding new code that activates a tracing mechanism when a method starts and terminates the tracing mechanism when the method completes. To better explain this concept consider the following exemplary pseudo code for a method called “exampleMethod.” This method receives an integer parameter, adds 1 to the integer parameter, and returns the sum:

One embodiment will instrument this code, conceptually, by including a call to a tracer method, grouping the original instructions from the method in a “try” block, and adding a “finally” block with a code that stops the tracer:

IMethodTracer is an interface that defines a tracer for profiling. AMethodTracer is an abstract class that implements IMethodTracer. IMethodTracer includes the methods startTrace and finishTrace. AMethodTracer includes the methods startTrace, finishTrace, dostartTrace and dofinishTrace. The method startTrace is called to start a tracer, perform error handling and perform setup for starting the tracer. The actual tracer is started by the method doStartTrace, which is called by startTrace. The method finishTrace is called to stop the tracer and perform error handling. The method finishTrace calls doFinishTrace to actually stop the tracer. Within AMethodTracer, startTrace and finishTracer are final and void methods; and doStartTrace and doFinishTrace are protected, abstract and void methods. Thus, the methods doStartTrace and doFinishTrace must be implemented in subclasses of AMethodTracer. Each of the subclasses of AMethodTracer implement the actual tracers. The method loadTracer is a static method that calls startTrace and includes five parameters. The first parameter, “com.introscope . . . ” is the name of the class that is intended to be instantiated that implements the tracer. The second parameter, “this” is the object being traced. The third parameter, “com.wily.example . . . ,” is the name of the class of which the current instruction is inside. The fourth parameter, “exampleMethod,” is the name of the method of which the current instruction is inside. The fifth parameter, “name= . . . ” is the name under which the statistics are recorded. The original instruction (return x+1) is placed inside a “try” block. The code for stopping the tracer (a call to the static method tracer.finishTrace) is put within the finally block.

The above example shows source code being instrumented. In one embodiment, source code is not actually modified. Rather, an application management tool modifies object code. The source code examples above are used for illustration to explain the concept of instrumentation in accordance with embodiments. The object code is modified conceptually in the same manner that source code modifications are explained above. That is, the object code is modified to add the functionality of the “try” block and “finally” block. In another embodiment, the source code can be modified as explained above.

In a typical implementation including an application performance management tool as provided herein, more than one application will be monitored. The various applications can reside on a single computing device or on different computing devices. An agent may be installed for each managed application or on only a subset of the applications. Each agent will report back to enterprise manager120with data collected for the application it manages. Agents can also report data for applications that they do not directly manage, such as an application on a different computing device. The agent may collect data by monitoring response times or installing scripts to collect data from a remote application. For example, Javascript inserted into a returned web page can execute to determine the execution time of a remote application such as a browser.

FIG. 3is a flowchart describing one embodiment of a process for tracing transactions using the system ofFIG. 1. In step200, a transaction trace session is started. In one embodiment of step200, a window is opened and a user selects a dropdown menu to start a transaction trace session. In other embodiments, other methods can be used to start the session. In step202, a dialog box is presented to the user. This dialog box will ask the user for various configuration information. In step204, the various configuration information is provided by the user typing information into the dialogue box. Other means for entering the information can also be used within the spirit of the present disclosure.

One variable entered by the user in step204is the threshold trace period. That is, the user enters a time, which could be in seconds, milliseconds, microseconds, etc. The system will only report those transactions that have an execution time longer than the threshold period provided. For example, if the threshold is one second, the system will only report transactions that are executing for longer than one second. In some embodiments, step204only includes providing a threshold time period. In other embodiments, other configuration data can also be provided. For example, the user can identify an agent, a set of agents, or all agents. In such an embodiment, only identified agents will perform the transaction tracing described herein. In another embodiment, enterprise manager120will determine which agents to use.

Another configuration variable that can be provided is the session length. The session length indicates how long the system will perform the tracing. For example, if the session length is ten minutes, the system will only trace transactions for ten minutes. At the end of the ten minute period, new transactions that are started will not be traced. However, transactions that have already started during the ten minute period will continue to be traced. In other embodiments, at the end of the session length, all tracing will cease regardless of when the transaction started. Other configuration data can also include specifying one or more userIDs, a flag set by an external process or other data of interest to the user. For example, the userID is used to specify that only transactions initiated by processes associated with a particular one or more userIDs will be traced. The flag is used so that an external process can set a flag for certain transactions, and only those transactions that have the flag set will be traced. Other parameters can also be used to identify which transactions to trace. The information provided in step204can be used to create a filter.

In other embodiments as will be more fully described hereinafter, variations to the trace period are utilized. A user may specify a threshold execution time for a type of transaction. A user may specify a threshold deviation from a normal execution time and capture faster or more slowly executing transactions. Transactions exceeding the corresponding threshold will be reported. In one embodiment, a user does not provide a threshold execution time, deviation, or trace period for transactions being traced. Rather, the application performance management tool intelligently determines the threshold(s). For example, the tool can average execution times of transactions of a particular type to determine a corresponding threshold execution time. The threshold time can be a static value or a dynamic value that is updated as more transaction data is collected. The threshold may be a running average based on a number of previous transactions. Other more sophisticated time series techniques may also be used as will be described hereinafter.

In step206ofFIG. 3, the workstation adds the new filter to a list of filters on the workstation. In step208, the workstation requests enterprise manager120to start the trace using the new filter. In step210, enterprise manager120adds the filter received from the workstation to a list of filters. For each filter in its list, enterprise manager120stores an identification of the workstation that requested the filter, the details of the filter (described above), and the agents to which the filter applies. In one embodiment, if the workstation does not specify the agents to which the filter applies, then the filter will apply to all agents. In step212, enterprise manager120requests the appropriate agents to perform the trace. In step214, the appropriate agents perform the trace. In step216, the agents performing the trace send data to enterprise manager120. More information about steps214and216will be provided below. In step218, enterprise manager120matches the received data to the appropriate workstation/filter/agent entry. In step220, enterprise manager120forwards the data to the appropriate workstation(s) based on the matching in step218. In step222, the appropriate workstations report the data. In one embodiment, the workstation can report the data by writing information to a text file, to a relational database, or other data container. In another embodiment, a workstation can report the data by displaying the data in a GUI. More information about how data is reported is provided below.

As noted above, agents perform tracing for transactions. To perform such tracing, the agents can leverage what is called Blame Technology in one embodiment. Blame Technology works in a managed Java application to enable the identification of component interactions and component resource usage. Blame Technology tracks components that are specified to it. Blame Technology uses the concepts of consumers and resources. Consumers request some activity while resources perform the activity. A component can be both a consumer and a resource, depending on the context.

When reporting about transactions, the word Called designates a resource. This resource is a resource (or a sub-resource) of the parent component, which is the consumer. For example, under the consumer Servlet A (see below), there may be a sub-resource Called EJB. Consumers and resources can be reported in a tree-like manner. Data for a transaction can also be stored according to the tree. For example, if a Servlet (e.g. Servlet A) is a consumer of a network socket (e.g. Socket C) and is also a consumer of an EJB (e.g. EJB B), which is a consumer of a JDBC (e.g. JDBC D), the tree might look something like the following:

Servlet AData for Servlet ACalled EJB BData for EJB BCalled JDBC DData for JDBC DCalled Socket CData for Socket C

In one embodiment, the above tree is stored by the agent in a stack. This stack is called the Blame Stack. When transactions are started, they are pushed onto the stack. When transactions are completed, they are popped off the stack. In one embodiment, each transaction on the stack has the following information stored: type of transaction, a name used by the system for that transaction, a hash map of parameters, a timestamp for when the transaction was pushed onto the stack, and sub-elements. Sub-elements are Blame Stack entries for other components (e.g. methods, process, procedure, function, thread, set of instructions, etc.) that are started from within the transaction of interest. Using the tree as an example above, the Blame Stack entry for Servlet A would have two sub-elements. The first sub-element would be an entry for EJB B and the second sub-element would be an entry for Socket Space C. Even though a sub-element is part of an entry for a particular transaction, the sub-element will also have its own Blame Stack entry. As the tree above notes, EJB B is a sub-element of Servlet A and also has its own entry. The top (or initial) entry (e.g., Servlet A) for a transaction, is called the root component. Each of the entries on the stack is an object. While the embodiment described herein includes the use of Blame Technology and a stack, other embodiments can use different types of stacks, different types of data structures, or other means for storing information about transactions.

FIG. 4is a flowchart describing one embodiment of a process for starting the tracing of a transaction. The steps ofFIG. 4are performed by the appropriate agent(s). In step302, a transaction starts. In one embodiment, the process is triggered by the start of a method as described above (e.g. the calling of the “loadTracer” method). In step304, the agent acquires the desired parameter information. In one embodiment, a user can configure which parameter information is to be acquired via a configuration file or the GUI. The acquired parameters are stored in a hash map, which is part of the object pushed onto the Blame Stack. In other embodiments, the identification of parameters are pre-configured. There are many different parameters that can be stored. In one embodiment, the actual list of parameters used is dependent on the application being monitored. The present disclosure is not limited to any particular set of parameters. Table 1 provides examples of some parameters that can be used.

TABLE 1ParametersAppears inValueUserIDServlet, JSPThe UserID of the end-user invoking thehttp servlet request.URLServlet, JSPThe URL passed through to the servletor JSP, not including the Query String.URL QueryServlet, JSPThe portion of the URL that specifiesquery parameters in the http request (textthat follows the ‘?’ delimiter).Dynamic SQLDynamic JDBCThe dynamic SQL statement, either in aStatementsgeneralized form or with all the specificparameters from the current invocation.MethodBlamed MethodThe name of the traced method. If thetimers (everythingtraced method directly calls anotherbut Servlets, JSP'smethod within the same component,and JDBConly the “outermost” first encounteredStatements)method is captured.Callable SQLCallable JDBCThe callable SQL statement, either in astatementsgeneralized form or with all the specificparameters from the current invocation.Prepared SQLPrepared JDBCThe prepared SQL statement, either in astatementsgeneralized form or with all the specificparameters from the current invocation.ObjectAll non-statictoString( ) of the this object of the tracedmethodscomponent, truncated to some upper limit ofcharacters.Class NameAllFully qualified name of the class of thetraced component.Param_nAll objects withtoString( ) of the nth parameter passed toWithParamsthe traced method of the component.custom tracersPrimary KeyEntity BeanstoString( ) of the entity bean's propertykey, truncated to some upper limit of characters.

In step306, the system acquires a timestamp indicating the current time. In step308, a stack entry is created. In step310, the stack entry is pushed onto the Blame Stack. In one embodiment, the timestamp is added as part of step310. The process ofFIG. 4is performed when a transaction is started. A process similar to that ofFIG. 4is performed when a component of the transaction starts (e.g. EJB B is a component of Servlet A—see tree described above).

FIG. 5is a flowchart describing one embodiment of a process for concluding the tracing of a transaction. The process ofFIG. 5can be performed by an agent when a transaction ends. In step340, the process is triggered by a transaction (e.g. method) ending as described above (e.g. calling of the method “finishTrace”). In step342, the system acquires the current time. In step344, the stack entry is removed. In step346, the execution time of the transaction is calculated by comparing the timestamp from step342to the timestamp stored in the stack entry. In step348, the filter for the trace is applied. For example, the filter may include a threshold execution time of one second. Thus, step348, would include determining whether the calculated duration from step346is greater than one second. In another embodiment, a normal value for the type of transaction is used with a threshold deviation. If the transaction's execution time deviates from the normal value by more than threshold amount, the threshold is determined to be exceeded. If the threshold is not exceeded (step350), then the data for the transaction is discarded. In one embodiment, the entire stack entry is discarded. In another embodiment, only the parameters and timestamps are discarded. In other embodiments, various subsets of data can be discarded. In some embodiments, if the threshold period is not exceeded then the data is not transmitted by the agent to other components in the system ofFIG. 2. If the duration exceeds the threshold (step350), then the agent builds component data in step360. Component data is the data about the transaction that will be reported. In one embodiment, the component data includes the name of the transaction, the type of the transaction, the start time of the transaction, the duration of the transaction, a hash map of the parameters, and all of the sub-elements or components of the transaction (which can be a recursive list of elements). Other information can also be part of the component data. In step362, the agent reports the component data by sending the component data via the TCP/IP protocol to enterprise manager120.

FIG. 5represents what happens when a transaction finishes. When a component finishes, the steps can include getting a time stamp, removing the stack entry for the component, and adding the completed sub-element to previous stack entry. In one embodiment, the filters and decision logic are applied to the start and end of the transaction, rather than to a specific component.

Note that in one embodiment, if the transaction tracer is off, the system will still use the Blame Stack; however, parameters will not be stored and no component data will be created. In some embodiments, the system defaults to starting with the tracing technology off. The tracing only starts after a user requests it, as described above.

FIG. 6provides one example of a graphical user interface that can be used for reporting transactions and components thereof, in accordance with embodiments of the present disclosure. The GUI includes a transaction trace table400which lists all of the transactions that have satisfied the filter (e.g. execution time beyond the threshold). Because the number of rows on the table may be bigger than the allotted space, the transaction trace table400can scroll. Table 2, below, provides a description of each of the columns of transaction trace table400.

TABLE 2Column HeaderValueHostHost that the traced Agent is running onProcessAgent Process nameAgentAgent IDTimeStampTimeStamp (in Agent's JVM's clock) of the(HH:MM:SS.DDD)initiation of the Trace Instance's root entry pointCategoryType of component being invoked at the root levelof the Trace Instance. This maps to the firstsegment of the component's relative blamestack: Examples include Servlets, JSP,EJB, JNDI, JDBC, etc.NameName of the component being invoked. This mapsto the last segment of the blamed component'smetric path. (e.g. for “Servlets|MyServlet”,Category would be Servlets, and Name wouldbe MyServlet).URLIf the root level component is a Servlet or JSP, theURL passed to the Servlet/JSP to invoke this TraceInstance. If the application server provides servicesto see the externally visible URL (which may differfrom the converted URL passed to the Servlet/JSP)then the externally visible URL will be used inpreference to the “standard” URL that would beseen in any J2EE Servlet or JSP. If the rootlevel component is not a Servlet or JSP, novalue is provided.Duration (ms)Execution time of the root level component in theTransaction Trace dataUserIDIf the root level component is a Servlet or JSP, andthe Agent can successfully detect UserID's in themanaged application, the UserID associated with theJSP or Servlet's invocation. If there is no UserID, orthe UserID cannot be detected, or the root levelcomponent is not a Servlet or JSP, then there will beno value placed in this column.

Each transaction that has an execution time beyond a threshold will appear in the transaction trace table400. The user can select any of the transactions in the transaction trace table by clicking with the mouse or using a different means for selecting a row. When a transaction is selected, detailed information about that transaction will be displayed in transaction snapshot402and snapshot header404.

Transaction snapshot402provides information about which transactional components are called and for how long. Transaction snapshot402includes views (see the rectangles) for various components, which will be discussed below. If the user positions a mouse (or other pointer) over any of the views, mouse-over info box406is provided. Mouse-over info box406indicates the following information for a component: name/type, duration, timestamp and percentage of the transaction time that the component was executing. More information about transaction snapshot402will be explained below. Transaction snapshot header404includes identification of the agent providing the selected transaction, the timestamp of when that transaction was initiated, and the duration. Transaction snapshot header404also includes a slider to zoom in or zoom out the level of detail of the timing information in transaction snapshot402. The zooming can be done in real time.

In addition to the transaction snapshot, the GUI will also provide additional information about any of the transactions within the transaction snapshot402. If the user selects any of the transactions (e.g., by clicking on a view), detailed information about that transaction is provided in regions408,410, and412of the GUI. Region408provides component information, including the type of component, the name the system has given to that component and a path to that component. Region410provides analysis of that component, including the duration the component was executing, a timestamp for when that component started relative to the start of the entire transaction, and an indication of the percentage of the transaction time that the component was executing. Region412includes indication of any properties. These properties are one or more of the parameters that are stored in the Blame Stack, as discussed above.

The GUI also includes a status bar414. The status bar includes an indication416of how many transactions are in the transaction trace table, an indication418of how much time is left for tracing based on the session length, stop button420, and restart button422.

FIG. 7depicts transaction snapshot402. Along the top of snapshot402is time axis450. In one embodiment, the time axis is in milliseconds. The granularity of the time access is determined by the zoom slider in snapshot header404. Below the time axis is a graphical display of the various components of a transaction. The visualization includes a set of rows454,456,458, and460along an axis indicating the call stack position. Each row corresponds to a level of components. The top row pertains to the root component470. Within each row is one or more boxes which identify the components. In one embodiment, the identification includes indication of the category (which is the type of component—JSP, EJB, servlets, JDBC, etc.) and a name given to the component by the system. The root level component is identified by box470as JSP|Account. In the transaction snapshot, this root level component starts at time zero. The start time for the root level component is the start time for the transaction and the transaction ends when the root level component JSP|Account470completes. In the present case, the root level component completes in approximately 3800 milliseconds. Each of the levels below the root level470includes components called by the previous level. For example, the method identified by JSP/Account may call a servlet called CustomerLookup. Servlet|CustomerLookup is called just after the start of JSP|Account470and Servlet|CustomerLookup472terminates approximately just less than 3500 milliseconds. Servlets|CustomerLookup472calls EJB|Entity|Customer474at approximately 200 milliseconds. EJB|entity customer474terminates at approximately 2400 milliseconds, at which time Servlet|CustomerLookup472calls EJB|Session|Account476. EJB|session account 647626 is started at approximately 2400 milliseconds and terminates at approximately 3400 milliseconds. EJB|EntityCustomer474calls JDBC|Oracle|Query480at approximately 250 milliseconds. JDBC|Oracle|Query480concludes at approximately 1000 milliseconds, at which time EJB|Entity|Customer474calls JDBC|Oracle|Update482(which itself ends at approximately 2300 milliseconds). EJB/Session/Account476calls JDBC|Oracle/Query484, which terminates at approximately 3400 milliseconds. Thus, snapshot402provides a graphical way of displaying which components call which components. Snapshot402also shows for how long each component was executing. Thus, if the execution of JSP|Account470took too long, the graphical view of snapshot402will allow user to see which of the subcomponents is to blame for the long execution of JSP account470.

The transaction snapshot provides for the visualization of time from left to right and the visualization of the call stack top to bottom. Clicking on any view allows the user to see more details about the selected component. A user can easily see the run or execution time of a particular component that may be causing a transaction to run too slowly. If a transaction is too slow, it is likely that one of the components is running significantly longer than it should be. The user can see the execution times of each component and attempt to debug that particular component.

In one embodiment, the application performance management tool automatically identifies and reports one or more components that may be executing too slowly. The identification and reporting is performed without user intervention in one embodiment. Moreover, normal execution times for transactions and components can be dynamically and automatically generated.

Transactions are identified and component data reported, such as through the GUI depicted inFIGS. 6 and 7, to enable end-users to diagnose the root cause of a performance problem associated with a particular transaction. To further facilitate the management of application performance, the root cause of a performance problem is programmatically diagnosed in accordance with one embodiment. The diagnosis is implemented in one embodiment by analyzing the component data for a selected transaction. After analysis, one or more components are identified as a potential cause of the application's performance problem. These components can be reported to the end-user as an automatic diagnosis of the cause of an identified performance problem. Such implementations enable abnormally performing components of transactions to be programmatically identified and reported without user intervention. By eliminating required human analysis of raw component data, designers, managers, and administrators can more quickly, efficiently, and reliably identify poorly performing components.

FIG. 8is a table depicting exemplary component data for four transactions of the same transaction type. The individual tasks performed for the illustrated transaction type are set forth in column502. In a Java environment for example, each task may be a set(s) of code that is instantiated and executed for the associated component of each transaction. The transaction component refers to an instance of the code for the task that is executed during a particular transaction in such an implementation. In some embodiments, however, different sets of code can be used or instantiated to perform the same task for different transactions of the same type.

Data for each component of individual transactions that perform each task is set forth in each corresponding row. Transactions1,2,3, and4each include a component for performing each of the identified tasks. Typically, each component of the transactions that perform the same task are of the same component type. Column504sets forth the data for transaction1, column506sets forth the data for transaction2, column508sets forth the data for transaction3, and column510sets forth the data for transaction4. By way of example, transaction1includes a first component that performs the task JSP|Account and has an execution time of 3825 ms. Transaction1further includes a second component having an execution time of 3450 ms for performing the task Servlet|CustomerLookup, a third component having an execution time of 2225 ms for performing the task EJB|Entity|Customer, a fourth component having an execution time of 990 ms for performing the task EJB|Session|Account, a fifth component having an execution time of 755 ms for performing the task JDBC|Oracle|Query, a sixth component having an execution time of 1310 ms for performing the task JDBC|Oracle|Update, and a seventh component having an execution time of 700 ms for performing the task JDBC|Oracle|Query a second time. Transactions2,3, and4also have components for performing each transaction.

Together, the execution times of each transactional component associated with a particular task forms time series data for that task. Time series analytical techniques can be used on this data to determine if a component of a transaction performs abnormally. For example, after determining that a particular transaction has an execution time outside a threshold, the time series data can be used to identify one or more components of the transaction that may be causing the performance problem.

Column512sets forth a normal execution time associated with each task. In one embodiment, the normal execution time for each task is determined by averaging the execution times of each transaction component when performing that task. The normal execution time is a static value in one embodiment that is determined from past component executions prior to beginning transaction tracing. In another embodiment, the normal execution time is a dynamic value. For example, the normal execution time can be recalculated after every N transactions using the component execution times for the last N transactions. More sophisticated time series analytical techniques are used in other embodiments. For example, determining a normal execution time for a task can include identifying trends and seasonal variations in the time series data to predict a normal value for the task's execution time. Holt's Linear Exponential Smoothing is employed in one embodiment to determine a normal execution time for a transaction. Holt's Linear Exponential Smoothing is a known technique that combines weighted averaging and trend identification in a computationally low-cost manner. This technique is very suitable for real-time updates to determine a normal value for task execution time.

Column514sets forth a threshold for each task. If the times series data for a component deviates from the normal execution time for the associated task by more than the threshold, the component is identified as a potential cause of a performance problem. These components can be reported when diagnosing the root cause of an identified transactional performance problem. In one embodiment, threshold deviations are applied so as to only identify components having an execution time that exceeds the normal value by more than the threshold. In other embodiments, if the execution time is below the normal value by more than the threshold, the component can be identified. In yet another embodiment, a threshold execution time is applied directly to the component rather than a threshold deviation.

Row516sets forth the total execution time of each transaction as well as a normal execution time and threshold. The total transaction time is equal to the execution time of each component of the transaction. The normal value can be calculated as previously described. Simple averaging of a number of transaction execution times or more sophisticated time-series techniques applied. The threshold can also be calculated as previously described. Static or dynamic threshold values can be used. The threshold can be expressed as a threshold execution time for the transaction or a threshold deviation from a normal value for the type of transaction.

The total transaction time can be compared to the normal value using the threshold deviation (or compared directly to a threshold transaction time). Those transactions having a total execution time beyond the threshold can be identified and reported, for example, as shown inFIGS. 6 and 7. For the reported transactions, the component data can be examined to determine if there were any abnormalities. For example, transaction3has a total execution time of 13,275 ms. This transaction time is beyond the threshold execution time so the transaction is reported. The JSP|Account component had an execution time of 3900 ms, which deviated from the normal value by more than the threshold. This component can be reported for transaction3. In some embodiments, only transactions having an execution time over the normal value by the threshold are reported. In one embodiment, if a transaction has an execution time above the normal value by more than the threshold, only components having execution times that are above their corresponding normal value are reported. That is, components that have an execution time below the normal by more than their threshold will not be reported. In other embodiments, components with execution times below their normal by more than the threshold amount can be reported as well. For transactions having execution times below the normal by more than the threshold, components above and/or below their normal values by more than the threshold can be reported as well.

InFIG. 9, an embodiment is depicted whereby component data is used to formulate time series data according to the systems involved in the type of transaction. In implementations where each component is directly associated with a particular system, system-level time series data may correspond directly to task-level time series data. In other implementations, such as where transactional components for the same task may execute on different systems in different transactions, such correspondence may not exist and the time series data will be different. Data for multiple tasks may also be grouped by system to consolidate data.

FIG. 9depicts time series data for a set of web-based transactions involving a browser, network, web server, identity server, application server, database server, messaging server, and CICS server. The individual systems are listed in column520. Common web-based transactions represented by the example inFIG. 9could include an initial browser request issued over the network to the web server to complete a purchase, request information, etc. The web server calls the identity server to authenticate the user and then calls the application server to complete the transaction. The application server issues a call to the database server, messaging server, and CICS server to perform the transaction. The application server then returns a result to the web server, which in turn responds to the browser over the network.

Columns522,524,526, and528list the execution times at each system by individual components of transactions1,2,3, and4, respectively. Each entry for a transaction may correspond to the execution time of one or more components of the transactions that are associated with the identified system. By way of example, transaction1includes execution times of 9.8 ms for the browser component(s), 99.8 ms for the network component(s), 9.9 ms for the web server component(s), 198 ms for the identity server component(s), another 10.1 ms for the web server component(s), 51 ms for the application server component(s), 98 ms for the database server component(s), 49.5 for the application server component(s), 101 ms for the messaging server component(s), 21 ms for the application server component(s), 200 for the CICS server component(s), 29.5 ms for the application server component(s), 10.1 ms for the web server component(s), 10.1 ms for the web server component(s), 99.8 ms for the network server component(s), and 10.3 ms for the browser component(s). Particular systems are listed more than once for the transactions to represent that these systems are involved in the transaction at multiple points. Different components of the transactions may be invoked to perform different tasks at the systems during these different points of the transactions.

Normal execution times are depicted in column530for each system during each individual part of the transaction. Like the values depicted inFIG. 8, the normal execution times can be static or dynamic values. Different analysis techniques including simple averaging, Holt's Linear Exponential smoothing, and more can be used to calculate the normal values as before. Threshold deviations from the normal values are set forth in column532. In the system-based technique ofFIG. 9, systems can be identified and reported when their execution time for a transaction is detected as having deviated from its corresponding normal value by the threshold amount or more. Again, deviations above and/or below normal can be used to identify systems, as well as threshold execution times.

Row534sets forth the total execution time for each transaction based on the execution time of each system involved in the transaction. A normal transaction time and threshold are set forth in columns530and532for the overall transaction. InFIG. 9, transaction3has exceeded the normal execution time by more than the threshold. The components corresponding to the database server are beyond the database server normal value by more than the corresponding threshold and can be reported as a potential cause of the performance problem associated with transaction3.

Another set of time series data for a set of transactions is depicted inFIG. 10. The set of transactions depicted inFIG. 10are similar to the set of transactions inFIG. 9. However, the execution times for each individual system have been grouped together and the raw execution times converted into percentages of total transaction time. Column550lists the systems involved in the transactions. Each system's total percentage of transaction time for transactions1,2,3, and4is set forth in columns552,554,556, and558, respectively. For transaction1, the browser makes up 2.0% of the total transaction time, the network makes up 20.0% of the total transaction time, the web server makes up 3.0% of the total transaction time, the identity server makes up 20.0% of the total transaction time, the application server makes up 15.0% of the total transaction time, the database server makes up 10.0% of the total transaction time, and the messaging server makes up 10.0% of the total execution time. Normal values for each system's total transaction time are set forth in column560as a percentage of total transaction time. Threshold deviations from the normal percentage values are listed in column562. In this embodiment, a system can be identified and reported when its percentage of total execution time for a transaction deviates from the normal for the transaction type by more than the threshold. Again, deviations above and/or below the normal value can be detected in various embodiments. Direct threshold percentages can also be used.

Row564sets forth the total execution time for each transaction based on the execution time of each system involved in the transaction. A normal transaction time and threshold are set forth in columns560and562for the overall transaction. While percentages are used for the individual component values, actual time values are used for determining if a transaction is beyond a threshold execution time value. InFIG. 10, transactions3and4have total execution times beyond the threshold. These transactions will be reported. The application server is reported as a possible cause of the performance problem with transaction3and the network is reported as a possible cause of the performance problem with transaction4.

FIG. 11is a flowchart of one embodiment for tracing transactions and providing programmatic root cause analysis of detected performance problems. At step600, the various agents implemented in the transactional system acquire data. Agents may acquire data directly from transaction components running on the same system. Agents may acquire data from other components, such as browsers, external database servers, etc. by monitoring response times and/or installing code such as Javascript to monitor and report execution times. An agent that initiates tracing, for example, may add a script to a web page to monitor the execution time of a browser in performing a transaction. At step602, the various agents report data to the enterprise manager.

In one embodiment, the agent(s) continuously acquire data for the various metrics they are monitoring. Thus, step600may be performed in parallel to the other steps ofFIG. 11. Each agent can be configured to report data to the enterprise manager at step602. For example, the agents may report data every 7.5 seconds or every 15 seconds. The reported data may be data for one or more transactions. In one embodiment, the agent(s) will sample data for a particular transaction at every interval. In one embodiment, an agent associated with a component that receives an initial request starting a transaction will operate as an entry point agent. The entry point agent can modify the request header (e.g., by adding a flag) to indicate to other agents in the system to report data for the corresponding transaction. When the other agents receive the header with the flag, they will report the monitored data for the corresponding transaction to the enterprise manager120.

The enterprise manager can be configured to wake-up and process data at a specified interval. For example, the enterprise manager can wake-up every 15 seconds and process the data from the agents reported during two 7.5 second intervals. This data may be appended to a spool file or query file at step602. More information regarding the collection of data by the agents and processing by the enterprise manager can be found in U.S. patent application Ser. No. 11/033,589, entitled “Efficient Processing of Time Series Data,” incorporated herein by reference in its entirety.

The enterprise manager formulates time series data for the various components of the monitored transactions at step604. The enterprise manager can create a data structure such as those depicted inFIGS. 8,9, and10in one embodiment, although other data structures can be used. The enterprise manager can formulate time series data by task as depicted inFIG. 8, or by system as depicted inFIGS. 9 and 10.

The method depicted inFIG. 11can be performed for each transaction being monitored. As such, step604can include appending component data for the selected transaction to previously collected data. At step606, the enterprise manager determines if the total transaction time exceeded a threshold. Step606can include comparing the total transaction time to a threshold time or determining whether the total time deviated from a normal transaction time by more than a threshold value. If the total transaction time did not exceed the corresponding threshold, tracing for the transaction completes at step608.

If the total transaction time exceeds the threshold, component data for the transaction data is identified at step610. The component data can be maintained by individual tasks with which the transactional components are associated as shown inFIG. 8, or by system as shown inFIGS. 9 and 10. At step612, the enterprise manager determines if a first component of the transaction exceeded the threshold for the associated task or system. The enterprise manager determines if the component execution time deviated from a normal value for the task or system by more than a threshold in one embodiment. In another embodiment, the component execution time (or percentage) is compared to a threshold execution time. If the component has exceeded the relevant threshold, the component is identified as a potential cause of a transaction performance problem at step614.

After identifying the component or determining that it did not exceed its threshold, the enterprise manager determines at step616whether there are additional components of the transaction to analyze. If additional components remain, the method proceeds to step612where the enterprise manager examines the execution time of the next component. After analyzing each component of the transaction, the enterprise manager reports the identified components at step618. Step618can include making an indication in the graphical user interface depicted inFIGS. 6 and 7. The identified components can be highlighted in transaction snapshot window402for example. Other indications can be used as well.

Thresholds for analyzing transaction execution times are dynamically updated using time-series analysis techniques in one embodiment. These analysis techniques can be performed in real-time for each transaction type.FIG. 12is a flowchart depicting one technique for providing dynamic thresholds in one embodiment.FIG. 12can be performed as part of step606inFIG. 11in one embodiment. At step702, the enterprise manager determines if the threshold for the particular type of transaction is to be updated. The enterprise manager may be configured to update the threshold for a particular type of transaction after receiving data for a certain number of transactions of that type. Other techniques may be employed to determine when to update a threshold for a type of transaction. If the threshold is to be updated, the enterprise manager identifies the execution time of the last N transactions for which the manager received data. The actual number of transactions can vary by implementation. A new threshold for the type of transaction is developed at step706. In one embodiment, step706includes determining a normal value for the execution time of the particular type of transaction. The threshold can then be set to a time at a certain level or percentage (variable) above and/or below the normal value. The threshold may also be expressed as a threshold deviation from the normal time (above and/or below). Thus, step706can include determining a new normal time for the transaction type and/or a new threshold to be applied. After developing the new threshold and/or normal value for the transaction type, the new values are applied for the particular transaction being analyzed at step708.

The thresholds used when analyzing the individual components of transactions can also be updated dynamically.FIG. 13is a flowchart depicting one method for dynamically updating a threshold for analyzing transactional components as the possible root cause of performance problems. In one embodiment,FIG. 13is performed at step614when analyzing a component execution time for a transaction. At step720, the enterprise manager determines if the task or system threshold information corresponding to the type of component being analyzed is to be updated. The enterprise manager updates threshold information after receiving data for a particular number of transactions that include a component associated with the particular task and/or system in one embodiment. Other update periods can be used. If the threshold information is not to be updated, the component is analyzed using the existing threshold and/or normal data for the particular task.

If the threshold data is to be updated, the enterprise manager identifies the execution times of components associated with the particular task during the last N transactions at step722. The number of transactions can vary by embodiment and particularly, on the type of analysis techniques to be employed at step724. A normal value for the particular task is determined at step724using the identified data. In one embodiment, the last N execution times are averaged. In other embodiments, trends and seasonal variations can be identified to predict a new normal value. Holt's Linear Exponential Smoothing is used in one implementation to combine weighted averaging and trend identification in a low-cost way for a real-time update of the normal value. At step756, the enterprise manager determines whether the threshold for the task is to be updated. In some embodiments, a threshold is used that is expressed as a deviation from normal. This value can remain the same regardless of the normal value determined at step724. In other embodiments, the threshold deviation is changed as well. If the threshold is to be updated, the enterprise manager updates the necessary value at step728. A new threshold deviation can be selected or a new threshold execution time selected. At step730, the new threshold deviation and/or normal execution time is applied to analyze the particular component.

FIG. 14is a flowchart describing one embodiment of a process for reporting data in the transaction trace table400. The process ofFIG. 14is performed by a workstation in one embodiment. In step800, the workstation receives transaction information from enterprise manager120. In step802, the data is stored. In step804, the data is added to the transaction table as a new row on table400.

FIG. 15is a flowchart describing one embodiment of a process for displaying a transaction snapshot. In step820, the GUI receives a selection of a transaction. That is, the user selects one of the rows of transaction trace table400. Each row of transaction trace table400represents data for one particular transaction. The user can select a transaction by clicking on the row. In other embodiments, other means can be used for selecting a particular transaction. In step822, the data stored for that selected transaction is accessed. In step824, the axis for the transaction snapshot is set up. In one embodiment, the system renders the time axis along the X axis. For example, in the embodiment depicted inFIG. 6, the time axis is from zero ms to 6000 ms. The zoom slider in snapshot header404(seeFIG. 6) is used to change the time axis. In some embodiments, configuration files can be used to change the time. In one embodiment, the actual lime representing the axis for call stack position is not rendered. However, the axis is used as described herein. In step826, the view for the root component is drawn. For example, in transaction snapshot402, the view for “JSP|Account” is drawn. In step828, views for each of the components of the root component are drawn. Additionally, the system recursively draws views for each component of each higher level component. For example, looking atFIG. 6, the first root component JSP|Account is drawn. Then, the components of the root component are drawn (e.g., “Servlets|CustomerLookup” is drawn). Then, recursively for each component, a view is drawn. First, a view is drawn for EJB|Entity|Customer, then the components of EJB|Entity|Customer are drawn (e.g. JDBC|Oracle|Query and JDBC|Oracle|Update). After the components for EJB|Entity|Customer are drawn, the view for EJB|Session|Account is drawn, followed by the component JDBC|Oracle|Query.

FIG. 16is a flowchart describing one embodiment of a process for drawing a view for a particular component. In step850, the relative start time is determined. In one embodiment, if the view is the root component the start time is at 0 ms. If the view is not from the root component, then the timestamp of the start of the component is compared to the timestamp of the start of the root component. The difference between the two timestamps is the start time for the component being rendered. In step852, the relative stop time is determined. By relative, it is meant relative to the root component. Thus, the stop time is determined for the component being rendered. The stop time of the component being rendered is compared to the stop time of the root component. The difference in the actual stop time of the root component as compared to the actual stop time of the component under consideration is subtracted from the stop time of the root component in the transaction snapshot402. In step854, the X values (time axis) of the start and end of the rectangle for the view are determined based on the relative start time, relative stop time, and the zoom factor. Based on knowing the relative start time, the relative stop time, and the extent of the zoom slider, the exact coordinate of the beginning of the rectangle and the end of the rectangle can be determined. In step856, the Y values (call stack position axis) of the top and bottom of the rectangle are determined based on the level of the component. That is, the Y values of all of the rectangles are predetermined based on whether it is the root component, the first component thereof, second subcomponent, third subcomponent, etc. In step858, the view is added to the transaction snapshot. In step860, an additional view box for the calling component is also added. The calling component is a component that invokes the component being drawn. For example, in the transaction snapshot of402, the calling component of Servlets|CustomerLookup is JSP|Account. At step862, the view for the component in transaction snapshot402is highlighted if the component data indicates that the component exceeded its relevant threshold. Step862is optional. In other embodiments, different indications can be made in transaction snapshot402for components that exceed a threshold during the transaction.

FIG. 17is a flowchart describing one embodiment of a process for reporting detailed information about a component of the transaction. That is, when the user selects one of the components in transaction snapshot402, detailed information is provided for that component in component information region408, analysis region410and properties region412. In step870, the GUI receives the user's selection of a component. In step872, the stored data for the chosen component is accessed. In step874, the appropriate information is added to component information region408. That is, the stored data is accessed and information indicating the type of component, the name of the component, and the path to the component are accessed and reported. Each of these data values are depicted in component information region408. In step876, data is added to the analysis region410. That is, system accesses the stored duration (or calculates the duration), the timestamp, the start of the component relative to the start of the root component, and determines the percentage of transaction time used by that component. These values are displayed in the analysis region410. The percentage of transaction times is calculated by dividing the duration of the selected component by the duration of the root component and multiplying by 100%. Step876can include providing an indication if the component exceeded its relevant threshold. In step878, data is added to the properties region. In one embodiment, the properties region will display the method invoked for the component. In other embodiments, other additional parameters can also be displayed. In one embodiment, regions408,410, and412are configurable to the display whatever the user configures it to display.

The user interface ofFIG. 8also includes a set of drop down menus. One of these menus can be used to allow the user to request a text file to be created. In response to the request by the user, the system will write all (or a configurable subset) of the information that is and/or can be displayed by the graphical user interface into a text file. For example, a text file can include the category, component name, timestamp, duration, percentage of the transaction time, URL, userID, host, process, agent, all of the called subcomponents and similar data for the called subcomponents. Any and all of the data described above can be added to the text file.

The above discussion contemplates that the filter used by the agent to determine whether to report a transaction is based on execution time. In other embodiments, other tests can be used. Examples of other tests include choosing based on UserID, provide a random sample, report any transaction whose execution time varies by a standard deviation, etc.