Abstract:
Systems and methods are described that analyze complex systems and identify potential performance bottlenecks that may affect capacity and response time. The bottlenecks are identified to resolve problems originating at a specific subsystem(s).

Description:
BACKGROUND OF THE INVENTION 
     The invention relates generally to monitoring complex systems in which some subsystem performances are either hidden from external monitoring or the transaction flows across these subsystems are unknown. More specifically, the invention relates to systems and methods that monitor and estimate complex system load and determine transaction capacities. 
     A complex system is defined as a large number of integrated subsystems or applications that are coupled together and are used to solve complex business problems such as provisioning of services. One system architecture approach taken to investigate the flow of the functions performed by a complex system is to use performance data of the functions collected by a Workflow Manager (WFM), develop a response time bottleneck model, and apply the data to estimate the model parameters to make predictions about current and future capacity and response times for the functions. Once the problem functions have been identified, a root cause analysis may be performed on the underlying subsystems. 
     For complex systems having many supporting subsystems and applications, it is common to monitor hardware health such as CPU consumption, memory usage, and other parameters, of all the supporting subsystems. However, the WFM response times and subsystem software bottlenecks are not typically monitored. 
     Identifying poor subsystem performance using response times may be complicated by focusing on running the complex system efficiently. While running hardware components such as CPUs at high utilization may reduce the number of hardware components required, to end users, running efficiently is not the same as running acceptably. The response times&#39; performances may be greater than stated targets. It is possible to have high efficiency but low acceptability. Aside from end users bearing the indirect costs of poor performance, the direct costs are increased staff. 
     There is a need to balance the tradeoff between additional systems costs and staffing costs, and the necessity to monitor response times for acceptability with respect to load. Both experience and analysis indicate that for complex functions traversing many subsystems, an average response time curve may be thought of as being flat or constant until reaching a specific load at which point the response times sharply increase. For these types of cases, there may not be an early warning such as a slow degradation in the response times to indicate a need to monitor and model response times. 
     What is desired are systems and methods that analyze complex systems and identify potential problems before they manifest into actual problems. 
     SUMMARY OF THE INVENTION 
     The inventors have discovered that it would be desirable to have systems and methods that analyze complex systems and identify potential performance bottlenecks. Embodiments identify functions having bottlenecks that may affect their capacities and response times. The bottlenecks are identified to resolve problems originating at a specific subsystem(s). 
     Embodiments use a data centric approach rather than a system architecture approach and know little about how the underlying subsystems work together to service the complex system and treat the complex system like a black box. Methods obviate examining all of the component subsystems for potential problems by identifying one or more subsystem(s) to be investigated and repaired. 
     One aspect of the invention provides a method for determining capacity in complex systems by setting capacity limits for transactions that assist in identifying performance bottlenecks. Methods according to this aspect of the invention include listing the service transaction types, identifying Workflow Manager (WFM) data where invocation data records for each service transaction type are located, retrieving the WFM data, parsing the WFM data to acquire the invocation data records for each service transaction type, for a given transaction type, determining average response time RT h  aggregation levels and volume counts count h , wherein the volume counts count h  is the traffic load and is a count of successful transactions within a predetermined measurement interval, performing a linear regression of data pairs (count h ,RT h ), determining an average Response Time Model RTM(x), estimating response time model z 0 , z 1  and z 2  parameters from the average response time RT h  and volume counts count h  data, calculating a maximum throughput 
             1     z   2           
that the complex system can handle, calculating a capacity warning value L 1 , calculating a capacity limit value L 2  and analyzing a time trend of volume counts count h  to determine when the capacity warning value L 1  and capacity limit value L 2  will be reached if current volume counts count h  trends continue.
 
     The details of one or more embodiments of the invention are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the invention will be apparent from the description and drawings, and from the claims. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is an exemplary complex system high level flow through many interrelated subsystems. 
         FIG. 2  is an exemplary system framework. 
         FIG. 3  is an exemplary method. 
         FIG. 4  is an exemplary plot of transaction type average response time vs. function load, with linear and quadratic limits that identify response time curve knees. 
         FIG. 5  is an exemplary plot of average response time vs. counts showing a response time model curve overlay and a table of results. 
         FIG. 6  is an exemplary plot of counts vs. time (date) showing a performance projection to predict when the counts will cross a capacity warning value L 1  and a capacity limit value L 2 . 
     
    
    
     DETAILED DESCRIPTION 
     Embodiments of the invention will be described with reference to the accompanying drawing figures wherein like numbers represent like elements throughout. Before embodiments of the invention are explained in detail, it is to be understood that the invention is not limited in its application to the details of the examples set forth in the following description or illustrated in the figures. The invention is capable of other embodiments and of being practiced or carried out in a variety of applications and in various ways. Also, it is to be understood that the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of “including,” “comprising,” or “having,” and variations thereof herein is meant to encompass the items listed thereafter and equivalents thereof as well as additional items. 
     The terms “connected” and “coupled” are used broadly and encompass both direct and indirect connecting, and coupling. Further, “connected” and “coupled” are not restricted to physical or mechanical connections or couplings. 
     It should be noted that the invention is not limited to any particular software language described or that is implied in the figures. One of ordinary skill in the art will understand that a variety of alternative software languages may be used for implementation of the invention. It should also be understood that some of the components and items are illustrated and described as if they were hardware elements, as is common practice within the art. However, one of ordinary skill in the art, and based on a reading of this detailed description, would understand that, in at least one embodiment, components in the method and system may be implemented in software or hardware. 
     Embodiments of the invention provide methods, systems, and a computer-usable medium storing computer-readable instructions. Components of the invention may be enabled as a modular framework and/or deployed as software as an application program tangibly embodied on a program storage device. The application code for execution can reside on a plurality of different types of computer readable media known to those skilled in the art. 
     By way of background, a complex system is a large number of subsystems and component applications integrated into a platform to solve a defined business problem. There may be tens to hundreds of subsystems that make up a complex system. Subsystems are for the most part developed independently of other subsystems and component applications and may service multiple business functions. One example of a business problem is the AT&amp;T U-verse provisioning process. The AT&amp;T U-verse is a group of services provided over Internet Protocol (IP) that includes television service, Internet access, and voice telephone service. To obtain AT&amp;T U-verse service, a customer would contact an AT&amp;T Customer Service Representative (CSR) who would provision the service for the customer. To do the provisioning requires several steps with each step performing a specific function or a specific set of functions that interact with multiple “backend” systems to complete the business transaction of provisioning. Example transactions include validating a customer&#39;s address, checking Digital Subscriber Lines (DSLs), selecting a tier of service, checking billing, and others. The CSR will often interface with a WFM. 
     At a high level, the WFM is the front end a system user or CSR would interact with to use the complex system. It provides transactions that a user can choose or it may be a simple interface to various backend systems. Once the user chooses a transaction, the WFM will handle any mundane interactions with the backend systems to complete the selected transaction and present the results to the user. In many cases, the WFM collects statistics on transactions such as transaction start and finish times, whether the transaction was a success or failure, and others. Other business functions interacting with various backend subsystems may or may not be known to the CSR or WFM. 
       FIG. 1  shows a high level flow of a complex system  101  through many interrelated subsystems (service nodes) A 1 , A 2 , A 3 , B 1 , B 2 , S 1 , S 2  and component applications function A, function B used to service various end user functions, and a WFM  103 . A function may span multiple subsystems with subsystems supporting multiple functions. All of the functions do not have to use all of the subsystems and all subsystems do not have to support all functions. The timing of various function invocations is critical to the individual subsystems. 
     None of the complex system&#39;s architecture, subcomponents&#39; hardware and software properties, work flows of the various transactions across the system components or other work being performed by the backend subsystems need be known in order for embodiments to determine the overall system&#39;s capacity for the various transactions. Embodiments are applicable even as the underlying architecture and components change over time and as new data becomes available. By definition, the complex system capacity for a given transaction is the number of functions per unit time that can be processed while still meeting the required response time of an end user for this function, if known, or below the limit at which small changes in load would cause disproportionate changes in response time. Embodiments employ a capacity warning value L 1  and a capacity limit value L 2 . 
       FIG. 2  shows an embodiment of a system framework  203  and  FIG. 3  shows a method. The framework  203  which may be part of a network management server  221  includes a network interface  205  coupled to a network and configured to acquire data from a WFM  103 , network reachability information, as well as network status information to perform performance and capacity modeling. The network interface  205  is coupled to a performance monitoring engine  207 , an inventory database  209 , a calculation engine  211  and a processor  213 . The processor  213  is coupled to storage  215 , memory  217  and I/O  219 . 
     The framework  203  stores acquired WFM  103  data in the database  209 . The framework  203  may be implemented as a computer including a processor  213 , memory  217 , storage devices  215 , software and other components. The processor  213  is coupled to the network interface  205 , I/O  219 , storage  215  and memory  217  and controls the overall operation of the computer by executing instructions defining the configuration. The instructions may be stored in the storage device  215 , for example, a magnetic disk, and loaded into the memory  217  when executing the configuration. The invention may be implemented as an application defined by the computer program instructions stored in the memory  217  and/or storage  215  and controlled by the processor  213  executing the computer program instructions. The computer also includes at least one network interface  205  coupled to and communicating with a network WFM  103  to interrogate and receive WFM  103  data. The I/O  219  allows for user interaction with the computer via peripheral devices such as a display, a keyboard, a pointing device, and others. 
     In order to find the performance and capacity bottlenecks for a set of transactions supporting a business service using a given complex system, a list of service transaction types or functions is compiled (step  301 ). For example, each business service is made up of a set of transaction types such as “get home address”, “get billing address”, “get billing information” and others. 
     Given a set of transaction types, embodiments identify where a transaction log for the transaction type data is kept and retrieve the data using agents, bots, spiders, or other network mechanisms (steps  303 ,  305 ). Typically, a WFM  103  has a transaction log. Analyzing the log data for a particular transaction type obtains records for each invocation of that transaction type such as “start time”, “end time”, “success”, “fail”, “duration” and others. Typically, the invocation data record time stamps are in the format of date:hour:min:sec, “success” is 1 (yes) or 0 (no) and “duration” (response time) may be in seconds or other time units. The formatting depends on the systems involved. Parsing the WFM data yields sets of invocation data records for each transaction type, such as for “get home address”. If the parsed WFM data does not include an invocation record for “duration”, the difference between “start time” and “end time” is calculated. “Duration” is a service transaction type invocation response time. Each transaction service type invocation made by either a customer or a CSR creates an invocation data record (step  307 ). 
     The data from complex systems that is readily available is the average Response Time (RT) which is the average of the response time for transactions completing within a time period and can be calculated per time interval. The count of transactions completing within the per unit time interval (carried load) is close to the count of transactions initiated within the time interval (offered load). By analyzing an RT vs. load curve, a determination of which functions are showing significant sensitivity to load may be made and subsystems that may be approaching capacity exhaustion may be identified. 
     Filtering incorrect data is performed on acquired invocation data records to arrive at an accurate average estimate (step  309 ). Filtering rules for each service transaction type may be developed and employed. In some cases it is clear that some data is in error, for example, an “end time” before a “start time” which is filtered out. 
     The specific statistical nature of the transaction traffic determines the length of the time interval over which the average is calculated and the carried load is counted. Generally, the measurement interval should be long enough that it has at least 30 or more completed transactions. Once the time interval has been set, for example, one hour, the invocation data records for a service transaction type are aggregated over the measurement intervals and those intervals which have too few transactions are excluded from further analysis. It is possible that as load increases in the future, the choice of the time interval, e.g., one hour, may be shortened to a quarter of an hour (step  311 ). 
     For a measurement interval h, average response time and carried load is calculated (step  313 ). Given the measurement interval h, a count h  is calculated of successful transaction type invocations and their associated average response time h  or RT h  over the measurement intervals having sufficient counts of transactions by transaction type. 
     Over time, it is possible that some of the “hidden” subsystems had a known abnormal behavior over the measurement interval, e.g., periodic backup or software upgrades or a subsystem failure causing an outage that stops servicing of transaction requests. The abnormal behavior may be detected by summing all “success” and “failed” invocation data records within the measurement interval and summing all “failed” invocation data records within the measurement interval. If the “failed” sum is greater than a predefined percentage of the “success” and “failed” sum, this behavior is not representative of normal system behavior and may be considered abnormal. The measurement intervals affected are discarded from the analysis unless upon inspection or via a Rules Based Expert System (RBES) it is determined that the transaction type was unaffected by the event in which case the measurement intervals need not be discarded (step  315 ). 
     Not all functions may show a strong RT h  to load sensitivity. This is due in part to the bottleneck capacity of the subsystems supporting a function possibly being large relative to the current load, or the function may have sufficiently high priority on the bottleneck resource as to not be sensitive to the current level of congestion, or the function&#39;s bottleneck resource is dynamically allocated. For such cases, the RT h  vs. load curves may be flat or decreasing. 
     From the transaction type invocation data records, the slope s of an ordinary linear least squares (straight line fitting) regression is calculated. The slope s is observed whether it is negative. For a given transaction type, the data pairs (count h ,RT h ) are acquired and regression coefficients are derived.  x  is the average over all the pairs of the count h  observations.  y  is the average of the RT h  observations. The centered counts variables x h  are calculated x h =(count h −  x ) and the centered average response time variables y h  are calculated y h =(RT h −  y ). Mathematically, the slope s is 
     
       
         
           
             
               
                 
                   s 
                   = 
                   
                     
                       ∑ 
                       
                           
                       
                       ⁢ 
                       
                         
                           x 
                           h 
                         
                         ⁢ 
                         
                           y 
                           h 
                         
                       
                     
                     
                       ∑ 
                       
                           
                       
                       ⁢ 
                       
                         
                           ( 
                           
                             x 
                             h 
                           
                           ) 
                         
                         2 
                       
                     
                   
                 
               
               
                 
                   ( 
                   1 
                   ) 
                 
               
             
           
         
       
     
     (step  317 ). 
     If the slope s is negative, then no further analysis is performed (steps  319 ,  321 ). The expectation is that RT h  is increasing with respect to increasing load. Transaction types showing a negative slope s are examined for bad invocation data records (step  323 ). 
     Embodiments make use of queuing theory and model the RT h  of a specific function for a given load x as the sum of the sojourn time in each of the service nodes required to process the function with special attention given to a bottleneck service node. A Response Time Model RTM(x) may be used to determine transaction type load at which expected response times cross predefined delay objectives. The resulting RTM(x) model is a variant of the standard processor sharing model and has the form 
     
       
         
           
             
               
                 
                   
                     
                       
                         
                           R 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           T 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           
                             M 
                             ⁡ 
                             
                               ( 
                               x 
                               ) 
                             
                           
                         
                         = 
                           
                         ⁢ 
                         
                           
                             
                               z 
                               0 
                             
                             ⁡ 
                             
                               ( 
                               x 
                               ) 
                             
                           
                           + 
                           
                             
                               z 
                               1 
                               ′ 
                             
                             
                               1 
                               - 
                               
                                 
                                   ρ 
                                   b 
                                 
                                 ⁡ 
                                 
                                   ( 
                                   x 
                                   ) 
                                 
                               
                               - 
                               
                                 
                                   z 
                                   2 
                                   ′ 
                                 
                                 ⁢ 
                                 x 
                               
                             
                           
                         
                       
                     
                   
                   
                     
                       
                         ≈ 
                           
                         ⁢ 
                         
                           
                             z 
                             0 
                           
                           + 
                           
                             
                               z 
                               1 
                             
                             
                               1 
                               - 
                               
                                 
                                   z 
                                   2 
                                 
                                 ⁢ 
                                 x 
                               
                             
                           
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   2 
                   ) 
                 
               
             
           
         
       
     
     where z 0 (x) is the amount of time to traverse all non-bottleneck service nodes in a function&#39;s work flow, z′ 1  is the unnormalized expected amount of time to traverse the bottleneck node under no or very low transaction load, z 1  is the normalized expected amount of time to traverse the bottleneck node under no transaction load, z′ 2  is the unnormalized average amount of time used per transaction of the bottleneck node and z 2  is the normalized average amount of time of the bottleneck node (direct and indirect) per transaction. The z 0 (x) sojourn time is taken to be a constant z 0  since as the load approaches the bottleneck&#39;s limiting rate, the non-bottleneck service nodes contribute significantly less to the RT h  relative to the bottleneck contribution. ρ b (X) is the additional utilization on the bottleneck node due to other functions&#39; load. This utilization is rewritten to reduce it into a background constant plus a part that varies in conjunction with the transaction&#39;s load
 
ρ b (x)≈ρ b0 +δx  (3)
 
     where ρ b0  is the utilization of the bottleneck node that is independent of the transaction&#39;s load and δx is the utilization that is dependent on the transaction&#39;s load x. Substituting, RTM(x) (2) may be rewritten in a simpler form 
     
       
         
           
             
               
                 
                   
                     
                       
                         
                           R 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           T 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           
                             M 
                             ⁡ 
                             
                               ( 
                               x 
                               ) 
                             
                           
                         
                         = 
                           
                         ⁢ 
                         
                           
                             
                               z 
                               0 
                             
                             ⁡ 
                             
                               ( 
                               x 
                               ) 
                             
                           
                           + 
                           
                             
                               z 
                               1 
                               ′ 
                             
                             
                               1 
                               - 
                               
                                 
                                   ρ 
                                   b 
                                 
                                 ⁡ 
                                 
                                   ( 
                                   x 
                                   ) 
                                 
                               
                               - 
                               
                                 
                                   z 
                                   2 
                                   ′ 
                                 
                                 ⁢ 
                                 x 
                               
                             
                           
                         
                       
                     
                   
                   
                     
                       
                         = 
                           
                         ⁢ 
                         
                           
                             z 
                             0 
                           
                           + 
                           
                             
                               z 
                               1 
                               ′ 
                             
                             
                               1 
                               - 
                               
                                 ρ 
                                 
                                   b 
                                   ⁢ 
                                   
                                       
                                   
                                   ⁢ 
                                   0 
                                 
                               
                               - 
                               
                                 δ 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 x 
                               
                               - 
                               
                                 
                                   z 
                                   2 
                                   ′ 
                                 
                                 ⁢ 
                                 x 
                               
                             
                           
                         
                       
                     
                   
                   
                     
                       
                         = 
                           
                         ⁢ 
                         
                           
                             z 
                             0 
                           
                           + 
                           
                             
                               z 
                               1 
                               ′ 
                             
                             
                               1 
                               - 
                               
                                 ρ 
                                 
                                   b 
                                   ⁢ 
                                   
                                       
                                   
                                   ⁢ 
                                   0 
                                 
                               
                               - 
                               
                                 
                                   ( 
                                   
                                     δ 
                                     + 
                                     
                                       z 
                                       2 
                                       ′ 
                                     
                                   
                                   ) 
                                 
                                 ⁢ 
                                 x 
                               
                             
                           
                         
                       
                     
                   
                   
                     
                       
                         = 
                           
                         ⁢ 
                         
                           
                             z 
                             0 
                           
                           + 
                           
                             
                               
                                 z 
                                 1 
                                 ′ 
                               
                               
                                 1 
                                 - 
                                 
                                   ρ 
                                   
                                     b 
                                     ⁢ 
                                     
                                         
                                     
                                     ⁢ 
                                     0 
                                   
                                 
                                 - 
                                 
                                   
                                     ( 
                                     
                                       δ 
                                       + 
                                       
                                         z 
                                         2 
                                         ′ 
                                       
                                     
                                     ) 
                                   
                                   ⁢ 
                                   x 
                                 
                               
                             
                             ⁢ 
                             
                               ( 
                               
                                 
                                   1 
                                   
                                     1 
                                     - 
                                     
                                       ρ 
                                       
                                         b 
                                         ⁢ 
                                         
                                             
                                         
                                         ⁢ 
                                         0 
                                       
                                     
                                   
                                 
                                 
                                   1 
                                   
                                     1 
                                     - 
                                     
                                       ρ 
                                       
                                         b 
                                         ⁢ 
                                         
                                             
                                         
                                         ⁢ 
                                         0 
                                       
                                     
                                   
                                 
                               
                               ) 
                             
                           
                         
                       
                     
                   
                   
                     
                       
                         = 
                           
                         ⁢ 
                         
                           
                             z 
                             0 
                           
                           + 
                           
                             
                               
                                 z 
                                 1 
                               
                               
                                 1 
                                 - 
                                 
                                   
                                     z 
                                     2 
                                   
                                   ⁢ 
                                   x 
                                 
                               
                             
                             . 
                           
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   4 
                   ) 
                 
               
             
           
         
       
     
     By using the observations on the pairs of carried load counts count h  and associated RT h  (count h ,RT h ), the z parameters z 0 , z 1  and z 2  may be estimated. One method to estimate the z 0 , z 1  and z 2  parameters is to let the estimates be the constrained values of z that minimize the sum of
 
(log(RT h )−log(RTM(count h   |z ))) 2   (5)
 
     subject to the constraints z 0 ≧0, z 1  and z 2 &gt;0. The z 0 , z 1  and z 2  parameters are found by using the standard conjugate-gradient method. Other methods for estimating the z 0 , z 1  and z 2  parameters are possible (steps  325 ,  327 ). 
             1     z   2           
is the estimated maximum function load or throughput for the transaction type that the complex system can handle. Functions with large z 0 +z 1  are functions that are the major contributors to the sojourn times of business requests when multiple functions must be called on sequentially to complete the request. z 0 +z 1  is the best expected response time that may be achieved.
 
     For transactions using a complex system, there may not be explicit RT h  requirements for each of the different transactions. Moreover, in some cases the requirements may be set poorly. For these cases, the general capacity rule is to avoid the “knee” of the response curve. Such requirements are useful but only if the definition of what is the knee is clear. In general, the response curve knee is a vague concept. However, Kleinrock&#39;s knees definitions are used for the response curve knees to set capacity limits. 
       FIG. 4  shows an RTM curve (solid line)  401 , a capacity warning value or linear limit L 1  is the x coordinate of the point  403  that is the tangent intersection of the line going through the origin (dotted line)  405  with the RTM curve  401 . A capacity limit value or quadratic limit L 2  is the x coordinate of the point  407  that is the tangent intersection of a quadratic curve going through the origin (broken line)  409  with the RTM curve  401 . These are Kleinrock&#39;s knees. 
     The capacity warning value L 1  is the first warning of load increasing to a point where the performance of the complex system may start to degrade based on the knees analysis. When z 0  is not zero, the capacity warning value L 1  is 
     
       
         
           
             
               
                 
                   
                     L 
                     1 
                   
                   = 
                   
                     
                       
                         
                           z 
                           0 
                         
                         + 
                         
                           z 
                           1 
                         
                         - 
                         
                           
                             ( 
                             
                               
                                 
                                   z 
                                   0 
                                 
                                 ⁢ 
                                 
                                   z 
                                   1 
                                 
                               
                               + 
                               
                                 z 
                                 1 
                                 2 
                               
                             
                             ) 
                           
                           
                             1 
                             2 
                           
                         
                       
                       
                         
                           z 
                           0 
                         
                         ⁢ 
                         
                           z 
                           2 
                         
                       
                     
                     . 
                   
                 
               
               
                 
                   ( 
                   6 
                   ) 
                 
               
             
           
         
       
     
     When z 0  is equal to zero, the capacity warning value L 1  is 
     
       
         
           
             
               
                 
                   
                     L 
                     1 
                   
                   = 
                   
                     
                       1 
                       
                         2 
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         
                           z 
                           2 
                         
                       
                     
                     . 
                   
                 
               
               
                 
                   ( 
                   7 
                   ) 
                 
               
             
           
         
       
     
     The capacity limit value or quadratic limit value L 2  is the safe operating point based on the knees analysis. When z 0  is not zero, the capacity limit value L 2  is 
     
       
         
           
             
               
                 
                   
                     L 
                     2 
                   
                   = 
                   
                     
                       
                         
                           4 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           
                             z 
                             0 
                           
                         
                         + 
                         
                           3 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           
                             z 
                             1 
                           
                         
                         - 
                         
                           
                             ( 
                             
                               
                                 8 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 
                                   z 
                                   0 
                                 
                                 ⁢ 
                                 
                                   z 
                                   1 
                                 
                               
                               + 
                               
                                 9 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 
                                   z 
                                   1 
                                   2 
                                 
                               
                             
                             ) 
                           
                           
                             1 
                             2 
                           
                         
                       
                       
                         4 
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         
                           z 
                           0 
                         
                         ⁢ 
                         
                           z 
                           2 
                         
                       
                     
                     . 
                   
                 
               
               
                 
                   ( 
                   8 
                   ) 
                 
               
             
           
         
       
     
     When z 0  is equal to zero, the capacity limit value L 2  is 
     
       
         
           
             
               
                 
                   
                     L 
                     2 
                   
                   = 
                   
                     2 
                     
                       3 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       
                         z 
                         2 
                       
                     
                   
                 
               
               
                 
                   ( 
                   9 
                   ) 
                 
               
             
           
         
       
     
     (step  329 ). 
     When there are stated performance requirements, the associated capacity limit is calculated (steps  331 ,  333 ). 
     Results are displayed and an analysis of the time trends suggest, if current trends continue, when the various capacity limits will be reached (step  335 ). Displays may include a plot of the data (count h ,RT h ). 
       FIG. 5  shows a plot of RT h  vs. count h    501  with an overlay of the resulting model curve  503  and a summary table of results  505 . The table  505  shows the capacity limit value (linear knee) L 1 , the capacity warning value (quadratic knee) L 2 , the estimated maximum load 
             1     z   2           
for this transaction type, the current estimated maximum utilization in percent, the estimated RT h  at low load, a model fit factor indicating how well the model and data agree and the number of data points M used in the analysis.
 
     A time series plot may also be displayed.  FIG. 6  shows count h  data vs. time (date)  601  with lines at a capacity limit value (linear knee) L 1    603  and at a capacity warning value (quadratic knee) L 2    605 . A linear growth performance projection  607  is fit and used to predict when the counts data is expected to cross the capacity limit value L 1    603  and the capacity warning value L 2    605 . 
     For the different transaction types, given the analysis and the projected time until capacity exhaust L 2 , it may be necessary to increase capacity or shift load prior to the capacity limit date and continue to track the load growth and response times for possible changes to the capacity limit date. An analysis may be performed after a new release of major subsystems to understand what the field capacity is and if the complex system can support the projected load increases until the next planned capacity increase. If not, systems engineers work to close the gap between demand and system capacity. 
     Knowing the capacity limits and the time series analyses for different transaction types allows operations to focus on transactions that are in overload or that are approaching overload. Rather than examining all subsystems in a complex system, embodiments indicate which transaction types are of concern which in turn identifies one or more subsystems. 
     One or more embodiments of the present invention have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the invention. Accordingly, other embodiments are within the scope of the following claims.