Productivity metrics for application software systems

A system and method for generating a class of weighted intensity performance metrics and a class of relative performance metrics for a computer system. Additionally, a method for accumulating data for generating the two classes of performance metrics. A record is generated that is associated with a transaction performed by an application process A.sub.i running on a computer system. A tick count is generated representing a total amount of a resource of the computer system that is consumed by application process A.sub.i for completion of the transaction. Each increment of the tick count represents a unit of consumption of the resource. A performance metric corresponding to a quality of performance of the computer system for application process A.sub.i and related to a metric value M1 is generated based on records and tick counts that are associated with the transaction performed by the application process A.sub.i.

BACKGROUND OF THE INVENTION 
1. Field of the Invention 
The present invention relates to the field of computer software systems. 
More particularly, the present invention relates to a system and a method 
of generation of performance metrics for application computer systems. 
2. Description of the Related Art 
Large-scale application system performance is a major issue in a computing 
system production environment. Unexpected performance problems may arise 
because some resources of a computing system may be quickly exhausted, 
while other resources remain abundant. Most system performance problems 
are of this kind of problem because software applications, or modules, 
that are simultaneously executed on a computing system compete for the 
limited resources of the system and, consequently, may adversely affect 
the overall performance of the system. 
Performance considerations for large-scale computer systems include, for 
example, performance tuning, job scheduling, and capacity planning. 
Traditionally, the key tools used for resolving performance problems have 
been performance metrics and include a spectra of performance metrics, 
such as hardware metrics, operating system related metrics, connectivity 
metrics, application performance metrics, etc. 
Conventional application performance metrics belong to two major 
categories: global metrics and vector metrics. Global metrics, sometimes 
referred to as total or integral metrics, accumulate information relating 
to consumption of a specific resource for an entire application system 
over a specified period of time. The resource consumption information is 
accumulated without differentiation between different applications and 
processes that execute transactions in consuming the resource. Global 
metrics based on system CPU usage, I/O utilization, or records per hour, 
for example, do not provide quantitative insight to system performance 
because system loads vary, and because contributions by different types of 
application processes, for example, batch transactions and queries, are 
not properly taken into account. That is, summing a total number of 
records (or a total of another unit of work) produced on a computing 
system by different application processes over certain period of time is 
the equivalent of adding apples and oranges because the combination of 
system components and their respective activities constantly change. Even 
qualitatively, conventional global metrics may be misleading. 
The second category of metrics, vector metrics, usually consists of few 
selected total metrics. Even if a vector metric consists of parameters 
related to every specific application process of the system, a vector 
metric is not particularly usable and does not provide significant insight 
to performance of a computing system. For example, it is difficult to 
decide whether a computing system having a vector defined as (CPU1=55, 
IO1=118, RECORDS1=83, CPU2=18, IO2=339, RECORDS2=117) performs better that 
another computing system having a vector defined as (CPU1=25, IO1=148, 
RECORDS1=63, CPU2=118, IO2=139, RECORDS2=137). 
What is needed is an objective performance metric that can be used for 
evaluating application system performance and that has a reasonable 
physical justification. What is also needed is a performance metric that 
is sensitive to a contention between different processes for all types of 
resources and data. 
SUMMARY OF THE INVENTION 
The present invention provides a family of metrics producing objective 
measures of system performance changes during the course of any repetitive 
task executed by complex application systems, have a reasonable physical 
justification, and are applicable to any application environment. The 
metrics of the present invention can be used directly and/or in 
application management software as a foundation for performance alerts, 
corrective events, and for production scheduling and planning. In either 
configuration, the metrics of the present invention are an application 
system performance tuning instrument that can be used for studying a group 
of applications, a type of application, or a whole system, and for 
providing clues for optimum system tuning. 
The present invention provides a method for constructing a class of 
weighted-intensity application computer system metrics. According to the 
invention, at least one computer system resource is selected with each 
selected computer system resource being consumed by at least one 
application process. A unit of work associated each computer system 
resource is selected with the selected unit of work associated each 
computer system resource being produced by consumption of the computer 
system resource by each application process consuming the computer system 
resource, the same unit of work produced by each application process 
consuming the computer system resource. A ratio of an amount of units of 
work to an amount of the selected computer system resource consumed by 
each application process is generated for each selected computer system 
resource. Lastly, a class of weighted-intensity application computer 
system metrics is formed with each member of the class of 
weighted-intensity application computer system metrics being a series of 
the generated ratios. 
The ratio of the amount of units of work to the amount of the selected 
computer system resource consumed for the i-th application process is a 
weighted-intensity for the i-th application process and is defined as 
##EQU1## 
wherein, Ticks.sub.i is the amount of the selected computer system 
resource consumed by the i-th application process, and Rec.sub.i is the 
amount of units of work produced by the i-th application process consuming 
the selected computer system resource. Preferably, both Ticks.sub.i and 
Rec.sub.i are measured over a same predetermined time interval and in a 
same application computer system environment. 
The method of the present invention also includes the steps of weighting 
A.sub.i for the i-th application process by using a respective share of 
the selected computer system resource of the total selected computer 
system resource consumed by the i-th application process, and generating 
each term X.sub.i of the class of weighted-intensity application computer 
system metrics for the i-th application process defined as 
##EQU2## 
wherein N is a total number of application processes consuming the 
selected computer system resource. 
Each term X.sub.i (t) of the class of weighted-intensity application 
computer system metrics for the i-th application process is generated with 
each term X.sub.i (t) being a relative weighted-intensity of consumption 
of the selected computer system resource by the i-th application process 
and being defined as 
##EQU3## 
wherein A.sub.i (0) is measured at a predetermined time. Each term X.sub.i 
of the class of weighted-intensity application computer system metrics for 
the i-th application process is generated defined as 
##EQU4## 
wherein, N is a total number of application processes consuming the 
selected computer system resource. 
The class of weighted-intensity application computer system metrics M1 is 
defined as 
##EQU5## 
wherein, K is a total number of computer system resources. 
The present invention also provides a method for measuring performance of a 
computer system. A measurement of consumption of a computer system 
resource during execution of an application computer system is defined 
with the application computer system including at least one application 
process. At least two measurements for an i-th application process are 
made with a time interval between the two measurements including at least 
one transaction successfully completed by the application process in 
consuming the computer system resource. Each measurement includes an 
amount of units of work (Rec.sub.i) produced by the i-th application 
process and an amount of the computer system resource (Ticks.sub.i) 
consumed by the i-th application process in producing the amount of units 
of work Rec.sub.i. A class of weighted-intensity computer system metrics 
is generated having at least one term, each term of the weighted-intensity 
class computer system metrics being based on the measurements for the i-th 
application process. Each transaction is a repetitive transaction that can 
be repeated any number of times, and the time interval between each 
measurement is a predetermined time interval and is equal between each 
measurement. 
According to the invention, the metrics of the present invention for the at 
least one application process on the computer system can be modeled. 
Similarly, the at least one application process can be executed and 
measured on the computer system. 
The present invention also provides the step of forming an intensity term 
for the i-th application process defined as 
##EQU6## 
Further, the present invention provides the step of generating a baseline 
intensity term for the i-th application process defined as 
##EQU7## 
wherein the time interval between the two measurements forming the 
baseline intensity term is defined as a base measurement interval. The 
relative intensity term for the i-th application process is generated 
defined as 
##EQU8## 
A class of relative weighted-intensity application computer system metrics 
for the i-th application process is generated defined as 
##EQU9## 
wherein, N is a total number of application processes. 
An incremental relative intensity for the i-th application process is 
generated defined as 
##EQU10## 
A weighted-intensity metric M3 for a predetermined computer system 
resource is generated defined as 
##EQU11## 
A relational class metric M4 corresponding to a number of averaged records 
of the computer system is generated defined as 
A metric M5 is generated corresponding to a relative measure of 
##EQU12## 
production change of the application computer system for the i-th 
application process and is defined as 
##EQU13## 
According to one alternative, the computer system resource includes a 
plurality of computer system resources consumed by the i-th application 
process, in which case the method of the present invention provides that 
an intensity term for the i-th application process and the j-th computer 
system resource is formed defined as 
##EQU14## 
A baseline intensity term for the i-th application process and the j-th 
computer system resource is generated defined as 
##EQU15## 
wherein the time interval between the two measurements forming the 
baseline intensity term is defined as a base measurement interval. 
A relative intensity term for the i-th application process and the j-th 
computer system resource is generated defined as 
##EQU16## 
A class of relative weighted-intensity application computer system metrics 
for the i-th application process and the j-th computer system resource is 
generated defined as 
##EQU17## 
wherein, K is a total number of computer system resources. 
An incremental relative intensity for the i-th application process and the 
j-th computer system resource is generated defined as 
##EQU18## 
A weighted-intensity metric M3 for the j-th computer system resource is 
generated defined as 
##EQU19## 
A relational class metric M4 corresponding to a number of averaged records 
of the computer system is generated defined as 
##EQU20## 
A metric M5 corresponding to a relative measure of production change of the 
application computer system for the i-th application process and the j-th 
computer system resource is generated defined as 
##EQU21##

DETAILED DESCRIPTION 
The present invention provides a method for creating of a family of 
performance metrics for the specific area of application computer systems. 
Areas such as hardware performance, operating system performance, etc., 
are outside the scope of the present invention. Application systems are a 
set of processes or tasks under control of a computing operating system. 
Each process of the set does not differ significantly from other processes 
of the set, but differs to the extent of providing a different end result. 
Processes of an application system produce "units of work" consuming all 
types of resources of the computing system and its operating system. An 
example of "units of work" are records inserted into a database. An 
example of a computing system resource that is consumed by an application 
in producing "units of work" is CPU time measured in units of ticks, such 
as seconds. For simplicity, the term "ticks" will be used herein for 
increments of a predetermined unit of resource consumption. A repetitive 
transaction is an integral set of operations forming a specific task 
within a job. A transaction is a set of operations forming a specific task 
within a job. The present invention is concerned with the number of units 
of work produced in a defined interval. 
An aspect of an application system is a multiplicity of data sources, of 
processes and "units of work". Such complex systems are easily 
characterized by a multidimensional set of parameters or a vector of 
parameters. Such a characterization is not usable for the major purposes 
that metrics are used, that is, system improvement by performance tuning 
and better job (process) scheduling. This is because in a real life 
situation, the contribution of different components of a system varies 
significantly over time and comparison of vector-based metrics for two 
different states of a system, for example, metrics measured on two 
different days, is not possible for a general case. 
A natural solution is a single-number, global (or integral) type metric 
that is based on contributions from different processes, such as CPU 
utilization. Nevertheless, such integral metrics are too crude of an 
instrument to be an effective tool for application system tuning. Attempts 
to translate multidimensional metrics into a single-number parameter have 
resulted in the equivalent of adding of apples and oranges. 
To solve the problem of a single-number metric, the present invention 
provides a method for generating of a family of metrics for a broad 
spectra of application systems. The essence of metrics of the present 
invention is based on the following principles. The first principle is 
that the metric is a function of a weighted relative improvement, or 
change, in the consumption of one or more resources by application 
processes producing units of work. The principle of relative improvement 
can be applied to functions of computing system resources. Another 
principle on which the present invention is based is that resources that 
are consumed to produce units of work by every application process are 
consumed in only one way. That is, resources are normalized (divided) by 
the number of units of work that are produced by consumption of the 
resource by the application processes. Application system activity is 
usually described in terms of transactions that are repetitive integral 
sets of actions and are usually specific tasks within a job. 
Time is not a part of the metric calculations of the present invention. 
Instead, a time interval is used over which transactions are executed that 
consume resources and generate units of work. The time interval sets the 
limits for metric granularity. The present invention does not consider how 
resource consumption is distributed within the time interval. The time 
interval over which a specific metric is calculated over is selected to be 
long enough for guaranteeing that the execution of a particular 
transaction is complete at least, for example, 95% of the time. 
Preferably, the time interval is selected so that a number of transaction 
execution periods are complete. 
The metrics provided by the present invention are equally valid in any 
system or subsystem having multiple application processes, and can be used 
for modelling an application or group of applications that run on a 
computer system. For example, the performance of a subsystem of 
database-related processes in a UNIX system can be measured and evaluated 
either directly as the processes are executed or modelled before their 
actual operation in a system. Each metric provided by the present 
invention can be calculated separately for each group of application 
processes of the subsystem, or the metrics can be calculated for all 
application processes of the subsystem collectively. There is no 
limitation on the time interval for collecting data for calculating the 
metrics of the present invention. The only requirement is that every 
process measured must produce repetitious "units of work" (transaction) 
for the time interval being analyzed. Of course, for the metrics of the 
present invention to accurately reflect system performance problems, all 
problems with individual applications or modules, such as programming 
errors or bugs, must be resolved in advance. 
To illustrate the present invention, consider the example of a simple 
metric M1 created for characterizing an application system that generates 
records in a database system as a unit of work. For this example, metric 
M1 reflects consumption of only one resource, such as CPU usage measured 
in seconds or ticks, by several application processes. Here, the variable 
Ticks.sub.i is used for identifying the CPU consumption (usage) by i-th 
process. The result of execution of repetitive transactions, in this case 
generation of database records, are units of work identified as Rec.sub.i 
for each i-th process. Another exemplary units of work that is equally 
applicable to this example is blocks of data copied locally from file to 
file or over the network. Again, a transaction is a set of operations 
forming a specific task within a job, and the present invention is 
concerned with the number of units of work produced in a defined interval. 
Let us consider transactions executed in a framework of an exemplary 
computing system 10 shown in the Figure. Computing system 10 has a 
plurality of workstations 11a-11d and host computers/database servers 13 
and 14 connected in a well-known manner to a network 12, such as a local 
area network (LAN) or a wide area network (WAN). A gateway 23 can also be 
connected to network 12 so that network 12 can be connected to other 
computer networks 24. Database server 13 includes an RDBMS 22, 
applications software 23, and a metrics module 21. Database server 14 
includes a system management module 20. Workstations 11 run client 
applications that request resource consumption. 
File server 14 runs a system management application 20 for managing 
computing system 10. A metrics module 21 can be run on any host running 
applications and provides metrics values for display or collection on 
workstations 11a-11d, or for use by system management application 20. 
Information and data are accumulated as a result of execution of every 
transaction, that is, a number of units of work (i.e., records) and CPU 
usage measured in ticks. Information and data regarding other resources 
are accumulated in the same manner. Metrics module 21 retrieves the 
resource consumption data with a time interval defined by the system 
administrator and calculates the metric M1 (defined below). Application 20 
collects applications metrics in addition to many other metrics that are, 
for example, hardware-related, operating system-related, 
connectivity-related, etc. The collected metrics are used by application 
20 primarily for two purposes: to set alarms that are based on conditions 
defined by a system administrator or by default, and for preparing and 
executing corrective actions based on metric values. For example, an 
operating system level metric can be used for setting an alarm if usage of 
a certain disk exceeds a threshold level. Extra space can be allocated on 
another disk as a corrective action. In the case of an application metric, 
the beginning of an execution of a new report might lead to an increase in 
data contentions and a strong degradation of the application metric value 
might trigger an alarm. A corrective action by an operator or by a system 
administrator might be to reschedule execution of the report to a 
different time slot. 
As previously mentioned, the metrics of the present invention are based on 
a discrete series of measurements of a specific resource consumption, for 
example, CPU consumption, and the units of work produced from the 
consumption of the resource by a specific process or task, such as 
records. That is, 
##EQU22## 
where A.sub.i (t) is the number of ticks per record for the i-th 
application process measured at time t, and where (Rec).sub.i is the 
number of records (units of work) generated by the i-th process as a 
result of CPU consumption (resource) by the i-th process as measured by 
the number of ticks (Ticks).sub.i. Resource consumption occurs in the 
course of executing any number of transactions, usually a sufficiently 
large number, that were successfully completed during a fixed time 
interval preceding time t. The fixed time interval is selected to be the 
same for all measurements at all times t for all processes, including t=0. 
(Ticks).sub.i and (Rec).sub.i are collected during the fixed time 
interval. 
A baseline for the i-th application process, A.sub.i (0), is measured at 
any arbitrary time, for example, at t=0, or when a particular record is 
generated for the first time for the i-th application. That is, 
##EQU23## 
is a total number of tick counts Ticks.sub.i (0) consumed by the i-th 
application per number of records Rec.sub.i (0) produced for the i-th 
application measured at time t=0 for a time interval preceding t=0. 
A measure of relative application process performance with respect to the 
baseline measurement is 
##EQU24## 
The basic metric M1 of the present invention is defined to be: 
##EQU25## 
where N is a number of application processes (tasks) involved in the 
metric. M1 is a relative improvement of the system performance weighted by 
ticks and represents an implementation of principles discussed earlier. 
The present invention provides a second relative performance metric M2 that 
has the same form as M1, but relates to incremental performance changes of 
a computer system. For M2, 
##EQU26## 
As an alternative for metric M2, a metric M21 is defined using 
##EQU27## 
where, t-1 is a base measurement time immediately preceding measurement 
time t. Metric M21 is an incremental relative improvement of the system 
weighted by ticks. 
An extension of metric M1 is a third metric M3, where tick values for the 
i-th application A.sub.i are used in Equation (25) instead of X values. 
Metric M3 is an averaged number of ticks per record for the application 
system and is defined to be 
##EQU28## 
Metric M3 is not a relative metric by itself, but is used for generating a 
relative production metric M5. 
Metrics M1, M2 and M3 are each measures of an execution performance of an 
application system on a computer system, and, as mentioned, each are 
equally valid in any system or subsystem having multiple application 
processes. For example, each metric M1, M2 and M3 can be calculated 
separately for a database subsystem for a group of application processes, 
or the metrics can be calculated for all application processes 
collectively. If a relative metric for several resources consumed by 
application systems in a course of execution of similar transactions for 
fixed time intervals are created, then metric M1 (and, similarly, metric 
M2) can be generalized as: 
##EQU29## 
where M1.sub.j is a relative metric for K resources of the same type, that 
is, j=1, 2, . . . , K. 
Metric M3 is not relative and can not be generalized in this way. Instead, 
metric M3 is used for generating another relative metric M5. 
The present invention also provides metrics for evaluating the production 
of the computer system by taking into account that the value 
.SIGMA.(Ticks).sub.i is a total number of ticks consumed by an application 
system and M3 is an average number of ticks per record. The number of 
"averaged" records produced by a system is then defined by 
##EQU30## 
Metric M4, like metric M3, not a relative metric. As such, these two 
metrics can be used for limited goals, for consideration of one resource, 
or in relative metric like M5. 
An improvement in productivity may come from an increase in the number of 
records or in a decrease of cost, that is, decrease of number of ticks per 
record. Thus, a fifth metric M5 is defined as 
##EQU31## 
and can be generalized for all K types of resources. j=1, 2, . . . , K, 
using 
##EQU32## 
While the present invention has been described in connection with the 
illustrated embodiments, it will be appreciated and understood that 
modifications may be made without departing from the true spirit and scope 
of the invention.