COMPUTER SCIENCE & TECHNOLOGY:

TECHNOLOGY ASSESSMENT:

ADP INSTALLATION PERFORMANCE

MEASUREMENT AND REPORTING

NBS Special Publication 500-53

U.S. DEPARTMENT OF COMMERCE

National Bureau of Standards
NATIONAL BUREAU OF STANDARDS

The National Bureau of Standards' was established by an act ot Congress on March 3, 1901. The Bureau's overall goal is to strengthen and advance the Nation's science and technology

and facilitate their effective application for public benefit. To this end, the Bureau conducts

research and provides: (1) a basis for the Nation's physical measurement system, (2) scientific and technological services for industry and government, (3) a technical basis for equity in trade, and (4) technical services to promote public safety. The Bureau's technical work is per- formed by the National Measurement Laboratory, the National Engineering Laboratory, and

the Institute for Computer Sciences and Technology.

THE NATIONAL MEASUREMENT LABORATORY provides the national system of

physical and chemical and materials measurement; coordinates the system with measurement

systems of other nations and furnishes essential services leading to accurate and uniform

physical and chemical measurement throughout the Nation's scientific community, industry,

and commerce; conducts materials research leading to improved methods of measurement,

standards, and data on the properties of materials needed by industry, commerce, educational

institutions, and Government; provides advisory and research services to other Government

agencies; develops, produces, and distributes Standard Reference Materials; and provides

calibration services. The Laboratory consists of the following centers:

Absolute Physical Quantities^ — Radiation Research — Thermodynamics and

Molecular Science — Analytical Chemistry — Materials Science. THE NATIONAL ENGINEERING LABORATORY provides technology and technical ser- vices to the public and private sectors to address national needs and to solve national problems; conducts research in engineering and applied science in support of these efforts; builds and maintains competence in the necessary disciplines required to carry out this research and technical service; develops engineering data and measurement capabilities;

provides engineering measurement traceability services; develops test methods and proposes

engineering standards and code changes; develops and proposes new engineering practices;

and develops and improves mechanisms to transfer results of its research to the ultimate user. The Laboratory consists of the following centers:

Applied Mathematics — Electronics and Electrical Engineering^ — Mechanical

Engineering and Process Technology^ — Building Technology — Fire Research —

Consumer Product Technology — Field Methods.

THE INSTITUTE FOR COMPUTER SCIENCES AND TECHNOLOGY conducts

research and provides scientific and technical services to aid Federal agencies in the selection,

acquisition, application, and use of computer technology to improve effectiveness and

economy in Government operations in accordance with Public Law 89-306 (40 U.S.C. 759), relevant Executive Orders, and other directives; carries out this mission by managing the Federal Information Processing Standards Program, developing Federal ADP standards

guidelines, and managing Federal participation in ADP voluntary standardization activities; provides scientific and technological advisory services and assistance to Federal agencies; and

provides the technical foundation for computer-related policies of the Federal Government.

The Institute consists of the following centers:

Programming Science and Technology — Computer Systems Engineering.

'Headquarters and Laboratories at Gaithersburg, MD, unless otherwise noted; mailing address Washington, DC 20234. 'Some divisions within the center are located at Boulder, CO 80303.
COMPUTER SCIENCE & TECHNOLOGY:

TECHNOLOGY ASSESSMENT:

ADP INSTALLATION PERFORMANCE

MEASUREMENT AND REPORTING

U.S. DEPARTMENT OF COMMERCE, Juanita M. Kreps, Secretary

Luther H. Hodges, Jr., Under Secretary

Jordan J. Baruch, Assistant Secretary for Science and Technology

NATIONAL BUREAU OF STANDARDS. Ernest Ambler. Director

Issued September 1979

Carol B. Wilson

Institute for Computer Sciences and Technology

National Bureau of Standards

Washington, DC 20234
Reports on Computer Science and Technology

The National Bureau of Standards has a special responsibihty within the Federal

Government for computer science and technology activities. The programs of the

NBS Institute for Computer Sciences and Technology are designed to provide ADP

standards, guidelines, and technical advisory services to improve the effectiveness of

computer utilization in the Federal sector, and to perform appropriate research and

development efforts as foundation for such activities and programs. This publication

series will report these NBS efforts to the Federal computer community as well as to interested specialists in the academic and private sectors. Those wishing to receive

notices of publications in this series should complete and return the form at the end

of this publication.

National Bureau of Standards Special Publication 500-53

Nat. Bar. Stend. (U.S.) Spec. Publ. 500-53, 37 pages (Sept. 1979) CODEN: XNBSAV

Library of Congress Catalog Card Number: 79-600154

U.S. GOVERNMENT PRINTING OFFICE WASHINGTON: 1^79

For sale by the Superintendent of Documents, U.S. Government Printing Office, Washington, D.C. 20402

Stock No. 003-003-02123-7 Price $2.00

(Add 25 pereent adiiltioinl for other than U.S. mailing)
Technology Assessment:

ADP Installation Performance

Measurement and Reporting

Table of Contents

1.0 Introduction 1 1.1 Overview 1 1.2 ADP Installation Model 2 2.0 Installation Performance Management ... 4

2.1 Upper Management .. 4

2.2 User Community . 5

2.2.1 Measuring User Satisfaction ... 5 2.2.2 Setting Service Levels 5 2.2.3 Measuring Service 6

2.2.4 Reporting to Users ........ 6

2.2.5 Correcting Poor Service 7

2.3 Resource Management 8

2.3.1 Operational Standards 8

2.3.2 Resource Utilization Standards . . 9

2.3.3 Forecasting 9

2.4 Vendor 10

2.5 Summary 11

3.0 Installation Performance Activities in the

Federal Sector 12

3.1 Auditor Perspective 12

3.2 Installation Perspective 14

3.3 Summary 15

4.0 Current Inadequacies in Installation

Performance Practices 16

4.1 Lack of Measurabil i ty 16

4.2 Lack of Well-Defined Terms 17

4.3 Lack of "Good" Values 17

4.4 Lack of Statistical CPE Techniques . . 18

5.0 Proposed Remedial Work . . . . , 20

5.1 Standard Performance Measures 20

5.2 Data Base for Normative Performance . . 20

5.3 Development of Statistical CPE

Techniques 21

5.4 Summary 21

6.0 Bibliography 22

Appendix A - Informational Model 27

iii
ACKNOWLEDGMENTS

The author wishes to thank Peter J. Calomeris and Wayne D.

Bennett of the National Bureau of Standards for their

contributions to this report. Mr. Calomeris performed the

literature search, provided copies of all the references

used in this report, and prepared the bibliography. Mr.

Bennett performed the auditing agency site visits and

contributed to the material in Appendix A.

i V
Technology Aissessment:

ADP Installation Performance

Measurement and Reporting

Carol B. Wilson

Abstract

This report compares the current status of ADP

installation performance measurement and reporting in

the Federal ADP community to the best practices as found in the Federal and private sectors and described

in the literature. The comparison reveals that more

effort could be expended by the Federal sites in the area of computer performance management. The principal

obstacles to more and better performance programs are perceived to be the lack of needed measures on many

systems and the magnitude of the effort involved in

accessing and analyzing the measures which are available. The report discusses several underlying causes

for these obstacles and makes three recommendations

which could partially relieve the situation: (1) development of standard performance measures, (2) developnent of a Government-wide data base for normative

performance ranges, and (3) development of statistical

computer performance evaluation techniques.

Key words: Computer performance evaluation (CPE);

computer performance management (CPM) ; installation

management; installation performance management;

performance measurement and reporting; resource

management; standard performance measures.

1.0 Introduction

1.1 Overview

The purpose of this report is to

examine the current status of ADP

installation performance measurement

and reporting in the Federal ADP

community, to identify and discuss any

current inadequacies in practice, and

to make recommendations for the cor- rection of those inadequacies. For

purposes of this report, an ADP

installation is considered to be the

complete set of resources dedicated to

meeting the automated processing needs

of an agency. Typically, the ADP

installation consists of the computer

hardware itself along with the

administrative, systems support, user support, and operational staff.

The major purpose of an ADP

installation should be to meet the

computing needs of its user population

at the least cost. While this qoal

may be simply stated, the measurement

of the degree of success in meeting

this goal is a complex problem.

Ideally we would like to be able to

solve this problem by using precise

quantitative measurements that are

1
objective and free of human biases and

intuitive judgments. This is not

possible because the performance we are trying to measure depends on complex human behavior and also on the

complexities of modern computer system

relationships which are not fully

understood. In addition, randomly

varying workloads, or even cyclical

workloads, pose problems in optimizing

service and cost.

Further, most of today's ADP

installations must meet the demands of

a multi-user, multi-application

environment. The work handled by the

installation is a composite of the

computing needs of all of the usersy

each putting different demands on the

installation. Because the demand for

resources varies by user, improving

the performance of some resources for

the benefit of specific users may be

at the expense of other users or may

even have an overall detrimental

effect on installation performance.

Hence, the goal of an installation

cannot be to provide optimal service

to each user but to provide "good"

service to all users with respect to

organizational goals and priorities.

It should be noted that state-of-theart techniques, n-th generation

machines and higher level languages do

not necessarily result in a "good"

installation but are only important in

that they enhance the potential for

better productivity, lower cost, or

better user service.

To meet the goal of providing

good service at the least cost, the

installation manager must do capacity

planning. Gilmore [GILMM 78-pl]

defines capacity as "the workload that

a well- tuned system can process without exceeding the limits of [defined

by] user performance objectives."

Capacity planning can thus be thought

of as optimizing the installation

resources (e.g., tuning the system) to

the user workload at the least cost,

for both the short and long terms. To

be successful in this function, the

ADP manager must undertake several

tasks: (1) communicate with upper

management, (2) serve and influence

the user community, (3) manage his

resources, and (4) maintain vendor

relations. A Computer Performance

Management (CPM) program supports the

ADP manager in all of these functions

by providing quantitative data for the

decision-making process. We will

discuss these tasks and how they

relate to installation performance by

introducing a model below.

1.2 ADP Installation Model

The model below represents a functional view of an ADP installation

as seen from the perspective of the

ADP installation manager.

UPPER MANAGEMENT

ADP INSTALLATION

USER — MANAGER — '— VENDOR

RESOURCES

Figure 1: ADP Installation Model

2
We choose to place the installation manager in the center of the

model because he/she controls or influences the factors which contribute to the success or failure of the

ADP installation. Although the problems and responsibilities that the ADP

manager must face are similar to those

encountered by managers in other functional areas in an organization, it is

becoming more widely recognized that

there are extra dimensions to the

control of the ADP installation. The

ADP manager must be accountable to

upper management for the economic

operation of the installation. He is

responsible to the users for providing,

an efficient and effective service.

He must oversee the actions and

obligations of the vendor. And

finally, he must control the resources

of his installation. However, the

on-rushing technological changes and

complexities of the computer field

affect the ADP manager's ability to

meet these responsibilities. For

example, there is rarely a large set

of precedents to aid him in making

many of the short or long range

•decisions which often arise. Secondly, staffing is uncertain as new

technologies require new skills.

Further, the ADP manager is required

;to assume multiple roles, to manage a multi-million dollar facility upon

which the success of many agency

programs depends and to remain

technically knowledgeable in a rapidly

changing technology. And finally, he

must also oversee the mundane

activities which can so easily degrade

installation performance if left

unattended

.

In Section 2 we briefly discuss

each of the four areas of our model by

describing the functions an ADP mana-

•ger must undertake to fulfill his

obligations with respect to each area:

upper management, users, resources,

and vendors. In Section 3 we discuss

the current practices in the Federal

sector. The discussion notes that few

ADP installations fully address the

four areas of concern. In particular,

the upper management relationship is

quite weak. Planning and managing of

installation resources has typically

digressed to a fire-fighting mode of

operation at many sites.

Section 4 discusses possible

underlying causes for this lack of

good resource management. These causes include current inadequacies

with respect to the measur abil i ty of

computer systems, the lack of welldefined terms for performance and

utilization measures, the lack of

"good" values for installation performance, and the lack of statistical

techniques which have been validated

for use in the computer environment

and which can be easily used by non- statisticians. The upper management

problem is not addressed since its

solution lies in the area of management science, not computer science.

Section 5 recommends work which could

be performed to increase installation

performance measurement and reporting:

development of standard performance

measures, generation of a data base

for normative performance, and the

development of "cookbooks" for

statistical techniques to be used in

the evaluation of computer systems.

The thrust of the three recommendations which are made is that more

effort would be expended by Federal

sites in the area of CPM if the task

did not require so much personnel time

and expertise.

3
2.0 Installation Performance

Management

Before we address the current

state of installation performance

practices in the Federal sector, we need to establish a basis for discussion. In this section we will describe the best practices as found in

the Federal and private sectors and

described in the literature. Section

6,0 (Bibliography) contains a list of

literature references which were

reviewed as part of this study. Each

of the four relationships of concern — upper management, user community,

resources, and vendor and the

corresponding functions will be

discussed. These functions should be

performed by the ADP manager to ensure

that his installation is meeting its

goal of providing good service to its users at the least cost. Then, having

established this baseline for installation performance practices, we will

be able (in Section 3) to compare

current practices in the Federal

sector with the baseline.

We will restrict our description

to the more traditional ADP installation centralized batch with

timesharing and remote batch. The

question of informational needs for

networks and other forms of distributed processing is beyond the scope

of this report. However, within the

best practices as set forth below, we will cJiscuss functions which could be

performed by the manager of any size

ADP installation whether it is

oriented toward data processing or toward research and development. Of

course the size of the installation,

its position in the organization, and

its mission will cause variations in

the details in carrying out the

functions given below, although most

of the tasks are necessary for any ADP

installation to be well-managed and

operated

,

2.1 Upper Management

The ADP installation manager has

a responsibility to report to his

upper management on the status of re- quirements, performance, and finances

of the ADP installation. It is best

that the installation manager report

to a person who has sufficient

authority to ensure that the ADP

installation will receive adequate

support for its policies. It is

necessary that the installation manager receive this support from upper

management for the controls, procedures, and management policies which

he establishes at the installation for

the assignment of resources to the users. To help him in this task, he

should present performance information

to upper management that is at a level

of abstraction which can be easily

digested and understood, but is sufficient to support the upper management decision-making process.

To permit the ADP installation to

fully support the goals and objectives

of the organization, upper management

must make the effort to include the

ADP manager in the planning process,

to recognize the impact that organizational plans will have on the ADP

installation, and to know where the

ADP installation fits into the plans.

To aid in the planning process, upper

management needs to know the costs of

various levels of service and their

effects on program schedules, personnel levels, and budget. The ADP

manager must, therefore, relate the

cost of providing service at various

levels to the current configuration

and, perhaps, to future configurations. The ADP manager should project the load which planned projects

will place on installation resources.

The ADP manager should encourage

management to reconsider committing to

ill-conceived or unrealistic ADP

projects. If the ADP manager is

successful in this task, he can save his installation from much of the busy

work which has minimal payoff but

which is a constant drain on resources

.

The ADP manager should keep his

upper management informed about the

current operations and the performance

of the ADP installation with respect

to organizational goals and objectives. That is to say, reports should

be concerned with the effectiveness of

the installation; i.e., how well the

installation is satisfying the user requirements for service. Depending

upon the management style of the

organization, upper management may be

interested in regular reports or may

4
only be interested in the reporting of

exceptions to the expected service

levels. The exact content of the

reports would also vary with management style but could address the

amount of work being done by the

various departments and the installation's ability to process the work

in a timely manner. Reports should

also be presented which are concerned

with budget, revenue, actual cost, and

financial tracking for the installation. Reports concerning component

utilization, etc. are probably

inappropriate for upper management,

although certainly necessary for

proper resource management.

2.2 User Community

The relationship between the ADP

manager and the user community is of

crucial importance to the success of

the installation. The ADP installation exists to serve its users, but

it must also influence their behavior

if the installation is to be both

effective and efficient. The ADP

manager's most difficult task probably

lies in working with and for the user community. The expertise in a typical

user community often ranges from the

very naive to the very sophisticated

user. But even the most sophisticated

users usually have only the vaguest

concept of the overall installation

operation and the functions which the

ADP manager performs. The ADP manager

must have rapport with his users and

instill in them a sense of mutual

cooperation and good will.

There are several general

measures of installation performance

which involve the user community.

They are based upon the acceptance of

the computer as reflected in the

number of departments using the

facility, the use each department

makes of it, and how well the computer

is accepted as an easy-to-use tool.

But these measures only provide a rough estimation of user acceptance

and do not indicate user satisfaction

with the installation. For example,

many departments may use the facility

with ease but still not be satisfied

with the installation because service

is poor. User satisfaction can be one

of the best general indicators of

installation performance. An iterative plan for ensuring user satisfaction is detailed below. It calls for

measuring the current level of user satisfaction, setting service levels,

measuring the service, reporting to

the users about the success or failure

in meeting the service level goals,

and taking corrective action for

missed goals.

2.2.1 Measuring User Satisfaction

One of the accepted means of

ascertaining user satisfaction is to use a well-designed survey. Developing a good survey may be hampered by

two facts: (1) satisfaction is

related to the user's expectation,

which may rise or fall rapidly with

the occurrence of a single event, and

(2) satisfaction is not a single

quantity but a blend of many factors.

Hence the survey must allow many

factors to be evaluated simultaneously

and provide a means of ensuring reliability of the data, but must not be

excessively long lest errors be introduced through fatigue. Pearson [PEARS

77] discusses the requirements for a survey designed to measure user satisfaction and the procedures he used to

develop a comprehensive thirty-nine"

factor questionnaire which takes only

thirty minutes to complete. He dis- cusses the factors used in his questionnaire and why they are important

to computer users.

2.2.2 Setting Service Levels.

The typical user is principally

concerned about the service he re- ceives in meeting his project deadlines and the consistency of the

service. If he is regularly receiving

poor service he is little consoled by

the fact that everyone else is more

fortunate than he. It is unusual for one user to receive consistently

poorer service than other users, but

it is not unusual for him to remember

only the reports that were late, the

jobs that did not come back when he

wanted them, or the day that timesharing response was abysmal. With no defined levels of service there is no basis for the ADP manager or the users

to judge the installation. Hence, in

.order to satisfy the users, it is

essential that the ADP manager know

what the users' service needs really

5
are. The ADP manager and the user community must together establish

service levels for timeliness, availability, accuracy and cost which are acceptable to the users (or classes of

users) and which are reasonable to the

ADP manager. These service levels

then provide one basis for measuring

installation performance. Depending

on the type of installation and users,

the service levels may be categorized

by class of job: batch, timesharing,

etc.; by class of user: payroll,

accounts, etc.? by resource class:

requested and used; or by some other

category. The categories should

reflect the needs of the user community and provide a basis for the

installation manager to understand

more clearly their needs. Satisfactory performance at each service level

is defined in terms of criteria that

must be met by a specified percentage

of jobs in each class.

2.2.3 Measuring Service

The measurement of service levels

depends on both the categories of work

and the types of service which the users demand. For example, measures

of timeliness for batch jobs could be

expressed as the number of jobs ontime (or late) , average turnaround

time, or average throughput. Measures

of timeliness for interactive processing normally include the average times

for responses, as well as the consistency of the responses for similar

tasks [see FIPS 57]. Availability is

measured for the total system, as well

as for each subsystem. Reliability

encompasses both hardware components

and software modules. Finally, revenue data is usually a function of

and obtained from the particular

chargeback system being used.

Much of the batch service level

measurement data may be collected from

the accounting logs found on most

vendor systems. However, most vendor

systems do not provide the necessary

data for interactive or remote batch

measures. Special hardware devices,

or modifications to the system software, may be necessary in these cases. Depending on the definitions of the

service measures, manual logging

procedures may even be necessary.

2.2,4 Reporting to Users

Many installations overlook the

benefits that can be derived from

organized reporting to the user community. Proper publicity can help

both to accurately report on the

service the users are receiving and to

influence their behavior. The attitude of the user community toward the

computer installation is a determining

factor in the effectiveness of the

installation. The installation may

not be meeting its service level

objectives, but may be performing its

functions in the best way possible.

That is, there are times when the

causes of poor performance may

actually lie in the practices of the

users, and changes in these practices

are needed to improve overall installation performance. However, dissatisfied users are often unlikely to be

cooperative in making changes which

would improve the efficiency and

effectiveness of the facility as a whole

.

Reports to the user community

should indicate how effectively the

installation has been meeting the

service requirements of the user and

what he can expect to receive in the

future. Users are generally more

interested in the number of reports

which are on-time, the batch turnaround time, or the timesharing

response time than in CPU utilization

or any other measures of hardware

performance. Other measures of

effectiveness encompass the areas of reruns and aborts, user complaints,

and the availability and reliability

of the system as seen by the various

user groups or classes of jobs. It is

important that the definitions of

turnaround time, availability, etc.

be well-defined y well-understood and

mutually acceptable to both the ADP

manager and the user community.

Examples have been given in the

literature showing the wide disparities between how the user perceives

(defines) a measure and what is

actually measured by the installation

(for example, see [STEVD 781).

The reports to the users should

display more than averages since a particular user's job is rarely "the

average job," Reports showing distributions for the various measures can

6
demonstrate to the user what service

he can expect to receive most of the

time and that occasionally things are not as good as they should be. Of

course, the reports may only serve to

show him that service is not as good

as he would like and that he should

alter his expectations accordingly or seek to renegotiate the service levels

and/or the cost for service. Reports

displaying service measures by class

over time can indicate to the user what to expect if he begins a different type of processing or if he

changes the characteristics of his

jobs. Reports by time of day, time of

month, or some other cyclical period

may indicate to the user that he may

obtain better service by performing

his work in the off-peak periods.

The reliability information used

in vendor relations, although usually

reported to the users in terms of

availability, is also useful to the

ADP manager in assessing the quality

of service he is providing the user community. An unreliable system

translates into a system which is

unavailable when the user needs it and

into a system which causes the user to

have to repeat lost work. It is quite

possible that an unreliable and

sporadically available system causes more user frustration and hostility

than a slow one. At least the user knows what to expect from a system

giving poor service.

2.2.5 Correcting Poor Service

Poor service may be caused by a true lack of hardware capacity or may

be the result of poor utilization of

existing resources. Some remedies for

poor utilization fall under resource management (see Section 2.3) but

others lie in changing the manner in

which the user community uses the

installation. With respect to the

users, increasing capacity may be

accomplished by influencing the user community to limit their resource

usage through charging practices, by

workload management, or by increasing

user proficiency. Each of these

possible remedies is discussed below.

A method often used by ADP

managers to limit resource usage is

charging the users for the resources

and services they receive. Without

some type of fiscal responsibility,

the workload at an installation may

grow in an uncontrollable manner. Chargeback is more effective when the user of the installation computer and

the person paying the bill are very

closely associated. In installations

where the user has little fiscal

responsibility, or the budget cycle

does not encourage economy of resources, the chargeback method alone

produces little change in user behavior. However, charging users for

their jobs is still often quite an effective means of controlling unrea- sonable requests and preventing

uncontrolled growth. The charges

should be directly related to the

actual resource utilization and should

be understandable to the user so that

he can control his usage of resources.

Further, the charges should be approximately repeatable when the identical

job is run again on the system. Cost

should be linked to the impact the job

has on the installation's performance.

The user who chooses resource demanding, high priority processing should

be charged accordingly. Perhaps, such

a user should be notified what extra

cost he is incurring through his

current mode of operations as compared

to alternate methods, priorities, or times of execution. The chargeback

system should provide economic incentives for the users to request only

cost effective services.

Workload management is a tool of

the ADP manager which is not used

often, although it is well known that

great gains can be made by eliminating

little-needed work. In times of low

usage and great capacity at an installation, the problem of resources

consumed by non-essential workload is

not so crucial. But, as excess capacity begins to dwindle, an audit

of the existing workload should be

performed and definite steps taken to

eliminate unnecessary, and perhaps

even marginal, workload from the

computer. In reality, the audit of

the existing workload for the installation should be an on-going project.

Not only can work of limited worth be

kept off the machine, but workload

statistics can highlight the top users

of resources who may then receive

training or guidance by the ADP

department to enhance their efficiency

in using the system.

7
Another way to reduce resource

usage is by upgrading the proficiency

of the user community through training

programs, meetings, newsletters, technical support, documentation, etc.

General areas for improvement can be

determined by the examination of

processing errors and their causes.

If errors are further categorized by

application or user, customized

approaches can be used for the top

offenders. If installation personnel

are available, they should work with

the users during their planning and

design phases to provide guidance to

better approaches to the solution. It

is often beneficial to optimize the

large production programs. Tools and

package? exist to aid the installation

in this task once problem programs

have been identified. It should be

noted that it is better and easier to

optimize programs during the design

phase than after development.

2.3 Resource Management

A systematic approach to resource

management requires a non-trivial

amount of time, effort, and money but

may provide for better operations in a better accepted installation. Efficient operation of the installation is

needed if the installation is to

operate at least cost. Operational

standards need to be set and tracked.

Standards for hardware utilization

need to be defined, monitored, and

corrective action needs to be taken

when necessary. Forecasting is

required to ensure the continuation of

good service in the future. Performance measurement for resource management may begin with observation and

simple manual techniques and extend

through use of sophisticated software

and/or hardware monitors. The size

and maturity of the installation

should determine the degree of

sophistication of the performance

measurement program.

2.3.1 Operational Standards

Operational standards and procedures should be defined since without approved standards and procedures

there is no basis for performance

evaluation in the short term. They

also provide the basis for long-range

planning needed to meet new service

criteria or simply to continue meeting

the same service levels in an expanding workload environment. Operational

standards might encompass such functions as reading cards, bursting output, handling media, such as mounting

tapes, security measures, etc. Procedural standards would define procedures to follow for unscheduled

restarts, cold starts, malfunctioning

units, etc. These standards and

procedures must not be rigid but

should be adaptable to technological

and environmental changes as they

occur. For a rather comprehensive

document on the subject, the reader is

referred to Standards in Operations

[RICKJ 75]

.

Operational procedures will work

only as well as the people performing

them. Without well-trained and

motivated personnel, the effectiveness

of the installation can be seriously

degraded. The ADP manager should take

steps to ensure that morale is and

remains good at his installation. Low

morale often leads to sloppy work

which will show up as operator errors

requiring reruns, delays in bursting

output that inhibit meeting timeliness

criteria, etc.

In order to compare the actual

performance of the operational procedures to the standards, special logging procedures may need to be implemented in order to capture the exact

information specified by the service

level definitions. For example, the

user may require turnaround time for

local batch jobs to be defined as counter-to-bin time instead of the

more prevalent and easily captured

measure of cards-read to linesprinted. Typically in the operations

spectrum, work flow studies are performed to measure times and volumes

of work in critical areas such as dispatching. Another source of data

is time records and the manual logging

of events. As in hardware management,

such historical data can then be used

to set realistic goals and can also be

used in planning by comparing future

requirements to available personnel

hours. Logs of errors and reruns can

be studied to detect areas which

require further operator training or a modification of software or procedures

for easier use by the operators.

8
2.3.2 Resource Utilization

Standards

Utilization measurements make up

the primary data used in resource

management. Utilization levels by

themselves are not a valid measure of

installation performance but are important only as they affect installation performance criteria, such as service to the user. High utilization

levels may indicate congestion and

consequent system degradation, but

congestion and degradation are not

necessarily related. It should be

remembered that utilization levels are not only a function of internal

computer system characteristics and

configuration but are also strongly

influenced by external factors such as scheduling policies, priority processing requirements, etc. The effects

that changes in configuration, workload and application, or operating

system have on utilization of components and subsystems should be examined

for possible causal relationships.

Information on such relationships

could be used both in analysis and

tuning in the operational environment

and in the planning process. The

inverse process of examining changes

in trends of utilization to ascertain

changes in the external environment is

also helpful in the planning process

and for optimizing operational

procedures

,

Bell [BELLT 75b] introduced the

concept of using control limits to

help the ADP manager to allocate and

plan for computer resources by tracking resource consumption to know when

to act and what action to take.

Simply stated, a control limit is a value of a variable which should not

be consistently exceeded if the

installation is to continue to operate

efficiently. If the control limit is

exceeded some action should be taken

to return the variable to its proper

range; i.e., the controlled area. Control limits for hardware units and

subsystems can be developed by determining the values at which the users begin to complain, by plotting service

levels versus component utilizations,

etc. Both resource usage and service

levels must be tracked to ensure the

acceptability of installation performance. When an installation is not

meeting a particular service level.

there can be many causes. By looking

at resource consumption relative to

the control limits set to keep service

levels under control, the ADP manager

can anticipate what corrective action

to take and when to take it to ensure continuation of proper service.

Once control limits are set there

is a need for a consistent set of

management policies for both control

limits and service levels. The level

of response will not necessarily be

the same for all control limit violations but should be commensurate

with the significance of the violation. The ADP manager must ensure

that enough time is allowed for the

evaluation of variances and that

corrective action is undertaken when

there really is a consistent problem,

but that no action is taken for random

occurrences. The use of control

limits can reduce costs due to excessive capacity and can enhance the

ability to consistently provide good

service by reducing the need for firefighting and hasty subjective

j udgments

.

2.3.3 Forecasting

Forecasting should be done by the

ADP manager so that his installation

remains responsive as new work is

added, the existing workload changes,

or when changes in installation con- figuration could result in less cost.

The goals of forecasting are to

predict future workloads and their resource consumption; to anticipate

problem areas, such as bottlenecks, in

time to prevent them from occurring;

and to predict performance changes

that may result from changes in system

configuration and workload. With the

long procurement times in the Federal

sector and difficulties in enhancing

existing systems speedily, forecasting

accurately over a several year span is

necessary. However, even when the ADP

manager can accurately forecast the

future workload for his installation,

he may be faced with the problem of

forecasting in the face of a rapidly

changing technology, which may affect

the manner of processsing in the

future. Examples in recent years

include the shift from batch to online

processing and the introduction of

data base management systems.

9
The process of forecasting can not be performed by the ADP manager in

isolation but must involve the users

who are the source of information on

the characteristics of new workload,

new service requirements, or the

expansion or contraction of existing

workload. The obvious patterns in

workload can be predicted from trend

data, but the effect from the introduction of new applications, etc.

must also be estimated and such information comes from the user community. The workload must be described

in terms which can be understood by

the user and can be used as the basis

for the user to project his future

requirements. But, the user description must also be translatable into

terms used by the forecasting technique. The process of forecasting may

also involve upper management,

particularly if the installation must

acquire capacity for future use and,

hence, may require an interim subsidy.

The forecasting process begins

with the characterization of the

current and projected workload. There

are many techniques for characterizing

workloads which range from simple job

stream analysis to determine the

average job characteristics to sophisticated clustering algorithms which

statistically categorize job classes.

The degree to which the workload can be accurately described strongly

influences the accuracy of any forecasting done by the ADP manager.

However, sometimes the ADP manager

needs only rough estimations and the

simpler techniques suffice in such

instances

.

Job stream profiles are typically

graphs displaying percentage of jobs

versus: CPU time per job, number of

tapes per job, elapsed time per job,

memory used or required per job, etc.

The graphs provide a visual means for

the ADP manager to partition his

installation's workload. Job stream

profiles also provide rough estimations of workload changes and can indicate when the composition of the

workload is staying steady. In most

installations, whether production or

not, the composition of the workload

will remain steady for substantial

periods of time. Hence, the average

job, or jobs, for the installation may

be defined by resource consumption

characteristics. For example, the

average compile-only job may use two

seconds of CPU time and require twenty

thousand units of memory. These average jobs can be used then as coarse

measures of workload when measuring

system productivity or doing "ball

park" forecasting of workload. Workload statistics of this type can point

to growth in specific applications or changes in the workload composition.

Continuing workload monitoring can provide data to support benchmarking

and simulation work used in the

planning process. More sophisticated

means of workload characterization —

for example, clustering — also exist

(for example, see [AGRAA 78], [DEVEE

78], [FERRD 72], [MEADR 78]).

The reason for forecasting

usually determines the specific

techniques which would be used (for

example, see [ABERF 78], [AGRAA 78],

[CANNR 75], [GILMM 78], [HERNE 76],

[MEEHJ 78]). The specific techniques

include benchmarking, regression

analysis, time series analysis,

response surfaces, analytical modelling, operational analysis, and many

more. Of course, the workload

characterization technique must

provide the proper type of data for

the forecasting technique being used.

Unfortunately, some of the forecasting

techniques are difficult to use or are inaccurate. Further clouding the

issue is the question of how to forecast user adaptability to future

changes in the system. Despite these

problems, forecasting at some level of

detail and accuracy must still be

done

.

2.4 Vendor

During the operational phase of

the ADP life-cycle, the ADP manager

must ensure that the vendor lives up

to his obligations under the contract.

Furthermore, the manager may need to

enlist the vendor's support in the

correction of persistent or increasing

problems with system failure. The

collection and review of concrete

reliability data will foster mutual

understanding of problems and cooperation between the ADP manager and the

vendor in finding solutions. When the

ADP manager must rely on supposition

10
and intuition for his arguments,

antagonistic and non-cooperative

relationships may evolve with the

vendor

.

When a system malfunctions, the

problem is usually the result of hardware or software malfunction, but

finding the culprit is not always a simple matter. The problem is difficult even when the installation has a single vendor supplying both the

software and the hardware. In multivendor environments, the problem of

assigning responsibility for the

malfunction is compounded and the need

for accurate information on the

reliablity of each module or device is

necessary for both pinpointing the

specific problem and for ensuring

solutions to recurring problems.

Information on error rates by

device or software module should be

collected, as well as information on downtimes for preventive (scheduled)

maintenances and the frequency and

duration of remedial (unscheduled)

maintenance. Displaying this data as trends and peaks can assist the ADP

manager in detecting declining performance and in identifying inadequate

vendor support areas. Collection of

reliability data may also allow the

installation manager to make comparisons with installations of similar

environment and configuration.

Time lost due to a computer

system malfunction translates into

loss of "availabil ity to the user community. The modularity of many

computers today clouds the issue of

availability since many systems can continue to operate in degraded mode.

For example, the system may operate

more slowly when a device is down and

similar devices pick up the work for

the malfunctioning unit. In other

cases, jobs may not be scheduled which

require devices for which there are no backup capabilities. Evaluation of

availability of such systems can be

extremely complex especially in the

latter case where the system could be

described as being "partially available;" i.e., it is unavailable for the user class requiring the malfunctioning device but available to all other

users

.

2.5 Summary

The sections above discussed many

functions which the ADP manager should

perform in fulfilling the goal of

providing good service to the users at

the least cost. The upper management

and user relationships are extremely

important to the proper management of

the ADP installation. The ADP manager

cannot plan well and provide the

necessary service unless he has strong

relationships in both of these two areas. In an installation where both

the users and the ADP installation are under the same upper management

structure the ADP manager should be

able to exert strong influence on his

user community and vice versa. These

relationships are vital to the planning process if the needed installation capacity is to be available when new work arrives. When the users'

upper management is not the same as that of the ADP installation, the user relationship becoihes even more important in the areas of planning and

workload management.

To support the ADP manager in

performing the tasks for all four

functional areas, phe CPM program must

supply him with the proper information. The data items making up this

information come from many sources —

accounting logs, manual logs, financial statements, hardware and software,

monitors, surveys, etc. A single data

item — for example, CPU time — may

also support several informational

needs and be used in multiple ways.

Appendix A presents a model depicting

the informational needs of the ADP

manager. This model was developed by

expanding the functions described

above into the classes of information

needed to support each function.

11
3.0 Installation Performance

Activities in the Federal

Sector

The functions described in Section 2 above constitute best practices

for managing installation performance.

These functions define a baseline for

comparison with common practices in

the Federal sector. As a result of

previous communication with personnel

at Federal and private sector ADP

installations, attendance at computer

performance conferences, and discussions with computer performance

consultants, certain hypotheses were

formed concerning the current state of

installation performance and measurement activities in the Federal and

private sectors. In order to verify

these hypotheses, site visits were made to nine Federal ADP installations

and the various computer performance

auditing organizations within the

Federal government. At the installations, the discussion included the

question of how the installation

evaluates itself, what information is

used or needed by the ADP manager and,

if more performance activities could

be done at their site, what prevents

them from being attempted. In the

interviews with the auditing agencies,

the topics of discussion were what

types of and how many audit functions

they perform, the state of installation performance from their perspective, and what could be done to

improve the situation. The views on

CPM practices expressed by the Federal

ADP performance auditor and the ADP

installation managers will be discussed in turn below. The perceptions

of the interviewees concerning the reasons why more activity is not

present are included in Section 4.

We do not directly address the

question of how the Federal sector

compares with private industry. The

situation with respect to installation

performance work appears to be similar

in both sectors although the emphasis

may be different. For example, the

private sector may not have the re- quirement for competitive procurement

or upgrade and, hence, may limit its

planning and forecasting to various

machine sizes within a single vendor

line.

3.1 Auditor Perspective

In order to verify the hypotheses

which were made concerning the current

state of installation performance ac- tivities, it was decided that computer

performance audit groups within the

Federal government should be interviewed. Each of these agencies does

performance audits at many Federal ADP

sites each year. Hence, they would be

quite knowledgeable about the status

of CPM activities. Below we briefly

discuss the activities of the agencies which were interviewed and sum- marize the remarks and points which

they made concerning their view of

installation performance activities in

the Federal government.

Six individuals from all four of

the known performance auditing agencies were interviewed. One of the

agencies is a true audit agency and

the others do performance audits in

conjunction with their role as invited

advisors on installation performance

problems. The first agency, the General Accounting Office (GAO) , performs

audits to establish how much capacity

exists at an installation and how much

of that capacity is being used. The

major thrust of their effort is to

encourage installations to plan more

effectively. The other agencies see

as their goal the encouragement of

good computer management practices at

installations. All of these latter

agencies are a part of the Department

of Defense (DOD) , although one serves more than DOD installations.

Each of the agencies had reviewed

dozens of sites but from different

perspectives. The less experienced

agencies concerned themselves primarily with utilization measurements

while the more experienced agency had

moved from an emphasis on utilization

statistics to the need for measurement

of user attitudes as a starting point

in the performance analysis process.

The problems with installation

performance activities which were most

often cited during the interviews

were: (1) failure to assign personnel

and to allocate funds to a CPM program, (2) lack of upper management

support, (3) failure of ADP managers

to consider service levels and their

attainment as one of the primary

12
measures of installation performance,

(4) failure to institute a chargeback

system as a means of influencing user behavior, and (5) a general lack of

knowledge and training in CPM. Each

of these problem areas is discussed in

more detail below.

The agencies indicated that one

of their primary objectives was to

convince the ADP manager that CPM is

necessary and useful and that CPM can supply information to support more of

his functions than is commonly

thought. Few of the sites reviewed by

the agencies had comprehensive CPM

programs as outlined in Section 2 above. The initial entry into a CPM

program was indicated to be the tuning

efforts of the installation. The

sites mentioned as having good CPM

programs tended to use a mix of com- mercial accounting packages, locally

developed monitors, and chargeback

systems

.

One agency indicated that it

rarely found sites where the ADP

manager regularly reported to his

upper management. And at the sites

which did report, the reports themselves were usually not appropriate

but were composites or summaries of resource management reports. They

pointed out that the lack of reporting

to and the observed lack of support by

upper management are probably related.

The agencies noted that a difference in installations having good

CPM programs and in sites having poor

CPM programs could be observed in the

attitudes of the installation managers. At the good sites there appeared to be a genuine concern about meeting service levels. Those installations viewed their mission as one of

providing service which must be.

measured and planned for in order to

meet realistic levels of service. At

other installations, the agencies

perceived several reasons for the

failure of ADP managers to consider

service levels with respect to

installation performance: (1) poor

user relations or lack of concern for

the users by the ADP manager, (2) ne- gative incentives for the ADP manager,

(3) lack of appropriate measuring

tools, and (4) lack of adequate forecasting tools. In most such sites,

poor user relations seem to arise from

the lack of both service goals or standards of performance and welldefined user reporting procedures.

With no performance goals the ADP

manager has no means of evaluating

user complaints and with no reporting

system the users tend to complain

frequently. Eventually the ADP

manager begins to believe that his

installation is performing poorly and

this attitude becomes a negative

incentive for establishing a CPM

program for fear that it will reflect

poorly on his work as a manager. When

early attempts at measuring or forecasting are inaccurate, the fear of

failing again leads the ADP manager to

search for infallible tools. Failing

in his search, the ADP manager moves

further away from implementing a CPM

program that could help him solve

perceived installation performance

problems

.

Chargeback ijs viewed by the agencies as more than simple cost recovery. They view it as another measure

of accountability and a means of con- trolling user behavior. The key

elements were defined to be high visibility among the users -- not just

their managers -- coupled with actual

low-level fiscal responsibility.

Under these circumstances the agencies

believed that the users would be

encouraged to exercise more care in

the development and implementation of

programs and would be more likely to

demand reasonable service and resource

levels. Unfortunately, they reported

finding few sites with good chargeback

systems

.

Lack of knowledge and training

had been observed by the agencies with

respect to both the user and the ADP

installation personnel. On the user side are found users who are not aware

of efficient ways to program nor otherwise utilize the full capabilities of the system. Additionally

they are rarely aware of the problems

of the ADP manager or the demands

being made by other users which could

affect their own use of the installation. On the installation side are found managers who often lack even a rudimentary knowledge of the CPM

process or its benefits. Lacking such

knowledge and facing turnovers and

training expenses, the ADP manager is

hesitant to invest the necessary funds

13
and personnel in CPM training.

3.2 Installation Perspective

Although the NBS staff is in

frequent contact with many Federal ADP

installations, it was decided that a formalized interview procedure should

be undertaken to ascertain the types

of installation performance measurement and reporting procedures which

are used in the Federal government.

Site visits were conducted to interview nine ADP managers, or the CPM

managers if the ADP manager was unavailable, for installations in the

greater Washington metropolitan area. The selection criteria was not formal

but attempted to include sites from

many of the major Federal departments,

sites that supported central agency

functions, sites that are "field

installations," sites that were known

to have at least some computer performance program, and sites about

which nothing was known concerning

their installation performance work.

The sites visited included

installations from the Departments of

Defense, Agriculture, Commerce, and

Health, Education and Welfare. In

addition, information from a previously conducted NBS review [CONTD 79] of

the U.S. Postal Service CPM program

was used. We will not attempt to generalize on the amount of CPM work

which is performed at Federal sites

throughout the country but will

restrict our comments to what we found

at these sites. Of the nine sites,

three could be characterized as having

good, though not complete, programs;

one as having a fair CPM program;

three as having poor efforts; and two

as having their programs in various

stages of development and specification, but little implementation.

During the interviews the discussion centered on the folir areas of

ADP manager responsibility which are given in the model above (Figure 1).

The intent was to find out what the

installation was currently doing in

those areas, what the ADP manager

thought they should be doing in those

areas, and what types of information

he needed to perform his job. It

might be of interest to note that the

ADP managers of sites with good CPM

programs were more aware of the

inadequacies in their programs than

the managers whose programs were some- what less than adequate. In fact,

during the interviews it was evident

that the latter managers were often

unaware of other practices or functions which could be performed or the benefits which could result from

such practices.

The relation to upper management

was found to be the least developed of

all the areas. Very few of the sites

did any reporting to upper management

and only one site presented information which was management oriented.

A few other sites sent various types

of utilization reports to their management. Often these reports were nothing more than "management reports"

produced from commercially-available

software packages (for example,

accounting log analyzers, software

monitors). In fact, these reports

would be more appropriately classified

as condensed resource management

reports. At several sites the ADP

manager expressed surprise at our inquiry and even questioned the need

for such a relationship with upper

manag ement

.

The user relationship area was

the best developed area and all of the

sites had some type of user community

liaison effort. The most popular

mechanisms were the use of regular,

usually monthly, meetings of user groups; user liaison personnel;

newsletters; and trouble desks for

answering questions. Few of the sites

had formal user reporting. The

installations with strong user support

groups, who believed their function to

be user service, seemed to have overcome the problem of user dissatisfaction. In addition, such sites usually

had good user training programs. Only

one site actually negotiated service

levels and charges with their users. Few of the sites had a chargeback

system which caused funds to be

transferred from the user project to

the ADP installation, and those

chargeback systems were oriented

toward cost recovery with little

emphasis on influencing user behavior.

14
All of the sites did some performance tuning in the area of re- source management but many were restricted to one-time, isolated

projects. The three sites with good

CPM programs all kept historical data

for trends and analysis and to help

them in setting control limits. One

of these three, however, was prohibited from openly setting control

limits for fear that the installation's performance would become more

visible. The other six sites appeared

not to use control limits but the ADP

managers were satisfied with a firefighting mode of operation. Workload

statistics were commonly reported to

the users in addition to some information in terms of responsiveness and

availability. Capacity planning and

forecasting were a rare occurrence

and, when attempted at all, were usually restricted to projections

based on the utilization of one component — normally the CPU.

In the area of vendor relations

the sites varied according to the

formality of the arrangement but all

did some tracking of equipment reliability. Several sites used commercially-available packages to track the

performance of their equipment versus

other sites with the same equipment.

Most of those sites believed that such

information gave them more leverage

with the vendor. Most of the sites

used their trouble reports in a fire-fighting mode — if a problem

seemed to persist, the ADP manager

went to the files and looked back

through the paper console logs and

trouble forms for any trends. One

site had its own automated system

which it ran regularly to try to

detect any equipment which might be

degrading and to use this information

in its dealings with its vendors.

the important upper management path is

quite weak while the strength usually

lies in the user community relationship, although this area could also be

enhanced with better reporting

practices. Dealing with the vendors

was the most uniform across sites

while the amount of resource

management performed varied the most

between sites.

As is noted above, it does not

appear that the Federal government

practices are any better or worse than

would be found in the private sector.

However, there would appear to be

ample room for improvement in all of

the four areas described in Section 2 above. In Section 4 we will discuss

possible underlying causes for these

inadequac ies

.

3.3 Summary

None of the findings of the

interviews were surprising, although

they tended to confirm previous

hypotheses concerning the quantity and

quality of installation performance

work being done in the Federal government. There is a general lack of CPM

programs in the Federal sector. At

the sites doing any performance work.

15
4.0 Current Inadequacies in

Installation Performance Practices

How do the current practices in

the Federal government compare to

state-of-the-art possibilities? In

Section 2 above we discussed a number

of functions which could be performed

by an ADP manager in relating to his

upper management, to users, and to

vendors and in managing his installation resources. Section 3 outlined

the practices at a small number of

Federal sites which were visited and

the collaborating views expressed by

personnel in several Federal computer

performance auditing agencies. The

site visits and the interviews confirmed the hypotheses which had been

made concerning current practices. A

gap exists between what is practiced

and what could be done. In particular, the upper management relation

rarely exists; fire-fighting is the

prevalent mode of operation instead of

managed performance; little user reporting is done; and very few sites

negotiate service levels.

This lack of wel 1 -developed and

widely-used CPM programs is the result

of the magnitude of the effort cur- rently required to have a CPM program

at an installation. This conclusion

is supported by the literature, by

conversations with computer performance conference attendees, by observation, and by remarks made during the

site visits. CPM programs can require

substantial personnel time to develop

and operate. The nature of the work

further requires that the personnel

who do the work be experienced and

quite knowledgeable about the system

being measured. In this section we will discuss several possible underlying reasons for the stringent re- quirements of installation performance

activities: (1) lack of measurability, (2) lack of well-defined terms,

(3) lack of "good" values, and (4)

lack of statistical Computer Performance Evaluation (CPE) techniques.

Section 5 w.i 1 1 outline a few initial

steps which could be undertaken to

alleviate some of these problems.

4.1 Lack of Measurabil ity

Before the ADP manager can manage

the performance of his installation,

he must have access to certain information. Unfortunately, most of the

computer systems in use today were not

designed to be measured and do not

supply all of the needed information.

The common measures which are provided

by most vendors were originally for

accounting purposes and not performance management. Additionally, most

vendors supply only a small fraction

of the information needed by the ADP

manager in ensuring good installation

per formance

.

Many of the measures needed in

the performance area are not readily

accessible but are buried in the

system. Hardware and software monitors are necessary to retrieve the

needed information but the monitors

themselves may cause degradation of

system performance. All software

monitors consume resources to a greater or lesser degree depending

upon the architecture of the system

and its operating system. In addition, the use of hardware monitors

requires specialized expertise.

Even when installations go to the

expense and time of installing monitors, measurement is not ensured.

Some measurements are prohibitively

expensive with respect to the substantial computer resources consumed

during the measurement, data collection, and analysis process. Other

measurements are not possible without

substantial system alterations and may

still consume many computer resources

when put in place.

From observation and from the

interviews, it would appear that the

lack of measurabil ity contributes

greatly to the lack of performance

programs at Federal ADP installations.

Many of the measures are not directly

provided by the vendors and collecting

and analyzing measurement data require

the time and effort of a highly

trained staff. Many installations

cannot afford the drain on their

resources which would be required for an installation performance program.

Designing system hardware and software

with performance measurement in mind

and liroviding eas.y access to the data

16
would reduce the effort required to do

performance management at individual

install at ions

.

4.2 Lack of Well-Defined Terms

Stevens [STEVB75b-pl] summed up

the terminology problem in the performance area when he stated:

As an industry we suffer from a complete absence of both standard

units of measure and standard

metrics. With standard units we could make direct comparisons

among equipment types available

to service the same workload, and

identify the precise configuration having the absolute minimum amount of capacity necessary

to do the job. ... Failing

that, we could utilize standard

metrics to at least describe to

each other the selection and

evaluation processes we use to assure ourselves that we are producing the best possible

result.

The lack of commonality of welldefined terms hinders communication

between practitioners. The computer

performance field is one of the few

"professional fields" which lacks

standard nomenclature and, as a result, has probably not progressed as rapidly as it might have. It is not

that the various systems collect

vastly different information which

prevents the sharing of ideas and

techniques. The problem lies in the

fact that although different vendor

systems collect essentially the same basic set of data, they mean slightly

different things and are recorded in

different formats. Hence, there is

little completely common ground.

Further compounding the problem,

is the lack of common names for the same measures. Some measures from

system A may be identical to those on system B, although different names are

used. A performance problem on system

A may be identical to a performance

problem on system B which has been

solved on system A. But, the analysts

on system B do not recognize that the

problems are the same because the

terms are different. Hence, they can

not avail themselves of an approach to

the solution or the solution itself.

The two problems — measuring

different things and measuring the same thing but naming it differently — combine to force much of the development work in CPM to be repeated

at each installation. The lack of

well-defined and common terms

seriously limits the ability of small

installations with limited resources

and installations new to CPM to use an established base of techniques,

problem approaches, and solutions

developed by other sites and verified

by repeated use. As a consequence,

many sites do not use CPM to as great

a degree as they could or they do not

start up any program at all. The

establishment of common and welldefined measures used in installation

performance work would also allow

greater communication among installations working on various performance

problems. Common terminology would

allow work to begin toward establishing standards of "good" performance

and ranges of normative values for

operating levels.

4.3 Lack of "Good" Values

One of the difficulties of an ADP

manager is knowing what constitutes

"good" performance. When a system is

meeting its service goals and no problems are visible in its operation,

how can the ADP manager know if his

system is well-tuned? Before this

question is summarily dismissed as not

being germane to installation performance, the following should be con- sidered. An installation may be operating close to its performance levels

but still have capacity to process

additional work. When new work is

presented to an installation, the ADP

manager needs to know whether sufficient excess capacity exists in his

system to accommodate the new work

through a performance enhancement

effort instead of an upgrade or procurement. Such delays of upgrades or procurements could save the installation money. The arguments that the

upgrade would be cheaper than the

performance enhancement effort (since

hardware is cheaper than personnel)

indeed have merit in an environment

17
where procurements or upgrades are easily justified and do not take long

periods of time. However, such an environment is rarely found in the

Federal sector.

Hence, the ability to compare

installations (i.e., to determine if a particular system is well-tuned) could

be beneficial in the Federal ADP

environment. The lack of industrywide standards of "good" performance

even prohibit the comparison of performance levels among installations

with like configurations and workload.

Such standards could provide a starting point for setting goals for a particular installation and provide a basis for the ADP manager to compare

the performance of his installation

with respect to a large number of

other "similar" installations. Such

normative values could assist the ADP

manager in assessing how well his

system is operating and how much

capacity could be obtained for new

work coming to his installation.

Furthermore, the installations which

are just starting performance programs

would have a baseline for setting

initial performance goals.

4.4 Lack of Statistical CPE

Techni ques

It would be of benefit to the ADP

manager if he had a means of determining what effects various components

of the system and workload have on the

service that his installation is

providing. How does one bridge the

gap between measures of effectiveness

in meeting service goals and efficiency measures in resource management? How does the performance of

certain hardware components affect

service levels? The ability to cor- relate the effects of computer configuration, operating system parameters,

etc. with the attainment of user service requirements has not been

fully studied because of the complexity of the interrelationships and also

because of the general shortage of

data which would help in the evaluation. Instead of bridging the gap,

the ADP manager often picks one goal

by deciding what is more important to

him — for the computer to run efficiently or for the computer to

serve its users by meeting their

goals. Since efficiency is often

easier to attain than effectiveness,

the emphasis has been historically on efficiency. However, efficiency of

the system may not produce the service

desired by the users. In recent

years, more emphasis has been placed

on user service level definition and

attainment. But, little work has been

done in bridging the gap between the

two alternatives and little work has

been done on statistical techniques

for either area. There are many reasons for

specific CPE studies but they all have

a common goals to improve the performance of the installation. This

goal may be accomplished by making the

system more responsive to the user requirements, or by reducing the cost

while maintaining the same service

levels, or both. The problems studied

may be simple with simple solutions or may be extremely complex involving

workload characteristics and their

effects on a system which has many

interdependenc ies and interrelationships. Simple problems may be

attacked by simple graphical methods

while others may require more rigorous

and precise techniques. The latter

studies, however, are often done without the rigor and preciseness which

comes with statistically based

techniques. Instead the "eyeball"

approach is used v/here practitioners

subjectively decide whether differences in a measure are significant

when a change was made or whether a change actually had the desired

effect. Furthermore, with a non- statistical approach nothing can be

said about how much confidence may be

placed in the conclusions.

There are two basic approaches to

analyzing CPE data: graphical and

mathematical. Graphical techniques

are quite straightforward and provide

a visual means of detecting patterns

and trends. Graphical techniques

include scatter plots for detecting

possible relationships, plots of time

versus service and resource measurement on the same graph for correlations or maximums, and plots for trend

analysis. Graphs will often indicate

where experimental studies should be

performed by visually pointing to

possible causal relationships between

18
system and service variables over

time. There are many mathematical

techniques which are applied to CPE:

correlation analysis, cluster analysis, multiple regression, analytical

models, operational analysis, curve

fitting, time series analysis, and

response surfaces. Currently there

are several problems with using any of

these techniques.

The major problem is that these

techniques are usually based on underlying assumptions and the exact nature — statistical or otherwise -- of

computer systems and workloads is not

well understood. For example, regression analysis assumes a linear relationship between the dependent variable and the independent variables,

equal variances, and an assumption

that the independent variables are additive. These properties, however,

have not been totally verified in the

general computer system environment.

A second problem is that future performance is often of interest. Modelling techniques are the most often

used methods for predicting performance in growth situations where the

predicted values may exceed the

current operating levels. But

analytical modelling has very

stringent assumptions and is still

fairly limited in modelling the true

complexity of modern computer systems.

The most stringent assumptions for

analytical modelling are removed by

operational analysis techniques but

many of the queueing parameters needed

are currently unavailable from most

operating systems. The techniques

•discussed above are not exhaustive but

are described to illustrate the

difficulties and problems associated

with the application of statistical

analysis to CPE.

With all of the problems associated with good CPE studies, it is no wonder that so little work is being

done at installations. The more

complex techniques are statistically

based and often not understood or known by the typical analyst. Little

study has been done to assess the

nature of the computer environment

V i s-a-vi

s

the statistical techniques

being used today. Furthermore,

"cookbooks" of procedures have not

been developed for the non-statistician analyst.

19
5.0 Proposed Remedial Work

The previous section discussed

four areas which were seen as inhibiting more wide-spread use of installation performance programs. The

inadequacies increase the effort

needed to develop and operate computer

performance management programs at

individual ADP installations. If

these inadequacies could be somewhat

relieved, less installation personnel

and fiscal resources would be required

for a CPM program. Thus the use of

performance programs in the Federal

sector might increase and the benefits

from such programs could be realized.

In this section, we will discuss some

remedial work which could be started

to alleviate at least partially the

problems mentioned above: lack of

measurabil ity , lack of well-defined

terms, lack of "good" values, and lack

of statistical CPE techniques. The

proposed remedies involve the development of standard performance measures,

development of a data base of normative standards for performance, and

development of statistical techniques

to be used in the CPE area.

5.1 Standard Performance Measures

Work should be undertaken to develop a list of standard performance

measures and definitions for these

measures. The list of metrics should

satisfy a minimal set of informational

requirements of the ADP manager in

assessing installation performance and

should be vendor independent.

Since the establishment of a set

of performance metrics does not ensure

their accessibility, standard reporting formats should also be developed.

Depending upon the type and use of the

data item, the format might take the

form of a record on a file or, maybe,

a special hardware register which

could be monitored. Most measures

which are needed by the ADP manager

could probably be written to a special

file whose format would be standard

across vendor lines. However, there

are some types of data which are needed but which would cause an excessive drain on the system if collected

in that way. For these measures, an alternate method of capturing and

recording would need to be specified

which would also be common across vendor lines.

The metrics should be grouped

according to purpose and use. There

are two reasons for considering

multiple groups. First, some measures

do not warrant continuous monitoring

and collection but are needed for

specialized studies. The overhead in

collecting some measurements is higher

than for others and the installation

may not be able to tolerate the drain

on resources. Second, depending on the sophistication and needs of a particular installation, not all

measures would be of interest to all

installations. Thus, the use of

multiple groups would allow the

installation to tailor the collection

of performance data to its own needs.

5.2 Data Base for Normative

Performance

Work should be instituted to

determine the feasibility of developing a Government-wide data base which

would contain measures of installation

performance classified by type of

installation (configuration) and type

of workload. The measures would have

to be quantitative if automatic

comparison of performance is to be

made across installation types and

workloads. The problems associated

with quantitatively describing workloads, installation configurations,

and organizational objectives have not

yet been solved. Without such quantitative descriptions, the utility of

the data base would be somewhat

limited. Some types of measures like

hardware reliability, for example, are readily quantifiable and normative

values would be useful. By examining

averages and ranges using the performance data in the data base, optimum

values or ranges could be established

which would provide the basis for

setting individual ADP installation

performance goals. These values would

be based on a large number of installations and would therefore provide an additional means for the ADP manager

to assess his installation's performance .

20
Of course, if such a system is to

be workable it must help the ADP

installations and require little in

return. It must have a well-designed

collection mechanism which would not

place a large burden on the individual

installations. The system must also ensure the proper feedback of information to the ADP installations. Last,

if the data is to be useful, it must

be accurate. It is anticipated that

installations which perceive such a data base as a threat will not provide

accurate data and the establishment of

normative data values will fail.

Hence, the system must be designed to

eliminate, or at least reduce, the

possibility that the information

supplied by the individual installations might be used against them.

5.3 Development of Statistical

CPE Techniques

Work should be started to develop

and validate techniques for bridging

the gap between measures of effectiveness and resource consumption

measures. The environment of the

computer installation and its interrelationships need to he studied to

determine which techniques used in

other control environments (such as production lines) are applicable to

the computer performance area. When

such ' techn i ques have been validated

for use in the computer environment,

guides need to be prepared for the

non-statistician analyst in terms that

he can relate to and understand.

Work should be continued to

develop techniques to model computer

systems, workloads, and service

levels. Such work is needed to help

the ADP manager in forecasting his

future work and the ability of his

installation to handle increased or different workloads. Techniques and

tools are needed which are relatively

easy to use and which use data which

is available to the installation.

because they lack the expertise in

their staff or the staff is not large

enough to spend the required time and

because there can be rather substantial costs involved. The benefits

from the three proposals above come

about largely because of their potential for reducing the effort to

implement installation performance

programs. The standardization of

performance measures and their

reporting format would allow greater

communication between installation

practitioners and could result in

better solutions to common problems

being developed in shorter time

periods. The approaches to solutions

would be more widely known and

available. Further, the standardization of the reporting format would

allow greater portability of packages

and programs which are currently

rather expensive to develop both in

terms of time and money. The benefits

of performance programs vary from

installation to installation, but no benefits can be realized if no such

activity is being done. By reducing

the effort to start and participate in

performance programs at installations,

it is hoped that more ADP managers

will be willing to institute such

programs at their sites.

5.4 Summmary

We believe that many Federal ADP

installations perform little or no work in installation performance

21
6.0 Bibliography

[ABERF 78] Abernathy, Frances S.

Communicating Capacity Plans to

Management. Wright, Linda S.;Buzen,

Jeffery P., eds. Proceedings of CMC

IX International Conference on Management and Evaluation of Computer

Performance; 1978 December 5-8: San

Francisco, California, cl978.

Bethesda, Maryland; Computer

Measurement Group, Inc.: 17-34.

[AGRAA 78] Agrawala, A.K. and Mohr,

J.M. Predicting the Workload of a Computer System. AFIPS Conference

Proceedings; 1978 National Computer

Conference (NCC) ; 1978 June 5-8:

Anaheim, California. Montvale, New

Jersey: AFIPS Press; cl978: [v47]

:

465-471

,

[ANPEH 75] Anderson, H.A.,Jr., Reiser,

M. and Galati, G.L. Tuning a Virtual

Storage System. IBM Systems Journal.

1975; 14(3): 246-263.

[AXELC 79] Axelrod, C. Warren-. How

Effective Is Your Computer?.

Infosystems. 1979 February; 26(2):

50,52-53.

[BATTV 72] Battaglia, Vincent J.

Cutting Computer Costs. Financial

Executive. 1972 August: no data

(n.d .) : 26-30

.

[BAZEM 77] Bazewicz, Mieczyslaw and

Petersiel, Adam. Use of Modelling in

Performance Evaluation of Computer

Systems — A Case of Installations in

the Technical University of Wroclaw.

Performance Evaluation Review. 1977

Fall; 6(4): 22-26.

[BELLT 72] Bell, T.E., Boehm, B.W.

and Watson, R.A. Framework and

Initial Phases for Computer Performance Improvement. AFIPS Conference

Proceedings (FJCC); 1972 Fall .Joint

Computer Conference ; 1972 December

5-7. Montvale, New Jersey: AFIPS

Press; C1972: [v41 part 2]:

1141-1154.

[BELLT 73] Bell, T.E., Boehm, B.W.

and Watson, R.A. How to Get Started

on Performance Improvement. Computer

Decisions. 1973 March; 5(3): 30-34.

[BELLT 75a] Bell, Thomas E. Set

Limits to Control Performance

Measurement. Computer Decisions.

1975 September; 7(9): 35-38.

(BELLT 75b] Bell, Thomas E. Managing

Computer Performance with Control

Limits. Proceedings of the Ninth,

Tenth, and Eleventh Meetings of the

Computer Performance Evaluation Users

Group (CPEUG); 1974 May 21-23;

Atlantic, Georgia; 1974 October

23-2 5; Columbus, Ohio; 197 5 September 23-26; Oklahoma City,

Oklahoma. Springfield, Virginia:

NTIS; PB 252-174: 169-172.

[BELLT 76] Bell, Thomas E. Computer

Performance Management Through Control

Limits. Transactions of the CMG.

1976 March. 11: 3-2 to 3-5.

[BEMER 73] Bemer, R.W. Standards in

Performance Evaluation & Measurement.

Computer Performance Evaluation:

Report of the 1973 NBS/ACM Workshop.

1973 March 27-30: San Diego,

California. NBS SP 406 (Issued August

1975) : 141-149.

[BENTT 76] Bentley, Trevor John.

Defining Management's Information

Needs. AFIPS Conference Proceedings;

1976 National Computer Conference

(NCC); 1976 June 7-10; New York

City, New York. Montvale, New Jersey:

AFIPS Press; C1976: [v45] : 869-876.

[BROTD 75] Brotherton, Dale E. and

Harder, Donald C. Computer

Performance Measurement. Data

Management. 1975 January; 13(1):

12-15.

[BROWS 77] Brown, Shellman H.,Jr. EDP

Capacity Management. Proceedings of

the 1977 SIGMETRICS/ CMG VIII;

Conference on Computer Performance:

Modelling, Measurement, and Management; 1977 November 29 to December 2:

Washington, D.C. . New York City, New

York: ACM Order Dept. Camp Springs,

Maryland; Computer Measurement Group,

Inc.: 301-305.

22
[BUZEJ 78] Buzen, J. P., Goldberg,

R.P., Langer , A.M., Lentz, E.

,

Schwenk, H.S., Sheetz, D.A. and Shum,

A.. BEST/1 - Design of a Tool for

Computer System Capacity Planning.

AFIPS Conference Proceedings; 1978

NCC; 1978 June 5-8; Anaheim,

California. Montvale, New Jersey:

AFIPS Press; cl978: [v47] : 447-455.

[CICA 70] The Canadian Institute of

Chartered Accountants. Computer

Control. Guidelines. cl970; The

Canadian Institute of Chartered

Accountants; Toronto, Canada: 135p.

[CANNR 75] Canning, Richard G.

(editor) . Do We Have the Right

Resources?. EDP Analyzer. 1975 July;

13(7): I2p.

[CHANJ 76] Chandler, John S. , and

DeLutis, Dr. Thomas G. A Methodology

for the Performance Evaluation of

Information Systems under Multiple

Criteria. CMG VII Proceedings of the

Seventh International Conference of

the Computer Measurement Group, Inc.;

1976 November 16-19: Atlanta,

Georgia. Camp Springs, Maryland;

Computer Measurement Group, Inc.:

221-229.

[CHANJ 77] Chandler, John S. , and

DeLutis, Thomas G. A Methodology for

Multi-Criteria Information System

Design. AFIPS Conference Proceedings;

1977 NCC; 1977 June 13-16: Dallas,

Texas. Montvale, New Jersey: AFIPS

Press; cl977: [v46] : 895-905.

[CHASD 73] Chastain, Dennis R. Guidelines for Planning and Organizing a Performance Improvement Project.

Proceedings of ACM 73: Annual Conference of the Association for Computing

Machinery; 1973 August; Atlanta,

Georgia. New York City, New York:

ACM Order Dept.; cl973: 332-335.

[CONTD 79] Conti, Dennis M. Review of

the U.S. Postal Service Computer

Performance Management (CPM) Program:

Task II Final Report. Washington,

D.C.: MIS-USPS; 1979 July 30.

Washington ,D.C. : Center for

Programming Science and Technology,

ICST, NBS. 35p.

[DEARJ 73] Dearden, John, and Nolan,

Richard L. How to Control the

Computer Resource. Harvard Business

Review. 1973 November/December;

5(6): 68-78.

[DEVEE 78] Deventer, E. N. van, and

Roode, J. D. Computer Resource

Performance Management - A Total Data

Centre Approach. Wright, Linda S. , and Buzen, Jeffrey P.,eds. Proceedings of the CMG IX International

Conference on Management and Evaluation of Computer Performance; 1978

December 4-8; San Francisco,

California. cl978. Bethesda,

Maryland; Computer Measurement Group,

Inc.: 67-79.

[DIEBG 74] Diebold Group, Inc.

Computer Performance Appraisal - A Top

Priority for Top Management. Automatic Data Processing Newsletter;

1974 December 9; XVIII (25) : 4p.

[DIESJ 79] Diesem, John L. Data

Centers Need Fair Management.

Computerworld . 1979 March 12;

XIII (11) : 59 & 62.

[DODSP 74] Dodson, Philip 0. Control

Data Corporation's Cyber 70:

Procedures for Performance Evaluation.

1974 May. Springfield, Virginia:

NTIS; AD 785129: 104p.

[DRUMM 69] Drummond, M.E., Jr. A

Perspective on System Performance

Evaluation. IBM Systems Journal.

1969; 8(4): 252-263.

[DUNLR 78] Dunlavey, Richard F. Workload Management. EDP Performance

Review. 1978 May; 6(5): 1-4.

[FERRD 72] Ferrari, Domenico. Workload Characterization and Selection in

Computer Performance Measurements.

Computer. 1972 July/August; 5(4):

18-24

.

[FIPS 49] Federal Information

Processing Standard Publication 49.

Guideline on Computer Performance

Management: An Introduction. 1977

May 1; FIPS PUB 49: 14p. Available

from: National Technical Information

.Service, Springfield, Virginia;

NBS-FIPS-PUB-49.

23
[FIPS 57] Federal Information

Processing Standards 57. Guidelines

for the Measurement of Interactive

Computer Service Response Time and

Turnaround Time. 1978 August 1; FIPS

PUB 57: 26p. Available from:

National Technical Information

Service, Springfield, Virginia;

NBS-FIPS-PUB-57.

[FIPS xxl Federal Information

Processing Standards Publication xx

.

Guidelines for Evaluation of Remote

Batch Computer Service. (Publication

is in process as of September 4 (. 1979) ? (n.d.)

.

[FORER 781 Forest, Robert B. Perspective: What are we measuring?. Infosystems. 1978 May; 25(5): 92 & 94.

[FRIEH 75] Friedman, H.P. and

Waldbaum, G. Evaluating System

Changes Under Uncontrolled Workloads:

A Case Study. IBM Systems Journal.

1975; 14(4): 340-352.

[FRIEL 72] Fried, Louis. Making Data

Processing Pay Its Way. Computer

Decisions. 1972 March; 4(3): 24-27.

[GELLH 751 Gellman, Harvey S. Evaluation of the Effectiveness of EDP

Facilities. Cost and Management.

1975 November/December; 49: 16-19.

[GILMM 78] Gilmore, Martha R, Capacity Management: A Definition and

Implementation Guide. Proceedings of

CMG IX International Conference on Management and Evaluation of Computer

Performance; 1978 December 4-8; San

Francisco, California. cl978.

Bethesda, Maryland; Computer

Measurement Group, Inc.: 1-11.

[GROCJ 72] Grochow, J.M. Utility

Functions for Time-Sharing System

Performance Evaluation. Computer.

1972 September/October; 5(5): 16-19.

[GSA 7"7] General Services Administration: Automated Data and Telecommunications Service. Management

Guidance for Developing and Installing

an ADP Performance Management Program.

Washington, D.C.: GSA; 1977 July.

252p. CS 77-5.

[GUDEE 75] Gudes, Ehud, and Sechler,

Charles. Measures for Workload and

their Relation to Performance

Evaluation. Proceedings of the Ninth,

Tenth, and Eleventh Meetings of the

Computer Performance Evaluation Users

Group; 1974 May 21-23; Atlanta,

Georgia; 1974 October 23-25;

Columbus, Ohio; 1975 September 23-26;

Oklahoma City, Oklahoma. Springfield,

Virginia; NTIS; PB 252-174:

115-121.

(GUIDI 76] Guide International

Corporation. Data Center Management

1975-1985. Transactions of the CMG.

1976 June; 12: 3-2 to 3-15.

[HALLS 75] Hallam, Stephen F. An

Empirical Investigation of the Objectives and Constraints of Electronic

Data Processing Departments. Academy

of Management Journal, 1975 March;

18 (1): 55-62.

[HANSJ 78] Hansen, John R. Getting A

Handle On Computer Performance.

Infosystems. 1978 June; 25(6): 82 & 87-88

.

[HERNE 76] Herndon, Edwin S. Computer

Systems Planning - Tools and

Techniques. ACM 76 Proceedings:

Annual Conference of the Association

for Computing Machinery; 1976 October

20-22; Houston, Texas. New York

City, New York: ACM Order Dept.;

C1976: 203-207.

[HOPKR 75] Hopkins, Robert H. It Pays

to Evaluate Your Systems Department.

Canadian Chartered Accountant. 1975

February; 106: 22-25.

[HOWAP 74] Howard, Phillip C.

(editor). What Management Should Know

About Performance. EDP Performance

Review. 1974 January; 2(1): 1-6.

[HOWAP 75] Howard, Phillip. Performance Reporting for Management - Theory

vs. Practice. 19'^5 BBUG/ CMG VI

Proceedings; 1975 October 7-10; San

Francisco, California. Long Beach,

California: Ian Roome (Treasurer,

CMG) : Computer Measurement Group,

Inc.: 253-267.

24
[HOWAP 76] Howard, Phillip C.

Performance Reporting and Management

Responsibilities. Supplement

Proceedings of Computer Performance

Evaluation Users Group: 12th Meeting;

1976 November 8-12; San Diego,

California. Springfield, Virginia:

NTIS; PB 290-700: 65-60.

[HOWAP 77a] Howard, Phillip C.

(editor). Performance Management

Information Systems: State of the

Art. EDP Performance Review. 1977

July: 5(7): 1-7.

[HOWAP 77b] Howard, Phillip C;

Stevens, Barry A., eds. The EDP

Performance Management Handbook:

Volume 1: Audit and Control and

Volume 2: Tools and Techniques.

cl977 (reference handbook is updated

periodically.). Phoenix, Arizona:

Applied Computer Research.

[HUNTH 70] Hunt, Holman. Is Your

Computer a .Success? Part 1. Data

Systems. 1970 February; n.d.:

44-46

.

[HURTC 72] Hurtado, Corydon D. A

System to Measure EDP. Journal of

Systems Management. 1972 January;

23(1): 32-35.

[IGERR 76] Igersheim, Roy H. Managerial Response to a Information

System. AFIPS Conference Proceedings:

1976 NCC; 1976 June 7-10; New York

City, New York. Montvale, New Jersey:

AFIPS Press; cl976: [v45] : 877-882.

[JOSLE 74] Joslin, Dr. Edward 0.

Nine Alternatives to a New Computer.

Journal of Systems Management. 1974

November; n.d.: 38-41.

[LEAVD 78] Leavitt, Don. Performance

Measures Not Good Enough. Computerworld. 1978 May 29; XII (22): 27.

[LYBRR 73] Lybrand , Ross Bros., and

Montgomery. Recognition of EDP

Operational Problems. Management

Advisor. 1973 January/February;

10(1): 55-57.

[McFAF 73] McFarlan, F. Warren.

Management Audit of the EDP

Department. Harvard Business Review.

1973 May/June; 51(3): 131-142.

[McKEL 79] McKell, Lynn J.; Hansen,

James V.; Heitger, Lester E. Charging for Computing Resources.

Computing Surveys. 1979 June; 11(2):

105-120.

[McLEE 73] McLean, Ephraim R. How

Effective is Your Data Processing?.

California Management Review. 1973

Fall; XVI (1): 95-100.

[MAMRS 79] Mamrak, Sandra A.; Amer,

Paul D. Computer Science & Technology: A Methodology for the

Selection of Interactive Computer

Service. Washington, D.C.: National

Bureau of Standards (NBS) ; Institute

for Computer Sciences & Technology

(ICST) . 1979 January; NBS Special

Publication 500-44. 77p.

[MATHJ 75] Mathews, J.R. A Survey of

EDP Performance Measurement for Local

Government. 1975 September.

Springfield, Virginia: NTIS; PB

254-351: 20p.

[MATHJ 78] Mathews, Joseph R. A

Survey of EDP Performance Measures.

Government Data Systems. 1978 July/

August; 7(4): 29-32.

[MEADR 78] Mead, Robert L. , and

Schwetman, Herbert D. Job Scripts - A

Workload Description Based on System

Event Data. AFIPS Conference

Proceedings: 1978 NCC; 1978 June

5-8; Anaheim, California. Montvale,

New Jersey: AFIPS Press; cl978:

[v47] : 457-464.

[MEEHJ 78] Meehan, James C. Computer

Capacity Planning: Evolution and

Implementation. Proceedings of CMC IX

International Conference on Management

and Evaluation of Computer Performance; 1978 December 5-8; San

Francisco, California. cl978.

Bethesda, Maryland; Computer

Measurement Group, Inc.: 17-34.

[MILLV 71] Miller, Victor E. Responsibility Time Reporting for Management

Control Over EDP. Arthur Anderson

Chronicle. 1971 June; n.d.: 8-11.

[NIELN 70] Nielsen, Norman R. The

Allocation of Computer Resources - Is

Pricing the Answer?. Communications

of the ACM. 1970 August; 13(8):

467-474.

25
[PALMC 76] Palmer, Carl R. A Brief

Review of the GAO Task Group's

Recommendations on Management Guidelines for Resource Utilization

Measurement of Data Processing in the

Federal Government. Transactions of

the CMG. 1976 June; 12: 3-16 to

3-22.

[PEARW 77] Pearson, Sammy W. Measurement of Computer User Satisfaction.

(PhD dissertation) Arizona State

University. 1977 August.

Springfield, Virginia; NTIS:

AD/A-046549: 315p.

[PEEPD 78] Peeples, Donald E. Measure

for Productivity. Datamation. 1978

May; 24(5): 222-230.

[RAUPA 72] Rau, Paul. Evaluating the

DP Function. Datamation. 1972

September; 18(9): 72-73.

[RICKJ 75] Rickerby, J. G.; Mellor,

G.; Co an, D. R. A. ,( eds) . NCC

Standards in Operations: Guideline

for DP Management. United Kingdom of

Great Britain and Northern Ireland:

The National Computing Centre (NCC)

Limited; 1975 November. (Distributed

in USA and Canada by Hayden Book Co.

Inc., 5PI Ess.ex Street, Rochelle Park,

New Jersey 07662). ISBN 0850121523.

[SCHUD 74] Schumacher, David. A

Graphical Computer Performance Report

for Management. Proceedings of BBUG

V; 1974 September 30 - October 3;

Montreal Quebec, Canada. Camp

Springs, Maryland: Computer

Measurement Group, Inc.; cl974:

200-216.

[SHEMJ 72] Shemer, Jack E. , and

Roberston, John B. Instrumentation in

Time-Shared Systems. Computer. 1972

July/August; 5(4): 39-47.

[SHIVR 72] Shively, Richard R. Performance Evaluation. Computer.

1972 September/October; 5(5): 12-15.

[STEVB 75a] Stevens, Barry A. Data

Center Optimization. 1975 BBUG/CMG VI

Proceedings; 1975 October 7-10; San

Francisco, California. Long Beach,

California: Ian Roome (Treasurer,

CMG): Computer Measurement Group,

Inc.: 268-300.

[STEVB 75b] Stevens, Barry A. Data

Center Optimization and Performance

Management. EDP Performance Review.

1975 November; 3(11): 1-4.

[STEVB 76al Stevens, Barry A. CPE

Measurement Activities for Data

Processing. EDP Performance Review.

1976 May; 4(5): 1-7.

[STEVB 76b] Stevens, Barry A. Elements of a Performance Management

Systems. EDP Performance Review.

1976 November; 4(11): 1-7.

[STEVB 78a] Stevens, Barry. Service

Levels and the Service Level Matrix.

Transactions of the CMG. 1978 March;

19: 3-1 to 3-18.

[STEVB 78b] Stevens, Barry A. Audit

and Control of Performance in Data

Processing. EDP Performance Review.

1978 January; 6(1): 1-8.

[STEVD 78] Stevens, David F. How To

Improve Your Performance Through

Obfuscatory Measurement. AFIPS

Conference Proceedings: 1978 NCC;

1978 June 5-8; Anaheim, California.

Montvale, New Jersey: AFIPS Press;

C1978: [v47] : 425-431

.

[STRAJ 72] Strauss, J.C. A Benchmark

Study. AFIPS Conference Proceedings

1972 Fall Joint Computer Conference

(FJCC); 1972 December 5-7; Anaheim,

California. Montvale, New Jersey:

AFIPS Press; cl972: [v41 part II]:

1225-1233.

[STRED 75] Streeter, D.N. Productivity of Computer-Dependent Workers.

IBM Systems Journal. 1975; 14(3):

292-305.

[SVOBL 76] Svobodova, Liba. Computer

System Measurabil ity. Computer. 1976

June; 9(6): 9-17.

[WOSFM 73] Wosfey, Marvin M. Management of ADP Systems. Philadelphia,

Pennsylvania; Auerbach Publishers,

Inc.; C1973: 197-218.

26
Appendix A - Informational Model

Based on the functions discussed in Section 2, the following

informational model is presented. There is no attempt to make this

model complete, or uniformly detailed, but the intent is to

illustrate the magnitude and variety of information used in

supporting the installation performance program. Usually the

detail levels are not given since the exact detail information

would depend on the objective of the function; for example, the

type of optimization in workload management - program, file

handling, etc. - determines the exact measures to be made.

I. Upper Management

A. Planning

1. Configuration alternatives

2. Pricing strategy

* workload forecast

* budget projection

* new service levels

3. Organization objectives

* type of processing

* priority

* schedule

B, Operations

1. Fiscal Accounting

* revenue

* project

* resources

* budget

* expenditures

* salaries

* supplies

* equipment

* packages

* maintenance

* space/utilities

* pricing strategy

* recovery

* behavior modification

2. Resources

* total system capacity

* personnel

3. Workload

* organizational unit

* trends

4. Service levels

* timeliness

* reliability

* availability

* cost when level not met

27

1
User Community

A. User Satisfaction

1. Survey

2. Suggestion forms

B. Setting Service. Levels and Pricing Strategy

1. Workload forecast (resource units)

2. Revenue projection

C. Workload Management

1. Audit (delete or reschedule)

* project(s) supported

* dependencies

* age of application

* age of version

* run frequency

* time of day

* resources requested

* resources consumed

2. Optimize

* execution characteristics

3. Quality assurance

D. User Proficiency

1. Trouble desk

* rel iabil ity

* availability (system)

* availability (personnel)

* types of problems

* utilities needed

2. User groups

* service level tracking

* workload statistics

3. Technical support

* modules being developed

* use

* schedule

* revenue

* staff time expended

* user

* utility

* system

4. Communication

* service level tracking

* cost (chargeback) information

* new packages, upgrades, etc.

* system schedule changes

* training notices

* efficiencies suggestions

5. Training

* workload statistics

* reliability (user errors)

* rates

* type

* jobs affected

* new procedures, packages

28
E. Planning

1. Service requirements

2. Conversion impact areas

* allocatable devices

* magnetic tapes • * disk packs

* data files

* data base management systems

* internal formatted

* programs

* non-standard features

* non-standard languages

3. Workload forecast

* functional

* resource

III, Resource Management

A. Operational and Procedural

1. Dispatching functions

* staff time

* volumes

* cards handled

* pages handled

* poor quality output

2. Schedules

* availability of staff and system

* schedule versus actual

3. Errors (system and personnel)

* number by type

* time to fix or restore

* fatal to system or subsystem

* which jobs affected

* direct

* indirect

* cost to projects

* wasted processing time

* wasted staff time

* cost to installation

* effect of user practices

* file save rate

* backup/restore activity

4. Backup/restore

* schedule

* frequency

* resources required

5. Training (operators)

* errors

29
Computer System

1. Control limits

* utilization

* total system

* device

* time periods

* subsystems

* availability

* time

* down

* scheduled

* unscheduled

* diagnostic

* waiting for repair

* repair

* up

* completely available

* subsystem down

* unstaffed

* total system

* hardware groups

* software modules

* responsiveness

* turnaround time

* time-sharing response

* number of jobs ontime/late

* throughput

* file

* space

* distribution

* allocatable device

* number busy

* time busy

* contention and overlaps

2. Tuning

* utilization

* by component

* by device

* productivity

* subsystem activity

* communication activity

* timesharing

* remote batch

* transaction

* line speeds

* profiles

* port activity

* service

* system queues

* user requested

* input

* output

* allocatable device

* processor

* misc. internal

* file

* organization

* placement

30
(2. Tuning continued)

* allocatable device

* service - utilization relationship

* system interrelations

* contention

* overlaps

* concurrency

3. Forecasting

* workload characterization

* function

* batch (over-the-counter)

* arrival distribution

* priority

* dependency

* remote batch (job entry)

* origination

* priority

* dependency

* timesharing

* number sessions

* functions

* order of functions

* volume of characters

* transaction

* number and type

* volume of characters

* language

* file storage

* permanent or scratch

* resources required

* workload - service relationship

* workload - utilization relationship

* service - utilization relationship

* system interrelations

* queue

* contention

* overlaps

31
Vendor

A. Hardware Reliability

1. Error

* jobs affected

* which

* cost to project

* cost to installation

* rates

* intervals

2. Retry

* rates

* intervals

3. Maintenance

* schedule

* unscheduled

* response

* delays

* error profile

B. Software Reliability

1. Error

* jobs affected

* which

* cost to project

* cost to installation

* rates

* intervals

* time to fix

* release/version

* between errors

2. Retry

* rates

* intervals

3. Maintenance

* time between releases

* event profile

* regression

32
MBS-IMA (REV. 9-7»)

U.S. DEPT. OF COMM. I 1. MlBLICATION OR REPORT NO. BIBLIOGRAPHIC DATA

SHEET SP 500-53

4. TITLE AND SUBTITLE COMPUTER SCIENCE & TECHNOLOGY:

Technology Assessment: ADP Installation Performance

Measurement and Reporting

7. AUTHOR(S)

Carol B. Wil son

9. PERFORMING ORGANIZATION NAME AND ADDRESS

NATIONAL BUREAU OF STANDARDS

DEPARTMENT OF COMMERCE

WASHINGTON, DC 20234

12. SPONSORING ORGANIZATION NAME AND COMPLETE ADDRESS (Street, CUy. State. ZIP) same as number 9.

15. SUPPLEMENTARY NOTES

Library of Congress Catalog Card Number r 79-600154

I

I Document describes a computer program; SF-185, FIPS Software Summary, is attached.

16. ABSTRACT (A 200-word or leaa lactaal aammary of moat aighificant inlormation. If doctanent inctudea a aignillcant bibliography or

Itterattae survey, mention It here.) This report compares the current status of ADP installation performance measurement

and reporting in the Federal ADP community with the best practices as found in the

Federal and private sectors and described in the literature. The comparison reveals

that more effort could be expended by Federal sites in the area of computer

performance management. The principal obstacles to more and better performance

programs are perceived to be the lack of needed measures on many systems and the

magnitude of the effort involved in accessing and analyzing the measures which are available. The report discusses several underlying causes for these obstacles and

makes three recommendations which could partially relieve the situation: (1)

development of standard performance measures, (2) development of a Government-wide

data base for normative performance ranges, and (3) development of statistical

computer performance evaluation techniques.

17. KEY WORDS (alx to twelve entriea; alphabetical order; capitalize only the firat letter of the firat key word imleaa a proper name;

aeparated by aemicoiona) Computer performance evaluation (CPE), computer performance

management (CPM), installation management, installation performance management,

performance measurement and reporting, resource management, standard performance

measures.

IS. AVAILABILITY (X) Unlimited n For Official Distribution. Do Not Release to NTIS

[X~] Order From Sup. of Doc, U.S. Government Printing Office, Washington, DC

20402, SD Stock No. SN003-003-a2123-7

1 1 Order From National Technical Information Service (NTIS), Springfield, VA. 22161

19. SECURITY CLASS

(THIS REPORT)

UNCLASSIFIED

21. NO. OF

PRINTED PAGES

37

20. SECURITY CLASS

(THIS PAGE)

UNCLASSIFIED

22. Price

$2.00

USCOMM-DC ^ U^. GOVERNMENT PRINTING OITICE : 1979-0-28 1-067

ANNOUNCEMENT OF NEW PUBLBCATIONS ON

COMPUTER SCIENCE & TECHNOLOGY

Superintendent of Documents,

Government Printing Office,

Washington, D. C. 20402

Dear Sir:

Please add my name to the announcement list of new publications to be issued in the series: National Bureau of Standards Special Publication SCO-.

Name —

Company

Address „

City State Zip Code

(Notiricatton key N-S03)

NBS TECHNICAL PUBLICATIONS

PERIODICALS

JOURNAL OF RESEARCH—The Journal of Research

of the National Bureau of Standards reports NBS research and development in those disciplines of the physical and

engineering sciences in which the Bureau is active. These

include physics, chemistry, engineering, mathematics, and

computer sciences. Papers cover a broad range of subjects, with major emphasis on measurement methodology, and

the basic technology underlying standardization. Also in- cluded from time to time are survey articles on topics closely

related to the Bureau's technical and scientific programs. As

a special service to subscribers each issue contains complete

citations to all recent NBS publications in NBS and non- NBS media. Issued six times a year. Annual subscription:

domestic $17.00; foreign $21.25. Single copy, $3.00 domestic;

$3.75 foreign. Note: The Journal was formerly published in two sections: Section A "Physics and Chemistry" and Section B "Mathematical Sciences." DEMENSIONS/NBS

This monthly magazine is published to inform scientists, engineers, businessrtien, industry, teachers, students, and

consumers of the latest advances in science and technology, with primary emphasis on the work at NBS. The magazine

highlights and reviews such issues as energy research, fire protection, building technology, metric conversion, pollution

abatement, health and safety, and consumer product per- formance. In addition, it reports the results of Bureau pro- grams in measlirement standards and techniques, properties of matter and materials, engineering standards and services, instrumentation, and automatic data processing.

Annual subscription: Domestic, $1 1.00; Foreign S13.75 NONPERIODICALS

Monographs—Major contributions to the technical liter- ature on various subjects related to the Bureau's scientific and technical activities. Handbooks—Recommended codes of engineering and industrial practice (including safety codes) developed in cooperation with interested industries, professional organizations,

and regulatory bodies.

Special Publications—^Include proceedings of conferences sponsored by NBS, NBS annual reports, and other special

publications appropriate to this grouping such as wall charts, pocket cards, and bibliographies.

Applied Mathematics Series—Mathematical tables, manuals, and studies of special interest to physicists, engineers,

chemists, biologists, mathematicians, computer programmers,

and others engaged in scientific and technical work.

National Standard Reference Data Series—Provides quantitative data on the physical and chemical properties of

materials, compiled from the world's literature and critically evaluated. Developed under a world-wide program co- ordinated by NBS. Program imder authority of National

Standard Data Act (Public Law 90-396).

NOTE: At present the principal publication outlet for these data is the Journal of Physical and Chemical Reference Data (JPCRD) published quarteriy for NBS by the American Chemical Society (ACS) and the American Institute of

Physics (AIP). Subscriptions, reprints, and supplements

available from ACS, 1155 Sixteenth SL N.W., Wash., D.C.

20056.

Building Science Series—Disseminates technical information developed at the Bureau on building materials, components,

systems, and whole structures. The series presents research

results, test methods, and performance criteria related to the

structural and environmental functions and the durability

and safety characteristics of building elements and systems. Technical Notes—Studies or reports which are complete in themselves but restrictive in their treatment of a subject.

Analogous to monographs but not so comprehensive in scope or definitive in treatment of the subject area. Often

serve as a vehicle for final reports of work performed at NBS under the sponsorship of other government agencies.

Voluntary Product Standards—Developed under procedures

published by the Department of Conmierce in Part 10, Title 15, of the Code of Federal Regulations. The purpose

of the standards is to establish nationally recognized require- ments for products, and to provide all concerned interests with a basis for common understanding of the characteristics of the products. NBS administers this program as a supple- ment to the activities of the private sector standardizing

organizations.

Consumer Information Series—Practical information, based on NBS research and experience, covering areas of interest to the consiuner. Easily understandable language and

illustrations provide useful background knowledge for shopping in today's technological marketplace.

Order above NBS publications from: Superintendent of Documents, Government Priming Office, Washington, D.C.

20402. Order following NBS publications—NBSIR's and FIPS from

the National Technical Information Services, Springfield, Va. 22161.

Federal Information Processing Standards Publications (FIPS PUB)—Publications in this series collectively consti- tute the Federal Information Processing Standards Register.

Register serves as the official source of information in the Federal Government regarding standards issued by NBS

pursuant to the Federal Property and Administrative Services Act of 1949 as amended. Public Law 89-306 (79 Stat

1127), and as implemented by Executive Order 11717

(38 FR 12315, dated May 11, 1973) and Part 6 of Title 15 CFR (Code of Federal Regulations).

NBS Interagency Reports (NBSIR)—^A special series of

interim or final reports on work performed by NBS for outside sponsors (both government and non-govemment).

In general, initial distribution is handled by the sponsor; public distribution is by the National Technical Information

Services (Springfield, Va. 22161) in paper copy or microfiche form.

BIBLIOGRAPHIC SUBSCRIPTION SERVICES

Hie following current-awareness and literature-survey bibli- ographies are issued periodically by the Bureau:

Cryogenic Data Center Current Awareness Service. A litera- ture survey issued biweekly. Annual subscription: Domestic, $25.00; Foreign, $30.00.

Liquefled Natural Gas. A literature survey issued quarterly.

Annual subscription: $20.00.

Superconducting Devices and Materials. A literature survey

issued quarterly. Annual subscription: $30.00. Send subscription orders and remittances for the preceding bibliographic

services to National Bureau of Standards, Cryogenic Data

Center (736.00) Boulder, Colorado 80303.
U.S. DEPARTMENT OF COMMERCE

National Bureau of Standards

Washington, D.C. 20234

OFFICIAL BUSINESS

Penalty for Private Use, S300

POSTAGE AND FEES PAID

U.S. DEPARTMENT OF COMMERCE

COM-21S

SPECIAL FOURTH-CLASS RATE

BOOK