*T 0F o0a

>e^r.

*1^

*£AU 0*

Q

J

NBS TECHNICAL NOTE 826

U.S. DEPARTMENT OF COMMERCE/ National Bureau of Standards

Cost-Benefit Analysis

of Computer Graphics Systems

QC

100

15155

1174-
NATIONAL BUREAU OF STANDARDS

The National Bureau of Standards ' was established by an act of Congress March 3, 1901. The Bureau's overall goal is to strengthen and advance the Nation's science and technology

and facilitate their effective application for public benefit. To this end, the Bureau conducts

research and provides: (I) a basis for the Nation's physical measurement system, (2) scientific and technological services for industry and government, (3) a technical basis for equity in trade, and (4) technical- services to promote public safety. The Bureau consists of the Institute for Basic Standards, the Institute for Materials Research, the Institute for Applied Technology,

the Institute for Computer Sciences and Technology, and the Office for Information Programs.

THE INSTITUTE FOR BASIC STANDARDS provides the central basis within the United

States of a complete and consistent system of physical measurement; coordinates that system

with measurement systems of other nations; and furnishes essential services leading to accurate and uniform physical measurements throughout the Nation's scientific community, industry, and commerce. The Institute consists of a Center for Radiation Research, an Office of Measurement Services and the following divisions: Applied Mathematics — Electricity — Mechanics — Heat — Optical Physics — Nuclear

Sciences 1' — Applied Radiation" — Quantum Electronics n — Electromagnetics'1 — Time

and Frequency 1 — Laboratory Astrophysics : — Cryogenics".

THE INSTITUTE FOR MATERIALS RESEARCH conducts materials research leading to improved methods of measurement, standards, and data on the properties of well-characterized materials needed by industry, commerce, educational institutions, and Government; provides

advisory and research services to other Government agencies; and develops, produces, and

distributes standard reference materials. The Institute consists of the Office of Standard

Reference Materials and the following divisions: Analytical Chemistry — Polymers — Metallurgy — Inorganic Materials — Reactor

Radiation — Physical Chemistry.

THE INSTITUTE FOR APPLIED TECHNOLOGY provides technical services to promote

the use of available technology and to facilitate technological innovation in industry and Government; cooperates with public and private organizations leading to the development of

technological standards (including mandatory safety standards), codes and methods of test; and provides technical advice and services to Government agencies upon request. The Institute consists of a Center for Building Technology and the following divisions and offices: Engineering and Product Standards — Weights and Measures — Invention and Innovation — Product Evaluation Technology — Electronic Technology — Technical Analysis — Measurement Engineering — Structures, Materials, and Life Safety ' — Building

Environment' — Technical Evaluation and Application 4 — Fire Technology.

THE INSTITUTE FOR COMPUTER SCIENCES AND TECHNOLOGY conducts research and provides technical services designed to aid Government agencies in improving cost effec- tiveness in the conduct of their programs through the selection, acquisition, and effective

utilization of automatic data processing equipment; and serves as the principal focus within

the executive branch for the development of Federal standards for automatic data processing

equipment, techniques, and computer languages. The Institute consists of the following

divisions: Computer Services — Systems and Software — Computer Systems Engineering — Information Technology.

THE OFFICE FOR INFORMATION PROGRAMS promotes optimum dissemination and

accessibility of scientific information generated within NBS and other agencies of the Federal Government; promotes the development of the National Standard Reference Data System and

a system of information analysis centers dealing with the broader aspects of the National Measurement System; provides appropriate services to ensure that the NBS staff has optimum

accessibility to the scientific information of the world. The Office consists of the following

organizational units: Office of Standard Reference Data — Office of Information Activities — Office of Technical

Publications — Library — Office of International Relations.

1 Headquarters and Laboratories at Gaithersburg, Maryland, unless otherwise noted; mailing address Washington, D.C. 20234.

3 Part of the Center for Radiation Research.

a Located at Boulder, Colorado 80302.

« Part of the Center for Building Technology.
Cost-Benefit Analysis

of Computer Graphics Systems

National Bureau of Standards APR 2 9 1974

Ira W. Cotton

Computer Systems Engineering Division

Institute for Computer Sciences and Technology

National Bureau of Standards

Washington, D.C. 20234

Sponsored by the

U.S. Army Electronics Command

AMSEL-GG-OD

Fort Monmouth, N.J. 07703

U.S. DEPARTMENT OF COMMERCE, Frederick B. Dent, Secretary

NATIONAL BUREAU OF STANDARDS, Richard W. Roberts, D/'recfor

Issued April 1974
National Bureau of Standards Technical Note 826

Nat. Bur. Stand. (U.S.), Tech. Note 826, 47 pages (Apr. 1974) CODEN: NBTNAE

U.S. GOVERNMENT PRINTING OFFICE

WASHINGTON: 1974

For sale by the Superintendent of Documents, U.S. Government Printing Office, Washington, D.C. 20402 (Order by SD Catalog No. C13.46:826). Price 90 cents.
TABLE OF CONTENTS

1Introduction 1

2. Computer Graphics System Design Alternatives 3

3. Analysis of System Performance 6

4. Cost Analysis 15

5. Cost-Effectiveness 18

6. Benefit Analysis 26

1. Synthesis: Cost-Benefit Analysis 30

8. Conclusions 37

Bibliography 38

in

Cost-Benefit Analysis of

Computer Graphics Systems

Ira W. Cotton

This report assesses the state-of-the-art in

cost-benefit analyses of computer graphics systems and

suggests an approach for developing improved methodology.

Cost-benefit analyses are distinguished from analyses of

system performance in that the latter are directed at

optimizing system performance at a given level of

investment, while the former are directed at justifying

the investment itself.

Computer graphic system design alternatives are first

outlined. Then methods of analyzing the performance and

costs of computer systems in general and graphic systems

in particular are discussed. With this information it is

shown how cost-effectiveness analyses may be performed.

The next crucial step is to conduct benefit analysis, an

ill-defined art. The results of benefit analysis must be

combined with cost-effectiveness analysis in order to

perform the desired cost-benefit analysis.

An experimental methodology is suggested for better

performing benefit analyses of computer graphics systems.

A more rigorous formulation of the cost-benefit procedure

is then outlined. No attempt is made in this report to

actually perform such an analysis.

Key words: Computer graphics, cost-benefit analysis,

cost-effectiveness, economics, performance

evaluation

1. Introduction

Computer graphics is a sub-discipline within computer

science which deals with the manipulation of digital

representations of pictorial information, including output

of the picture on an appropriate display device and

modification of the image in response to operator-initiated
actions. There are many systems based on this technology in

use today, primarily in the area of computer-aided design,

and many cost savings are attributed to such systems. Hard

evidence to support such claims is mostly lacking, however.

Few studies have been published relating economic

principles to the design and implementation of computer

graphics systems. Those that have been published deal with

only one side of what is essentially a two-faceted problem.

Either a detailed analysis of the costs and performance of

an optimized system is presented, without corresponding

analysis of the economic benefits to be derived; or expected

benefits are estimated without discussion of the costs of a system to achieve those savings. The first type of study

may be characterized as a cost-effectiveness evaluation; the

second as a benefit analysis.

Cost-effectiveness studies are aimed at optimizing the

performance of a system according to stated criteria and for

a given level of investment. The sensitivity of system

performance to changes in level of investment may also be

considered within the scope of cost-effectiveness analysis.

No value, however, is assigned to the benefits from improved

performance. Benefit analysis, on the other hand,

investigates just this issue, the value of improved

performance, but without regard to the cost of achieving it.

Cost-benefit analysis provides a link between

cost-effectiveness analysis and benefit analysis.

Cost-benefit analysis is concerned with providing adequate

information for making the investment decision. Is the

stream of benefits resulting from the use of a system with a given level of performance greater than the required

investment? What is the optimal level of investment to

maximize net benefit? Such are the questions addressed by

cost-benefit analysis.

Cost-effectiveness analysis is necessary but not

sufficient for cost-benefit analysis. A valid cost-benefit

study requires that the optimal level of performance for a given level of investment be considered. Alternatively

stated, for each given level of investment, the value of the

benefits considered should be the maximum possible. This is

precisely the information provided by cost-effectiveness

studies. What cost-effectiveness studies fail to provide is

any assessment of the value of the benefits. Such

assessments are often difficult to perform, but they are

crucial to the investment decision.

We will be concerned in this report with the methodology
of cost-benefit analysis. Methodologies for

cost-effectiveness studies will be considered too because,

as has just been explained, they are essential for valid

cost-benefit studies. The relevant literature will be

critically reviewed and suggestions offered where published

accounts are lacking. No attempt is made here to actually

perform such an analysis.

2. Computer Graphics System Design Alternatives

Before any consideration of economic analysis is possible

it is first necessary to outline the technology of the

systems under consideration: It is convenient to assign

graphic systems to one of three main classes, as follows.

Basic System

Computer graphics systems of the type under analysis

consist, in their simplest form, of an analog display device

similar to an oscilloscope, a display processing unit which

converts digital display commands to analog signals to drive

the display, and a general purpose computer system which

generated the digital commands in accordance with some

application program. This system is illustrated in Figure

1.

Satellite System

This basic system may grow in a number of ways [22],

More and more features may be added to the display processor

until it comes to resemble a computer itself* This permits

the display device to be located remotely from the main

computer, connected by a communications link. Since the

link is generally slower than the processing rate of the

display, local storage is required to hold the commands sent

from the central computer. However, in this type of a system we permit only primary storage; secondary or mass

storage such as a disk or drum is not permitted. This type

of configuration is called a satellite graphics system

because the graphics processor is located at a distance from

the central computer but is dependent on it. The general

configuration is illustrated in Figure 2.

Such a system offers a number of tradeoffs for the

designer [25]. The speed of the communications link, the

power of the display processor and the amount of local
Central

Computer

Display

Processing

Unit

Disploy

Mass

Storage

Figure 1. Basic Computer Graphics System

Central

Computer

Communications Link

Satel lite Computer

and Display

Processing unit Display

A

Mass

Storage

Figure 2. Satellite Computer Graphics System

Central

Computer

Mass

Storage

Communications Link Satellite Computer

Mass

Storage

Display

Processing

Unit

Display

Display

J

J

Figure 3. Network Graphics Configuration
primary storage all may be increased at increased cost,

Also, certain tasks may be performed either in the main

computer or in the local processor. The decision as to

which components to upgrade and where to perform processing

tasks is not obvious.

Network Configuration

The situation can get even more complex. It is well

within the capability of today's minicomputers to control

multiple displays located remotely from the central

computer. In such configurations, it is again desirable to

separate the digital processing functions from the analog

conversion. In addition, it will probably be necessary at

this stage to permit the graphics processor to have a mass

storage subsystem. All of this results in a true network

coflf iguration where multiple users share facilities at each

end of the communications link. This is illustrated in

Figure 3-

The attachment of input devices has not been considered

in order to simplify the problem. In general, input devices

send their data to the closest processor. In the first case

this would be the main computer. In the latter two cases

input signals would be initially processed by the satellite

computer. The degree of processing performed at the

satellite in response to inputs, however, is a design

variable and may vary widely from system to system [7],

Tradeoffs

For each of these types of systems there are three types

of tradeoffs which can be made: hardware/hardware,

hardware/software and software/software. Hardware/hardware

tradeoffs refer to the possibility of diverting costs from

one component to upgrade another. For example, mass storage

capacity at a satellite might be reduced in order to pay for

a faster data link, or vice versa. Hardware/software

tradeoffs refer to the possibility of implementing certain

functions either in hardware or software. Character

generation is a good example of a function which can be

provided either way. Software/software tradeoffs refer to

the possibility of performing certain processing tasks

either in the main computer or the satellite. Display

generation, or the process of formatting display commands

from a description of the graphic design in a data

structure, is an example of a function which may be

performed in either place. All of these types of tradeoffs
challenge the system designer to come up with the optimum

for a particular set of criteria. This is what the art of

system design is all about, and efforts to express these

tradeoffs analytically is what is meant by making system

design into a science.

3. Analysis of System Performance

There are essentially two basic approaches to evaluating

the performance of a computer system: observation and

analysis [30]. The system is either set various tasks

(benchmarks) to perform and is graded on how well it

performs them; or the system is analyzed in terms of

specific parameters in order to derive a measure of

performance. The tradeoff between these two approaches is

between the detail and accuracy of the analysis and the cost

of performing it. The first method has the appeal of direct

measurement: the system is assessed as a whole. However,

benchmarks are expensive to prepare and to run. Parametric

analyses are less expensive to perform, but their results

are less reliable since the effects of small errors in

determining the parameters may be cumulative. Specific

techniques of system analysis fall somewhere on a continuum

between pure observation and pure parametric evaluation.

Feature Analysis

The first serious attempt to provide a formal basis for

comparing computers was a listing of the various

characteristics of different computers published by Adams

Associates. This listing has been called an Adams chart; it

is still published by Keydata and typifies a simple

portrayal of the classical features of computers

manufactured worldwide. In the graphics area, Adams

Associates also published the <Computer Display Review>

which listed the features of graphics equipment; it was

continued by Keydata and is now published by GML

Corporation.

As pointed out in [17], the limitation of the Adams chart

is the inference that performance can be meaningfully

predicted from these classic features. It has been well

established that the use of such raw speed parameters as

clock speed, arithmetic speed, memory speed, word size or

I/O rate can be misleading in predicting comparative

performance between different systems. Over the years, many

people have taken different combinations of these classic
features and have used these combinations as figures of

merit to infer performance measures.

Any listing of simple features fails to incorporate the

differences that each system designer included to improve

his system. The power of each instruction set combined with

each system's architecture is not indicated; the

effectiveness of system software is not considered; and

factors for evaluating special capabilities such as

multiprogramming or multiprocessing are lacking.

Instruction Mixes

A major advance over the feature analysis approaches was

the development by the Auerbach Corporation of a standard

set of small problems which could be programmed by experts

for the variety of different machines and configurations

under consideration. The results yielded the amount of

memory required, the amount of code and similar detail. The

implication is that performance on these standard problems

is indicative of the performance on actual user problems,

and that small-problem performance is indicative of

large-problem performance.

As Johnson observes, however, all instruction mixes [17]

suffer from the limitations that the actual mix in use may

or may not correspond to that of the "standard" mix.

Further, none of the instruction mixes yet devised provide a reliable indication of full load behavior of computing

systems. This is an especially serious shortcoming when

attempting to predict the performance of multiprogramming or

multiprocessing systems. The full load performance of such

systems does not appear to be proportional to its

small-problem performance. Nevertheless, simple instruction

mixes have been widely used over the years because they are

simple, easy to use, and provide a reproducible evaluation

methodology.

The application of this technique to the analysis of

graphic systems is reflected in the set of test patterns

which was developed by Adams Associated for their <Computer

Display Review>. These patterns were designed to be coded

for each display system to be evaluated. One immediate

result would be a comparison of the number of words

necessary to code the pattern in its entirety. The results

of running each pattern were the percentage of the pattern

that could be displayed flicker free - a measure of the

capacity of the display for applications of which the test
patterns were supposedly representative. The patterns used

were the following:

Alphanumeric Test Pattern - The alphanumeric test

pattern is representative of applications requiring the

display of tabular or free-form character data- Figure 4

illustrates this test patternWeather Map Test Pattern - The weather map test

pattern is representative of a class of applications

including geographical or mathematical contour mapping as

well as sheet metal styling. Figure 5 illustrates this

test pattern.

Graph Test Pattern - Graphs and charts are very common

displays, both in business and scientific applications.

Figure 6 illustrates this test pattern.

Architectural Drawing Test Pattern - The architectural

drawing test pattern is representative of applications

where a large quantity of unstructured data must be

displayed with high precision. Such applications include

circuit board layout and mechanical design. Figure 7

illustrates this test pattern.

Electronic Schematic Test Pattern - The electronic

schematic test pattern is representative of applications

where pictures are constructed hierarchically, using

repeated instances of certain patterns or subpictures.

Such applications include logical design and certain

layout problems and mechanical analysis systems. Figure

8 illustrates this test pattern.

Kernels

A more sophisticated version of the instruction mix

approach to the anlysis of system performance uses

"kernels." A kernel is a complete nucleus problem;

meaningful kernel problems are selected according to the

type of application of interest, e.g., a matrix inversion

for scientific problems or a small sort for business

applications. The execution time of a set of such problems

is assumed indicative of the system's execution of the whole

application. When kernels are programmed by experts on

their respective machines and the actual software of that

system is used, the results can be good indicators of that

small-problem behavior of that system.
BURNER SYSTEM

System Valves System Status System Signals

SUPERHEATER C *TRIPPED* STEAM PRESSURE N REHEATCR C FURNACE PRESSURE N

MAIN GAS SIIUTOFF GAS TEMPERATURE N

VENT COMBUSTION M

MAIN OIL SirUTOFF C FORCED DRAFT FAN R

RETURN C AIR HEATER

NORTH C SOUTH C IGNITOR CHARGED Y

GAS READY Y

IGNITOR SinJTOFF

VENT C OIL READY N

DUAL PERMISSIVE N

PENTHOUSE NO. 1 NO. 2 MODE SWITCH C VENT PURGE REQUIRED Y

BURNERS 1234S6789 10

STATUS S * * * I -

IGNITOR

SHUTOFF

FLAME

TRANS CCCCCCCCCC ----__-_-_ ----______

MAIN FIAME N F F F - - - -

GAS SHUTOFF CCOOOOCCCC

OIL SHUTOFF

PURGE

ATOM

UNIT

CCCCCCCCCC

CCCCCCCCCC OOOOOOOOOO IIIIIIIIII

COOL

DAMPER OOOOOOOOOO

CONTROL

IGNITOR

GASOIL ----------

---------- -XXXXX

DUAL - - - - - - - - - " - OFF XX - - »- - -XXX

Figure 4

ALPHANUMERIC DISPLAY TEST PATTERN

©Copyright GML Corporation

9
Figure 5 WEATHER MAP TEST PATTERN

)Copyright GML Corporation

10
r \

)

x

l - / ; - / { - / \\

""•N >-

\ / -

>

( -

10 A-

)-

V) CD i

111 *D y'

— /

_l /

<

1 -

^TM / -

z < -

3 \

> \ < \

-

Q.

CO v >-

a

) /

-

/ /

\ \

) \ - \ >-

y< -

1

I 1 1 1 1

1 1 ^ i

1 1

i i

>•<

2

a.<

a:<

5

COUJ

~3

o o o o o o o

o 00 tf> <r CVJ o CO m sr <T <r t <r to

vD

ao•**

COuII2

«-» J5M

"m>.a

oO

©

ex: w

H

H

<a.H

w

w

H

a,2

o

sdvnioa jo saNvsnoHi

li
CUPBOARD

" r8"X3'2" 5SQ FT

TOTAL FLOOR AREA 875 SQ FT

CIRCULATION !89 SQ FT 22%

Figure 7

ARCHITECTURAL DRAWING TEST PATTERN

JCopyright GML Corporation

12
<k>WV

fAA/V

»-WV^

Figure 8

ELECTRONIC SCHEMATIC TEST PATTERN

©Copyright GML Corporation

13
Benchmarks

The most reliable and a has been developed to anal

actually code and run a re

programs selected from tho

actual application. Such

"benchmarks," However,

benchmark programs run in

represent the nature of th

several benchmarks partial

choosing the right combina

difficult. Still, as long

large and small mixes expe

good indication can be obt

problem behavior of the sy

ccurate measurement technique that

yze total system performance is to

presentative mix of complete

se that are to be run in the

sample programs are called

pitfalls still exist: the

sequence may not accurately

e work to be performed. Combining

ly alleviates this problem, but

tion of benchmarks is also

as benchmarks are run in the

cted in the actual operation, a ained of both the large and small

stem.

The real limitations of the benchma

it takes a large effort to develop the

to prepare realistic data. The task i

batch-oriented computer systems for wh

programs written in standard compiler

developed for benchmark purposes. It

impossible task for computer graphics

programming systems for each are compl

Also, the results of running benchmark

the benchmarks are representative of t mix. Graphics programs typically are

of different modules with radically di

characteristics. Furthermore, an esse

graphics programs are the sequences of

direct the execution paths of such pro

sequences are difficult to model and e generate in a test situation,* Thus,

for graphics systems which are truly r actual applications is difficult to do

are equally difficult to run.

rk approach are that

sample programs and

s difficult enough for

ich some libraries of

languages have been

is nearly an

systems where the

etely different,

s are only as valid as

he real application

composed of a number

fferent

ntial part of the

operator inputs which

grams. These

xpensive to actually

developing benchmarks

epresentative of

and the benchmarks

Simulation

A final technique which can be used used in performance

analysis is simulation. Simulation cuts across the two

general approaches of observation and analysis. To use this

technique, a model is constructed of the system under

consideration and the performance of this model is

"observed" under the desired conditions. The model

* Work is in progress by a number of researchers at The

Mitre Corporation and at the National Bureau of Standards on

"Terminal Environment Simulators" which can serve as drivers

for benchmark programs for interactive systems.

14
represents a simplification of the actual system which is

constructed from analysis of the system. If the model

accurately represents all the salient features of the

actual, when it is run with the desired (simulated) workload

the result is analogous to observing the actual system.

Of course, the results of simulation can be no better

than the many assumptions that go into the construction of

the model and into the design of the simulator.

Constructing a computer system simulator for performance

evaluation purposes is, as has been pointed out,[H] not an

easy task. If the level of simulation is too fine, the

simulator will be too costly to use. If the level is too

gross, not enough information will be yielded. Despite

these difficulties, simulation is becoming; much more widely

used, and a good simulator properly used nay be one of the

better tools for predicting system performance accurately.

As we shall see presently, it is also a technique which can

and has been used to analyze computer graphics systems.

*J . Cost Analysis

The principal problem of the cost analysis of computer

systems is to express all the costs of alternative system

proposals in a form suitable for analysis. This requires

that ALL relevant costs be included, and that all costs be

expressed in a comparable form. The question of which costs

to include has been well treated in the literature. Joslin

[18] presents a detailed analysis of all the types of costs

which must be included in evaluating a system design

alternative. The main complication in many cases is the

need to include development costs in the analysis which are

not known in advance but which must be estimated.

Cost Estimates

If two or more proposed systems are to be compared, it is

necessary to compare the amount of development effort

required for each. Data for such an analysis is "softer"

than data describing the pricing of components or even

charges for the use of components. Substantial work has

been done in developing formal techniques for estimating

such costs [20, 2*0, but their main effect has been to

reduce the margin of error from outrageous to large. Yet

the estimates made of the cost to implement functions in

either the central site or the satellite may be the dominant

factor in system selection.

15
For example, large systems today all come with operating

systems, and most with communications handlers as well. On

the other hand, few minicomputers come with the type of

operating systems which would be required to shift some

functions from the central site to the satellite. The cost

to implement such an operating system may be critical to

determining the cost-effectiveness of the proposed division

of labor.

Of course, given a large enough number of satellite

systems over which to share the total development costs, the

average development cost may be made arbitrarily low. Thus,

an honest and realistic assessment of the expected number of

replications must also be made. This is primarily a market

research problem.

Comparable Form

The other problem of cost analysis is to express all

costs in comparable forms. The assignment of the

appropriate unit costs to the use of the various resources

is of critical importance. A controversy may arise in the

analysis of graphic systems over whether to price at average

or incremental costs. For resources which would not be

present if there were no graphics system, incremental and

average costs are the same. However, if existing facilities

are used part time for graphics, the incremental costs will

be less than the average costs. If any tradeoff is to be

valid, costs for resources in the central system and at the

satellite must be comparable. This can be accomplished by

using, in all cases", equivalent monthly rental charges or

average resource utilization charges, as appropriate.*

For example, a graphics processor cannot be utilized at

no cost simply because it is owned and always available. An

equivalent monthly rental charge must be determined and used

to derive unit-time operating charges so that they may be

aggregated with the charges for the central processor, which

are likely in this form already. Likewise, file storage

charges for devices at either location must be represented

in equivalent, non-zero terms.

* Actually, the costs used for the analysis may vary

according to the purpose of the analysis. An individual

company considering adding a graphics capability to an

existing facility will want to use incremental costs in

order to determine if the addition is warranted. An analyst

wishing to compare two alternative graphics systems in the

abstract would prefer to use a measure which assigns a cost

to resources irrespective of the use to which they are put.

16
Fixed and Variable Costs

The concept of fixed versus variable costs is also a relevant one. Development costs such as design and

programming costs are fixed, as are hardware components in

the graphics terminal since they cannot be used for any

other purpose (their cost is fixed regardless of utilization

rate). In contrast, the use of central site resources such

as storage and the execution of programs are variable costs,

since the central computer is shared by a large number of

users who only pay for the units of each resource as they

are used. The costs for some resources may be either fixed

or variable depending on the particular arrangements. For

example, communications charges would be fixed if a leased

line were used and variable if dial-up facilities were used.

The significance of expressing the charges in this manner

is that where fixed charges dominate, total costs are

relatively constant but the unit operating costs for a Riven

configuration are very sensitive to the level of usage. In

such situations, only high utilization of the graphic

equipment can bring down the unit cost [26], The

sensitivity of unit costs to level of usage may be reduced

if fixed costs can be converted to variable costs. For

example, a higher speed communication link may make it

possible to store programs and data at the central computer

which are used by the graphics program at the satellite.

This would permit the amount of memory at the satellite,

which represents a fixed cost, to be reduced in favor of use

of storage at the central site, which usually represents a variable cost.

However, the argument goes two ways. Above a certain

level of utilization, called the "breakeven point", it is

more economical to convert variable expenses to fixed ones.

Communications charges are an excellent example -- above a certain level of utilization it is cheaper to lease a full

time link than to make a call each time. So estimates of

the level of utilization for a system may still be critical

to system design decisions.

Generally, the problems of cost analysis for computer

systems are well-understood (though they may not be

well-solved). Computer graphics systems are not so unique

in this regard that they present any special problems.

Perhaps the greatest challenge for cost analysis is to

obtain results in a form such that the marginal costs of

performance were obvious or could be easily determined.

This would permit the costs for improved performance or the

17
savings from reduced performance to be more easily used in

making design decisions. A parametric formulation

explicitly relating costs to level of performance and

utilization rates could serve as input to a decision model.

5, Cost-Effectiveness

Cost-effectiveness analysis is a technique used to

compare alternative systems where the benefits to be derived

cannot be determined quantitatively. The object of

cost-effectiveness analysis is to select either the system

with lowest cost for a given level of performance or

greatest performance for a given level of cost. The method

requires that either the costs or the performance associated

with all the systems under consideration be equal. If both

costs and performance are different for any two systems to

be analyzed, then cost-effectiveness analysis is impossible

since the parameters have different dimensions and may not

be compared.

The application of cost-effectiveness analysis is fairly

straightforward. First it must be decided which parameter

to hold constant -- cost or performance. Cost is frequently

held constant when there is no absolute level of performance

which is required. In this case, a fixed amount of money is

made available and the analysis seeks the system with which

offers the greatest performance, A measurement criteria is

selected and the systems are evaluated according to that

measure. The system which measures the greatest --for

identical cost-- is the most cost-effective. Tradeoffs

between system components may be investigated by varying the

mix of components subject to the constraint that total

system cost remain the same.

The alternative approach is to specify in advance the

performance required of the system. The system is selected

which offers at least that level of performance for the

least cost. Performance in excess of the specified

capabilities is ignored in the analysis. For such systems,

tradeoffs focus on the reduction of performance (and with

it, cost) to the point where performance just equals that

required.

Many studies have been published discussing the tradeoffs

involved in the design of graphics systems [16, 22, 25, 31,

32], However, few published results show any serious

attempt to quantify these tradeoffs. Quantification is

necessary if a true cost-effectiveness analysis is to be
performed. One of the first published results of a quantitative analysis of design, tradeoffs was Foley's study

of satellite graphics systems [11].

Foley s Analysis

Foley defined optimum design of a computer graphics

system as "maximizing a display system's performance subject

to a cost constraint." In his view, optimum display design

can be thought of as a resource allocation problem. The

resources are dollars which can be allocated to the purchase

of display subsystems of differing individual performance.

System response time was employed as the sole measure of

system performance. (Minimizing response time means

maximizing performance). Total system performance is

determined from a model of how the subsystems fit together.

The parameters of the model are functions of the

capabilities of the graphic hardware and of the

computational requirements of the graphics application. The

model can be analyzed using numerical queueing analysis or

simulation to obtain an average response time prediction.

By applying an optimization procedure, the "best" graphics

system configuration, subject to a cost constraint, may be

found for several applications. The optimum configurations

are in turn used to find general system design guidelines.

Foley used a combination of kernels and instruction mix

techniques to evaluate the computer power of the remote

display controller. Basically a set of "display commands"

was defined. These commands are at a level higher than

machine instructions but lower than a high level language.

The commands are coded in the machine language of the

controller and timed. Each display command timing is

weighted by a factor representative of the relative

frequency of execution of that command for a particular

application. The greater the value of the weighted

instruction execution rate (the reciprocal of the command

execution timing) the better suited is the corresponding

controller for the application. The full account of Foley's

research [10] indicates that he also employed the Adams'

test patterns in a similar manner to compare the suitability

of display controllers for different applications.

Foley's analyses yielded curves of the form shown in

Figure 9. Such curves show the price which must be paid for

a given level of responsiveness. Performance is actually

the reciprocal of response time, so that figure 9

illustrates the relation between performance (output) and

cost (input). The decreasing slope of the curve reflects

19
RESPONSE

TIME

(SECONDS)

MONTHLY COSTlfc)

Figure 9. Typical Result of Foley's Analysis

20
the decreasing marginal productivity of additional units of

cost in Improving performance. The analyses also revealed

that different regions on the curve were associated with the

upgrading of different components in the system. Simply

stated, this means that different components were beinpupgraded when performance was improving most rapidly than

when performance was increasing less rapidly. Thus the

analysis yielded a set of decision rules indicating the

order in which a limited amount of additional money should

be spent.

Cislo's Approach: Simulation

Cislo expanded on Foley's approach by considering a more

complex model of a graphics system in greater detail [6].

Foley neglected the cost of resources at the central

computer; Cislo's study includes them. Cislo also modeled a multiterminal system in which each terminal is used for a different application; this went beyond Foley's analysis.

Finally, Cislo employed simulation in contrast to Foley's

use of queueing analysis.

Cislo's model of a graphic system decomposed the

operation of the system into a number of low level

operations which might be viewed as the kernel problems of a display system. The activities included in the model are

listed in Table 1; Figure 10 illustrates the model itself.

A GPSS program based on the model was written and used to

exercise it. The program simulated operation of the system

by generating transactions which correspond to user

requests. The interarrival time between requests was

determined randomly from a negative exponential

distribution. Similarly, the selection of the appropriate

routines to process the request was determined

stochastically.

21
DTRFH

DTRAN

/ {

HOST

DCONV

i ^

DTRTH

STORG

t

ASUBR

-* 1 TRANS

!

, i

' . !i

UPDTE

TXTPR

i'

GENGO TRACK ATNPR

'

-

f 1 1

'

WAIT FOR NEXT REQUEST REQUEST ATTENTION WAIT FOR NEXT REQUEST

Figure 10. Cislo's Graphic System Model

22
Activity Mnemonic

Attention Processing ATNPR

Transformation TRANS

Text Processing TXTPR

Tracking TRACK

Application Processing ASUBR

Data Transmission to Host DTRTH

Host Computer Processing HOST

Data Transmission from Host DTRFH

Data Conversion DCONV

Secondary Storage Operation STORG

Update Data Structure UPDTE

Graphic Order Generation GENGO

Table 1. Model Activities

The parameters of the model are very similar to those of

Foley's model and the data were similarly obtained.

Published data were used for simple parameters such as

instruction execution and data transmission rates, and

simple test programs were generated for the estimation of

such parameters as the time required to generate graphic

orders. Application-dependent parameters such as the

probability of generating a given number of graphic orders

were much more difficult to estimate, and in many cases were

simply assumed.

In exercising the model, it was decided to hold these

latter parameters constant while varying the hardware

parameters. Accordingly, the operation of eight different

hardware configurations, involving changes to the main CPU,

display CPU and data transmission facility, was simulated.

The results were in accordance with Foley's findings, but

the method accorded much more detailed descriptions of

system activity. The expanded version of the model which

would handle multiple independent graphics consoles was

described but not implemented.

Validity of Indicators

These techniques which have just been described offer

much hope of permitting the analysis of computer graphics to

be performed on a much more systematic and quantitative

basis. The value of the analysis, however, is dependent on

the validity of the variable(s) which is optimized as an

indicator of total system performance. Accordingly, this

23
discussion of cost-effectiveness is concluded with a discussion of this question.

Response Time

The response time provided to a user has long been

considered a critical parameter in the performance of

interactive systems. The general approach of system

designers to this parameter has either been to "minimize

it", subject to available monetary resources, or to insist

on a minimum response time, e.g., 2 second response 95$ of

the time, regardless of the cost. This approach is overly

simplistic and may be far from cost-effective for a number

of reasons.

rirst of all, not all interactive tasks require the same

responsiveness of the computer. Miller [231 has shown that

there are several classes of interactive activity with quite

different response requirements. One class is the input of

data by various means. An immediate response of no longer

than .1 - .2 seconds is required for this class to signify

acceptance of the data. A second class is characterized by

a user engaged in high-intensity "brainstorming" requiring

the ready access of data from his own "short-term memory."

Such activity requires no longer than a two-second response

in order that the chain of thought not be broken. A final

class includes those activities which complete a subjective

(sub)task or (sub )purpose , More extended delays (up to 15

seconds or more) may be permitted following such an activity

completion, or "closure", than in the process of obtaining a closure.

These findings may have an important impact on system

design, since alternative methods exist to provide each type

of response. For example, verification feedback and simple

data manipulations may be supported on a so-called

"intelligent terminal" with its own local mini-computer.

Since the response from complex application-dependent

computations may be permitted to be less rapid, these may be

provided by a large time-shared computer, possibly located

at some remote location and connected by a communications

circuit. A cost-effective design will assign the function

of providing each type of response to the facility which can

meet the required response and meet it at least cost.

Other factors may also be considered relating to response

time. A uniform response time has come to be recognized as

very important (in contrast to a response time which varies

widely from transaction to transaction) -- so much so that

24
in some applications the responsiveness of the system is

artificially delayed when it falls below the desired value,

so that the variability of response is thereby reduced (at

the expense of lower average response time).

Finally, some experimentation has been performed into the

effects of restricting the free access to the computer after

it has responded to each major request (similar to a closure) [3]- It was found that "users tend to become

dissatisfied if mild restraint is placed on their free

interaction with the computer." However, "they also tend to

problem solve more effectively, using less computer time and

less of their own time in the process." The authors suggest

that "the results cast doubts on the validity of user

acceptance as a general index of system effectiveness."

Clearly, the matter of the responsiveness required of a graphics system is not a straightforward question, and it

does require serious consideration on the part of the system

designer.* The dilemma is exacerbated by the demonstrated

sensitivity of responsiveness to additional investment [11],

Other Indicators

Another effect of the over-reliance by analysts on

response time as the sole measure of performance has been

the neglect of a number of other important performance

characteristics. These have been conveniently divided into

task-dependent factors and human-dependent factors [2].

Task-dependent factors include screen size, message size,

message format, erasability, color and half-tone capability

in addition to response time. These factors will vary with

importance according to the particular application; hence a weighting approach should be used when evaluating systems

according to these factors.

Human-dependent factors, or more simply human factors,

permit the system to be evaluated on its usability by human

beings. Such factors include brightness, contrast,

resolution, readability and the visual fidelity of the

display. These factors are not as importance as they were

in the early days of display system design when they were

less-well understood, but they should still be validated for

any system under consideration. Other ergonomic

* An interesting application of response time data to

design tradeoff problem is presented in [33].

25
considerations might also be important in special cases, for

example if the display were meant for proup viewing. Again,

the needs of the particular application will dictate

requirements. A table in [2] presents a cross reference of

desirable characteristics for specific display applications.

Both task-dependent and human-dependent factors are

included.

6. Benefit Analysis

The benefits from a computer system arise from performing

some tasks faster or qualitatively better than previous

methods. Benefits are generally measured relative to the

previous method. The problem is to assign actual dollar

values to the benefits measured.

Methodologies for performing benefit analysis of computer

systems are analogous to the methodologies for evaluating

system performance. There are two general approaches:

empirical test (benchmarks) and parametric analysis.

An empirical test to determine the benefits of a given

system would require two control groups and a typical task.

One group would perform the task according to the old or

baseline method; the other group would perform the task with

the new system. Two types of benefits could be measured

from such an experiment: cost savings from using the new

system to achieve the same level of performance as the old

method, and benefits from any performance levels achieved in

excess of what was normally accomplished the old way.

The other approach of assessing benefits by parametric

analysis involves decomposing tasks into a number of

elements. The benefits from improved performance in each

element must be assessed, and then the old and new system

can be compared on each element. For each application, the

required tasks can be reconstructed from an appropriate

aggregation of elements (where the parameters specify the

proportions for each). In this way the benefits from doing

the particular application with the new system can be

estimated.

Unfortunately, very few published results of either type

of study are extent in the literature. Productivity claims

of 500$ and above for the first type of study have been

bandied about at professional society meetings, but the

supporting evidence has not been published, either because

it did not exist in satisfactory form or because such data

26
was considered proprietary. What are most commonly found

are comparisons of an installed system with the old way of

doing things, without controls for any other variables.

Sackman [28] has presented a summary of several such studies

comparing on-line vs. off-line performance on a number of

different tasks [1, 9, 12, 1U, 27, 29]. The shortcomings of

the experimental methodology employed in most of these

studies has been critiqued [21].

Occasional studies of the second type have been

published, but the focus has generally been so narrow that

the results could not be generalized [5]. An example of a better study of this type is Gold's study of problem-solving

performance in time-sharing versus batch processing [13]*

Gold's Study

Using a programming language available in both batch and

on-line environments, Gold "focused on a development of a methodology through which time shared computer system usage

could be evaluated." five categories of variables were

included in the methodology:

Cost of using the system

System performance

Turnaround time

The mode of learning resultant from system use

The attitudes of the system users

Principal variables associated with the measurement of

the computer system features included computer response

delays, degree of interaction between the user and the

computer system, and quality of response in feedback

mechanisms and programming systems.

User attitudes are defined in the study as "the degree to

which the computer system characteristics appear to the user

to facilitate or hinder him in the attainment of his short

and long term objectives." User behavior is further

characterized in the study as "the degree to which the user

relegates programmable problem solving to the computer

system." Performance for the purposes of the study is

defined as "a measure of the output of the man-computer

system which is arrived at independently of the user's

behavior or the computer system used."

The following are the significant findings of the study

with respect to user behavior:

27
1 , The users of the time sharing system interacted

with their computer system more than three times as often

as did the batch-processing users*

2„ There was no significant difference between the

reasons advanced by the time sharing or batch-processing

users for initiating computer actions.

3. There was a strong relationship between a batch-processing user's performance level and the number

of computer interactions which produced usable output.

For the time sharing user, the correlation between

performance and the number of sessions with the time

shared computer console is strongest.

M. Much more favorable attitudes toward the time

shared computer system, its use and the results produced

through it were evidenced.

With respect to quality and cost of results, the

following results were reported:

1, Use of the time sharing system resulted in a higher

level of objective performance than use of the batch

processing system. Also, the "perception and

understanding of the problem" of the time sharing users

was evaluated as significantly higher than that of the

batch users.

2. The total cost of time sharing and batch usage did

not appear to differ appreciably. Higher computer costs

and lower man costs were found with time sharing usage,

however. There were indications that the total cost for

equivalent objective performance would have been less for

the time sharing users.

A Suggested Approach

Studies of the type just discussed are limited in that

they may not be generalized beyond the specific sample

activities under investigation. There has been no attempt

to decompose the overall activity into more basic

application-oriented functions, and to model different

activities on the basis of these functions.

Application-oriented functions are different in character

than the type of model activities developed by Cislo. For

example, text editing and geometric sketching might be two

tasks elements which would be used in different proportions

28
in a particular graphics application. Determine the

benefits of improved performance on each of these tasks

would provide the necessary data to determine the benefits

for an application comprised solely of those two activities.

A sample list of kernel activities for benefit analysis is

presented in Table 2. Other elements need to be identified

and analyzed.

Text Editing - original input and modification

Sketching - original input of geometric data

Updating - modification of geometric data

Inquiry - transaction processing

Directing - on-line control of other operations

Monitoring - of other operations

Mechanization of Output - production of finished drawings

Rapid Turnaround - should be isolated as a separate

benefit

Table 2. Sample Kernel Graphic Tasks for Benefit

Analysis

The paradigm suggested would employ first analysis, then

experimentation, then synthesis and finally validation by

means of experimentation again. The goal. is to isolate the

individual task activities in an application, experimentally

determine the individual benefits for each, parametrically

aggregate these benefits to determine the benefits for the

entire application and finally to validate these results by

determining directly the benefits for the entire

application. Since part of the paradigm includes

experimentation using the application in its whole or total

form, an application of limited complexity should be chosen.

Having chosen such an application, it will be necessary

to try to decompose it into discrete task areas. This is a procedure for which no formal rules exist. Common sense and

some understanding of what constitute "closures" [ 23] in the

psychological sense will have to guide this analysis. The

activities should be large enough that they may be analyzed

and the benefits from each determined, but they should also

be sufficiently limited in scope that they may be

experimentally manipulated reasonably well and that they may

be used as building blocks to characterize full

applications

Having chosen the task areas, or kernels, experiments

will be designed after Gold's model to determine the

benefits derived from performing each task on a model

29
graphics system as compared with doing it some other

(baseline) way. Experimental procedures employing two or

more sample groups and a control group are well understood

and should be followed. Statistical techniques such as

analysis of variance may be useful both in establishing the

level of benefits and validating the initial choice of the

groups.

The final validation of the approach comes after an

application is modeled by the task groups and the benefits

are predicted and verified by experimentation. This will

require an experiment which treats the entire application -- which we have been trying to avoid -- but this is necessary

a few times in order to validate the approach. If the

approach can be validated, experiments on that scale will

not be necessary again.

Studies such as this are but a first step in the

development of kernel problems for the assessment of benefit

values for particular systems. Many more studies, and

preferably a systematic project with the stated objective of

disaggregating the application-oriented functions of a graphics system and determining the benefits of each, are

required before a parametric approach to determining the

total benefits of an arbitrary system can be perfected.

7. Synthesis: Cost-Benefit Analysis

Cost-effectiveness analysis is valuable because it shows

how to best allocate limited resources so as to optimize

some specific performance objectives. However, such an

analysis is inadequate for making an investment decision.

Such decisions require that total system cost be compared

with the value of the benefits provided by the system. This

requires that acceptable levels of performance be defined,

so that a point on the cost-performance curve (corresponding

to a particular system) can be selected. Comparing total

costs with the benefits to be derived from the system

determines whether or not the system selected is

cost-beneficial. Unfortunately, the problem is rarely

expressed this way in the literature.

Cost-Value Technique

One approach which has been described in the literature

is the so-called "cost-value technique ."[ 19 ] This approach

describes a formal technique for comparing computer system

30
proposals which all offer slightly different features beyond

the basic mandatory requirements. The procedure involves

assigning a value to each of these "extras" (expressed in

terms of dollars) which are credited to the cost of each

proposal. The value of expansion potential, for example, is

assessed and subtracted from the costs of systems which

offer it. The approach is crude and subject to all the

perversions of subjective judgement, but at least it forces

the evaluators to think in terms of the benefits to be

gained from particular features.

Figure of Merit

In another vein, Dunn [8] has sought to define a figure

of merit for computer graphics systems which was free of

bias towards any particular type of configuration but which

is applicable to all types of configurations. However, Dunn

claimed that "useful measures for interactive graphics

systems are dependent upon subjective judgements." His

argument was as follows:

"Any measure that is devised must reflect more than

dollar costs and/or usefulness to the human and/or

satisfaction achieved via a particular configuration. An

interactive graphics system should and does function as

an 'amplifier' of the interactive human activities in

conjunction with the use of computers. The amplifying

function affects quality of results, quantity of work

accomplished, cost-effectiveness of methodology,

efficiency of effort, minimization of elapsed time for

work efforts, and so on. Amplification in this sense,

then, incorporates a variety of dollar costs along with

usefulness and satisfaction for the human user,"

He suggested a figure of merit called "degree of

amplification" or "amplification factor", symbolized as "A!"

and based upon four factors: productivity, degree of

interaction, extent of graphics capability and total system

direct dollar costs per graphics console.

The productivity factor, P, is defined as the change in

the rate of units of acceptable output relative to the same

function being performed via a non-graphics console. The

units of acceptable output are application-dependent (e.g.,

engineering drawings completed),

Graphics capability is a measurement factor that reflects

both ease of use of the graphics console and extent of load

on supporting computing systems. The graphics capability

31
factor, GC , is defined as the weighted sun of capability

indices for all desired features. The index values are if

a desired feature was not available, 1 if a desired feature

was available, and -1 if a required or critical feature was

not available. The weights indicated the relative

importance of each feature.

The degree of interaction is measured by the interaction

quotient, IQ, and is defined as a function of the ratio of

the effective data rate of the connecting channel to the

interaction data flow rate requirement. The IQ is expressed

as a weighted sum of quotient terms for each type of

interaction which could occur. Each term has the form

DR/(DO+DI), where DR is the effective data rate of the

channel and DO and DI are the output and input data rates,

respectively, at the graphics console required to request

and receive response for that type of interactive service

request.

Direct costs are the pro-rata portions of system

resources attributable to graphics support. These include

the fair share of all remote, interconnecting and local

hardware and software that is necessary to implement the

graphics svstem.

Finally, the amplification factor, A!, was defined as

follows

dl P (d2 10 + d3 GO

A! = d4 $/C

where P is the productivity factor; IQ is the interaction

quotient; GC is the graphics capability; $ is the total

system direct dollar costs; and C is the number of graphic

consoles that may be concurrently active in the system. The

di are the assigned weights that reflect the significance

attached by installations towards each of these factors. A!

is thus a weighted measure of the change in the rate of

productive output per dollar of total graphics system direct

cost per active console.

A Cost-Benefit Formulation

As has been indicated, the formulation of a cost-benefit

analysis must combine the results of cost-effectiveness

studies and benefit analysis. Cost-effectiveness analysis

identifies those systems with the greatest performance for a given level of investment, or the least costly system for a

32
given level of performance, depending on how the analysis is

performed. No guidance is provided, however, for the

selection of a system from a group of systems at different

levels of investment and performance, because there is no

way to judge if incremental improvements in performance are

worth their incremental costs. Benefit analysis addresses

this point by placing a value on improved levels of

performance. To perform cost-benefit analysis, the benefits

from different systems are systematically compared with

their respective costs in a search for the system with the

greatest net benefit (gross benefit minus cost).

When cost-effectiveness analyses are performed for a variety of systems at different levels of cost and

performance, an optimal price-performance function is

obtained. Specifically, this function is obtained either by

a) successively finding, for different levels of

investment, the best level of performance obtainable for

each; or

b) successively finding, for different levels of

performance, the least costly system which will provide

it.

Whichever procedure is followed, the result is an

"efficient frontier" of systems which are optimized at every

level both for cost as a function of performance and

performance as a function of cost. This frontier may be

expressed either as

C = fKP) (1)

or

P = f2(C) (2)

A similar frontier may be obtained from benefit analysis,

representing the maximum level of benefits possible for any

level of investment. The derivation is easier to visualize

for the curve expressing benefit as a function of

performance. To obtain this function, the benefits for

systems SELECTED FROM THE EFFICIENT PRICE-PERFORMANCE

FRONTIER at different levels of performance are successively

determined. The result is a function giving the maximum

benefits possible for each level of performance. This may

be expressed as

B = f3(P) (3)

33
Cost-benefit analysis seeks the system with the maximum

net benefit (the maximum of benefits minus costs). Thus,

the problem is to maximize

NB = B - C = f 1 (P) - f3(P) (4)

Typical curves for the functions under consideration are

illustrated in figure 11. Very little must be assumed about

either function to assure that a solution is possible. The

benefit function must exceed the cost function at some point

or the system will never even be considered for use. Both

functions are also assumed to be asymptotic, for the

following reasons:

1) The cost function is asvmptotic in the sense that

at some point, successive increments of investment

provide successively smaller increments of Derformance.

Economists would call this an illustration of the

decreasing marginal efficiency of capital. In simple

terras, as performance is improved, it becomes more and

more costly to provide still better performance.

2) The benefit function is asymptotic in the sense

that at some point, successive increments of Derformance

will provide successively smaller increments of benefit.

Economists refer to diminishing returns; in simple terms,

the more of a good or service is provided, the less is

the value of still more of that product.

With these justifications for drawing the curves as

illustrated in figure 11, an analytic solution may be

obtained by the use of calculus. The desired maximum occurs

when the first derivative of the net benefit function is

zero

= d(NB)/dP = d(f1(P))/dP - d(f3(P))/dP (5)

Subject to NB > 0.

Reformulation

Actually, the cost-benefit formulation illustrated thus

far has been somewhat simplistic in that it ignores

utilization level. Consideration of utilization level is of

primary importance because of its effect on the cost

function -- increased utilization results in lower average

costs. Utilization level may be of secondary importance

because of its effect on benefits, e.g., the effects of

scale.

34
Costs

Benefits

B-f 3

(P)

/ Net Benefit

NB = B-C

= f 3 IP)- f

1 (P)

r Loss

Performance

Figure 11. Graphical Presentation of Cost-Benefit Formulation

35
In the reformulation of cost-benefit analysis, costs and

benefits are recognized to be functions of both performance

and utilization level

C = f1(P,U) (6)

B = f2(P,U) (7)

If the expected utilization level is known, it can be

held constant for the analysis and the previous formulation

is valid. If it is desired to permit utilization level to

vary in the analysis, the graphical formulation can only be

expressed as a three-dimensional graph. A solution is

insured by arguments similar to those used previously. As

utilization level increases incrementally, benefits will

also increase, but at some point will increase less rapidly;

likewise, costs will decrease, but at some point will

decrease less rapidly. Thus, the effect of utilization

level is asymptotic both for benefits and for cost.

The analytic solution to this formulation requires the

solution of a simultaneous set of equations obtained by

setting the partial derivatives equal to zero:

= d(NB)/dP = Mf1(P,U))/dP (8)

= d(NB)/dU = d(f2(P,U) )/dU (9)

Subject to NB > 0.

Comparison With Previous Approaches

The cost-value technique may be viewed as an attempt to

adjust for differences in system performances by modifying

system cost. The approach is in keeping with the spirit of

cost-benefit analysis; it is just too simplistic to be

useful in a complex case.

Dunn's approach may be reduced to a form closer to the

formulation suggested for cost-benefit analysis.

Performance is actually a function of both the interaction

quotient and graphic capability. Thus, the figure of merit

reduces to a performance function over a cost function. All

that is lacking is to relate performance to value, but this

is facilitated by the way in which performance is expressed.

By expressing performance in terms of productivity gains, it

is easy to determine value by considering the value of

performance achieved the old way. The unit benefits of the

36
new system are just the unit value of output done the old

way times the rate of increase of productivity.

The real shortcoming off Dunn's approach for the purposes

of cost-benefit analysis is that it starts with the data it

should be seeking. Data such as the graphics capability

factor, the various weights, and in particular, the

productivity factor, are unknown in advance. The crux of

the problem is to determine the productivity gains from

using a particular system. Once this is known, and the cost

for using the system is determined, then the cost-benefit

ratio for the system can be determined.

8. Conclusions

The quantitative analysis of computer graphics systems is

performed at a very crude level of sophistication today.

This is partially because it has a basis in the evaluation

of computer systems in general, which is still a very

inexact science, and partially because it deals with a system involving humans, whose performance is always

difficult to assess.

As we have explained, performance evaluation of computer

systems in general and graphics systems in particular can be

done reasonably well. Cost analysis can be done well also;

consequently so can cost-effectiveness analysis. On the

other hand, benefit evaluation cannot vet be done very well,

so neither can cost-benefit analysis.

Cost-effectiveness analysis is necessary but not

sufficient for making investment decisions. Cost-benefit

analysis reauires the output of cost-effectiveness analysis,

since it requires optimal levels of performance for given

levels of investment. What must then be added are results

from benefit analysis to determine the optimal level of

investment. However, procedures for performing benefit

analysis are not well-developed. An experimental approach

was outlined in this paper which applies the same types of

tools to benefit analysis which have been applied to

performance analysis.

Still, cost-effectiveness analysis will likely continue

to be the principal tool for choosing among alternative

systems on the same order of cost or of performance.

Cost-benefit analysis is too gross a tool to detect small

differences between systems. Cost-benefit analysis will be

37
increasingly used, however, to place the "go or no-go"

investment decision on a more analytic basis.

Acknowledgments

This work was originally motivated by Dr. Guy Black of

the School of Government and Business Administration, George

Washington University. The continued assistance- through

many manuscript drafts of Dr. James Dei Rossi of the

Institute for Computer Sciences and Technology, National

Bureau of Standards, is greatly appreciated.

Figures 5 through 9 are copyrighted by the GML

Corporation and are used by permission.

BIBLIOGRAPHY

1 Adams, Jeanne and Cohen, Leonard. "Time-sharing vs.

instant batch processing." <Computers and Automation>,

March 1969, pp. 33-34.

2 Auerbach Info, Inc. "Design and applications of

automated display systems." <Auerbach Standard ED?

Reports>, Special Report, September 1968

3 Boehm, B. W., Seven, M. J. and R. A. Watson.

"Interactive problem-solving -- An experimental study of

'lockout' effects." <Proc> Spring Joint Computer

Conference, 1971, pp. 205-210.

4 Calingaert, Peter. "System performance evaluation:

survey and appraisal," Communications of the ACM>, 10:1

(January 1967), pp. 12-17.

5 Carlisle, James H. "Comparing behavior at various

computer display consoles in time-shared legal information."

RAND Corporation Report P-4448, September 1970.

6 Cislo, Robert A. "Graphic system performance

evaluation." <Proc> 1972 ACM National Conference, Dp.

432-442.
7 Cotton, Ira W. "Network graphic attention handling."

<Proc> Online 12 Symposium, Brunei University, September

1972.

8 Dunn, Robert M. "Computer graphics: capabilities and

usefulness." <Proc> SHARE XL (Denver, Colorado, 7 March

1973).

9 Erikson, W. J. "A pilot study of interactive versus

noninteractive debugging." System Development Corporation,

TM-3296, December 13, 1966.

10 Foley, James D. <0ptimum design of computer driven

display systems>. Doctoral Dissertation, University of

Michigan, March 1969.

11 Foley, James T. "An approach to the optimum design

of computer graphics systems." "(Communications of the ACM>

,

14:6 (June 1971), op. 360-390.

12 Gold, Michael M. "Methocolorv for evaluating

time-shared computer usa^e." Unpublished Ph.D.

dissertation, 1967, Massachusetts Institute of Technology,

Alfred P. Sloan School of Management.

13 Gold, Michael M. "Timesharing and batch-processinf?

an experimental comparison of their values in a problem-solving situation." Communications of the ACM>,

12:5 (May 1969), pp. 249-259.

14 Grant, E. Eugene and Harold Sackman. "An exploratory

investigation of programmer performance under on-line and

off-line conditions." <IEEE Trans, on Human Factors in

Electronics>, March 1967, pp. 33-48.

15 Hillegass, John R. "Systematic technioues for

computer evaluation and selection." <Management Services>,

July-August 1969, pp. 35-38.

16 Jacobs, Lesley D. "CRT graphics consoles - an aid to

selection." Rome Air Development Center Technical Report

71-61, November 1971.

17 Johnson, R. R. "Needed: a measure for measure,"

<Datamation> , December 15, 1970, pp. 22-30.

18 Joslin, Edward 0. "Costing the system design

alternatives." <Data Management^ April 1971, op. 23-27.

19 Joslin, Edward 0. and Martin J. Mullin. "Cost-value

39
technique for evaluation of computer system proposals."

<Proc> Spring Joint Computer Conference, 1964, pp. 367-381.

20 LaBolle, V. "Development of equations for estimating

the costs of computer program production." System

Development Corporation, June 1 Q 66 (Published as U. S. Air

Force Electronic System Division Technical Report No.

66-350),

21 Lampson, Butler W. "A critique of 'An exploratory

investigation of programmer performance under on-line and

off-line conditions.'" <IEEE Trans. Human Factors in

Electronics> , pp. 48-51.

22 Meyer, T. H. and I. E. Sutherland. "On the design of

display processors." <Comm. AC.M>, 11:6 (June, 1968), Dp.

410-414.

23 Miller, Robert B. "Response time in man-computer

conversational transactions," <Proc> Fall Joint Computer

Conference, 1968, pp. 267-277.

24 Nelson, E. A. "Management handbook for the

estimation of comouter programming costs." System

Development Corporation, Technical Memorandum No. 3225,

March 20, 1967.

25 Poole, Harry H. "Comouter display system tradeoffs."

In <Computer Graphics: Techniques and Applications> , Eds.:

Parslow, Prowse & Green. New York: Plenum Press, 1969.

26 Prince, M, David. interactive Graphics for

Computer-Aided Design>. Reading, Massachusetts:

Addison-Wesley , 1971.

27 Sackman, Harold. <Man-Computer Problem Solving>.

Princeton, N. J.: Auerbach Publishers Inc., 1970.

28 Sackman, Harold. <Mass Information Utilities and

Social Excellences Princeton, N. J.: Auerbach Publishers,

1972.

29 Schatzoff, M. , Tsao, R. and R. Wiig. "An

experimental comparison of time sharing and batch

processing." <CACM> , May 1967, pp. 261-65.

30 Smith, J. Meredith. "A review and comparison of

certain methods of computer performance evaluation."

<Computer Bulletin>, 12:1 (May 1868), op. 13-18.

40
31 Thomas, E. M. "System considerations for graphic

data processing." <Computers and Automation>, November,

1967, pp. 16-19.

32 Ward, John E. "Systems engineering problems in

computer-driven CRT displays for man-machine communication."

<IEEE Trans. System Science and Cybernetics> , 3:1 (June

1967), pp. 47-54.

33 Yule, A. "Human response time in a graphic

environment." <Computer Bull.>, June 1972, pp. 304-305.

41

NBS-114A (REV. 7-73)

U.S. DEPT. OF COMM. BIBLIOGRAPHIC DATA

SHEET

1. PUBLIC ATION OR REPORT NO. NBS TN-8262. Gov't Accession

No.

3. Recipient's Accession No.

4. TITLE AND SUBTITLE

Cost-Benefit Analysis of Computer Graphics Systems

5. Publication Date April 1974

6. Performing Organization Code

7. AUTHORS Ira W. Cotton 8. Performing Organ. Report No.

9. PERFORMING ORGANIZATION NAME AND ADDRESS

NATIONAL BUREAU OF STANDARDS

DEPARTMENT OF COMMERCE

WASHINGTON, D.C. 20234

10. Project/Task/Work Unit No. 6505373

11. Contract/Grant No. 73-90042

12. Sponsoring Organization Name and Complete Address (Street, City, State, ZIP)

U. S. Army Electronics Command

AMSEL-GG-OD

Fort Monmouth, N. J. 07703

13. Type of Report & Period Covered

Final 7/72-6/73

14. Sponsoring Agency Code

15. SUPPLEMENTARY NOTES

16. ABSTRACT (A 200-word or less factual summary ol most significant information. If document includes a significant

bibliography or literature survey, mention it here.)

This report assesses the state-of-the-art in cost benefit analyses of

computer graphics systems and suggests an approach for developing improved

methodology. Cost-benefit analyses are distinguished from analyses of

system performance in that the latter are directed at optimizing system

performance at a given level of investment, while the former are directed

at justifying the investment itself.

Computer graphic system design alternatives are first outlined. Then

methods of analyzing the performance and costs of computer systems in

general and graphic systems in particular are discussed. With this

information it is shown how cost-effectiveness analyses may be performed.

The next crucial step is to conduct benefit analysis, an ill-defined art.

The results of benefit analysis must be combined with cost-effectiveness

analysis in order to perform the desired cost-benefit analysis.

An experimental methodology is suggested for better performing benefit

analyses of computer graphic systems. A more rigorous formulation of the

cost-benefit procedure is then outlined. No attempt is made in this report

to actually perform such an analysis.

17. KEY WORDS (six to twelve entries; alphabetical order; capitalize only the first letter of the first key word unless a proper name; separated by semicolons)

Computer graphics; cost-benefit analysis; cost-effectiveness; economics;

performance evaluation

18. AVAILABILITY $T Unlimited

[ For Official Distribution. Do Not Release to NTIS

I Order From Sup. of Doc, U.S. Government Printing Office Washington, D.C. 20402, SD Cat. No. C13 .46:826

I Order From National Technical Information Service (NTIS)

Springfield, Virginia 22151

19. SECURITY CLASS

(THIS REPORT)

UNCLASSIFIED

20. SECURITY CLASS

(THIS PAGE)

UNCLASSIFIED

21. NO. OF PAGES

47

22. Price 90 cents

USCOMM-DC 29042-P74

NBS TECHNICAL PUBLICATIONS

PERIODICALS JOURNAL OF RESEARCH reports National Bureau of Standards research and development in physics, mathematics, and chemistry. Comprehensive

scientific papers give complete details of the work,

including laboratory data, experimental procedures,

and theoretical and mathematical analyses. Illustrated with photographs, drawings, and charts. Includes

listings of other NBS papers as issued.

Published in two sections, available separately:

• Physics and Chemistry (Section A)

Papers of interest primarily to scientists working in these fields. This section covers a broad range of

physical and chemical research, with major emphasis

on standards of physical measurement, fundamental

constants, and properties of matter. Issued six times

a year. Annual subscription: Domestic, $17.00; Foreign. $21.25.

• Mathematical Sciences (Section B)

Studies and compilations designed mainly for the mathematician and theoretical physicist. Topics in mathematical statistics, theory of experiment design,

numerical analysis, theoretical physics and chemistry,

logical design and programming of computers and

computer systems. Short numerical tables. Issued quarterly. Annual subscription: Domestic, $9.00; Foreign,

$11.25.

DIMENSIONS, NBS

The best single source of information concerning the Bureau's measurement, research, developmental, cooperative, and publication activities, this monthly

publication is designed for the layman and also for the industry-oriented individual whose daily work

involves intimate contact with science and technology —for engineers, chemists, physicists, research managers, product-development managers, and company

executives. Annual subscription: Domestic, $6.50; Foreign, $8.25.

NONPERIODICALS Applied Mathematics Series. Mathematical tables, manuals, and studies. Building Science Series. Research results, test methods, and performance criteria of building materials, components, systems, and structures. Handbooks. Recommended codes of engineering

and industrial practice (including safety codes) de- veloped in cooperation with interested industries, professional organizations, and regulatory bodies. Special Publications. Proceedings of NBS conferences, bibliographies, annual reports, wall charts, pamphlets, etc. Monographs. Major contributions to the technical

literature on various subjects related to the Bureau's

scientific and technical activities. National Standard Reference Data Series.

NSRDS provides quantitative data on the physical

and chemical properties of materials, compiled from

the world's literature and critically evaluated. Product Standards. Provide requirements for sizes, types, quality, and methods for testing various industrial products. , These standards are developed co- operatively with interested Government and industry groups and provide the basis for common understanding of product characteristics for both buyers and

sellers. Their use is voluntary.

Technical Notes. This series consists of communications and reports (covering both other-agency and

NBS-sponsored work) of limited or transitory interest. Federal Information Processing Standards

Publications. This series is the official publication

within the Federal Government for information on

standards adopted and promulgated under the Public Law 89-306, and Bureau of the Budget Circular A-86

entitled. Standardization of Data Elements and Codes

in Data Systems.

Consumer Information Series. Practical information, based on NBS research and experience, covering areas of interest to the consumer. Easily understandable language and illustrations provide useful background knowledge for shopping in today's tech- nological marketplace.

BIBLIOGRAPHIC SUBSCRIPTION SERVICES

The following current-awareness and literature-survey bibliographies are issued periodically by the Bureau

Cryogenic Data Center Current Awareness Service (Publications and Reports of Interest in Cryogenics).

A literature survey issued weekly. Annual subscription: Domestic, $20.00; foreign, $25.00.

Liquefied Natural Gas. A literature survey issued quarterly. Annual subscription: $20.00. Superconducting Devices and Materials. A literature survey issued quarterly. Annual subscription: $20.00. Send subscription orders and remittances for the preceding bibliographic services to the U.S. Department

of Commerce, National Technical Information Service, Springfield, Va. 22151. Electromagnetic Metrology Current Awareness Service (Abstracts of Selected Articles on Measurement

Techniques and Standards of Electromagnetic Quantities from D-C to Millimeter-Wave Frequencies) . Issued monthly. Annual subscription: $100.00 (Special rates for multi-subscriptions) . Send subscription order and

remittance to the Electromagnetic Metrology Information Center, Electromagnetics Division, National Bureau

of Standards, Boulder, Colo. 80302. Order NBS publications (except Bibliographic Subscription Services) from: Superintendent of Documents, Government Printing Office, Washington, D.C. 20402.
U.S. DEPARTMENT OF COMMERCE

National Bureau of Standards

Washington. D.C. 20234

OFFICIAL BUSINESS

Penalty for Private Use. $300

POSTAGE AND FEES PAID

U.S. DEPARTMENT OF COMMERCE COM—215

U.!

O^T'O/v