col

A111D3 Qfi^aBS

NATL INST OF STANDARDS & TECH R.I.C.

All 103089825

Mamrak, Sandra A/A methodologv for the s QC100 .U57 NO.500-, 44. 1979 C 1 NBS-PUB

A Methodology for the

Selection of

Interactive Computer Services

500-44

;9

Co NBS Special Publication 500-44

U.S. DEPARTMENT OF COMMERCE

National Bureau of Standards
NATIONAL BUREAU OF STANDARDS

The National Bureau of Standards' was established by an act of Congress March 3, 1901. The

Bureau's overall goal is to strengthen and advance the Nation's science and technology and

facilitate their effective application for public benefit. To this end, the Bureau conducts

research and provides: (1) a basis for the Nation's physical measurement system, (2) scientific and technological services for industry and government, (3) a technical basis for equity in trade, and (4) technical services to promote public safety. The Bureau's technical work is performed by the National Measurement Laboratory, the National Engineering Laboratory,

and the Institute for Computer Sciences and Technology.

THE NATIONAL MEASUREMENT LABORATORY provides the national system of

physical and chemical and materials measurement; coordinates the system with measurement

systems of other nations and furnishes essential services leading to accurate and uniform

physical and chemical measurement throughout the Nation's scientific community, industry,

and commerce; conducts materials research leading to improved methods of measurement,

standards, and data on the properties of materials needed by industry, commerce, educational

institutions, and Government; provides advisory and research services to other Government

Agencies; develops, produces, and distributes Standard Reference Materials; and provides

calibration services. The Laboratory consists of the following centers:

Absolute Physical Quantities^ — Radiation Research — Thermodynamics and

Molecular Science — Analytical Chemistry — Materials Science.

THE NATIONAL ENGINEERING LABORATORY provides technology and technical

services to users in the public and private sectors to address national needs and to solve national problems in the public interest; conducts research in engineering and applied science

in support of objectives in these efforts; builds and maintains competence in the necessary

disciplines required to carry out this research and technical service; develops engineering data and measurement capabilities; provides engineering measurement traceability services; develops test methods and proposes engineering standards and code changes; develops and

proposes new engineering practices; and develops and improves mechanisms to transfer

results of its research to the utlimate user. The Laboratory consists of the following centers:

Applied Mathematics — Electronics and Electrical Engineering^ — Mechanical

Engineering and Process Technology' — Building Technology — Fire Research —

Consumer Product Technology — Field Methods.

THE INSTITUTE FOR COMPUTER SCIENCES AND TECHNOLOGY conducts

research and provides scientific and technical services to aid Federal Agencies in the selection, acquisition, application, and use of computer technology to improve effectiveness and

economy in Government operations in accordance with Public Law 89-306 (40 U.S.C. 759), relevant Executive Orders, and other directives; carries out this mission by managing the Federal Information Processing Standards Program, developing Federal ADP standards

guidelines, and managing Federal participation in ADP voluntary standardization activities; provides scientific and technological advisory services and assistance to Federal Agencies; and

provides the technical foundation for computer-related policies of the Federal Government.

The Institute consists of the following divisions:

Systems and Software — Computer Systems Engineering — Information Technology.

'Headquarters and Laboratories at Gaithersburg, Maryland, unless otherwise noted; mailing address Washington, D.C. 20234. 'Some divisions within the center are located at Boulder, Colorado, 80303.

The National Bureau of Standards was reorganized, effective April 9, 1978.

For sale by the Superintendent of Documents, U.S. Government Printing OflBce, Washington, D.C. 20402

Stock No. 003-003-02018 Price $2.50

(Add 25 percent additional for other than U.S. mailing).
FEB 2 1 1979

COMPUTER SCIENCE & TECHNOLOGY: ^^^r^

A Methodology for the Selection of no

Interactive Computer Services ^^^^]

Sandra A. Mamrak

Paul D. Amer

Institute for Computer Sciences

and Technology

National Bureau of Standards

Washington, D.C. 20234

U.S. DEPARTMENT OF COMMERCE, Juanita M. Kreps, Secretary

Jordan J. Baruch, Assistant Secretary for Science and Technology

NATIONAL BUREAU OF STANDARDS, Ernest Ambler, Director

Issued January 1979
Reports on Computer Science and Technology

The National Bureau of Standards has a special responsibility within the Federal

Government for computer science and technology activities. The programs of the

NBS Institute for Computer Sciences and Technology are designed to provide ADP

standards, guidelines, and technical advisory services to improve the effectiveness of

computer utilization in the Federal sector, and to perform appropriate research and

development efforts as foundation for such activities and programs. This publication

series will report these NBS efforts to the Federal computer community as well as to interested specialists in the academic and private sectors. Those wishing to receive

notices of publications in this series should complete and return the form at the end

of this publication.

National Bureau of Standards Special Publication 500-44

Nat. Bur. Stand. (U.S.) Spec. Publ. 500-44, 82 pages (Jan. 1979) CODEN: XNBSAV

Library of Congress Catalog Card Number 78-600149

U.S. GOVERNMENT PRINTING OFFICE

WASHINGTON: 1979

For sale by the Superintendent of Documents, U.S. Government Printing OflBce, Washington, D.C. 20402

Stock No. 003-003-02018 Price $2.50

(Add 25 percent additional for other than U.S. mailing).
A Methodology for the

Selection of Interactive Computer Services

CONTENTS

Page

1.0 INTRODUCTION 1 1.1 A Computer Selection Model 2

1.2 Classification of Performance Criteria .... 4

1.3 Application of Performance Criteria 6

2.0 THE COMPARISON METHODOLOGY 7

2.1 Comparison Criteria 8

2.2 A User Scenario 9

2.2.1 Developing a Representative Scenario . . 9

2.2.2 Constraints on Scenario Development . . 10

2.3 Script Generation 11

2.4 Data Collection and Analysis 11 2.4.1 Selection of Services Satisfying

a Mandatory Requirement: Based

on Proportions 13

2.4.2 Selection of the Best Service .... 16

2.4.2.1 Selection Based on Means . . 16

2.4.2.2 Selection Based on Percentiles 18

2.4.2.3 Selection Based on Proportions 19

3.0 A CASE STUDY 20

3.1 Scenario and Scripts 20

3.2 Data Collection and Analysis 22

3.2.1 Measurement of Computer Services ... 23

3.2.2 Assessing Correlation 25

3.3 Comparison Results 27

3.3.1 Selection of Services Satisfying a Mandatory Requirement: Based on Proportions 29

3.3.2 Selection of the Best Service .... 30

3.3.2.1 Selection Based on Means . . 30

3.3.2.2 Selection Based on Percentiles 32

3.3.2.3 Selection Based on Proportions 33

4.0 DISCUSSION AND CONSLUSIONS 34

4.1 Estimated Cost of Using the Methodology ... 34

4.2 Appl icabil ity of Techniques 36

4.3 General Recommendations 37

5.0 ACKNOWLEDGMENTS .

" 38

6.0 REFERENCES 39

i i i
FIGURES

Page

Figure

1. Computer Selection Model 3

2. Hardware/Software Configuration for the Case Study . . 24

TABLES

Table

1. Examples of Performance Criteria 5

2. Summary of Ranking and Selection Terminology .... 14

3. Scenario for the Case Study . 21

4. Estimated Correlation Coefficients . 26

5. Sample Statistics from the Case Study 28

6. Example Kolmogorov-Smirnov Test for Normality:

Service C Cost Data 31

7. Estimated Cost 35

APPENDICES

Appendix

A. Correlation 43

B. Tables 45

C. Figures 76

iv
A Methodology for the

Selection of Interactive Computer Services

Sandra A. Mamrak

Paul D. Amer

ABSTRACT

This publication addresses the comparison and

selection of remote access interactive computer

services. The comparison methodology presented relies

principally on the statistical analysis of measurement

data obtained from the interaction between a computer

service and a user. One of the most important

properties of the methodology is that it incorporates

confidence statements about the probability of having

made a correct selection. Experimental data are

presented to illustrate an application of the

methodology, and serve as a basis for a discussion of

the cost and appropriateness of using the methodology

in various procurement efforts.

Key words: comparison of computers; computer

measurement; computer selection; computer services;

ranking and selection; selection methodology.

1.0 INTRODUCTION

The comparison of services delivered by a computer

to a terminal user is a frequent, important and

essential effort for those involved in selecting from

among alternative computer services. This is

especially true for large industrial and government

installations engaged in procurement efforts which may

involve millions of dollars [FIP57, TSP77]. The

comparison and selection process is a complicated one,

based on various criteria. Some of the criteria are

measurable (such as system response time) , and some are

not (such as ease of system use and coherence of system

documentation). For those criteria which are

measurable, comparison requires collecting and

analyzing relevant performance measurements for the

various computer services under consideration, in order

Note: Reference to any commercial products in this

report is for the purpose of identification only and

does not imply endorsement by the National Bureau of

Stand ards

.

1
to choose the best one. A simple rule that analysts

have used in the past for choosing the best service is

the same one that a sound statistical methodology would

dictate: pick the service that yields the best

measurements, on the average. However, if test

conditions are such that the measurement error is

relatively large compared to the difference between the

services being compared, then the chance of selecting

the best service by comparing measurement data is no

better than the chance of picking it by random drawing

[YOU59]

.

The comparison of computer services, therefore,

requires a methodology designed to lead to the

selection of the best computer service, and to provide

control of the probability of having made a correct

choice. Methodologies dictated by classical

statistical designs (which are inappropriate for

computer service selection) lead to regression analyses

of the data, employing either analysis of variance or

curve-fitting techniques [DAV63]. The questions that

can be answered using such methodologies are of the

type "Is the performance of several alternative systems

the same, i.e. are the distributions of performance

measurements identical from a statistical point of

view?" or "How does one particular system performance

parameter depend upon the other system parameters?".

In most computer comparison efforts, however, these

questions are not appropriate. The question of real

interest is: "Which computer service is the best?" or

"How do the services rank from best to worst?". It is

for problems of this type that statistical ranking and

selection procedures were developed [GIB77]. These

procedures are applied in the selection methodology

described below.

1.1 A Computer Selection Model

A model of the computer service selection process

is presented in Figure 1. This model will be used to

specify those phases of the selection process v;hich

this publication addresses. The model is not intended

to represent the entire procurement process. Readers

are referred to FIP42 and J0S77 for a broader view of

the competitive procurement environment.

Figure 1 shows three phases of the selection

process that lead progressively from the set of all

computer services under consideration to the isolation

of the best one. Each phase involves evaluation of the

services with respect to different classes of

performance criteria. At each phase, those services

2
ALL ALTERNATIVE COMPUTER SERVICES

0©©© •..©

^ ^ ^ ^ ^

©

BEST SERVICE

Figure 1. Computer Selection Model [AME78a]

3
which fail to satisfy the respective performance

criteria are eliminated. The kinds of performance

criteria that are evaluated are described briefly in

Section 1.2. Those phases of the selection process for

which the statistical methodology presented here are

applicable are then specified in Section 1.3.

1.2 Classification Of Performance Criteria

In choosing the best computer service from several

alternative computer services, performance criteria

must be defined which describe what is meant by best.

These criteria can be divided into those for which no

empirical measurement is necessary and those whose

values are derived from actual system measurements.

For example, evaluating system documentation and amount

of main memory does not require measurement collecting

activity, whereas evaluating system turnaround time and

response time clearly requires measurement.

Performance criteria can be further divided into

those which are mandatory and those v/hich are

desirable. A mandatory criterion is defined to be any

performance requirement which must be satisfied by the

computer services being considered for selection.

Desirable criteria, on the other hand, are those which

are not absolute requirements for system acceptance,

but which would make the implementation of the

purchaser's work easier. Therefore, if a given

computer service does not include some desirable

features, it would continue to be considered for

selection, but the lack of each desirable feature would

perhaps invoke some penalty on the respective computer

service [JOS77].

Based on the two characteristics described above,

performance criteria can be classified as: Mandatory

Nonmoasur able (MN) , Mandatory Measurable (MM)

,

Desirable Nonmeasur able (DN), and Desirable Measurable

(DM). Examples of each class of criteria are provided

in Table 1.

Clearly, there are a large number of performance

criteria which may be used as a basis for computer

comparison. (See ABR77b, FER78, and GRU75 for

examples.) Ultimately the decision about how many and

which performance factors are most important in an

evaluation is a policy issue. One user may view system

reliability as the over-riding consideration in a system selection effort while another might view cost

and response time for short edit commands as the most

important factors. Usually some combination of various

4
Table 1

Examples of Performance Criteria

Type Example

Mandatory

Nonmeasurable

1. The system must be fully delivered and operational

no later than September 1, 1979.

2. Timesharing service must include FORTRAN, Basic, Lisp,

SNOBOL and editing facilities.

Mandatory

Measurable

1. The mean-time-to-failure for a specific one month

period must be greater than 4 hours.

2. 95% of all trivial command response times must be

less than 1 second.

Desirable

Nonmeasurable

1. It is desirable that the system include Pascal and

COBOL facilities.

2. It is desired that the system provide a text editing

capability.

Desirable

Measurable

1. It is desired that the system provide a mean turnaround time for the benchmark run of 5 minutes

or less.

2. It is desired that 95% of all trivial command response

times be 0.5 seconds or less.

5
measures of system response time and system costs is

used in the selection process.

1.3 Application Of Performance Criteria

Figure 1 illustrates a sequence in which the

classes of performance criteria should be applied in

the process of choosing the best computer service.

This sequence is composed of three phases. Phase I involves the application of MN criteria. The phase is

easily managed since each computer service either does

or does not have the required characteristic. All of

those systems which do not satisfy the MN criteria are

el im ina ted . Phase II involves the application of MM criteria.

In general for each MM criterion, performance

measurements are gathered from every computer service

and a decision is made as to whether or not the

criterion is satisfied. Failure to satisfy a single MM

criterion results in a service's elimination.

Finally, determination of the best alternative is

made in phase III. This phase is separated into two

parts, phase IIIA for the application of DN criteria

and phase IIIB for the application of DM criteria.

(Note: It is not implied that DN and DM criteria can

necessarily be applied separately or in any particular

order.) The information required in phase IIIB is

similar to the information required in phase II in that

it can only be obtained by system measurement. Data is

collected from each alternative computer service being

evaluated and compared. On the basis of relative

performance, one system is selected as the best.

In both phases II and IIIB, comparison requires

collecting and analyzing relevant performance

measurements for the various computer service

alternatives under consideration: in phase II, to

select those satisfying certain mandatory performance

requirements, and in phase IIIB, to select the best

remaining one. It is for these two phases that the

selection methodology presented in this publication is

appl icable

.

It should be noted that the selection methodology

presented here is capable of specifying the best system

only with respect to a one performance criterion at a time. For example, it will guarantee, with a certain

maximum probability of error, selection of the best

6
system with respect to script turnaround time, or

selection of the best system with respect to response

time for short edit commands, and so on. The task of

integrating the information provided by the methodology

is left to the analyst. Typically, a simple policy

which defines the overall best computer service as that

one which ranks best v^ith respect to the greatest

number of single criteria is satisfactory [LEE77]. A

weighted function of several criteria could also be

used for final selection, if the function was based on

information regarding which computer service was the

best with respect to each criterion, rather than on a relative ranking of the computer services from best to

wo r St

.

The main components of the selection methodology

are

:

1. determination of performance criteria which

will form the basis of a service comparison.

2. development of a representative of a

3. translation of a scripts executable

under test, and

user scenario that is

projected workload,

scenario into individual

on the respective systems

4. collection and analysis of the data required

for a comparison.

The remainder of this publication is devoted to

discussing each component in turn. The main emphasis

is on discussion of the fourth component, for which

procedures are presented based on statistical ranking

and selection theory. The details of executing each

step of the comparison methodology is clarified by

reference to an actual case study of computer service

selection. An important consideration throughout the

execution and discussion of the case study is the cost

incurred in computer service selection.

2.0 THE COMPARISON METHODOLOGY

Typically, a comparison case study to select a computer service consists of running an identical set

of tasks (scripts) on all systems under test during

specified periods of the day, and collecting

performance and cost measurements during script

7
execution. Scripts, discussed in more detail below,

may be driven and the performance measurements made

under human or automatic control (see RTE75 for a discussion of automatic script drivers and ABR77a,

ROS76, WAT76 for a discussion of an automatic

measurement machine) . The computer services are then

evaluated based on the measurement results and the best

one is selected.

Performance and cost data are usually collected in

an uncontrolled environment. That is, a user exercises

no control over the software, hardware or workload of a computer service during the test periods. This mode of

testing is in accord with the actual mode of

utilization of a computer service once it has been

selected. Users do not have, nor do they desire, such

control when they purchase computer s'ervices (in

contrast to computer systems) . However, they do demand

a guaranteed level of service at their terminal

interface and that is precisely what is measured.

2.1 Comparison Criteria

There are a large number of measures which may be

used to describe the demands and needs of a user at an

interactive terminal. Abrams and Treu rABR77b] have

tabulated more than fifty of these measures, describing

the time, length, rate and ratios of user-computer

interactive behavior. Three measures are likely to be

of primary interest in an evaluation and selection of

computer services: response time, turnaround time and

cost

.

The importance of response time is generally

acknowledged. A preferred definition of response time

is the elapsed time from the last user keystroke (which

terminates a service request) until the first

meaningful system character is displayed at a user's

terminal rFIP571. Response times for various

individual commands, and for various classes of

commands are useful for service comparison. Individual

commands of interest include compile and

load-and-execute requests. Classes of commands include

short editor commands, status inquiry commands, and

database queries.

Interactive turnaround time is measured as the

elapsed time required to complete a given sequence of

tasks in an interactive mode. As compared to response

time, turnaround time requires fewer measurements over

a longer interval. In general, these measurements are

less expensive to obtain. This advantage, however, is

8
offset by the increased number of tests required to

obtain the necessary number of data points [FIP57]

.

Sporadicity problems, system verbosity, and

complexity of the command structures or syntax of a computer service are all reflected in a turnaround time

measure. Two services which yield similar response

time measurements for a given scenario may yield

dissimilar turnaround times for that same scenario.

The cost of a computer service, if not the primary

selection criterion, is usually high on a list of

important considerations upon which a selection is

based. The costs incurred for computer services are

typically incorporated into a charging a]gorithm that

is based on the various resources used during an

interactive session, such as CPU time, connect time,

and so on.

2.2 A User Scenario

A "scenario" is a functional description of an

interactive benchmark which is to be run and measured

on each service being compared [WAT77] . There are

three major workload types which describe remote access

interactive computing requirements. They are 1)

interactive program development including use of

compilers, editors, etc., 2) transaction processing,

and 3) remote job entry [CON76]. A scenario being

developed for a computer service comparison study may

or may not contain elements representing all three

types of computing depending on a comparison study's

objectives. The number and kind of interactions of

each type which are included depend both upon the

representativeness which is desired and upon the

statistical methodology for the comparison study.

2.2.1 Developing A Representative Scenario -

The degree to which a scenario is required to

represent an actual projected workload is a critical

issue in an evaluation and selection process.

Normally, a scenario is required to closely reflect the

functional requirements of a real workload. The

identification and quantification of significant real

workload characteristics, and the generation of a scenario based on these characteristics, is a major

undertaking which has been reported upon elsewhere in

the literature rcR074, NOL74, WRI76] and will not be

9
discussed here.

2.2.2 Constraints On Scenario Development -

The statistical ranking and selection procedures

used in the data analysis phase of the methodology

described in this publication, which are described in

more detail below, typically require 1) that a relatively large number of measurements (about one

hundred) be made for each comparison criterion and 2)

that the measurements be independent (data independence

is a statistical concept which deals with how

consecutive data points are related). These two

requirements, described below, introduce constraints on

scenario development.

Since a large number of measurements may be

required, and for some comparison measures like

turnaround time, only one measurement can be made per

scenario run, the scenario should be relatively short.

On the other hand, a representativeness requirement may

dictate a relatively long scenario. Thus a balance

must be reached by an analyst between including all

types of interactive transactions which are

representative of a real workload, and keeping the

total scenario length to a manageable size. The

scenario length directly affects both the cost of a comparison study and the total time required to conduct

a study. If only a few runs per day are possible, a comparison study could easily extend over several

months

.

The requirement that the data be statistically

independent poses a problem that must be given careful

consideration. If the data are not independent, they

are "correlated". This means that the value of a current measurement is affected by the value of a previous measurement. In a comparison study,

correlation among data points would be large if

consecutive measurements for a given measure were of

constantly increasing or decreasing value. This would

happen, for example, if there was an unstable

environment in which a system had just recovered from a crash, and users were logging on and initiating work in

quick succession. Under such circumstances,

consecutive response time measurements of the time to

log on are very likely be constantly increasing.

10
Appendix A contains an extensive discussion of the

statistical notion of correlation and how it relates to

the realities present in computer system data

collection. In general, if data are correlated, the

number of measurements needed for a given probability

of making a correct selection increases dramatically,

thus further constraining the scenario length. There

are many techniques, however, for avoiding correlation

in the data, and often these can be used easily without

violating experimental objectives. Thus, with careful

planning (and reasonably well-behaved computer

systems) , a correlation problem can be avoided.

2.3 Script Generation

A machine- independent scenario must be translated

into a machine-dependent benchmark called a script for

each system under study. This process involves

translating general scenario commands into commands

that are compatible with each particular system, and

possibly modifying small sections of programs which are

not directly transportable across heterogeneous

computer systems. Care must be exercised to ensure

that the scripts which are executed on different

systems are as similar as possible.

Typically, the translation must be done in close

cooperation with vendor personnel who have a detailed

knowledge of the respective computer service.

Producing a workable script may require interfacing

with system I/O devices, editors, compilers, linkers

and loaders, database management systems, and so on.

2.4 Data Collection And Analysis

In any study involving measurement of computer

services, the efficiency of the data collection process

and the validity of the data analysis depend upon the

choice of an appropriate statistical methodology.

Ranking and selection procedures (see GIB77 or KLE75

for a survey of these techniques) provide an

appropriate foundation for computer service comparison

and selection methodology. These procedures can be

roughly characterized as following three lines of

development: ranking computer services 1) by comparing

sample means, 2) by comparing sample percentiles, and

3) by comparing sample proportions. In all three

cases, the procedures specify the number of data points

which must be collected from each service in a comparison study in order to guarantee that the

n
probability of a correct selection be greater than or

equal to a predetermined minimum value. A detailed

description of how the three classes of ranking and

selection procedures can be applied to computer

comparison studies has been described in two recently

published papers [AME78b, MAM77]. The main points of

these two papers are summarized below.

The use of a mean, percentile or proportion

statistic for computer service comparison is an analyst

choice based on considerations about the objectives of

a comparison study, the statistical properties of the

data and the statistical requirements of the selection

methodology. (The reader unfamiliar with the

statistical terms employed in this report is referred

to NAT63.) Means are often used for comparisons when

criteria like script turnaround time or script cost are

of interest. For comparison criteria such as response

time, which tend to have exponential-like distributions

[FUC70, TOT65] , the mean is not as meaningful a statistic. Percentiles or proportions are more

appropriate rAND71, BR075].

A percentile and a proportion approach to

performance are very similar in that they both rely on

the cumulative distribution of a single performance

criterion. The difference lies in whether an analyst

prespecifies a desired percentage value or a desired

comparison criterion value. In a comparison based on

percentiles, a percentage is predetermined. Results

are produced of the form:

"if computer service A has exactly 90% of its

response times less than 3 seconds, and computer

service B has exactly 90% of its response times

less than 3.5 seconds, then A is better than B,"

where "90%" is prespecified by the analyst. In a comparison

based on proportions, a criterion threshold is prespecified.

Results are produced of the form:

"if computer service A has exactly 80% of its

response times less than 3 seconds, and computer

service B has exactly 87% of its response times

less than 3 seconds, then B is better than A,"

where "3 seconds" is prespecified by the analyst.

12
Mean, percentile or proportion statistics can be used

in either phase II (evaluation of mandatory measurement

criteria) or phase IIIB (evaluation of desirable measurement

criteria) of the selection process. The next two sections

of this publication present the details of the ranking and

selection procedures applicable for selection in phase IT

(section 2.4.1), and for selection in phase IIIB (section

2.4.2). A basic set of definitions common to ranking and

selection methodologies is presented in Table 2 for easy

reference. Table 2 should be read in conjunction with the

next sections. The notation used is consistent with that

used in the statistical ranking and selection literature and

will facilitate further reference to that literature.

The actual step-by-step procedures that are presented

in this section, combined with the tables in Appendix B, are

complete in that they may be applied without reference to

other books or articles. No statistical justification for

the procedures is presented. The interested reader is

referred to AME78a, C-IB77 and KLE75 for the appropriate

statistical theory. Also, for those readers for whom this

is a first exposure to ranking and selection procedures, it

is suggested that section 2.4.1 and 2.4.2 be read in

parallel with section 3.3.1 and 3.3.2, respectively, where

example applications of each procedure are presented. This

mode of reading will provide an intuitive feel for the

procedures and will place their necessarily abstract and

general steps in an environment with which the reader is

already familiar.

In the procedures below, assumptions are made such as

the analyst wishing to find the "smallest" mean or the

"largest" proportion of one or the other performance

measure. These assumptions are made to reflect what

typically would be desired in a computer service selection

process. Procedures do exist for other selection

parameters, e.g. the "largest" mean, and the reader should

refer to the ranking and selection literature for these

procedures.

2.4.1 ^Selection Of Services Satisfying A Mandatory

Requirement: Based on Proportions -

In the case of selecting those computer services which

satisfy a mandatory requirement (phase II in Figure 1), a methodology is required which leads to an analysis of the

data that answers the question "Which services are at least

as good as a minimum requirement?". The motivation for

conducting this type of selection is to screen out some of

the alternatives which fail to meet certain minimum

performance requirements, thereby reducing the overall

number of alternatives from which a final selection is to be

13
Table 2

Summary of Ranking and Selection Terminology

A. Notation Common to All Procedures

k: number of alternative computer services in the study

P* : desired level of confidence in the correctness of the selection results;

P* is a probability of correct selection, and is a value chosen by the analyst

n: number of measurements required from each computer service to guarantee P*

B. Notation Common to Selection Based on Means

x(i,j): jth sample measurement from ith computer service, j = l,2,...,n

xT(i): first stage sample mean of the ith computer service

nl : number of measurements to calculate the xl(i)'s

x2(i): second stage sample mean of the ith computer service

n(i): number of measurements to calculate x2(i)

x(i): weighted sample mean of the ith computer service

b: weight used to calculate the weighted sample mean

d*: the minimum difference to be detected between the sample mean of the best

computer service and that of the next best service

2

s (i): sample variance of the ith computer service

C. Notation Common to Selection Based on Percentiles

a: a-quantile used to denote the a-th percentile; e.g., a = .9 for the 90th

percentile

d*: the minimum difference to be detected between the smallest a-quantile value

and all other a-quantile values

q(i): sample a-quantile of the ith computer service

^

D. Notation Common to selection Based on Proportions

C(thld): criterion threshold value; i.e., mandatory value for a particular

performance measure

X(i): number of measurements from the ith computer service which are less than C(thl

p(min): minimum proportion of measurements which must be less than C(thld)

p(i): proportion of measurements from the ith computer service which are less

than C(thld)

p(i ) : estimate of p(i

)

14
made. The ranking and selection literature refers to

selection of this type as selection "better than a standard". The ranking and selection techniques which have

been developed for selection better than a standard are not

appr opr ia te for computer service selection v^/hen pe r f o rmance

criteria are being compared at their mean or percentile

values. The procedures make assumptions about the data that

are clearly not justified in computer service selection

experiments (such as equal variance for all systems). But,

an appropriate procedure does exist when proportions are the

basis for selection.

For selection based on proportions, it is assumed that

there are k >= 2 computer services from which it is desired

to select a subset of minimal size which contains all

computer services satisfying a specific performance

requirement. The goal of this procedure is to perform the

subset selection in a way which guarantees a probability at

least P* that all computer services which meet or surpass a specific performance requirement are contained in a selected

subset. P* is chosen by the analyst. A joint confidence

statement which can be made with confidence P* is that all

of the computer services which are rejected do not meet the

performance requirement.

A mandatory performance requirement is stated in terms

of a measure threshold value, C(thld) , and a minimum

required proportion, p(min). For example^ if a criterion of

interest is script turnaround time, then a possible

performance requirement might state that at least eighty

percent of the script executions must take less than 10

minutes. In this case p(min) = 0.80 and C(thld) = 10

minutes

.

The choices of C(thld) and p(min) are based on

nonstatist ical considerations such as past performance or

projected performance needs. Selection is accomplished via

collecting measurements from each computer service under

study and estimating the true proportion of measurements

lying below C(thld). The analyst is then able to choose

those computer services which pass this phase of the

selection process.

This procedure for selection assumes that measurements

are independent and that measurements from the same computer

service have a common probability of being less than

C(thld). The procedure makes no assumption about their

underlying distribution. The procedure is a four step

process

:

Step 1: Choose appropriate P*, C(thld) and p(min)

values according to nonstatist ical considerations.

15
step 2^: Collect n independent measurements from each

of the k computer services. The analyst chooses n based on

nonstatistical considerations, bearing in mind that as n increases, more accurate estimates of each alternative's

proportion will be attained.

Step 3^: Let X(i) = number of measurements from service

i which are < C(thld). Note that since n, the total number

of measurements made, is identical for each computer

service, the X(i) values can be used to indicate a ranking

of the true proportions.

Step 4^: For each service, compare X(i) to a constant

M, which is derived from ranking and selection theory. If

X(i) >= M, then include the computer service in the selected

subset; otherwise eliminate it. M is determined by table

lookup based on the parameters k, n, P* and p(min)

.

Extensive values for M are tabulated in AME78a and have been

reproduced in Tables 1-19 in Appendix B.

2.4.2 Selection Of The Best Service -

In phase IIIB of the computer service selection

process, a methodology is required which leads to an

analysis of the data that answers the question "Which

service is the best one?". Ranking and selection procedures

exist for choosing the best computer service based on

performance criteria expressed in terms of mean, percentile

or proportion statistics.

2.4.2.1 Selection Based On Means -

The goal of this procedure is to choose as best that

computer service which has the smallest mean value of a single performance criterion. It is assumed that there are

k >= 2 computer services from which the best one is to be

determined based on the mean of a set of performance

measurements for each service. Let represent the

sample mean from the ith service, i = 1, 2, k. The

probability of a correct selection is desired to be at least

P* when the true mean of the best service is at least a distance d* away from the true mean of the next best

service. For example, it may be desired to select with 95

percent confidence the best service whenever a difference of

0.5 seconds or more exists between its and the next best

service's mean response time. In this case P* = 0.95 and d*

= 0.5.

16
The introduction of d* is required by the nature of the

statistical theory upon which the procedure is based.

However, it is a useful tool for the analyst in that it

provides the ability for stating how close two computer

services can be before they are considered identical in the

analyst's mind. The statistical theory is such that if the

best service and any other service(s) are not separated by a distance d*, then the selection of any one of these close

services is considered a correct selection of "the best".

This procedure for selection assumes that measurements

from each service are independent and are drawn from a distribution that is normal. A normality assumption holds

for certain types of performance data, as will be

demonstrated in the case study. Because of a normality

assumption, it is necessary to perform a preliminary

investigation before using this procedure to verify that

such an assumption is valid. Statistical tests for

normality like the chi-square or Kolraogorov-Smirnov tests

are appropriate [BRA68]. An alternative procedure such as one based on percentiles or proportions will be necessary

when the data cannot be assumed normal. The procedure is a five step process:

Step 1: Collect an initial sample of nl independent

measurements from each service and calculate the first stage

sample means of the k computer services:

nl (1) xl(i) = E x(i,j)

il]

and the sample variances':

(2) s^d) = ""^-^ilf'"''

j = l

Step 2: Calculate the total sample size n(i), required

,2

for service i

(3) n(i) = max nl + 1

where h is a constant whose value is dependent on k, and

analyst supplied values of P* and d*. Values for h which

are likely to be used in a comparison of computer services

are given in Table 20 in Appendix B. These values assume an initial sample size of nl = 30. As nl decreases, the values

of h increase, thus requiring more second stage sampling.

17
other values of h have been tabulated by Dudewicz [DUD72,

DUD75].

Step _3= Collect n(i) - nl additional independent

measurements from each service i and calculate the second

stage sample means:

n(i) x(i,j)

(4) x2(i) = z - nl

Step A_: For each service calculate the weighted

average of the first and second stage sample means:

(5) x(i) = b xl(i) + (1-b) x2(i)

where the weight b is given by

(6) b = ^ II . (

1 . "(i) [] _ n(i) - nl

\ (hs{i)/d*)2

Step 5: Select the service with the smallest weighted

mean as the best.

2.4.2.2 Selection Based On Percentiles -

A percentile, like the 90th percentile, may also be

referred to as an al pha-quan t i 1 e (denoted a-quantile) , e.g.

the .9-quantile for the 90th percentile. The a-quantile

terminology is consistent with the ranking and selection

literature and will be used in the remainder of this

section. The goal of this procedure is to choose as best

the computer service which has the smallest a-quantile. For

example, a computer service which has a .9-quantile value of

3.2 seconds for response time (i.e., 90% of all response

times are 3.2 seconds or less) would be chosen over all

other services which had values larger than 3.2 seconds.

Selection of the best computer service is accomplished via

estimates of a-quantile values which are called sample

a-quantiles. The procedure quarantees a probability of

correct selection at least P*, provided there is a distance

d* between the smallest a-quantile value and all other

a-quantile values. The significance of d* in this procedure

is similar to its significance in the procedure for

selection based on means.

18
Determination of n, the required number of

measurements, using this procedure is accomplished by

straight-forward table lookup based on values of k , P* and

d* . Appropriate tables for use in computer service

selection exist only for .5-quantiles [SOB67] and for

.9-quantiles [MAM??] and are provided in Tables 21-26 in

Appendix B. The tables for selection based on a-quantiles

are very limited at present, but work is currently in

progress to produce a more extensive set [DUD79].

This procedure for selection assumes that measurements

from each computer service are independent and that the

underlying data distributions are continuous, but it makes

no other assumptions about the data distributions. The

procedure is a two step process:

Step 1.: Collect n independent measurements from each

of the k computer services under study, where n is

determined by table lookup based on analyst supplied values

of k, a (for the a-quanti]e), P* and d*. Calculate the ith

sample a-quantile, q(i), for each service, where q(i) is

defined as follows. Let the positive integer r be defined

by r <= (n+l)a < r+1, and let x denote the rth largest

measurement. Then ^

q(i) = (r + 1 - (n+l)a)x^ + ((n+l)a -

Step 2\ Select the service which produced the smallest

sample a-quantile value as the best.

2.4.2.3 Selection Based On Proportions -

In this procedure it is desired to select the best

service according to a threshold value of a single

criterion, C(thld). Let p(i) represent the true proportion

of measurements from computer service i which lie below the

threshold value C(thld) . The goal of this procedure is to

choose as best that computer service which has the largest

p(i) . For example, suppose C(thld) = 3.0 seconds for a response time criterion. A computer service which has 92%

of its response times less than 3.0 seconds would be chosen

as best if all other services had smaller proportions of

values less than 3.0 seconds. Selection of the best

computer service is accomplished via estimates of p(i) which

are denoted p(i). The procedure guarantees a probability

of correct selection at least P*, provided there is a distance d* between the largest and next largest proportion

value. The significance of d* in this procedure is similar

to its significance in the procedure for selection based on

means. Extensive tables and graphs of n for various values

of k, d* and P* can be found in S0B57. Values of n

19
appropriate for use in computer service selection have been

reproduced in Tables 27-30 and Figures 1-8 in Appendix B.

This procedure for selection assumes that measurements

from each service are independent and that measurements from

the same computer service have a common probability of being

less than C(thld). The procedure makes no assumption about

their underlying distribution. The procedure is a four step

process

:

Step 1: Collect n independent measurements from each

of the k computer services, where n is determined by table

lookup based on analyst supplied values of k, P* and d*.

Step 2: Let X(i) = number of measurements from service

i v;hich are < C(thld).

Step 3: Compute p(i) = X(i)/n.

Step _4: Select the service which produced the largest

estimated proportion value as the best. (In case of ties

for the largest p(i) , the probability statement is

satisfied by randomly selecting among the services whose

p(i) values were tied.)

3.0 A CASE STUDY

A large scale feasibility case study was conducted at

the National Bureau of Standards to evaluate the time and

cost required to apply the computer service comparison

methodology in an actual procurement environment. Four

heterogeneous remote access time sharing services were

compared: a DEC System-10 running a TOPS-10 Monitor, a Honeywell fil80 running MULTICS, an IBM 360/65 running TSO,

and a UNIVAC 1108 running Exec 8. The specifications for

the case study and the experimental results are presented

her e

.

3.1 Scenario And Scripts

A scenario for the case study, presented in Table 3,

was designed to be reasonably representative of the

functional requirements of a real workload (se-^ discussion

in section 2.2). The scenario consists of commands

representing two types of remote access interactive

computing: transaction processing and interactive program

development and execution.

20
Table 3 Scenario for the Case Study

General Specific ations: Think time {response-to-stimulus

delay) for all user commands is 6 seconds.

File Character istics ; the files listed below are to be

resident on the respective systems before the start of

script execution.

Files for executing a COBOL sparch

bibliographic database:

SELECT: compiled COBOL program

BIB: bibliographic database

ACCESS: bibliographic entries to be found

CHOSEN: list of entries retrieved

RESULT: summary of search

of

Files for executing a FORTRAN version of the BELL

benchmar k

:

BELL: compiled FORTRAN program

BELLIN: input specifications for the program run SELLOUT: output of program run File for interactive FORTRAN program, INTERA, a source program (with errors) to compute prime

numbers

Functiona l Scr ipt

;

1. Logon to the system.

2. Execute SELECT.

3. Execute BELL.

4. Copy file INTERA to file INTRl

.

5. Edit INTRl, correcting a syntax error by changing line

14 to read: GO TO 10.

6. Edit INTRl, correcting a logical error by changing line

23 to read: PP=PP+1

.

7. Compile INTRl.

8. Execute INTRl. This will initiate the following

d ialogue

:

ARE YOU TESTING A NUMBER? (0 or 1)

Enter : 0 (CR)

ARE YOU GENERATING PRIMES? (0 or 1)

Enter: 1 (CR)

LIMIT UNDER WHICH PRIMES ARE TO BE GENERATED:

Enter : 100 (CR)

25, 2, 3, 5, 7, 11, 13, 17, 19, 23, 31, 37, 41, 43, 47,

53, 59, 71, 68, 71, 73, 79, 83, 89, 97

ARE YOU TESTING A NUMBER? (0 or 1)

Enter : 0 (CR)

ARE YOU GENERATING PRIMES? (0 or 1)

Enter : 0 (CR)

DO YOU WANT TO QUIT? (0 or 1)

Enter: 1 (CR) 9. Delete INTRl file.

10. Logoff system.

21
The transaction processing was implemented in a COBOL

program which executed a sequential search of a bibliographic database in order to retrieve a given set of

entries. The database consists of 2400 records, each of

which is 132 characters long. The transaction processing

was accomplished by command no. 2 in the scenario (see Table

3). A synthetic module which was capable of being adjusted

for varying amounts and types of CPU and I/O activity

[BUC69] was executed next (item no. 3 in the scenario) . The

last section of the scenario (items no. 4-9) consists of

commands to debug, compile and execute an interactive

FORTRAN program which computes prime numbers.

The translation of the scenario into scripts executable

on each computer service required interaction with personnel

who possessed a thorough knowledge of the. respective

operating system. For each computer service two activities

were required. First, it was necessary to establish

permanent program and data files for use in each script

execution (as noted in Table 3). Then, instructions for the

actual script execution were determined, sometimes including

complicated "control language" constructs.

The specifications for running the scripts are typical

of a real selection procedure. The user is often concerned

with the quality of the computer service offered during a particular time period. If this quality is within

acceptable limits, then service at all other times is

assumed to be acceptable. In this case study, the scripts

were used to collect performance data from each service only

between 8:30 a.m. and 4:45 p.m., Monday through Friday, in

order to compare the services under normal work day

cond i t ions

.

3.2 Data Collection And Analysis

Eight performance measures were chosen for computer

service comparison in the case study. They are:

1. cost,

2. turnaround time for the entire script execution,

3. response time for the bibliographic retrieval (no.

2 in the scenario)

,

4. response time for the FORTRAN program (no. 3 in

the scenar io) ,

22
5. response time for the copy command (no. 4 in the

scenar io) ,

6. response time for the first edit command (no. 5 in

the scenar io) ,

7. response time for the second edit command (no. 6

in the scenario) , and

8. response time for the interactive calculation of

all prime numbers less than 100 (no. 8 in the

scenar io) .

3.2.1 Measurement Of Computer Services -

The hardware/software configuration for the case study

is illustrated in Figure 2. The scripts were executed on

each system under the automatic control of a remote terminal

emulator called the Network Access Machine [ROS75].

Turnaround time, response time and cost data were

automatically collected for each execution of each script by

a minicomputer called the Network Measurement Machine

[ABRVVa, ROS76]. The data were analyzed using a statistical

package called OMNITAB [H0G71]. Correlation coefficients,

means, percentiles and proportions were calculated. Since

all of the selection procedures described above require

independent measurements if the probability of correct

selection is to be at least P*, the correlation coefficients

were evaluated first. The method of ensuring independence

for the experimental data is described in section 3.2.2.

Readers are advised to read Appendix A before proceeding

with this discussion if they are unfamiliar with

correlation. Section 3.3 then presents example applications

for each of the procedures described in section 2.4

From this point on, the computer services will be

referred to as service A, B, C and D. The letters have been

randomly assigned to the four services to ensure their

anonymity. The selection results discussed below are

primarily a function of the load on a given system and not

of the individual hardware/software combinations providing

the computer service. Thus, it cannot be concluded from

this study that a given computer system is better than the

others, but only that a given computer serv ice is better.

The results of the case study are not to be construed as an

endorsement of any one computer service.

23
I— o z

ME SIS AB <

LU >- t~- LU

H- 1

IDUK < NW >- v> z to <LU < O

UJQ

>-

m

Q£U I- < p: ^ i2 O ^ I— Q. < < UJ 3 I- Q£ to to

O

_j lO 00 5 zl 'O o

il^ UJ UJ

Q S Z CM

O UJ - \ ^ ^ ^ > u ^ < °-

UJ

u

LUz

LU—> X

CO

u

O < o

UJ o ^ Q I E5 3 —I d: LD < O ^

LU ^ O

>> •oao

+j

s-ococoo

<u

s-ns +JO

i-(0so

CM

O)

S-3

CD

to

O < z < u

24
3.2.2 Assessing Correlation - An experiment was designed so that each script run

contributed one measurement for each of the eight measures

listed above. Date and time information was also recorded

for each script run. Thirty consecutive script runs were

performed on each of the four computer services and

correlation coefficients (defined in Appendix A) were

estimated for each performance measure for each service.

These are presented in Table 4. The 95% confidence interval

(defined in Appendix A) for a true correlation coefficient

of zero v^hen n = 30 was computed to be (-.36,. 36). If the

estimated correlation values fell within the 95% confidence

interval, then for purposes of the case study, the

measurements were taken to be independent.

For computer service D all estimated correlation

coefficients for all eight measures were within the 95%

confidence interval. Thus, it was concluded that these

measurements were independent and no care need be taken to

space script runs at any minimum time intervals. For the

other three computer services, all of the estimated

correlation coefficients were not within the 95% confidence

interval, thereby invalidating an independence assumption

(see Table 4). For these services, thirty more script runs

were executed no less than 15 minutes apart. The new

interval lengths were chosen in an ad hoc manner based on

the average time to run a script on the respective system

and on the magnitude of the estimated correlation

coefficients from previous script runs. Correlation

coefficients were then estimated for the second thirty

script runs. For services B and C, the correlation

coefficients were found to be within the 95% interval (see

Table 4), but service A script runs required an even longer

interval between consecutive runs. Thirty more scripts were

run on service A no less than 40 minutes apart and finally

no less than 60 minutes apart to obtain measurements that

could be assumed independent.

Measurement collection then proceeded on all computer

services according to a schedule that maintained the minimum

interval required for each service for independent

measurements. Since the purpose of the experiment was to

demonstrate several selection procedures, each of which

might require a different number of measurements, a maximum

number anticipated for any selection process of 120 was

collected from each computer service. Thus, a script was

run 120 times on each of the four systems. (In a real

application of these procedures the exact number of

measurements is determined in advance based on the

predefined performance criteria.) Using a 99% confidence

interval, data independence was tested on the set of 120

observations. These included the set of 30 data points that

were initially taken to be independent for each performance

25
oCO

II

in

t->

0) o

(J >

S- >r (U oo

0)O S- O +J c 3 o Q. 1— E +J Oo

cu

S-ooo<u

+-> E C •!- o o Ln CM CTl

•r- E to Ln CTl CM u o VD LD Ln CM CM o O

(o o ro CM

O CMO OO CLUD 1

CD

C c •-

O in CO Ln «^

•-- E Ln 00 (Ti CM CTl CTl O O 00 o oo Ln

n3O oo CM CM CM o CM

cnc CO CO CO to

•.- <u 00 o CM to CO o c

rO

O 00 00 o CM OO , oo

QC 1 c !- Ln

•r- E CTl oo CO CM CM o Ln CM Ln CO CM * CM

rO LT) oo OO o o o O CLi

1 1 CO

Ol

c: o CO CM CM

•1- <L) CM i-~^ o OO o c un O oo OO

n3

O LD ID o CM o

Q.

C • to 1 1 1

cj) c: c Ln 00 o Ln CO CM

•.- E CO CM CO

O o Ln tn CM CM oo to Ln

(O Ln CM CM

OO o o O 1

1 1 1 1

CD

•r- O) r-- CO CM (Ti tn

O o c: uo to oo CM LD oo

n3

O en CTi CTl oo ID o

CO OO o CM

00 1

Ln OO CO

•1— Q) CM to oo

O CM

Q o c 00 cyi oo Ln <y\ CM Ln

fo o o o o CM o O o , oo t 1 1 1 1 1

"O

CM c OJ

s_ E

ro o to

S- o i- o

CD

01 o O +-> c: U S- Q. s_ +-> o

O) +-> 13 (O a. o

(C Su 4-> S- 1— TD cu +-> E 13 0) CD 7Z OJ to to

S- (/I (/) +J

O > cu o fO Q. •1- OJ 4-> Z3 E

4- oi cu

M 1— >>

m o O

S- to S- JD S_ q; Q- Su S- 0) I/) O o r- 4-> o O cu fO Q. O oo CO Ll- <_) u. CO CJ

-o

0}3

<+-Ooc

03O

!-> C

CU -o

26
measure. For services C and D, a data independence

assumption had been maintained, but for services A and B the

correlation coefficients for some of the performance

measurements did not fall within the 99% confidence

interval. For these two services, the 90 most widely spaced

script runs (of the 120) were determined, and these new data

sets were again tested for independence. In both cases,

independence could be assumed for these reduced data sets.

For simplicity in the presentation of example applications

in section 3.3, the data sets from computer services C and D

were also reduced to 90 points (the first 90 points is each

case) .

In general, the computer services with a relatively

uniform load throughout the measurement period had the

lowest estimated correlation coefficients. For those

services whose load varied widely with the time of day, the

correlation estimates were highest. In these latter cases,

scripts which \^ere run in quick succession tended to produce

"runs" of measurements either below the mean (light loading)

or above the mean (heavy loading). As discussed in Appendix

A, it is just this type of data that is likely to be

correlated. Thus, an analyst should take extra care to

widely space script runs on computer services whose loading

varies widely.

3.3 Comparison Results

Table 5 presents various summary statistics for the

four computer services. Many of these statistics were used

in the examples below. The rest are presented to provide a complete comparison picture. All of the statistics are

based on 90 measurements. Since 90 independent measurements

were available for all performance variables on all computer

services, all 90 of them are used in the examples below,

even though in some cases fewer were required for a given

level of confidence. It is statistically acceptable to use

more than the required number of measurements, and advisable

to do so if they are available.

In the examples, the values of those parameters which

are to be chosen by an analyst were arbitrarily specified.

Hov;ever, they represent values that are likely to be chosen

in a computer service selection process. The steps in the

examples parallel the steps in the original presentation of

the procedures in section 2.4. In references to the "ith

computer service", i assumes the values A through D (rather

than 1 through 4 )

.

27
Table 5 Sample Statistics from the- Case Study

Variable

1. Cost ($)

2. Script turnaround

(min.

)

3. Response time for

bibliographic re- trieval (sec.)

4. Response time for

FORTRAN synthetic

program (sec. )

5. Response time for

copy command (sec.

)

6. Response time for

first edit (sec.

)

7. Response time for

second edit (sec.

)

i. Response for

calculation of prime

numbers (sec.)

)mputer Mean 5-Qijfint i 1 p . QV — nii^n1""ilo i^Uuil L 1 1 c iNU Mn MoA C 1 1 y*Qman c . fleet J U 1 clllci 1 Lo 'rvice (Median) <C(thld)

C(thld) = 2.00 A 3.44 3. 93 d c;i

1 0 B .81 . / u 1 no 1 . Uo 90

c 1 . 69 1 fid /-7 0/

D K72 1.71 1.81 87

C(thld) = 10.00 A 4. 53 0 . oy onyu

B 4. 97 7 9 yy

c 12 77 1 Q 91 1 o . ^ 1 6c

D 7.85 7.70 10.36 77 C(thld) = 10.00

A in fi^

1 o . 40 52

B 1 08 . DO 9 c;9 90

c 147 8? 1 1 D . M- 1 9C;/I 17 CO", , \ 1 0

D 6.36 3.37 12.19 80

C(thld) = 30.00

A 12 91 9PCO 1 n . 1 u

OA

B 1 19 . fin OU 9 fin yu

c 30 27 ?1 ^1 c^n Ai U . D 1 bo

D 11.32 7.00 20.22 84

C(thld) = 3.00

A 2 27 1 59 00 o . yy / 1 B 2 53 9 nR /I /I 4 . 40 by

1 fi /l/I "39 44 . Oil U

D 4.26 3.69 9.12 43

C(thld) = 3.00

A 1 70. 9 C3 d . DO O A 84

B • 11 5^ 1 ^ 1 K onyu

c 4 IR C , 1

'\ 0 AO y . 4y cnbU

D 3.50 2.06 6.27 57 C(thld) = 3.00

A 1 . 05 1 Q"?

1 . y J P7O/

Ru 1 1;

. 1 3

1 c nn

yu

r t • oo c . Of Q 1

"3 O. 1 J D 3.63 1 fi2 4 fi7 fii 0 i C(thld) = 1.00

A 1.21 .92 2.37 49

B .13 .12 .15 90

C .76 .60 1 .30 74

D 2.62 1.66 6.80 32

28
3.3.1 Selection Of Services Satisfying A Mandatory

Requirement: Based On Proportions -

Exampl

e

(parallels section 2.4.1)

Step j^: Suppose it is a mandatory requirement that a computer service be able to interactively calculate the

prime numbers between 1 and 100 (no. 8 in Table 5) in less

than 1.00 second at least 80% of the time. Further, suppose

it is desired to select a subset containing all such

services with a probability of correct selection at least

.90. Then P* = .90, C(thld) = 1.00 and p(min) = .80.

Step 2\ Choose n = 90. Ninety independent

measurements have been made on each computer service.

Step ^: For the computer services under study, X(A) = 49, X(B) =~90, X(C) = 74 and X(D) = 32.

Step _4: Table 3 in Appendix B shows that M = 64 for

this example. Therefore select services B and C and reject

services A and D.

Exampl

e

(parallels section 2.4.1)

Step 1^: Suppose it is a mandatory requirement that a computer service be able to process the first edit command

(no. 6 in Table 5) in less than 3.00 seconds at least 70% of

the time. So, C(thld) = 3.00 and p(min) = .70. Let P* = .95.

Step 2\ Choose n = 90. Ninety independent

measurements have been made on each computer service.

Step 3: For the four computer services, X(A) = 84,

X(B) ^T0,~X(C) = 50 and X(D) = 57.

Step 4: Table 3 in Appendix B shows that M = 53 for

this example . Therefore, select services A, B and D and

reject service C. Note that if ranking and selection

techniques were not being applied in this example, then

service D might have been rejected since 70% of 90 is 63 and

X(D) < 63. But the ranking and selection procedure

indicates that because of experimental variation service D

cannot be rejected at the 95% confidence level.

29
3.3.2 Selection Of The Best Service -

3.3.2.1 Selection Based On Means -

Example

(parallels section 2.4.2.1)

Step It Suppose it is desired to select the computer

serviFi which has the lowest mean cost (no. 1 in Table 5)

from among services B, C and D. (Service A already may have

been rejected in an earlier selection process.) This

procedure for selection is applicable only if the

measurements are normal, so a Kolmogorov-Sm i rnov test for

normality was applied to the cost data from computer

services B, C and D. Sample results for computer service C

are presented in Table 6. The test is described fully in

many statistics textbooks (BRA68 has a particularly good

discussion) . The cost data from all three services passed

the Kolmorgorov-Smi rnov test at a confidence level of .95.

The fi_rst stage sample means (nl = _30) from each service

were xl(B) = .80, xl (C) = 1.64 and xl (D) = 1.74. The sample

2 2 2 variances were s'(B) = .20, s^(C) = .53 and s (D) = .17.

Step 2: Choose P* = .95 and d* = .25 (this d* value

implies that an analyst v\7ishes to detect differences in cost

greater than $0.25). For nl = 30 and k=3. Table 20 in

Appendix B shows that h = 2.81. Using h to calculate the

total number of measurements required from each service

yields:

n(B) = max[ 31 , ( ( . 45 ) (2 . 81 ) / . 25

) ^ 1 = 31

n(C) = max[ 31 , ( ( . 73 ) ( 2 . 81 ) / . 25

) ^ ] = 68

n(D) = max[ 31, ( ( . 42 ) ( 2 . 81 ) / . 25

) ^ ] = 31

Step 3^: Service C requires (68-30) more data points

and services B and D require one more measurement each.

Since 90 independent measurements already exist for each

computer service, use (90-30) more measurements to calculate

the second stage means for every service. Hence, x2(B) = .81, x2(C) = 1.76 and xl (D) = 1.70.

Step _4; The weighted mean for each service is 'x(B) = .81, x(C) = 1.69 and x(D) = 1.72.

Step 5: Select service B as the best.

30
o o o o o o O o o o o o o o o O o o o o o o

to

4-> fO Q

+J

inoooo

•r->

S-(U

vooc oo

n3 •r- O > 4-5 •1— 13

4-> XI

O) 03 •r" S- S- ra o 3 -M E E CO S- =J •r~ o

1— o Q 2: o o o o o o o o o o CM CO CO CTi o o o o o o o o o

•1—

(0E

s-o

scu o

Ifra 4-> t—

cu 1—>o£

s-E.

CO1 >o

s-ooEo^

cuE

fOX

UJ

szo

cu -1- > 4-5 -a •!- rs cu 4-> JD > fO -r- s- I— scu 3 4-> to E to J2 13 -r- O C_) Q O Un3

+-> >

CO S- O CU '-^

C_) 4-J E o to o o o o CO tn to CO CO o o o o o o o o o o

II

cu> >>

T3 •rO

CU 4-J

C > (T3 CU

i- 1— Z3 1— COm cu 3 CT 1 cn

to E CU 3 S- o O U.

>i -a o

cu c ^ > cu o s- 3 cn

cu cr II 1— r-- LT) CO cu c 1

s-

— o

00

LO

CM

1^

00 00 o QVO• o

II

X

00 to to

cooo^cNjcni^Lntoto coi— ooLotocoocMun Oi— — I— r— r— CMCVJCNJ I I I I i I I I I O'^'^t-Lnrooootor^ ocoi— roLnr-~oooc\j 00>— 1— I— I— I— CVJCM

LO

31
Example

(parallels section 2.4.2.1)

Step 1: Suppose it is desired to select the computer

service which has the lowest mean turnaround time (no. 2 in

Table 5) from among services A, B and D. The turnaround

time data from these three services passed the

Kolmorgorov-Sm i rnov test for normality at a confidence level

of .95. The first stage sample means from each service were

xl (A) = 4.63, xl(B) = 4.92 and xl (D) = 7.87. The sample

2 2 2 variances were s (A) = .82, s (B) = 1.96 and s (D) = 1.74.

Step 2\ Choose P* = .90 and d* = . 5 (an analyst wishes

to detect differences of .5 minutes) . For nl = 30 and k =

3, Table 20 in Appendix B shows that h = 2.30. Using h to

calculate the total number of measurements required from

each service yields:

n(A) = max[ 31, (

( . 91 ) (2 . 30 ) / . 5

) ^ ] = 31

n(B) = max[ 31 , ( ( 1 . 39 ) ( 2 . 30 ) / . 5

) ^ ] = 41

n(D) = max[ 31 , ( ( 1 . 32 ) (2 . 30 ) / . 5

) ^ ] = 37

Step 3^: As in the previous example, since 90

measurements have been made on each computer service, use

(90-30) measurements for the second stage means for all

services. Hence x2 (A) = 4.48, x2 (B) = 5.00 and x2 (D) =

7 . 72.

Step 4^: The weighted mean for each service is x(A) = 4.53, ^(B)~= 4.97 and 3?(D) = 7.85.

Step 5: Select service A as the best.

3.3.2.2 Selection Based On Percentiles -

Exampl

e

(parallels section 2.4.2.2)

Step 1_: Suppose it is desired to select the computer

service which has the lowest median value for response time

to execute the bibliographic retrieval (no. 3 in Table 5).

Then a = .5. Choose P* = .90 and d* = .15. For k = 4,

Table 22 in Appendix B indicates that n = 67 independent

measurements are required from each service. Since 90

measurements already exist for each computer service, use 90

measurements. Hence, q(A) = 9.33, q(B) = .66, q(C) = 116.41

and q(D) = 3.37.

32
step 2\ Select service B as the best.

Example

(parallels section 2.4.2.2)

Step 1^: Suppose it is desired to select between

services A and D the computer service which has the lowest

.9-quantile value for response time to the second edit

command (no. 7 in Table 5). Than a = .9. Choose P* = .75

and d* = .04. For k = 2, Table 25 in Appendix B indicates

that n = <S9 independent measurements are required from each

service. Since 90 measurements exist, use all 90 to

calculate q(A) = 1.93 and q(D) = 4.87.

Step 2: Select service A as the best.

3.3.2.3 Selection Based On Proportions -

Exampl

e

(parallels section 2.4.2.3)

Step 1^: Suppose it is desired to select the computer

service which has the largest proportion of response times

for the execution of the FORTRAN program (no. 4 in Table 5)

which are less than 30.00 seconds. Then C(thld) = 30.00.

Assume that P* = .80 and d* =.10. Table 29 in Appendix B

indicates that for this example n = 90 independent

measurements are required from each service.

Step 2: For the four computer services, X(A) = 84,

X(B) ^W,~X(C) = 56 and X(D) = 84.

Step 2= For the four computer services, p(A) = X(A)/n

= 84/90 = .93, p(B) = 1.00, p(C) = .62 and p(D) = .93.

Step _4: Select service B as the best.

Example

(parallels section 2.4.2.3)

Step 1: Suppose it is desired to select the computer

servic^^ which has the largest proportion of response times

for the copy command (no. 5 in Table 5) which are less than

3.00 seconds. Then C(thld) = 3.00. Assume that P* = .80

and d* = 10. Then Table 28 in Appendix B indicates that n = 59 independent measurements are required from each service.

The total 90 measurements will be used.

33
step 2: For the four computer services, X(A) = 71,

X(B) = 6 9,~X(C) = 0 and X(D) = 43.

Step 3: For the four computer services, p(A) = .78,

p(B) = .75, p(C) = 0.0 and p(D) = .48.

Step _4: Select service A as the best.

4.0 DISCUSSION AND CONCLUSIONS

4.1 Estimated Cost Of Using The Methodology

The total cost of the case study was estimated to be

$16,500. Personnel and equipment costs are itemized

according to different tasks in Table 7. Each task was

performed by one professional and one technical person

working in cooperation on a full time basis. Personnel

costs are estimated at $200/day for each person. This

represents the averaged "burdened" (all overhead included)

cost to the Federal Government per day for each person.

In a typical procurement study, the equipment cost from

developing and running the scripts would be absorbed by the

vendors. Therefore, although these costs are noted in Table

7, no figures are entered. It is assumed that the equipment

costs for the data collection and analysis are part of a selection effort. These activities require a specialized

hardware and software configuration for about six weeks,

with an estimated cost of $1,500.

Since care was taken to design and execute the case

study in the same way that a real procurement would be done,

the total cost figure of $16,600 covers the measurement

phases of a f ul 1-scal

e

procurement study. This figure,

therefore, can be used as a basis for a reasonable cost

estimate when making a decision about using the selection

methodology in any real procurement. The impact on cost due

to an increase in the number of computer services or due to

a change in the number of selection criteria is discussed

below.

If more than four computer services are to be compared,

then various personnel and equipment costs will change, but

in different proportions. Time for scenario development

will remain the same since only one scenario is needed

regardless of the number of services. The time required for

script generation and for writing control programs for

remote terminal emulation will increase in direct proportion

to the increase in the number of services being compared.

This is because time is necessary to translate the scenario

into a system compatible script for each service. The total

elapsed time for data collection (script runs) may remain

34
Table 7

Estimated Cost

a. Personnel

Task Person Cost {$200/day)

Days

Scenario development 5 $ 1000

Script generation (4 computer 12 2400

services)

Control programs for remote 8 1600

terminal emulator

Script runs (data collection) 30 6000

Data analysis 15 3000

Report generation 3 600

Overhead (equipment failure, 2 400

bad data etc.)

75 $ 15000

b. Equipment

Task

Script development (4 computer

services)

Script runs (150 runs/service)

Data collection and analysis

Total Cost = $15,000 + $1,600 =

Cost ($) *

*

$ 1600

$ 1600

$16,600

*Supplied by the vendor

35
the same, or increase slightly, inasmuch as the requirement

for independent measurements forces spacing of script runs

anyway. As more time is required to collect measurements,

more money must be allocated for data collection equipment.

The total increase for both data analysis and report

generation would be about two more person days for each

additional computer service.

A change in the number of selection criteria to be

applied will not have much impact on the cost of the study.

Most automatic measurement devices collect data about every

interactive transaction, even if it is not all required.

So, increasing or decreasing the number of criteria actually

used in the selection has little effect on the time and cost

of data collection. Also, most statistical analysis

packages are as easy to run for one variable as for ten, so

changing the number of criteria will have little effect on

the time and cost of data analysis.

4.2 Applicability Of Techniques

All of the techniques for selection of the best

computer service (section 2.4.2) require that the analyst

supply a value indicating how close together the best

service and any other service(s) can be before they are

considered equivalent from both a statistical and a practical viewpoint. This value is represented by the

parameter d*. The statistical theory has been derived such

that a correct selection is defined to be either the best

service, or any other service that is within d* of the best.

Since the analyst chooses d* to be that value for which it

is not cost-effective or desirable to distinguish among the

services, this is an acceptable solution from one point of

view. The methodology guarantees selection of the best, but

when several services are close to the best (specifically

within d* of the best) the best may not be unique.

From another point of view, however, the statistical

solution may not be acceptable. This is true, for example,

if selection as the best implies being awarded several

points toward a final merit value, while all other services

are awarded no points. If this kind of point system is

used, as is often the case in Federal Government

procurement, all services which are within d* of the best,

and are both practically and statistically equivalent to the

best, should be awarded points. Hence, in certain

circumstances, depending on the selection objective, it is

possible that ranking and selection techniques for choosing

the best should not be used.

36
As an alternative, the subset selection technique

(section 2.4.1) can be applied in selection phase IIIB.

When applying the technique to mandatory criteria in phase

II, those services which meet the criteria are selected for

the next phase and those which do not meet the criteria are

eliminated. In phase IIIB, those services which meet the

criteria can be awarded an equal number of points, and those

which do not can be awarded no points.

4.3 General Recommendations

The selection methodology presented in this publication

is important because it provides statements about the

probability of having made a correct selection. The number

of independent measurements required from each computer

service in a comparison is clearly specified for a given

level of confidence in the selection results. If it is

determined prior to the measurement phase of a procurement

study that, due to either time or cost constraints, a required number of measurements cannot be made, then it is

not an unreasonable decision to eliminate the entire

measurement phase from the selection process.

A collection of measurements can often give an illusion

of fairness and objectivity to a computer service selection

when in reality the measurements contain little information

of value. This can be the case when a selection is based on

methods and techniques which are not statistically sound.

Suppose, for example, that an analyst executes thirty script

runs on a computer service, intending to use the

measurements which result as a fair and objective evaluation

of that service. Suppose also that these runs are performed

in quick succession to "save time". As was demonstrated in

this case study, it is possible that as few as two or three

of the thirty measurements are independent because "saving

time" may result in too short a spacing interval. In a situation where too short a spacing interval is used, thirty

measurements may be quoted as the basis for an evaluation,

but in reality the actual information is equivalent to that

provided by a much smaller set of measurements. Thus, only

an illusion of fairness and objectivity exists.

So, if statistically sound methods cannot be applied to

the measurement phases of the selection process, it may be

more effective to concentrate on the other phases of the

selection process (phase I and IIIA in Figure 1) than to

waste time and money producing measurements which are of

questionable value.

37
5.0 ACKNOWLEDGMENTS

The authors would like to thank Marshall D. Abrams for

sharing his extensive knowledge about the government

procurement process and for his many substantive and

editorial contributions to this publication.

38
6.0 REFERENCES

ABR75 Abrams, M.D. and I.W. Cotton, The Serv ice Concept

Appl ied to Computer Networks , NBS Technical Note 880,

National Bureau of Standards, August 1975.

ABR77a Abrams, M.D., I.W. Cotton, S.W. Watkins, R.

Rosenthal, and D.E. Rippy, "The NBS Network Measurement

System," IEEE Transactions on Commun icat ions , Vol.

Com-25, No. 10, October 1977, pp. 1189-1198.

ABR77b Abrams, M.D. and S. Treu, "A Methodology for

Interactive Computer Service Measurement," Communications

of the ACM , Vol. 20, No. 12, December 1977, ppT

936-944.

AME78a Amer , P.D., "Experimental Design for Computer

Comparison and Selection," Ph.D. Dissertation,

Department of Computer and Information Science, The Ohio

State University, December 1978.

AME78b Amer, P.D. and S.A. Mamrak, "Statistical Methods in

Computer Performance Evaluation: A Binomial Approach to

the Comparison Problem," Proceed ings Computer Sc ience and

Sta t i sties: El eventh Annual Symposi um on the I n ter f ace , Institute of Statistics, North Carolina State Un iversi ty

,

Raleigh, NC , March 1978, pp. 314-319.

AND71 Anderson, M.A. Jr. and R.G. Sargent, "A Statistical

Evaluation of the Scheduler of an Experimental

Interactive Computing System," Sta t i st ical Compute r Per fo rmance Eval ua tion , ed . W. Freiberg, Academic

Press, New York, 1972.

BRA68 Bradley, J. V., Distribution-Free Statistical Tests , Prentice-Hall, Inc., Englewood Cliffs, NJ, 1968.

BR075 Brown, R.S., "A Case Study of a Multi-Access

Scientific Benchmarking Application," Benchmarking , ed . N. Benwall, Hemisphere Publishing Co., Washington, DC,

1975, pp. 133-147.

BUC69 Buchholz, W. , "A Synthetic Job for Measuring System

Performance," IBM Journal , Vol. 8, No. 4, 1969, pp.

309-318.

CON76 Conti, D.M., "Federal Agency Practice and Needs in the

Area of Teleprocessing Workload Specification," CS 77-2,

General Services Administration, Automated Data and

Telecommunications Service, Washington, DC, October 1976.

CR074 Crothers, C.G., Wo rkload Determination and

Represen ta t ion fo r On-Line Computer Systems , ESO-TR-74-54 , The MITRE Corporation, Bedford, MA,

NTIS-AD-779 818, January 1974.

DAV63 Davis, O.L. ( ed . ) , The Design and Analysis o f

Ind ust r ial Exper iments , Hafner Publishing Co., New York,

1963.

DUD72 Dudewicz, E.J. and S.R. Dalai, "Allocation of

measurements in Ranking and Selection with Unequal

Variances," Sankhya , Series B, Vol. 37, 1975, pp.

28-78.

DUD75 Dudewicz, E.J., J.S. Ramberg and H.J. Chen, "New

39
Tables for Multiple Comparisons with a Control (Unknown

Variances) /' Biometr ische Ze i tschr i ft , Vol. 17, 1975,

pp. 13 --2 6.

DUD77 Dudewicz, E.J. and N.A. Zaino Jr., "Allowance for

Correlation in Setting Simulation Run-length Via

Ranking-and-Selec tion Procedures," St ud ies i

n

the

Manag emen t Sc iences , North-Holland Publishing Co., 197T^

DUD78 Dudewicz, E. j7 and Y.J. Lee, "How to Select the

Best Contender, " Annual Techn ical Conference Transactions

of the Amer ican Soc i e ty f o r Qual i ty Control , Vol . 32

,

1978, pp^ 546^552.

DUD79 Dudewicz, E.J., I.H. Dhariyal and P.D. Amer,

"Sobel's Nonparametr ic Selection Procedures for

Quantiles: Extension and Tables," Techn ical Repo rt , Department of Statistics, The Ohio State Un iversi ty

,

Columbus, OH, in prepartaion.

FER78 Ferrari, D. , Computer Systems Per fo rmance Eval ua tion , Prentice-Hall, Englewood Cliffs, NJ, 1978.

FIP42 Guidel ines f o r Benchmarking ADP Systems in the

Compet i tive Procurement Env i ronment , Federal Information

Processing Standards Publication (FTPS PUB)42-1, May

1977.

FIP57 Guidel ines f o r the Measur ement o f Interactive Computer

Serv ice Turnaround Time and Response Time , Federal

Information Processing Standards Publication (FIPS PUB)

57, August 1978.

FUC70 Fuchs, E. and P.E. Jackson, "Estimates of Random

Variables for Certain Computer Communications Traffic

Models," Commun icat ions o f the ACM, Vol. 13, No. 12,

December 1970, pp. 752-T5 7~

GEI62 Geisler, M.A., "The Sizes of Simulation Samples

Required to Compute Certain Inventory Characteristics

with Stated Precision and Confidence," Memorandum RM

3242 - RP, The RAND Corporation, Santa Monica, CA,

October, 1962.

GIB77 Gibbons, J.D., I. Olkin and M. Sobel , Selecting and

Order ing Popul a t ions

:

A New Sta ti st ical Methodology , John Wiley and Sons, Inc., New York, 1977.

GRU75 Grubb, D.S. and I.W. Cotton, Criteria fo r the

Per fo rmance Eval ua tion of Da ta Commun icat ions Serv ices

fo r Computer Networks, NBS~Techn ical Note 882 , National

Bureau of Standards, September 1975.

HIL69 Hillegass, J.R., "Systematic Techniques for Computer

Evaluation and Selection," Manag ement Sc ience , July-August 1969, pp. '^5-38.

H0G71 Hogben, D. , Peavy S.T. and R.N. Varner, OMNI TAB II

User ' s Reference Manual , NBS Technical Note 552, National

Bureau of Standards, October 1971.

JOS 7 7 Joslin, E.O., Computer Selection : Augmented Ed i tion , The Technology Press , Inc Fairfax Station, VA , 1977.

KLE75 Kleijnen, J.P.C., Statistical Techn iques i n Simulation , Part II, Marcel Dekker, Inc., New York, 1975.

LEE77 Lee, Y.J., "Winner Selection," Proceed ings 1977 Winter

Simulation Conference , Washington, DC, December 1977, pp.

40
86-91.

MAM77 Mamrak, S.A. and P. A. DeRuyter, "Statistical Methods

for Comparison of Computer Services," Computer , Vol. 10,

No. 11, November 1977, pp. 32-39.

NAT63 Natrella, M.G., Exper imental Statistics , National

Bureau of Standards Handbook 91, August 1963.

NOL74 Nolan, L.E. and J.C. Strauss, "Workload

Characterization for Timesharaing System Selection,"

So f twa re--Prac t ice and Exper ience , Vo 1 . 4 , No . 1

,

January-March 1974, pp. 25-39.

POR67 Porter, J.D. and B.M. Rudwick, "Application of

Cost-Effectiveness Analysis to EDP System Selection,"

MTR-327, ESD-TR-67-412, AD 667 522, The Mitre

Corporation, Bedford, MA, 1957.

ROS75 Rosenthal, R. , "Accessing On-Line Network Resources

with a Network Access Machine," IEEE In tercon 1975 , IEEE

Inc., 1975, pp. 25/3:1-4.

ROS76 Rosenthal, R. , D.E. Rippy and H. Wood, The Ne two rk

Measurement Machi ne — A Da ta Col lection Dev ice for

Measur ing the Per fo rmance and Ut il ization o f Compute r Networks , NBS Technical Note 912, NationaT Bureau of

Standards, April 1976.

RTE76 "Summary of the NBS/General Services Administration

Federal Workshop on Remote Terminal Emulation," CS 76-1,

General Services Administration, June 1976.

SHA69 Sharpe, W. F. , The Econom ics o f Computers , Columbia

University Press, New York, 1969.

SOB57 Sobel, M. and M.J. Huyett, "Selecting the Best One

of Several Binomial Populations," Bel 1 System Techn ical

Journal , Vol. 36, March 1957, pp. 537-576^

SOB67 Sobel, M., " Nonpa r ametr ic Procedures for Selecting the

t Populations with the Largest a-Quantiles , " Annal s o f

Mathematical Statistics , Vol. 38, 1967, pp. 1804-1816.

TIM73 Timmreck, E.M., "Computer Selection Methodology,"

Computing Surveys , Vol. 5, No. 4, December 1973, pp.

199-222.

TOT65 Totschek, R.A., "An Empirical Investigation into the

Behavior of the SDC Time-Sharing System," System

Development Corporation, Report SP-2191, AD622003, Santa

Monica, CA, 1965

TSP77 Spec ial Notice Concerning the Teleprocessing Serv ices

Prog ram , General Services Administration, Automated Data

and Telecommunications Service, Washington, DC, April

1977.

WAT76 Wat kins, S.W. and M.D. Abrams, Data Treatment and

Analysis for the Ne two rk Measurement System , NBS

Technical Note 897, National Bureau of Standards, March

1976.

WAT77 Watkins, S.W. and M.D. Abrams, "Remote Terminal

Emulation in the Procurement of Teleprocessing Systems,"

AFIPS 1977 Conference Proceed ings , National Computer

Conference , Dallas, TX, June 1977, pp. 723-727.

WRI76 Wright, L.S. and W.A. Burnette, "An Approach to

Evaluating Time Sharing Systems: MH-TSS A Case Study,"

41
SIGMETRICS , Vol. 5, No. 1, January I97fi, pp 8-28 y0U59 Youclen, W.J., "How to Pick a Winn;r!" stati^ical §f|ian^^^^gli2dustrlal Engineering, Reprint, p p. 81-82,

42
APPENDIX A

Correlation

The independence of a set of measurements can be

assessed by calculating a statistic called the correlation

coefficient [GEI62]. The correlation coefficient for n measurements can be estimated by:

. n-1

(Al) r = E (x(i) - x) (x(i+l) - x)

if]

- 2 where i = l,2,...,n, x is the sample mean and s is the

sample variance of the data. The correlation coefficient is

alternatively called the autocorrelation coefficient, and is

a measure of the correlation between adjacent data points.

A method for testing if the data are independent is to find

the 100(l-a)% (0 <= a <= 1) confidence interval for the true

correlation coefficient, r. If the estimated correlation

coefficient falls within the interval, then the data can be

taken to be independent. To form this confidence interval

for any value of r, determine if:

(^2^ - ^HTHTI) (n-3,l-|)

where t(d,q) is the 100q percent point of the Student's-t

distribution with d degrees of freedom.

Independence of the data is an underlying assumption in

almost all ranking-and-selec t ion techniques. V/hen

measurements are not independent, more observations are

required to come within a desired confidence interval for

having made a correct choice. The problem of correlation

has been addressed in the case of ranking based on means

[DUD77] and an expression for the increased sample size has

been derived. Dudewicz found that for a comparable level of

confidence in the results a comparison based on measurements

which are even slightly correlated (r = 0.3) required twice

as much data as a comparison based on independent

measurements. No such theoretical v/ork has been done to

handle correlation in the ranking procedures based on

percentiles or proportions.

The one approach for dealing with a correlation problem

is to design an experiment so as to be reasonably sure that

the data as collected are independent. Series of

performance measurements which are consistently below or

above the mean of the measurements suggest correlation.

43
There are some system conditions under which this sort of

behavior can be anticipated. In order to reduce correlation

in the data, therefore, data collection should be avoided,

or at least minimized, under these conditions, if such

action is compatible with the experiment's objective. Two

such periods to be avoided are the times immediately after

system start-up and immediately before an announced system

shut-down. Another obvious measurement period to be avoided

is the recovery time after a major system crash. In this

period system usage is inevitably heavier than normal as

users try to re-establish their previous working

env i ronmen t s

.

Not all conditions which may lead to correlated

observations can be so readily recognized and avoided.

Periodic factors in workload with respect to the time of

day, week, month or year, may also cause correlated

measurements. Discussion with system support personnel, or

frequent system users, is often sufficient to discover what

these factors may be. Whenever possible, the experiment

should then be designed so as to avoid data collection

during these times.

Since correlation is essentially introduced into the

observations because they are collected too close together

in time, a further means for avoiding correlation is to

collect data such that the time between consecutive

measurements is as large as is practically possible. For

example, if measurements are being taken on several

different systems, then alternating them in a round-robin

fashion will produce a natural spacing interval. Or, if

several measurements of different kinds are being made on a single system, then a similar spacing technique can be

incorpo rated

.

An estimate of the minimum required spacing between

consecutive measurements in order to prevent correlation can

be made by running a pretest in which 30 measurements are

collected in succession. The estimated correlation

coefficient can then be used to determine if measurements

are independent and this information can be translated into

a spacing interval. There are no strict rules for choosing

an interval of adequate length, but previous experience

suggests that if successive runs are correlated, then

choosing an interval that is about three times as long as

the average script run itself, is a good first estimate.

Thirty more measurements should be collected with this

minimum interval between data points and again tested for

correlation. This process should continue until an interval

which produces independent measurements is derived.

44
CO t- >0 •* M

* ift >o N ca _ 1 f4 f4 C5 K cr

APPENDIX B

Tables

This appendix contains tabled values which are most

likely to be needed to apply the ranking and selection procedures described in this publication to computer service

selection. The tables are arranged to parallel the description of the procedures in section 2.4 and the example

applications of the procedures in section 3-3. The notation

used in each table Is consistent with that used in section

2.h, with a few noted exceptions. A demonstration of the use of each set of tables can be found In section 3.3.

The tables, with their original source noted in brackets,

are divided up in the following way (all tables reproduced

with permission of the copyright holder)

:

Table 1-19 [AME78a] Selection of computer services

satisfying a mandatory requirement

based on proportions; in all of

these tables, the notation P(CS)

represents "the probability of

correct selection" and the

standard proportion refers to

p(mln)

,

Table ?0 [DUD75] : Selection of the best computer

service based on means. Copyright

1975, Akadamie-Verlag

.

Tables 21-26 [SOB67]: Selection of the best computer

service based on percentiles.

Copyright 1967, Institute of Mathematical Statistics.

Tables 27-30 and

Figures 1-8 [SOB57]: Selection of the best computer

service based on proportions.

Reprinted with permission from

the Bell System Technical Journal,

Copyright 1957 , The American

Telephone and Telegraph Company.

In those cases where values other than the ones tabulated

are desired, the analyst is encouraged to refer to the original

source. If the desired values do not appear in the original

source, they may be in preparation and the analyst is advised

to consult the authors of this publication.

45
« © W W N " >C © M « « C5 *

CO N « * «

* in N CO

a M in CO ca in 0> W ® «

— N fj f5 «

— 00 vc C5 * * in >« N

N• — * * a> N in CO — * 00 - — W N M^ © N CO ©

fj * * in * i

1 « — « Ift N * N * N ® M

- — - M W

1 N C9 CO •* a> 1 C^ C5 CO * *

10 S M « in N * — ej ifi N

N CO CO CO

• * ® -< IN « W * CO - CJ \^ \^

1

C5 99 "dm « in >c N CO © CO in CO ©

— — - — N

a N N « C5 * vC N CO © ?4

® © © - CI CO CO *

lA S lO ® lA

-H — N N

© in © in © w w ^ ^ in VO N CO 9^ ©

CO « !

a fc< < ^

i

N I S - I O I

Bd « I O • I ft. I 9 I

su in I • : a I § i §

•

i H I (a I

9

C^] © ^ 03 M © in 9^ M N CO CO CO N * CO PI * 10 >C N CO

- * CO — * 00 — Ift » «

— N « fj CO © N * N #> * <* in vC ^c

" ^ \C CD " * N * rt V© - - fj fj

CO * in — CO « CO * in in

9 Oi ^ ^ CO ve CO - - — — N

ve — N CM N

CJ CO C3 ^ 'J'

© CO ^ ^ CO © M '<J< ^« © ^ tf* CO N

PI N CM C5 C5

»©'-«•* in N CO © in CO — * CO — « CI CM

© ® — M CO * 10 vO N «^ PI >©

© © © © — N CO CO in * N * ©

— — PI CO «

in © in © in © in © in © ©©©©© — PI CM CO C5 * * in v© N CO «> ©

on

iB w

CDCOo

CD S U

— Q CD

O a «i

II s tn ^ O II - «< U S M

w tn u u

a. 2 H Vi _ o _ tc >- as Id -<cpo

P ii] CD •PnCO • O O B Pk CO b. fc O

f- J© u^r • wo J_

CD S pp

O 0.

- H<» & 3b II fa5m

W N * ©

— - M

00 CO N

PI PI es CO * ® d> N >© in

in in <© N CO

GO PI V© CO r- ® * 00 — »

PI pj PI CO CO

CO © 00 >© CO * in in >© N

N• PI in CO — * N © CO ^ © — PI PI PI o

CO O >© PI

CO * * W ^©

• CO ^© CO ^ (MM Tf — - — PJ CM © in — V© PI CO c5 If * in

in — CM >C CO « CO in N Tf 00 c; N PI

PI PI « CO

© PI CO in v© cc CO ^ CO — Lo CO PI - P4 fJ PI CO w ® - CM C5 f in >© N 9^ ® PliO N © C5 - - - PI PI

PI © © - « PI c5 CO * in ^© N «^ ® 2 J

— — — — CM PI CO * in in

1 10 ®—

in ® m ® in ® in ® ©©©©© ''PICM CO C5 * V© N CO iS> © s

* !

n Cb <: gO>IR

p w

cns

* CO N — in 0^ CO CO PI v© —

PI PI CO CO *

1 * CO in *

* 10 V© N ffl 1

CO CM lO *• CM N© tf> CO N -1 *

— PI PI CO CO

1

PI as N * PI 1 * * lO ^C N 1

1 N ^ N © CO v© a* PI in « - PI PI PI

1

1 w - eo * — 1 CO * * in 1© 1

1

•« — CO in CO © CO in CO CO — — — PI PI

a^ ^ a* 10 ©

PI CO CO * m

m © PI « >© CO ® PI '<}• V© CO PI N — >© ©

PI PI « CO *

* © - PI * lO N CO ® PI CO N ® N — 1 — PI PI PI CO 1

CO © ® « PI CO * 10 1- CO a> — *»©*' — — — — — PI

PI © ® ® ^ ^ CM CO CO * w vC CO * — PI

© - - - - PI CO CO * in

in © in ® in — - PI PI © 10 o in ©

CO CO * •* 10 ^© N CO ^ ®

CD « 2

W H

S fa '•s -> S w

BE U

COn

o 46
CM V© ® * CO t) N — ifj CM CM C5 CO CO

ffl * W * CM * Ift N CO

a N lO CO W CO ^ 0^ CO — CM CM CM CO © CO CO ©

* * W N

N — M ^C OmN 115 03 r- «f ts. - - N M CI

CO © >^ CM t?* 1 CO * * in in 1

• ® M TP N CM •* N 1^ CM — - - - CM

1 N CM N CO CO M CO CO * *

® CO Ift N <j> -< CO in N « in o> * CO

CM CM CM f5 CO ©-we?* vC N ® M lO 05 CVl lO CO " - CM rj :i

® ® — CM M CO 'if w N CO ® CM in N a>

® ® ® ® ^ - CM CO CO * in N 00 »

© © O ® - CM M CO *

in © in © in © in ® in © ©©©©© — — CM CM CO CO * 'f in vo N CO a< ©

GCB ^ I

s

M >S « '4< CO

— * N — •*

t- I — CO in 00 —

CM * ® "t CM CM CO CO CO

CO - in CO CM - CM CM CO

^ N © CO ^ — — fJ CM CM

in •* CO - * in * N CO

o> « — a> I CO * in >o >c I

a in - I CO m m I

»c I © PI * -c CO — rt in CO ©

— CM

© — CM * >0 ffi ® 01 If %o

* I o©-":*!* inocoa^ —

— fs. CM CO CO *

® * CO CM N W fj CM CO CO

* t- © + N I — — CI CJ I

Q ® ® — CM CO * in vC N — CO vC 10

CM I ©©©©© — — cicoco *^c^•eo©

©©©©© ©©©«© — — iN iM CO I

in ® in © in — -< CI CM ® in ® Lo ®

CO CO ^ ^ m

CO « !

m fc. -a: ^ g o p» z; D w

Qin

o

N CO 9" ®

O ^Q

cnguo

CO >-] CO 2 »

-1= o 3

* o a <

ij te cn - <

w CO n u

c a P (/! O CO E

— CS K

. Hao — f- u

r^: J => 5 a a

.fl, cq . « o s a. cfj Ek " s; O

S;^2 . W I- E- O

J g < S

U O mJ ^ » b ^

pfi pa o -< o a. IS

fe- eo „ J CO

II •<o

fat

CO N — w © a CJ © CO N *

— - d ca cj CO CO * in in vo N CO

GO N vC 05 ® * — in CM © N in CO CM d CM CO CO * in in N

N N * N © « C5 vc a> in CJ 03 m CJ M M CJ eo •* * in

— CO CO '-' CO ^ CO 'f — — - CM CJ

— ci'*>coo ©CI

© * N o< © c) *

CM CO * Tf in I

CO N M

CJ CM CO CO

t- — Tf C3 — I - N CJ CJ Ci I

© — — CM* W^CNCOO^ CJ * N a> CM

® ® ® -< d CJ CO * in N 03 ® C5 I - _ _

^

i ®©©©© ® — — « CJ CJ CO CO * in I

I

in ® in ® in - - CJ CJ © in ® in ©

CO CO ^ ^ in

COK

«W !H a <: ^ S o > 2;

§

\0 N CO 9> ®

CON— in* coNCi>c© »N>CLnco — — cjcjcoco* 'finvCNco

CJ in CO CJ in art*®* — ^ >c * -. — -' — CJ CM « rt * * m * h.

_ _ _ (M {M CJ rt * * in vc

— CO in N © CM in N © « CO CO 5- * ©

„ ciwco^in

©dcoinN o> — rtinco CJ>c — ino* cjcirtrtrt

I © — CM^LO »Ca3© — M OOvCOvC® I

I „ „ — „ ,

I I

j . ® ® — CM CO Tj> tt5 vC N CO — CO >© CO

— — — — CJ

® ® „ „ CJ CJ e5 * * \0 N » © CM ©©®®® © ® — — — cjcjeoTfTf

in ® in ® in ® in ® in © ©©©©;?

r- -< CJ CM ««** in * N CO * ®

GO « s

fCfa < — OS

CBoao 47
(^ 01 »o ® f CO cj N — w a> CJ fJ Ci 05 «

00 >p ift CO cvj Tf 10 vc N CO

CO — * 00 — Ift CO cj w isn CO — fifl CVl C5 ® N in M ®

* * lO i« N

N• M >c a< oj * N — Tf h" — — fl fJ OJ CO a< lo N CO CO CO * in Lo

® w * vo a> — * vC ^

vO M N CM CO PI CO CO * U"

© — P5 Ift NO CO ® IM * vO e> no i?< CO CO

CM CM OJ CO CO

•*

• ® - M -0 « N ^ ® C) lo CO — in CO — -ttaci

® ® - - CO Tt" W v© N ® CM •<»' N a>

• ® ® ® ® ^ - W CO * in 03 «^ —

© © © ® - CM Cl CO CO

10 ® in ® lA — - W fj ® m ® m ®

CO CO ^ ^ lo N CO ^ ®

« 2

OK

o

o

a.

0. in

C'l NO ® CO fj »C © CO

CM N CO CO CO h- » * CM 1 * in V© N CO 1

^ ^ N ® ^ N — * CO —

- CM CM N CO o» CO -< 00 1 CO * in v© 1© 1

CO in 00 — * N © «i \C - - CM CM CM CM CO * ® N 1 CO CO * Lo in 1

® CM * V© CO ® CO in CO ® in ® v© — >© 1 CM CO CO ^ 1

© — CM * * N CO in a> 05 CO N >© — CM W 05 05 i

® ® - CV] * If; vO CO <3> * N ® CO N 1 — — CM M OJ 1

® ® ® - CM CO •* * in N * 05 Ift 00 1

- — CM CM CO * in N 03 ® 1

® ® ® ® ® © © © © © — — CM CM 05

j

1

in ® in © in © in s in © ® s ® s ®

— — WCM 05 05 •* •* in >© N CO <^ ®

GO

fa <: ^ o > a

Cfi w

OlCOo

fao ^o

aguo

f/i u

en K u

** 3

-SOB?

j( o a <

II S (7; ^. O II — <

U § GO w ai fo td

a, a H :/) o _^ »

S Bfi fa H f- y u

« -3! H U

rJ 3

5 H r/j 'P- CO - o o

is 0. i/j fa r 85©

. J s < s U K 5 B — (1. P

05 CQ O —

O »

fa *

II < O

fa

05 N — in * 00 CM V© — W CM 05 05

0^ CO VD lO *

* in V© N CO 1

1

CO M in 9n CM vc © 05 N — •* CM CM CM 05 05

1

1 CM ON N * PJ * * in V© N

— * t>« ® 05 >© » OJ in — — 01 CM CJ

1

1 in — 3 -1 1 05 TP * in V© 1

1

V© — 05 in CO ® 05 LO CO — CO — — — CM CM

1 * * <jMn ® 1 CM 05 CO ^ in

j

in ® CM * V© M © CM ^ ^ 00

1

CO N "-^ >© ® I CM 04 05 05 •* 1

1 * © — fJ ^ in N a> © OJ 05

1

1 N C * N - 1 M CM CM 05 1

1

05 © ® -< CM 05 * in N CO

1 — *>©«> — - « - - CM 1

1

CM © © © ^ ^ CM 05 05 in

1

1 V© 00 * — 05 1 — - 1

1 ® © ® © © CM CO 05 * LO 1

1

in © in ® in - CM M

© in © in ©

CO CO ^ ^ in

1

\C N CO ^ ®

u

o

n

o

05 V© — in «N CO « v© «

CM CM 05 05 •*

1 00 N© Tf 05 * in 1© N CO

M in CO M in 0> CM »© © 05 — CM CM 05 CO — 00 >0 05 — 1 •* * » <C N 1

1

1 * 1© aviM LO CO — CO — — fj W CM

1 ® N CO © 1 05 ^ * LO >© 1

1 — CO in N ij> C\J N © fj — — ~ CJ CM

1 N CO CO * »> 1 CM 05 05 * 1

1 ® CM 05 in N On — CO in N

1

•-^ N© ® ^ (3n CM CM CO 05 05 i

1

1 O — fJ CO w V© CO <^ — CM

1 O f'l V© 3n 1 -• — OJ CM CI 1

I © © — CM 05 * in vo N 00

1 © 05 in 00 ® 1 - - - - CM 1

t ® ® ® ^ ^ CM CM 05 05 *

1

1 V© N CO © CM 1

© © © ® © © © © - CM 05 05 *

in©in©in ®m®Ln® s®®©© — CM CM 05 CO * in >© t« CO a> ©

COo

OS -

« H

@fa < ^ o > s

CD§

48
-H -H - M CJ «

-*<

N «> « \C

-I

— fJ CJ iM

h. * CO <M I <*> in * N CO I ® N * <N 0> I * * lO vC * I M 3^ W iM CO I M « •* lo in I © \C c> — CO \0 CO — Is. fj N — — — N M CO CO * Tf ® — CO * >o CO ® W •* vD © •* ^ CO N

CM CM CM CO CO * ® ® —' ?r 10 N CO <S 10 CO - ^

CI

<5 ^ _

w (sj P5 10 !<.

N 0^M »c 9v II ®®®®— CJ M CO CO 10 N ® I1 ®®®®® ©©®o® — ^CJCOCO I I0®10©10 ®10® « -I CM M CO CO

PS

10

© * lO

ViO

N CO ^ © , < ~. I > K

OS

w i

0> CJ ^ ^ CO

N

— ^ © ^ 03 ^ lO CO Ci • M M CO CO CO lO < N 5 1 - © ^ fs* — ^ CO — CO CO © 00 1 • CI CM fJ CO CO •<» lO ^0 * 11 N © CO lO CO CO ^ CI to — CO ^ ^ 11 • — — CI CI CO CO •+ LO LO 1 ® CJ ^ CO © CO lO N © LO © lO © 1 • CM C5 CO * -4" 1

lO

®

— CI ^ lO

N ^ — CO lO ^ CON

— <i • — CI CI CO CO ® © — 01 CO 10 O "3^ © CO N © CO -0 • CJ f 1 ri

CO®®© -1 CJ CJ CO * lO >C CO

— C5 lO

N

C^l

©

©®®®

—— CI CJ C5 * lO

NX

©©©®© ^1 --.-,(>] CO

10®I0©10 ©io©io®

.^"CMCM COCO'J'^IO

COzo

Cfi - W H

§fc, < ^ OSOS

01no ®

®®®® * N CO o

Ep Q CO «c <( 3 * o s ^ " ^ «

II S V) ^ O 11 -H <5 /s

oi § s - u <5 m w CO a (d

ft,aH cfl o _ w

a - OS

w

?- EU 'J - <fl F Ij: -J H S S W Cfl -O, CQ - O O s 0. CO h - so U ^ -H .

ta o J

CO z P P3

0-

h

OS 03O - O ft- >

li. to

II 5 o

CO

N

— lO (J- CO CO CI ^ ® 3^ CO LO • M CI CO CO * lO vO I-- CO

CO CJ lO (5- CI vO On CO

N© * — CTv vC * CJ CO • — CJ CJ CO CO * * lO N N - * N ® C5 >c f1 LO 03 10 — N Tf ® N • - - CI CJ CJ CO T(< * LO o — M LO N © C] LO CO © CO CO Tf * © cS • - — — -H CM CJ CJ CO « * lO o

Oh

lO

© CI vc

N 0> CM * >C 00 CMN

— lO

© Oh lO

CJ CI C5 C5

QK

<C

©

— CI 'f !0

N CO

®

— CO

O© CO f>»©

Q - CJ CJ c CO H

CO

CO©® -1 CI CO * LO « N <?-

— CO >o CO CO

CI

®©®

—

— CJ CO CO 10 vO CO ^ " CI CI

1^ ^5 ©©---' CJ CJ CO * *

lO

® lO

® lO

® lO

® lO© ^5 ^5 - — CM CI CO CO ^ ^ lo N CO

©

w! H fc. <( 5 O > K 6 a; w ©

C0vO®'*<> CON

— lO©

——— CJCIC5C0*

CJ lO CO CJ 10 0^ CI vO 0^ CO — — — CI CI CJ C5 — CO^CC^CJ lOCQ— -fN — - - CJ CJ CI

CO

N LO * CO I * lO V© N CO I © CO lO CO © I * * lO vC N

Tf

© \C> CO ^ 1 CO * lO LO I — ~ - - CJ

©

— CO lO

N a>

— CO lO

N I ® — cicolo vco3<r — CJ

N CI CO

CM CO CO — lO ©

CJ CJ CO

lO CJCJ

CO ^^

CO CO

lO o> I CI CI I I ©® — CICO tt-lOvCNCO ©CMLON® iII fj i ®®®®- — CJ CO CO lo N CO © 

I ©®®®® ®©o-- — CJ CI CO I

10®10®10 ®LO®IO® — — CJCJ COCO'J'^IO

GQIE2

eg !i. «s S o > z 5 OS w i ®

®® ^ N CO

49
1 ® * £0

- - *

t" I -< KUC CO —

04 vO ® W

M W fS CO «

CO — lO CO IN — fj ca CI c5

* N ® «

— — CM M M

N ^ CO ~ I * WS v£> N CO I

0^ N 0^ I 00 * 15 vO vO

CM a- W — CO I 05 f) LO W I

1

1 ve 1 • 1

1 © M * CO -• o: 05 — — — -* — CM — O CM N

CM CO CO ^ ^

1

1

1

1 o ^ CO ^ CO © M * »C © * CO W NCM CM f! 05 05

1

1

1

1

1 © ® CO K5 N CO © — N N ^- —' *J ?J C3

1

1 ^ 1

1 ® ® ® -< <M CO 'f W V© N «- — * C3

1

1 fj 1 • 1

1 © ® ® o ^ — CM CM 05 o; l« N 3^ ©

1

1 - i

1 © o o © © © © © © © — -I CM CM 05

1 lO ® W ® lO « — CM CM © u5 © in ©

05 CO ^ U? © © © © ©O N CO

GOo

W H

g fr, <( ^ S O > Z 5 OS w

t»

P3o

o

b

o

Q-, no

CSl vC 05 N " ift © * CO

CM CM 05 05 05

>C LO 05 CM ® * 10 \C N CO

'f* N © ^ N © CO — CM CM C5CO ^ CO © CO

CO ^ lO *o

O 05 115 03 ® CO « M w

— — — fJ CM — N * ® kO

05 05 * LO LO

© '-' CO CO © M 10 N ®

— - - - CJ

lO © LO © LO

CM 05 05 * *

© — CM ^ Ift N a< — 05 U5 <?> 05 N — lO — OJ 05 05

© ® — CM 05 K5 vP N ® 05 « © 05 \C

' fJ C I ^M

© © ® « CM 04 05 10 * CO ® 05 lO t-

© © ® © © — — CM CM 05 lO vP 03

® ® © © ® 1^ 'S* ® — — CM CM

i IO®IO®L0 ®tO©LO® ©©©e® -> — CM CM CO 05 * * lO N 03 ff> ®

en "

^ 2

a ti. <t s © > s n

o

< < , H <!i y a; O 2

o a < u <s - < ^ g

s <M (M a Cd S E- (« O CO ci H

f- M U

< p

saw

a. tQ - e

C-. CO fc, ^2 . --HO 2 <5 Z z 5 a

0-1 H M O "

0- &

iJ CO 1,^1

P O,

I I a^ 05N— lO0^ CONCJ^C® t3^Nv£iLO05 .| _ — _ C'JCMO5 05'<f <i"lOv©I>C0i

I I

I I

I I CO I CM U5 CS CM vO ^5^ CO © ^ „ ^ ^ | •

I — - — CM CM 05 05 Tf lO v£> Nj

I I

I I N I -'*N©M vc^MiOOO Tf — N-*® 1 •I — — CMCMOJ 05'*'*IOvO

I i

I

;

I I -O 1 —OSLON® CMLON®05 03 05ff''t© 1

•

j — — CM CM CM 05 05 * lO

j

I ! lO I ® CM 05 lO N (^ -> 05 lO CO CM * -< lO On I •I — — — ->r>ICMO505O5l

I I

i 1

I I

I © — MTt'Lo Ncio— 05 \p©n^® I

• I — - — - 01 w fj CO I

I I

I

I I 05

I ®®— CM05 * lO ^ N CO -'05>C'C0-<

j

I I

I I 0]| CMCM05Tf* v©N»©MI

'

I " I

I I

i I -<

j ©©©©© — _ CM CM 05 * *

I

I I

I IO©IO©IO ©IO©tO© ©©©®® — OJ CJ 05 05 ^*10 ^ N CO ®

(»Zo

A - W H

g o > z

Z H

tn§

CM vc ® * 03 05 N — lO 0>

CM W 05 05 05

1 CO V© LO * N

* lO vC N CO

M lO CO — lO CO CM vD On 05

-1 CM CM CM 05 ® CO LO 05 S

•* * lO 1© N

— 05 * On CM 10 CO — * N

— -1 N CM

00 ® V© CM o> 1 05 * * lO lO 1

1 © CM * N O- C^l * N » CM - - - - N

1 N CN| N 05 CO 1 CM 05 05 * * 1

j © ^ 05 lO N 0> — CO lO N — lO <J> * CO

CM CM CM 05 05 i

1

1 « — M 05 <? vO N <JN S CM

1 lO CC M lO 0> ! — CM CJ OJ 1

1 © © - CM CM 05 * 10 N CO © CM lo r- o> ©©©©" - M 05 05 •* 10 N CO On —

© ® © © O ©©©©—* — CM CM 05 *

1 I0©10©10 ®io®io® ©©©©© — — OJ OJ 05 05 **10 N© N CO On ©

GOZ

o

« r w H

a ci, <c s o > z 5 cfi w z a§

50
— — CM « W ?5 lO vC N CO

— N — * CO -I Ift CO N o> vc

<s-<CM'*>vc coaM**© e^fcoc^N — — — — CMCMPJPSfS OS—'P-lTf ICNC0*~ TfN — 'i'N

,5, ® ® — C\j CO W * N -I C5 CO ®®®®® — — fleece * * N ®

o®®®® ®®®®® -^-^wcjco m ® 10 ® 10 ® m ® in ® ®®®® — — CJ fj CO CO * * IfS v© N CO iS>

(» « 2

O > K

n

o

CJ 10 0^ CO N — LO <>• •c 0 CO CJ «

CO * LO vC (. CS

^ 'f' ® CO N ® t- CO in CO © N

- cj fj CO CO * in vc >c

® M 10 CO ® CO v© in - N CO ® c « CM rt CO * in in

2 •

I

Oi in

® — coinoo ©cimN© in©Ln®in 1 ____CM C4C0C0*'* I

© — CM * in N o> — CO in <^^ CO r- — in 1 _i _ « — fJ M CO CO I

a •<• I ®o — Mrt TfoNa>c co^o-covc - •—cj f 1 ®®® — — CM CO ij" in >© CO ® CM in t- ®®®®® — — -> N CO * in >o CO !?> ®®®®® ®®®®® ®«^**c]c]

in®in®in oin®in® ©®®®© — — CM CM CO CO * * in vC N CO o> ®

o

B- -

W H

gfc. < ^ D OSOS — K W

CCCPo

g I

CD S U

Q W

-S c 3

it o z

c-i £

II B CC ^ W M >- <r /\ M » a - o 2 CO w CO ffl w

Bu e H t/i o 00 a — c H

H H U (J — P u 5t J D

W (B -c- oa -

Ed a« COso

t3^2 . a - H o

J r < z;

CO z 5 a

cu p

(3^ n o »

O Oh *

II < o

Oi4

CO N — in s^ CO N — * ®

CJ iM CO CO *

<3s h- 0 CO

in \D t> CO 1

1

CO CM in CO CM in flv CO «0 ®

-< CM CM CO CO

1 _ 00 v© CO — 1 * * in v© N

1 N — * >4> OmN in 03 CM Lo CO - - C] CM CM

1

•* ® N M ® 1 CO * * Ln v© 1

1

\0 — CO in N © C'J Ln t- ® CM — — — CM CM

1 00 CO CO * 3v 1 CM CO CO •* •* 1

1

1

in ® CM CO in N a- — CO in N CM >o © in 1 CJ CM CO CO CO 1

1

1 * c — f1 ?Mn V© 33 o> — ?;

1 \C> ro \© ® I — - « fa CC 1

1

CO ® ® — W CO * LO ^© N CO

i CO in CO ®

- - - - CM i

1

CM ® ® ® ^ CM M CO 4' *

1

1 V© N * ® M 1 " " 1 © ® - - —

1

1 CM W CO CO * 1

I

1

in ® m ® in

-1 - CM CM ® m ® in ®

CO CO ^ ^ in

CO * !

« <c ^ a o > K

D BC w

CDg

•© 'S' 'S'

\Ci ^« CO ON ®

CM t© 0 ^ CO CM N — in

CM CM « CO CO

1 CO V© in CO M

* 1ft V© N CO 1

1

CO CM in CO — in CO CM in a< CO — CM CM CM CO

1 © N in C4 © 1 * * in \c N

1

1 N• — rt -© smtm-

-

in CO — * N

— - fl fl CM

CO 3< v© CM i3> 1 CO 0 * in in 1

1

V©

• ® CM * N — * V© a< — — - — — CM

1 N CM N CO 00 1 CM CO 05 •* * 1

1

1

in © CO in V© 00 ® CM * V©

1 — ift a> CO CO 1 CM CM CM CO CO 1

1

1 *

• ® — CM w * vC N * © CM

1 10 X c^i 10 CO 1 — — ri CM f 1 i

CO

• ® ® — — CM CO * in * CO

1 ® CM * N

j

CM ® ® ® ® — M CM CO

1

1 in V© CO (j> 1 -

1

^& ^5 © © ® ®

1

1 — CM CM C? CO

j

1

in ® in ® in — - CM CM © in © in ©

CO CO ^ ^ in v© N CO 9> S

GO a 2

w f- o > z s a w e u

00n

o

51
0^ d ^ ® t(" CO >C> <S> T}' 03 iM N M 05 «

1 N W ^ ^ 1 * in vc N CO 1

1

CO ^ ^ fs* ^ ^ 03 ^ lO CO — M W W t5

1

1 ^ ^ ^ ^ 1 05 ^ lO *0

j N ^ CO W CO N © P5 — - W W

1

1 W CO ^ ^ N 1 05 0) K5 in

j

vC> ® C4 ^ 00 ^ w w CO ® — — — — CJ

1 SO ^ \0 \£i 1 05 0) * •* 1

1 ® CM ^ >C CO ® rt K5

1

s> Tf' CO c] 1 CM CM 05 05 1

1 ^ C O CM ^ lO vfil CO ^ —

1

1 ^ N © ^ N 1 - - fj fJ 01 1

1

GO Q Q ® ^ ^ CO * W N

i ^ 05 V© 03

j

CI 'S' 'S* '© '© - - c:

1 •* V© N CO ® 1 - 1

1

S* s*

1 — - OJ 05

j

1 tf5®10®ll5 ®IA®I15® — — Nd OSOS-**!!?

tn SOS D BS§

® ® ® O ®

^ ^ CO ^

• C^I 1(5 ^ 05 N ^ 1^5 ^ 00

CI W CJ 05 05

ve 05 —' ®

* W >C N CO

CO ^ ^ |s* O 05 N ® ^ N

— CI CI CI 05

03 lO CI © N

05 ^ lO ^© ^

N• ® OJ W N ®—

05 ^C ^ CI lO - — - CI CI " t"^ 05 0» ^©

05 05 * •* K5

• ® 05 W CO ® CI W5 N ^ ^ lf5 ® 1(5 M CI 05 * *

lO ® W CO K5 N ^ CO 00 d N " 115 — CI CI 05 05

•*

• ® © « CI 05 1}' C N O ® CO vp a> d ve - - - CI M

CO ® ® ® ^ ^ CI 05 * Ift ^ CO ® d IS N - - -

• ® © ® ® ® - — - CI CI •* W * N

- ® © ® © © ® © ® o ® ® — d d

U5 ® U5 ® 115 - - CI CM © 115 e U5 ©

05 05 K5 o ® ® o ®

N CO O

en

o > s

as ^

BQO

©

E-

«)§ O)

CO Is < < t f- • - z

X- o

a, u

u • o

SB —

E

o o

H O

cn s

ob CO

U 'J H !=>

pa -

HO

O M

baa

a, H

o Io.

J CO < O

H

CO I • 1

1

1

• I

05 N — W

dlft CO d 115

05 N — ®

N d 05 05

^ 04 O © CO — d d 05 05

LO CO — * 00 - d d CI

CO N vO Tf 05 I ^ 115 N CO I

_ CO vC 05 - I * 115 >C N I

^ ® N 05 © I CO * * Ifi * I

— 05 115 N CI * N © C! — — — d CI

© d 05 115 N * — 05 lO N

I © — CI051O v^CC5^— d

N 05 CO * <3^ I d 05 05 * *

-I 1© ® Tfi ijv I d d 0^ 05 05 I

* ^! 1© vj> I

^ ci ca c! I ®® — dCO •*i«>«NCD ® 05 115 CO ©

— - - - d

® ® 5, _ « deles'"}"* N CO ® d

®©®®® ®®®^^ ^dCOCO* j

115 ® US S 115 — d d

© 1(5 © 115 ®

05 05 * 115

cn ^ 2

o > a

o

^5

vD N 03 ©

CO I

• I

I

d I

d ® * CO d N — W 3^ d 05 05 05

CO >£> LO 05 CI * 115 VC Iv CO

— 115 CO — 115 03 d 115 ij» et — d d d 05 © N 115 d ®

* * 115 v© N

— 05 VC d LO N — N

- - d W d

1 05 ^© d CO 1 0) 05 * 10 LO 1

1 ® d ^© 0^ ~

— — — — d

1 d N d CO 1 d 05 CO * •* 1

1

! © « 05 115 »C ffi ® CI *

1 ® 115 i5> CO CO 1 CI d d 05 05 1

1

1 0 -< CI ri V* N C< ® CI

1 10 a — 10 CO 1 — ci d CI

1 ® ® — (M 05 •* LO N ® d 2 N 2

® ® © ® ^ - d d 05 * LO V© 00 9> —

® ® © ® © ® © ® © — d d 05 05

1 lO®115®10 ®LO®tO® ®®®®® — — dd 05 05 * 1(5 v© N ea

®

CO2

a fc. < —

S O > EI 5 cc^

CO

ISo

52
M C «• CO OJ © 00

e\j CM CO « «

N lo a — * ro N CO 1

1

00 - * N N •* CO N

— IM CM 05

1

1 0^ vO * — ffl 1 CO * lO iC >c

N — « w CO -1 * t>- © e5 V© - - O fj CM

1

1 CM CO * -< N 1 CO CO * W 10 1

1

ve © 01 * V© CO © Ot) to CO ©

1 lO — '-C 1 CJ CO CO Tf *

1

in ® — CJ * vO CO (Jv — « lO

1

(J^ T^ CO CM V©

-> CM CM W CO 1

1 * ® O iM * w >f> CO *

1

1 * N ® CO N 1 -< CM ?J fj 1

1 n © ® ® — C4 « * W vO N

1 9\ <— n CO """""" 1

1 w o © o © © - - CM CM CO

1

1 * VP N OB © 1

1 © © © © © - - N M CO

j

lA ® in © in — — W N

© m © m ©

CO CO ^ ^ 10

1

<Si SiCO ®

o

n

o

CM 10 » CO h- — lO Ov CO CO - - OJ CJ CM CO CO

CO I -i^NSCO N©*N —

-< — — CM M M CO

©CMION® C0vf>3>f4lO _ — _ _ fJ fj

© * CO — ®

>* lO -C N CO

CO lo ej © N I CO lO * >©

— N CO a- I CO CO * * lo I

>© I ©— COION ©W'+N*

lO I © — CM CO lO N l^^ — CM *

I « - -

I

I * I ®o-ic*jeo 'fvCNffl®

* 0^ * © lO

fj 01 CO * -ICO CM * — lO — 01 CM CO CO

CO 5 3- OJ lO I — CM 01 I

I

1 CO I ®®®-.-. OJCOtMO^C C0®OI'<fN

'

I

I

I CM I ©©©©© © — — fJCM ^lOvCNa^

— I «©©©© s©©®® ©- — fif] I

IO®IO®IO ©IO©IO©

CMCM COCO-^^IO

CO « 2

Ct. < ^

o > a=

CDn

o

V© N CO ^ ©

CI

CO u — q CO

- g Q

^

II s cn

II

s w o

- <cP (J -a. CQ • OO COK © O

HO<( a M O

CO s o e. Is E

11

<!

111 ^

O >

CO V© © lO av CO N — lO ®

CM CM CO CO •* CO N lO * CO * lO * N 03

oa

• CMIO CO 01 lO Ov CM \© © CO — CM CM CO CO — CO v© CO —

* * lO V© N

N• — V© CS CMlO CO — * N

— — 01 01 CM ^ ® vC 05 cjN CO * lO lO

V©

•

- — CM lO N 01 N !3v CM N CO 00 CO <^

01 CO CO * *

- © M CO lO N Ov — CO irt N — V© © * av

01 CM CO CO CO

•*

- © - 01 CO lO >© CO » — f 1 V© 3< 01 V© On - - f 1 01 01

CO ® © — CM CO rf LO v© N CO ® CO 10 N ©

01

© ® ® — — OJ 01 CO c: 10 N 03 ® —

_

• © © 9 ® ® © ® © ^ CM CO CO *

IO©IO©IO ©IO©IO© ©©©©© — — OJ 01 CO c5 * 10 ^c N CO a> ©

CO ^ 2

a fc< <c S o > a 5 a; — s u

COm

o

CM ^© ® ^ CO CM >© — lO On 01 01 CO CO CO

CO >© lO CO CM 1 lO >© N CO 1

- * CO - lO 00 CM lO 9> 01 — 01 01 01 CO © N * 01 On i * * lO >o >C

j

1 ^ CO ^© On — ^ N ® CO N

- — CM 01 01

CO ON lo 01 X 1 CO CO * LO LO 1

1 ® CM Tf< 1© c ^ Tf" \© O* —

'

— - — — 01

1 N© CM N 01 N 1 CM CO CO >r * 1

1 ® ^ CO ^ V© CO ® CM * N©

1

1 ® ^ On CO N 1 01 01 01 C5 CO i

1

1 ® S 01 « \0 N CO ® —

'

1 n CO — -If X i CM OJ 01 1

1 ® ® — — CM CO * lO N© N

1 0> CM •* N© ON

1

1 ® ® © © "* " 01 01 CO *

1

1 O N© CO ON ® 1 - 1 ©©©©"*

1 — 01 01 C5 05 1

1

1 IO®LO®IO ®io®io®

— — 0101 0505'*-*lO

GO « 2

gfc. < ^ OK D flS w S U

C/) CQo

© © © © ©

N© N C3 On ©

53
M vc ® * CO M vC ® * CO anm mm

N W CM — * w v© N eo

00 — * N ® * N - * CO M

— (M CM W C5 ^C 05 —' 00

05 * 1(5 vC vC

1 N• — « LI CO —

-

-

Tf* N O 05 *C — CM CM CM

1

1 CM CO * ® N 1 05 05 * lO W 1

1

• © M * CO o cc to CO ® — — — — CM

1 in ® — \C 1 M 05 05 * * 1

1

i

in ® « CM * vc N iJ> -< C3 W <?> 05 CO CM 1 — CM Oi 05 05 1

1

1 * ® ® — CM * i«5 \c CO —

1 * IS. ® 05 N 1 — — OJ ?! OJ 1

j « ® ® ® — w 05 If If 10 N

1 0^ — 05 W CO - _ - 1

^

• ® ® ® ® o - — CM N 05

1

1 * W N 05 ® i

• ® ® ® ® ® *s* ^s^ - — CM CM 05

in ®-I to ® to ® to o to ® ®®®®® — CJCM 05 05 * 10 \0 N CO tfv ®

g

« 2

gfc.OS< D OS w n

o

oH

o

0-

0. lO

9

Pi

M 1(5 05 N — lo a^ 05 N

CM fj CM 05 05 ^ CO ^ ©

10 »e N CO

^ N ® 05 N ® N ^

— CM CM CM 05

CO 10 CM © N

05 * lO *

® CM lO N ® 05 •« ij> CM 10 - - - JM M

— N 05 iJM(5 05 05 * * 10

® — 05 to N © CM Tf N 0> * 0> 3- lO

CM CM 05 05 *

® — CM 05 10 N <S< ® CM CO CM >C © lO

CM CM 05 05

® ® — CM 05 N CO ® 05 « a> CM lo " — - CM CM

® ® ® ^ ^ CM 05 * "5 CO © CM •* N

^1 ^1 ^1 <^ © - « CM CM 05 to V© N 0^

© - - CM CM

to©io©io ©io©io® ©©©©© — — CM CM 05 05 * * 10 >© N CO ©

GO « !

S O > K 5 flS

CO

pao

0 -

COgMu

Ul -J

CO u

" a CO

** §

-z ci a

5t O Z ^

II Bcn

II - <! ^

01 § a ' (J 3 CO - CO H Cd

0- z H

O CO M >- te W

P H W U - <c p 5: J H S » (J CO -O- B - O O

S Ou CO b

- K©

^^2 , W I- H O

J S <C K

U O (J

c» s s n

^ " p- H

OS M O -

o a.*

b. ©

^ii^§

fa©Ul

1

05^C©*9^ 05N-'tO© CONIO'*05 ___ CM CM 05 05 * lO t> ffl

OlIOCOCMlO U>CJvO<3^05 ©C01O05© — — — CM CM CM 05 'J' * lO vC N

— 05»CiS-M lOCO'-'^N <r©^05*

— — -CMOJCl CS^TflOLO

— C)10Na> M'*Na>CM N>CM305tf<

CM05 05'*'*

© — 05 lo N a^ — 05 lo N — lo ® a

01ICM050505

O — CMOtto ^CCOC^ — fl n<J^MlO(J- — - — -I CM CI fj ©© — 0105 * lO * N 00 © CM lO N ©

— — — - CM

I CM I ©®®©— « ij] p) 05 * to r- CO © —

-

1

CM M 05 * I

lO © 10 © to © lO © to © ©©©©© — CM CM 05 05 **10 vc N CO ©

CO "

« !

D PS w

CO

COo

N I

Mv©©TtoS M-0 — 10a^ Nv©'*05CM — — — CM CM 05 05 05 * lO vC N CO

— — — CM CM CM 05 * * to

— 05 '«S— *N©05>0 OSOMOCMOO — — — CM OJ CM 05 05 •* lO 10

^ _ « « 01 05 05 >*"*

©—05**© 00©CM*>© ©*a»05N

— — — — CMCMeM0505 ®® — r5* WNM®— lOCO — *M — - - CM S^3 CI

® © — — CM CO * lO >© N CM * v© cS-

— CMCM0505 LOv©Na>© ©®©®® ©©©©© — — CM05 05

54

in © to © lo © in ® 10 ® ®®®®© — — CM CM 05 05 ** in \e N 00 a< ®

CO

Bs 2

W H m fc, < S o > s

5 PS w

M

PSo
o

s

Oli

^ ® CO 00 CM vo ® Tj" 00 « fj CO CO CO N W M - 1 * in * N CO 1

03 — Tf N © * N — * CO —

— M CM CO

i <3\\D n ^ CO 1 CO * in >c N ® « Ifl CO — Tf VO ^

— — -' CM CM

1

1 iM 03 * ® N 1 CO CO 'J- LO in 1

1

• ® N Tf »Ci CO ® K5 CO ® - — — — M

in ® LO \o 1 CM CO CO * ^ 1

1 ® — 01 * vO N O — CO 110

I CO N vO — CI CM CO CO

• ® ® — M CO LO -0 00 tfv —

1 * N ® CO * 1 - fl CM fl 1

1 ® ® ® — CM CO CO ^ LO

I 0^ — CO W CO

• (J) 'S' — — CM CM CO in N CO

j • 1

1 ® ® ® ® o ® ® ® ® ® — « CM CM CO

1 in ® irt ® lo ~ iM © 10 ® K? ®

CO CO * to

"S* N CO (3^ ®

on2

gfc, «4 ^

O > S » (S w

GO03o

o

o

g

c>i in a> CO N — in 9> CO !<• fl fJ f I CO C5

>C CO — ©

in >e t- CO

00 — * N © CO N © CO N ©

— CI CM fj CO 00 LO C4 (J> N

CO "f LO in »o N © in t- © : >£> » CM If — — - n CI — N CO CMn 1 CO CJ f * in © — CO LO N © CI * <y

1 •*<?>•* 5« in 1 CJ OJ C5 C5 *

1

in © © CM CO in N CO © Cl *

1

00 CI vD © Ln 1 — CI CI CO C5 1 O © — C^l CO LO N CO ©

1

1 CO * 0- Clio 1 — - CI CI 1

I ©©©'-*-• CI CO * LO vf) CO © CI Ti<

j

1

CM

• © © © © ® © - - CJ CJ

1

1 CO in v5 N 0^ 1

1

1

^& ^5 ©' *s

1 © — - N CI 1

1 in © in © in

-< — CI CI © in © in ©

CO CO «}• * in

'S ^& ^» N CO ^ ®

GO 9 Z

lS> 2

11 *

tx

cqO

O ^

O

GOguU

GO U

CO a: cd

>- Q (B

** ** 5 - -Z D 2

» O K <

II as CO

1-1 <s ^

iz; • w (M M H

0, K E- t/1 O GO

35 — o; M

P H U U

- <a H u

3: J c- 3

to w cc -0, H - o o

z; 0. (/J b

t5^2 . J a < a

u o

GO z 5

g

o cu ;s JGO

II «! O

li,

GO iJ

covc©^o> «N — LO® cONLn-*ci

cjcicoco* *invDi>ca

cjinoo-'in a^MvCl3^c; ®coloco®

CJ ci CI CO * in * N

— co-co^cJ inco — *N co®^^>coo^ — « ci CI CJ CO Tf * in LO

vO 1 © CM LO N 3^ CJ N CM N CI CO CO CO

CJ CO « * *

in 1 ® — CO in N o» — CO in N — in © * CO CM CI CO CO CO

1 ® — ci CO in ^0 N — CJ LO <y CJ LO — CJ CJ CJ

CO

1 ® © — N P5 <f in vo N 00 ® CJ in N e

--•-"-CJ

M 1 © ® ® ® - CI CO C5 * in N 03 ® —

® ® ® ^ — CJ CI C5 *

in ® in © in — - CJ CJ ® in ® in ®

CO CO ^ ^ in vO N CO 9< ®

GO2

fd H

a fc< <c S o > a

GO

eao

d >r> © * CO

— * CO —

N I CO CO

CI vc ® in l^^ CJ CJ CO P5 CO

CO — in o> CJ — Cl CJ CJ CO

* N ® CO — — CI CJ CJ

N V© * in

* N

C5 CI ^ N 03

•* CJ 3- I in vo vc I

CJ a^

CO f5

in — 00 I * in in I

I © CJ ^ ^« * CO ^ CO — ~ — — CJ

in I © — CO * CO © CJ * v£>

CJ CO

© *

Cl CJ

vD CJ N I f5 * Tf 1

CO CO N

CJ C5 CO

I ®® — CO* inNco®— TfO — <fffl I

•

I ~ — -' - fI CJ CJ !

© © © — Cl CO m ^ CJ * vCKj'

Cl I ® ® ® © — — CJCJCOCJ in V© N «> © I - I ©©©O® Q^^^o "^^CJCOCO

in © in © in — — CJ CJ © in © in ©

CO CO ^ * in GO ^ 2

gbi,< —

O > K

COCQo

>0 N CO ff> ®

55
0^ N >C (5> « CM ic ® * CO CM CI « CO « N 1ft CO M — 1 Ift * N CO 1

CO ^ ^ N ® N — •* CO —

« ei M CM ?5 OS CO CO i CO * tft iC 1

N « tt lO 00 -1 CO \c 0^ ea 1ft - — f! CM — CO © ^ 1 « CO Tl< Ift 1ft 1 © 01 ^ CO © CO Ift N ©

— - - - N

in o in " ^ 1 CJ CO CO ^ ^ j « N * v« N ff> — CO W Cs CO N I — fj CM CO CO 1

® O — C\| M lO CO o N © CO s© 1 — -< CM Ci OJ 1 ® ® ® — CM CO •* lf> v© C3 — CO 1ft N 1

• ® o o o ® - -1 CM CM CO 1ft N CO 0^ 1

- ® ® o o o o ® o o o - - - CM CO

j W © lO ®

- — N CM © lO ® 10 ®

CO CO * * w

1 © © © ® ©

\£> N CO ^ ©

iz;

8= 2

ffl fc. <9! ^ S O > K P OS w a w

COno

CM LO OS CO N

^ CO N © CO

S CM ift !>• ©

— LO CO N

CM M fJ « CO

\C © CO N ©

— C^J CM CM CO

CO vo CO — *

- — - CM

« * CO — © I * ift C N CO I

CO 1ft N «v N I CO * Ift 1ft \0

© « CO OMfl I CO CO Tfi Ift I

©©CMCOlft NCO®CMTf<

C5 © c>l JO LO ts. CO ©

* as * I CM fJ CO CO * I

CO CM © ^

CM fJ CO CO

CO as M ift I ®®® — — CO ^ 1ft CO © CM * sc

CM I ©®®®© © ^ — 01 CO 1ft ve N 00 I l

i 1-4

I 00©©0 I

I

1ft ® 1ft ® 1ft — CM CM © Lft © 1ft ©

CO CO * * ift a>

OS 2

s ^ a fe, <a; ^ S o E> a

© © © © ©

N CO 9> ©

e w

Q m

Z Q S

o a 5

t-s U li - < ^

M u

e E- te o _ CO - cc a

H M u

P u

J H 3

5 w (Z) ©

cn fc. o ^2 . "HO r < s

ea o -

a. CM- -3 on

C0.O©*C0 CON — lftas CSsCLOTfCM — — — CM CJ CO CO CO * 1ft sC N CO

CvllftCO-"lft COOlsOOsCO ©CO Lft CO® — — "CMOJCMCO 'J'^lftsCN

— cosfcoscM looo — co®sccMa-

\0 1 ® M * N «^ CM * N a- 0! « -< -< - CM N 01 N CO CO OJ CO CO ^ ^

lft 1 © — CO lft N a> " CO lft N — lft OS * CO

01 01 01 CO CO

5t" 1 © — CM CO O N OS ® 01 Lft CO CM lft OS

^ 01 01 fJ

CO

1 ® ® — CM CM CO so N CO ® Ol lft N 9s

CM 1 © © © © " — C>i CO CO * lft N CO © "

® © © © © ©©©'*'-" — CM CJ CO *

lft © lft © Lft — — CM CM © lft © lft ©

CO CO ^ ""J* lft ® ® o o o

^ N CO 0^ ®

on « 2

ffl <e ^

S O > 35 D a; -

is M

ain

o

OlsOSTfOS 01s©®LftOs — — — OJOICOCOCO

CO I — * CO — * CO— IftCOOl — — - OJ CM 01 CO

I —COsCCO— ThN©C0sO

I - - iM CM fI

N sC CO —

lft so N 03

OS N * 01 fls CO ^ lft sO sO

01 <Mft - CO I CO CO * lft lft I

-O I ©Ol^sOCO — COvOCO —

.

I „ _ ^ _ 01

© -< CO * vC CO © 01 If so

a I ® ® -< CO lft N 00 ®

sC — vO Ol N I C>i CO CO Ti* ^ 1

© * CO CO N

Ol OJ 01 C5 CO

N — * N I - Ol 01 OJ I

O © ® — CM CO lft so t- OS -< * so «

CM I ©©©©— -sOlCMCOCO LftvCNOs®

— I ®©®©© ©©©©© ^^oiCMCO

lft © lft © lft © lft © lft ®

— — OIN COCO^-^lft

QC

iz; « 2

m fc, <( ^ SOS o as w

iz; W

X

n

o

'© ^ N X 9^ 9

56
(M ^<> « N

—

C© * CO Cicim <n m « o nfi

© * W vO N CO

CO

— * N

® * N

— * 00

—

— CM

WM C5

CO CO

© CO

CO ^ m ^ N © M W CO -< « 3v (M If) — - — iM M 11 — CO •* © vO 1 CO CO * w 11 « CI ^ CO © CO m N © A © lA © 1 CM 05 CO * * 11 w ® — w * in b, (j> « CO Ift 1 CO N ^ ^ — CM CM CO CO © ® — JM rt w * N *. © CO N © CO \C> 1 — — CM CM CM 1 m © ® © — N CO — CO 10 N a 1^ 1^ - - w N « * lO N CO i> © © © © © © © © © © - -> - CM CO 1 m©iA®io ©io®io© ©©®iss -> — CM CM CO CO 'J" * 1(5 ^C N CO a> ®

on " « H a < S o > a P OS w

IS

H

cn§

CMWtf'CON— «50<CON C'ffI— 

— — CM CM W CC rj * L'5 * t- t-

'-P5>C©C0 -C®CON® tvlOCIaN — ~ CM CM CM CO CO * 1ft m * ©CMinN© co>ceo — * ®*coo-m _ COCO^'^'LO

©—COION OCM^>CO> ^^O-'* -" — -•-> NCMCOCO*

©©CMCOIA NC0©CM^ C0CM>O©^ — — — — fJ C>J CO CO ®0 — CICO *l(5N0O© CMLOJ^fll'S I -I III CM CO * irt vO CO © N »}• vC ®®®®© ©^^CMCM COl/5>©NC0 ©©©©© ^^^^Q ^^«-«^^] I

II5SU5SIO ©m©io© s(Sioso — — CM CM CO CO * in vc N oa <r ©

CO " « 2 w h

CDno O a

t-U

EC

CO U3

<: < a • z o cd

if

O !S 3

II is '/I /\ U II

CfJ * S5 • O < CO w GO a u

D- S H CO O (B S" as M . H W U — «• p c;i 5 Id cfl -a. ta • o o s a. CO - a©

&^2 . W-HO J S < S

Ed 0|-l w a 5 n

Cfl CQ

O - O 0.

Cb CO

II <: o'—'

OM Ea If I M * © * CO CO

N

— in CO vo in * CM

——— CM CM CO CO CO * in o N 00

CM in CO " in 00M « 9> CO ©coiocm©

—— -< CM CM CM CO * * in vO

N

— CO v© cm in o3— * n co

© vc ci i — — — (M CM CM CO * •* in in I

©M'-J'N* « •* * 0^ CM NCMNCOCO — MCOCO** © ~ CO in N o> ® M in N — in i9> * CO eMCjCMc5eo ® — Nco* vCN5>®CM incocMinos . — - ' CM CJ Ci 1

®®-<CMCM CO'^inNCO ©CMWN*

®®®®—

— CM CO CO * int«eoo^

— ©©©©© o®©©^ *^^]iNCO^

in© in

© in© in© in© ©©©©© -> — CM w CO CO * * in vo N 00 ©

CO « ! a fc. <c ^ K W

ODCQO oHoc

Obo> in

CO

CO

M© CO CM * © * <> CM CM f5 CO CON CO

— * in vo h. CO 1 — « N 03 — in CO CM - CM CM CM CO

a>N * — 1 CO •* m * * 1 — M vO CO — ^ N © CO ^0 - - CM CJ M f1 00 W - N 1 CO t: * in in i © CM ^ ^ CO — CO CO - — - - — CM

»C

— vD CI

N 1 CM CO CO * Tf 1 © CO ^ ^ CO © CM ^ ® <t CO CM N

CM CM CM CO es

® S — C5 * inN 00 s —N

— * 1^ 1 - -> fI fJ CM 11 ® © © - CM n fiao 1 (J, — >f> CO 1 ©©©©" — - CM CO CO in * N » ® 1 - 111 — — N CM r5 11

in

®m

® in© in

©m s

-<

— CM CM CO CO in ve

N CO *>

© « 2

CdH

nb < '^ £ O > S5 K U

QDoao 57
oi

o

N V* * « N — V© ® * CO N Ct 05 05 05

v« Ift 05 CI ©

* lO N CO

00 - * N © * N - * CO -

- CJ W d 05 CO >C 05 © CO 05 * lO ^C vO

N © « 10 00 — 05 0* CI w

- — - W M

— N * © «

05 05 * lO 10

^e © C^l CO CO © CM U5 N ©

- - - - N

lO © lO © vC CJ 05 05 •* *

lO ® — CI * Ift N 0> — 05 W 05 N — iC « CJ CJ 05 05 * © © — IN 05 10 vC Is. C\ ® 05 ^ © 75 ^

- — 01 CI M

05 © © © — OJ O 05 'f 115 « CO © 05 in N

• © © © © © — - CJ CI 05 * lO CO »

**^ ^ ^ d CI

lA © 115 ® U5 — — CI N

© U5 © 115 ©

05 05 ^ ^ lO \C N CO 9« ©

« 2

O > K

i

<^ CI lO 05 Is. — lO 05 N

d d d 05 05

»0 d — ij> Tf lo >e N N

CO ^ 05 ^C © 05 ^ © 05 ©

— d d d 05 N lO d OmC

05 'J' lO U5 N © CI W N © 05 W 00 — * — - — d d

1 ® >C d OMO 1 05 05 Tf Tf lO 1

1 « © 05 » N <^ d * \C

t ^ ^ ^ ^ ^ 1 d M 05 05 * 1

1

lO © © d 05 to N CO ® d *

1

CO d vO ® « 1 -I d d 05 05 1

1 * © © - d 05 ^ lo N B a>

1

1 d 10 CO d LO 1 — - — d d 1

i

05 © © © © ^ d 05 10 10 t» © d * ..... ® - - d d 05 * V© N 00 ..... © - - — d

1 lO © to © lO © 10 © lO ® ..... — — dd 05 05 **10 * N CO flN ©

CD " 2

a H

H p

QO U — aw

** 5

it o Is < H a

II » CD U II - «< /N U ^ 01 w CO M W

0- S H Cfi O CO E - C5 Ed H H M U - E u P CO •3- ja ' O O K OL, CO b

r KO

S^2 . M-HO

W O J _ (» 2: 5 n

O Pk EE „ - J CD

II < o

d © * 00 - - _ d N — 10 d d 05 05 05CO * lO 05 d * W v© N CO <^ d « © * 00 d vO © * ^ d d 05 05 05 N V© * 05 — * lO V© N 00 i

1

00 d lO 00 — » 00 d lO 0> 05 — d d d 05 © N lO d ©

* * lO * N

a — * N - * 00 - lO CO d - d d d 05

lj> v© * -> IJ> i 05 * lO >© >©

j

1 N• — 05 ^C d—

lO 00 - * N

- - d d d

05 a> >© d (5> 05 05 * 10 lO

ION N -< 05 * 00 * N © CO >© — - d d d d CO !0 — N 1 05 05 * lO lO 11

OS »© I © d * N ff> « .fl< ,© as — N d N 05 00 O • d 05 05 *'<l' a.

CU lO © — 05 10 N CO © d * V© — lO o> * 00 ddd0505

Q * I © — d 05 * \C N <3» ® d lO CO d lO CO I

I « — -. d 01 CJ ! H

'

CO . . — fvj 05 * lo 1© 00 © d N a<

— d d 05 * lo ^© CO a< ..... ....w ^ddO5C0

lO © lO © lO © U5 © lO © ..... — — d d 05 05 **10 V© N 00 ©

« 2

W H

PQ fc. <C EOS

10

1©

• © d * V© CO -« 05 -© CO — - — - d

« ^ (s, d 05 05 * *

lO © — d * >© CO © d * ^© © * 00 d N

d d d 05 05

* ® ® ^ 05 * lO N CO © — * N — * N

— — W CI CJ

05 ®® ®-d 05 * 10 V© N ffi * * CO

d - - d 05 05 * N© N » ©

— - d d 05

10 ® lO ® lO « « d d© lO © lO ©

CO 05 * * lO <© N 00 9> ©

COK

«Ed !H

n lb cc ^

QB

CQO

58
O ® * 03

CI CI C9 CO M

>C W CO CM ®

TMft \0 N CO

CO *-4 ^ |S* ^ lf4 N ® CO —

-> CI CI CI CO

CO vo CO ® X

CO * U5 >0 vO

N <d |r) CO CO vc a^ CI lO

-I — — CI CI

fs* ^ ®

CO CO * 11? 10

® C^] CO vO CO ® CI » N ®

>- — — — M

in ® ire ® in

CI CO CO * *

to ® ^ N * W t» O CO Ift <J> CO N — »

— CJ CI CO CO

® ® -> CM p: W N ® CO \0 ® CO — — CI W CI

1 ® ® ® « C>l d CO •* w CO ® CO 10 N

'S' — — CI CI « •* 10 vc CO »

Si 1^ ® ® ® ® ® ® — -< CI M

10 ® lO ® lO ® lO ® lO ® ® Si ® ® ®

— CI d CO CO * * lo >c N 00 ff-

®

en

a fa <e E © > !? P OS

o

o

Oh

CLi lO

CI lo (5> CO r- " lo * CO r- CJ CM CI CO CO

>0 * C4 — »

•* 1ft vC t-

^ CO ® CO so ® CO N ®

— CI CI CI CO N N >©

CO -4- m lo

® CI •* N ® CO in 00 — *

- - - CM M ® X CM W

CO CO * lO 1

1 ® -H CO lO N CM ^ ^ ^ 1 CJ CI CO CO * 1

1 ® ® CI CO lo * CO ® CI * CO CM vC ®

-< CM N CO CO

® ® — CM CO * lO N CO <5^ CI 10 CO CM i^n 1 - -< - fJ CJ 1

1 ® ® ® ® ^ cj CO * lo N ® CI •* -©

® — — CJ CJ CO * vC N CO

1

1 ® - - - CI

j

1

10 ® lo ® lo ® lo ® in ® ®®®®® — -'CMCI 0'.'C0'**lO 'CNCOU"® «

. I

a fa < ^ E O > 2

B o ^

SCO

P fcj M)

Oh n - o BO

. »H fH O

o 2

— a. H

Oh

II ^ O

B

O

b

o

Oh

Oh IO

Cl*®'*00 CIN-^lOa" OO^CIOCOCI w -< CM CI CO CO CO ^ IO * N 00 d IO CO to CO Clio 9> CO ® N IO CM ® H- 'XCMCICMCO * * IO V* N

-HcovcijiCi to CO — * N CO * >c CI CO I — " — CI CI d CO « * to lO I ®d*Na^ — NdNdCO

CMCJCO*'*

® — CO IO 00 ® d * « 10 o> CO CO — -H-"-" dddCOCO

c * 1 <s cm v£l^•c^®d wco'-ioeo

I - - M d d

CO ® ® d eo f » v© t- ® d * N

d ® ® ® ® ^ - d d CO •* W vO CO * -<

*?* ® ® ® ® ^ - d d CO «

IO ® IO ® 10 — — d d

® in ® IO ®

CO CO ^ ^ 10 ® O ® "9 ®

^0 N CO 9^ B « !

n fa o ^ B w

QQs

d >c o ^ CO d *© ® ^ ^ d d CO CO CO N IO * CO — * IO vC N CO

OS CO - lo CO d

— d d d n

a> vC ^ — O

CO * IO * >c

N — « IO CO — * N ® CO * -H - CI d d

1 d 00 in — N 1 CO CO * in in

| « ® d 'f >c CO -H CO >C CO — - - - d

1 VO -H N© -H N 1 d CO CO * *

j

10 ® -H d •* >C CO ® d « « ® * 00 d t» 1 d d d CO CO 1

© ® d * in N CO a< — ^ N ® ^ N 1 - — CJ d d 1

CO ® ® ® — d CO in VP r» o> CO CO

d 'S' — — d CO CO * v© N CO ®

<^ 1^ iS* — -H d d CO

1

IO ® IO ® IO -«dd ® in ® in ®

CO CO ^ ^ IO N CO ^ ®

B

w !h

S fa <: ^ 2 O > K p oe: w B H

59
|^^ N Irt »• 05 t» — Ifl © Tf 00 ^C W C9 CM «

CM CM f) CO CC * W \0 N CO i

1

CO — <# N ® * N ® « N

1 CO m CC ® 00 1 — CM N CM « CO * 1ft vC

j N ® CM W CO <S C5 vC «^ N m

1 — r- c5 ® vo 1 — — - CM CI

1

o

0. Ift

CO

® — COLOCO OMIAN® lA®lftOlft I — — — _ (vj CMCOCO*'* I ® CM * Ift N » CO W a>CON"» I — — — CMCMC5C0 I

® © — CM CO W V© N * © CO « * « * I - _ _ _ 1^ f^^ I © © © — dW'PIO^e 00 © CO l?5 N

©©©©© _ ^ fj 51) * tft V© eo ©©©©© ©©©©© ©<-«»^CMM

io©in©tn ®ift©ift© ©©©©© — CM CM CO CO Tf* 1ft * N CO ij> ©

CD

z.

<fi 2

W H

S o > s

D OSs

o

Oi Ift

CM Ift* CO N — W <Jv CO N

CM CM CM CO CO

Ift Tfi cj — <r * Ift >C N N

— CO »C © « V© © CO N ®

- CM CM M CO N * CM 0> »C

CO * 10 Ift <c

® CM •* N ® CO Ift OS — 'J* — -• — CM CM ® <C CM CO IS

CO CO ^ ^ 1ft ® — CO ifl r» «> CM •* «. ^ ^ ^ ^ ^ W CM CO CO •*

© s CM nift 00 © CM « CO CM <C ©

CM CM C5 «

© ® — CM CO Tf Ift N CO * CM 1ft CO ~ Ift 1 — CM CI 1

1 © © © © « CM CO f 1ft t» ^ CI * >©

® - - CM CM CO ^ ^ h« CO

© © © © © ® - — — CI

lft®10©lft ©lft©lft® ®Q©©© CI CM CO CO * * 1ft vC N 00 0< ©

s

« H

a «c

is u

GOn

o

S en S

en u

Q tB < <5 2

-K Q 2

If o IS <^ a- u <

II ts en A O II N- «a; ^

yj a - w on m M

0- K H cn O cn ta -> (S u

-ft. OQ - o o

0. cn r a o t3^2 . J S < iE ^ E ^

OSfflO- o a.» _ - J on

II < o

oH

Oi «

O • I CL, I g i Bu in

CM vC ® Tf 00 CI N — Ift * CO >C Ift CO CM

CI cj CO CO CO * 10 N 00

CO — 10 CO 1ft CO CM Ift I?- CO ® in CJ ©

-1 M W CM CO * T»« (ft vO N

N — CO >o <J> CJ 10 00 — * N CO 0^ « CM 00 - — CI CI CI CO CO * in 1ft

® CI * N — v© — ve CI N CM 00 I — — — — CM CM CO CO 't' * I

® — CO 1ft ve CO ® CM * vo ®lft(?COC0 I _ cjCMcieoco I

o >} I ® — CMco^ vCNff>®o iftO — inM I — — — CI CI CM j H

en ®®rt — CM CO rf Ift (s. © CM '<f N 9'

©©©©-. — CM CM n 1ft <C CO fl> — I ©©©©© ©©©©^ — CICMCOCO

Ift © Ift © Ift © in © in © ©©©©© — — CM CM CO CO * * in vc N CO o^ ©

GO

Bi 2

W H

a fc. <: E O > K D Ofi w

BE H

o

o

g

Cm in

CI so © * 00 CI « © ^ CO

CI CI CO C5 P5

1 N Ift * CI — * 10 * r» CO

- * N - * CO - in CO CI — CI CI CM CO

0» ^C ^ —

CO * Ift vc >e

— CO 1ft 00 — * N ® CO vC - - CI CI CM

CI CO * - N 1 CO CO * in Ift 1

® CM * vC CO — CO Ift CO ®

- - - - CI

V© — N© — N 1 CM CO CO ^ ^ 1

© — CI CO © — CO 1ft © 41 CO CI >c CI CI CI 05 C5

© © — CJ * Ift 00 0^ — ^ N © ^ N 1 — — CM CM N 1

1 © © © — CI CO •<(" Ift >c t» ff' — CO >o CO

— — CJ CI CO « >© N CO ®

- - N M 05

1

in ® in ® in ® in ® m ® ©©©©© — — CM CM 05 05 * * Ift V© N CO ®

GO

So>s P flS w

en

60
ISo

o

On

CM Ift 0< « N — 1ft ® Tf CO

CM CM CO » CO

^£l 1ft CO CM ®

* 1ft vC N CO

^ h* ® 05 N © * —

— CJ CM CM «

CO Ift CO ® N

CO * Ift >0 NO

O M L') CS ® n ^ ij' fi to - - - « fj N CO ® ^C CO CO * ift in

® — CO W CO ® CM Ift N ®

- — — - CM

1ft ® Ift ® 1ft CO CO ^

® " M * lis tv 0> -1 CO lO (5> CO N 1ft — CM CM CO CO

© O - CO ^ \0 N (3^ © n \o n ^ 1^ *-l ^

^ Q 9 M ^ CVi CO W »© CO «i M 1ft N

— « W CM CO 1ft « CO 9-

o — — c^a

lft®lft®lft ®lftSlft® ®®®®o -> « CM CO CO * 1ft vC N CO a^ ®

CO

«: 2

W H

O > IS

(fi

cnn

o

£ •

I g

CU i 1ft I • I

CM 1ft » CO N — Ift » CO N

CJ CJ CO CO

Ift d — »

* 1ft -c t- t05 ® CO nC ® 05 N ®

— M CM iM CO N CM (3> >C

05 4< Ift lO ^C

® CM •* N ® c»i 1ft CO — *

- - - W CM ® « 01 00 Ift 05 05 * * W

® — CO 1ft N ^ — ^ ^ * * *

CM CM 05 05 *

SuSi a m m >C CO ® CM « CO CM « ® *

CM CM CO C5

® © — Pi CO ^ Ift >C CO CM ift 00 — 1ft —' CJ CM

® ® ® ® CM 05 05 <f W t- 0> 01 * -©

® — - N CM <c N CO

^5 ® ® - - CM

lft®lft®lft ®lft®10® — — 01 CM 05 CO * tC Ift vC N CO * ®

GO "

s ? OK

IS u

§

O ~o

Q W

* o s; «i

II 35 73 ^ O II

J M X K U

iSHW O OD

«t F ^

-T E- a

a w CO

0, CQ - o

Oh CO fa

Ss o

. s^n — a. H MO-ClH N

- J CO

©!>< H

> w

CM vc ® * CO CM * — 1ft *

OJ CM 05 CO 05

CO »C Ift 05 CM

'i' 1ft V© N CO

ce — * CO lO CO CM 1ft ij> C4 — fl M CM 05 © N lO 01 ©

* * ift N

N — 05 vO CM * N ® * N

— - 01 CM OJ

05 1ft 01 CO

CO 05 Tf Ift 1ft

® CM * >C ON * >C On " — — — — CM

vC' CM N CM CO CM 05 05 •*

m ® — 05 1ft >0 CO © CM * >e © 1ft ON 05 CO

CM CM M 05 «

* ® - 0* 05 * VP as © CM 1ft CO ift CO — — CM CI 01

OJ ®® — «CM « f Ift Nfil f © W •* N 0>

CM ® ® ® ® - W CM 05 * 1ft vC CO

C5 ^5 ^5 © © © © - CM CM 05 05

1ft ® 1ft ® 1ft -< " W CM © 1ft © 1ft ©

CO CO ^ ^ 1ft

"S" ^C N CO ®

« !

pq fc. i< ^ £ O > K 5 a: w

s

o> CM nC O If CO

_, _, -

oj V© © * CO

CM W 05 05 05 N 1ft * CM — * Ift >C N CO

• - * N - *

r- - N - 1ft CO CM - 01 CM 01 05 *• nC * — CO

CO 1ft N© «C

N• — 05 1ft 03 —

- ^ N © 05 \C - - IM 01 01 01 03 •* - N

03 CO * Ift lO

• © Tf N© ca - 05 1ft CO ©

— — — — 01

1ft — nC — NO

CM 05 05 * *

ift

• ® — OA * NO CO ® — 05 Ift On CO CM nO — CM CM 05 05

* © ® - 01 •* 1ft « CO ON - * N ® * h" - - 01 01 CI

CO ® ® ® - CM CO * 1ft -C N On — 05 >0 ffl

CM — — CM 0^ 05

1 nO N CO ©-

1

• © © © © ©

1 - - OJ 01 05 1

1

1

1ft © 1ft © Ift - — 01 01 © 1ft © 1ft ©

05 05 •* * 1ft ®' ^5 ®' * N ffl On ®

« 2

U E- o > a

COao

61
(N ift ^ W N Ift ^ ^ CO W W C5 «

^ CC> ® * Iff >C N CO

® CO [s. © ^ ts» ^ W W M COCO ID CO O tv « * IB vO

N ® 01 IC 03 O CO ^ w

— — ej c^i N CO O vC CO CO * to Ift

'•O © '-^ W DO © w w © ID S> ID O t'? CJ CO CO ^ ^

LO ® W * W N ^ "-^ CO W |J> CO N ID - CJ W CO CO

O © - i^J « ^ vC ^ © CO vO SmM *

- - - W

« © © © W CO ^ to ^f- CO ® CJ ID N

CI ©) ©) ©1 - c? * ID * CO

© © © © © ©1 (© ©1 ©1 © - - M

© If? © lO

C4 W

® m ® m ®

CO P5 ^ ^ ID ^ N CO 9^ O

CO « 2

a <! — s o > s

5 a; ^

COn

o

*• N ID CO N

- -

— ID * CO N

CM CO CO

ID •* fv) — * ID \0 N N

CO —* CO ^ CO © CO ©

— a IN «

N * N lO

CO * ID ID V©

N © 0^ N © ID CO ^ — - - CI © C] CO ID

rt CO * ID

. © ^ CO ID N ^ —' ^ vO 0» CO CO ^ ^ ^ « w CO « *

ID © © IM CO ID vC 00 © CI ^ CO M © *

« CM N CO CO

® © - M « * ID V© ffl ». ID CO —

- - CJ w

CO © © © © ^ d CO CO ^ ID ^ ^ ^

^9 ©) ©* © - — M fj CO * v©N CO

© © - - (M

ID © ID © (D © ID © ID © © © © ©

— CJ W CO CO * * ID >« N CO ©

m fc, < ^ S o > a 5 OS w

«)g

: < L H < tJ a Q 2

O IS <

U II - < ^ lis-

• 1/1 a u

. Z H 71

' J- w u

< H U

: J r- 3

S W Vj

lu oa • ©

0. r/l (ib KOO

— 0. F

(0 0-

p. ;£ CO« J w

II ^ o

p p.,

cj © * CO CM vC — ID M M CO CO CO CO ID CO CI * ID N CO

CO * CO ID CO CM ID » M

— CM CM CM CO © N ID CM a>

If ID « v©

N

• ^ CO >c —

•

-

-

Tf N © CO N

— - N fj f1 CO i> ID d CO

CO CO * ID ID

• © w Tf * a^ — •* O —

— - — - CM O CI N CJ CO

CM CO CO *

ID

• © " CO * >« CO ® CM If so © ID ^ CO N

CJ W O CO CO

© © CM CO V* N CO © fJ ID 00 — ID CO

CI M f I

1

CO © © CM CO ID ^© N (5> CM

• © © © © *^ — W CM CO •* ID « CO *i —

© © © © © © © © © ^

1 — N CM CO CO

1 ID © ID O ID — - W CJ © ID © ID ©

CO CO ^ ID

CO

OS 2

ffl tt, <( — S O > K

5 OS w

COn

o

© © ©

^ N CO © ©

CI © * CO CI © * CO CI CI CO CO C5 N ID * CM — * ID * N CO

- * N — * N - * CO CJ

•- d N CI 05

1

\© * —1 00

CO * ID >C «

CO ID 00 -" * N © CO — — N f1 CI CI ffl * — N 1 CO CO ID ID 1

© CM * CO © CO ID 03 ©

- — — — CM

ID — <* \© 1 CM CO CO ^ ^ 1

® CI * CO ON — CO ID ffi * CO CM >e> 1 — CI CI C5 CO 1

© ® CI Tf> ID O X — * N © r5 N 1 — — fJ CI CI 1

© © © - CI CO * ID V© N a> CO >© CO

— — CI CM C5

1

1 * v© (>. CO © 1

— - N CI CO

1 ID©ID©ID ©ID©ID© ©©©©© — — CMC! CO CO * ID V© N CO <5> ©

K

«: 2

W E- & OKo; w

COCOo 62
N in (j^ CO N — ift Tf CO w CO ®

— — CJ ?J CO CO Tf irt \o N CO — *N®CO N ® * N -< 00 ID CO ® N — — N M N CO CO-'J'IftvOvO

N I ® Ift 00 ® CO >0 <?- iM lO N CO ® \C

• I — — -• iM IN CO CO W Ift I

I I

CC vC I ® -< CO Irt CO © W Ift N

o I _____ dcocoir^f

a.§ ® CM * 1(5 N * CO W OOCON— lO ___ — CIOCOCO ©©—MCO **^•lS^® CO^OcTCM^S - — — M CI

© ® ® — fj CO * W 00 ® CMIA ®®®®© '--' — MO ^ w ^c CO a> ©©©©© ®®©©® ©—« — MCI

in®in®iio ®io®tn© ©®©©s — — W CM CO CO * * 10 \0 N 05 0> ®

CO

a fc. -s S o «

m

oao

H

od «

o

CM in (S> CO N — « CO N

CM CM M CO CO

in * CM — ir m vc 1- N

—• CO *c ^ CO © CO vC ®

— CM CM CM CO

(N. ^ _ O. ,£) CO * in w -c ® CM * N ® M in CO *

— - - f) M

® vC CM CO *

CO « * * in

© — CO W N ^ _ if ^ CO CO » a> *

CM CM CO CO If ® ® CM CO in vo CO © CM « N — >C © Tf — fl f1 CO CO

(9 O — CM CO Tf in o CO a- iM in CO — * ---r.ci

® © ® ® — CM CO CO * in N ^ — ^

© ® ® ©® ® - r- — CM CO •* in N CO

© © — - M

in © m © in _ _ CM CJ ® in ® in ®

CO -0 ^f * in

S> Si t- CO a> ©

« !

g o > z 5 a: ^

CO

eao

^ a

a e

en u

Sen >J GO K a

" Q CO <<^

K o K <

II 9

^ O II - «« ^

w tn w u

0. z t- t/j o X

K - BC W

H E- M U I- <C H O

5 la 01 -0. m • o ©

s a< « b

X s 5 H)

as CQ o «

o 0-3= JCD

II <o

on i-i

CM >c © Tf CO CM « — in l3^ N in CO CM i — — — ciCMcococo *in>cNco i

I

I — ifCO — * 00iMlOi3<M ©N-*fJ<J> I __ — MiMMCO -"ftJ-WievC

— covco^— *N®«v« coaMOCMCo _ __fjfjfj coeiTt-inin

• © CM * «^ — Tf V© a^ —

— — — - CM

\fi -1 N CM N

CM CO CO * *

w © _ « * V© 00 ® CM ^ <C ® ^ ^ CO N

CM C4 CM CO CO

•* © S CM CO * in N CO © -> in CO — •* 03 — " c^i rj M

CO ® © _ _ w CO * in >c t- (J> CM * *

CM ® ® © ® -* - CM CM CO CO in « 00 tf' ®

^1 ©©©©'- — CM CM CO CO

in©in©in ®in®in® ©©©©© CMCM COW't'fJn vONC0O>®

en « !

® §

»)§

z

o

O

2 ' PC I a. n

<

H

CO

M >e © ^ CO CM ® * CO

CM CM CO CO CO N in f1 — * in >c N CO

- * N - * N - * CO W -< CM CM « CO

o> « CO 00 i CO * in vo >c

1 — CO in 03 — ^ N © CO ^0 — — N M fI

cvj CO * - rv 1 05 CO * in m 1

® CM •* * ca ® CO in CO ©

— - — - CM

in s V© i CM 05 C5 * * 1

® CM * v© ffl <?> — CO in On CO CO M ^ 1 — CM fj C5 05 1 ©©-<«* in « CO — * r- ® 05 N 1

^ 01 CM CJ 1

© © © - CM CO 'f in vo N

1 «> — 05 « CO

— - CM CM 05 * W N CO ® 1

— — CM CM rt 1

in®in©in ®in®in® o®©©© — — CM CM 05 05 * * m * N CO a> ©

CDZ

« 2

8 o > K D (C^ Z U

cnn

o 63
M

W * « N

— in *> <*" 00 w CM fj eo CO ^ ^ CO

©

in

N CO

CO f— ^ ts«© CO N,

® ijiN " — M M fj CO

CO in CO©N

CO ""f in 1©

N

®M 10N <S CO vC a<

ww - - - N ^ N CO ^ ^

CO CO * * in

® eo ID ta

© CM inN *. 1 •* J^ in © w

CM CM CO * *

lA

© -" CM COw ^ CO ^ CO

NN

— in

—

M CM CO CO

® S - CM ?o * \0

N i3>® CO \C CM v£> - - — M fa

w®®

®N CO

W * CO© CM

mN

*S - - — CM

M in vo CO »

'S' 1^ (jj) (J)

©

—— CM

M

v> 9 m 
 in « — Cl® lo© in© « CO in

iS* 'S' 'S' ^ N ® 9^ C>

O S> EI (S ^

isHo

fao • I CL, I o I

cl, in II P I

CM in > CON

— in «• CO

N

CM CMN CO CO in * C-l

— I3^ in * N IV « CO V© CO O ^ CO ^0 © — -> Cl M CO N * vC « * in 10 <e © CM * N ® M in CO - * — — — CM CM © « CM 03 1 CO CO ^ in 1 © — CO in N — •* ve CO CO CO CO "J» ^ i CM CM CO CO •* 1 © © w CO in vO CO © Cl * 1 N — in © *

>- CM Cl CO CO 1 © © - - CO * ift >a s * CM in CO - 1 - — — CI CI 1 © © © © cj CO CO in t» ^ TfH©

^1 ^1

©

—— - N CO * in

N 00

©

© - — Cl

in

© in

© in© in

© in

© ©©©©© r» « CM CM CO CO * in V© N CO 9> ©

cn

a fa <s ^

IEU

GOnS o o " I

CO S 1^

•i «; 2

-K

Q 3 X O is <

II S GO O II — «I ^ ) S -

GO

GCWw (B

0,aH

M

O GO

n >-A

H P H H U

5c 1-3 ED 5 W Ml •0, ffl - o o rp^2 .

-J

a < s u o tJ

i» 5 ca OS CQ O -< O 0. 17

Ct.

©

_ CIJ GO

II <( o

fa©

<!

GO

J

fa ^ fa

CM vc® * CO Cl v© -< in <^

n v© * co ci " — -< Cl Cl CO CO CO * in >c N CO * 03 — Tfi CO— ioo>ci © t» * Cl i> — '-' — Cl Cl N CO * 'i' in i© >©

— covoa^— ^NScOvc eoa^incioo I _ — -fjcici cocO'*inin I

so• © Cl * k© o» CO >© CO

—

——

—— CI

^©

N ClN

CM CO CO * *

in© ri CO ^ *© CO

® Cl if « © ^ Oh CO

Cl Cl Cl CO C5 • © © Cl 03 * in N 03 © — in CO CO — — Cl Cl C! ® ® - — Cl CO <}• in V© t» Cl •<}»© OH © © © ® — — CM N CO C5 in V© N * ® © © ® ® ® ® © © © — — — w CO «

in

© in© in

—— Cl CM

© to® in

©

CO CO * * in

®' 'S *© '©

^0

NX

O

GO

a: 2 w f- « fa <( ^ S o > s 5 ss w

isu

GO§ o

fa§

fa in

Cl >©

® CO Cl >©© -* CO

Clw CO CO CO

N in * Cl - * in V© N CO

— ^ N

© ^ N - * CO CM — Cl CM Cl CO 1

^© CO

— CO 1 CO * in >c V© 1 — CO in CO — * N © CO <© — — Cl Cl Cl

Cl so * ©N 1 CO CO •* in in i ® CM V© 03 ® CO in CO © in © V© --I V© CM 03 CD ^ ^ © — N * <© N <J> — CO in (7> CO CO Cl <© — Cl CM 03 03 ® © — Cl in v© CO Oh — <J" N © CO N — Cl Cl Cl 11 ® ® ® — w CO Tf in in N 9> — CO lO CO - - M M CO * in h> CO ® — — Cl CJ 03

in

® lo® in© in©m© ©©©©©

— Cl CM 03 03 ^ * in ^©

N CO OH©

«i

«: 2

Cd

H

g fa ^ ^ 5 as w

GOcqo 64
c II o

+-> o

01

c: <a lO *" 0)E •(-> u +-> <u

s- (U

s-oo lOE

to

O

0)

•r- ^o c

s- o

Q.

+J JC (O 3 Q.O

Q. c

+-> +-> o

(U </) co

(/) +-> s- (O o r~

M- 3

Q. T3 O

<u Q.

(Uc Eo

s-

+-> *'

+J

(0 IJ 3 CO cr ro(U

•!-> (0

00

CM

00

Ln o LT) CM

CM CM CM CO CO

LD <X)

O 1— CO o o VO C7^ CM 1— <X> CM CM CM CO CO O I— LO ID

00 i—

CO

W3 o un CM CM CM CO CO >=1-

CO ID

O I—

00 I—

CM CM CM

CO

CO CO

O Ln

cyt LOO

(Ti Ln

Ln <y\

r— CM

CM <d- 00 <o cr.

^£5 CO

CO CTl r— O CM CO O CM

0\ I— CO O <x>

CT>

O r—

CO CO

CM 1—

CM Ln

CO

CO o Ln

CO 00 oCTl

LO

CM CM Cvj CO CO CO 'd- O

CM CM CM CM CO CO ^

c>o cy> I— cn CM CO 1— CO in cr> CO 00 CO

CM CM CM CM CO CO

CO in

CM

CM CM CM CM CO CO

00 CM ^ ,— LD 1—

CM CM CM CO CO

CO Ln I— <d- CO CO I

r- CJi CM Ln o «^O 1— 1— CM CM CO CO 1

CM CO I— I— 1— CM CM CO CO

Ln

1— CM CM CO

Ln

CTl <T> CTl

LnCM

CM

CO

CM

CM

CMOCM

00/

CO CO vo

CO o CM Ln

CO CO CO

00

CJ^

1X3

CO ^ o »d- cn

t— 1— 'Cj- 00 O CMLO 1— CM CM

LO o 'S- cri CM CO CO CO

CTl

CTl

OCM

CTl

CTl

O oen

CM CM CO CO CO

U3 <X5

O I— CO CO 1^ CTl CM Ln CTl CO 00

CM CM CO CO f>0

CM CM

<n

00

CO CO

00 Ln

CTl <£) CO LO

CO

CM OJ lO LO

r~. CTl r— «i- CM «dCTl CO 00

CM CM CO CO CO

CTl 00CO

CO

LO

O I—

LO CO r— ^ ocn

CM 1—

CO CO

<Ti CTl CM

CM CM C>0 CO

CO -vf CO CO Ln

LD 00

O CO CO CM LO

CM CM CO CO CO ^ o -—

ID 00

LnoLn

CO

•=3- CM

CO CM CO CO CO 1

P^ o

CM CTl CO r--OCM

CM CM CO CO CO ^ ^

CO <^

Ln

CM CM

CO

CTl CM

CO 1— LD

CM CO CO oCO o

LOcn

cncn

65
4-> COn3 OJ O) 1—

r— •r—

+-> -M C

03 fO3

o c cr

•r- o t -!-> •r- LD

fO -)-> . r— U

O) -)-> Q. 1— t/> O Ol OJ CL CD

sS_cu O 1—

Q. O)

s_ ^

a> O +J u

•1—3 O

CT O CM

<U

S- 5^

+-> O) II co •r- C

fO c r— O

1— o•1— -Q O)

+-> J3 +-> > o

S- CO

<u Q. E

(A o

J= -1- o +-> +-> •r- rtJ 2 .— O 13

»> CL E CD O

E CL

E

3 >— O

o *

Q.

CTi

CO

cn CO r— CM

Lo r-^ 1

I— f— CM

CM

Ln

CM

LT)

CO

CM

CO LO CT» OO — I— I— CM

CO

CM

CM

CO

IDCO

CO

CO

LD

CO

LD r—

ID

cr> r—

CO I—

COLn

CTl

I

ID to

CO

CO00

o

IDo

1— Lnr-»i— ID I— cTir^LOi— co r— I— I— CMCMCOCO-xftOOOO

I— CO LO CT> CO CT) I— r— I— r— CM CM

cr> I— CO I— LD r— I— r— CM CM

CO

LD r—

CO 1—

CO

cr> I— CO 1— cr> r— 1— I— CM CM1

CO Ln

CO CO CO 1

I— r— CM CO

Ln

CO Ln :d crv r— Ln r— Ln I— I— CM CO

CO1^

1-^

cnLn

CTi

CO

C3^

CTl CO

Lno o o o o o o o O in o o Ln o Ln o Ln o Ln cr> Ln <£) vo 00 CO CTl cn C3^

56
fO QJ

4J

-M

C

O

d o~

•r—

O 1 4_> .p- LO

fl3 +-5 • O CO O OJ

CJ CJ r—

QJ "O i- -C

QJO -M

•r- f

^ L| CT O •'~

(I) ^ LO

CM -(-> QJ CM CO -I- E 1 Q 11 QJ o *

•1— (11 )( J2 -!-> fO JC -0

fO fO -Q +-> 1— > O

S- S- 00

O) QC

CO

O

-rO +-> 4-> •1- ro M- 2 .— O 3 E CD O E CL

S- -r- QJ +-> -Q a E QJ E

ZS r— o

2: QJ S- co MS- " 0 »< H- DCT)

00

CO

CTi

1^CM

00

CM

CO

CO

00

CM

CM

CO

CM

00

CO

ID

cn

roLn

ro

cri CM

LnCM

CO

Lnroro

ro

ro

cri Ln

Ln

LnCM

in

cri ro

cri

ro

1X5

ro

Ln

Ln

roCM

ro 1— Ln CTi CM

cri

Ln

CTi

CTi

cn

CO

00

Ln

U3

LnLn

cn

ro

cnCM

Ln

CM

CTi O

roO

ro

cn

CO

LO

cn

Ln

LnLn

cn

cnro

CMO

cn

00

cn

rocn

cn

00

cn

CO

cn

1X5

cnLn

ro

cn 0 0 0 0 0 0 0 0 0 Ln 0

Ln 0 Ln 0 Ln 0 Ln 0 Ln cn Ln LO LO CO CO cn cn cn cn

67
4-> CO

fO (U

(U

•r4-J £

(O rO c 3 o C o-

•r- o 1 4-> •r- LO

<o +-> O

QJ 4-> Q. to O O) <D Q. (A C7) SS- 4-> fO 0) O r—

Q.

re 0) -o O -M

s- o

•r- J=

+-> o- O •r— Q} 2 o S- >^ CO +-> 0)

CVJ to •f— c E o II O) O •1— <u x

-!-> 03 x: o to ro J3 +-> h- > O

S- S- 10

OJ Q.

10 o

JD •1— O +-> at *S O n nd c c od

s- •p- a> +-> ^

J3

a E OJ E 3 O

'oj s- o *

*+- Q_ o LO

r-. o CVJ LO CVJ cn LO

CVl CM ro

m CO ro ro ro C3^ CT> 00 o CVJ 00 ro

CVJ CM ro ^

CO ID LO CO LO LO LO ro

LO cr> ro LO

O r>. ro CM

CM CM ro

oo LO LO LO LO

LO 00 o CM LO LO CM

CM ro

fO CO LO LO ro cr> CO

LO LO CTi 00 o CM ro

IT) LO LO LO LO LO CTv ro r>. CO

ro LO LO 00 o ro LO ro cr» 00 1

CM CM CO

1^ LO LO LD <T> LO cy»

CVJ ro LO r->. 00 sj- LO

CM CM ro

ro CO ro LO ro ro LO

CSJ LO cy> CVl 00 CM

r— t~ CM ro

CVJ ro ro LO ro cy> LO CM ro LO 00 ro CT> CM o o O O O O O o O LO O / LO o LO O LO O lO o LO LO LO LO 00 00 CT> CT>

68
o CO

m CTl IT) 1

IT) CO cn 1

1

1— to 1 ^o CO o CM 00 to o CO CO in to 00 1

oo J— 1

^ m r~ r— 1^ LO r-v CO LD 1

4-> CTl 1

00 CTl un CO

O to

{/) C>vJ CM CO LO to 00 1 fO (U 1 1

O) r— 1— 'r-

+-> -M C

fO fO

C 15 CO 1

CO LO 1 CO LD CO

O

C CT C\J to CO 00 Ln to CM to r—

•1-

O 1 CM CM CO CO in to 00 o CO 1^

+J T- lO 1— 1— r—

fO +-> • 1— u

Q.1

— t/> o CU o)

C2. to cn oo CO CO tn 1— CO f

CTl 1^

S- CTl LO CM 1

CM oo 1— to I

CM CM CO Ln to

O CO to

<U

O 1

f— r— r—

Q. CU

S- CU •o S- -C

CU

O -M

S- (J

•r- x: r— ro 1

cn r—

Ol CO CTl 3 Cf- -»-> O 1^ 1 to 1 00 1^ CO CTl LO o OsJ O- O •!- 1

r— CM CM CO oo Ln CTl CM to

CU S • r— r—

(U S- >)

r~~ -M cu II J3

fO

C r—

O -K 1— O -r- -a

•p- JD <U 1

CTl CO r— 1in CO 1— Ln

4-> n3 ^ CM C^J CM CO cn CO 1

1 OJ CM CO un to cn r

in > o I

r—

S- S- CO

cu Q. E

to o

ja ^ •- O -•-> 4-> •I- 1X3 CTl CTl r-^ LO CTl 1

r— CO CTl ^- S r- o CO CM 00 to in o Ln o CO O 3 V NJ C\J CO to «^ " Q. C CD O C Q.

cu 4-> ^ E CU E n CO un LT) CO 1 CO CTl 3 r— O CM to 00 o CO o ^ CU S- OJ CM CO in CTl CO

to <+-

s_ -> o *

CM 1

LO 1

CO CO '-~ CO to CO CO LO to to 00

CM CO LO o o O o O o o O o o LO O O o LO o Ln o in CTl /* to to 00 CO cn CTl CT^

69
in

CM

<o a> <u 1— c o-

+j

»o co

•I- o

-M 'r- cr> ro +J • »— u 3 Oi 4-> Q-i— CO O (U <U Q. to cn

SS- •<-> 03

0)O I—

Q. O)

S- 0) •O i. JC

<U

O -M

S-3 »4- cr o

<u

s-

+-> to •!- C r— O

•I- JD

-M (O > O

S-(U Ql

to ^ J= O +JS i—

4_> •«- o 2

CD I! Co *

toco

4-> c cnc

S- •!- J3 U E <U 3 I—

to

CLO

Q.

O -K <4- Q_

00

to

ID

CO

C\J

CTt cr> cy> CT> cr> CT> cn

CSJ «^ o C3^ tn cn CM CM CM CO ro CO Lf) 00

cn

cn

<n cn cn

CM

cn cn

to cn

<n cnro

cnoCM

cn

CMCO

<n

<no

cn cn cn cn cn cn cn cn cn cn cn o CO to cn CO CM CM CM CM CVJ CM CM CO CO tn to 00 o

cn cn cn cn cn cn cn cr> cn cn cn

cn tn o in o cn

CM CM CM CO CO in to CO cn

cn cr> cn <n cr> cn cn cn cn cn

cn CM in cn CO cn CM to

CM CM CM CO CO to cn

cn cn cn cn o> <n cn cn <n (Ti cn in cn CO to to in cn CO CM CM CM CO CO in 1^ cn

cn cn cn cn cn cn cn cn cn cn cn

CM <^ o CO CO in 00

CM CM CM CO 5- tn to 00

cn cn cn <n cn cn cn cn cn cn

CO to cn CO cn to o CM

CM CM CO to 00

cn cn cn cn cn cn cn cn cn

cn CO CO o CO to in CM CO in

cn

to

ino o o o o o o o o in o o LO o in o in o in cn in to to rv. 00 CO cn <n cn cn

+-> fO>>

fO o NO <oo

Qi T33 1 O

to • o "3 *r- +J "O to

S- •r- <o +-> "O +-> UJ oo

s- CD o s:

(A -M

</) 0) 4-

O

O

&. to Q. 0)O ^

•r—CL

W)3

OJ ro

*->3 CU E +-> Oo S-(U

OJ T3

s>

C

OJ 3 s

(U 'ro cn- J3 •1— (0 &- +J ro to

4-> H-> 3 C Q

Ol ro 3

CU

+-> ro

+-> OO 2 O I ^ro to ^ > I—1 O

CU . OJ ^ S- ^

70
10

O) 1— 1 •(

+-> -M C

fO ro

C 3 O C CT

•rO 1 --) •!- cr>

(TJ +J . 1— O 3 0) +J

C1.I

— CO

O 0) <U Q. CO

s_ S- -!-> (C

<U

O 1—

O. 0)

S- 0) o s- ^

O) o +->

•r- JC 3 M- •<-> + O" O •!- <U S

(SI s- >, ir> P O) CM

0) CO c o E r— O O "-

•r- J3 Ol II t— M fO ^ <0 J3 4-> > O

S- S- CO CU Q- C

iA

O

J3 JC T- O +-> +-> •1- fO

i+- s ,- O 3

•> Q. E 01 O E CL

S- •!- O) +J ^ X) O E <U E 3 «— O S OJ St/) M- o * o CTi cr> cn cn cn cn cn O CO UD 00 <o

«^ CO cn r— CO 1

r— r—

C3^ en cr> cn cn cn cn cn

CO CM CM cn 1

cn

cn in CO cr» o CO VD

r— 1

00 cr> cri cn cn cn cn

1^ CO o 00 cn <^

un 00

O CM 1

1

1

CTl cri CTl cn cn cn cn cn

CM CO cn CO CM cn o LO UD 00 cn CM Lf) cn r— 1

[—

cr> cn cn cn cn cn cn

1^ CO CTl vo CM CO cn LD

CO cn UD cn 1

CO 1 1

1

un cn CT) cn o-i cn cn cn cn

CO o cr> cn cn LO

CO CO Lf) 00 o CO I— 1

cr> cn cn cn cn cn cn

cn o CO ro CM

CM CM co <^ 1^ cn CM1

1

ro cn CT^ CTl cn cn cn cn cn cn cn

CO CO CT> vo cn o o

CM CM CO «^ LO cn

CM cr> CTl cn cn cn cn cn cn cn cn

CO Ln CO CM 00 CM p~ CO

CM CO LO 00 LO E

4-> 03

M

>i . S-O OO

4->0

JO CM

a; -Q fO -a "3 x: 3 —I +-> Q CO J_ . O 0)

CO QJ

••- S_ •P cn

uj <y) t—

s- a> o -£= CU

(/J +-> osCO

IfO CO o OJ S_ CO 3

CL. O) 1— u to

CO CD -o =S C

QJ (O ••— +J CO zj CU 1/1 Q. .E T- E E 8 s- OJ

a» -E

(L) -O I—

S- E

(U ^— >,

ro -t-> cn-r-

(-1 •<— CO

ro S- S- +j (T3 CU .E >

00

Q -r-

•I- E ^ -»-> ZD 3 CU E Q +-> •r- n3

•r- +J

00 S_ 00 3 S O 1 ^ •!—

(T3 CO x: > I—I o

CU o o o o O o o o o Ln O O) <u Ln o in o LO o Ln o Ln cn -E

Ln CO CD CO 00 cn cn cn cn 1— +

s: 1—

71
J3

03 " (T3 C7>T- E E

•IO

4->

C

U T-

<U CO I

CD ^

S-o O

C

SO <4- -rO )< •r- Q_

+J

fO 4-i 1— </) Z3 fO Q. CU O r—

Q.^

-a

O) -!-> s- u

•I- OJ 3 1

cr <u

OJ </) s-

+-> U) (J

c:o

4-> O >

5- <+- O) o

(/) -Q >> o +->

•I

M- I— O

XI

C fO J3

SO

a» s- E

s-o

Q.o

S-Q.

1/1 O)CD

i-n3EO

O)

-l->

COo

•r"

4-> 03

r 3

Q.O

Q.

CVJ

II

(TV

LO

CM o o CM 00 CM <£) CM o CM

I— «X) CM I—

CO

LO

LO

CO

O CO

CMCMLO I— IT)

Ocy>

LO

00 o00 O

CT) CM

CO

CO

00 1—

CU 1—

in

CM ID Owl Ln

CM

CO

(£1 CM CM

C^J

cr> oo r—

CM rCM

CO CM CM CM r— o LO o tn o LD o un o O CM CM CO ro IT) O o o o o o O o o o

72
00

CM

Q)

(0

cn-i-

£= E

•IO

+->

C o •-

O) CQ

(UCO o c

i- o

<+- -r-

-l-> S- * o

a. Q.o

+-> s-

(/) Q.

<a

O) 4-> I— trt O) +J D1

03 S-m E r- o

•r- ^O

S-oo

+-> fO ! =3Q.O

Q.

SO)Q.T3

CU

S-

_

cr

s->

S-OJ

CO JDO

i)-O

(UE3

0) <u

CO co

4-> U (U O) ^5- +-> s-O CO u c:o

If- T- o 4-> (0

+-> rs

•p- Q. — O

•r- Q. J3

roO

S_a. ^+->

CO

II

CTi

cn

CTi o

in

00oCOoCOo

00 LT) CVJ in CO o LT) o CO CO in CO CM CO CO

CM

CO 00 1^ r—

CM

00 CO

co un Lo I

cn CM CO CO

-v!- I—

«^ I— I—

CD CTi "^l- CO

CO

CM

CTl CMOCM

LO

OCM

LO I—

00 CD CO

00 CD

CO 1—

1^ CD CO I

CM

CO CD LO CO

CO CM o un

CM

CO CO

CM I—

cn CO CO oCM

CTl LO CO CM

CO

CO <^ CO CM CM CM

r—

LOo o LO oCM

LO

CM oCO

LO

CO o LO oLO o o o o o o O o O O

73
(Tl

cn-i- E E

•rO

+J

C

O

<U OQ

<U ^to E

s- o c o s- o

<4- 4- -r4-> o

Q.

D.

CUQ.

CO

O) +J

SO

•I- O)

13 1

CT O)

Oi to >

sO)

(/) JDO

^-o

CUE

Z3 o

DlO

S-Q.

-M

to

CUCD

i.

(T3

CUCo

<u

toEO

II

»-> o

CU

iS-oO 4J

fC

-l-> Z5

•I- CL r— O

•I- a.

03O

sCTi m(Tl oCT)

LO00o00

inooLO

CVJ o O CTl cr> 00 to 00 ID CO CM CNJ CO o

ooo

00

LO

(Ti ID

CO

CO

00

CM

CO

CO

C^J

(Ti COLn ro

oo

CM

cn 00 oLn to

00

oo

CTi CM

CM

00

CM to

cn to Ln oCTi o

CM

oo

oo

ooCM

00

CTi

to Ln

CM

to

00 to Ln oo

oo CM CM

00 00 Ln oo oo CM CM CM CM

LnO o Ln1

oCM

LO

CM ooo Ln

oo o Ln oLn o o o o o o o o o o

74
oCO

<u jQ

(O

cn-r- E E

•rO

4-> E O 'r-

<L) CO a ^to E

5- o o s- M- M- E " O *

•I- n_

4-> 03 +-> I— I/) Q. <U O rQ.

+->

Q. EO o •-

<U -l-> 5- U

•I- O) EO

•r—

4-> S-o

o.o

i_Q.

•*-> «/) O)Ui

icr

(U

ScoEo

•r—

+-> >

(/I o

to

+-> u

O)

Ss_ou

4-o

>>

-M M- r— 0 •-

J3 E fO

S_

O

01 S- ^ Q. E

ZS -E Z +-»

+-> •r—sEo

toEo

+J

<o3

Q.O

Q.

II

CO 00 o CO LO to o to Q «^ o*> CO

CX5 '~

O

tf) cn CM to CM CO rto CT>, CVJ CO

CVJ oo o OsJ 00 If) LO «^ 00 ro

cr> CTv C\J cn LO CM

00 C\J o

If) to LO t—^ LD O)

CM CO 00 'd- CM CVJ oo00 to 00 CO o CO to o LT) to CO CM r—

to

tn CO 00 r-. CM o 00 to LO

CvJ If) CM

to ooto If)

O LO

CO CM >— o CO oIT) 00 LO CTi LO ro ro

Lf) CM o CVJ

Lf) o If) o LO o LO o LO o o CM CM ro CO LO T3 o o o o o o o o o o

75
zo>X

1- o 0 f < a X a

lU < 1 i \\ 1 1\1. ! ' i k »\\ 11 1 (\ p . 4. - -\\V- e\

6\

•fti °l 1''^ V\ leg 6\ 6\ 16 \\\ 1 1 1 1 1 1 t 1 1 ^ ^> •/ON \\\ \\\

-t-tt 1 1 1 1 1 1 1 1 1 1 1 1 1 11 1 II 1-1 

vv i

x>;

u

(UD.

(Uu

•H3

cr

<uu

(Qo

•H

4-1 to>

!-l (UCOO

4-1 •H

r-i •Hou

Q. x: u-i 4J •H • S 11

60C ^-i •H (0

4-> (13 U 01

(U iH

iH

to to M C O O O 0) •H iH

4-1 <U to to 3 W O 01

EXACT

"—

APPROXIMATION

A 11 \\ 11 --\\ t1 -V\

»-\\x_ 1, ii \\ I ! I 1 ' 1 ' ' ' „ 1 ll s V —_ * \\o\\

*/$

V —V \V

V>v

v. S \ "

l-l

O

(U o s cx to

U-t •a O o

0)

Q

l-<

Q

•H 4J

O 3 •H m

01 •H

1-1 p, o

to

g

Q

f\j (/) .CI c 0 o

•H cx [ 1 4-1 to> 4->

O jc

u •d

0) if to

<o to * iH p 1 O x:

*o 4J ij IH •H

tn o u to & u 0)

rj c 0) 0) iH n u 0) o

<u 0) to qm

he 3 O Q 4J

M-l •H

4-1

C (J c O 0) o

•H .H •H

4-1 Oi 4-J 01 to to to

1-1 iH 1-1 > 3 3 4-1 3

bO O.

O tx

•H

O o; 0 u a.

(0

4-"

C

u o

0) -H

1-1 4-1

1-1 to O rH O 3

P- t3 M-( O

Q)

O

u

•H >M-( 3 4J ^

cr -H

01M -l-l

(0 XIo

1-1 p.

to

B C! > x: o O

1-1 4-1 l-l •H

(U •H 4-1 CO >^ X2 o o o p.

60 in o

M-1C »-l o •H II p<

4-1 * c O Ph 4J

o; CO

1-1 iH 4-1 01

01 0) to 60

(0 to 1-1 1

QJ to

r-l I-l z o

MM 4-1 x;

to 4-1

C •H 0 c &

•H o

4J •H 0)

01 to 4-1

C

iH

U

O 3 3 OJ 60 D. "-I (U •H O 0) fa (X CD u \-I.- - I 1'. \-Af\; N\ \» .4\ \ 1^

'{ 1-1,1 - »- —I\ —

9\

61 1 ll 11 >

'•i 1 11 \ 111 \ 1 11 > \ \ \\\\\

»-\\\\\

EXACT

APPROXIMATION

— >\- ' 1 1

CO

4-1 O o

OJ •H

1-1 4-1 ^ to

1-1

O

(U o 3

P- IH do 13 o a

<Uu iH

•H 4-1 to 3 •H •H

ty iH

(U •H i

1-1 In

to XI XI c o o 1-1 •H p.

4-1 c

to x: o

4-1

u •H

•H <H 4->

1-1 CO o

XI o p. o 60 VD o C 1-1 4-1 •H II p. o 4-1 * O Oh 4-1 c <U CO iH 4J (U M (U to 60

(U 10 tou

0) (0 -i u rH iH 3 o

2: <4-l c

at

ith o c

CN •H o

4J •H tu

(U to 4J c

|4 o o 3 3 01 60 P- rH 0) •H O 01 x:

PM P. CO 4-1

76
o>X < S" tu < 1 j 1 1

\. \ it \\ —yI —

rii ' 1 I. 1- I 1 1 1 1 

\

111 \ \

-- 1

6\\ « i 1 II oV— — \J[III \»A \ v A " >V\ 1 \\s\ \ T'' « ll \ ll « It \ ll < III 1 lu

U

Oy iH

a to

>«-< •H o eo c u c o

•H •H

D •H XI

)-l <u H o «n a. g o

to o u o l-l o. o 4-1 p. 4J

CO

to 1-m (U > ON c>o

II to

0) « n Pu o x:

4J 4-1 CO •H o 4_) to

(J

d) c

4J o

to to CD 1

l-l x: o o 4-1 4-1 •H

4J CO c U c o <U o

•H iH •H

(U 4-1 (V to CO to

i-i l-l iH 3 3 4.J 3

60 D. o C •H O (U o

O.

M

a oo 1l-l 14 O o u

a.

>4H la o TI O B *c

In »j >> c o

•H 4-1 •H o 3 -H 4-> D- i-l <U *^ o

^, •s

Cl. o

COc Oi o o w rt •H 4J

to

to 2? 01 > 60

1-4 •H ' u

tu 11 CO o to *^ iH

XIo 6C x: c *^ ij

i*-i tH •H o *^ to

CJ (U c 0) iH 0)c

4-1 o

ID to to

qui he 3 o o 4-1 Z >4-< ti

CO c o c 0 OJ O

CO •H l-l •H

4-1 OJ 4-1 <u CO to to u rH •H 3 3 4-1 3

60 D. o D. •H O O

PM D. l-la

N-A 1\ \l i \\V-

~^1 _ .i ._ \ 1 . —» - \\

_t .

6«\\1I 1 l< 1 l<

^.ikli'^l

—\\\

EXACT

—

APPROXIMATION \> \\ >\\ — \\ ^ 1 ' • • k 1 1 1 1

ii

\\\> ' 'I

CO

4-1 c O o

01 •H

;-l 4J u CO

u o cH

01

u 3 a

>4-l do •a o (X

0)u

•H 4-1 CO 3 •H •H

cr iH e

0) •H o U ab in

CO X) C

•§

Ou

•H D.

4J

C

rva ith fro tlo

(U l-l CO o

X3 D. o 60 00 0 c l-l 14-1 •H II o 4-1 * O P4 4-1 c 01 CO rH 4-1 01 u 0) CO 60

(U CO CO

u

tu CO i l-l t-H iH 3 O

at

ith o c

iri •H o

4-1 •H 0)

0) CO 4J

B

iH

O

O 3 3 0) 60 a l-( 01 •H o a; x:

D. CO 4-1

CO

4-1

B

CJ o

01 •H

4-1

u to

l-l o rH

o; o 3

a. O. -a o od

<uu

•H 4.1 CO 3 •H •H

cr iH

0) •H 1

l-l XI

B

CO •H

CD XI XI C 0 0 u

•H o.

4J e B

CO x: 0 O

4J

M •H

•H <4-l 4-1 <u

CO

O

O a.

•g 60 0\ 0 c l-l H-l •H II o. o 4-1 * O Oh 4-1 B 01 CO iH 4J 01

l-l 01 to 60

01 CD to

M

X2 01 to e l-l iH 3 0 z 14-1 4-1 x;

to 4J c tI o B s

vO •H

O

U •H 0)

0) CO 4-1 E

l-l .-1

OO 3 3 0) 60 O. H 01 •H 0 OJ x:

pti o. CO 4J

77
NBS-114A (REV. 9-781

U.S. DEPT. OF COMM. BIBLIOGRAPHIC DATA

SHEET

1. PUBLICATION OR REPORT NO.

SP 500-44

2. Gov't Accession No. 3. R«eijjj!5nfs Ac(S«$sioB l!lo.

4. TITLE AND SUBTITLE CCMPUTER SCIENCE & TECHNOLDGY:

A Methodology for the Selection of Interactive Computer

Services

5. Publication Date January 1979

5. P^rm^fisg Of^sartizatlon Code

7. AUTHOR(S)

Sandra A. Mamrak and Paul D. Amer

8. Performing Organ. Report No.

9. PERFORMING ORGANIZATION NAME AND ADDRESS

NATIONAL BUREAU OF STANDARDS

DEPARTMENT OF COMMERCE

WASHINGTON, DC 20234

10. ProjecS/tasfc/Wo^ dnit No.

11. Contract/Grant No.

12. SPONSORING ORGANIZATION NAME AND COMPLETE ADDRESS rsfee(. City, st^ie. ziP)

same as item 9

13. Type of Report & Period Covered

Final

14. ^»»isoriflg Ag«>cy Code

15. SUPPLEMENTARY NOTES

Library of Congress Catalog Card. Number 78-600149

I

\ Document describes a computer program; SF-185, FIPS Software Summary, Is attached.

E 16. ABSTRACT (A 200-word or less factual summary of most significant information. If document includes a si gni ficant bibliography or

literature survey, mention it here.) This publication addresse

interactive computer services,

principally on the statistical

interaction between a computer

properties of the methodology

about the probability of havin

are presented to illustrate an basis for a discussion of the

in various procurement efforts

s the comparison and selection of remote access

The comparison methodology presented relies

analysis of measurement data obtained from the

service and a user. One of the most important

is that it incorporates confidence statements

g made a correct selection. Experimental data

application of the methodology, and serve as a cost and appropriateness of using the methodology

17, KEY WORDS (six to twelve entries; alphabetical order; capitalize on/y the first letter of the first key word unteaa a proper name;

separated by semicolons) Comparison of computers; computer measurement; computer

selection; computer services; ranking and selection;

selection methodology.

18. AVAILABILITY [^Unlimited 19. SECURITY CLASS

(THIS REPORT)

21. NO. OF

PRINTED PAGES

1 1 For Official Distribution. Do Not Release to NTIS UNCLASSIFIED 82

(XD Order From Sup. of Doc, U.S. Government Printing Office, Washington, DC

20402, SD Stock No. SN003-003-

20. SECURITY CLASS

(THIS PAGE)

22. Price $2.50

Q Order From National Technical Information Service (NTIS), Springfield, VA. 22161 UNCLASSIFIED USCOMM-DC
ANNOUNCEMENT OF NEW PUBLICATIONS ON

COMPUTER SCIENCE & TECHNOLOGY

Superintendent of Documents,

Government Printing Office,

Washington, D. C. 20402

Dear Sir:

Please add my name to the announcement list of new publications to be issued in the series: National Bureau of Standards Special Publication 500-.

Name

Company

Address

City Stale Zip Code

(Notification key N-503)
1
NBS TECHNICAL PUBLICATIONS

PERIODICALS

JOURNAL OF RESEARCH—The Journal of Research

of the National Bureau of Standards reports NBS research and development in those disciplines of the physical and

engineering sciences in which the Bureau is active. These

include physics, chemistry, engineering, mathematics, and

computer sciences. Papers cover a broad range of subjects, with major emphasis on measurement methodology, and

the basic technology underlying standardization. Also in- cluded from time to time are survey articles on topics closely

related to the Bureau's technical and scientific programs. As

a special service to subscribers each issue contains complete

citations to all recent NBS publications in NBS and non- NBS media. Issued six times a year. Annual subscription: domestic $17.00; foreign $21.25. Single copy, $3.00 domestic;

$3.75 foreign. Note: The Journal was formerly published in two sections: Section A "Physics and Chemistry" and Section 6 "Mathematical Sciences." DIMENSIONS/NBS

This monthly magazine is published to inform scientists, engineers, businessmen, industry, teachers, students, and

consumers of the latest advances in science and technology,

with primary emphasis on the work at NBS. The magazine

highlights and reviews such issues as energy research, fire protection, building technology, metric conversion, pollution

abatement, health and safety, and consumer product per- formance. In addition, it reports the results of Bureau pro- grams in measurement standards and techniques, properties

of matter and materials, engineering standards and services,

I instrumentation, and automatic data processing.

Annual subscription: Domestic, $1 1 .00; Foreign $13.75

NONPERIODICALS

I Monographs—Major contributions to the technical liter1 ature on various subjects related to the Bureau's scientific and technical activities. Handbooks—Recommended codes of engineering and industrial practice (including safety codes) developed in cooperation with interested industries, professional organizations, and regulatory bodies.

Special Publications—Include proceedings of conferences sponsored by NBS, NBS annual reports, and other special

publications appropriate to this grouping such as wall charts,

j pocket cards, and bibliographies.

j Applied Mathematics Series—Mathematical tables, manuals, and studies of special interest to physicists, engineers,

chemists, biologists, mathematicians, computer programmers,

and others engaged in scientific and technical work.

National Standard Reference Data Series—Provides quantiI tative data on the physical and chemical properties of

j materials, compiled from the world's literature and critically

I evaluated. Developed under a world-wide program co-

: ordinated by NBS. Program under authority of National

I Standard Data Act (Public Law 90-396).

NOTE: At present the principal publication outlet for these data is the Journal of Physical and Chemical Reference Data (JPC'RD) published quarterly for NBS by the American Chemical Society (ACS) and the American Institute of Physics (AIP). Subscriptions, reprints, and supplements

available from ACS, 1155 Sixteenth St. N.W., Wash., D.C.

20056.

Building Science Series—Disseminates technical information developed at the Bureau on building materials, components,

systems, and whole structures. The series presents research

results, test methods, and performance criteria related to the structural and environmental functions and the durability and safety characteristics of building elements and systems. Technical Notes—Studies or reports which are complete in themselves but restrictive in their treatment of a subject. Analogous to monographs but not so comprehensive in scope or definitive in treatment of the subject area. Often

serve as a vehicle for final reports of work performed at NBS under the sponsorship of other government agencies.

Voluntary Product Standards—Developed under procedures

published by the Department of Commerce in Part 10, Title 15, of the Code of Federal Regulations. The purpose

of the standards is to establish nationally recognized require- ments for products, and to provide all concerned interests with a basis for common understanding if the characteristics of the products. NBS administers this program as a supple- ment to the activities of the private sector standardizing

organizations.

Consumer Information Series—Practical information, based on NBS research and experience, covering areas of interest to the consumer. Easily understandable language and

illustrations provide useful background knowledge for shopping in today's technological marketplace.

Order above NBS publications from: Superintendent of Documents, Government Printing Office, Washington, D.C.

20402. Order following NBS publications—NBSIR's and FIPS from

the National Technical Information Services, Springfield, Va. 22161.

Federal Information Processing Standards Publications (FIPS PUB)—Publications in this series collectively constitute the Federal Information Processing Standards Register.

Register serves as the official source of information in the Federal Government regarding standards issued by NBS

pursuant to the Federal Property and Administrative Services Act of 1949 as amended. Public Law 89-306 (79 Stat. 1127), and as implemented by Executive Order 11717

(38 FR 12315, dated May 11, 1973) and Part 6 of TiUe 15 CFR (Code of Federal Regulations).

NBS Interagency Reports (NBSIR)—A special series of

interim or final reports on work performed by NBS for outside sponsors (both government and non-government).

In general, initial distribution is handled by the sponsor;

public distribution is by the National Technical Information

Services (Springfield, Va. 22161) in paper copy or microfiche form.

BIBLIOGRAPHIC SUBSCRIPTION SERVICES

The following current-awareness and literature-survey bibli- ographies are issued periodically by the Bureau:

Cryogenic Data Center Current Awareness Service. A litera- ture survey issued biweekly. Annual subscription: Domestic, $25.00; Foreign, $30.00.

Liquified Natural Gas. A literature survey issued quarterly.

Annual subscription: $20.00.

Superconducting Devices and Materials. A literatiu-e survey

issued quarterly. Annual subscription: $30.00. Send subscription orders and remittances for the preceding bibliographic

services to National Bureau of Standards, Cryogenic Data

Center (275.02) Boulder, Colorado 80302.
/.S. DEPARTMENT OF COMMERCE

National Bureau of Standards

/Vashingcon, O.C. 80S34

OFFICIAL BUSINESS

Penalty for Private Use. S300

POSTAGE AND FEES PAID

U.S. DEPARTMENT OF COMMERCE COM-219

SPECIAL FOURTH-CLASS RATE

BOOK



1

I

I


INTERFACES BETWEEN SMF AND OS

OS/360 MVT

1= I PL/NIP

Job Management

Wait time initialization

Job Scheduler

Master

Scheduler

Initialization Routin'

SMF Initialization Soutine

1 1 lEESHFIT

(Initialize

SMCA)

(vWi :;vc 78

I'ur Type 19

IEE3MFI3 (Process

S11FDEFLT)

SMF Open Initialize

( lEESMFOI

SMF Writer

( ffiESMFWB

Initiator/ Terminator

1

Command Scheduling

Routine

Record Transfer

Routine

(IGCOOO8C)

1

VARY Syntax

Check Routine

HALT/ SWITCH Command

Processor

Allocation

Routine

( lEESMFAL)

SW VARY

Record

Handler

SVC 83 Open

Routine

(lEESMFOP)

f.vrc T;/pe Type Type

7 9 11 12

Record Record Record Record

Output Writer

Reader/

Interprete:

User Job Validation

Exit

(lEFUJl)

EXEC Statem

Proces

(Builds JOT, JMR, and

Job act)

SCT and Step ACT)

Task Management

Main Storage Timir g Super Supei visor

Routine

(CMSMFCRE) Routine

(FM3MFCRE)

SMF Time

Limit Ex Routine (IFATLEX

/svsajT piratlon

T)

User Time

Limit Expiration

Routine

( lEFWTL)

System Wait

Time Col- lection Routine

(lEAQWArr)

Data Management

Rofilon Size

Dete mlnation

Routine

(TWEIPIOI^

Exit

Inltlalii-.ntlon Routine

(.lEFSMFIE^

(Builds TCT, reads jm and

act's ip from

Job queue)

User Job Initiation

Exit

(lEFUJl)

User Step

Initiation

Exit: (lEFUSl)

I/O D

Alloc

Routi

2vice ition

nes

Alloc Recov

Routl

(lEFX

atlon

sry

lie JBIP)

Attach

Routine

IEFSD263

TCTIOT Build

Routine

(lEFSMFAT)

(Builds TCTICT

and fills in TCT)

JOB Statement

Condition Code

Processor

( lEFVJIMP)

(To vfr'ite Type 1; Record for all but

last Job step)

Step Terminatior Control Routine (lEmiMP)

Job Termination

Control Routine

(IEFZAJB3)

(To write Type h Record for last step in job)(To write Type 5 Record)

Input /out put open/cloge/

Supervisor End-of-Volume

1

User Accounting

LinkEige Routine

(lEFSHFIK)

SMF Termination

Record Construction Routine

(lEFSMFWl)

EXCP

Counter

(lEASHFEX)

3YS0UT Limit Expiration

Routine

( lEATLETXT

BSV

Rout ln<;

( IGi';OO09A

User Step and Job

Termination Exi

( lEFACTPTl

luilds Type h

• Type 5 Record)

Type

k or 5 Record

User SYSOUT

Limit Expir- ation Exit (lEFUSO) <3,

EutVi-Jr rFfJ0202H

Close/EOV Record DASD

Extena Ion Routin (irG0202I

Catalog

Management

nCRA'J'CH Routine (ICGO29OD)

(Note; these two components ari unaccounted for by SMF and are Hdown only Tor the soJte of

.

RENAME

Routine

(IGG03OO3)

Type 18 Record

LSPACE Routine

(IGC0107H)

Type 19 Record