A11103 D-^IQll^

NATL INST OF STANDARDS & T^^^^

11030901 19 ICE 8l technology

Application of Measurement

Criteria in the Selection of

Interactive Computer Services

NBS Special Publication 500-58

U.S. DEPARTMENT OF COMMERCE

National Bureau of Standards

500-58

lO
NATIONAL BUREAU OF STANDARDS

The National Bureau of Standards' was established by an act of Congress on March 3, 1901. The Bureau's overall goal is to strengthen and advance the Nation's science and technology

and facilitate their effective application for public benefit. To this end, the Bureau conducts

research and provides: (1) a basis for the Nation's physical measurement system, (2) scientific and technological services for industry and government, (3) a technical basis for equity in trade, and (4) technical services to promote public safety. The Bureau's technical work is per- formed by the National Measurement Laboratory, the National Engineering Laboratory, and

the Institute for Computer Sciences and Technology.

THE NATIONAL MEASUREMENT LABORATORY provides the national system of

physical and chemical and materials measurement; coordinates the system with measurement

systems of other nations and furnishes essential services leading to accurate and uniform

physical and chemical measurement throughout the Nation's scientific community, industry,

and commerce; conducts materials research leading to improved methods of measurement,

standards, and data on the properties of materials needed by industry, commerce, educational

institutions, and Government; provides advisory and research services to other Government

agencies; develops, produces, and distributes Standard Reference Materials; and provides

calibration services. The Laboratory consists of the following centers:

Absolute Physical Quantities^ — Radiation Research — Thermodynamics and

Molecular Science — Analytical Chemistry — Materials Science. THE NATIONAL ENGINEERING LABORATORY provides technology and technical ser- vices to the public and private sectors to address national needs and to solve national problems; conducts research in engineering and applied science in support of these efforts; builds and maintains competence in the necessary disciplines required to carry out this research and technical service; develops engineering data and measurement capabilities;

provides engineering measurement traceability services; develops test methods and proposes

engineering standards and code changes; develops and proposes new engineering practices;

and develops and improves mechanisms to transfer results of its research to the ultimate user. The Laboratory consists of the following centers:

Applied Mathematics — Electronics and Electrical Engineering^ — Mechanical

Engineering and Process Technology' — Building Technology — Fire Research —

Consumer Product Technology — Field Methods.

THE INSTITUTE FOR COMPUTER SCIENCES AND TECHNOLOGY conducts

research and provides scientific and technical services to aid Federal agencies in the selection, acquisition, application, and use of computer technology to improve effectiveness and

economy in Government operations in accordance with Public Law 89-306 (40 U.S.C. 759), relevant Executive Orders, and other directives; carries out this mission by managing the Federal Information Processing Standards Program, developing Federal ADP standards

guidelines, and managing Federal participation in ADP voluntary standardization activities; provides scientific and technological advisory services and assistance to Federal agencies; and

provides the technical foundation for computer-related policies of the Federal Government.

The Institute consists of the following centers:

Programming Science and Technology — Computer Systems Engineering.

'Headquarters and Laboratories at Gaitliersburg, MD, unless otherwise noted; mailing address Washington, DC 20234. 'Some divisions within the center are located at Boulder, CO 80303.
COMPUTER SCIENCE & TECHNOLOGY:

Application of Measurement

Criteria in the Selection of

Interactive Computer Services

Paul D. Amer

Center for Computer Systems Engineering

Institute for Computer Sciences and Technology

National Bureau of Standards

Washington, DC 20234

U.S. DEPARTMENT OF COMMERCE, Philip M. Klutznick, Secretary

Luther H. Hodges, Jr., Deputy Secretary

Jordan J. Baruch, Assistant Secretary for Productivity, Technology, and Innovation

^. NATIONAL BUREAU OF STANDARDS, Ernest Ambler, Director

Issued April 1980
Reports on Computer Science and Technology

The National Bureau of Standards has a special responsibility within the Federal Government for computer science and technology activities. The programs of the NBS Institute for Computer Sciences and Technology are designed to provide ADP

standards, guidelines, and technical advisory services to improve the effectiveness of computer utilization in the Federal sector, and to perform appropriate research and

development efforts as foundation for such activities and programs. This publication

series will report these NBS efforts to the Federal computer community as well as to interested specialists in the academic and private sectors. Those wishing to receive notices of publications in this series should complete and return the form at the end

of this publicaton.

National Bureau of Standards Special Publication 500-58

Nat. Bur. Stand. (U.S.), Spec. Publ. 500-58, 90 pages (Apr. 1980) CODEN; XNBSAV

Library of Congress Catalog Card Number: 80-600035

U.S. GOVERNMENT PRINTING OFFICE

WASHINGTON: 1980

For sale by the Superintendent of Documents, U.S. Government Printing Office, Washington, D.C. 20402

Price $4

(Add 25 percent additional for other than U.S. mailing)
Application of Measurement Criteria in the

of Interactive Computer Services

Selection

CONTENTS

Page

1.0 INTRODUCTION 1

2.0 SELECTION OF INTERACTIVE COMPUTER SERVICES . . 3

2.1 Classification of Performance Criteria • • 5

2.2 Application of Performance Criteria • • 6

2.3 Methodology for Selection 9

3.0 APPLICATION OF PROPORTION CRITERIA .... 11

3.1 Proportion Criteria 12

3.2 Procedures for the Application of Proportion

Criteria 15

3.2.1 Selection of Services Satisfying a Single Proportion Criterion ... 16

3.2.2 Selection of Services Satisfying a Dual Proportion Criterion .... 20

3.2.3 Selection of Services Satisfying an

Upper-Lower Proportion Criterion • • 25

4.0 COMPARISON RESULTS 29

4.1 Summary of a Case Study 29

4.2 Application of Proportion Criteria ... 31

4.2.1 Selection of Services Satisfying a

Single Proportion Criterion ... 33

4.2.2 Selection of Services Satisfying a Dual Proportion Criterion .... 37

4.2.3 Selection of Services Satisfying an

Upper-Lower Proportion Criterion • • 37

5.0 DISCUSSION AND CONCLUSIONS 38

6.0 ACKNOWLEDGMENTS 39

7.0 BIBLIOGRAPHY .... 40

i i i
FIGURES

Page

Figure

1. Computer Service Selection Model (CSSM) .... 8

2. Application of a Single Proportion Criterion ... 18

3. Equivalent Formulations of a Dual Proportion Criterion 21

4. Typical Cumulative Distribution Function F(i) of

Observations from Service (i) 22

5. Application of a Dual Proportion Criterion ... 24

6. Application of an Upper-Lower Proportion Criterion . 27

7. Hardware/Software Configuration for the Case Study . 32

TABLES

Table

1. Examples of Performance Criteria 7 2. Summary of Ranking and Selection Terminology • • 14

3. Scenario for the Case Study 30

4. Statistics from the Case Study 34

APPENDICES

Appendix

A. Tables for Dual Proportion Criteria 43

B. Tables for Upper-Lower Proportion Criteria ... 64

iv
DISCLAIMER

This publication discusses three procedures which may be

utilized in the experimental design - data analysis

components of a remote access computer service selection.

It is not intended to represent the entire government

computer service procurement process. Consideration of

testing costs, bidding procedures, environment test

conditions, and other selection issues are beyond the scope

of this publication.

V

Application of Measurement Criteria in the Selection

of Interactive Computer Services

Paul D. Amer

ABSTRACT

This publication emphasizes the data analysis component

of the computer service selection process. A computer

service selection model, introduced in NBS Special

Publication 500-44, is presented and three binary type

selection procedures applicable in the measurement phases of

that model are given. A binary type procedure determines

which competing computer services perform above and which

perform below a specified performance level. Those services

determined to perform below a specified performance level

can be eliminated or penalized depending on whether the

performance specification is mandatory or desirable. The

procedures explicitly specify prior to measurement an

appropriate decision rule and/or the number of test

measurements required in a comparison effort to attain a given level of statistical confidence in the results.

Experimental data from a previous case study are reanalyzed

to illustrate application of the selection procedures. If

the collection of measurements is necessary in a computer

service selection, then these procedures (or similar

statistically sound procedures) should be employed because

of their mathematical justification.

Key words: Binary selection; computer comparison;

computer performance measurement; computer service; data

analysis; experimental design; performance evaluation;

ranking and selection; selection methodology.

1.0 INTRODUCTION

NBS Special Publication 500-44 [MAM79a] (herein

referred as SP 500-44) addresses the comparison and

selection of remote access interactive computer services in

terms of a statistical comparison methodology. That

methodology relies principally on the collection and

statistical analysis of measurement data obtained from the

interactions between a computer service and a user. Four

1
general test procedures or experimental designs are

described for use in the data analysis component of the

comparison and selection process.

Three of the procedures in SP 500-44 determine the

"best" of a set of computer services where best is defined

by a single performance index and either a mean, percentile

or proportion statistic. For example, one of the procedures

might be employed to determine the computer service with the

shortest mean response time for all text-editor

transactions

.

The fourth procedure in SP 500-44 determines a subset

of computer services that includes all services which, for a particular performance index, perform at or above a specified "single proportion" performance level. For

example, the procedure will determine a subset of computer

services that includes all services for which at least 80%

of the response times are less than two seconds. This

procedure is a "binary" or "yes-no" type procedure since for

each service a decision is made either to include it in or

to exclude it from the selected subset. A binary type

procedure provides an analyst with reliable performance

information about which services perform above and which

services perform below a specified performance level.

This Special Publication also emphasizes the

experimental design - data analysis components of the

computer service selection process. Introduced are two

binary type selection procedures which are extensions of the

binary type procedure in SP 500-44. The first procedure

determines a subset of computer services that includes all

services which, for a particular performance index, perform

at or above a specified "dual proportion" performance level.

For example, the procedure will determine a subset of

computer services that includes all services for which at

least 50% of the response times are less than two seconds

and at least 80% of the responses are less than fWe

seconds. A dual proportion permits greater flexibility than

a single proportion in defining measurable performance by

specifying two, rather than one, statistical values of

interest of each service's performance distribution.

The second procedure determines a subset of computer

services that includes all services which, for a particular

performance index, perform at or above a specified "upper

proportion" performance level and excludes all services

which perform at or below a specified "lower proportion"

performance level. For example, suppose computer services

2
are defined to be in one of three categories: (1) those

with 80% or more responses less than two seconds, (2) those

with 50%-80% responses less than two seconds, and (3) those

with 50% or less responses less than two seconds. This

procedure will determine a subset that includes all services

in category (1) and excludes all services in category (3).

An upper-lower proportion combination permits greater

control than a single or dual proportion by regulating just

how poorly a service can perform and still be selected due

to variability present in a measurement study.

An important characteristic of the procedures in

SP 500-44 and here is the analyst's ability to control

(limit,' bound) the probability of obtaining an incorrect

selection (further explained in Section 3.2). These

procedures explicitly specify prior to measurement an

appropriate decision rule and/or the number of measurements

required in a computer service comparison to attain a given

level of statistical confidence in the results. Although

each procedure compares services for a single performance

index, together they may be employed for several indices

with the individual results then integrated in a comprehensive selection methodology (see Section 2.3). The

procedures specifically are applicable in a comparison

effort whenever a selection methodology requires binary

information about a set of competing computer services, i.e.

which services do and which services do not provide a certain level of performance.

If the collection of measurements is necessary in a computer service selection, then these procedures (or

similar statistically sound procedures) should be employed

because their mathematical foundation permits an analyst to

justify and document the selection making process.

2.0 SELECTION OF INTERACTIVE COMPUTER SERVICES

Federal Information Processing Guidelines for the

Measurement of Remote Access Interactive Computer Service

Response Time and Turnaround Time [FIP57] indicates that

when two or more computer services are being compared,

statistics and statistical techniques should be employed for

the description and analysis of the measurements which are

collected. Further, since the statistical techniques which

are employed can influence the results, the techniques must

be clearly specified in advance. With regard to the kind of

3
statistics to employ, the guidelines more specifically

state

:

"Quite often the distributions of response time

and turnaround time are not known (in closed

mathematical form) . Since most of the data

distributions which have been collected and

analyzed have been non-normal (non-Gaussian)

,

Gaussian statistical descriptors such as the mean

and standard deviation are not appropriate. It is

therefore recommended that non-parametric

statistical descriptors be employed when

specifying response time and turnaround time."

Performance data in a computer service selection

usually are collected in an uncontrolled environment. That

is, an analyst exercises little or no control over the

software, hardware or external workload of a computer

service during the test periods. (+) This mode of testing

is in accord with the actual mode of utilization of a computer service once it has been selected. Users do not

have, nor do they desire, control of the technical aspects

of how services ..are produced or how they are delivered when

they purchase computer services (in contrast to computer

systems) [ABR77b] . However, they do desire a guaranteed

level of service at their terminal interface and that is

precisely what is measured. Due to the many uncontrolled

variables present in a typical computer service evaluation

effort, there is an unknown degree of variability in the

collected measurements. Therefore, it is necessary to

employ valid statistical techniques and to collect multiple

measurements to confidently obtain an accurate estimate of a computer service's performance. This publication addresses

the questions of what data analysis technique and how many

measurements are necessary for a desired level of

statistical confidence in the comparison and selection

results.

A computer service selection model (CSSM) is presented

in SP 500-44. Based on that model, a methodology is given

which relies principally on the statistical analysis of

measurement data obtained from the interactions between a computer service and a user. The methodology incorporates

(+) Complete specification of the testing environment is

beyond the scope of this publication.

4
confidence statements about the probabilities of correct

selections made at various phases of a computer service

selection process. The technical and economic feasibilities

of applying the selection methodology are verified in a case

study comparing four heterogenous remote access time sharing

services. (It is recommended that the reader study

SP 500-44 for a global view of the selection process. Some

portions of SP 500-44 have been summarized in this SP for

completeness.

)

This publication presents selection procedures beyond

those previously published in SP 500-44 for use in the

measurement phases of a computer service selection when the

criteria for selection are measurable and are expressed in

terms of proportion statistics. Specifically, they are

applicable when the analyst wishes to know only if a service

performs above or below a specified performance level.

One of the four experimental procedures that are

presented in SP 500-44 (Section 2.4.1) is repeated here to

assist the reader in understanding proportion criteria and

to facilitate presentation of the two additional procedures

that follow. In conjunction with SP 500-44, this

publication furthers the development of a statistical

methodology for the comparison and selection of interactive

computer services.

2.1 Classification of Performance Criteria

The computer service comparison and selection process

is a complicated one, based on various performance indices.

Some indices are measurable (such as service response time)

,

and some are not (such as ease of service use and coherence

of service documentation) . For example, evaluating service

documentation and the amount of main memory does not require

measurement collecting activity, whereas evaluating service

turnaround time and response time clearly requires

measurement. When criteria are based on measurable indices,

comparison requires collecting and analyzing relevant

performance measurements from the various computer services

being considered.

Performance criteria can be divided further into those

which are mandatory and those which are desirable. A

mandatory criterion defines a service requirement which must

be satisfied by the computer services being considered for

selection. Failure to satisfy even one mandatory criterion

results in a service's elimination. Desirable criteria, on

5
the other hand, are those which are not absolute

requirements for acceptance, but which if satisfied would

facilitate processing of a user's workload. Therefore, if a

given computer service does not fulfill some desirable

criteria, it would continue to be considered for selection,

but depending on the overall selection methodology failure

to satisfy each desirable criterion would perhaps invoke

some associated penalty on the respective service.

(Likewise, for each desirable criterion satisfied, a given

service could be compensated with an associated reward.)

Based on the two characteristics described above,

performance criteria can be classified as: Mandatory

Nonmeasurable (MN) , Mandatory Measurable (MM) , Desirable

Nonmeasurable (DN) , and Desirable Measurable (DM) . Examples

of each class of criteria are provided in Table 1.

There are a large number of performance indices which

may be used for defining comparison criteria in a computer

service selection. Abrams and Treu [ABR77b] have tabulated

more than fifty measures which may be used to describe the

demands and needs of a user at an interactive terminal.

Those measures which describe the time, length, rate and

ratios of user-computer interactive behavior, are likely to

be of primary interest in the evaluation and selection of

computer services, especially the three measures of response

time, turnaround time and cost.

Ultimately a decision about how many and which

performance factors are most important in an evaluation is a

management policy issue. One user may view system

reliablility as the over-riding consideration in a service

selection effort while another might view cost and response

time for short edit commands as the most important factors.

Usually some combination of various measures of service

response time and costs is used in the selection process.

This publication makes no statement about which performance

indices should be used in defining selection criteria . It

does, however, assume that selection criteria exist and can

be stated in advance of any measurement effort.

2.2 Application of Performance Criteria

The CSSM in SP 500-44 indicates a sequence in which the

four classes of performance criteria typically are applied

in most service selection methodologies. This model is

repeated here in Figure 1. Phase I involves the application

of MN criteria. The phase is managed easily since each

6
Table 1

Examples of Performance Criteria

Type Example

Mandatory 1. The system must be fully delivered and operational

Nonmeasurable no later than September 1, 19T9-

2. Timesharing service must include FORTRAN, Basic, Lisp,

SNOBOL, and editing facilities.

Mandatory 1. The mean turnaround time for all student jobs must

Measurable be less than 15 minutes.

2. 95^ of all trivial command response times must be

less than 1 second.

Desirable 1. It is desirable that the system include Pascal and

Nonmeasurable COBOL facilities.

2. It is desired that the system provide a text editing

capability.

Desirable 1. It is desired that the system provide a mean

Measurable turnaround time for the benchmark run of 5 minutes

or less.

2. It is desired that 95% of all trivial command response

times be 0.5 seconds or less.

7
PHASE il

PHASE

ALL ALTERNATIVE COMPUTER SERVICES

0©©© •••©

^ ^ ^ >^

MANDATORY

PHASEl\ NONMEASURABLE

CRITERIA

MANDATORY

MEASURABLE

CRITERIA

DESIRABLE

MIA

NONMEASURABLE

CRITERIA

IIIB MEASURABLE

CRITERIA

REJECT

REJECT

REJECT

©

BEST SERVICE

Figure 1 Computer Service Selection Model (CSSM)

8
computer service either does or does not have the required

characteristic. All of those services which do not satisfy

the MN criteria are eliminated.

Phase II involves the application of MM criteria. In

general for each MM criterion, performance measurements are

gathered from every computer service and a decision is made

as to whether or not the criterion is satisfied. Failure to

satisfy a single MM criterion results in a service's

elimination. A typical selection criterion that might be

applied in phase II is "consider only those services which

provide a turnaround time of five minutes or less for at

least 80% of the remote job entry requests."

Finally, determination of the best computer service is

made in phase III. This phase is separated into two parts,

phase IIIA for the application of DN criteria and phase IIIB

for the application of DM criteria. (Note that it is not

implied that DN and DM criteria are necessarily applied

separately or in any particular order.) Phase III involves

gathering both measurement and subjective data and

integrating all of the information in a complex manner to

determine which computer service is the best one. A

service's failure to satisfy a desirable criterion results

not in its elimination, but rather in a penalty to

compensate it with those services that do satisfy the

desirable criterion. Different selection methodologies

incorporate these penalties into the overall selection

process in different manners, yet the effect is the same.

Typical questions asked in phase III are "does a service

provide adequate software and hardware documentation?" and

"does a service have a desired x% of its response times less

than a five second threshold?".

In both phases II and IIIB, comparison requires

collecting and analyzing relevant performance measurements

from the various computer services under consideration: in

phase II, to eliminate those services failing to satisfy

mandatory performance requirements, and in phase IIIB, to

penalize those services failing to perform at desirable

performance levels.

2.3 Methodology for Selection

To understand better how the procedures described in

Section 3 fit into the overall selection process, two

general approaches to computer service selection are now

discussed. The first approach applies a set of mandatory

9
criteria and then chooses the least expensive service

remaining. In terms of the CSSM, phase III is greatly

simplified with this approach (since minimum cost becomes

the only desirable criterion) . When possible such an

approach is suggested due to its ease and objectivity. Sole

use of mandatory criteria to define computer service

requirements, however, is appropriate only when a user has a

well justified set of functional (nonmeasurable) and

performance (measurable) requirements. The latter occurs if

there exist threshold levels of performance for which

"better" performance represents no significant improvement

in satisfying processing needs. For example, in a particular application, a service for which 80% of its

responses occur in less than two seconds may be for

practical purposes just as good as a service for which 80%

of its responses occur in less than one second. That is, as

long as 80% of its responses occur in less than two seconds,

a service is considered acceptable. Once a certain

threshold level of performance is assured, other

considerations such as cost and available software become

more important factors in the selection process.

A second approach for use in more complex selections

applies a set of mandatory criteria (thereby reducing the

original set of services to a set of minimally acceptable

ones) and then applies a set of desirable criteria. For

each desirable criterion satisfied, a service is rewarded an

amount (either in dollars or points) which represents the

effort required for the user to make up for the service

feature had it been lacking. After all desirable criteria

are applied, the service with the lowest cost or score is

selected. This technique is part of a cost-value selection

methodology detailed in [JOS771 and summarized in [TIM73]

.

Other selection methodologies define a weights and scores

approach [DOW67] and a figure of merit approach [ABR77b]

.

The reader should be familiar with one or more of these

techniques to understand fully how the procedures described

below fit into the overall selection process.

Regardless of the selection methodology used,

information about the computer services being compared with

respect to particular performance indices is needed. The

procedures presented in this publication determine which

services perform at specified performance threshold levels

when those levels are expressed in terms of proportion

statistics. I^ the procedures are applied in phase 11_ of a

selection process , the included services continue to be

considered for selection, while the excluded services are

eliminated . If the procedures are applied in phase IIIB of

10
a selection process ^ the included services are properly

rewarded and/or the excluded services are properly

penalized . In general, the procedures are employed as part

of a binary decision making process. Each procedure

provides the analyst with a confidence statement which

characterizes a comparison study's probability of success.

That is, confidence statements similar to, "we are 90%

confident that services x, y, z provide a specified level of

performance while services a, b, c do not" can be made after

measurement.

For computer service measurement analyses, there are

four steps which must be followed to collect meaningful

measurements from the computer services being compared.

They are:

1. determination of performance criteria which

will form the basis of a service comparison

[ABR77b]

,

2. development of a user scenario that is

representative of a projected workload [CR074,

NOL74, WAT77, WRI76]

,

3. translation of a scenario into individual

scripts executable on the respective services

under test [MAM79a] , and

4. collection and analysis of the data required

for a comparison.

The procedures now presented address issues relevant to step

four; some references for the others are provided.

3.0 APPLICATION OF PROPORTION CRITERIA

The application of proportion criteria in a computer

service selection occurs when an analyst wants to know which

services provide a minimum level of performance. The

efficiency of the data collection process and the validity

of the data analysis in making the binary decisions required

in such an application depend upon the choice of an

appropriate experimental design. An area of statistics

entitled "Statistical Ranking and Selection" provides an

appropriate foundation for the data collection and analysis

component of a computer service comparison and selection

(see [DUD78] or [GIB77] for a survey of ranking and

selection techniques)

.

11
The statistical ranking and selection literature refers

to the binary decision making process as selection "better

than a standard" or "subset selection". Reference in this

publ ication to a service as being better than standard is an

attempt to be consistent with this literature and does not

imply a "standard" such as a voluntary national standard or

a Federal Information Processing Standard . Subset selection

procedures based on a proportion statistic are appropriate

for the application of measurable criteria in a computer

service selection. Three of these procedures are described

here.

The actual step-by-step procedures that are presented

in this section, combined with Tables 1-19 in Appendix B of

SP 500-44 and Appendices A and B here, are complete in that

they may applied without reference to other books or

articles. No statistical justification for the procedures

is presented; the interested reader is referred to [AME79]

and [AME80] for the statistical theory. Other references

describing the application of ranking and selection theory

to computer selection include [AME78] and [MAM77]

.

To those readers for whom this is a first exposure to

ranking and selection procedures, it is suggested that

Sections 3.2.1, 3.2.2 and 3.2.3 be read in parallel with

Sections 4.2.1, 4.2.2 and 4.2.3, respectively, where example

applications of the procedures are given. This mode of

reading will provide an intuitive feel for the procedures

and will place their necessarily abstract and general steps

in an environment with which the reader is already familiar.

3.1 Proportion Criteria

As quoted in Section 2.0, [FIP57] recommends the use of

non-parametric statistical descriptors when specifying

computer service response time and turnaround time. In

particular, it is recommended that in specifying acceptable

computer service response, one or more classes of

interactive tasks be specified to the degree required by the

complexity of the workload. The examples given are:

1. In the use of the text editor for insertions,

deletions or changes involving n characters or

less, the response time shall be less than w

seconds 50% of the time.

2. In the compilation of PL/I programs of less

than m lines, the response shall be x seconds or

12
less 90% of the time.

3. Retrieval requests to a bibliographic

information retrieval service, in which p criteria

(keys) shall all be satisfied in order for an item

to be displayed, shall be serviced with a response

time of u seconds or less 95% of the time.

These are all examples of "single" proportion criteria;

each one defines a single threshold level of performance (w

seconds, x seconds, u seconds) and an associated minimum

proportion (50%, 90%, 95%). (Note that percentages such as

50%, 90%, 95% are equivalent to proportions of 0.50, 0.90,

0.95, respectively.)

It also is possible for the user to increase the

complexity of a selection criterion (and likewise the

complexity of the associated data analysis) by using

multiple proportion statistics in specifying acceptable

response. For instance, number 1. above could be restated

as:

1'. In the use of the text editor for insertions,

deletions or changes involving n characters or

less, the response time shall be less than w

seconds 50% of the time and less than 2w seconds

85% of the time.

This is an example of a "dual" proportion criterion since it

defines two threshold levels of performance (w seconds, 2w

seconds) and two associated minimum proportions (50%, 85%).

Procedures for the application of single and dual

proportion criteria are discussed in Sections 3.2.1 and

3.2.2, respectively. A procedure for the application of an

"upper-lower" proportion criterion, a variation of a single

proportion criterion, is discussed in Section 3.2.3.

A basic set of definitions common to ranking and

selection procedures is presented in Table 2 for easy

reference. Table 2 should be read in conjunction with the

next section. The notation is consistent with the

statistical ranking and selection literature and will

facilitate reference to that literature.

13
Table 2

Summary of Ranking and Selection Terminology

A. Notation common to all procedures

k:

p*.

n:

service

niomber of computer services in the study

desired level of confidence in the correctness

of the selection results; P* is a probability of

correct selection

number of measurements collected from each computer

service

(i) the ith computer service; i = 1, 2,

B . Notation common to selection based on single proportion criteria

C(thld)

:

X(i):

p(min)

:

p(i):

measure threshold value for a performance index

number of measurements from service(i) which are less than C(thld)

minimum proportion of measurements less than C(thld)

required to include a service in a selected subset

true proportion of measurements from service(i)

which are less than C(thld)

C. Notation common to selection based on dual proportion criteria

Cl(thld)

:

C2(thld)

XIUJ, X2(i):

pl(min) , p2(min)

pl(i), p2(i)

first and second measure threshold values for

a performance index

number of measurements from serviced) which

are less than Cl(thld), C2(thld), respectively

minimum proportion of measurements less than Cl(thld)

C2(thld), respectively, required to include a service in a selected subset

true proportion of measurements from service(i)

which are less than Cl(thld), and greater than

or equal to C2(thld) , respectively

D. Notation common to selection based on upper-lower proportion criteria

C(thld)

:

X(i):

p_upper (min)

p_lower(max)

p(i):

measure threshold value for a particular index

number of measurements from serviced) which

are less than C(thld)

minimum proportion of measurements less than C(thld)

required to include a service in a selected subset

maximum proportion of measurements less than C(thld)

required to exclude a service from a selected subset

true proportion of measurements from service(i)

which are less than C(thld)

14
3.2 Procedures for the Application of Proportion Criteria

For selection based on proportions, it is assumed that

there are k > 0 computer services from which it is desired

to determine a subset of minimal size which includes all

computer services satisfying a specific performance

requirement. Each procedure achieves a correct subset

selection with probability at least P* (0 < P* <1) . The

value P* is prespec if ied ; its complement 1-P* is the upper

bound on the probability of an incorrect selection. (That

is, the selection procedures limit the probability of an

incorrect selection to no more than 1-P*.)

For each procedure, it is necessary to obtain n independent measurements from each computer service. The

requirement that the data be statistically independent poses

a problem that must be given careful consideration. If the

data are not independent, they are "correlated". This means

that the value of a current measurement is related to or

dependent on the value of one or more previous measurements.

In a comparison study, correlation among data points would

exist if measurement collection did not account for the

periodic nature of the workload demands. For example, if

the processing requests on a computer service significantly

change during certain periods of the day, consecutive

measurements taken over short time intervals are likely to

be correlated. That is, a fast (slow) response time data

point is likely to follow a fast (slow) response time data

point if the time interval between collection is not long

enough to negate the periodic effect.

SP 500-44 contains an extensive discussion of the

statistical notion o£ correlation and how it relates to

computer service data collection. In general, if data are

correlated, the number of measurements needed for a given

probability of making a correct selection increases

dramatically since correlated measurements provide less

"information" than do independent measurements. There are

many techniques, however, for avoiding correlation in the

data, such as properly spacing measurement collection in

order to assume independence. With careful planning,

adequate time over which to spread measurement, and

reasonably well-behaved computer services, a correlation

problem often can be avoided without violating experimental

objectives. (+)

(+) In some cases a correlation problem is unavoidable. The

author is developing procedures which take correlation into

account

.

15
A significant advantage of these procedures is that

they make no assumptions about the underlying distribution

of the measurements

.

That is, one need not assume that

response times, turnaround times, etc., follow a normal,

exponential, or any other often assumed distribution.

The goals of the procedures are:

Section 3.2.1 - Determining S(single), a subset of

computer services that includes all services

better than a single proportion criterion.

Section 3.2.2 - Determining S(dual) - a subset of

computer services that includes all services

better than a dual proportion criterion.

Section 3.2.3 - Determining S (upper-lower ) - a subset of computer services that includes all

services better than an upper proportion criterion

and excludes all services worse than a lower

proportion criterion.

3.2.1 Selection of Services Satisfying a Single Proportion

Criterion

A single proportion criterion is stated in terms of a measure threshold value, C(thld), and a minimum proportion,

p(min). The object of the procedure is to determine a subset of the k computer services being compared which

includes every service whose performance distribution (for a particular index) has values less than C(thld) at least

p(min) proportion of the time. (+) The true proportion of

values from the ith computer service, denoted service(i),

which are less than C(thld) is denoted p(i)

.

For example, if a performance index of interest is

turnaround time for a certain sequence of user-computer

interactions, then a single proportion criterion might state

that at least eighty percent of the time the sequence must

be executed in less than 10 minutes. In this case, p(min) = 0.80 and C(thld) = 10 minutes.

(+) One can determine a subset which includes every computer

service whose performance distribution has values greater

than C(thld) at least p(min) proportion of the time by

replacing "less" (<) with "greater" (>) wherever values of

C(thld) are referenced.

16
The values of the parameters C(thld) and p(inin) (and

analogous parameters in the two procedures that follow) are

analyst choices based on nonstatist ical considerations such

as past performance or projected performance needs. These

values also may be the results of psychological studies of

human-computer interaction studies which indicate threshold

levels of computer performance necessary for efficient

and/or acceptable human performance [CAR68, MIL68, WIL77]

.

For example, [MIL68] indicates that most response delays in

conversational interaction between human and information

systems must be less than 15 seconds. (In this case, "most"

might be interpretted as 90%.)

Selection is accomplished by collecting measurements

from the computer services under study and estimating each

service's p(i) value. Using an appropriate decision rule

based on statistical ranking and selection theory, the

analyst is then able to determine those computer services

which satisfy the single proportion criterion.

Denoting S (single) as the selected subset of computer

services, by definition a correct selection occurs if and

only if:

V i[p(i) > p(min) —> service(i) e S(single)]

That is, for each and every service, if the proportion of

values less than a threshold is greater than or equal to a minimum proportion value, then that service is included in

the selected subset (see Figure 2)

.

When applying proportion criteria, an analyst would

prefer a procedure which perfectly separated all "better

than (or equal to) standard" computer services from all

"worse than standard" services. With only a finite number

of measurements from which to draw inference, any rule

separating "better" services from "worse" services will have

positive probability of two kinds of selection error: that

of excluding better services, and that of including worse

services. Based on practical considerations in procurement

situations, it is assumed better to minimize the chance of

the latter type of selection error (i.e. it is better to

include a "worse" service than to omit a "better" one)

.

This attitude is justifiable in government procurement where

efforts must be made to minimize the chance of incorrectly

claiming a vendor's product to be below specifications.

17
EXAMPLE: SELECT ALL SERVICES WHICH HAVE AT LEAST 80%

TURNAROUND TIMES LESS THAN 10 MINUTES

C(thld) = 10 MINUTES p(min) = 0.80

WORSE THAN STANDARD BETTER THAN

STANDARD

0.0 p(m in) 1.0

BETTER

BETTER

WORSE

WORSE

BETTER WORSE

ALL BETTERS

(MAYBE OTHERS)

NO BETTERS

FIGURE 2: APPLICATION OF A SINGLE PROPORTION CRITERION

18
Therefore this procedure determines (with probability

at least P*) a subset of services which includes all

services that perform better than C(thld) at least p(min)

proportion of the time. Other assumptions regarding the two

kinds of selection errors result in different subset

selection procedures; they are not investigated here. This

selection goal is extended in the procedure in Section 3.2.3

where an analyst has some control over how far below

standard a service can be, and still be included in the

selected subset.

Note that for a subset selection policy, there is also

an implicit objective of minimizing the size of the selected

subset (subject to a 1-P* constraint on the probability of

incorrect selection) . One can always attain a correct

subset selection with probability equal to one by including

all services in a selected subset (since such a subset

indeed would include all better than standard services) , yet

this contradicts the purpose of applying proportion criteria

- that of detecting as many worse than standard computer

services as possible.

The procedure is a four step process:

Step 1: Choose appropriate P*, C(thld) and p(min)

values according to nonstatistical considerations.

Step 2: Collect n independent measurements from each

of the k computer services. The analyst chooses n based on

practical considerations, bearing in mind that as n increases, more accurate estimates of each service's

proportion will be attained, thereby increasing the chances

of excluding those services whose performance does not

satisfy the single proportion criterion.

Step 3; Let X(i) = number of measurements from

service(i) which are < C(thld). Since n is identical for

each computer service, the X(i) values provide information

about the true proportions.

Step A: For each service, compare X(i) to a constant

M, which is derived from statistical ranking and selection

theory. If X(i) > M, then include service (i) in the

selected subset; otherwise exclude it. M is determined by

table lookup based on the parameters k, n, P* and p(min)

.

Extensive values for M are tablulated in [AME79] and are

reproduced as Tables 1-19 in Appendix B of SP 500-44. (In

these tables p(min) is referred to as the standard

proportion.)

19
3.2.2 Selection of Services Satisfying a Dual Proportion

Criterion

A dual proportion criterion is stated in terms of two

measure threshold values, Cl(thld) and C2(thld), and two

minimum proportions, pl(min) and p2(min). The object of the

procedure is to determine a subset of the k computer

services being compared which includes every service whose

performance distribution (for a particular index) has values

less than Cl(thld) at least pl(min) proportion of the time

and has values less than C2(thld) at least p2(min)

proportion of the time. (+) (Assuming without loss of

generalization that Cl(thld) < C2(thld), then to be

meaningful pl(min) < p2(min).) The latter part of a dual

proportion criterion equivalently can be stated as "has

values greater than or equal to C2(thld) at most l-p2(min)

proportion of time" (see Figure 3)

.

For example, if a performance index of interest is

response time for program compilations, then a dual

proportion criterion might state that at least 60% of the

compilations must take less than thirty seconds and at least

90% of the compilations must take less than one minute. In

this case pl(min) = 0.60, p2 (min) = 0.90, Cl(thld) = 30

seconds and C2(thld) = 1 minute.

The values of the parameters Cl(thld), C2(thld),

pi (min), and p2(min) are analyst choices based on

nonstatistical considerations. Selection is accomplished by

collecting measurements from each computer service under

study and estimating the true proportions of measurements

from service(i) lying below Cl(thld) and C2(thld).

Performance measurements from the computer services

fall into one of three intervals; interval 1 = [0,C1 (thld) ) , interval 3 = [CI (thld) ,C2 (thld) ) and interval

2 = [C2(thld), infinity). (Note that the intervals are

ordered 1,3,2 and not 1,2,3.) The true proportions of

measurements from service (i) which lie in the intervals

1,3,2 are denoted pl(i), 1-pl ( i

) -p2 ( i ) , and p2(i),

respectively (see Figure 4) .

(+) Justification for determining a subset which includes

all better than standard services rather than one which

precisely separates the better than standard services from

the worse than standards services is discussed in Section

3.2.1.

20
zg

»—0

LU o LUX

to

LU

'i <> a UL. o

CO<

LU<

.EO "J

Iz £ . I- ^ ^ ^ < o 3

UU O- ^ o > < Qu

li

CS ~ a O

Q-o

LUI a H- to \ I LU 1 ^ n 3 ^ toO <> E z

to

LU o 3

to !r

—I < o < uu vJ ^

_J

I- Q- H- 2

C/3

C9

21
1.0

INTERVAL! | INTERVALS | INTERVAL 2

Cl(thld) C2(thld)

FIGURE 4: TYPICAL CUMULATIVE DISTRIBUTION

FUNCTION F(i) OF OBSERVATIONS FROM SERVICE(i)

22
A dual proportion criterion defines service(i) to be

better than standard if pl(i) > pl(min) and l-p2(i) > p2(min). Stated in other terms, pl(min) is a minimum

probability of a measurement lying in interval 1, and

p2(min) is a minimum probability of a measurement lying in

the union of intervals 1 and 3. (Note that the latter part

of this statement is not equivalent to saying

p2 (min) -pi (min) is a minimum probability of a measurement

lying in interval 3.)

Denoting S(dual) as the selected subset of computer

services, by definition a correct selection occurs if and

only if:

Vi[{pl(i) > pl(min) and l-p2(i) > p2(min)} —> service(i) csjdual)]

That is, for each and every service, if the proportions of

values less than both threshold values are greater than or

equal to two minimum proportion values, respectively, then

that service is included in the selected subset (see Figure

5) . A dual proportion criterion permits greater flexibility

than a single proportion criterion in defining performance

requirements in a computer service comparison effort by

specifying two, rather than one, statistical values of

interest of a performance distribution. The added

flexibility of a dual criterion is attained at the expense

of a more complex experimental design.

The procedure is a four step process:

Step 1: Choose appropriate P*, Cl(thld), C2(thld),

pi (min) and p2 (min) values according to nonstatistical

considerations

.

Step 2: Collect n independent measurements from each

of the k computer services. The analyst chooses n based on

practical considerations, bearing in mind that as n increases more accurate estimates of each service's

proportions will be attained, thereby increasing the chances

of excluding those services whose performance does not

satisfy the dual proportion criterion.

Step 3: Let XI (i) and X2(i) = number of measurements

from service (i) which are < Cl(thld) and < C2(thld),

respectively. Since n is identical for each computer

service, the XI (i) and X2(i) values provide information

23
EXAMPLE: S ELECT ALL SERVICES WHICH HAVE AT LEAST

60% RESPONSE TIMES LESS THAN 30 SECONDS

& AT LEAST 90% RESPONSE TIMES

LESS T HAN 60 S ECONDS

Cl(thld)= 30 SECONDS

C2(thld) = 60 SECONDS

pl( min)=0.60

p2( min)=0.90

WORSE THAN STANDARD 1 BETTER THAN STANDARD 1 —H

0.0 p 1 (min) 1.0

0.0

WORSE THAN STANDARD 2

BETTER THAN

S TANDARD 2

^-^^H^

p2(min) 1.0

WORSE-WORSE

BETTER-BETTER

WORSE-BETTER

BETTER-BETTER

BETTER-WORSE

ALL BETTER-BETTERS

(MAYBE OTHERS)

NO BETTER-BETTERS

FIGURE 5: APPLICATION OF A DUAL PROPORTION CRITERION

24
about the true proportions.

Step For each service, compare XI (i) and X2(i) to

constants Ml and M2 , respectively, which are derived from

statistical ranking and selection theory. If XI (i) > Ml

and X2(i) > M2 , then include service(i) in the selected

subset; otherwise exclude it. Ml and M2 are determined by

table lookup based on the parameters k, n, P*, pl(min) and

p2(min). Extensive values of Ml and M2 are tabulated in

[AME79] and are reproduced as Tables 5-24 in Appendix A.

(In these tables pl(min) and p2 (min) are referred to as

standard proportions 1 and 2, respectively.)

3.2.3 Selection of Services Satisfying an Upper-Lower

Proportion Criterion

This procedure is a variation of the procedure

described in Section 3.2.1. In that section, it is assumed

that there are k > 0 computer services being compared from

which a subset is determined that includes all services

satisfying a specific single proportion criterion. Such a procedure does not control how far below standard a computer

service can perform and still be included in the selected

subset. (Although the farther below standard a service

performs in general, the greater the probability that it

will be excluded as a result of a particular empirical

study) The procedure only achieves with confidence P* that

all better than standard services are included. Therefore

it is possible to obtain a correct selection and include

several worse than standard services.

For this procedure it is assumed that there are k > 0

computer services from which a subset is determined that

includes all services better than an "upper" proportion

criterion and that excludes all services worse than a "lower" proportion criterion. An upper-lower proportion

criterion is stated in terms of a measure threshold value,

C(thld), and minimum upper and maximum lower proportions,

p_upper (min) and p_lower (max) . The object of this procedure

is to determine a subset 1) that includes every computer

service whose performance distribution (for a particular

index) has values less than C(thld) at least p_upper (min)

proportion of the time and 2) that excludes every service

whose performance distribution (for the same index) has

values less than C(thld) at most p_lower(max) proportion of

the time. (Note that if p_upper (min) = p_lower (max) , the

problem reduces to perfectly separating better services from

worse services for a single proportion criterion. As

25
discussed in Section 3.2.1, with only a finite number of

measurements to estimate the true proportions, perfect

separation is not possible.)

For example, if a performance index of interest is edit

command response time, then a performance criterion might

state that at least seventy percent of the response times

must be less than three seconds. Using the procedure in

Section 3.2.1, a comparison study could determine a subset

of services which includes all services for which at least

70% of the responses are less than three seconds. Using the

procedure below, an analyst could design the same study also

to exclude, with probability P*, all services whose

percentage (proportion) was less than 50% (0.50). In this

case, p_upper (min) = 0.70, p_lower (max) = 0.50 and C(thld) =

3 seconds.

The values of the parameters C(thld), p_upper(min) and

p_lower(max) are analyst choices based on nonstatistical

considerations. Service selection is accomplished by

collecting measurements from the computer services under

study and estimating each one's p(i) value.

Denoting S (upper-lower ) as the selected subset of

computer services, by definition a correct selection occurs

if and only if:

Vi[p(i) > p_upper (min) —> service(i) e S (upper-lower

)

and

P(i) £ p_lower(max) —> service(i) / S (upper-lower )

]

That is, for each and every service, if p(i) is greater than

or equal to a minimum upper proportion value, then that

service is included in the selected subset. And if p(i) is

less than or equal to a maximum lower proportion value, then

that service is excluded from the selected subset. No

statement is made about those services for which

p_upper (min) > p(i) > p_lower(max) (see Figure 6).

An upper-lower proportion criterion provides more

control than a single proportion criterion does in applying

proportion criteria in the selection process. It does so by

permitting an analyst to regulate how poorly a service can

perform and still be selected due to variability in the

collected measurements.

An upper-lower proportion criterion differs from a single or dual proportion criterion in that for the latter

two types of criteria, the analyst chooses n bearing in mind

26
EXAMPLE: SELECT ALL SERVICES WHICH HAVE AT LEAST 70%

RESPONSE TIMES LESS THAN 3 SECONDS ALSO

GUARANTEEING TO EXCLUDE ALL SERVICES WITH

AT MOST 50% RESPONSE TIMES LESS THAN 3 SECONDS.

C{thld)= 3 SECONDS p -upper (min) = 0. 70

p-lower(max) = 0.50

WORSE THAN BETTER THAN

LOV/ER STANDARD ? UPPER STANDARD

r II I II I—

h

0.0 p- lower (max) p- upper (min) 1.0

BETTER

BETTER

?

WORSE

? BEHER

WORSE

INCLUDE

EXCLUDE

ALL BETTERS

NO WORSES

(MAYBE ?'S)

NO BETTERS

ALL WORSES

(MAYBE ? S)

FIGURE 6: APPLICATION OF AN UPPER-LOWER

PROPORTION CRITERION

27
only practical considerations and a desire to detect as many

worse than standard services as possible. Applying an

upper-lower criterion, however, requires a minimum number of

measurements which must be collected for a given combination

of parameter values. This is because n must be large enough

to differentiate at the desired confidence level between

those computer services better than the upper standard

proportion and those services worse than the lower standard

proportion

.

The procedure is a four step process:

Step 1: Choose appropriate P*, C(thld), p_upper (min)

and p_lower (max) values according to nonstatistical

considerations

.

Step 2: Collect n independent measurements from each

of the k computer services. The analyst chooses n based on

table lookup in Appendix B.

Step 3: Let X(i) = number of measurements from

service(i) which are < C(thld). Since n is identical for

each computer service, the X(i) values provide information

about the true proportions.

Step £: For each service, compare X(i) to a constant

M, which is derived from statistical ranking and selection

theory. If X(i) > M, then include service(i) in the

selected subset; otherwise exclude it. M is determined by

table lookup based on the parameter values k, n, P*,

p_upper (min) and p_lower (max) . Extensive values for M are

tabulated in [AME79] and are reproduced as Tables 25-44 in

Appendix B. (In these tables p_upper(min) and p_lower(max)

are referred to as upper and lower standard proportions,

respectively.

)

Table entries in Appendix B are computed as (n,M) pairs

such that if n measurements are collected in Step 2, a decision rule will exist in the form stated in Step 4.

Three (n,M) pairs are computed for each combination of k,

P*, p_lower (max) and p_upper (min) . The first pair indicates

the minimum n for which a decision rule that satisfies the

P* requirement exists. The next two pairs are for larger n in the situation when an analyst can afford more data and

wants the same P* value. Larger n values result in more

accurate estimates of p(i) and therefore in a more accurate

selection process.

28
4.0 COMPARISON RESULTS

Using the procedures above, this section reanalyzes

computer service measurement data collected in a case study

performed at the National Bureau of Standards during Summer

1978. The purpose of reanalyzing past data is to illustrate

the selection procedures in terms of an actual comparison

effort. The original study, discussed in detail in

SP 500-44, was conducted to evaluate the time and cost

required to apply various statistical ranking and selection

procedures in an actual computer service procurement

environment. Case study results in terms of the

theoretical, technical and economic feasibility of comparing

interactive computer services can be found in [MAM79b]

.

4.1 Summary of a Case Study

Four heterogeneous remote access time sharing services

were compared: a DEC System-10 running a TOPS-10 Monitor, a Honeywell 6180 running MULTICS, an IBM 360/65 running

MVT/TSO, and a UNIVAC 1108 running Exec 8. A "scenario", or

functional description of an interactive benchmark which is

to be executed and measured on each computer service being

compared [WAT77] , was designed to be reasonably typical of

the functional requirements of a real workload. It was not

expected nor supposed to be representative of a specific

user's workload; rather the scenario was designed to be a realistic example which could be used to investigate the

overall selection methodology. The scenario is presented in

Table 3.

The scenario was translated into "equivalent" scripts

executable on the four services. Eight performance indices

based on the activities in the script were chosen for

computer service comparison in the case study. They are:

1. cost,

2. turnaround time for the entire script

execution,

3. response time for the bibliographic retrieval

(no. 2 in the scenario)

,

4. response time for the FORTRAN program (no. 3

Note: Reference to commercial products in this report is

for identification only and does not imply endorsement by

the National Bureau of Standards.

29
Table 3 Scenario for the Case Study

General Specifications ; Think time (response-to-stimulus

3elay) Fo'r all user commands is 6 seconds.

File Character i sties ; the files listed below are to be

resident on the respective systems before the start of

script execution.

Files for executing a COBOL search

bibliographic database:

SELECT: compiled COBOL program

BIB: bibliographic database

ACCESS: bibliographic entries to be found

CHOSEN: list of entries retrieved

RESULT: summary of search

of

Files for executing a FORTRAN version of the BELL

benchmark

:

BELL: compiled FORTRAN program

BELLIN: input specifications for the program run BELLOUT: output of program run File for interactive FORTRAN program, INTERA, a source program (with errors) to compute prime

numbers

Functional Scr Ipt : 1. Logon to the system.

2. Execute SELECT.

3. Execute BELL.

4. Copy file INTERA to file INTRl

.

5. Edit INTRl, correcting a syntax error by changing line

14 to read: GO TO 10.

6. Edit INTRl, correcting a logical error by changing line

23 to read: PP=PP+1

.

7. Compile INTRl.

8. Execute INTRl. This will initiate the following

dialogue

:

ARE YOU TESTING A NUMBER? (0 or 1)

Enter : @ (CR ) ARE YOU GENERATING PRIMES? (0 or 1)

Enter: 1 (CR)

LIMIT UNDER WHICH PRIMES ARE ~T0 BE GENERATED:

Enter: 10^ (CR)

25, 2, 3, 5, 7, 11, 13, 17, 19, 23, 31, 37, 41, 43, 47,

53, 59, 71, 68, 71, 73, 79, 83, 89, 97

ARE YOU TESTING A NUMBER? (0 or 1)

Enter : 0 (CR)

ARE YOU GENERATING PRIMES? (0 or 1)

Enter: 0 (CR)

DO YOU WANT TO QUIT? (0 or 1)

Enter: 1 (CR)

9. Delete INTRl file.

10. Logoff system.

30
in the scenario)

,

5. response time for the copy command (no. 4 in

the scenario)

,

6. response time for the first edit command (no.

5 in the scenario)

,

7. response time for the second edit command (no.

6 in the scenario), and

8. response time for the interactive calculation

of all prime numbers less than 100 (no. 8 in

the scenario)

.

The hardware/software configuration for the case study

is illustrated in Figure 7. The scripts were repeatedly

executed on each of the computer services under the

automatic control of a remote terminal emulator called the

Network Access Machine [ROS75] . Turnaround time, response

time and cost data were automatically collected for each

execution of each script by a minicomputer called the

Network Measurement Machine [ABR77a, ROS76] . The data were

analyzed using a statistical package called OMNITAB [H0G71]

.

Correlation coefficients, means, percentiles and proportions

were calculated.

In the case "^study comparison results, the four computer

services are referred to as service 1, 2, 3 and 4. The

numbers were randomly assigned to the four services to

ensure their anonymity. The selection results discussed

below are primarily a function of the load on a given system

and not of the individual hardware/software combinations

providing the computer service. Thus, it cannot be

concluded from this study that a given computer system is

better than the others, but only that a given computer

service is better. The results of the case study are not to

be construed as an endorsement erf any one computer service.

4.2 Application of Proportion Criteria

The purpose of the case study was to demonstrate

several selection procedures each of which might require a different number of computer service measurements.

Therefore a maximum number anticipated for any selection

procedure was collected. In a real application of

performance criteria using ranking and selection procedures,

the number of measurements as well as the selection criteria

are determined in advance. Initially the scripts were

executed 120 times on each service, resulting in 120

measurements for each of the eight performance indices

listed in Section 4.1. Due to the independence requirement.

31
>- U1U t— 1 < (— /

CO

O / <

EP( ZD 1— to CO o

_i

m 00

>o

O ^ uj tu o

25 $ Z CM O "J T ^ ^ 5 S ^ S < S z < s s:

U 2

CO

u

o<

>- >- U 2 ^ - 2 ^

CO ^ Q z3 < O '"t

LU

UJ

13 Q.

CQ

(UCO

cli O

(U -PO

«Hao

•H

-pau

:sbO •H

<H

C3OO

<UU

-Po

02 \u

(U

bO

32
each data set was eventually reduced to 90 (independent)

measurements for each index. (See SP 500-44: Appendix A

and Section 3.2.2 for a discussion of reduction techniques.)

In the examples that follow, proportion criteria which

likely would be used in an actual procurement are chosen.

Each example employs one of the experimental procedures

presented in Section 3.2. (Additional examples can be found

in [AME79] and [MAM79a].) In those examples requiring n < 90

measurements from each computer service for a given level of

confidence, the first n measurements were chosen. In a real

comparison, when application of one criterion requires fewer

measurements than are collected to satisfactorily apply

another criterion, it is statistically acceptable (and, in

fact, strongly advisable) to use more than the required

number of measurements. The examples use the minimum number

of measurements for purposes of clarity only.

In the examples, the values of those parameters which

are analyst choices were specified to make the examples as

clear and explicit as possible. Efforts also were made to

select parameter values that are likely to be chosen in a computer service selection process. The steps in the

examples parallel the steps in the original presentations of

the procedures in Sections 3.2.1, 3.2.2 and 3.2.3.

Table 4 presents various summary statistics for the

four computer services. Only the proportion information

(i.e. the column titled "No. Measurements < C(thld)") is

used in the examples below. The mean and quantile

statistics are presented to provide a more detailed

comparison picture. The complete set of n = 90 independent

measurements were used for the calculation of all statistics

in Table 4 except where a different n value is indicated.

4.2.1 Selection of Services Satisfying a Single Proportion

Criterion (2 examples)

Example

(parallels Section 3.2.1)

Step 1: Suppose it is a mandatory criterion that a computer service be able to interactively calculate the

prime numbers between 1 and 100 (no. 8 in Table 3) in less

than 1.00 second at least 80% of the time. Further, suppose

it is desired to select a subset which includes all such

services with a probability of correct selection at least

33
np

C

3

to •-(

O V

ON

O.

cvj

II

t3 C\i o o o H u^ CT\ o CO

X3

•p o o • c— CO t~ ^

II IIC O TJ CO o t— t— t— ^ooNfn • CO o o oo •HrHOWDOOrH t— ^ OO m ON O t—

s: s: II +> 4a ^ 00^ 73

rH

+> 5

^ O VO ^ CO OS l/NCO

CO m H O -3- 00

iH CVJ H

0\ OJ i-l VD

vo ir\ cvi on VO OJ On

l/N r-l r-l 00 CVJ ^ C\J H ir\ r-i OJ

O O H CM H NO VD CVJ CO CVJ O O

CVJ ir\ cvi

EH

t+»CO

0)

+>i

Sh

CEO

•H

+>

to •H^

CQ

0) rHtI C

•P cd C -H

a; -a

I ITN

e

cd

(40) a; & > O 9) a CQ

cd •H

rH CVJ

+>

a]o

cn

CTn NO

t— JNO H m

UN l/N

iTN

CVJ on 0 00

en NO

NO

rH e'- NO OHO ^ CO 0 CO CO 0

m * rH rH -3- CVI C3N NO 00 0 H H H H CVJ rH

-a- H CJN CVJ on t~ iTN 00 CO OJ NO rH C7\ t— CVJ CO NO iTN On 00 NO 0 CO 00 o\ rH CVJ 00

CO * rH rH -a- CVJ c— 0 H NO OJ •-i 0 r-i H H -3- H 00 H

H

H CVJ co-a- H CVJ 00-a-

§

o

Ih P

P '-^

-d

CO ^

CM

U I O 0) <H U

o .

PO

o • •H «

A3 01 P4 CO

Cd ^

bO rH O OS

H CVJ 00.

0) •H

Vi•HU

0)

Q)V

(0

0) P

O

(U

0)

o

X

0) o

On

IIa

34
on

II o

Tl II O CVI iH a

^II om

•d II rH C

LA O VO

OJ rH

II o o t—

rH OO ON l/N'U^

CO

II TJ t— O OJ rH

rH CO a\VD VO

H

II T) On O -3- C\J H -3- On t— on

Cy\ ITN CM CVJ ON-a- CO H ^ ON

on IfN On t~ VO iH J- 'CVJ CMCO i/N CO ^-

CJN rH rH OO CO J00

CVJ H 00 < H VO

ON ITN^ On

ITS O UNVD

rH CM CO 00 H

CM UN-a- NO On H t- O

CM CM

VO

CM

Cvj rH

CM CM O NO

ON H NO NO

t- 00 J- NO

CM UN I— CM CM CNl 00^

CM

-3- l/NCO O

CM rH iH LTN

-a- OO

ITN UNCO CO O H CO NO

00 H 00 NO CM CM n t-NO

CM

•rt •PCo

u

H

OS &4

H CM OO-UO <U 4h (Q

0) 1

(0 o

C O

o

0) P4

a) o K u

H CM 00^ rH cj 00-a- H CM 00 JU0 • P

•

0) 4h

<U CQ 0 (0 e .3 u

•H -p C

+> -p 0 ,0 •H •H

(U •P

CO 0) (0 03 c

c c n c H

0 +3 0 a 0 0) . P4 CO A 0 0 S 0

(0 u (0 0 (C rH •H (U

0) (V i) a U (0 « CO K 0

• NO OO*

35
0.90. Then k = 4 , P* = 0.90, C(thld) = 1.00 and p(min) = 0.80.

Step 2: Choose n = 90. Ninety independent

measurements have been made on each computer service.

Step 3: For the computer services under study, Table 4

indicates that X(l) = 49, X(2) = 90, X(3) = 74 and X(4) = 32.

Step 4^: Table 3 in Appendix B of SP 500-44 shows that

M = 64 for this example. Therefore include services 2 and 3

and exclude services 1 and 4, i.e. S(single) = {service(2),

service (3)}. Since the criterion was mandatory, services 1

and 4 would be eliminated from the selection process.

Example

(parallels Section 3.2.1)

Step 1: Suppose it is a desirable criterion that a computer service be able to process the first edit command

(no. 5 in Table 3) in less than 3.00 seconds at least 70% of

the time. So, k = 4, C(thld) = 3.00 and p(min) = 0.70. Let

P* = 0.95.

Step 2: Choose n = 90. Ninety independent

measurements have been made on each computer service.

Step 3^: For the four computer services. Table 4

indicates that X(l) = 84, X(2) = 90, X(3) = 50 and X(4) = 57.

Step £: Table 3 in Appendix B of SP 500-44 shows that

M = 53 for this example. Therefore, include services 1, 2

and 4 and exclude service 3, i.e. S(single) = {service(l),

service(2), service(4)}. Since the criterion was desirable,

services 1, 2 and 4 would be appropriately rewarded.

Note that if ranking and selection techniques were not

being applied in this example, then service 4 might have

been excluded since the "expected" number of responses less

than 3.00 seconds for a service providing minimally

acceptable response is 70% of 90 or 63, and X(4) < 63. But

the ranking and selection procedure indicates that because

of experimental variation service 4 cannot be excluded at

the 95% confidence level.

36
4.2.2 Selection of Services Satisfying a Dual Proportion

Criterion (1 example)

Example

(parallels Section 3.2.2)

Step 1: Suppose it is a desirable criterion that a computer service be able to perform the bibliographic

retrieval (no. 2 in Table 3) in less than 8.0 seconds at

least 40% of the time and in less than 10.0 seconds at least

70% of the time. Further, suppose it is desired to select a subset which includes all such services with a probability

of correct selection at least 0.90. Then k = 4, Cl(thld)

8.0, C2(thld) = 10.0, pl(min) = 0.40, p2(min) = 0.70, and P*

= 0.90.

Step 2: Choose n = 90. Ninety independent

measurements have been made on each computer service.

Step 3: For the four

indicates that:

Xl(l)=38

Xl(2)=90

Xl(3)= 0

Xl(4)=78

computer services. Table 4

X2(l)=52

" X2 (2)=90

X2(3)= 0

X2 (4)=80

Step j4: Table 8 in Appendix A shows that

(M1,M2) = (27,52) for this example. Therefore include

services 1, 2 and 4 and exclude service 3, i.e. S(dual) = {service(l), service(2), service(4)}. Since the criterion

was desirable, services 1, 2 and 4 would be appropriately

rewarded.

4.2.3 Selection of Services Satisfying an Upper-Lower

Proportion Criterion (1 example)

Example

(parallels Section 3.2.3)

Step 1: Suppose it is a mandatory criterion that a computer service be able to perform the entire script ( 1-10

in Table 3) in less than 8.0 minutes at least 80% of the

time. Further, suppose it is desired to select, with

probability of correct selection 0.90, a subset which

includes all such services and excludes any service which

performs the script in less than 8.0 minutes less than 60%

of the time. Then k = 4, C(thld)= 8.0, p_upper(min) = 0.80,

p_lower(max) = 0.60, and P* = 0.90.

37
step 2: Choose n = 77 (based on Table 28 in Appendix

B) . Ninety independent measurements have been made on each

computer service; ^ choose the first 77 measurements.

Step 3: For the four computer services. Table 4

indicates that X(l)=77, X(2)=76, X(3)=9 and X(4)=43.

Step £: Table 28 in Appendix B shows M = 55 for this

example. Therefore include services 1 and 2 and exclude

services 3 and 4, i.e. S (upper-lower ) = {service(l),

service(2)}. since the criterion was mandatory, services 3

and 4 would be eliminated from the selection process.

5.0 DISCUSSION AND CONCLUSIONS

One requirement for effective computer service

comparison in a selection methodology is a set of procedures

which can provide accurate appraisals of service behavior.

Because services are compared in an uncontrolled

environment, the data collection and analysis component of

the selection methodology must account for the variability

in the collected measurements. This publication indicates a general model of the computer service selection process

based on the application of four classes of performance

criteria. For use in phase II or phase IIIB of that model,

where measurable criteria are applied, three binary type

selection procedures are given. Based on statistical

ranking and selection theory, assuming one can collect

statistically independent measurements, the procedures have

valid application when measurable selection criteria define

computer service performance in terms of a threshold

level (s) and a proportion statistic (s)

.

Each procedure described provides information regarding

the performance of a set of computer services for a particular performance index. To arrive at a final

selection of the best overall service, it may be necessary

to integrate the information obtained from individually

applying these procedures for several performance indices

with cost estimates and other analysis results. Selection

methodologies for integrating all of the available

information have been referenced.

The procedures explicitly specify prior to measurement

an appropriate decision rule and/or the number of

independent measurements required from each computer service

38
to attain a given level of statistical confidence in the

comparison results. The procedures are valuable in the

computer service procurement process because they design

comparison studies to result in statistical confidence

statements about the probability of having accurately

estimated computer service performance. This is in contrast

to those studies which make statements about which computer

services satisfy certain performance specifications, but

omit statistical justification in the form of probabilistic

confidence statements.

If the collection of measurements is necessary in a computer service selection, then these procedures (or

similar statistically sound procedures such as those in

SP 500-44) should be employed. Their mathematical

derivation provides an analyst with the ability to justify

the empirical results employed in the computer service

selection decision making process.

6.0 ACKNOWLEDGMENTS

The author is indebted to Sandra A. Mamrak and Edward

J. Dudewicz whose encouragement and guidance motivated the

research on which this document is based, and to Marshall D.

Abrams and the many reviewers for their significant

editorial contributions.

f

39
7.0 BIBLIOGRAPHY

(Note: Not all works are referenced.)

ABR75 Abrams, M.D. and I.W. Cotton, The Service Concept

Applied to Computer Networks , NBS Technical Note 880,

National Bureau of Standards, August 1975.

ABR77a Abrams, M.D. , I.W. Cotton, S.W. Watkins, R.

Rosenthal, and D.E. Rippy, "The NBS Netv/ork Measurement

System," IEEE Transactions on Communications , Vol.

Com-25, No. 10, October 1977, pp. 1189-1198.

ABR77b Abrams, M.D. and S. Treu, "A Methodology for

Interactive Computer Service Measurement," Communications

of the ACM, Vol. 20, No. 12, December 1977, pp.

936-944.

AME78 Amer, P.D. and S.A. Mamrak, "Statistical Methods in

Computer Performance Evaluation: A Binomial Approach to

the Comparison Problem," Proceedings Computer Science and

Statistics : Eleventh Annual Symposium on the Interface , Raleigh, NC, March 1978, pp. 314-319.

AME79 Amer, P.D., Experimental Design for Computer

Comparison and Selection , Ph.D. Dissertation, Department

of Computer and Information Science, The Ohio State

University, March 1979.

AME80 Amer, P.D. and E.J. Dudewicz, "Two New Goals for

Selection Based on Proportions," Communications in

Statistics , submitted for publication.

BR075 Brown, R.S., "A Case Study of a Multi-Access

Scientific Benchmarking Application," Benchmarking , ed.

N. Benwall, Hemisphere Publishing Co., Washington, DC,

1975, pp. 133-147.

BUC69 Buchholz, W. , "A Synthetic Job for Measuring System

Performance," IBM Journal, Vol. 8, No. 4, 1969, pp.

309-318.

CAR68 Carbonell, J.R., J. Elkind and R. Nickerson, "On the

Psychological Importance of Time in a Time Sharing

System," Human Factors , Vol. 10, No. 2, April 1968, pp.

135-142.

CON76 Conti, D.M., "Federal Agency Practice and Needs in the

Area of Teleprocessing Workload Specification," CS 77-2,

General Services Administration, Automated Data and

Telecommunications Service, Washington, DC, October 1976.

CR074 Crothers, C.G., Workload Determination and

Representation for On-Line Computer Systems , ESO-TR-74-54 , The MITRE Corporation, Bedford, MA,

NTIS-AD-779 818, January 1974.

DOW67 Dowkant et. al., "A Methodology for Comparison of

Generalized Data Management Systems: PEGS (Parametric

Evaluation of Generalized Systems)," Mitre Corp.,

40
Bedford, MA, ESD-TR-67-2, AD 811 682, March 1967.

DUD77 Dudewicz, E.J. and N.A. Zaino Jr., "Allowance for

Correlation in Setting Simulation Run-length Via

Ranking-and-Selection Procedures," Studies in the

Management Sciences , North-Holland Publishing Co., 1977.

DUD78 Dudewicz, E.J. and Koo, J.O., "A Categorized

Bibliography on Ranking and Selection Procedures,"

Technical Report 163, Department of Statistics, The Ohio

State University, Columbus, June 1978.

FER78 Ferrari, D., Computer Systems Performance Evaluation , Prentice-Hall, Englewood Cliffs, NJ, 1978.

FIP42 Guidelines for Benchmarking ADP Systems in the

Competitive Procurement Environment , Federal Information

Processing Standards Publication (FIPS PUB) 42-1, May

1977.

FIP57 Guidelines for the Measurement of Interactive Computer

Service Turnaround Time and Response Time , Federal

Information Processing Standards Publication (FIPS PUB)

57, August 1978.

FUC70 Fuchs, E. and P.E. Jackson, "Estimates of Random

Variables for Certain Computer Communications Traffic

Models," Communications of the ACM, Vol. 13, No. 12,

December 1970, pp. 752-757.

GIB77 Gibbons, J.D., I. Olkin and M. Sobel, Selecting and

Ordering Populations

;

A New Statistical Methodology , John Wiley and Sons, Inc., New York, 1977.

GRU75 Grubb, D.S. and I.W. Cotton, Criteria for the

Performance Evaluation of Data Communications Services

for Computer Networks, NBS Technical Note 882, National

Bureau of Standards, September 1975.

HIL69 Hillegass, J.R., "Systematic Techniques for Computer

Evaluation and Selection," Management Science,

July-August 1969, pp. 35-38.

H0G71 Hogben, D., Peavy S.T. and R.N. Varner, OMNI TAB I^

User ' s Reference Manual , NBS Technical Note 552, National

Bureau of Standards, October 1971.

JOS77 Joslin, E.O. , Computer Selection ; Augmented Edition , The Technology Press, Inc., Fairfax Station, VA, 1977.

KLE75 Kleijnen, J.P.C., Statistical Techniques in

Simulation , Part II, Marcel Dekker, Inc., New York, 1975.

MAM77 Mamrak, S.A. and P. A. DeRuyter, "Statistical Methods

for Comparison of Computer Services," Computer , Vol. 10,

No. 11, November 1977, pp. 32-39.

MAM79a Mamrak, S.A. and P.D. Amer, A Methodology for the

Selection of Interactive Services , NBS Special

Publication 500-44, National Bureau of Standards, January

1979.

MAM79b Mamrak, S.A. and P.D. Amer, "Comparing Interactive

Computer Services — Theoretical, Technical and Economic

41
Feasibility," AFIPS 1979 Conference Proceedings , National

Computer Conference, New York, June 1979, pp. 269-275.

MIL68 Miller, R.B., "Response Time in Man-Computer

Conversational Transactions," AFIPS 1968 FJCC Conference

Proceedings , Vol. 33, Part 1, pp. 267-277.

NOL74 Nolan, L.E. and J.C. Strauss, "Workload

Characterization for Timesharaing System Selection,"

Software—Practice and Experience , Vol. 4, No. 1,

January-March 1974, pp. 25-39.

ROS75 Rosenthal, R. , "Accessing On-Line Network Resources

with a Network Access Machine," IEEE Intercon 1975 , IEEE

Inc., 1975, pp. 25/3:1-4.

ROS76 Rosenthal, R. , D.E. Rippy and H. Wood, The Network

Measurement Machine — A Data Collection Device for

Measur ing the Performance and Utilization of Computer

Networks, NBS Technical Note 912, National Bureau of

Standards, April 1976.

RTE76 "Summary of the NBS/General Services Administration

Federal Workshop on Remote Terminal Emulation," CS 76-1,

General Services Administration, June 1976.

TIM73 Timmreck, E.M., "Computer Selection Methodology,"

Computing Surveys , Vol. 5, No. 4, December 1973, pp.

199-222.

TSP77 Special Notice Concerning the Teleprocessing Services

Program , General Services Administration, Automated Data

and Telecommunications Service, Washington, DC, April

1977.

WAT76 Watkins, S.W. and M.D. Abrams, Data Treatment and

Analysis for the Network Measurement System , NBS

Technical Note 897, National Bureau of Standards, March

1976.

WAT77 Watkins, S.W. and M.D. Abrams, "Remote Terminal

Emulation in the Procurement of Teleprocessing Systems,"

AFIPS 1977 Conference Proceedings , National Computer

Conference, Dallas, TX, June 1977, pp. 723-727.

WIL77 Williams, J.D. and J.S. Swenson, "Functional

Workload Characteristics and Computer Response Time in

the Design of On-Line Systems," Proceedings CPEUG - Computer Performance Evaluation Users Group , NBS Special

Publication 500-18, National Bureau of Standards, Wash.,

D.C., October 1977, pp. 3-12.

WRI76 Wright, L.S. and W.A. Burnette, "An Approach to

Evaluating Time Sharing Systems: MH-TSS A Case Study,"

SIGMETRICS , Vol. 5, No. 1, January 1976, pp. 8-28.

42
APPENDIX A

Ta"bles for Dual Proportion Criteria

This appendix contains tabled values which are likely to he needed

to apply the selection procedure described in Section 3.2.2. A demonstration of the use of these tables can be found in Section h.2.2.

The original source of the tables is [AMET9]. In all of these tables,

"population" represents a computer service, "dual standard" represents

a dual proportion criterion, "P(CS)" represents the probability of

correct selection, and "standard proportions 1 and 2" represents pl(min)

and p2(min), respectively.

43
B) lO 10 Ifi 10 in lO n M M N M CJ l ) N

******** eOSCO®® NNt-Nt-N

nnmnnmnm o><J.o^^^o>o•9 looiotiioio « — — —

— - — N fUN N M

MfiflMNNfJM t-Nr»NNNN N^JflDfin (SOtoaStS

NNNNfJMCJN cOvC-CC** i»©0«3« ***** O-Oxs-*

10 W to 5* ^ ^

« M M M «a M 05 W rt W CO P5 »*>eco — c?\e c;*vs»©c3io

OlOKIOtOffi QCOOCOCC * * N o D 10 « O © « m NN

N*

oortoom *<'*** BOO — — wi

I

1 »-NN«I0vSpN — C4N«I0«N — NCT*I0N NN«*ve C5*«N *«eN 10* N I © © 9 9 O ® 4

© a

©

«e^^e^^**^^« l01OL'?101OI.'?IO

9

K IS in 10 10 10 10 10 mnnKnnm — — - — — - — — — — — — — CJMMMNCJ

IS to in le 10 10 nin 9^ ff" 9^ 9* 9^

N i <e I - - I CO I « I

ft

! i

******** ©©©«©©« \eve'Cve««

rcooeo c5«oo ce CO CO 0? 2 cc cc ****** c c J

t£ >c c « ^&

MMNMCJNNN innismoion scecooeo mcjnn?i ««c<e

9

Mcsint-O' — *« 99©©©©©9

9-Me9*10*O

c; * <c a © in 10

— MO*io>eeo

in N o> — r)

«

CO W CO CO w

cieo*in>eO

N a- — a-c

10 10 in 10 in co*io«r»

» I N I -- I

CO CO CO ?s n CO CO «««««« o>9>i>8-» — — — — to nm eon

©•^CO^C -*CO»C COO NNNN —

*io>st» inoN 'Clcocococo *** mis

5P

44
w in m Iff in IS n o M M rj w m m ci — - C) n ri ri ?j ci nr^ntr.T.

05 C5 P3 C5 rt P5 C5 75 C^ ff* 0^ 0> ^ C >C vC i C J

nraomncson c; la c; co co x o *+*•»•*• *

e

h-N N N

C CO CO

cjfirjMfirifi ****** M^r.f.N ©isoe nnn -c* — pj*«iS®Nin M*inNa'Pj!0 *ioN©Min iffNO>fi* NO> — * o — * N* lo ®®®©®©®o — — — — — — — MMCMCgriM ***** ^c^e«'C as sax «© M «® — Mn*inN o — f)n*iff^ — .nci*ion Nc^*inN c;*in(» *«n ton n ©OO©®©©© ©©©©©©© ©©©©©© — — « — -. WNWM C5C5e5 ** »

* £

eisinioinnisie Mc^cHNfjrjM

>«'******* 0©®®S©0 NNNNNN

* —

NNCJCJOMflN ^vfnc**** ®©©C®® ***** 0-0^0-9-

* _

e^ £

******* r-r-NSNN

C5 W CO C5 W CO CO

CO *\o 10 ® CO in

IS in in n lo w

* « CO s M in

CO CO CO CO CO

in ON® CO

LO w m in C'O'O'

***

CO © COLO ©rjio

» CO — fi NN

PJ * ©©©©©©©© ^ — — ^ — — eoeocococooi ninnisis cscoo — — eo ©-ClNCOlS-CN — MMCOmvCN — NeO*lftN NCOCJIO* CO*'eN inoN N ©©©©©©©© ©©©©©a® — — — — — — f)f4PJMM pjfjfJM eococo eio «

E

f©C4 pi

45
a

-J" •* * * * r;fifirifjc!fi — — — - — ------- NdfKKIM

C9 Ct> C9 CO CO n «c>£)^o%C^ COCOCOCO NNN

« £ ®®®*®®®® NMNNNMfJ ****** ^,c^,c>C 9-0>ff>i5> fJfIN »l — M*ir5r»o>Mn Nm»>co< — * owno — * ***©« N9>Min o>Mio ft a *i ©®®©®®®® « — _ — NMNcirK-i ***** loiftion r-r-N O'O' ni »« — Mn*ifflN o — Mc;*ioN — pm*ir5N c^o*!-)-© Nnm* nio* ni ©s©**®©® ©®o®e®® ®©eco® ----- nmnn neon * mi

J -

ccazQ

• oe- »-»

coa. VvrH £S

— ZHE

CO,

UH P j-Pz

_ 05 b: Vj VI Z ©ZZ©

u.©©P

©oc

J C < '/J

©

9" £

«

£

li

HOininiBtnminwifj nnnnonn

********

nnnnn   ninio

9" O^C^ 9^ 9'

nnnnnnnm ®®©oo®® NNNr-Nr»

einnnnnnn eoxocscooco ** **** ------ MNfJMM

MNMNflMNfJ toiomiflioiom »C'0>0-!>5- ***** O'S-O^O'

******* Nt«t-NNr» ©©©©© **** escoffl

©©©®©®®©

— N*'CS®01ff

N N N M « N N N*«B©M1»

m lo 10 ift ifl ts **0©MtO >£cs©C4tn

® ® ® © rt rt rt

©riin Nil) * I *©©©©©©© — — — — — — — fJNNNNN ***** «««« a*« ©» n I — - « I

I ® — — Nn*iON - — Nn*>eN — NNnin* Mn*B5N n*iaN *i'5N »n «i

IS®©©©©®© ciNNn n«n ** <© I

46
Si * * * ^ T^ * fjciNcicjfifi

« I S J O C3 CO CC CO O OOOOCOO OOOCOC3CC3

coeococococococo cooaoscocooca lAininv^mt-o

N - I E I

* el ^ »^ ^ ^ ^ « ^ M ^ \{) ^ ^ ^ ^ ^ ^ ^ ©© ^^I^^V^

E I

4>eoeo®os nnnmpjwn ««««« »»o>o- now liio e.i

S =1E I

I • ®«os©®® «»©®eo® *o«®®® (^NiNM ooeo b i

CO0

Eh

n I ^= I IE i h-\a. I

* E

• E

o^ E

" E

" E

Eo

E

lo lij in in m in in in csnnrtcscsn fKaNfjMf) neonnn

CO CO CO CO CO C3 ?3 ^ ^ 9^ ^ ^ 9* vp vp >p ^ ip vp

CO eo CO C5 CO CO CO n «eecetS!8eo»

NNNNN

mtoininininm o\o-o>a>as< eocococ5co isocscs

cococococ?r;o NNNt-t-N ®e®®® ceescs ciirN®n«®«

— M •* -c cs ® M in f) <*• » to » M in «N®CMin NNNNN ®«0© CiCOCO inr-»cj* N» — * * — * NO IS I *®®»©©e® —- — — -I — — NfjfifiNra ***** ve*** cocoa «® c» «« — '4n*iot» © — Mso*inN — f)co*ii5S fico*inN co*int» *i»n n NNCMN COCOCO ** 10 I

47
— I ^^^^^^^^

^ ^ ir; o N 1

I M-r; -M Ml

NMMNNMNn NNh-NNNN MMflMfirj 0>0- 0-0-0-

ir5ir5ir3if5inioio ©®®wO® wiowirsw — — — —

N O l-! N «• M <) int»o- —

N 0- ^ ^ — ^ — f n G©0©©0©© iS* <S> O ® ® O 9 C3 CO O CO C9 in LO m N N N CJ

M M I ©©©©©©O© OO©©©©© G©©©©© Nfio nm in

fI CI fj fj r4 fj

ecsocscac:

CO C5 CO CO CO CO CO CO 3- O- O- ^ O- O" ipi©vc^^%c

NC^MMflNMN COtOCSCBXBCO COCOCOCOcOCO

fJ « N P) CJ M W fl « * -C « -C C

5«5L0ii? o-o-a-o-o-o- eococococo cscscaco

J5 C5 CO CO CO CO CO « -5 O « OC ©©©©© (B CB CO

©©©©©©©© MC^fJNPJNM NNt»NW O-^'O-O- NMf) -C* ^1

I — N*-cr»©ciiA NC0IONO- — * mNC'Citn ion*"- * icNO^M ©flin — * ni "~ I ©©®*©©©9 NC^MCiriN ***** Nt»t- ©® NI -1- ~ I

I ©» — Neo*ioN © — fjco*i»t- -<wco*u5N pjn*iot- co*io* *«« io« l»

j ®©®©®©©® ©S©©©©® ®®®©®® — NNflM COCOCO ** lOI

48
tf* c>

W C5 CO CO CO C5 C5 P5 ^ fiN tf» ^ fs. t« |s. N

W C4 CJ CJ W W CVJ M N N N N> N h> N C4 fJ fJ CJ OJ CO S CO CO CO

© © © © © © © © C5C5r5rtOCO' MNM win CO I

©©©«©©©© ©OO©®®© MPlNPaMW 00OC5O Olftmis t>NN ff'0> —I

©©©O©©®® ©O®©®®© ©©O©©® — -H^ — — C4dCJ coco Ifli

ii ! 5© I HQ- i

ti <! 'U^ -J

«<s u

(npE - e!- — CO u mesa

« -o i^55

05 B. K r- w JO-

<-[». a E©

Ptn, — eHz

<:<- EO,

wop

MB--

ec: Ti ©zzs

Nf-P<

J O <i 7: <ZCl< >-Zij

»

©N —

©

SON

Hg-

* ^l• * * f1 JJ fJ W M fI fj N M M ri CJ CI ©©©©©©© COCOCC03SC0

eoe^ nncocjnco o- o- o c- o- o> o- >c>c>c-e«>t>

vO ^ ^

*«'>* * * coco3«aco co«eococ5 cscocics

CO CO CO CO CO n CO ©©©©© n CO CO CO Nt>N

©®®©©©®® CJMNfJMfJN 4'*'<I"J'* Nr-t-NN O-O-O-O- NMN »j

— MfUONaOJlO fJCOlONO"-"* COIOI»0>C1* fCCOOCi N0>MlO S'CJIO ©CO *l e®©®©*©® — — — — — — MNNNNM ***** lOBOUMS t»Nr» ©© Ml

, -• - _ - I ©© — CQe5*lftN ©•-NC0*1!5N -Cl»*«r» MC0*LO>C CJC01O>C C01O« io« nI ©«©©©©©© ee®®©®© «©©«o® —-. — --I NCifiei meim * le i

49
COOC5C3COrt WC5 O^O^ffiC^©**^ N N N N N

CO E

S w lo LT n !C L'; c- o © © o © t*; w L* w — — —

»0999999 — — — ncJnwnn >cve«>e« csaacs — mm to

s -

9-eoinNe»-'>j' — ni»t»o> — * nmNo^ — •r — -ceoso cS9« — f * 99909999 ^^^^^^^ wt^MMv-iM CO CO >) CO lOIACOlO t^t^h* ^ 99 — — Mni»« 9 — — NC310* — — ficom* —MC^'fio fiwiove mf^ ta<o « 99999900 9999999 999009 — - ____ cifjfj eOrj lO

9

9

9

B

at

c5oc»«eon«e) 9903930 otooocsce

CO c5 CO CO c5 CO CO .c.c»%o.^.c^ ^ ^ o ^ o. cocoroco NMCO

99909909 NNNNNNM O.*?.*. MMfJ 101(5 0v| — NmoNO-Nio — eo*.ecB — * eoior-a- — * ONO-riift no. — t o-fJ* fJ't 99999999 — — — — — - MNMNMN COCOtOCOCO lOloniO O.0> Nl

1 «o — NW^mN 9 — MCO*inN -"MeO*WI» MMCOIOvC NfS** CO** ts« «l

e^lNNfl COCOTS IS I

50
ff> (J* ^^ Q\ ij>

MMWCaWNClM Qseococsotae <i- ^ * 4-

fj CJ M J) CI

u: w w 1*5 1*5 ir5 10 ©oocco mo town

1 * E B

M ei M M PJ N N

CO C5 W CO c? w

CO ^ * ^ — ^ ^ »e X —

ococonn mnioLO ««« (to- — i ©® — — Mcom* « — — pjcoio* - — fjcoiftve — -ci-.om Nco*>c n*>o lOi

I ®o©e®«©is> ©«©©«o® ©©©©©© — — — fiMM ccn loi

ii

|£

Cm

u

II < 'U

gi-«

II l£<< ^^ =

o-e;<S 25

w»i©'—

(SO

ws - — esse B<<g

gp^w

pis

©

»

»

" £

"a£

'a

*

S

£

CO CO CO CO CO CO CO CO ©©©©©©© CSSCOCQCOS

concoc^eococo?) coa)coo3Qto ialoiolololo

NMMNMPJMN N.NNMv.h.N CSCOCOCOCOCO o-o****

\C >© *C >£> *C ©®©©0® *CCvO>OiC — — -1 w- _____ fJ fJ fI ci

COCSCOCCOCS MNflflN Nf-NN

©©©©©©©© coconcocoTso t^ff*?^^*^' " PI PI CO

CO rt CO CO

P)C1?5 NNtQO 1 »' © © © © © © © O © © © © ©

0- gop) 1 CO * lO « N CS 9.

II ii 1 K 11-

1

• • •

0. n

©©©©©©©© PJPIPJPIPJPIPI «ve««« lMJ>0>0> PINP) lOlO 2j

— PicsioNC'rjin — pa*%BC0ec5 coin«o< — * ksnO' — * t»8> — — * * ©»©*©©©© — — P(p)PlP^P)Pt COCOOCJCO 10101010 Nt»N PI *© — p)co*ioiv © — pin^iot. ->pi»*iot» '-weoio* pipj** n*i5 * * ©©©©©© ^ • piPiciN eomco us i

51
E

»

no jofsccnctn oooooc® e>e-cM-j-o>

W CO CO CO P5 CO CO P5 a» ^ ^ c^ o *c *c ^

NpjfjMcjfjfiN ococscoocecs

Ift LO 10 UO W to 1ft a C * Lft '^^ LO L*

« « « « — — —

cocococococo oo>c«^ cocococo - — — C8 I

-CT-*-** — * c5*vccc--* *>ecoc>co ^oco- C8 — ' ®C0 * 1

eoc5cococo o^ff- — i ©so — CMcO'*!© ®® — r)co*« ®— pjco** — fjcof* cjco** co** «i

I ©«is®®®®o ocosaao ©®«so© ---- npim coco * i

OP)- 1 in S 1 o o o o o o © © c o © o © ©o©co © © © © © © © e©

NCO'J'LOONCOO' CO ? 1ft £ N S 0- * * N 3 O- 1ft C N CO 0^ N CO 0* eoo>

II ii i §2 1 • e o © o

a. IS N

cnes 1 B. 1

II «: -u3

jitz: b: ii.s:<

tot-s -

<!/3 § -CZ0-E- — 1»

II < 3

5Sh«

_ * OKZO

EC.::. W

b.c.^0 © -

© £

" £

03

ii

1*

HQ- ! gtiS I a. I

TO CO CO CO CO CO CO CO ©©©©©©© N h» N h» N N

cococococorococo cacsscacscocs tftisiftifttnm

IftlOWIftlftlftlO ©©©©©© >*>\£'VC^0VC> —

^'j'* * 13 see CO a CO Mririrjn t-t-NN

©©©©©©©© CO CO CO CO CO CO CO ^^^^^ cocococo nnn

I N I CO ! r» I

I CO 1

•"MC0IOt»O>C>ILO — — C^*vOO-C>l CO'J'VCCO — * LON*— t»a- — f 0> — * — * *l _ _ .

^ ®®®®®®®® —-. — — -.-•-< MfJPJMNM COCOCOncO lOlOLOtJ I'^Nt- 0>0> Nl

I

I ©© — flN*IOvC © — NN-tlOC — faM*10« -"MCOif-C Nra*io <'10>C *IO yO I

I ®»®©©©©® ®®®®©©© ©®®©©® — — — — — NMnr] fjflfj 101

52
n * * rlo lA ic 10 If? 10 !n e^o^c^^o^^ w t*!' w w w

ficicj fin n I *00000®0 MMNMCMfJM l»lBlBi»mift fflcecOCOa (MNMN ve««

CO CO CO CO ___ Q

,

I O I e5 't O — — e5^\Cffl-*^ C^^CCO — ^^ *n©C0OC5 ^©C0 — CO — * —

©©©®©®©© ©©©O©©© — COCOC505C5 **** ypvo** COCO

©©©0©©00 ©©©©©9© 90©0©© ©©©©© -ri M *N *< CICIN on *4' I

I

09

CO W CO C5 05 CO P5 ©©©©©©© NNNNNN

CINMC^tJMNM CO 03 CO CO CO a CO in » in » LO in

inminLomintn ©©o©©© ininininin

f****** cocosaaco ciMciMrj nnnn

©©©©©©©o CO n 0 CO CO CO CO cocoeoco nnn

©©©©©©©© ****** vCk^O^nO C'C^O^O*

*NPJ

©pjcoint-o>fj* Me5int»9>fj* eo*«co — * inN»-"* n* — * O' — * — * co ©©o©©e©© ©©©©©©© cuNMMNCi coconnco inLoinLo nnn »9>

^^---•OJCoin*© © — — Mc^io* — — wcoin»c — Nco*\c pjcoin^e coin%o in*© * ©©©©©©©© ©©©«©©© ©©©©©© — — — — — cac^cj nco in

53
9

CO s

C5C5C5C5C5C5C5CO tf* ^ 5^ 0^ O^CO^ivO^C

------ fj ca fI N cj

«««««ve« riMficMriN COCSC0C0C3

>©©®9©©S» COC5P?C3C3rtrt NNNNNN

9

€> ^ CO * C CO " 00®<&CO®S> ^^^^ >PsC>0 CO CO

©©®©<&0© ©^o®®© O®®®® —I — ^ NC^Cl COCO

9

9

9

n rt C5 CO O C5 ®©0©©C>® NNh-NNN

MrJMNnnrJN cecoecotnoa inmininL'jL'i

mwiftiffliniftio ooooo© uiiaiovtm — — — —

99©®©®®© C5 C5 C9 CO CO CO CO sc^^c^^^ c^o^^c^^ CO CO CO CO Nt*"N

- - - - - — I SNC01BN0- — CMCOWNC— * C5*«C0©C0 1.0Na> — * «C3 — * » — *

99 — — MC5io>c © — — cieoiio<e — — ncoib* — fiC3*« cmcoi»« nifj* lo* *i

©®®©©®©« ®®®®®®® ©©©999 — — — — — — — - CJCIM nco mi

54
I inio vc. I

I OC5 n I

I

< -< fi ci M :i CI

« NO VO «0 >0 >0

C9 CO C3 CO O !^ Ct> NNNNNN *£> >C * « «

— CO I

©OO — 99 — fJC5*« 9—Nf5*>C '^Hmt'O Mrt** n<** «l

I 99999999 9990000 900009 O0S90 — — -- eitifi On « I

(» 1 19 E r 99999999 9 9 9 O 9 9 9 gON 1 §H 1 n * r « Nca o>

urn OR <eu 1 e 9

a. HO- 1 CbCS 1 Cu 1

CI

'MO * N CO »

9 S I 2 - I SI

» vc * r) CJ — — O * O 3 N N a> * M - - - M no- -a -son — — N pjco - CM fJ O * K sS t) C; * IB « N C3 + IS c r- * w c r» N

? -1 wrtrtcocococoo 9999999 N N t» I- N t» in 1(5 m n w ococo NN W E 1 CI M N fJ N CO n CO CO ** in IS

NMrjNCtinNM CO CO CO eg CO IS o ininmLoinin

SI « 1

vo — N* — co-cn — — NCJCOtlO 9 N© PI a* o m * — -NMCO*in CO ^ — N *0 — flcoco*in

f? — t- «

cjcoO'Mn — *«incoco Tfifj e-NLO

eo*LO

IS

10 e 1 N M CJ M M M CO CO CO CO CO 10 Kioto

Cl C J M N C? CO CO 99 9*

9

* = I E I lOlOlOIorOLOIO 99999© lOIOLOLOLO — — -

COcecOCOCBce MriflM?! NI-NN

9 I & I

9 I «

-I

N O- tf5 N CI CO — — CICJ 99900999 CO CO CO Ci CO 05 CO *c^^^^^ o^ffi^o*©' COC5e9CO NNN —

---- --- fifl

cococoococo ««>c«ve *o>o>o> nmcj iois

gl 9 I ©NC5IONO> —

NOU5N0> —

fI*int09C0 «C0 — * CO-* J

MNCIMCIM C5 CO CO CO CO to 10 10 10 r-NN o>o> 99 — — N«IO« 9 — -MCSIO* — — CJC510NC — CI C5 * >C PIC5IO* coio« IS« >e 9 1 - SI

1

9 99 9990 9 9 9 9 9 9 NC^CI fse« 0

1 m 1 r- 1 gON 1 9 5999999 M C5 «!• » * t- CO 0» 9 9 9 9 9 9 9

CO 10 « t- CO ffN 99C009

*10>C 99 sc;9

IOnON^C" ©999

N CO 9< 9 9 9

NC0O> 99

ca» 9

9>

55
e SI — M rt ^ ift >o a N *f> W W CO ©

OJ i'S ^ W If CO

h- v& VD O CS o

04 C5 TT W C CO

>C O >P CO o

r5 V l.'S CO

ID W *0 ^

* « N N ^ CO c ® 1 ce 1 e Z 1 C- o c o c o & ------

tN> r** r • r** w

:kicjmo4 N N r-" N N N N N N

IS in

1 ^^ 1 <o 1 9

^ ! — CI C5 vC N

O CO CO ff^ o

01 C5 § ^ v: r- c: ^- ? -c N

C O M N ^ CO Lt t © w

« N

1 N 1 c 1

Al •M A1 ^1 Al Al Al

ff* ^ ^ ?^ c* ^ ^ ^ ^0 sC" <f> tT" ^ ^ ^

fJCJfl fjfl W CJ W N a r3 ci

1 «l S

1 — 04 CO ^ w ^

c? o h- w ^ .'^ rt <- OJ CI ?3 ^ s;j 0^ 1.^ O OJ OJ" — OJ P5 W CJ K ^ C «W ^ ?^ P5

P5 ^ W \C ^ C5 CO C5 1 <e

j £ E 1 CJ 01 W N 01 PI W CJ t» N N N N N t" ^ ^ ^ ^ ^ « « f1 01 fl

CO CC Q CO CI ?j fJ C ^

rt .in m 1/) ** * 1 in

1 £1 ! e -.0 CO O N L* t N CO O CO sP Ift n c « w C N LT* to L"5 * * 1 — — MCTO* in — — Cv CO w Mcora-tm cjco'j'in cj'-j'in ii-in in i N ! * I -- I in ci c^i N CI M N I w in in ir5 m in 9'?'3-0^a-0» * * *

»

"a

>C vC V© — — —

®«*o»»«e MPJMNPJC'iei minmininin cotocococo NfJMM

conncocoo intnininw cecocece — — — Oj

oinNo-N •ceo — * CO — * Ol

^rm^^^mm C5 C5 CO C? O **** •© >© O CO CO — 1 - I ©® « — N es ©© — NO** © — Neo*>c — Mco*>c Nn*\c eo** «i

©©©so©oo ©©©©oe© ©©©©©© ©e©o© — — — - ciciN con *l

BSON

pa.

to —E

» E

ft* E

IS —

*

* E

ft E

01e i§

II.

^ ^ ^ ^

wconconcococo co^aa-aco" nnnnnn

CJNNriCMNiNn ccocccscoaco ininmtninin

inin « I

I * I t»« « I

I vB* 10 I

NN(1CINNINC>I NNNNl»M>. C^flCMPJflN eOCOCOCOta

ininininioinin ©©©©c-o intoininto — — — — _____ NPIC-JM

eofflcoccceca wcifiMN Nr-Nr-

©©©©©©©« coeocsescococo mioinmoin a>o>fl. o\a> wcjfjpi vcvcc —

____ ___ Nfi

— — — inio

CSCM N I .____ cococotonco o-o-** col ©McoinNa — * McoinN* — wc5»^^co in*co — ieeo®co co©co —

_ _ _ - _ , ft©©©©©©© ©©©e©©© MNCic^MN cococococo inininin nnn "i— I

I ©® — — Ncomo ® — — cjcoin* — — fjcom* — Me5*« Ncotn* cooip in« <ei

I ®®©®®®®® ®®©®®®© ©*®®©® _____ ____ M*l(a COCJ K II

56
^ ^ \f ^ ^ ^ ^

I

I ClfJ o> I

I ^ *c * — — ^

IOCS — mcTj-c t» — if5ij'*e^ — rtc^-*© ^cooo- ones — — — Mr; — — — cici — — — fjn — — rifj — nci

ce I .r I © © © © © ^ ^ ^

90©©©©©©

©© — NO** © — NO** — NO-tvC Mn**

©©©©O©© ©©©©©© ©©©©© >rt M w M MNfJ

©9©©©©©© ©©©©©©© ©©©©©© C©©<:

"* I W W C? C? C5 W C5 W ^ ^ ^ ^ ^

I nn * I is« « I

I NNNNC^lMNn cDcoooaeote ****** vS« O I

nioioiQisinn ©©o©©o irttnwwn — ------ ----- fKNNN ******* COCOMtecOCO NNNfJM >C«nC*

©©©©©©©© MNNNMNN ISt5l010in« c>0>»0>0> NNDCJ >C«^0 —

> — OKSNO' — *" — nu5N» — * oi»t-o> — * *xet8 — * «coeM ce©cj — * *i

— — CNjnio^ ® — — NCOiaO — — NCDtO*© — C4W*li5 C4C9IO>e C5*»C L5»0 ^0 I

57
>C> ^ >D >C O ^

* ^^ C ^ — —

9

— — — — — — nnnnnn mmmiam eecoasco — — oi

— — — — — — I ® — f)f\oco — 't — ri**c — * flT^^t>a — * — * vccoon coon on nj

cos®®®©® ®®®o®®® — — — — NNMfjr] ««« coco — I ®o® — pin*« o® — on** ® — on** -"Hn** NO*<e n** ** «i

o®®o®o®s oee®®®s ®®o®®® eooo® nnri nn 4* I

goiN

IE

<a. i-O- I Vice

nnnnrtnnn c- ff* ff- ^ n — t o- :5 —

fJfJflMCl

n I

I » I e ! ®

wwwwtoirtio ®®©©®o i^nmirtw — — — -

— ^^^^^^^ f.|^t,t-Nf-. MMfJNCI >e«ve«

CO I O I

I n I 09 I ®®®o®©®© PJONMONN mmiomioio a^a.o>»0' mnnm N I -- I nnmnnos «««te\o csascoco — — loio ® — miot»o> — » -MioNo- — * r3m>ca> — * **ee — * \eoe®n ffl®n — n ©®®®®®®® ®9®Q99© wmr^r^rm^r^ W W CO W W 1(9101(910 N N h» tf* ff^ ©® — — M«lO» ®— -PjniO* — — INC5I0* -• — P1*I0 Nm*>C B'^vB 1(9 ©®®©*©®® so®®©®® ®®®®®® — — — — CINN wn 10

gdcM

p

58
rjl CCC5COCOe5C5C5c: ©©oooc© cocococaso — wej^rtscis. "Cjco^io^oN cjcico^ior- * ^ w 1

^ ^ ^ ^ ^ ^ N N N N N

— - pjpiK ---w;^ -««cv)cg «-_,cJCJ --cirt piwrt ©©©©©©©© MNf'jwwwN wwir?ir>»w cotococOX — — « ininiio

— — — M ^ — — N — — — CJ — — — cj — — — rj — — fj ©©©©©©©©

© © © © © © © © © © © © © © © *^ ^ *M M >

9eO«<00&A SeOCOC — — — -• MMN nc9 4'!

WP5 ^lft\er-C0O* O^lftCNCO* ^IftvfsNO^ lft>ots.CO^ vCh-COC* NCO^ C3 ^

^ £

C5C5OWC9C5P0P5 ^ ^ ^ ^

* ID « N US * N

C5COCOC5r5WWC5 0\©\c»0»^^0' NNNNNN

NMNNNNNf) OCOatOCCO -----

— —- — — _

isir, loinuntn ©occoo iBWrtiran

----- MNM

(S> — ni»N9< — ^ — ninNO< — * mo\co> — * *\ecBCO CO CO CO — — — nio vcN®ci No-M ec; oo®co®®® o®©®0(S(S — — — — — — mnnnn mmmu) nnn o>» — o© — -"Nnio* ® — — Ncflis* — — Mcsiffl* — — NOiB fjcj** n*\e lo ®®®®®®®© ®0®®®®© ®®®®®® — — — — — — — — — C4C^C4 C5C5 »

0) 1 B 1 o®e©©®e©

QOCJ 1 ii i

n <c lo « N 00 e>

nK §£ 1 s

a. HQ- 1 OlBi 1 0. 1

©cossa© ®©©o®o ©®so® ®os9

c;*i»>CNcaa' ^mvcNCOa- ibonso- *Ncaa59
e

«

^ lA >o tf^^c> \o

— nio CO — -f ^ N N in N o n t- — N © n N - - - -N © ?; « © NO© --rj M 1 M 1 r 1 n C5 C5 C5 C5 W irt 1ft "J? ift CO Q CO Q N 1 « — PJtvCCO— * M*vCC©0 * c © * vOCOSC^ o®o ®7S n 1 » C 1 o © ® ® ® © ® \0 * * COS «® — fin-** ©-Mr>*-f) — fjn** «*« *10 * 1

£ \ ©o®©®o®® ® © © o © o c © © © © o c CJOJ * 1

m 1 t» • B 1 ©©O©©©©®

1

1

Mc5*iffl>t>Ncea-

« §£ ! Ck. HQ- 1

B. 1

<

J - <aa

-e:<eCJ

II < -a -J

J«E Ed

t«p= -

*•-©

6. = 3; - C id I» t. '/. r- ^ JO-

— << — EC

UC--

^ !» ee V. 7: a

= ?2g

©sea o

JC<Va

©

^ 5

©

* —

»„ -

»

ait

tB§~

c?coconc?c7c?cs

CI CI fJ M fj

cieiricic4C40cj ^^^^^^ ^ ciMMcicacj csaacscs

13 o w K Lti in w C © o ® © © irMrt in w

******** r-r-NNNN MCIMCICI >C>C>£'ve — — -. ____ MMM

©©®©®®«© cjcjjjNCicjci 19 mm in mm o.o.e^a-o^ ciNCifj >c>««

©©©©©©©© P5C3P3COC5C5 O * * * CO CO CO CO — — — "f* ^ ® — etW'*©^ — ^ — C5in*^ — * CO*}**©* — '* — * wn^cj e^ — * ®C9 * ®e©o®®s® ©o®®®©© — — — — — — c5eoc)c5C5 m in in in o>t>« o>o< ®® — — CQeo*vc © — — wre** -• — fjcit*

60
C O O C C 5

I*? O CO CO I C O i N N ce I 1-5115 « I

I

* inio

CO XE «

^ ^ ^ ^ ^

C5 W C5 W

CO

* E

" E

®-"N*>ocO--* — d -^p in CO — Tf ^•^coxsn *icco©c5 icccsc; co©n ®ej ni

©©©©©©©© O®®©©©© — — ^ MNNNN ««« coco ""I ®©® — NO*!© ©© — NP5*« ® — NCI** — fJC«*« NO** W*lfl *I0 * I ®®©©©©e© ®®®®©®® ©©©9©© ©©®o© ciiMci mc9 ^l

PSCSOCiCiCOCOOS ^ J\ c ^

«

* E

® E

MNNriNciNN coe«ac3caco

cn »

65 t» - O B NOIMONMNM ««««««« f]N(Mcgr]N CSSCOCOca iri LO W

« n n N > ©

(S * ^ W E

05g g

»

feO SE

»

* E oseo©©oe rtc^nnriMM nismninis cococoas nmnn >o««

eJcscoCTc^cc **>c** cococD© """"^ 2^ j ® — C5*>ce> — * CO "J' \e e> * cxj-icco — * *>oo©n vt>co — * • Tf ©o * I ©©©e©®®® ©©©®®®© — — — C5«C5«C5 ««« —I ©©©-ncs** o© — ® — Mco*>c ->nc5*« Nft'Cvp «"* vej ©®®®©©©® ®®®©©©© ©©©©©© ®©®®® dciN no ^1

0)B

JON

30

61
E

N f1 « fJ eg M N fj B a CO X o e o n w « l-? w w

Bin « I 9

CO s

s

ID o w in» w w It?

IS g

i s -

B * S

I

©o®©®©©© ©©©©©©© — Ne]^]c^c4 ^^^Tj- v©\c«© coco ©©© — Mo*vo ©© — Ncs** ® — pjn*« — Mcs** Mnu'>e N>tif»~*io~i ©©©©o©©© ©©©©©©© ©©©©©© © © © © o - MCIM COW * I

<B 1 in E 1 O O ® G O S> O ft^ DCCJ 1 §B 1 do: 1 * «

a. mm 1 C< 1

*-< Q <t r~

[J

CO

§1

• OE -

a<f---i/i <H J

&s rc cc -et:w

U KiS ^ JO- 0.-3 « BO

Pw,

b: in 01

2

0E!=0 (hOOspl

he- 0.0

J © < w

CO

© g

©

s

» E

„ IE

a. HOC5 C5 CO CO CO C5 C5 C3 ©OOC©©© ^ fl^ ^ ^ ^

C5 C5 C5 CO C5 CO CO <y* ^ ^

NNWNfJMfJCg COCSStOCDCOCO - _ - -

NNWMNNNCJ vC-CvC**** NNfJfJC\T4 CSCOCOCOCO

110 1.0 LO w w w w o- ^ 0^

CO CO CO n CO CO 73 r»Nr-r-NN

®C®©©00© MMNNNPIM lOISLOLOlBLO COCSaea fJC^rjfJ

©OO©©©©© cocococor.co ninLoinus saaa —

cococococo ve** »o> —

i

I ® ® ® — fj ra ©© — Mcot'C ©-ipjco** —NO** cjco** rt** •*« «i

I

62
N N N t- I-

" s

1 * =

(e ®®®®®®®® NNNNiNPJf) ininmisinio ecococeo — — — mwm

®®®*®®®® nnonnn ninioinio i.nnn o©® ni

»

" 5

® — ci'**o®'j' — M*\i)«©-«' M*«eo®K *\eco©c5 «a®o ea®n ®n ni

®®®®©o®® ®©c®©®® — ®®© — rjo** ©® — Nr^^r"* ® — Nc5*« — fi?5*>c Mn*B fl*io !« <ei ®®®eo®®® ®®®*®©o ®®®s®® ®®®©® — ciofi nn ^!

«l

II 4 -U -J

cef-s - gH — C! M

OODEO •oa- »-©

a.E- — QD

II iJ<3

Cults!

t»Ei.Ujr- V ca& ^ JO-

* — MO <<-

^^^^

am — —

OS OKSO

oos

>- = t3

— —I nnrtrencinn ®o®©®®© o>3>e-o>a-onnocicjnncj «. o. » c. o^ e^ "C-eve***

csn « I

I nn * I --- — I nnnnonn r-r-t-NNN — — —

®®®®e®®® NNCiNCinM lonioininin coacecoco nmncm

nnnnnn wioiBion cBoeoffl — to i

I oo®o®®®® ©©©©©®* — — — — — — nnnncfi «ovo>o i ®®© — Nn*c ®® — Mn^vp ® — Mn** — fin*vo fin** n*\c ** >e i eo*®®®®® ®®®®®®® ©*©*©© ®®®®® ———— ONM nn fi

63
APPENDIX B

Tables for Upper-Lower Proportion Criteria

This appendix contains tabled values wRich are likely to be

needed to apply the selection procedure described in Section 3.2.3.

A demonstration of the use of these tables can be found in Section

k.2.3. The original source of the tables is [AMET9]. In all of

these tables, "population" represents a computer service, "P(CS)"

represents the probability of correct selection, and "standard

proportions upper and lower" represents p_upper(min) and p_lo'wer (max)

respectively.

64
Table 25

VALUES OF n,M FOR SELECTING, VITH P(CS) >= P*, A SUBSET OF k = 1 BIHOMIAL POPULATIONS WHICH CONTAINS ALL POPULATIONS BETTER THAN AN

UPPER STANDARD AND NO POPULATIONS WORSE THAN A LOWER STANDARD

(POPULATIONS WITH NO. OF 'SUCCESSES' >= M OUT OF n TRIALS ARE SELECTED)

P^jc = 1 P* = .95 I

1 STANDARD 1 1 1 PROPORTIONS! 1 1 UPPER LOWER

1

n M n M n M 1 n M n M n ft 1

.90

j .80 1 86

1 741 172

1 1501 2581 226 1 135

1 116 1 270

1 235 1 4051 354 1 .70 1 25 211 50 42 1 75 64 1 41 341 82 69 1 123 105 1

. 60 1 1 "tI 1 O 1 O 1 30 25 1 45 OO 1 1 Q 1 4ift 40 1 72 60 1

. 50 1 9 7 i 18 15 1 27 221 13 101 26 21 1 39 32 1

. 40 1 5 41 10 8 1 15 121 8 6 1 16 121 24 19 1 .30 1 4 31 8 6 1 12 9 1 7 5 1 14 1 1 1 21 16 1 .20 1 4 31 8 6 1 12 9 1 6 4 1 12 9 1 18 141 10 1 1 1 1 3 2 1 3 2 1 3 2 1

1 6 41

1 9 6 1

1 .80 .70 1 127 96 1 254

1 195 1 381

1 295 1 204 1541 408 313 1 6 12 473 1 .60 1 36 26 1 72 53 1 108 81 1 60 43 1 120 89 1 180 135 1

. 50 1 19 13 1 38 27 1 57 42 1 28 19 1 56 40 1 84 61 1 '.40 1 9 6 1 18 12 1 27 19 1 17 1 1 1 34 23 1 5 1 36 1 .30 1 6

A 1 4 1 12 8 1 18 1 u ^O 1

1 1 1 30 20 1 .20 1 5 3 1 10 6 1 15 10 1 7 41 14 9 1 21 14 1

, 10 1 4 21 8 5 1 12 81 6 31

1 12 71

1 18 1 1 1

1 .70 .60 1 loo

1 1 AO 1 312

1 208 1 468

1 QO 1 "tI 1 1 O 1 OOV 1 744 500 1 .50 1 39 241 78 49 1 1 17 76 1 67 41 1 134 85 1 201 130 1 .40 1 19 11 1 38 23 1 57 35 1 28 16 1 56 33 1 84 521

.30 1 y O 1 1 ft 1 Ct 1 27 1 7 y 1 34 19 1 5 1 30 1 .20 1 6 3 1 12 6 1 18 10

1

10 5 1 20 1 1 1 30 17 1

. 10 1 A. ^ 1 8 41 12 f, 1

1 7 3 1 14 7 1 21 1 1 1

1 .60 .50 1 168

1 931 336

1 190 1 504 2881 268 1481 536 303 1 804 459 1 .40 1 41 211 82 43 1 123 67 1 67 34 1 134 71 1 201 109 1 .30 1 19 9 1 38 19 1 57 29 1 28 13 1 56 281 84 43 1 .20 1 9 41 18 81 27 13 1 17 71 34 16 1 51 25 1

. 10 1 5 21 10 41 15 71

1 8 31

1 16 6 1

1 24 10 1

1 .50 .40 1 168

1 76 1 336

1 156 1 504 2381 268 121 1 536 249 1 804 379 1 .30 1 39 16 1 78 33 1 1 17 52 1 67 27 1 134 57 1 201 89 1 .20 1 19 7 1 38 15 1 57 24 1 28 101 56 22 1 84 341

. 10 1 9 31 18 6 1 27 101 13 41

1 26 9 1

1 39 14 1

1 .40 .30 1 156

1 55 1 312

1 1141 468

1 174 1 248 871 496 181 1 744 276 1 .20 1 36 1 1 1 72 231 108 371 60 181 120 39 1 180 61 1

. 10 1 15 41 30 9 1 45 14 1 24 6 1

1 48 14 1

1 72 22 1

1 .30 .20 1 127

1 32 1 254

1 671 381

1 1031 204 51 1 408 107 1 612 165 1

. 10 1 25 5 1 50 1 1 1 75 171 41 81

1 82 181

1 123 29 1

1 .20 . 10 1 86

1 131 172

1 281 258

1 431 135 201 270 43 1 405 681

65
Table 26

VALUES OF n.M FOR SELECTING, WITH P(CS) >= A SUBSET OF k = 2

BINOMIAL POPULATIONS WHICH CONTAINS ALL POPULATIONS BETTER THAN AN

UPPER STANDARD AND NO POPULATIONS WORSE THAN A LOWER STANDARD

(POPULATIONS WITH NO. OF 'SUCCESSES' >= M OUT OF n TRIALS ARE SELECTED)

i P* = .90 I P* = .95 I STANDARD 1 PROPORTIONS!

UPPER LOWER

1

n M n M n

1

1 M 1 n N n M n

1

1 M 1

1 .90 . 80 1 1351 116 1 2701 235 1 4051 354 1 190

1 163 1 3801 330 1 570

1 499 1 .70 1 41 34 1 82 69 1 123 105 1 57 471 1 14 96 1 171 146 1 .60 1 20 16 1 40 331 60 50 1 29 231 58 47 1 87 72 1 .50 1 13 10 1 26 21 1 39 32 1 17 131 34 27 1 51 41 1 .40 1 8 6 1 16 12 1 24 19 1 14 101 28 221 42 34 1 .30 1 7 5 1 14 11 1 21 16 1 9 6 1 18 131 27 21 1

. 20 1 6 4 1 12 9 1 18 14 1 6 4 1 12 8 1 18 13 1

. 10 1 3 21 6 4 1 9 6 1 5 3 1 10 7 1

1 15 1 1 1 .80 . 70 1 200

1 151 1 4601 307 1 600

1 464 1 285

1 215 1 570 437 i 855

1 661 1 .60 1 57 41 1 1 14 84 1 171 1281 80 57 1 160 1 181 240 180 1

. 50 1 28 19 1 56 40 1 84 6 1 1 37 25 1 74 52 1 1 1

1

80 1

. 40 1 17 1 1 1 34 23 1 5 1 36 1 22 14 1 44 30 1 66 46 1 .30 i 10 6 1 20 13 1 30 20 1 15 9 1 30 19 1 45 31 1 .20 1 7 41 14 9 1 21 14 1 9 5 1 18 11 1 27 17 1

. 10 1 6 3 1

1 12 7 1 18 121 6 31 12 7 1 18 1 1 1 .70 .60 1 245 1601 490

1 326 1 7351 4941 3511 229 1 6751 449 1 1000

1 671 1 .50 1 67 41 1 134 85 1 201 130 1 92 56 1 184 1 16 1 276 1781

.40 1 28 16 1 56 341 84 52 1 44 25 1 88 531 132 621

• t 11 o^ O 1 1 y 1 i i 1 OA 11 ^ 1 1 .20 1 10 5 1 20 1 1

1

30 171 15 71 30 16 1 45 25 1

. 10 1 7 3 1 14 7 1 21 1 1 1 9 41 18 9 1 27 14 1 .60 .50 1 266

1 147 1 5321 301 1 7981 456 1 3771 2081 6881 3881 1000

1 570 1 .40 1 65 331 130 69 1 195 106 1 95 481 190 101 1 285 155 1 .30 1 28 13 1 56 281 84 43 1 44 20 1 88 44 1 132 681

.20 1 17 71 34 16 1 51 25 1 22 9 1 44 20 1 66 321

. 10 1 8 31 16 6 1

1 24 10 11 14 5 1 28 121 42 19 1 .50 .40 1 2661 1201 532 2471 798 376 1 3771 1701 6881 3181 1000

1 469 1 .30 1 67 271 134 581 201 89 1 92 371 184 79 1 276 1221

.20 1 28 10 1 56 22 1 84 35 1 37 131 74 29 1 1 1

1

45 1

. 10 1 13 41 26 9 1

1 39 14 1 17 5 1 34 1 1 1 51 19 1 .40 .30 1 245

1 86 1 490 1781 7351 272 1 3511 1231 6751 245 1 10001 3701

.20 1 57 17 1 114 37 1 171 581 80 24 1 160 52 1 240 81 1

. 10 1 20 5 1 40 1 1 1 60 181 29 7 1 58 16 1 87 26 1 .30 .20 1 200

1 501 4001 105 1 6001 1621 2851 711 5701 1501 8551 231 1

. 10 1 41 81 82 181 123 29 1 57 1 1 1 1 14 25 1 171 40 1 .20 . 10 1 135

1 20 1 2701 431 4051 681

1- 190

1 281 3801 61 1 5701 96 1

1

66
Tatle 27

VALUES OF n,M FOR SELECTING. WITH P(CS) >= A SUBSET OF k = 3

BINOMIAL POPULATIONS WICH CONTAINS ALL POPULATIONS BETTER THAN AN

UPPER STANDARD AND NO POPULATIONS WORSE THAN A LOWER STANDAHD

(POPULATIONS WITH NO. OF 'SUCCESSES' >= M OUT OF n TRIALS ARE SELECTED)

I P* = .90 I P>tc = .95 I STANDARD I I I PROPORTIONS! I I UPPER LOWERI n M n M n MIn M n M n Ml

.90 .80

.70

.60

.50

.4©

.30

.20

. 10

.80 .70

.60

.50

.40

.30

.20

. 10

.70 .60

.50

.40

.30

.20

. 10

.60 .50

,40

.30

.20

. 10

.50 .40

.30

.20

. 10

.40 .30

.20

. 10

.30 .20

. 10

.20 .10

169 145 1 4C> 1

1 20 1

1 7 1 3 1

1 1 81

7 5 1

(f 4 1 \j 249

1 1881

tj V 1 34 23 1 20 131

13 8 1 7 4 1 6 31

305

1 199 1 82 50 1 37 21 1 19 10 1 13 6 1 7 31

328

1 181 1 81 41 1 37 171

20 81

1

1

4 1 328

1 1481

82 33 1 34 12 1 17 5 1 305

i 107 1 70 21 1 25 6 1 249

1 62 1 51 10 1 169

1 25 1

I 338 2941

1 fti> Rfi o\} 1

1 4< 1 1 27 1 22 17 1 14 10 1

1 2 0V 1 4i 1 498

1 3821

1 d.ft 1 ft'^ 1 68 481

40 271

26 17 1 14 81

12 7 1 610

1 406 1 164 1041

74 45 1 38 21 1 26 14 1 14 71

656

1 371 1 162 86 1 74 371

40 181

22 9 1

1 656 305 1 164 701

68 271

34 12 1 610

1 2221

140 46 1 50 14 1 498

1 131 1 102 221

338

1 541

507 444 1 1 1io 1 1 1 1

€ %J TJo 1 O 1 tt^ 1

1 33 26 1 21 16 1 1 o 1 4i 1 I 1 Q AO 1

1 747

1 5781

1 17 1 102 74 1 60 421

2fi 1 21 131

18 111

915

1 615 1 246 159 1 1 1

1

69 1 57 331

39 22 1 21 1 1 1 984

1 5621

243 1321

1 1

1

571

60 29 1 33 15 1

1 984 4631

246 109 1 102 42 1 51 19 1

! 915 339 1 210 71 1 75 22 1

1 747 201 1 153 36 1

1 507 85 1

I 225 1931

t)0 Do 1 ^ t t O 1 ID 1 15 1 1

1

10 7 1 oQ O 1 o Qo 1

1 337

1 2541

Q A. O < 1 43 29 1 27 17 1 1 1 1 1 1

1

6 1 8 4 1 414

1 270 1 107 65 1 48 27 1 27 14 1 17 81

10 4 1 446

1 246 1 1 1

1

56 1 48 221

27 1 1 1 15 5 1

1 446 201 1 107 431

43 15 1 21 6 1 414

1 145 1 94 281

34 81

1 337 841

68 13!

1 225 33 1

I 450 39 11

1 oo 1 lo 1 OO DO 1 QQ 1 30 231

20 15 1

1 o 1 1 1

1 k) 7 1 668

1 512 1 loo loo 1 86 61 1 54 371

22 131

16 9 1 707

1 469 1 214 135 1 96 571

54 301

34 181

20 9 1 723

1 406 1 222 1 181

96 471

54 25 1 30 121 723

1 3331

214 921

86 33 1 42 14 1 707

1 255 1 188 611

68 19 1 668

1 176 1 136 30 1 450

1 721

675 591 1 IT"* 1 1 /AO Qc; 1 OO \ OO O 1 1 45 36 i 30 23 1 lo 1 1 Pt lo 1 1 1 1 1 1 1000

1 7731

O 1 1 1 ^111

129 931

81 571

O 1 QA 1 33 211

24 15 1 1000

1 669 1 321 2071

144 89 1 81 481

51 29 1 30 16 1

1 1000 5671

333 181 1 144 741

81 39 1 45 201

1000

1 466 1 321 1421

129 521

63 231

1000

I 367 1 282 95 1 102 30 1 1000

1 2701

204 481

675

1 1131

67
Table 28

VALUES OF n,M FOR SELECTING, WITH P(CS) >= P*, A SUBSET OF k = 4

BINOMIAL POPULATIONS raiCH CONTAINS ALL POPULATIONS BETTER THAN AN

UPPER STANDARD AND NO POPULATIONS WORSE THAN A LOWER STANDARD

(POPULATIONS WITH NO. OF 'SUCCESSES' >= M OUT OF n TRIALS ARE SELECTED)

P* = .90 1 P* = .95 1 STANDARD 1 1 1 PROPORTIONS! 1 1 UPPER LOWER

1

n M n M n M 1 n M n M n K 1

. 90 . 80 1 190

1 163 1 3801 330 1 5701 499 1 252

1 216 1 5041 4381 756

1 66 1 1

. 70 1 57 47 1 1 14 96 1 171 146 1 78 641 156 1321 234 200 1

. 60 1 29 23 1 58 47 1 87 73 1 38 301 76 621 1 14 95 1

. 50 1 17 13 1 34 27 1 5 1 4 1 1 24 181 48 381 72 59 1 !40 1 12 9 1 24 181 36 29 1 15 1 1 1 30 23 1 45 36 1 .30 1 9 6 1 18 13 1 27 21 1 12 8 1 24 181 36 28 1

. 20 1 6 4 1 12 9 1 18 13 1 8 5 1 16 1 1 1 24 18 1

. 10 1 5 3 1 10 7 1 15 1 1 1 5 3 1 10 6 1 15 1 1 1 .80 .70 1 281

1 212 1 562

1 431 1 843

1 652 1 374

1 282 1 687 526 1 1000

1 771 1 .60 1 77 55 1 154 1 13 1 23 1 173 1 104 74 1 208 153 1 312 233 1

. 50 1 37 25 1 74 52 1 1 1

1

80 1 49 33 1 98 69 1 147 106 1

. 40 1 22 14 1 44 30 1 66 46 1 30 19 1 60 41 1 90 63 1 ^30 1 15 9 1 30 20 1 45 31 1 17 10 1 34 22 1 51 34 1 .20 1 9 5 1 18 1 1 1 27 17 1 1

1

6 1 22 13 1 33 21 1 10 1 6 3 1

1 12 7 11 18 1 1 11 8 4 1 16 9 1 24 15 1 .70 .60 1 348 227 1 674 448 1 1000 672 1 457

1 298 1 728 482 1 10001 667 1 .50 1 92 56 1 184 1 17 1 276 1781 120 73 1 240 1521 360 232 1 OiCi 1 *T\r 1 44 25 1 88 53 1 132 82 1 55 31 1 1 10 66 1 165 1021

.30 1 23 12 1 46 26 1 69 41 1 29 15 1 58 33 1 87 5 1 1 .20 1 15 7 1 30 16 1 45 25 1 17 81 34 181 5 1 281

. 10 1 9 4 1 18 9 1 27 14 1 12 5 1 24 121 36 19 1 .60 .50 1 377

1 2081 6881 3881 10001 5701 4951 2731 7471 4181 10001 565 1 .40 1 93 47 1 186 99 1 279 151 1 123 62 1 246 130 1 369 200 1 .30 1 44 20 1 88 44 1 132 681 55 25 1 1 10 54 1 165 85 1 .20 1 22 9 1 44 20 1 66 32 1 30 12 1 60 27 1 90 44 1

. 10 1 12 41 24 101 36 16 1 15 5 1 30 12 1

[ 45 20 1 .50 .40 1 377

1 1701 6881 319 1 1000

1 469 1 4951 2231 747 343 1 10001 465 1 .30 1 92 37 1 184 79 1 276 122 1 120 481 240 103 1 360 159 1 .20 1 37 13 1 74 29 1 1 1

1

45 1 49 17 1 98 381 147 60 1

. 10 1 17 5 1 34 1 1 1 51 19 1 24 7 1 48 16 1 72 27 1

1 .40 .30 1 343

1 122 1 6741 245 1 10001 370 1 4571 1601 7281 2621 1000 366 1 .20 1 77 23 1 154 501 231 781 104 31 1 208 681 312 106 1

. 10 1 29 7 1 58 16 1 87 26 1 38 9 1

1 76 21 1

1 1 14 34 1

1 .30 .20 1 281

1 70 1 5621 1481 843

1 227 1 374 93 1 687 180 1 1000 2681

. 10 1 57 1 1 1 1 14 25 1 171 40 1 78 15 1 156 341 234 55 1

1 .20 . 10 1 190

1 281 3801 611 570

1 96 11- 2521 371 5041 81 1 756 127 1

1

68
Table 29

VALUES OF n,M FOR SELECTING, WITH P(CS) >= ?*, A SUBSET OF k = 5 BINOMIAL POPULATIONS WICH CONTAINS ALL POPULATIONS BETTER THAN AN

UPPER STANDARD AND NO POPULATIONS WORSE THAN A LOWER STANDARD

(POPULATIONS WITH NO. OF 'SUCCESSES' >= M OUT OF n TRIALS ARE SELECTED)

1 P* = .90 ) P* = .95

STANDARD I I PROPORTIONS! I UPPER LOt^ERI nMnMnMlnPInNnM I

. 90 . 80 1 210 180 1 420 365 1 630 551 1 266 228 1 532 462 1 798 6981

.70 ! 62 511 124 104 1 186 159 1 83 681 166 140 1 249 213 1

. 60 1 33 26 1 66 54 1 99 83 1 42 33 1 84 69 1 126 105 1

. 50 1 20 15 1 40 32 1 60 49 1 25 19 1 50 40 1 75 6 1 1 .40 1 14 10 1 28 22 1 42 33 1 18 13 1 36 28

1

54 43 1

. 30 1 10 7 1 20 15 1 30 23 1 12 81 24 181 36 281

.20 1 6 4 1 12 81 18 13 1 9 6 1 18 131 27 20 1

. 10 1 5 3 1 10 7 1 15 1 1 1 5 3 1

1 10 6 1

1 15 10 11

. 80

1

. 70 1 309

1 233 1 618

1 474 1 927

1 716 1 402 303 1 701 536 1 1000 770 1 .60 1 87 62 1 174 1281 261 195 1 1 1

1

79 1 222 163 1 333 249 1 .50 1 40 27 1 80 56 1 120 87 1 52 35 1 104 73 1 156 113 1

. 40 1 25 16 1 50 34 1 75 53 1 30 19 1 60 40 1 90 63 1 .30 1 15 9 1 30 19 1 45 301 20 12 1 40 26 1 60 40 1

. 20 1 9 5 1 18 1 1 1 27 17 1 lo ( 1 1 ^ 1 ID 1 t

. 10 1 6 3 1 12 71 13 11 1 9 41

1 18 1011 27 16 1

1 .70

1 .60 1 377

1 246 1 688

1 457 1 1000

1 6701 486 3171 743 491 1 1000 666 1 .50 1 102 62 1 204 129 1 306 1981 130 79 1 260 165 1 390 252 1 .40 1 46 26 1 92 55 1 138 85 1 57 32 1 1 14 681 171 106 1 .30 11 25 13 1 50 281 75 44 1 31 16 1 62 35 1 93 55 1 .20 11 15 71 30 16 1 45 25 1 20 9 1 40 21 1 60 34 1

. 10 11 10 41 20 10 1 30 16 1 12 5 1

1 24 11 1

1 36 19 1

1 .60

1 .50 1

I

i 415

1 229 1 707

1 3981 1000

1 5681 535 295 1 767 429 1 1000 564 1 .40 11 103 52 1 206 109 1 309 1681 133 67 1 266 141 1 399 217 1 .30 I1 46 21 1 92 46 1 138 71 1 57 26 1 1 14 56 1 171 831

.20 1 25 10 1 50 23 1 75 36 1 30 121 60 27 1 90 43 1

. 10 1 14 5 1 28 1 1 1 42 19 1

1 18 6 1

1 36 15 11 54 24 1

1 ,50 .40

1

1 415

1 187 1 707

1 326 1 1000 4681 535 241 1 767 351 1 1000 463 1 .30 1 102 41 1 204 87 1 306 135 1 130 521 260 nil 390 172 1 .20 1 40 14 1 80 31 1 120 49 1 52 181 104 40 1 156 64 1

. 10 1 20 6 1 40 14 1 60 221

1 25 7 1

1 50 17 1

1 75 231

1 .40 .30

1

1 377

1 132 1 688

1 249 I 1000 369 1 486 1701 743 266 1 1000 3641

.20 1 87 26 1 174 57 1 261 881 1 1

1

331 222 72 1 333 1 13 1

. 10 1 33 81 66 181 99 301 42 10 1

1 84 23 1

1 126 3811 .30 .20

1

1 309

1 771 618

1 1621 927

1 2501 402 100 1 701 182 1 1000 267 1

. 10 1 62 12 1 124 27 1 186 43 1

1 83 16 1

1 166 36 11 249 581

1 .20 . 10

1

1 210

1 31 1 420

1 681 630 106 1 266 39 1 532 85 1 798 1341

69
Table 30

VALUES or n,M FOR SELECTING, WITH PCCS) >= P*, A SUBSET OF k = 6

BINOMIAL. POPULATIONS WHICH CONTAINS ALL POPULATIONS BETTER THAN AN

UPPER STANDAiU) AND NO POPULATIONS WORSE THAN A LOWER STANDARD

(POPULATIONS WITH NO. OF 'SUCCESSES' >= M OUT OF n TRIALS ARE SELECTED)

r - 1

1 P* = . yo 1 STANDARD 1 1 1 PROPORTIONS! 1 ! UPPER LOIVERI 11 M n M n M 1 n M n M n M !

.90

1 .80 1 2241 1921 4481 389 1 672! 588! 286I 245 1 572

! 497! 858! 751 ! .70 1 oo 1 Dt) 1 11 OtJ 1 1 1 1 i D 1 1 '7A 1 1 < ^ 1 ODO Q 1 * o OA ^ oo 1 .60 1 34 27 1 68 56 1 102 85 ! 46 36 ! 92 75 1 138 115 ! .50 1 21 16 1 42 33! 63 51 ! 28 21 ! 56 45 1 84 68!

. 4W 1 15 1 1 1 30 231 45 36 ! 18 131 36 281 54 43!

.30 1 10 71 20 15 ! 30 23 ! 13 9 ! 26 19 ! 39 30 ! .20 1 6 4 1 12 8! 18 13 ! 9 6 ! 18 13! 27 20!

. 10 1 Qo I

1 1 Ck 1 VJ T 1 1 1 1 o 1 1 1 1 1 1

! 9 ! 21 lO 1

'"7 f\ 1 . 70 1 333

1 251 1 666! 511! 999 772 ! 426! 321 ! 713! 544 ! 1000

1 769 1 .60 1 94 671 188 139 ! 282 211 1 1 18 84! 236 174 ! 354 265 1 .50 1 43 29 1 86 61 ! 129 93 1 55 37 ! 1 10 781 165 1 19 ! .40 1 1 ^ r 1 o 1 ou Q-d. 1 o"* 1 < O OA 1 43 1 96

. dO 1 17 10

1

34 22 ! 51 34 ! 22 13! 44 281 66 45 ! .20 1 11 6 1 22 13! 33 21 ! 13 7! 26 16 1 39 25 !

. 10 1 6 3 1 12 6 1 18 1 1 ! 9 4! 18 10 ! 27 16 ! .70 .60 1 409 267 1 704 467 ! 1000 669 ! 520 339 ! 760

! 5021 1000 665 I

. 50 1 107 65 1 214 135 ! 321 207 ! 135 82! 270 171 ! 405 261 1 .40 1 OT 1 Oo 1 1 /H, A, DQ 1 QoO 1

1 1 O/l. 74 1 186 1 lO 1 .30 1 27 14 1 54 31 1 81 48! 33 17! 66 37 1 99 58!

.20 1 17 81 34 181 51 29 ! 22 10 ! 44 231 66 37!

. 10 1 10 4 1 20 10 ! 30 16 ! 13 5 ! 26 121

1 39 20 1 .60 .50 1 444

1 245 1 722! 405 ! 1000

1 5671 5661 312! 783 437 ! 1000

1 5631

. 40 1 109 55 1 218 1 15 ! 327 177 ! 139 70 ! 278 147 1 417 226 ! .30 1 48 22 1 96 47 ! 144 74! 62 28! 124 61 1 186 96 ! .20 1 25 10 1 50 23 1 75 36 ! 32 13 ! 64 29 1 96 46 !

. 10 1 15 5 1 30 12 1 45 20 1 18 6 1 36 15 !

1 54 24!

.50 .40 1 444

1 200 1 722

1 333! 10001 467 1 566! 255 ! 783 358! 1000! 462!

.30 1 107 43 1 214 92 ! 321 142 1 135 54 ! 270 1 15 1 405 179 ! .20 1 43 15 1 86 33! 129 53 ! 55 19 ! 1 10 43 1 165 671

. 10 1 21 6 1 42 14 ! 63 23 ! 28 81 56 19 1

1 84 31 1 .40 .30 1 409

1 143 1 704! 254 ! 1000! 367 ! 5201 1821 760 272! 1000! 363!

.20 1 94 281 188 61 ! 282 96 ! 1 18 35 1 236 77! 354 120!

. 10 1 34 81 68 19 1 102 31 1 46 1 1 !

! 92 26 1 138 4211 .30 .20 1 333

1 831 6661 175 ! 999

1 269 ! 426 106 ! 7131 185 1 1000 266 !

. 10 1 68 13 1 136 30! 204 48! 88 171 176 39 ! 264 62!

! .20 . 10 1 224

1 33 1 448! 72! 672

! 1131

!- 286

! 42 ! 572

! 92 1 858 144 !

1

70
Table 31

VALUES OF n,M FOR SELECTING, WITH P(CS) >= F*, A SUBSET OF k = 7

BINOMIAL POPULATIONS WHICH CONTAINS ALL POPULATIONS BETTER THAN AN

UPPER STANDARD AND NO POPULATIONS WORSE THAN A LOWER STANDARD

(POPULATIONS WITH NO. OF 'SUCCESSES' >= N OUT OF n TRIALS ARE SELECTED)

P* = . 90 1 P* = . 95 1 STANDARD 1 1 1 PROPORTIONS! 1 1 UPPER LOWER

1

n M n N n M 1 n M n M n M 1

.90

1 .80 1 2381 204 1 4761 414 1 7141 625 1 3001 2571 6001 521 1 9001 787 1 .70 1 73 60 1 146 123 1 219 187 1 89 73 1 178 150 1 267 228 1 .60 1 37 29 1 74 61 1 1 1

1

93 1 46 36 1 92 75 1 138 1 15 1 .50 1 21 16 1 42 33 1 63 51 1 28 21 1 56 44 1 84 631

ACt 1 . TJV 1 15 1 1 1 30 23 1 45 36 1 18 13 1 36 27 1 54 43 1 .30 1 10 ^ 1 20 15 1 30 23 1 13 9 1 26 19 1 39 30 1 .20 1 8 5 1 16 1 1 1 24 181 10 6 1 20 14 1 30 22 1

. 10 1 5 3 1 10 7 1 15 1 1 1 7 4 1 14 9 1 2

1

15 1 R0 . 70 1 354

1 267 1 6771 519 1 1000

1 772 1 4461 336 1 7231 552 1 10001 769 1 !60 1 100 71 1 200 147 1 300 225 1 125 89 1 250 184 1 375 281 1 .50 1 46 31 1 92 65 1 138 100 1 58 39 1 1 16 82 1 174 126 1 .40 1 27 17 1 54 36 1 81 57 1 35 22 1 70 47 1 105 74 1

. 30 1 17 10 1 34 221 51 34 1 22 131 44 281 66 441

!20 1 1

1

6 1 22 13 1 33 21 1 13 7 1 26 15 1 39 25 1

. 10 1 8 4 1 16 9 1 24 15 1

1 10 5 11 20 1 1 11 30 1811 .70 .60 1 434

1 2831 717

1 475 1 1000 6681 543 354 1 771 5081 1000 664 1 .50 1 1 12 681 224 142 1 336 217 1 145 88 1 290 184 1 435 281 1 .40 1 53 30 1 106 64 1 159 99 1 64 36 1 128 77 1 192 1 19 1 .30 1 27 141 54 301 81 481 35 181 70 39 1 105 62 1 .20 1 17 81 34 181 51 281 22 101 44 23 1 66 37 1

. 10 1 10 4 1 20 9 1 30 15 1

1 13 5 11 26 12 1

1 39 20 11 .60 .50 i 466

1 2571 733

1 411 1 1000 566 1 595 3281 797 444 1 1000 5621

.40 1 117 59 1 234 1241 351 191 1 147 74 1 294 156 1 441 239 1 .3© 1 53 24 1 106 53 1 159 821 64 29 1 128 63 1 192 981

.20 1 27 1 1 1 54 25 1 81 39 1 35 141 70 32 1 105 51 1

. 10 1 15 5 1 30 121 45 201 18 6 1

1 36 14 11 54 24 11 .50 .40 1 466

1 2101 733

1 3371 1000

1 466 1 595 2681 797 3641 1000 461 1 .30 1 112 45 1 224 96 1 336 1481 145 581 290 124 1 435 192 1 .20 1 46 16 1 92 36 1 138 56 1 58 201 1 16 45 1 174 71 1

. 10 1 21 6 1 42 14 1 63 23 1 28 81

1 56 19 1

1 84 31 1

1 .40 .30 1 434

1 152 1 717

1 2581 1000

1 366 1 543 190 1 771 275 1 1000 362 1 .20 1 100 301 200 65 1 300 1021 125 371 250 81 1 375 127 1

. 10 1 37 9 1 74 21 1 1 1

1

33 1 46 1 1 1

1 92 26 1

1 138 41 11 .30 .20 1 354

1 881 677

1 1771 1000

1 269 1 446 1 1 1 1 723 187 1 1000 265 1

. 10 1 73 141 146 321 219 511 89 171

1 178 39 1

1 267 62 11 .20 . 10 1 238

1 35 1 476

1 771 714

1 1201 300 44 1 600 971 900 151 1

71
Table 32

VALUES OF n,M FOR SELECTING, WITH P(CS) >= P*, A SUBSET OF k = O

BINOMIAL POPULATIONS WHICH CONTAINS ALL POPULATIONS BETTER THAN AN

UPPER STANDARD AND NO POPULATIONS WORSE THAJf A LO^/ER STANDARD

(POPULATIONS WITH NO. OF 'SUCCESSES' >= M OUT OF n TRIALS ARE SELECTED)

P* = .90 1 P:ic = .95 1 STANDARD 1 1 1 PROPORTIONS! 1 1 UPPER LOWER

1

n M n M n M 1 n M n M n K 1

.90

1 .80 1 245

1 2101 4901 426 1 7351 643 1 3071 2631 614

1 533 1 921

1 805 1

. 70 1 73 60 1 146 123 1 219 187 1 94 77 1 188 1581 282 241 1

. 60 1 38 30 1 76 621 1 14 95 1 47 371 94 771 141 1171

. 50 1 24 18 1 43 38 1 72 59 1 28 2 1 1 56 44 1 84 68 1

. 40 1 15 1 1 1 30 23 1 45 36 1 21 15 1 42 32 1 63 50 1 .30 1 10 7 1 20 15 1 30 23 1 15 10 1 30 22 1 45 35 1 .20 1 8 5 1 16 1 1 1 24 181 10 6 1 20 14 1 30 22 1

. 10 1 5 3 1 10 7 1 15 1 1 1 7 4 1 14 9 1 21 15 1

. 80 . 70 1 370

1 279 1 6851 524 1 10001 771 1 4661 35 1 1 733

1 559 1 10001 7681

!60 1 101 72 1 202 149 1 303 227 1 128 9 1 1 256 188 1 384 287 1 .50 1 49 33 1 98 69 1 147 106 1 6 1 41 1 122 86 1 183 132 1

. 40 1 30 19 1 60 4 1 1 90 63 1 35 22 1 70 47 1 105 73 1

. 30 1 17 10 1 34 22 1 5 1 34 1 22 13 1 44 281 66 441

!20 1 1 1 6 1 22 13 1 33 21 1 15 81 30 18 1 45 29 1

. 10 1 8 4 1 16 9 1 24 15 1 10 5 1 20 1 1 1 30 181

,70 .60 1 454

1 296 1 7271 481 1 1000

1 6681 566

1 369 1 7831 516 1 1000

1 664 1 .5© 1 1 17 71 1 234 1481 35 1 226 1 150 9 1 1 300 190 1 450 290 1 .40 1 55 3 1 1 110 66 1 163 102 1 66 37 1 132 79 1 198 122 1

• 1 29 15 1 58 331 87 51 1 37 19 1 74 421 1 1

1

65 1 ,20 1 17 81 34 181 5 1 281 22 10 1 44 23 1 66 37 1

. 10 1 10 4 1 20 9 1 30 15 1 15 6 1 30 15 1 45 241

.60 ,50 1 495

1 273 1 7471 4131 1000

1 565 1 615

1 339 1 8071 449 1 10001 561 1 .40 1 121 61 1 242 1281 363 197 1 153 771 306 162 1 459 249 1 .30 1 55 25 1 1 10 54 1 165 85 1 66 301 132 65 1 198 102 1 .20 1 30 12 1 60 27 1 90 44 1 35 14 1 70 32 1 105 501

. 10 1 15 5 1 30 121 43 20 1 21 71

1 42 171

1 63 2811 .50 .40 1 495

1 223 1 747

1 343 1 10001 465 1 615 277 1 807 3681 1000 461 1 .30 1 1 17 47 1 234 100 1 351 155 1 150 601 300 1281 450 199 1 .20 1 49 17 1 98 381 147 60 1 61 21 1 122 471 183 75 1

. 10 1 24 7 1 48 16 1 72 27 1 28 81 56 19 1 84 311

.40 .30 1 454

1 159 1 7271 262 1 10001 366 1 5661 1981 7831 279 1 10001 362 1 .20 1 101 30 1 202 65 1 303 102 1 128 381 256 83 1 384 130 1

. 10 1 38 9 1 76 21 1 1 14 34 1 47 1 1 1 94 26 1

1 141 42 11 .30 .20 1 370

1 92 1 685

1 179 1 10001 2681 4661 1 16 1 733 189 1 1000 264 1

. 10 1 73 14 1 146 32 1 219 51 1

1 94 1811 188 41 11 282 66 11 .20 . 10 1 245

1 36 1 490

1 79 1 735 123 11- 307 45 1 614 99 1 921 155 1

1

72
Table 33

VALUES OF n.M FOR SELECTING, WITH P(CS) >= ?*, A SUBSET OF k = 9

BINOMIAL POPULATIONS WHICH CONTAINS ALL POPULATIONS BETTER THAN AN

UPPER STANDARD AND NO POPULATIONS WORSE TOAN A LOWER STANDARD

(POPULATIONS WITH NO. OF 'SUCCESSES' >= M OUT OF n TRIALS ARE SELECTED)

I P* = .90 I P* = .95 I STANDARD I I I PROPORTIONS I I I UPPER LOWERI n M n N n MIn M n M n Ml

I I I I I .90 .80 1 259 2221 518 450 1 777 6801 321 275 1 642 5581 963 8421

TO{a 1 loo loll OAA 1

1 Q 1 1 Ol 1 lyti lO ( 1 00*7 <sy < ^04< 1 .60 1 38 30 1 76 621 1 14 95 1 51 401 102 83 1 153 1281

.50 1 24 181 48 381 72 59 1 31 231 62 49 1 93 76 1 .40 1 17 121 34 26 1 51 41 1 21 15 1 42 32 1 63 501

.30 1 12 81 24 181 36 281 15 101 30 221 45 35 1

1

1 QO O 1 1lO^ 1 1 1 24 1 O 1 1

1

"7 1 ( 1 i D 1 OO 1

1

. 10 1 5 3! 10 6 1 15 10 1 7 41 14 9 i

1 21 15 11 .80

1 .70 1 386

1 291 1 6931 5301 10001 771 1 479

1 361 1 739 5631 1000 7671

^ A 1 . t)W 11 1 AO '7'7 1 ( f 1 159 1 324 O/tO 1 ^tti 1 1 oO OA 1 OvO 1 .50 11 52 35 1 104 741 156 1 131 63 421 126 89 1 189 1371

.40 11 30 19 1 60 41 1 90 631 35 221 70 471 105 731

1 1 1 oo 241 57 oo 1 Ir* 1 1O 1 1 1 1 .20 11 13 71 26 16 1 39 25 1 15 81 30 181 45 29 1

. 10 11 8 41 16 9 1 24 14 1 1

1

5 1

1 22 12 11 33 20 1

1 .70

1 .60 11

1 469

1 306 1 734

1 4851 1000

1 667 1 586 3821 793 5221 1000 663 1 .50 11 122 741 244 154 1 366 236 1 155 941 310 196 1 465 300 1 .40 11 55 31 1 1 10 66 1 165 1021 70 39 1 140 841 210 130 1 .30 11 31 16 1 62 35 1 93 55 1 37 19 1 74 421 1 1

1

65 1 .20 11 19 9 1 38 20 1 57 321 24 111 48 25 1 72 401

. 10 11 12 51 24 121 36 19 1 15 61 30 14 1

1 45 23 1

1 .60

1 .50 1

1

1 515

1 2841 757

1 4231 1000

1 565 1 635

i 3501 817 455 1 1000 561 1 .40 11 127 641 254 135 1 381 2071 157 79 1 314 166 1 471 256 1 .30 11 55 25 1 1 10 541 165 85 1 70 321 140 69 1 210 1081

.20 1 30 121 60 271 90 431 35 141 70 321 105 501

. 10 1 17 6 1 34 141 51 23 1 21 71

1 42 171

1 63 281

1 .50 .40

1

1 515

1 2321 757 3471 1000

1 464 1 635 286 1 817 3721 1000 4601

.30 1 122 49 1 244 1041 366 161 1 155 621 310 1331 465 205 1 .20 1 52 181 104 401 156 641 63 221 126 49 1 189 77 1

. 10 1 24 71 48 16 1 72 26 1

1 31 9 1

1 62 21 1

1 93 341

1 .40 .30

1

1 469

1 1641 734 2641 1000 365 1 586 205 1 793 2821 1000 361 1 .20 1 108 321 216 701 324 1 10 1 135 401 270 881 405 1371

. 10 1 38 9 1 76 21 1 114 341 51 121

1 102 29 1

1 153 46 1

1 .30 .20

1

1 386

1 96 1 693 181 1 1000

1 2671 479 1191 739 191 1 1000 2641

. 10 1 78 15 1 156 341 234 55 1 99 19 1

1 198 431

1 297 70 1

1 .20 . 10

1

1 259

1 381 518

1 831 777

1 131 1 321 471 642 1031 963 1621

73
Table 3^

VALUES OF n,M FOR SELECTING, VITH P(CS) >= ?*, A SUBSET OF k = 10

BINOMIAL POPULATIONS VHICH CONTAINS ALL POPULATIONS BETTER THAN AN

UPPER STANDARD MB NO POPULATIONS WORSE THAN A LOWER STANDARD

(POPULATIONS WITH NO. OF 'SUCCESSES' >= M OUT OF n TRIALS ARE SELECTED)

STANDARD 1 PROPORTIONS!

UPPER LOWER

1

n. M

P* =

n

.90

M n

1

1

1

1 M i n ri

PJf: =

n

.95

Mn n

1

1

1

1 nM 1

1

.90

1 .80 1 2661 2281 532

1 4621 798

1 6981 3281 281 1 6561 5701 984 861 1 .70 1 83 681 166 1401 249 2131 100 821 200 1681 300 256 1

. 60 1 OO 1 oo 1 OA 1 IMo 1 O 1 4M 1 1 AO OO 1 i J? O loo 1 0*7 1 .50 1 1 O 1 OU 70 ^ 1 1 6 1 1 o o £^ A R 1 1 o 1 1 7o 1

. 40 1 18 lo 1 36 E? A 54 4o 1 O -1 15 1 A O42 OO 1 63 50 1 .30 1 12 n 1 8 1 O A t O 1 18 1 O £. 36 <^8 1 15 10 1 O /X 30 OO 1 22 1 45 35 1 .20 1 8 5 1 16 1 1 1 24 181 1

1

71 22 16 i 33 25 1

. 10 1 5 31 10 6 1 15 10 1 7 41 14 9 1 21 15 1 .80 .70 1 398

1 300 1 6991 534 1 10001 770 1 49 11 370 1 7451 5671 10001 767 1 .60 1 111 79 1 222 163 1 333 249 1 135 96 1 270 199 1 405 303 1 .50 1 52 35 1 104 73 1 156 1 13 1 64 43 1 128 901 192 139 1

. 40 1 30 19 1 60 401 90 631 38 241 76 51 1 1 14 801

.30 1 20 12 1 40 26 1 60 40 1 24 14 1 48 31 1 72 48 1 .20 1 13 7 1 26 16 1 39 25 1 15 8

1

30 181 45 29 1

. 10 1 8 4 1 16 9 1 24 14 1 1 1 5 1 22 121 33 201

.70 .60 1 483

1 315 1 74

1

1 490 1 10001 666 1 603

1 393 1 801

1 5271 1000

1 662 1 .50 1 127 771 254 161 1 381 246 1 155 94 1 310 196 1 465 3001

.40 1 57 32 1 1 14 681 171 106 1 73 41 1 146 881 219 136 1 .30 1 31 16 1 62 35 1 93 55 1 39 201 78 44 1 117 69 1 .20 1 20 9 1 40 21 1 60 34 1 24 1 1 i 48 25 1 72 40 1

. 10 1 12 5 1 24 1 1 1 36 19 1 15 6 1 30 14 1 45 23 1 .60 .50 1 526

1 2901 7631 426 1 10001 564 1 6551 361 1 827

1 460 1 10001 5601

.40 1 131 66 1 262 139 1 393 213 1 163 821 326 1731 489 2651

.30 1 57 26 1 1 14 56 1 171 881 73 331 146 72 1 219 1131

.20 1 30 121 60 271 90 431 38 15 1 76 35 1 1 14 55 1

. 10 1 18 6 1 36 15 1 54 241 21 71 42 17 1 63 281

.50 .40 1 526

1 237 1 7631 350 1 1000

1 464 1 655

1 295 I 827 377 1 10001 459 1 .30 1 127 51 1 254 109 1 381 1681 155 621 310 1321 465 205 1 .20 1 52 181 104 40 1 156 64 1 64 221 128 50 1 192 781

. 10 i 25 71 50 171 75 281 32 9 1 64 221 96 35 1 .40 .30 1 483

1 169 1 7411 266 1 10001 3641 6031 211 1 8011 285 1 10001 3601

.20 1 1 1

1

33 1 222 721 333 1 13 1 135 401 270 881 405 137 1

. 10 1 42 10 1 84 231 126 381 51 121 102 281 153 46 1 .30 .20 1 398

1 99 1 6991 1821 10001 267 1 4911 1221 745 1921 10001 2631

. 10 1 83 16 1 166 36 1 249 531 100 19 1 200 44 1 300 701

.20 . 10 1 266

1 39 1 5321 86 1 798

1 134 i1- 3281 481 6561 105 1 984

1 165 1

1

74
Table 35

VALUES OF n.M FOR SELECTING, WITH P(CS) >= P*, A SUBSET OF k = 11 BINOMIAL POPULATIONS WHICH CONTAINS ALL POPULATIONS BETTER THAN AN

UPPER STANDARD AND NO POPULATIONS WORSE THAN A LOWER STANDARD

(POPULATIONS WITH NO. OF 'SUCCESSES' >= M OUT OF n TRIALS ARE SELECTED)

P* = .90 1 P* = .95

STANDARD 1 1 PROPORTIONS!

-

1 UPPER LOWER 1 n M n M n M 1 n M n M n M

.90 .80 1 273

1 2341 5461 4741 8191 716 1 3351 2871 6671 579 1 1000 875

.70 1 84 69 1 168 1421 252 215 1 100 821 200 1681 300 256

.60 1 f\Ot V V 1

1 1

1

5 1 11 OiO V/ u Rfl IJO 1

1 1 o o 1 27

.50 1 25 191 50 401 75 611 32 24 1 64 51 1 96 78

.40 1 18 131 36 281 54 43 1 21 15 1 42 321 63 50

OA 1 12 81 24 181 36 281 15 10 1 30 22 1 45 35

.20 1 9 6 1 18 131 27 20 1 1

1

71 22 16 1 33 25

. 10 1 5 31 10 6 1 15 10 1 7 41 14 9 1

1 21 15

.80 .70 1 410

1 309 1 7051 539 1 1000

1 7701 5031 379 1 751 572 1 1000 767

. O0 1 1 14 81 1 228 1681 342 256 1 141 1001 282 2081 423 316

.50 1 55 371 1 10 781 165 120 1 66 44 1 132 93 1 198 143

.40 1 30 19 1 60 40 1 90 63 1 38 241 76 51 t 1 14 80

.30 1 O 1 oo 1 4i 1 1 *x 1 O 1 1 c ^

. 20 1 13 71 26 16 1 39 25 1 15 81 30 181 45 29

. 10 1 9 4 1 18 10 1 27 16 1 1

1

5 1

1 22 121

1 33 20

.70 . 60 1 RAO

1 OOO 1 Oi^O 1 <o i

1 AQ^ 1 TJVD 1 1 AHA

1 OOO 1 D 1 O 1 1 t AAA

.50 i 132 801 264 167 1 396 256 1 160 971 320 202 1 480 310

.40 1 61 341 122 731 183 1 13 1 73 411 146 871 219 135

.30 1 o o 17 1 DO Oo 1 OA 1 ( o A<dj 1

! lit

.20 1 21 101 42 22 1 63 35 1 24 1 1 1 48 25 1 72 40

. 10 1 12 5 1 24 11 1 36 19 1 15 6 1 30 14 1

1 45 23

.60 .50 1 546

1 301 1 773

1 4321 1000

1 564 1 673

1 371 1 836 465 1 1000 559

.40 1 135 681 270 1431 405 220 1 167 841 334 177 1 501 272

.30 1 61 281 122 601 183 94 1 73 331 146 721 219 1 12

.20 1 30 121 60 27 1 90 431 38 15 1 76 34 1 1 14 55

. 10 1 18 6 1 36 15 1 54 24 1 21 71

1 42 171

1 63 28

.50 .40 1 546

1 246 1 773

1 3541 1000

1 4631 673 3031 836 3801 1000 459

.30 1 132 531 264 1131 396 175 1 160 641 320 137 1 480 212

.20 1 55 19 1 110 431 165 671 66 231 132 51 1 198 81

. 10 1 25 71 50 171 75 271 32 9 1

1 64 22 1

1 96 35

.40 .30 1 503

1 176 1 751

1 269 1 1000

1 3641 615 215 1 807 2871 1000 360

.20 1 1 14 34 1 228 741 342 1 16 1 141 421 282 92 1 423 143

. 10 1 42 10 1 84 23 1 126 381

1 51 121

1 102 281

1 153 46

.30 .20 1 410

1 102 1 705

1 1831 1000 266 1 503 125 1 751 1931 1000 263

. 10 1 84 16 1 168 37 1 252 59 1 100 19 1

1 200 441

1 300 70

.20 . 10 1 273

1 40 1 546

1 881 819

1 1371 335 49 1 667 1071 1000 168

75
Table 36

VALUES OF n,M FOR SELECTING, WITH P(CS) >= ?*, A SUBSET OF k = 12

BINOMIAL POPULATIONS WHICH CONTAINS ALL POPULATIONS BETTER THAN AN

UPPER STANDARD AND NO POPULATIONS WORSE THAN A LOWER STANDARD

(POPULATIONS WITH NO. OF 'SUCCESSES' >= M OUT OF n TRIALS ARE SELECTED)

STANDARD 1 PROPORTIONS!

UPPER LOWER

1

n M

P* =

n

. 90

M n

1

1

1 M 1 n M

P* =

n

. 95

M n

1

1

1 M 1

. 90

1

. 80 1 286

1 245 1 5721 497 1 8581 75 1 1 3481 2981 674

1 585 i 1000

1 874 1

. 70 1 84 69 i 168 141 1 252 215 1 105 86 1 210 177 1 315 269 1

. 60 1 43 34 1 86 70 1 129 107 1 55 43 1 1 10 90 1 165 138 1 .50 1 28 21 1 56 '45 1 84 69 1 32 24 1 64 51 1 96 781

.40 1 18 13 1 36 28

1

54 43 1 21 15 1 42 321 63 50 1

. 30 1 13 9 1 26 19 1 39 30 1 15 10 1 30 221 45 35 1

. 20 1 9 6 1 18 13 1 27 20 1 1

1

7 1 22 15 1 33 25 1

. 10 1 5 3 1 10 6 1 15 10 1 7 4 1 14 9 1 21 15 1 .80 .70 1 422

1 3181 711

1 5431 1000

1 769 1 5151 3881 757

1 576 1 10001 766 1 .60 1 118 84 1 236 174 1 354 265 1 142 10 I 1 284 209 1 426 319 1

. 50 1 55 37 1 1 10 78

1

165 119 1 69 46 1 138 981 207 150 1 .40 1 32 20 1 64 43 1 96 67 1 40 25 ! 80 54 1 120 84 1 .30 1 21 12 1 42 27 1 63 42 1 24 14 1 48 31 1 72 481

.20 1 13 7 1 26 16 1 39 25 1 15 81 30 181 45 281

. 10 1 9 4 1

1 18 10 1 27 16 1 1

1

5 1 22 121 33 20 1 .70 .60 1 517 337 1 758

1 500 1 10001 665 1 6291 410 1 814

1 535 1 10001 66 1 1 .50 1 135 82 1 270 171 1 405 261 1 165 1001 330 209 1 495 319 1 .40 1 62 35 1 124 74 1 186 1 15 1 75 42 1 150 901 225 139 1 .30 1 33 17 1 66 37 1 99 581 41 21 1 82 46 1 123 72 1 .20 1 21 10 1 42 221 63 35 1 24 1 1 1 48 25 1 72 40 1

. 10 1 13 5 1 26 12 1 39 201 15 6 1 30 14 1

1 45 23 1 .60 .50 1 564

1 311 1 7821 436 1 10001 563 1 6861 3781 843 4681 1000

1 559 1 .40 1 139 70 1 278 1471 417 226 1 169 85 1 338 179 1 507 275 1 .30 1 62 281 124 61 1 186 96 1 75 341 150 741 225 1 16 1 .20 1 32 131 64 29 1 96 46 1 40 16 1 80 36 1 120 581

. 10 1 18 6 1 36 15 1 54 24 1 21 7 1 42 17 1 63 27 1 .50 .40 1 564

1 254 1 7821 3581 10001 4621 6861 309 1 843 3831 10001 4581

.30 1 135 54 1 270 1 16 1 405 179 1 165 66 1 330 141 1 495 2181

.20 1 55 19 1 1 10 43 1 165 67 1 69 24 1 138 54 1 207 85 1

. 10 1 28 81 56 19 1 84 31 1 32 9 1 64 221 96 35 1 .40 .30 1 517

1 181 1 7581 271 1 10001 3631 6291 2201 8141 289 1 10001 359 1 .20 1 1 18 35 1 236 77 1 354 1201 142 421 284 921 426 1441

. 10 1 43 101 86 241 129 39 1 55 131 1 10 31 1 165 501

.30 .20 1 422

1 105 1 7111 185 1 10001 266 1 5151 1281 757 194 1 10001 2621

. 10 1 84 16 1 168 37 1 252 59 1 105 20 1 210 46 1 315 74 1

I .20 . 10 1 286

1 421 5721 921 8581 144 11- 3481 511 6741 1081 1000 1671

1

76
Table 37

VALUES OF n,M FOR SELECTING, WITH P(CS) >= F*, A SUBSET OF k = 13

BINOMIAL POPULATIONS WHICH CONTAINS ALL POPULATIONS BETTER THAN AN

UPPER STANDARD AND NO POPULATIONS WORSE THAN A LOWER STANDARD

(POPULATIONS WITH NO. OF 'SUCCESSES' >= M OUT OF n TRIALS ARE SELECTED)

STANDARD 1 PROPORTIONS 1 UPPER LOWER

1

n M

P* =

n

.90

M n

1

1

1 M 1 n M

P* =

n

.95

M n

1

1

1 M 1

.90

1 .80 1 2931 251 1 586

1 509 1 8791 769 1 3551 304 1 6771 588 1 10001 874 1 .70 1 89 731 178 1501 267 2281 105 86 1 210 177 1 315 269 1 .60 1 46 36 1 92 75 1 138 1 15 1 55 43 1 1 10 90 1 165 138 1 28 21 1 56 44 1 84 681 32 24 1 64 5 1 1 96 78 1 .40 1 18 13 1 36 28 1 54 43 1 21 15 1 42 32 1 63 50 1 .30 1 13 ^ 1 26 19 1 39 30 1 15 10 1 30 22 1 45 34 1 .20 1 10 6 1 20 14 1 30 23 1 1

1

7 1 22 15 1 33 24 1

. 10 1 7 41 14 9 1 21 15 1 7 41 14 9 1 21 15 1 .80 ,70 1 434

1 327 1 7171 547 1 1000

1 769 1 527

1 397 1 763

1 580 1 10001 766 1 .60 1 1 18 84 1 236 174 1 354 265 1 145 103 1 290 213 1 435 325 1 .50 1 58 39 1 1 16 82 1 174 126 1 69 46 1 138 97 1 207 150 1

1 35 221 70 481 105 741 40 25 1 80 54 1 120 841

.30 1 22 13 1 44 281 66 45 1 24 14 1 48 31 1 72 48 1 .20 1 13 7 1 26 16 1 39 25 1 17 9 1 34 21 1 5 1 33 1 10 1 10 5 f 20 1 1 1 30 181 1

1

5 1

1 22 12 1

1 33 20 11 .70 .60 1 526

1 343 1 763

1 503 1 10001 665 1 646 421 1 823 541 1 1000 66 1 1 .50 1 140 85 1 280 177 1 420 271 1 170 1031 340 215 1 510 329 1 .40 1 64 36 1 128 771 192 119 1 75 421 150 90 1 225 139 1 .30 1 33 17 t 66 37 1 99 581 41 21 1 82 46 1 123 72 1 .20 1 22 10 1 44 231 66 371 24 1 1 1 48 25 1 72 40 1

. 10 1 13 5 1 26 121 39 201 15 6 1

1 30 141

1 45 2311 .60 .50 1 575

1 3171 787

1 439 1 1000

1 5631 704 3881 852 473 1 1000 559 1 .40 1 143 721 286 1521 429 233 1 173 871 346 1831 519 282 1 .30 1 64 29 1 128 631 192 99 1 75 341 150 741 225 1 15 1 .20 1 35 141 70 321 105 511 40 16 1 80 36 1 120 581

. 10 1 18 6 1 36 14 1 54 241 21 71

1 42 171

1 63 2711 .50 .40 1 575

1 259 1 787

1 3601 1000

1 4621 704 3171 852 3871 1000 4581

.30 1 140 56 1 280 1201 420 185 1 170 681 340 146 1 510 225 1 .20 1 58 201 1 16 45 1 174 71 1 69 241 138 531 207 84 1

. 10 1 28 81 56 19 1 84 31 1

1 32 9 11 64 21 1

1 96 35 11 .40 .30 1 526

1 1841 763

1 2731 1000 3631 646 226 1 823 2921 1000 359 1 .20 1 118 35 1 236 76 1 354 1201 145 431 290 941 435 1471

. 10 1 46 11 1 92 26 1 138 421 55 131

1

1 10 31 1

1 165 5011 .30 .20 1 434

1 1081 717

1 186 1 1000

1 265 1 527 131 1 763 196 1 1000 2621

. 10 1 89 171 178 391 267 621

1 105 2011 210 46 1

1 315 7311 .20 . 10 1 293

1 431 586

1 941 879 1481 355 521 677 1081 1000 1671

77
Table 38

VALUES OF n,M FOR SELECTING, WITH P(CS) >= P*, A SUBSET OF k = 14

BINOMIAL POPULATIONS WHICH CONTAINS ALL POPULATIONS BETTER THAN AN

UPPER STANDARD AND NO POPULATIONS WORSE THAN A LOWER STANDARD

(POPULATIONS WITH NO. OF 'SUCCESSES' >= M OUT OF n TRIALS ARE, SELECJED)

P* = .90 1 P* = .95 1 STANDARD 1 1 1 PROPORTIONS! 1

1

1

1 TTTTi nKT" Tl UFPEFi LO ivLR 1 n n Mrl n M 1 ri 1 n n ri n n 1

. 90

1

. 80 1

11 oy o

t

1 07 t

1 too \

1 o IW 1

1 oy 1

1

1 o74 1

. 70 1 S9 731 178 150 1 267 2281 1 10 90 1 220 185 1 330 282 1

. 60 1 46 36 1 92 75 1 138 1 15 1 55 43 1 1 10 901 165 137 i

. 50 1 yl A, 1 Oo I QO JL^ 1 ovi 1 yo

.40 1 1 CI1 o io 1 oo ^ i 1

1 O 111 Pf 1 A O O O 1 0.2 1 ^ o Kin 1 OW 1 .30 1 1 o 1^ 1 QAoU 1

1 IO 1 €k 1 oo 1 40 o4 1

. 20 1 1 £)k 1 \} t) 1 1 A. 1 1** 1 1 -1 1 1 ( t OO 1

1

lo 1 OO

. 10 1 7 41 14 9 1 21 15 1 7 41 14 9 1 21 15 1 .80 .70 1

1 OOO 1 "70 11 OOW i 1 WWW1 7ftQ 1 « 1 OoO1 t!Wo 1 (K> <

1 Ooo C?flQ 1

1 11 AAA \J\jy3

1 i OO 1 .60 1 1 O 1 OO 1

1 1 1 I < o\ OT^(11t 1 1

1 1 1 Wt) 1 O 1 O 1 oo^ 1

. 50 1 Do 1 1 1 D 1 < * 1 OA 1 /dl,fl 1 ^cf 1 1

1 AO t O 1 A loo 1

. 40 1 35 221 70 47 1 105 74 1 40 25 1 80 54 1 120 84 1 .30 1 22 13 1 44 281 66 44 1 26 15 1 52 331 78 521

.20 1 1 O

'7 1 1 O 1 QQ OPT 1 1I 7( Q 1 OA 1 O 1 OO 1

. IM 1 O 1 I 1 1 I I 1 1 O 1 1 1 I 1 O 1 oo 1 O 1 oo OA 1

.70 .60 1 543

1 354 1 7711 5081 10001 664 1 6551 4271 8271 5431 1000

1 661 1 .50 1 OO 1

1 1 T7 1 ^(11 1 < U 1 UO i O'tW O 1 5^ 1 1 O 1 O Iv o.iy 1

. 4v> 1 64 36 1 128 77 1 192 1 19 1 77 43 1 154 92 1 231 143 1 ,30 1 35 181 70 39 1 105 62 1 41 21 ! 82 46 1 123 72 1 .20 I 22 10 1 44 23 1 66 37 1 26 12 1 52 271 78 43 1

. 10 1 1 O O 1 11 o D 1 Ov OQ 1 .60 .50 1 586

1 3231 7931 4421 1000

1 5621 7151 394 1 8571 475 1 10001 5581

.40 1 145 73 1 290 154 1 435 236 1 177 89 1 354 187 1 531 2881

.30 i 64 29 1 128 631 192 99 1 77 35 1 154 76 1 231 1131

.20 1 35 141 70 32 1 105 5 1 1 40 16 1 80 36 1 120 57 1

. 10 1 18 6 1 36 141 54 24 1 21 71 42 17 1 63 271

.50 .40 1 586

1 2641 7931 3621 10001 462 1 7151 3221 8571 389 1 10001 4581

.30 1 140 56 1 280 120 1 42^ 185 1 170 681 340 145 1 510 225 1 .20 1 58 20 1 1 16 45 1 174 71 1 72 25 1 144 56 1 216 881

. 10 1 28 81 56 19 i 84 31 1 32 9 1 64 21 1 96 35 1 .40 .30 i 543

1 1901 7711 275 1 1000

1 3621 655

1 229 1 8271 2931 10001 359 1 .20 1 121 36 1 242 781 363 1231 149 44 1 298 971 447 151 1

. 10 1 46 11 1 92 26 1 138 41 I 55 131 1 10 30 1 165 49 1 .30 .20 1 442

1 1101 721

1 1871 1000

1 265 i 5351 1331 7671 197 1 10001 2621

. 10 1 89 171 178 39 1 267 621 1 10 211 220 481 330 771

.20 . 10 1 299

1 44 1 5981 96 1 897

1 151 11- 362

1 531 6811 109 1 10001 1671

1

78
Table 39

VALUES OF n,M FOR SELECTING, WITH P(CS) >= P*, A SUBSET OF k = 15

BINOMIAL POPULATIONS WHICH CONTAINS A1.L POPULATIONS BETTER THAN AN

UPPER STANDARD AND NO POPULATIONS WORSE THAN A LOWER STANDARD

(POPULATIONS WITH NO. OF 'SUCCESSES' >= M OUT OF n TRIALS ARE SELECTED)

P* = .90 1 P* = .95 1 STANDARD 1

1

1 1

1 PROPORTIONS! 1

1 11 UPPER LOWER

1

n ri n n n nM 1\ n n n Pi n Fl 1

. 90 . 80 1 300

1 2571 600

1 521 1 900

1 7871 368

1 3151 684 594 1 1000

1 8741

. 70 i 94 771 188 1581 282 241 1 111 91 1 222 187 1 333 284!

. 60 1

1 o7 1 94 77 1 t A t 14

1

1 lo 1 06 >f A 1 44 1 1 1<2 9 1 1 loo 14w 1 .50 1 28 2 1 1 56 44 I 84 68 1 35 26 1 70 55 1 0 e? 1 o5 1 .40 1 18 13 1 36 27 1 54 43 1 24 17 1 48 37 1 72 57 1

. 30 1 13 9 1 26 19 1 39 30 1 16 1 1 1 32 24 1 48 37 1

. 20 1 10 6 1 20 141 30 221 1

1

71 22 15 1 33 24 1

. 10 1 7 41 14 91 21 15 1 7 41 14 9 1

1 21 15 1 .80 .70 1 454

1 342 1 7271 555 1 1000

1 768 1 547

1 412 1 773 588 1 1000

1 765 1 .60 1 125 89 1 250 184 1 375 280 1 152 108

1

304 224 1 456 34 1 1

. 50 1 60 40 1 120 85 1 180 1301 72 481 144 102 1 216 156 1 .40 1 35 221 70 471 105 731 40 25 1 80 54 1 120 841

.30 1 22 13 1 44 28

1

66 44 1 26 15 1 52 33 1 78 52 1 .20 1 15 81 30 181 45 29 1 17 9 1 34 20 1 5 1 33 1

. 10 1 10 5 1 20 1 1 1 30 181 1

1

5 1 22 121

1 33 2011 .70 .60 1 549

1 3581 7741 5 10 1 1000

1 664 1 6691 436 1 834 548

!

1000 660 1

. 50 1 145 881 290 184 1 435 281 1 175 106 1 350 221 1 525 339 1 .40 1 64 36 1 128 77 1 192 119 1 80 45 1 160 96 1 240 1481

.30 1 35 181 70 39 1 105 621 43 221 86 481 129 76 1 .20 1 22 10 1 44 23 1 66 37 1 26 12 1 52 27 1 78 A 0 1 43 1

. 10 1 13 5 1 26 121 39 201 16 6 1

1 32 15 1 48 25 1

1 .60 .50 1 597

1 329 1 798

1 445 i 1000

i 562 i 726 4001 863 479 1 1000 5581

.40 1 149 75 1 298 158i 447 2431 179 901 358 1901 537 291 1 .30 1 64 29 1 128 631 192 981 80 36 1 160 79 1 240 1231

.20 1 35 141 70 321 105 51 1 40 16 1 80 36 1 120 57!

. 10 1 18 6 1 36 14 1 54 231 24 81 48 201

1 72 321

1 .50 .40 1 597

1 269 1 798

1 3641 1000

1 461 1 7261 3271 863 3921 1000 4571

.30 1 145 581 290 1241 435 1921 175 701 350 1501 525 2321

.20 1 60 21 1 120 47 1 180 741 72 251 144 56 1 216 881

. 10 1 28 81 56 19 1 84 311

1 35 1011 70 241

1 105 39 11 .40 .30 1 549

1 1921 774

1 276 1 1000 3621 669 2341 834 296 1 1000 3581

.20 1 125 371 250 81 1 375 1271 152 45 1 304 99 1 456 1541

. 10 1 47 111 94 26 1 141 42 1

1 56 13 1

1 112 311

1 168 50 11 .30 .2© 1 454

1 1131 727

1 1881 1000 265 1 547 136 1 773 1981 1000 261 1

. 10 1 94 181 188 41 1 282 66 1

1

1 1

1

211

1 222 49 1

1 333 7811 .20 . 10 1 300

1 441 600

1 96 1 900 151 1 368 541 684 109 1 1000 166 1

79
Table ho

VALUES OF n,M FOR SELECTING, WITH P(CS) >= A SUBSET OF k = 16

BIWOMIAL POPULATIONS ¥HICH CONTAINS ALL POPULATIONS BETTER THi\N AW

UPPER STANDARD AND NO POPULATIONS WORSE THAN A LOWER STANDARD

(POPULATIONS WITH NO. OF 'SUCGE-SSES' >= M OUT OF n TRIALS ARE, SELECTED)

P* = .90 1 P* = .95 1 STANDARD 1 1 1 PROPORTIONS! 1

1

1

1 Ur r h.tv n n M n M 1 Fx 1 n M n rl n n 1

. Q€lk 1

1 O I**

11 Oo^ (^Qd. 1

1 OO 1

1

1 OO^f

1

1 O lO 1

1 O>;qo tO 1

1 \.\)\)yt

1 Oi o I

. (V \ 94 771 188 1581 282 241 1 1 1 1 911 222 1871 333 2841

. dW I 47 371 94 771 141 1171 56 44 1 1 12 91 1 168 140 1

. OW ! on O 1 1 OD AiA 1 *r*r i 0*r ARVO 1

1 OO Oft 11 ( y} OO 1 1 wo RKOOi 1

1 .40 1 11 Ro 1 o i 1f> OO 07 1 o*r id.*! 1 4.R O ( 1 TO O ( t .30 1 1 o O 1 1 Q 1 QQO V '\€k 1 <y\f \ 1 O 1 OO OT 1 o^ A.O 1

. J0 1 1 (h 1 yJ f\O 1I 1 <1 1 OO 1 1 1 1 1 i 1 OO 1lO^ 11 OO OA 1

. 10 1 7 41 14 9 1 21 15 1

1 7 41 14 9 1 21 14 1 .80 .70 1

1 O^Q O^O 1

1 r o 1

1 CI SIT 1 TAR 1 < DO 1

1 /L 1 R 1 T! lo 1 tit1 OVECO 1 1 1 11 too 1 .60 1 11 OQ<^o Q 1 1 Oct A loo 1 oo'i 1 P^O 1mo€bn 1

1 OW'l' *oo

. 50 1 D 1

A. t 1 I 1 1 1 oo 1 OO 1 loo 1 OO 1 loo 1 ^^o 4o 1 \ A.A 1 AO 1 O 1 ^lo loo 1

. 40 1 35 221 70 47 1 105 73 1 43 271 86 581 129 90 1 .30 1 22 131 44 281 66 441 26 15 1 52 331 78 52 1 .20 1 lo D 1 oW 1 Q 1 1 O 1 A Ct OCk 1 1J 7

Q 1 y 1 O A O"* OA 1 O 1 O O 1 0<£ 1

. 10 1 1 A10 O 1 J0 1 1 1 1 1 1 O0 lo 1

1

1 1 K 1 1 O 1 1<2 1 o o OO OA 1 <20 1 .70 .60 1 563

1 3671 7811 515 1 1000 664 1 675

1 4401 837

1 549 1 1000

1 660 1 ,50 1 145 88 1 290 183 1 435 280 1 a WET 175 106 1 350 22 1 1 525 338 1 .40 1 66 371 132 79 1 198 1221 82 46 1 164 981 246 1521

.30 1 35 181 70 39 1 105 62 1 43 221 86 481 129 76 1 .20 1 22 101 44 231 66 371 26 121 52 271 78 431

. 10 1 13 5 1 26 12 1 39 20 1 18 7 1 36 17 i 54 28 1

1 .60 .50 1 613

1 3381 3061 449 1 1000

1 56 11 7371 406 1 8681 481 1 1000 5571

.40 1 151 76 1 302 160 1 453 246 1 183 921 366 194 1 549 2981

.30 1 66 301 132 65 1 198 1021 82 37 1 164 81 1 246 126 1 .20 1 35 141 70 321 105 50 1 43 171 86 39 1 129 62 1

. 10 1 18 6 1 36 14 1 54 231 24 81 48 19 1 72 32 1

1 .50 .40 1 613

1 276 1 806

1 3681 10001 46 11 7371 3321 8681 3941 1000 457 1 .30 1 145 581 290 1241 435 1921 175 701 350 1501 525 231 1 .20 1 61 21 1 122 471 183 75 1 72 25 1 144 56 1 216 831

. 10 1 28 81 56 19 1 84 311 35 101 70 24 1 105 39 1

1 .40 .30 1 563

1 1971 7811 279 1 1000

1 3621 6751 236 1 837

1 296 1 1000 3581

.20 1 128 381 256 831 384 130 1 152 45 1 304 99 1 456 1541

. 10 1 47 1 1 1 94 26 1 141 42 1 56 131 1 12 31 1 168 501

1 .30 .20 1 462

1 1 15 1 7311 189 1 1000

1 264 1 555

1 1381 777i 199 1 1000 261 1

. 10 1 94 181 188 41 1 282 66 1 1 1

1

21 1

1 222 481

1 333 7811 .20 . 10 1 307

1 45 1 6141 99 1 9211 155 11- 369 541 684 109 1 1000 166 1

1

80
Table kl

VALUES OF ii,M FOR SELECTING, WITH P(CS) >= P*, A SUBSET OF k = 17

BINOMIAL POPULATIONS WHICH CONTAINS AIL POPULATIONS BETTER THAN AN

UPPER STANDARD AND NO POPULATIONS WORSE THAN A LOWER STANDARD

(POPULATIONS WITH NO. OF 'SUCCESSES' >= M OUT OF n TRIALS ARE SELECTED)

P* = .90 1 Pif: = .95 1 STANDARD 1 1 1 PROPORTIONS 1 1 1 UPPER LOWER

1

u M M n M 1 EL M M 1

* 7\y 1 314 269 1 628

1 546 1 9421 824 1 3761 3221 688 597 1 10001 873 1 7ft 1 94 771 188 1581 282 241 1 1 15 941 230 194 1 345 294 1 fttt 1 . 1 47 37 1 94 77 1 141 1 17 1 59 46 1 118 96 1 177 148 1 .50 1 3

1

23 1 62 49 1 93 76 1 26 1 70 55 1 105 85 1 .40 1 2

1

15 1 42 32 1 63 50 1 24 17 1 48 37 1 72 57 1 15 10 1 30 22 1 45 35 1 13 12 1 36 27 1 54 42 1

. 20 1 1

1

7 1 22 16 1 33 25 1 1

1

7 1 22 15 1 33 24 1

1 1 7 4 1 14 9 1 21 13 1 7 4 1 14 9 1 21 14 1 .80 .70 1 470

1 354 1 7351 560 1 10001 763 1 5631 424 1 7811 593 1 1000

1 765 1

. 60 1 128 9 1 1 256 188 1 384 287 1 155 110 1 310 228 1 465 348 1

. 50 1 61 41 1 122 86 1 183 132 1 75 50 1 150 106 1 225 163 1 .40 1 35 22 1 70 47 1 105 73 1 43 27 1 86 58 1 129 90 1 .30 1 22 13 1 44 28 1 66 44 1 28 16 1 56 36 1 84 57 1

. 20 1 15 8 1 30 181 45 29 1 17 9 1 34 20 1 5 1 32 1 10 1 1

1

5 1 22 13 1 33 20 1 1

1

5 1 22 121

1 33 20 1

1 .70 .60 1 569

1 37 1 1 7841 5 16 1 1000

1 663 1 6891 449 1 844 554 1 1000 660 1

. 50 1 150 91 1 300 190 1 450 290 1 180 109 1 360 2281 540 3481

.40 1 66 37 1 132 79 1 193 122 1 82 46 1 164 98 1 246 152 1 .30 1 37 19 1 74 42 1 1 1

1

65 1 43 22 1 86 481 129 76 1 .20 1 22 10 1 44 23 1 66 37 1 28 13 1 56 29 1 84 471

. 10 1 15 6 1 30 14 1 45 24 1

1 18 7 11 36 171

1 54 281

1 .60 .50 1 624

1 344 1 812

1 452 1 1000 56 1 1 746 4111 873 484 1 1000 5571

.40 1 153 77 1 306 162 1 459 249 1 185 93 1 370 196 1 555 301 1 .30 1 66 30 1 132 65 1 198 101 1 82 37 1 164 81 1 246 126 1 .20 1 35 14 1 70 32 1 105 50 1 43 171 86 39 1 129 62 1

. 10 1 21 7 1 42 17 1 63 281

1 24 811 48 19 11 72 32 1

1 .50 .40 1 624

1 281 1 812

1 370 1 1000 460 1 746 336 1 873 396 1 1000 457 1 .30 1 150 60 1 300 1281 450 1981 180 721 360 154 1 540 2381

.20 1 61 21 1 122 47 1 183 75 1 75 26 1 150 581 225 921

. 10 1 31 9 1 62 21 1 93 34 1 35 101

1 70 24 1

1 105 39 1

1 .40 .30 1 569

1 199 1 784

1 279 1 1000

1 361 1 689 241 1 844 299 1 1000 3581

.20 1 128 381 256 83 1 384 130 1 155 46 1 310 101 1 465 157 1

. 10 1 47 1 1 1 94 26 1 141 42 1 59 14 1

1

1 18 33 1

1 177 53 1

1 .30 .20 1 470

1 1 17 1 735

1 1901 1000

1 264 1 563 1401 781 200 1 1000 261 1

. 10 1 94 181 188 41 1 282 66 1 1 15 22 1

1 230 50 1

1 345 81 11 .20 . 10 1 314

1 46 1 628

1 101 1 942

1 1581 376 55 1 688 109 1 1000 166 1

81
Table k2

VALUES OF ii,M FOR SELECTING, VITH P(CS) >= ?*, A SUBSET OF k = 18

BINOMIAL POPULATIONS WHICH CONTAINS ALL POPULATIONS BETTER THAN AN

UPPER STANDARD AND NO POPULATIONS WORSE THAN A LOWER STANDARD

(POPULATIONS VITH NO. OF 'SUCCESSES' >= M OUT OF n TRIALS ARE SELECTED)

STANDARD 1 PROPORTIONS!

UPPER LOWER

1

n M

~

n M n

1

1

1

1 M 1 n M

—

n

.95

M n

1

1

1

1 M 1

. 90 . 80 1 314

1 269 1 628

1 546 1 9421 8241 3831 3281 691

1 599 1 100©1 8731

. 70 1 O 1 1 1 Q5l 1 y o 1 AT 1 .S V < ^o*« 1 1 1

A

1 lO 1951 348 00*7 1 Jif * \ .60 1 47 371 94 77 1 141 1171 59 46 1 118 96 1 177 1471

.50 1 31 231 62 49 1 93 76 1 36 271 72 571 108 881

. 40 1 21 15 1 42 321 63 5©l 24 171 48 371 72 571

. 30 1 15 101 3© 221 45 35 1 18 121 36 271 54 421

. 20 1 1

1

71 22 16 1 33 25 1 1

1

71 22 15 1 33 24 1

. 10 1 i Q 1 y 1 4£, 1 lo 1

1

(

A. 1 9 I 21 1 A 1 14 1

. 80 . 7© 1 475

1 3581 7371 5621 10©0 7681 571

1 4301 7851 596 1 100©

1 764 1

. 60 1 132 941 264 194 1 396 296 1 159 1131 318 234 1 477 3571

.50 1 63 421 126 89 1 189 1371 75 501 15© 1©6 1 225 163 1 .4© 1 oO 1 (y) ^ i \ i O 1 1 OD 581 129

. 3© 1 24 14 1 48 31 1 72 49 1 28 16 1 56 36 1 84 56 1 .2© 1 15 81 3© 181 45 29 1 17 9 1 34 2© 1 51 321

. 1© 1 1

1

5 1 22 121 33 20 1 1 1 5 1 22 121 33 20 1

. 70 . 6© 1 583 38© 1 791 521 1 1000 6631 695

j 4531 847

1 556 1 1000 659 1 .5© 1 15© 91 1 3©0 19© 1 45© 2901 180 109 1 360 2281 540 3481

.4© 1 OO 1 1 Q A lOD ol0. 1 11 OA/4. 1 OA 1 A.^ 1 981 246

.3© 1 37 19 1 74 42 1 1 1

1

65 1 45 231 9© 51 1 135 79 1 .2© 1 24 11 1 48 25 1 72 4© 1 28 131 56 29 1 84 47 1

. 1© 1 15 6 1 30 14 1 45 23 1

1 18 71 36 17 1 54 281

.60 .5© 1 633

1 349 1 8161 4541 1©00 56 1 1 7641 421 1 8821 489 1 10©©1 5571

,4© 1 157 79 1 314 166 1 471 256 1 187 94 1 374 1981 561 3041

.3© i 68 31 1 136 67 1 204 105 1 82 37 1 164 81 1 246 126 1 .2© 1 35 14 1 7© 321 105 5©1 43 171 86 39 1 129 621

. 1© 1 21 71 42 171 63 281 24 81 48 19 1

1 72 321

,50 .4© 1 633

1 285 1 816

1 3721 1©©©1 460 1 7641 3441 8821 4©© 1 1©©©1 456 1 .3© 1 150 6©l 3©© 1281 45© 1981 180 721 36© 1541 54© 2381

.2© 1 63 22 1 126 49 1 189 771 75 26 1 150 581 225 921

. I© 1 31 9 1 62 21 1 93 34 1 36 101 72 24 1 108 4© 1 .40 .3© 1 583

1 2©4I 7911 2821 1©0O1 361 1 6951 2431 847 30© 1 1©©01 3571

.2© 1 132 39 1 264 86 1 396 1341 159 471 318 103 1 477 161 1

. 1© 1 47 1 1 1 94 26 1 141 421 59 141 1 18 331 177 531

.30 .2© 1 475

1 1181 7371 190 1 10©©1 264 1 5711 1421 785

1 2001 10©©1 2601

. 1© 1 99 19 1 198 44 1 297 7© 1 1 16 22 1 232 51 1 348 81 1 .20 . 10 1 314

1 46 1 628

1 101 1 9421 1581

1- 383

1 56 1 6911 1101 1©©©1 166 1

1

82
Table h3

VALUES OF n,M FOR SELECTING, WITH P(CS) >= P*, A SUBSET OF k = 19

BINOMIAL, POPULATIONS IVHICH CONTAINS ALL POPULATIONS BE^ITER THAN AN

UPPER STANDARD AND NO POPULATIONS WORSE THAN A LOWER STANDARD

(POPULATIONS WITH NO. OF 'SUCCESSES' >= M OUT OF n TRIALS ARE SELECTED)

1 P* = .90 I P* = .95 i STANDARD II I PROPORTIONS! I I LUWLn. 1 n n n

TVf n *.n n 1 n n rl n nM 1

1

. 90

1

. oU 1 O O 1

1 ^<o 11 e Q 1 Q^o1 Q>4.0 1 O^til 1 ooo1 AO 1 DV 1

1 oyy 1 1 CkiAA1 o< O 1

1 . 7W 1 yy o 1 1 1 VO ID r 1 ^297 Oct o 1 I lo ooo 1 O C^ 1 ivo 1 O^O OOT 1 € 1

. tyio 1 O 1 A £i 1 "iW 1 oo 1 loo \<S.O 1 4 i i 1 OA 1 lo^l 1 QA low 1

. 50 1 3

1

OO 1 l^o 1 6^ 49 1 76 1 o6 <±7 1 T^O 72 ct T 1 57 1 1 Wo Q Q 1 OO 1 .40 1 21 15 1 42 321 63 501 24 171 48 371 72 571

.3© 1 15 101 30 22 1 45 35 1 18 121 36 27 1 54 421

. 20 1 1 1 7 1 22 16 1 33 25 1 13 8 1 26 1 O 1 lU 1 cfy oo 1 2y 1

. 10 1 7 4 1 14 9 1 21 lo 1 O 1 lo 1 O 1 \z 1

1

0*7 27 ly 1 .80

! .70 1 483

1 364 1 7411 565 1 1000 7671 579 436 1 789 599 1 1000

1 7641

.60 1 135 96 1 270 199 1 405 303 1 159 1131 318 234 1 477 357!

. 50 1 64 43 1 128 90 1 192 139 1 75 50 1 150 106 1 ooIT 223 16o 1

. 40 1 35 22 1 70 47 1 105 73 1 43 27 1 o6 Oo 1 i2y QA 1 yw 1 .30 1 24 14 1 48 31 1 72 49 1 28 16 1 56 36 1 84 56 1 .20 1 15 81 30 181 45 29 1 17 9 1 34 20 1 51 321

. 10 1 1

1

5 1 22 12 1 33 20 1

1 13 6 11 o^26 lo 1

1

or* cjy OA 1 24 1

1 .70

1 .60 1 589

1 384 1 794

1 523 1 1000 663 1 707 461 1 853 559 1 1000 6591

A 1 • OU 1 O 1

V

1 1 ^> 1 o 1 V .SOT* 1 O^O 1 .40 1 71 40 1 142 85 1 213 132 1 84 47 1 168 101 1 252 156 1 .30 1 37 19 1 74 411 111 65 1 45 231 90 51 1 135 79 1 .20 1 24 1 1 i 48 25 1 72 40 1 28 131 56 29 1 84 471

. 10 1 15 6 1 30 14 1 45 231 18 71

1 36 17!

1 54 281

1 .60

1 .50 1 644

1 355 1 822

1 4571 10001 560 1 766 4221 883 489 1 1000 537 1 .40 1 159 80 1 318 1681 477 259 ! 191 96 1 382 202 1 573 311!

.30 1 71 32 1 142 701 213 109 1 84 381 168 831 252 129 1 .20 i 35 14 1 70 311 105 501 43 171 86 39 1 129 62!

. 10 1 21 7 1 42 17 1 63 281 24 81

1 48 19 1

1 72 31 1

1 .50

1 .40 1 644

1 2901 822

1 375 1 1000

1 460 I 766 345 1 883 400 1 1000 456 1 .30 1 155 62 1 310 1331 465 205 1 185 74 1 370 1581 555 245 1 .20 1 64 22 1 128 501 192 781 75 26 1 150 581 225 921

,10 1 31 9 1 62 21 1 93 34 1 36 101

1 72 24 1

1 108 401

1 .40

1 .30 1 589

1 206 1 794

1 283 1 1000

1 361 1 707 2471 853 3021 1000 357 1 .20 1 135 40 I 270 831 405 1371 159 471 318 1031 477 161 1

. 10 1 51 12 1 102 281 153 46 1 60 141

1 120 33 1

1 180 54 11 .30

1 .20 1 483

1 120 1 741

1 191 1 1000

1 264 1 579 144 1 789 201 1 1000 260 1

. 10 1 99 19 1 198 431 297 69 1 1 16 22!

1 232 511

1 348 81 1

1 .20

1

. 10 1 321

1 47 1 642

1 1031 963

1 1621 383 56 1 691 1101 1000 166 1

83
1

Table hk

VALUES OF n,Il FOR SELECTIWG, WITH P(GS) >= F*, A SUBSET OF k = 20

BINOMIAL. POPULATIONS WHICH CONTAINS ALL POPULATIONS BETTER THAN AN

UPPER STANDARD AJND NO POPULATIONS WORSE THAN A LOWER STANDARD

(POPULATIONS WITH NO. OF 'SUCCESSES' >= M OUT OF n TRIALS ARE SELECTED)

P>^ = J7 T* — 1

1 P:*:^ =— . 0^ 1 STANDARD 1 1

1 PROPORTIONS 1 1 1 n M n M n n 1 n n n N n n 1

.90 .80 1 328

1 281 1 656

1 570 1 984

1 86 1 1 3901 334

1

695

1 602 11 873 1 .70 1 100 82 1 200 168 1 300 256 1 1 16 95

1

232 195 1 348 297 1 5 1 40 1 102 83 1 153 128

1

60 47

1

120 98 1 180 1 50 1

. 50 1 32 24 1 64 51 ! 96 781 36 27 ! 72 571 108 881

Ad 1 21 15 1 42 32 1 63 50 1 24 171 48 37 1 72 57 1 15 10 1 30 22 1 45 35 1 13 12 1 36 27 1 54 42 1 .20 1 1

1

7 1 22 16 1 33 25 1 13 8

1

26 18 1 39 29 1

. 10 1 7 4 1 14 9 1

1 2

1

15 1

1 9 5 1 18 12 1 27 19 1

> 0\f • * V 1 491

1 370 1 745 5681 1000 767 1 5871 442 1 7931 6021 1000

1 764 1 135 96 1 270 199 1 405 303 1 162 1 15 1 324 2381 486 363 1 .50 1 64 43 1 128 90 1 192 139 1 78 52 1 1 56 1 10 1 234 169 1

. 40 1 88 24 1 76 5 1 1 114 80 1 43 27 1 86 58 1 129 90 1 'id 1 24 14 1 43 3 1 1 72 48 1 28 16 1 56 36 1 84 56 1 Oa,^ 1 . 1 T5 81 30 181 45 29 1 19 10 1 38 23 1 57 371

. 10 1 1 1 5 1

1 22 12 i1 33 20 11 13 6 1

1 26 15 1

1 39 24 11 '7A . 1

1 000 1 » " <

1 1 f\ft'\ 1

1 i 1 .J V ^ 1

1 1

1 .50 1 155 94 1 310 196 1 465 300 i 185 1 12 1 370 234 1 55" 3581

.40 1 71 40 1 142 85 1 213 132 1 84 47 1 168 101 1 252 156 1 .30 1 ov 1 t 0 1 TT" 1 1 1 < ^00^ 1

1 OQ. 1 loo 70 1 ( 7 1 24 1 1

1

48 25 1 72 40 1 28 13 1 56 29 1 84 47 1

. 10 1 15 6 1 30 14 1

1 45 231 18 7 1 36 171 54 2811 .60 .50 1

1 000 1 *xOO i

1 ^j^y? 1 ( i i1 0001 .40 1 161 81 1 322 171 1 483 262 1 193 97 1 386 204 1 579 314 1 .30 1 71 32 1 142 701 213 109 1 84 381 168 83 1 252 129 1 .20 1 38 15 1 76 35 1 1 14 55 1 43 171 86 39 1 129 621

. 10 1 21 7 1 42 17 1 63 281 24 81 48 19 1 72 31 1

1 .50 .40 1 646

1 291 1 8231 375 1 1000

1 460 1 7771 3501 8881 4021 1000 456 1 .30 1 155 62 1 310 133 1 465 205 1 185 74 1 370 1581 555 245 1 .20 1 64 22 1 128 50 1 192 781 78 27 1 156 61 i 234 96 1

. 10 1 32 9 1 64 22 1 96 36 1 36 10 1 72 241

1 108 40 I

1 .40 .30 1 595

1 2081 797! 284 1 1000

1 361 1 7151 2501 857 303 1 1000 357 1 .20 1 135 40 1 270 881 405 137 1 162 481 324 105 1 486 164 1

. 10 1 51 12 1 102 281 153 46 1 60 14 1 120 33 1

1 180 5411 .30 .20 1 491 122 ! 745

1 192 1 1000

1 263 1 5871 146 1 793 2021 1000 260 1

. 10 1 100 19 1 200 44 i 300 70 1 116 221 232 51 1 348 81 1 .20 . 10 1 328

1 48 i 656

1 106 1 9841 165 11- 390

1 571 695

1 1 10 1 1000

1 165 1

1

U.S. GOVERNMENT PRINTING OFICE : 1980 0-31 1-046 (62)

84
NBS-114A (REV. a-7e)

U.S. DEPT. OF COMM. BIBLIOGRAPHIC DATA

SHEET

1. PUBLICATION OR REPORT NO. SP 500-58

Gov't Accession No^

m^MMf/d^r,

4. TITLE AND SUBTITLE r^rr„^,,-i-^^ Co-;^^,.^ o t«^i,.«^i . Computer Science & Technology:

Application of Measurement Criteria in the Selection

of Interactive Computer Services

5. Publication Date Aoril 1980

<. Performing Organization Code

7. AUTHOR(S)

Paul D. Amer

8. Performing Organ. Report No.

9. PERFORMING ORGANIZATION NAME AND ADDRESS

NATIONAL BUREAU OF STANDARDS

DEPARTMENT OF COMMERCE

WASHINGTON, DC 20234

10. ProjectAasl</Work Unit No.

11. Contract/Grant No.

12. SPONSORING ORGANIZATION NAME AND COMPLETE ADDRESS rsrreet, city, state. ZIP)

same as Item 9

13. Type of Report & Period Covered

Final

14. %»nsor}ng Agetwy Code

15. SUPPLEMENTAL^ NOTES

Consress Card Catalog Number 80-600035

Document describes a computer program; SF-185, FIPS Software Summary, is attached.

16. ABSTRACT (A 200-word or leaa {actual aummary of moat significant information. If document includes a significant bibliography or

literature survey, mention it here.)

iriii |juui iLouiuri duurtrbbcb Lilt: uacd analysis componeni, OT me compuxer

service selection process. A computer service selection model, introduced in

NBS Special Publication 500-44, is presented and three binary type selection

procedures applicable in the measurement phases of that model are given. A binary type procedure determines which of a set of competing computer services

perform above and which perform below a specified performance level. As a result of employing these procedures, those services that perform below the

performance level can be eliminated or penalized depending on whether the

performance specification is mandatory or desirable in nature. The procedures

explicitly specify prior to measurement an appropriate selection rule and/or

the number of test measurements required in a comparison effort to attain a given level of statistical confidence in the results. Experimental data from

a previous case study are reanalyzed to illustrate application of the selection

procedures.

17. KEY WORDS Cs(x to twelve entriea; alphabetical order; capitalize only the firat letter of the first key word unlesa a proper name;

separated by semicolons) Binary selection; computer comparison; computer measurement; computer service;

data analysis; performance evaluation; ranking and selection; selection methodology.

18. AVAILABILITY Unlimited

1 1 For Official Distribution. Do Not Release to NTIS

1—1 Order From Sup. of Doc, U.S. Government Printing Office, Washington, DC * 20402

1

1 Order From National Technical Information Service (NTIS), Springfield, VA. 22161

19. SECURITY CLASS

(THIS REPORT)

UNCLASSIFIED

21. NO. OF

PRINTED PAGES

90

20. SECURITY CLASS

(THIS PAGE)

UNCLASSIFIED

22. Price

$4.00

USCOMM-DC

ANNOUNCEMENT OF NEW PUBLICATIONS ON

COMPUTER SCIENCE & TECHNOLOGY

Superintendent of Documents,

Government Printing Office,

Washington, D. C. 20402

Dear Sir:

Please add my name to the announcement list of new publications to be issued in the series: National Bureau of Standards Special Publication 500-.

Name

Company

Address

City State Zip Code

(Notification key N-503)

Subscribe now—

The new

National Bureau

of Standards

Journal

The expanded Journal of Research of the National Bureau of Standards reports NBS research and

development in those disciplines of the physical

and engineering sciences In which the Bureau is active. These include physics, chemistry, engineering, mathematics, and computer sciences. Papers

cover a broad range of subjects, m\U major

emphasis on measurement methodology, and the basic technology underlying standardization. Also

included from time to time are survey articles on

topics closely related to the Bureau's technical and scientific programs. As a special service to subscribers each issue contains complete citations

to all recent NBS publications in NBS and non- NBS media. Issued six times a year. Annual subscriptions: domestic $17.00; foreign $21.25. Single copy, $3.00 domestic; $3.75 foreign.

• Note: The Journal was formerly published in two sections: Section A "Physics and Chemistry" and Section B "N^athematical Sciences."

NBS Board of Editors Churchill Eisenhart, >. Executive Editor (Mathematics)

John W. Cooper (Physics)

Donald D. Wagman (Chemistry)

Andrew J. Powell (Engineering)

Joseph O. Harrison (Computer Science)

Howard J. 1^. Hanley (Boulder Labs.)

Jbscription Order Form

per my subscription to NBS Journal of Research

1^17.00. Add $4.25 for foreign mailing. No additional postage is Quired for mailing within the United States or its possessions.

liRA—File Code 2N)

id Subscription to:

Credit Card Orders Only

Total ctiarges $ Credit Card No.

Name-First, Last r

Company Name or Additional Address Line

Street Address

City State Zip Code

(or) COUNTRY

EASE PRINT

in the boxes below.

Master Ctiarge Interbank No.

Expiration Date

Montti/Year

Remittance Enclosed. Domestic: Check or money order. Foreign:

International money order,

draft on an American or Canadian

Bank, or by UNESCO coupons, Made payable to the Superintendent of Documents. Charge to my Deposit Account No.

M IL ORDER FORM TO:

Su^^. intendent of Documents

Government Printing Office Washington, D.C. 20402
IIII
NBS TECHNICAL PUBLICATIONS

PERIODICALS

URNAL OF RESEARCH—The Journal of Research of the

itional Bureau of Standards reports NBS research and developnt in those disciplines of the physical and engineering sciences in ch the Bureau is active. These include physics, chemistry,

jineering, mathematics, and computer sciences. Papers cover a

)ad range of subjects, with major emphasis on measurement

thodology and the basic technology underlying standardization,

io included from time to time are survey articles on topics

sely related to the Bureau's technical and scientific programs,

a special service to subscribers each issue contains complete

[itations to all recent Bureau publications in both NBS and non- BS media. Issued six times a year. Annual subscription: domestic

17; foreign $21.25. Single copy, $3 domestic; $3.75 foreign. OTE: The Journal was formerly published in two sections: Secn A "Physics and Chemistry" and Section B "Mathematical

ciences." iiVIENSlONS/NBS—This monthly magazine is published to in- orm scientists, engineers, business and industry leaders, teachers,

udents, and consumers of the latest advances in science and

ichnology, with primary emphasis on work at NBS. The magazine

ighlights and reviews such issues as energy research, fire protecjion, building technology, metric conversion, pollution abatement,

lealth and safety, and consumer product performance. In addiion, it reports the results of Bureau programs in measurement

landards and techniques, properties of matter and materials, ngineering standards and services, instrumentation, and

utomatic data processing. Annual subscription: domestic $11;

oreign $13.75. NONPERIODICALS

l^lonographs—Major contributions to the technical literature on

arious subjects related to the Bureau's scientific and technical aciMties. jlandbooks—Recommended codes of engineering and industrial Mactice (including safety codes) developed in cooperation with in- lerested industries, professional organizations, and regulatory

[lodies.

^ppecial Publications—Include proceedings of conferences sponsored by NBS, NBS annual reports, and other special publications

jippropriate to this grouping such as wall charts, pocket cards, and

bibliographies.

I^pplied Mathematics Series—Mathematical tables, manuals, and

Itudies of special interest to physicists, engineers, chemists,

'idlogists, mathematicians, computer programmers, and others

:ngaged in scientific and technical work.

'National Standard Reference Data Series—Provides quantitative

.lata on the physical and chemical properties of materials, compiled from the world's literature and critically evaluated. Developed under a worldwide program coordinated by NBS under

he authority of the National Standard Data Act (Public Law

10-396).

NOTE: The principal publication outlet for the foregoing data is the Journal of Physical and Chemical Reference Data (JPCRD)

published quarterly for NBS by the American Chemical Society

(ACS) and the American Institute of Physics (AIP). Subscriptions,

reprints, and supplements available from ACS, 1 155 Sixteenth St., NW, Washington, DC 20056.

Building Science Series—Disseminates technical information developed at the Bureau on building materials, components,

systems, and whole structures. The series presents research results,

test methods, and performance criteria related to the structural and

environmental functions and the durability and safety characteristics of building elements and systems.

Technical Notes—Studies or reports which are complete in themselves but restrictive in their treatment of a subject. Analogous to monographs but not so comprehensive in scope or definitive in treatment of the subject area. Often serve as a vehicle for final reports of work performed at NBS under the sponsorship of other government agencies.

Voluntary Product Standards—Developed under procedures

published by the Department of Commerce in Part 10, Title 15, of

the Code of Federal Regulations. The standards establish nationally recognized requirements for products, and provide all concerned interests with a basis for common understanding of the

characteristics of the products. NBS administers this program as a supplement to the activities of the private sector standardizing

organizations. Consumer Information Series—Practical information, based on NBS research and experience, covering areas of interest to the con- sumer. Easily understandable language and illustrations provide

useful background knowledge for shopping in today's tech- nological marketplace. Order the above NBS publications from: Superintendent of Documents, Government Printing Office. Washington, DC 20402. Order the following NBS publications—FIPS and NBSlR's—from

the National Technical Information Services, Springfield. VA 22161

.

Federal Information Processing Standards Publications (FIPS

PUB)—Publications in this series collectively constitute the Federal Information Processing Standards Register. The Register

serves as the official source of information in the Federal Govern- ment regarding standards issued by NBS pursuant to the Federal Property and Administrative Services Act of 1949 as amended.

Public Law 89-306 (79 Stat. 1127), and as implemented by Executive Order 11717(38 FR 12315. dated May II, 1973) and Part 6 of Title 15 CFR (Code of Federal Regulations).

NBS Interagency Reports (NBSIR)—A special series of interim or

final reports on work performed by NBS for outside sponsors

(both government and non-government). In general, initial dis- tribution is handled by the sponsor; public distribution is by the National Technical Information Services, Springfield, VA 22161,

in paper copy or microfiche form.

BIBLIOGRAPHIC SUBSCRIPTION SERVICES

the following current-awareness and literature-survey bibliographies

pre issued periodically by l|ie Bureau:

'iDryogenic Data Center Current Awareness Service. A literature survey issued biweekly. Annual subscription: domestic $25; foreign

k30. Liquefled Natural Gas. A literature survey issued quarterly. Annual

I Subscription: $20.

Superconducting Devices and Materials. A literature survey issued

quarterly. Annual subscription: $30. Please send subscription or- ders and remittances for the preceding bibliographic services to the National Bureau of Standards, Cryogenic Data Center (736) Boulder, CO 80303.
U.S. DEPARTMENT OF COMMERCE

National Bureau of Standards

Washington. D.C. 20234

OFFICIAL BUSINESS

Penalty for Private Use, $300

POSTAGE AND FEES PAID

U.S. DEPARTMENT OF COMMERCE COM-215

US.

SPECIAL FOURTH-CLASS RATE

BOOK