Research Note 85-18

^7

CO

CO

CO

An Annotated Bibliography of the

Manned Systems Measurement Literature

CD

<

O

<

Joan M. Edwards, Richard F. Bloom, John F. Oates, Jr.,

Susan Sipitkowski, Paul A. Brainin,

Richard J. Eckenrode, and Peter C. Zeldler

Dunlap and Associates. East, Incorporated

8

ARI Field Unit at Fort Bennlng, Georgia

Seward Smith. Chief

Training Research Laboratory

Harold F. O'Neil, Jr., Director

rf- » »

U- S. Army

<$* \

Research Institute for the Behavioral and Social Sciences

February 1985

Approved for public rtltate; dittnbution unllmittd.

^5 10 ig i r«

- •

• *• "_'»

'•-"- -'• •-•."• >/ .•:•.
U. S. ARMY RESEARCH INSTITUTE

FOR THE BEHAVIORAL AND SOCIAL SCIENCES

A Field Operating Agency under the Jurisdiction of the

Deputy Chief of Staff for Personnel

EDGAR M. JOHNSON

Technical Director

L.NEALE COSBY

Colonel. IN

CommanHcr

Accc;ion For

NT1S Cl

DTIC 1

D

,

Tbit rtport, at tubmittad by the contractor, hai batn claarad ic ralcaw to Dcfente Tachntcal Information Ctni«r

(DTIC) to comply «with regulatory raquiramanti. It hat btcn given no primary distribution other than to DTIC

and will be available only through DTIC or other reference »ervicet tuch at the National Technical Information

Service (NTIS). The view*, cpinirnt, and/or fmdingt contained in thit report are thotc of the authordl and

thould not be conttrueu at an offici«! Depaitmem of the Army position, policy, or decision, unless so designated

by other official documentation.
SECURITY CLASSiriCATION OF THIS PACE (Whaf> Dmim Enfft)

REPORT DOCUMENTATION PAGE READ INSTRUCTIONS

BEFORE COMPLETING FORM

1. REPORT NUMBER

ARI Research Note 85-18

2 COVT ACCESSION NO

E&bfc&L££J

I RECIPIENT'S CAT ALOC NUMBER

«. TITLE (and Subtltlm)

AN ANNOTATED BIBLIOGRAPHY OF THE

MANNED SYSTEMS MEASUREMENT LITERATURE

• TYRE OF REPORT t PERIOD COVERED

Final Report

t PERFORMING ORO. REPORT NUMBEB

293-6

. AUTMORf«;

Joan M. Edwards, Richard F. Bloom, John F. Oales,

Jr., Susan Sipitkowski, Paul A. Brainin, Richard J.

Eckenrode and Peter C. Zeidler

• CONTRACT OR GRANT NUMBER.)

MDA903-80-C-0345 A-

» PERFORMING ORGANIZATION NAME AND ADDRESS

Dunlap and Associates East, inc.

One Parkland Drive

Darien CT 06820

10. PROGRAM ELEMENT. PROJECT. TASK

AREA • WORK UNIT NUMBERS

2Q263743A794

11. CONTROLLING OFFICE NAME AND ADDRESS

U.S. Army Research Institute for the Behavioral

and Social Sciences

5001 Eisenhower Avenue, Alexandria VA 22333

12. REPORT DATE

February 1985

11 NUMBER OF PAGES

524

TJ MONITORING AGENCY NAME B ADDRESSC" dllltttnl Irom Ctmtmlllni Oltle») IS. SECURITY CLASS, (ol thl, r.pon.

Unclassified

IS« DECL ASSlFiCATION DOWNGRADING

»CMEDULE _ _

1«. DISTRIBUTION STATEMENT (ol Ihl» KftoH)

Approved for public release; distribution unlimited

17. DISTRIBUTION STATEMENT (of th« mbtumcl m>fr»d In Block 30, II iltlotmt from Rrpori)

IB. SUPPLEMENTARY NOTES

Seward Smith, Contracting Officer's Representative

IB. KEY »ORDS (Cantlnuo on torottm »Id» II n»e»»mmy and Idmntlty by bloc* niaibar;

Manned System; Evaluation, Performance Measurement, Effectiveness Measurement

ta AMTHACT (TmXfnm urn f% •(* fl n*c«««a>y tod lamltj by block mm*m)

A comprehensive computerized literature search was conducted on the subject

of systems and measurement theory and practice. As a result of this search,

244 citations were identified as very likely relevant to the area of interest,

and were abstracted for this document.

W IJAM71 1473 EDITION OF I MOW «US OBSOLETE Unclassified

»ECumTT CLASSIFICATION OF THIS PAGE fWwn Dmlm EnffO)

•' •- . -. -%V W.^"

- "•- ■■" • -•--• . .••. . . 1 ■■> . -.•" vV.
ACKNOWLEDGEMENTS

The authors depended greatly upon the assistance of many individuals during

the course of this project. In particular, we wish to acknowledge the invaluable

technical assistance provided by Mr. Hal C. Str?sel pno Ms. Dorothy L. Finley.

of AR1 and Drs. Marvin Grunzke and Edward Youngling of Litton/Mellonies, all at

Fort Benning.

We would also like to thank the word processing department of AR1 (Fort

Benning) for their hard work in producing the typed copies of the abstracts.

Constant support and assistance on technical and administrative issues were

generously provided by Messrs. Joseph T. Fucigna and Edward W. Bishop, the former

and present Responsible Officers for this project at Dunlap and Associates. Inc.

Finally we acknowledge the help provided by the Dunlap technical assistants,

namely, Cynthia Gilchrist, Susan Davies-Bloom, Karen Kolenda-Schoelch and Janet

Vartuli.

i

'

in

.. --.• . - ' .. . • - • v.- :
'

SUMMARY

A comprehensive computerized literature search was conducted on the subject of systems and measurement theory and practice. As a result of this search,

244 citations were identified as very likely relevant to the area of interest, and

were abstracted for this document.

S

•

i

.

- - " -

%

- - - - • - - ^ -. . - _ . • . - _ • _ * -. •.. - „% „v. „^
TABLE OF CONTENTS

Page

I. INTRODUCTION 1

n. REVIEW 01 SUBTASKS 3

A. Subtask 3a{l): Obtain Relevant Literature 3

B. Subtask 3a(2): Review and Annotate Literature 5

APPENDIX A - Annotated Bibliography A-l

APPENDIX B - Bibliography (without annotations) B-l

vii

.»»'.
1. INTRODUCTION

I This report describes one of the earliest tasks under the "Study of Effectiveness of Infantry Systems: TEA, CTEA and Human Factors in Systems Development

and Fielding" (Contract No. MDA903-80-C-0345). Dunlap and Associates, Inc., is

responsible for Task 3 (Systems Development and Evaluation Technology) of t lat

contract, under subcontract^LMo^ J)5628) to the Mellonics Systems Development Divi-

'. m sion of LifTon Systems, Inc. This present report is partial fulfillment of Task 3a,

"Review of the Manned Systems Measurement Literature."

Published research and other literature pertinent to systems and measurement

theory and practice were identified, acquired, reviewed and annotated during this

task. The purpose of these activities was to obtain a clear understanding of the

state-of-the-art of manned systems measurement. This would indicate the necessary

direction and scope of the effort required to expand and update the Systems Taxonomy

Model (STM) and the components of the Overall Conceptual Process Model (OCPM).

Task 3a was divided into two subtasks:

•

i

1. Obtain Relevant Literature

2. Review and Annotate Literature

The methods used for completing these tasks are discussed in the next sections.

Appendices A and B are the bibliographies themselves, in annotated and unannotated

form, respectively.

. «

.':vv:v:>:-:::-:v:-;-;.;. /y^.
-,

II. REVIEW OF SUBTASKS

A. Subtask 3a(l): Obtain Relevant Literature

The identification and acquisition of relevant manned systems measurement

literature was built on an existing base of documentation. This base consisted of

the searches conducted by ARI of the National Technical Information Service

(NTIS) and Defense Documentation Center (DDC), now Defense Technical Information Center (DT1C), data bases in February 1977. The Contracting Officer's Technical Representative (COTR) provided Dunlap and Associates, Inc., with the strategy

utilized in these searches and also approximately 200 documents which had been

obtained in that 1977 search. Key words used in the 1977 searches were: operational test and evaluation; performance standards measurement; systems development,

test and evaluation; performance measurement; systems human operators; measurement methodology; systems and methodology; measure theory; and effectiveness

measurement.

Dunlap updated and extended the ARI literature file by conducting searches

using the same data bases and key words to acquire new entries since the original

search was performed in 1977. For the NTIS search, additional key words were

selected from the candidate terms listed in Table 1. In addition to the NTIS and

DTIC searches, the search was expanded to include the PASAR and COMPENDEX

data bases. Complete bibliographic citations were obtained, including abstracts

and printouts from the four searches.

Two of these searches (NTIS and COMPENDEX) were conducted in-house

using the set of DIALOG data bases maintained by Lockheed's Palo Alto facility.

The Defense Techncial Information Center conducted the DTIS search and the

American Psychological Association performed the PASAR search under the direction of Dunlap staff members.

The next step was to determine the relevance of the results of the computerized search. A triage of the results was performed and abstracts were

coded into one of the following three mutually exclusive categories: V for "very

likely relevant," P for "possibly relevant" and L for "very likely not relevant."

The triaging results for each data base are summarized in Table 2. The percentages given in the table are approximate. In the "very likely relevant" category,

there is one duplication of a report and that occurs in the NTIS and DTIC lists.

Since there may be duplications among the other categories as well, the actual

totals may be less than the values indicated in the table.

A copy of the triage ratings was submitted to the COTR and based upon

discussions with the COTR and in view of the volume of materials already available, it was decided to obtain, at the present time, only those documents classed

as "very likely relevant." Approximately 50 such documents were acquired.

-3- P

PREVIOUS PAGE

IS BLANK ©

./.-: ,-: >:>>. :>:>: • - - - • - ^- • - - -••_- ^ »• . -v .- .-/•.
Table 1. Candidate Key Words for Literature Search

System (Design, Analysis, Effectiveness, Definition, Attributes,

Testing, Constraints, Performance)

Development (Methods, Process, Models, Technology)

System Development (Methods, Process, Models, Technology)

Manned System Development (Methods, Process, Models,

Technology)

Mission Definition

Measurement (Methods, Process, Models, Analysis, Techniques,

Standards)

System Measurement (Methods, Process, Models)

Manned System Measurement (Methods, Process, Models)

Analytic Methods

Taxonomy (Models)

System Taxonomy (Models)

Evaluation (Methods, Process, Models, Technology, Criteria,

Techniques)

System Evaluation (Methods, Process, Models, Technology)

Manned System Evaluation (Methods, Process, Models,

Technology)

Cost Benefit (Analysis, Evaluation, Methods, Measures)

Cost Effectiveness (Analysis, Evaluation, Methods, Measures)

Measures of Effectiveness

Performance (Requirements, Criteria, Analysis, Measures,

Assessment)

Test (Plans, Planning, Methods, Development)

Effectiveness (Evaluation, Testing, Measures, Criteria, Analysis,

Assessment)

Proficiency (Measures, Measurement)

Man-Machine Systems (Evaluation, Methods, Process, Models,

Technology, Assessment)

Human Factors (Analysis, Evaluation)

State of the Art

-4-

. .•-.--.

. -v.-.--v. ■■ . . -. -w .
»

Table 2. Triage Results for Four Data Base Searches

Data Bases Relevance Total V P L

NTIS

DTIC

PASAR

COMPENDEX

23

19

7

9

29

49

10

15

62

32

15

27

114

100

32

51

TOTAL 58 103 136 297

PERCENTAGE 20% 35% 45% 100%

B. Subtask 3a(2): Review and Annotate Literature

Following the step? described above, abstracting of the relevant material

began. The format of literature annotation/abstracting conformed to the components of the overall conceptual process model. That is, an abstract was

prepared for each document included in the annotated bibliography using a standard abstracting form as illustrated in Figure 1. This standard abstracting form

indicates all of the conceptual process model components to which the particular

document applies; the abstract itself consists primarily of separate summaries of

the document's contents relevant to the indicated model components.

An annotated bibliography of the relevant literature is presented in Appendix

A. The bibliography indicates all of the documents obtained as a result of the

search performed in 1977 and those documents that were identified as "very likely

relevant" in the new searches and obtained. A listing of documents, without abstracts, is presented in Appendix B.

-5-

, V . i
APPENDIX A

ANNOTATED BIBLIOGRAPHY

A-l

PREVIOUS PAGE

IS BLANK ©
AC Spark Plug Division, General Motors Corporation. Inertial guidance system

107A-2—Category II, Personnel Subsystem Test and Evaluation (PSTE) and

Maintenance, Logistics, Reliability and Readiness (MLRR) test and

evaluation—Objective achievement status report (AF On (69^)-177 AFBM Exhibit

60-20A). Milwaukee, WI, 15 February 196^. (AD-829 7U9).

I Topics Relevant I I

1 to System Development !Topic!

land Evaluation Technology I No. I ABSTRACT

1. State of the Art Review

of the Process

1.1 General Systea

Heasuresents

1.? System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPU)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1» General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3- Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3-3 Performance

Requirements, Specific

3.'* Performance

Criteria, Specific

3.5 Measurement Procedures

4. Planning Components

of the Process

1.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoaaendations

6. Further Research Areas

6.1 Measurement Systea

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

2.2 This report specified the Personnel

Subsystem Test and Evaluation (PSTE)

objectives for the TITAN II IGS.

2.5 Two system measures were developed: a

measure of system "adequacy" and a measure

of system "efficiency". The former measure

subsumed the concepts of "availability" and

"accuracy", while the latter measure

included "expenditure per unit of output"

as the primary yardstick.

3.3 The PSTE objectives were broken down into

three components that involved weapon

system testing as it related to human

engineering, personnel, training, and

validation of technical publications:

(1) Personnel Performance

(2) Safety

(3) Technical Data

The MLRR (Maintenance, Logistics,

Reliability, and Readiness) test and

evaluation included the collection and data

analysis items relevant to weapon system

testing.

3.4 Operationalization of this concept included

the following representative items: for

"maintenance", an indicator might be the

determination of whether the support

activities for missile and group equipment

maintenance are adequate; the

A-2

. . "> V 'v'--.
I Topics Relevant I

I to System Development iTopic I

land Evaluation Technology ! No. | ABSTRACT

"logistics" concept includes spare parts

consumption and compatibility requirements;

"reliabilitj" involves collection of

failure data regarding all subsystems;

"readiness" involves an evaluation of the

interrelated effects of operations,

maintenance, and reliability on the

"in-commission" rate. Finally, weapon

system capability is determined by the

product of alert readiness reliability x

launch reliability x in-flight reliability

x warhead reliability.

5.1» The main conclusion for the Personnel

Subsystem section found personnel operating

within tolerance limits although

contributing 20% to the total downtime of

the system. A morale problem was found due

to the minimum level of capability required

for the guidance system that underutilized

personnel skills. Corrective training and

reorientation were required.

A-3

v>>>v»»:

...... . . .
Akashi, H. 4 Mahmood, S. Performance of human operators under various system

parameters (NASA CR-672S). National Aeronautics and Space Administration,

(undated)

Topics Relevant I

1 to System Development 1 Topic 1

!and Evaluation Technology ! No. ! ABSTRACT

1.

2.

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2 2

Contextual Components

of the Process ? 3

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.H General Constraints ? i

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria. Ultimate

Analytic Components

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.'i Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoonendations

6.2

6.3

Further Research Areas

6.1 Measurement System

Limitations

Research Potentials/

Priorities

Research Planning

2.1 A closed loop system, consisting of a

display, control stick, an analogue

computer, an operator and procedures, was

used in this evaluation.

The performance of operators under various

system parameters was measured.

Untrained operators were tested in a

laboratory situation.

The analogue computer compared the random

input signal with operator tracking signal

and computed an error signal.

3.2 A performance index which is the fraction

of the total time during which the error

signal exceeded an arbitrarily chosen

threshold was used as the measure.

3.3 The objective was to measure the

performance of hunan operators under

various system parameters.

^.3 A display (oscilloscope), control stick,

and a controlled element (analogue

computer) were used for this study.

*».U Operators were taken from 3 groups:

(1) persons with both licensed flying and

driving experience; (2) persons with

average driving experience; (3)

non-drivers. Ages: 20-40.

A-4
Topics Relevant

I to System Development I Topic

land Evaluation Technology ! No. ABSTRACT

5.3 The results show that the performance of

operators can be represented in the

parameter plane of time constant and gain

by a hyperbolic curve, its listance from

the origin showing the contr«! ability of

the operator and its general shape

indicating the operator's adaptability to

the variation of the two parameters.

5.1 The effect of changing the nature of random

noise, the display device and the manual

control have yet to be studied.

i

i- -:

A-5

A -.-.•- ,.. y --•---- •- - - • • • '"wfc
Analytics, Inc. Measuring the performance of operational decision aids

(N0O014-75-C-06OO, Final Rep. 1161-B). Willow Grove, PA: Analytics, Inc.,

April 1976. (AD-A024 795).

Topics Relevant 1

I to System Development I Topic

land Evaluation Technology ! No. ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 Systen Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3-3 Performance

Requirements, Specific

3.,( Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

1.1 The problem addressed is the development of

a "methodology" for testing the

acceptability of Navy operational decision

aids (or systems). Resulting tests must be

scientifically sound, statistically

reproducible, objective and unbiased,

extrapolatable to real world conditions,

plausible, defensible, and evaluatable in

terms of liabilities, performance and risk.

The test methodology must assess the

functional performance of the system and

cannot be designed to match the system

itself, lest the result be determined by the

evaluation method.

3.1 Classes of systems were expected to be

identified, such as information storage and

retrieval systems. This methodological

stuJy addressed the formulation of measures

of performance (MOP's) applicable to

information storage and retrieval type

systems. Two types of measures emerged:

(1) asymptotic measure of improvement in the

decision process, and (2) a tine constant

identifying the rate of improvement. The

measures are calculable from experimental

measures of reliance, irrelevance and time.

The specific and tentative measures are

shown to leave a common interpretation as

Bayesian updated probabilities.

Application Coatponents

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-6

.-. ... . _ . ---'-•- - - - - - - - -
r. .'•

Anderson, R. Measures of aircraft effectiveness (AOS-TR-73-5). Kirkland AFB,

NM: Office of The Assistant for Study Support (OAS), May 1973. (AD-913 306).

Topics Relevant

to System Development iTopicj

and Evaluation Technology j Ho. ! ABSTRACT

.

u.

State of the Art Review

of the Process

1.1 General System

Heasurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environaent Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

11.2 Parameter Determinations

1.3 Apparatus for Tasting

1.1 Personnel for Tasting

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.1 Conclusions and

Rec ommenda ti ons

2.5 The utility of a tactical interdiction

aircraft depends upon kill potential,

probability of reaching the target,

probability of survival, and availability.

The aircraft's worth cannot be assessed by

considering these factors in isolation.

3.2 Measures of effectiveness must be developed

which quantitatively account for the

interaction of these characteristic

effectiveness parameters. Any valid

measure of effectiveness must account for

the cumulative effect of repeated sorties.

4.1 To obtain a measure of effectiveness of an

aircraft in a given scenario, it seems

reasonable to keep the scenario fixed

(fixed characteristic parameters) and to

determine cumulative effectiveness under

repeated sorties in that fixed scenario.

5.2 The expected number of targets destroyed

was expressed as a function of several

probablistic variables, including kill

potential, probability of reaching target

and releasing weapons, probability of

survival, etc. The expected number of

targets destroyed was computed for the

aircraft's total lifetime as well as for an

arbitrary number of sorties.

5.U It was shown that survivability is of

utmost importance since it determines the

average number of sorties an aircraft can

complete.

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

. • _ • -

A-7

-
Anderson, W.H. Development of performance measures for organizational level

aviation maintenance managers. NTIS Weekly Government Abstracts. October 17,

1977. (Abstrict)

I Topics Relevant I 1

I to System Development I Topic I

iand Evaluation Technology i No. I ABSTRACT

1. State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.h Central Constraints

2.5 Perlormance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3- Aialytic Components

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

4. Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendaticns

6. Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

2.1 The assigned functions of organizational

level aviation managers are the subject of

this evaluation process.

3.2 Functions are defined in terms of their

objectives and "appropriate" measures are

developed which reflect the effectiveness

and efficiency with which these objectives

are accomplished.

3.3 The use of these measures is intended to

provide effective feedback data for

planning and controlling functions as well

as for objective performance appraisal.

A-8

^ ' •• - 1 _

/VVV .;

M

•<*fr
Andrews, L.B. Detailed Test Plan (DTP) for the Initial Operational Test and

Evaluation (I0T4E) (OT-II) of the AN/IPS-59 radar set (C 0041-0-07-7).

Quantico, VA: Conmanding General, Marine Corps Development and Education

Command, August 1977. (\D-B020 8U8L).

Topics Relevant I !

i to System Development !Topic I

and Evaluation Technology ! No. I ABSTRACT

State of the Art Review

of the Process

1.1 General Syste«

Measurements

1.? Systeic Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environaent Definition

2.U General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Ana.'ytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

l|.2 Parameter Determinations

l|.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

2.1 The radar system in question was a

lightweight, long-range, three-dimensional

system designed for air surveillance and

ground control intercept.

2.2 The radar was designed for a range of up to

300 nmi and 100,000 feet altitude to

detect, identify, and classify targets

within a defined air space to the Tactical

Air Operations Center (TAOC). The purpose

of the actual test was to provide data

analysis on the operational effectiveness,

suitability, and military utility of the

Radar Set.

3.3 The Operational Testing (OT) objectives

were the verification of radar's ability to

meet stated operational requirements and to

estimate the radar's military utility,

operational effectiveness, suitability, and

any need for modifications.

4.5 During 0T, the radar was to be tested in

all primary modes and in its secondary

modes vis-a-vis the TAOC. Varying flight

profiles will be used to assess radar

detection ability and testing, itself, will

be conducted under all weather conditions

on a 24-hour basis.

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoonendatlons

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-9

_^_^_^_w. •- -• •- - • T. ^ ^ ^ - ^ ^ , -.' O -." %,'

- "- - «Ml - -•--
Baker, J.D. Quantitative modelling of human performance in information

systems. Ergonouics. 1970, _1^(6), 615-664.

|. Topics Relevant

I to System Development

!and Evaluation Technology ABSTRACT

State of the Art Review

of the Process

1.1 General Systea

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1 General Constraints

2.5 Performance

Requirements, ultimate

2.6 Performance

Criteria, Ultimate

Analytic Coaponents

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Compon-nts

of the Process

4.1

1.2

*.3

1.4

1.5

Analytic Methods

Parameter Determinations

Apparatus for Testing

Personnel for Testing

Test Plans

1.1 This paper summarizes an approach toward

developing a general information system

model which focuses on man and considers

the computer only as a tool. The ultimate

objective is to produce a simulator which

will yield measures of system performance

under different mixes of equipment,

personnel, and procedures. In structuring

the framework for this model, the

assumption was made that men have five

basic and critical operations to perform in

an information system: screen, transform,

input, assimilate, and decide. These

operations, or functional areas, are

interrelated along three dimensions: (1)

a data flow and processing dimension; (2) a

task analysis dimension for each event in

the data flow sequence; and (3) a source of

variation dimension, such as level of

training. The model approach described has

several major points of payoff. Among the

immediate benefits is the potential for

using the model to quantify human

performance by employing system measures

and the value of the model as a tested,

usable tool for developing test and

evaluation plans which will provide human

factors data as part of the information

system design verification checkout.

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.1 Conclusions and

Rec omaenda 11ons

Further Hesearch Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-10

• - -- - • - - m --------- . • . ~ « '
Basso, G. L. A methodology for measurement of vehicle parameters used in

dynamic studies. Ottawa, Canada: National Aeronautical Establishment, July

1973.

I Topics Relevant

I to System Development

land Evaluation Technology

iTopic!

i No. | ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1) General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3-3 Performance

Requirements, Specific

3.4 Performance

Criteria, Specific

3-5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Method:

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

2.1 The system under evaluation is the highway

vehicle in interactions with its occupants,

the terrain and physical obstacles, during

the course of ordinary operations and in

the event of accidents.

3.1 The parameters selected for measurement

using the air-bearing device described in

this study were chosen to match those in an

existing mathematical simulations model, so

that later comparisons could serve to

validate the model.

3.5 The measurement equipment, comprised of a

system of air bearings, was designed to

help measure vehicle parameters in a manner

suitable for dynamic studies of vehicles.

By providing the data needed in the

mathematical model, the measurement system

helps to facilitate studies involving

various aspects of the vehicle occupant

terrain obstacle system. The air bearing

configuration of the measurement device was

determined by the requirement for

flexibility in the types of experiments to

be conducted and by the availability of

local expertise in the technology. The

majority of this document describes the

apparatus and its utilization.

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Reconwendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potenti^s/

Priorities

6.3 Research Planning

A-ll

- - -- - ... , lb M • ---
Beau, J.F. Management of the human element In the physics of failure,

presented at The Third Annual Symposium on The Physics of Failure in

Electronics, Chicago, IL, September 29. 1964. (AD-812 5*3).

Paper

1 Topics Relevant

I to System Development

land Evaluation Technology ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 Systec Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultima-e

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.I Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

4.2 Parameter Determinations

t.3 Apparatus for Testing

n.n Personnel for Testing

4.5 Test Plans

Application Coaponents

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoonendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potential«/

Priorities

6.3 Research Planning

2.1 The human element in electronic production

systems Is the focus of this report.

2.5 Reliability of human performance in factory

processes is the issue of concern.

3.1 The measurable attribute for assessing

reliability and the worker's contribution

is the rate of escape into the field of

products with unacceptable workmanship

defects.

3.2 An audit of final products, in combination

with defects found by inspection, leads to

the calculation of an Estimated Outgoing

Quality Level (EOQL), which is an overall

performance effectiveness measure.

3.3 The specific requirement is to maintain an

overall Average Outgoing Quality Limit

(AOQL). The overall AOQL is budgeted to

establish contributing AOQL's for

production subdivisions. These figures, in

turn, are used to specify a Submitted

Quality Level (SQL), for products submitted

for inspection.

3.5 The study is concerned with measuring

output quality ano setting acceptable

limits for poor workmanship escapes and

product degradation. This includes

establishment of a 3-level classification

scheme for defects, based on the expected

impact of defects on product performance.

Consistent measurement also requires the

standardization of vocabulary for

describing defects found uuring the

inspection process.

A-12

■►"- •

---•-•-- ---- • rial 

----- ----- -
't

Berson, B. L. & Crooks, W. H. Guide for obtaining and analyzing human

performances data in a materiel development project (Tech Memo. 29-76).

Woodland Hills, CA: Perceptronics, Inc., September 1976.

Topics Relevant

1 to System Development

!ind Evaluation Technology

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.« General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

I.** Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

1.1

4.2

4.3

%,%

1.5

Analytic Methods

Parameter Determinations

Apparatus for Testing

Personnel for Testing

Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

1.3 This report contains guidelines for

conducting, analyzing and reporting Human

Factors Engineering (HFE) tests according

to the specifications of DI-H-IBS^A. The

requirements imposed by the specification

are presented together with suggested

sources of information.

4.5 A brief description of the test activities

necessary to meet the requirements of

DI-H-ISBAU is presented below:

(1) Test administration — Includes

milestone development, manpower

specification and budget preparation.

(2) Task group description — This task

requires task group identification

(all operations and maintenance tasks

assigned to a single personnel

position); task analysis (defines in

detail the behavioral requirements of

the task), and performance standards

identification (the identification of

the specific functions that the system

must program).

(3) Test planning and design — Test

planning begins with a statement of

test objectives. It Is felt that the

more precisely the test objectives are

defined the easier it is to develop

the test plan. The next task is to

further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-13

.-.-. %-.-. ,-. .

- -. . •.- -.

-'--'-•

-"-..'• . ^. .

1 - - - - ^—.
Topics Relevant

to System Development

and Evaluation Technology

Topic

No. ABSTRACT

select and design the test equipment.

The nature of this configuration

depends upon the stage of system

development. Test environment

conditions and -he need to simulate

those environmental conditions likely

to affect task group performances are

considered. After the required

environmental conditions have been

specified, provisions for measurement

of critical conditions must be made.

For example, changing environmental

conditions will necessitate more

frequent measurement. It is noted

that to a large extent, the validity

of the HFE test results depends on

selection of test personnel. Therefore, detailed guidelines are

presented in the report regarding

personnel selection and personnel

training.

With regard to data acquisition and

analysis planning, xt is noted that

both subjective and objective data

collection techniques are required to

meet the specifications of D1-H-133^A.

Subjective data can be collected by

such means as ratings, rankings,

questionnaires and interviews.

However, it is suggested that

objective measures be employed as much

as possible to better provide

comparison of the obtained measures

and to determine the degree to which

performance standards are met. In

addition, errors arising from human

judgment are minimized.

(1) HFE test execution — It is recommended that a pretest be conducted and

procedures are set forth for conducting such an activity.

A-14

• •- -•- - - - Mi «--«.'. ^. «* v " > v " •

- - - u~*. ^i mhttt MdK - -
I to System Development iTopic I

!and Evaluation Technology ! No. ! ABSTRACT

(5) Data analyses— Various data analysis

techniques and the applicability of

these techniques to the type of data

collected are discussed.

(6) Sunmary — A checklist of the

activities associated with human

factors engineering testing is

included in this report.

i

A-15

".' '. • ... '- S '. . V

ri - - v •- ^ ^ ,
Blanchard, R.E. & Smith, R.L. Field test of a Technique for Establishing

Personnel Performance requirements. 1969 Annals of Assurance Sciences,

272-277. N.Y.: Gordon and Breach Science Pjblishers, July 1969.

I Topics Relevant

I to System Development

land Evaluation Technology ABSTRACT

State of the Art Review

of the Process

1.1 General Systen

Heasurenents

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3-3 Performance

Requirements, Specific

3.4 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Coaponents

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoanendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.3 A technique was developed to quantify and

relate human performance to the operability

component of systems effectiveness. This

man-machine modeling technique is titled

"Technique for Establishing Personnel

Performance Standards (TEPPS)," and two

field tests of its application are reported

here.

4.1 A graphic mapping technique (similar to a

block diagram or flow-chart) is employed to

show how the system is intended to operate,

how it can operate (unintended), its

various required operating states, and the

logic for developing a conditional

probability model. Similar to the

conventional reliability equation, the

TEPPS mathematical model is used in

derivative fashion to determine contributing probabilities of successful subsystem performance when the overall system

required performance probability

(reliability) is known. The model is used

in integrative fashion to determine the

overall probability of successful performance when the contributing subsystem

probabilities are known or assumed. Due to

the lack of actual human capability data,

TEPPS was designed to accept relative

estimates of human capability, which are

obtained by subjective scaling techniques.

6.1 The unavailability of valid human

performance data seriously limits the

utility of the model. Since the lack of

relevant performance data is not likely to

change soon, subjectively derived data

continues to be given prime emphasis. In

addition, there is a need to simplify,

apply and test this technique so that ic

can be made more practical for evaluating

the human component in systems.

A-16

- -.

•-~-•"- - - M^^kl*A^MA*A^te^^BAM - ^ - - - - - • - - - -

>V. \%--.-.-'
Bloom, R.F., Pepler, R.D., Schimenz, M.V. & Lenzycki, H.P. IFV/CFV personnel

selection analysis (Army Research Institute Research Note 80-M1) Darien, CT:

Dunlap and Assoc, Inc., July 1979.

Topics Relevant I

I to System Development iTopic!

land Evaluation Technology ! No. I ABSTRACT

State of the Art Review

of the Process

1.1 General Systen

Measurements

1.2 System Taxonomy

Model (CTM)

1.3 Overall Conceptual

Process Model (LPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definitioi.

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Rec oonendat i ons

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.3 This is the final report of an analysis to

determine extranormal selection

requirements for crew members of the

Infantry Fighting Vehicle (IFV) and the

Cavalry Fighting Vehicle (CFV).

The procedures developed *".o achieve the

research objectives began with a

clarification of objectives and

assumptions. That clarification served

mainly to emphasize the investigation's

concern with extranormal attributes only.

It was then determined from othe:

concurrent efforts that the two vehicles

(IFV and CFV) were similar enough so that a

single consolidated set of five crew

positions was appropriate ''or this

analysis: Track Commander, Driver, Gunner,

Firing Port Weapon Operator (IFV only) and

Observer (CFV only). Next, a taxonomy of

62 personnel attributes was constructed,

and a representative set of IFV/CFV mission

scenarios was developed. The operator's

task and subtask demands occurring during

exercise of tne mission scenario were

analyzed to identify which of these

attributes in the taxonomy were required to

perform the task or subtask. Current

Infantry and cavalry tasks were analyzed to

determine the soldier attributes required

to perform the tasks. These attributes

were then compared with those required to

perform the IFV/CFV mission to identify

those attributes that were new or unique to

IFV/CFV. Six potentially extranormal

attributes were identified for the Track

Commander (TO and Gunner positions, and

A-17

. vufev:-..;-.-.-.-;.-:-
1 Topics Relevant 1

I to System Development I Topic I

land Evaluation Technology ! No. I ABSTRACT

three for the Driver position, on the basis

that they appear tr be new to current MOS

11B or 19D personnel. Those attributes are

especially needed to perform the new or

unique IFV/CFV tasks, and they are not now

used individually for personnel selection.

Any of the attributes is considered

extranormal if it must be possessed at the

level of the mean or higher so that 50% or

less of the personnel pool will provide the

necessary level.

U.I Means, standard deviations, and ranges for

any subset of scores were used in the

analyses.

A-18

* . • .

•:>:-.•: >. :."v"-
Boehm, B.W. Computer systems analysis methodology; Studies In measuring,

evaluating, and simulating computer systems (R-520-NASA). Santa Monica, CA:

""he RAND Corporation, September 1970.

I Topics Relevant I

I to System Development I Topic

land Evaluation Technology ! No. ABSTRACT

1. State of the Art Review

of the Process

1.1 General System

Heasureaents

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.H General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Coaponents

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.H Performance

Criteria, Specific

3.5 Measurement Procedures

4. Planning Components

of the Process

4.1

1.2

«t.3

«.4

4.5

Analytic Methods

Parameter Determinations

Apparatus for Testing

Personnel for Testify

Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

1.1 Three modest, coordinated efforts were

carried out to help in providing better

techniques for the design, evaluation, and

analysis of computer systems: (1) the

development of design principles for

languages to model and simulate computer

systems, (2) the evaluation and extension

of measuring and analyzing the performance

of complex computer systems, and (3) the

analysis of controlled experiments in

man-computer problem-solving. The first

effort provided a set of terms and phrases

described by the author as convenient and

natural while maintaining the flexibility

and power of a general-purpose simulation

language. The second effort was a brief

review of previous studies and a

description of needed further studies. The

third effort reported on an experiment to

test the effect of forced temporal lockout

intervals on human performance In

man-computer problem-solving. It concludes

that the relationships Involved in

man-machine problem-solving are neither

obvious nor simple, and further

investigation is necessary.

3.1 An appendix addresses the development of

performance criteria that are

discriminating and measurable. It

describes a productive thought ratio

(P.T.R.), based on the ^ime spent thinking

about the project in comparison to time

spent thinking about programs and waiting

for computer responses.

6. Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-19

..V. ^v -V ^ . •- _v •- •-' •-' •

i»^ii-iil^ij!il£
Bohdanovych, V.l., Kulyk, V.T. 4 Uyomov, A.I. (Development and practical

application of systems analysis.) Filosofs'ka Dumka, 1973i J.« *

,0

-

1

6.

(Arlington, 7A: Joint Publications Research Service, 1973. JPRS 58935).

Topics Relevant

j to System Development {Topic

land Evaluation Technology I No. ABSTRACT

1. State of the Art Review

of the Process

1.1 General System

Measurements

1.2 Systen Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3 2 Practical Attribute

Measures

3-3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinatio.'S

I4.3 Apparatus for Testing

H.k Personnel for Testlna

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

5.

6.

1.1 In this Russian paper, there is a general

discussion of the need to make broader use,

in the practice, planning and management of

the national economy, of novel methods, in

particular systems analysis.

The history of systems analysis, its

essentials, and its application to the

solution of problems in the concrete

sciences are discussed. Presented are the

general characteristics which emerge in the

planning and management of the national

economy in its p» esent stage and require

systems analysis application.

Outlined is the perspective of the

development of systems analysis into

systemology, a complex science of systems.

The role of Marxist-Leninist philosophy as

the methodology for further improvement and

development of systems analysis is also

discussed.

A-20

"'• '•. •' '.' ••" ••' •-" •.' •.' •- *.' %' • V . '*M\ .' .

, - - • ^Z^l^m. .-. .-.
Bond, N.A. & Rigney. J.W. Measurement of training outcomes (Tech. Rep. 66).

Los Angeles, CA: University of Southern California, Behavioral Technology

Laboratories, June I^O. (AD-711 302).

I Topics Relevant |

I to System Development !Topic

land Evaluation Technology ! No.

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

k.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Coaponents

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoonendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 The report is concerned mainly with methods

of assessing the effectiveness of training

programs, materials, and techniques, with

special focus on Computer-Aided and

Computer-Managed Instruction (CA1, CMI).

Concern with "effectiveness" of training

implies concern with items of information

that show how well the teaching objectives

are realized in the students receiving the

treatment (training).

1.3 The basic assessment/evaluation methodology

can be outlined as follows:

(1) A cle^r statement, in observable

terms, of the expected results of the

treatment. Including the time span

over which a specific result can be

measured.

(2) Development of relevant, reliable

yardsticks (MOEs) which measure

progress toward the stated objectives

(expected results).

(3) Application of the yardsticks within

the time spans of the objectives.

(4) Establishment of an evaluation design

allowing the treatment effects to be

distinguished from intervening

contaminants.

(5) Establishment of the kinds and sources

of information required to evaluate

the treatment in terms of the

objectives.

A-21

.--v- "-• - - - • • - - - -• -

. .-•.• a « >. v• .
I Topics Relevant

I to System Development |Topic I

land Evaluation Technology ! Wo. I ABSTRACT

(6) Specification and examination of

underlying personality and situational

factors which explain the identified

change.

2.6 Only relatively few indices have much

practical use as criteria for evaluating

learning. These include:

3.2

(1) High degree of accuracy in performing

the learned response.

(2) Significantly shorter reaction latency

than at the beginning of practice.

(3) Increased rate or speed of correct

response.

(U) Increased amplitude of response.

(5) Increased resistance to experimental

extinction.

(6) Increased resistance to retroactive

inhibition from subsequent learning as

compared to the amount occurring when

learning stops short of mastery.

(7) Increased positive transfer to

subsequent learning in similar

situations.

(8) A degree of generalization to similar

status events.

General learning measurements that can be

applied to practical training include the

following:

(1) Gain scores (difference between

post-test and pre-test scores).

(2) Process scores (assessment based upon

application of procedures rather than

overall success in problem-solving).

A-22

' -V.V -iL - •- •

•:•.-■■ .^•'- ii -- - -
r

Topics Relevant —T

to System Development I Topic 1

and Evaluation Technology I No. I ABSTRACT

(3) Time to criterion (time required to

complete some work or achieve some

level of success).

(1) Error rate.

(5) Persistence measures (staying with

some specific training sequence).

(6) Transfer measures (generalizability of

the learning to other situations).

(7) Time vs. achievement measures.

(8) Retention measures.

The authors describe the strengths and

weaknesses of these various measures, and

suggest certain types of training programs

for which particular measures might be most

applicable.

4.1 Evaluation designs that are considered

applicable to assessment of training

effectiveness include the classic Solomon

four-group design; iterative adaptation to

individual student progress; response

surface designs; adaptive control models;

decision theory models; simulation models.

5.4 Principal conclusions are that the classic

four-group design is impractical for most

training evaluation; that "adaptive

research for big effects" is apt to be

scientifically and administratively

desirable; and that current measurement of

training outcomes still uses fairly simple

methods.

A-23

^_-

Vf,^•- .

- - - - - um
Bovalrd, R.L. 4 Zagor, H.I. A systems approach to predicting and measuring

Polaris fire control system operational availability (RM 59TMP-57). Santa

Barbara, CA: General Electric Company, Technical Military Planning

Operation, December 1959. (AD-901 773).

Topics Relevant I |

i to System Development !Topic I

and Evaluation Technology ! No. i ABSTRACT

State of the Art Review

of the Process

1.1 General System

Heasurenents

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.M General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Psrformance

Requirements, Specific

3.it Performance

Criteria, Specific

3-5 Measurement Proceci'irea

Planning Components

of the Process

4.1 Analytic Methods

U.2 Parameter Determinations

4.3 Apparatus for Testing

1.« Personnel for Testing

1.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5-3 Findings Interpretation

5.4 Conclusions and

Recooaendations

1.1 The report describes a systems approach to

predicting and measuring the operational

availability of a system. Operational

avallrabllity, along with the performance

capability level is a major determinant of

the system's operational effectiveness. A

multl-moded system is operationally

available whenever it is not down, i.e.,

whenever its performance equals or exceeds

the required level.

2.1 A multl-moded system is described as a

collection of functionally connected but

independent subsystems. Each subsystem is

a set of identical functional groups of a

given type.

M.I This viewpoint of systems and subsystems

permits a simple mathematical prediction of

the expected operational availability of

the system at each performance level. The

operational availability of 1 subsystem

Is shown to be a function of: probability

that any functional group in the 1

subsystem Is non-failed at a random point

in tlme;.nunber of such functional groups

in the 1 subsystem; minimum number of

non-failed functional groups required for

the i subsystem to operate in the

particular mode in question. The total

system operational availability is the

product of the subsystem availabilities.

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-24

------ - - -- -• -

V.-V-.'

- - 

««. «

- - - - -
Boycan, G.G. & Warnick, W.L. Training requirements for the armor crewman and

reconnaissance specialist Advanced Indivi'''ual Training programs (HumRROCR-D2-72-7). Alexandria, VA: Human Resources Research Organization, November

1972. (AD-759 569).

i Topics Relevant I 1

to System Development ITopicI

and Evaluation Technology ! No. I ABSTRACT

1. State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

Of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.'* Performance

Criteria, Specific

3.5 Measurement Procedures

4. Planning Components

of the Process

1.1 Analytic Methods

i(.2 Parameter Determinations

4.3 Apparatus for Testing

U.I* Personnel for Testing

4.5 Test Plans

5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

6. Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.3 This report summarizes the results of the

initial phase of a three phase study. The

study is designed to provide (a) instructional goals for each program that are

stated in measurable terras and (b) corresponding performance-based Go/No Go test

items suitable for evaluating trainee

achievement of these goals.

2.1 The systems addressed were the armor

crewman and reconnaissance specialist

Advanced Individual Training programs

(AIT).

3.1 In this phase, job related tasks addressed

in the AIT programs were examined and

tentative proficiency levels were

established.

3.2 All tasks were individually reviewed and

coded to reflect the estimated level of

mastery required at the end of training and

prior to Job entry.

3.5 Tasks were coded into one of four

categories which represent degrees of

proficiency ranging from complete

qualification to basic orientation.

Definitive Go/No Go performance criteria

were established for those requirements

considered relevant to job entry.

5.^ Data collected during this first phase will

be used as a basis for formulating

performance training objectives and tests

to be subsequently incorporated in new Army

Subject Schedules.

A-25

- - - - Erii1. -....-..-.-.

«-•«-^ ..1-

-'— -
Breaux, R. Training characteristics of the automated adaptive Ground

Controlled Approach Radar Controller Training System (GCA-CTS)

(NAVTRAEQUIPCEN-TN-52). Orlando, FL: Naval Training Device Center, July 1976.

Topics Relevant

I to System Development ITopic

iand Evaluation Technology ! No. ABSTRACT

State of the Art Review

of the Process

1.1 General System

Heasurenents

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definltlcn

2.3 Environment Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

1.2 Parameter Determinations

1.3 Apparatus for Testing

1.1 Personnel for Testing

1.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5-3 Findings Interpretation

5.1 Conclusions and

Reconmendatlons

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

2.2 The automated adaptive controller system

had, as its primary objectives, an increase

in training effectiveness by application of

automated performance measurement,

self-paced instruction, and a type of

adaptive training oriented around cognitive

skills development.

3.1 The risk or problem areas outlined in the

system design included the following:

(1) Performance Measurement: to evaluate

trainee performance, frequency counts

of errors were made in each category;

error frequency counts are then

combined in a linear combination and

weighted to produce a single composite

score.

(2) Adaptive Logic: the problem here is

determining the sequence of problems

within the automated performance

measurement system; the assumption of

the arrangement of the syllabus is

based on an increasingly linear

difficulty of problems.

(3) Adaptive Variables: the arrangement

of the syllabus in terms of increasing

difficulty is based on various

adaptive variables such as wind

factors, aircraft type, pilot

response, and pilot variability.

(H) Student Feedback: the trainee is

given knowledge of his results at the

end of each "run."

A-26
I Topics Relevant

to System Development !Topic I

and Evaluation Technology i No. ! ABSTRACT

3.2 Measures were made on system output

variables and on control input variables.

The former referred to the simulated

history of the aircraft around the

glidepath and runway centerline; the latter

consisted of trainee behavior measures,

e.g., elapsed time between advisories.

A single composite score was created by

weighting a combination of these measures.

The difficulty level of the next problem

was then selected using this score on the

adaptive logic.

5.3 The laboratory feasibility model showed

and that the trainee must be consistent in his

5.1 speaking voice before voice reference

pattern data is collected.

With regard to the training, the key to

effective teaching was found in the order

or sequence of the problems: tasks were

systematically introduced in ascending

order of complexity.

A-27

ffivirivr;-^ .-:..v:: :•:•,.:::-•:-:^^^
Brown, D. Statistical guide for CDEC experimental design (Rev. Preliminary

Draft). BDM Scientific Support Laboratory, October 1976.

I" Topics Relevant

! to System Development

land Evaluation Technology ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3-3 Performance

Requirements, Specific

3.1* Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Rec ommenda 11ons

1.1 This document provides selected experimental descriptors which can be used to

estimate sample size requirements and

compare potential design schemes during the

planning phase of military field tests.

The data is compiled and organized under

six general categories: intervisibility,

detection, identification, localization/pinpoint, engagement, probability of hit/kill.

1.2 Appendix B provides a listing of measures

and of effectiveness by the following experiment

3.2 types:

(1) Mounted Unit Operations.

a. Mounted Unit

Organization/Employment

b. Antitank Weapors Fire Effect

(2) Dismounted Combat Operations.

a. Dismounted Unit

Organization/Employment

b. Small Arms Effectiveness

c. Detection

(3) Indirect Fire Support.

a. Forward Observer and Gunner

Operations

Further Rcrearch Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-28

.- .- • .•-. - .-•. •".--._

_. — ^ - -^ - ^-^ '
I Topics Relevant

i to System Development I Topic 1

land Evaluation Technology i No. I ABSTRACT

b. Gunnery Operations

(U) Army Aircraft Operations.

a. Air System-Ground Target

b. Ground System-Air Target

c. Air System-Air Target

d. Aircraft Operations

(5) Special.

a. Combat Support Operations

b. Might Operations

c. Line of Sight/Exposure

d. Two-Sided Experiments

A-29

I -.-^•!.-/.\v;-..^ .-. ...-..•,. ..•..•;.;•.;.;. •,• .--.•..- .•-•lv ]
Brown, J.D. Field evaluation of Wi8k5 tank product improvementst~ Fort Knox,

KY: U.S. Armor and Engineer Be , 1U January 1977. (AD-B016 139).

I Topics Relevant

I to System Development

land Evaluation Technology ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definltio.

2.2 Mission Definition

Environment Definition

General Constraints

Performance

Requirements, Ultimate

Performance

Criteria, Ultimate

2.3

2.0

2.5

2.6

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

1.2 Parameter Determinations

1.3 Apparatus for Testing

H.H Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.I Conclusions and

Recoonendations

further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

2.1 The purpose of this study was to assess the

operational effectiveness of the product

improved items on the M48A5 tank to include

crew duties and maintenance.

The selected product improvements included

a low profile commanders' cupola, externally mounted 7.62 mm machine guns and a

redesigned 54-round main gun ammunition

storage rack.

2.3 The tests were conducted with ^s much

tactical realism as possible and included

operation on primary, secondary, and

croas-country terrain.

3.1 Reliability, availability, and maintainability data were collected on the product

improved items. Personnel skills and

training requirements were also identified.

M.5 The test plan utilized was the USAARENBD

Test Design Plan for Field Evaluation of

M48A5 Tank Product Improvements described

in TRADOC Project Number 1-VC-080-MU8-602,

May 1975.

For each product item, detailed

descriptions were presented of test

procedures including objective, method,

analysis, and results.

A-30

v- -"S;-: .. >;■■•: : >.

—-^ - , v........

- •••." . . .• . i

\ v •-• .• - • •. -\-. ■■. J

• - - - - - ^
Topics Relevant

! to System Development I Topic

land Evaluation Technology ! No. ABSTRACT

5.1 There were several deviations from the test

plan. These included no firing from a

moving tank, the unavailability of the

simulated mission firing target, and the

impossibility of accurately scoring gun

engagements. Conventional silhouettes were

used and some of the scoring was

subjective.

5.3 It was concluded

improvements prov

increased capabll

system. However,

factors engineer!

addressed within

framework of the

Other corrections

minor nature were

that the product

ided the system with

ities over the present

several safety and human

ng problems should be

the current cost and time

development of the system.

and improvements of a

recommended.

A-31

•. V V -. v' •.
Buckley, E.P., Goldberg, B., Rood, A., Hamilton, H. & Ciamplon, P.

Development of a performance criterion for Enroute Air Traffic Control

personnel research through air traffic control simulation; Experiment

I-parallel form development (FAA-RD-75-186, Interim Rep-). Atlantic City,

NJ: Federal Aviation Administration, National Aviation Fc ilities

Experimental Center, February 1976. (AD-A023 Uli).

I Topics Relevant

I to System Development

land Evaluation Technology

Topic!

No. I ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STW)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.H General Constraints

2.5 Performance

Requirements, Ultiaate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3-3 Performance

Requirements, Specific

3.'I Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

«.I Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

1.1 This report describes one of a series of

experiments used in the development of a

standardized performance criterion for

journeyman enroute air traffic controllers.

The final performance measurement system

will be used in personnel research such as

the evaluation of aptitude tests as to

their capacity to predict suitability for

entrance into training. The criterion

measure being reported here will be based

on the use of realistic dynamic simulation

of the radar air traffic control situation.

Its specific purpose is to explore one

method of constructing parallel forms of a

measurement system.

2.1 This system is an Enroute Air Traffic

Control system consisting of the National

Aviation Facilities Experimental Center

(NAFEC) dynamic air traffic control

simulator and two controllers working

independently.

2.3 Two widely divergent sector structures were

chosen to be examined with three traffic

density levels each.

2.1» Controllers work without assistant

controllers and communicate with "pilots"

over simulated radio frequencies.

5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoonendations

2.5 The ultimate performance requirement of the

Air Traffic Control (ATC) system is the

safe and expeditious movement of aircraft

through the sector.

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-32

.> -••.

•. .
f:

I Topics Relevant

I to System Development I Topic

land Evaluation Technology I No. ABSTRACT

3.1 The measurable attributes were as follows:

Confllctions, delays, completed flights,

oommunlcatlons, identifications, aircraft

handled, physical effort of controllers.

3.2 The attribute measures for this experiment

were as follows:

(1) Number of confllctions.

(2) Number of delays.

(3) Cumulative delay time.

(4) Number of completed flights.

(5) Number of air/ground contacts.

(6) Cumulative air/ground communication

time.

(7) Number of aircraft handled.

(8) Number of identifications requested.

(9) Number of aircraft in sample.

(10) Number of completable flights.

(11) Number of conflictlons/number of

aircraft handled.

(12) Number of conflictlons/number of

delays.

(13) Number of delays/number of aircraft in

sample.

(11) Cumulative delay time/number of

aircraft in sample.

(15) Number of completed flights/number of

completable flights.

(16) Number of contacts/number of aircraft

handled.

A-33 i- ^

. . . - v. -j -.. -^ -j -j _. •_, j
Topics Relevant

I to System Development

land Evaluation Technology ABSTRACT

'

(17) Communication time/number of contacts.

(13) Number of aircraft handled/number of

aircraft in sample.

(19) Correlation hold-delay transformation.

(20) Number of identifications requested

minus number of aircraft in sample.

(21) Controller heart rate.

3.5 All performance measures were recorded, and

ratios computed, by ^he simulator computer.

Heart rates were rec ded continuously

throughout the experiment and compared to a

resting rate.

4.1 Six subjects worked the traffic control

problem in each of two sectors at three

traffic levels. The traffic was generated

by a large-scale digital simulator and

directed by simulator operators who

represented pilots in the real ATC system.

The computer recorded all aircraft events

and printed the performance measure scores

at the end of one hour.

4.2 The test parameters that wert controlled

were as follows:

(1) Sector 1 or 2.

(2) Traffic density of 40, 50, 60 aircraft

per hour.

4.3 The test apparatus was the National

Aviation Facilities Experimental Center

dynamic air traffic control simulator.

4.4 Six qualified enroute air traffic

controllers from the NAFEC evaluation group

served as subjects.

A-34

-.•.•.-, - - - --.*'. ^"_ ... ^. • _ -
Topics Relevant

I to System Development I Topic I

land Evaluation Technology ! No. j ABSTRACT

5.2 The basic data for each subject were

produced in histogram form for each of the

performance measures. A three-factor

analysis of variance was performed

involving the variables of subjects (6),

sectors (2), and traffic densities (3).

5.3 The results indicated that the hypothesis

of interaction between sector and density

in affecting performance was not sustained.

There was little difference shown in the

measures between sectors. Great difference

was shown between the three levels of

traffic density.

5.1 There are wide differences among air

traffic controllers in their ability to

handle identical traffic. It is possible

to measure these differences in a

completely objective manner. The sample of

subjects was small and the data points few,

thus the results of this experiment are

only indicative. Future experiments must

consider the problem of minimal optimal

traffic sample length; one hour is not

enough.

A-35

. -.•.--. -•.

' -' - ,' - J!.«^- _
Burgin, G.H. and Fogel, L.J. Air-to-air combat tactics synthesis and analysis

program based on an adaptive maneuvering logic. Journal of Cybernetics, 1972,

2(M), 60-68.

Topics Relevant

1 to System Development

land Evaluation Technology ABSTRACT

1. State of the Art Review

of the Process

1.1 General Syste«

Heasui ements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Conte.ttual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2." General Constraints

2.5 Performance

Requirements, liltixate

2.6 Performance

Criteria, Ultimate

3- Analytic Cooponents

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.14 Performance

Criteria, Specific

3.5 Measurement Procedures

i|. Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

1.1 Two digital computer programs synthesizing

optimal maneuvers in one-on-one

air-to-air combat situations are described.

The method develops intelligently interactive maneuvers without relying on human

pilot experience. One program drives one

of the interacting aircraft, thus replacing

one of the human pilots on the NASA Langley

Research Center's Differential Maneuvering

Simulator, this in real time. The other

program operates in a normal batch processing mode. Both programs use the same

technique which maps the physical situation

of the two aircraft into a quantized,

abstract situation space. The outcome in

this situation space is predicted for

several trial maneuvers, a value is

associated with the outcome of each trial

maneuver, and finally, the maneuver with

the highest predicted value is executed.

These programs, operating with six degrees

of freedom and realistic aerodynamic

representation for both aircraft, provide a

means for objective evaluation of weapons

systems and pilot performance.

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Prioritiss

6.3 Research Planning

A-36

• *^ o ^" -' • •»•

••» ^ •» V-. r-i

-•.-,• - •

• •- »-«-1.1-'
w -

V

I

Burington, R.S. Concerning the reliability and effectiveness of weapon

systems and their measurement (R-m-6). Washington, DC: Bureau of Naval

Weapons, Office of Chief Mathematician, April 1961.

I Topics Relevant

I to System Development I Topic

land Evaluation Technology ! No. ABSTRACT

State of the Krl Review

of the Process

1.1 General System

Neasureaents

1.2 System Taxonomy

Model (UN]

1.3 Overall Conceptual

Process Model (CPK)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.<4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of tne Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3-5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Teat Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recomnendatlons

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 In discussing the effectiveness,

reliability or readiness of a weapon

system, the following items must be

considered:

(1) General characteristics of the system.

(2) Operational, tactical and strategic

situation for which the system is

envisioned: its missions.

(3) Importance of the system relative to

other systems.

(U) The effectiveness of the system

against various target types, under

various operational situations.

(5) Opportunity for using the system.

(6) Comparative effectiveness with other

existing or possible competitive

systems.

(7) Cost.

(8) Suitability of system for use on

ships, planes, etc.

(9) Types of ships, planes, etc. on which

the system must be used.

(10) Ease of operation and maintenance.

(11) Reliability and operability.

;:::

:2 A-37

->• w - ,•• >v .-. .N VVV-

-^_^. ^ • * v •
I Topics Relevant I

I to System Development I Topic

{and Evaluation Technology I Wo. ABSTRACT

3.1

(12) Susceptibility to countenneasures.

(13) Susceptibility to interference by or

with other systet.s on the same 0

neighboring vehicle.

(14) Mobility and flexibility.

Analysis of potential effectiverass of a

typical weapon system is concerned with

such factors as:

(1) Ability to detect target.

(2) Ability to locate and identify target.

(3) Ability to designate target.

(4) Ability to track target for fire

control purposes.

(5) Ability to bring the vehicle bearing

the weapon into the neighborhood of

the target soon enough to permit use

of the weapon.

(6) Ability to bring the weapon to bear

against the target soon enough to be

effective, once in range.

(7) Ability of weapon system to place the

missile within the desired damaging

radius of the target.

(8) Ability to detonate the warhead at the

proper place, in the proper manner at

the proper time.

(9) Ability of warhead to inflict the

quality of damage desired.

(10) Ability to fire repeatedly with the

necessary degree o' rapidity.

(11) Number of targets that may be engaged

simultaneously within a given interval

of time.

A-38
•.

I Topics Relevant

i to System Development iTopic

land Evaluation Technology ! No. ABSTRACT

3.2 To each of these factors one can define

suitable measures of merit. In general

each such factor must be taken into account

in estimating or evaluating the worth of a

weapon system.

One of the common measues of worth of a

weapon system is the probability that one

burst of a missile will inflict "Kill."

'

A-39

.

".- .' V V V

« I Il.t.l.i'i m i.t.ki.^^ • " • »- »- J
Campbell, S.C., Feddern, J. & Graham, G. A-6E Systems Approach to Training;

Phase I (NAVTRAEQUIPCEN-75-C-0099-1). Bethpage, NY: Grumman Aerospace Corp.,

February 1977. (AD-A037 468).

Topics Relevant

to System Development iTopicl

and Evaluation Technology I No. I ABSTRACT

State of the An Review

of the Process

1.1 General System

Hetsurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.« General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

S.t Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Appl ication Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoonendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.3 This report describes the efforts and

results of the application of a Systems

Approach to Training (SAT).

2.1 The system addressed was the A-6E Pilot and

Bombardier (B/N) training program. The

A-6E tram aircraft is a two man-subsonic,

mid-wing attack aircraft. It is manned by

a pilot and bombardier/navigator.

2.2 The mission of the system is to perform

high and low altitude all-weather attacks.

It can provide close air support for ground

forces or can conduct long or short range

interdiction raids. It is capable of

delivering a large selection of

conventional and nuclear weapons.

3.1 The analysis encompassed the identification

of all the pilot and bombardier/navigator

job performance requirements. In all, over

700 tasks were identified. Task

criticality, frequency of occurrence,

inherent difficulty, changes in knowledge

and skills required were also identified.

3.2 A Taxonomy of Training Objectives was

specifically developed for this program.

Criterion objectives were determined and

criterion referenced tests were developed.

A-40

••> "-'-' ^—^ >..v^ -j _^ ^ ^^J
7*

} %

Chaikin, G. STINGER human factors engineering final report (Tech. Memo. 25-76).

Aberdeen Proving Ground, MD: U.S. Army Human Engineering Laboratory, July

1976 (AD-B014 866).

I Topics Relevant

I to System Development

land Evaluation Technology ABSTRACT

IS

Stat« of the Art Review

of th« Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultiaate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4. 3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5-4 Conclusions and

Recoonendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.3 This report summarizes design-influencing

human factors engineering requirements

applied to the development of the STINGER

weapon system. The system, its use, and

operating procedures are provided as

background for the human factors

engineering components of program planning,

analysis, design, and test parts of the

orogram.

3.1 The following characteristics of the total

system were discussed and operationalized:

(1) Program Planning

(2) General System Description

(3) Function Allocation

(4) Critical Tasks

(5) Weapon System Weight and Size

(6) Launch-Induced Environment

(7) Launcher Controls

(8) Human Engineering Handling Tests

(9) System-Handling Performance

(10) Training Equipment

A-41

.v -

r

-

-' . '.- . '. . - .-• . - .-.
1 Topics Relevant 1 I

! to System Development I Topic I

and Evaluation Technology ! No. | ABSTRACT

3.2 Under program planning, the established

goal involved the design and development of

equipment, facilities, and procedures which

would provide a work environment conducive

to effective work patterns and personnel

safety, minimize discomfort, distraction,

etc., and downgrade human performance

and/or increase error.

A second part of the program planning

involved the "test efforts" with an eye

toward securing data-relevant selected work

cycles, tests in which human participation

is critical (speed, accuracy), use of

personnel representative of the military

population, collection of task performance

data, identification of discrepancies

between required and obtained tasks

performance, and establishment of criteria

for acceptable performance of tests.

(Further sections of this document

concentrate on narrow, technical aspects of

the STINGER weapon system, and are not

relevant to the objectives of this review.

A-42

_^: *_ J.

■■>.<••:••> ':-•>••:•••.■■•. ".-:; .-..-.. .; -^.
Chaney, F.B. & Thresh, J.L. Diagnosis and correction of quality problems:

human factors approach. Paper presented at the ASQC Seminar on Product

Quality Audit, Milwaukee, WI, April 19-20, 1968. (AD-855 919).

I Topics Relevant I I

I to System Development iTopic!

land Evaluation Technology ! No. ! ABSTRACT

.>

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPU)

Contextual Components

of the Process

?.1 System Defiriltion

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 .Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1J Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

application Components

oj" the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Concluaions and

Recoanendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 Effective quality auditing requires

consideration of human factors at each

phase of the process from initial concept

and engineering design through

manufacturing, inspecti n and testing

operations. The purpose of this paper is

to describe human factors techniques used

to obtain better understanding of basic

quality problems.

3.2 The first requirement for improving

inspection accuracy was development of a

standard procedure for measuring Inspection

effectiveness. Job sample performance

teats were developed by selecting

representative hardware Items with a number

of known defects. These measurements

provided basic data for pinpointing problem

areas and evaluating potential

effectiveness of various methods for

improving Inspection accuracy.

5.1» Research has indicated that low inspection

accuracy may be due to a number of specific

factors, such as:

(1) Product factors.

(a) Equipment complexity

(b) Defect rate

(2) Job (inspection) factors.

(a) Procedures

(b) Tools

(c) Visuals

A-43

•-V'V-".-V v-v- _^ . .'•-**:..': .-. • . . .
K

i Topics Relevant "T

I to System Development iTopic

jand Evaluation Technology I No. ABSTRACT

(3) Human factors.

(a) Selection of inspection personnel

(b) Training of inspection personnel

(c) Motivation of inspection personnel

A-44

1 -'- -'• -'• -"• -"' -'• - • -'• -'• -'• -"' - • -'• -^ -'• -"• -"• - • -'* -"» -"»-"' -"' -'« -'» -"*-'* -'«-"» --i-'k '» ' '. -- • .•.• " • ;.' -', ."
Chapanis, A. Relevance of physiological and psychological criteria to

«an-machine systems: The present state of the art. Ergonomics, 1970, 13,

337-316. (AD-751 W.

Topics Relevant

I to System Development I Topic

land Evaluation Technology 1 No. ABSTRACT

-

St«te of the Art Itevlew

of the Process

1.1 General System

Heasureaents

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPU)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 ErvlroriBent Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.'* Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

1.2 Parameter Determinations

4.3 Apparatus for Testing

l.l Personnel for Testing

1.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

S.I Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitation)

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 As shown by content analyses of symposia

papers and journal articles, significant

differences exist between U .' related

disciplines of "Ergonomics^ ^European) and

"Human Factors Enginee-ing" (American).

Ergonomics appears to be more

physiologically—oriented. Human Factors

more psychologically-focused. In America

there has been more concern with

integration of man into large machine

systems. In Europe there has been more

concern with the welfare of the individual

worker.

1.3 The difference in orientation, in essence,

is a methodological problem. Ergonomists

land human factors engineers) are more

concerned with methodological problems than

are physical scientists and engineers,

because precise answers about the behavior

of man are hard to find in any handbook or

textbook. Huch work is taken up with

studies in which methodology is of such

great importance.

The methods, techniques, apparatus, and

variables used by psychologists tend to be

different from those used by physiologists.

Trying to decide which experimental methods

are appropriate to any practical problem is

a very complex question.

2.6 The value or worth of a system is normally

Judged by several criteria, not necessarily

all compatible. Criteria vary greatly from

system to system, and many iriteria are

specific to particular systems. Typical

man machine system criteria include:

A-45
I Topics Relevant I

I to System Development I Topic

land Evaluation Technology ! No. ABSTRACT

3.1

(1) Anticipated system lifetime.

(2) Appearance.

(3) Comfort.

(U) Convenience.

(5) Ease of operation.

(6) Familiarity.

(7) Initial cost.

(8) Maintainability.

(9) Manpower requirements.

(10) Operating cost.

(11) Reliability.

(12) Safety.

(13) Training requirements.

Common ergonomic and human factors research

dependent measures used to assess system

performance include:

. Accuracy

. Cardiovascular

response

. Critical flicker

fusion

. EEC

. Energy expenditure

. Muscle tension

. Psychophysical

thresholds

. Ratings (of

comfort,

annoyance, etc.)

. Reaction time

. Respiratory

responses

. Spare mental

capacity

. Speed

. Trials to learn

A-46
i Topics Relevant

I to System Development

land Evaluation Technology

I Topic

I No. ABSTRACT

5.3 One of the most important methodological

questions with which one has to come to

grips is how can measures like these be

matched to the system criteria? One needs

to concentrate on finding combinations of

experimental variables, and proper weights

to assign to them, to arrive at an overall

index of what is relevant and important.

i

; S

A-47
Chasteen, C.L. IOT&E of an AWADS radar imagery recorder (MAC Project

74C-110U). Eglin AFB, FL: Military Airlift Command. Operating Location F, May

197r. (AD B003 562).

Topics Relevant

I to System Development |Topic I

land Evaluation Technology ! No. I ABSTRACT

State of the Art ftoview

of the Process

1.1 General System

Measurements

1.2 System TaxonoiEy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

Z.k General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.^ Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recomnendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.3 This report addresses the evaluation of a

radar imagery recorder.

2.1 The equipment evaluated was an adverse

weather aerial delivery system radar

imagery recorder installed in a C-130

aircraft.

2.2 The primary purpose of the evaluation was

to determine the operational effectiveness

and suitability of the prototype recorder

system for use in navigator ground

training, radar prediction, and

reconnaissance/intelligence gathering.

3.1 Specific objectives were to determine

whether the system provides imagery of such

quality that radar returns are readily

identifiable, whether the system can be

installed in the aircraft without hindering

operations, and the operational suitability

of the navigator-operated controls.

Additional objectives were to determine

whether the system can record all range

marks, leading marks, and cursors; whetht

the system is capable of operating at

altitudes up to and including 25,000 feet

above mean sea level with the aircraft

pressurized and unpressurized.

4.3 An aircraft simulator was designed and used

by the engineers to exercise the prototype

system. The system was installed in a

field training detachment Adverse Weather

Aerial Delivery System (AWADS) simulator to

check compatibility with the AWADS. ihe

C-130 AWADS equipped aircraft was used as a

test bed for the prototype system.

A-48
I Topics Relevant !

I to System Development iTopic I

and Evaluation Technology ! No. ! ABSTRACT

(

4.4 Ten navigators, three of whom were

qualified weapons and tactics officers,

participated in the study.

4.5 Six missions of eleven sorties were flown

under controlled test conditions; known

checkpoints and offset aiming points were

used by the navigators to provide

independent evaluation of the system. Each

sortie was flown at preselected altitudes

ranging from 500 feet Above Ground L-vel

(AGL) to 25,000 feet Mean Sea Level (MSL).

Routes were preselected and enroute

position coordinates were recorded on data

forms along with intensity/gain used. A

camera/periscope assembly recorded the

radar display and auxiliary data throughout

the mission. Debriefing meetings, attended

by various specialists, included review and

analysis of the recorded imagery; comments

and recommendations were solicited from the

attendee relative to his area of expertise.

Questionnaires were also completed by the

navigators who participated in the study.

•

•

5.4 It was concluded that the prototype system

is operationally effective and suitable for

use in navigator training and radar

prediction and has limited capability as a

reconnaissance/intelligence gathering

device. Specific recommendations for

improvement of the system were made.

.-. v

A-49
Chop, A. Capability measures for system effectiveness (RADC-TR-72-26, Final

Tech. Rep). Sunnyvale, CA: Lockheed Missiles and Space Co., February 1972.

(AD-892 863).

tl

Topics Relevant

to System Development I Topic I

and Evalua .on Technology j Wo. ABSTRACT

1. State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.'! Performance

Criteria, Specific

3.5 Measurement Procedures

4. Planning Components

of the Process

4.1 Analytic Methods

1.2 Parameter Determinations

0.3 Apparatus for Testing

I.M Personnel for Testing

4.5 Test Plans

5- Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

1.1 System effectiveness is based on a

quantitative measure of the extent to which

the system is expected to meet its assigned

role in a specific mission. The measure is

dependent upon system parameters of

availability, dependability and capability.

The capability parameter measure the

ability of a system to achieve specific

mission objectives, given that the system

is in a particular operating condition.

2.2 A system's required overall capability is

directly related to its set of defined

mission objectives.

2.5 System capability is a focal parameter in

that it is the top performance parameter of

a system against which all other parameters

are funneled, evaluated, cross-traded and

optimized. It provides the linkup of

system performance with mission objectives.

3.2 The most practical and realistic frame of

reference for categorization of capability

measures is by specific and discrete types

of major missions assigned to Air Force

squadrons, wings, or unit equipment. A

logical initial refinement of that broad

categorization is to drop down and recast

the measures by force type. The final

refinement is to group the forces by common

force missions and stratify the resultant

force missions by discrete types if major

missions.

6. Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-50
:

Topics Relevant

i to System Development I Topic

land Evaluation Technology ! Wo. ABSTRACT

The report presents a compendium of

capability measures for each of the

different types of stratified force

missions. F«urteen (1U) stratified

missions are listed, for which a total of

forty (40) capability measures are defined.

5.2 Because overall performance response of the

system with time will inherently fluctuate,

the capability measures require use of

statistical methods based on probability

distribution laws or use of empirical

methods for their evaluation.

i

.--

•'. A-51
Churchman, C.W. Systems analysis and organization theory: A critique

(Internal Working Paper No. 3). Berkeley, CA: University of California,

Space Sciences Laboratory, June 1971.

Topics Relevant

\ to System Development I Topic!

and Evaluation Technology ! No. I ABSTRACT

State of the Art Review

of the Process

1.1 General Systes

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.I Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

<t.2 Parameter Determinations

1.3 Apparatus for Testing

H.H Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Frioritiea

6.3 Research Planning

1.1 This paper describes, in five basic points,

the normative technique approach to the

study of organizations.

(1) Organizations are goal oriented and

the goal structure can be translated

into a "measure of performance" such

as profitability, benefit minus cost,

social utility, etc.

(2) Organizations can be subdivided into

components which themselves have

sub-goals. However, these sub-goals

must necessarily be in partial

conflict.

(3) In order to be feasible, it is

necessary to set boundaries of the

system so that analysis can proceed in

an orderly fashion. The boundaries

are set by identifying a decision

maker.

(U) The systems analyst's task is to

Identify one or more important

problems of the decision maker and to

formulate the problems so that they

can be expressed in terms of a model.

The model must be rich enough to lay

out the alternatives available to the

decision maker and to enable the

systems analyst to estimate optional

solutions.

(5) The systems analyst should have an

active role in Implementing his

"solutions."

A-52
'K

I Topics Relevant

! to System Development I Topic

land Evaluation Technology | Wo. ABSTRACT

1 _

2.1 A system is defined as that which a

decision maker can control and change.

2.3 The environment of a system is thf set of

things which the decision maker cannot

control but which nevertheless affect the

performance of the system.

•

A-53 ! ^
City of Reading. Systems analysis completion report. Vol. 2; Systems

analysis methodology (USAC-RPAO-005). Reading. PA: City of Reading,

September 1971. TPS-ZOS 500-2).

1 Topics Relevant

I to System Development

land Evaluation Technology ABSTRACT

1. State of the Art Review

of the Process

1.1 General System

Measurements

I.I System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

it. Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

5- Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoonendations

1.1 This volume of the study describes the

system methodology employed by the City of

Reading, Pennsylvania for urban information

systems.

2.5 The objectives of the system are:

(1) A system design that is transferable

to other Municipalities.

(2) Production of fact and methodology for

rationalizing the Information flow of

the Municipality.

(3) Definition of horizontal loops.

(4) Developing a regional environment for

sharing system operation.

(5) Laying groundwork for improvement.

(6) Consideration of future linking with

the state-wide information system.

(7) Utilization of other on-going Federal

projects.

(8) Taking advantage of existing

Information system technology.

(9) Viewing the city as a "natural

information system" with inputs,

processes, and outputs.

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-54
I Topics Relevant

! to System Development iTopic

land Evaluation Technology | No. ABSTRACT

,

(10) Scheduling to utilize project funds

appropriately.

3.5 The system analysis process was divided

into eight phases:

(1) Organization of project.

(2) Research - literature, field

trip/technology, studies, workshops,

(3) Survey and data collection.

(1) Synthesis and coarse analysis.

(5) Data conversion.

(6) Detailed analysis.

(7) Report preparation.

(8) Reanalysis.

4.1 The methods associated with the above

phases used the following:

(1) Technology studies.

(2) Policy guidelines, application

inventories.

(3) Annual reports from departments,

summaries of codes and ordnances.

(4) Flow charts of current systems.

(5) Reading analysis technique/event

matrices, decision flow charts.

(6) Statistical tables.

A-55
Clovis, E.R. 4 rtuller, T.H. Development of procedures for evaluating unit

performance {TRA-75/009, Final Rep. Vol. 1). Monterey, CA: Litton

Mellonics Defense Sciences Laboratories, March 1975.

Topics Relevant I

! to System Development I Topic

land Evaluation Technology I No. ABSTRACT

State of the Art Review 1 .J

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM) «> . I

1.3 Overall Conceptual

..'ocess Model (CPM)

Contextual Components 2 .2

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Enviromaent Definition

2.1 General Constraints 2 .5

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria. Ultimate 3 .1

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.^ Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

>l.2 Parameter Determinations

l*.3 Apparatus for Testing

H.H Personnel for Testing

1.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.1 Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

3.4

3.5

A Unit Performance Assessment Model (UPAM)

was designed to evaluate simulated 2-sided

actions.

The systems addressed are rifle squads,

rifle platoons, and tank platoons.

The missions are selected kinds of

engagements (e.g., attacking, defending)

with the enemy for each system.

The ultimate requirement is to destroy the

enemy's ability to wage war.

The effectiveness measures include

capturing, immobilizing or defending an

objective in a given time and at a given

cost. In the study example, 20 cost and

achievement measures were selected by

military experts who rated a larger number

of objective, quantitative, face-valid

measures.

Performance criteria are established by

having experts rate the significance of the

various cost and achievement measures,

calculating and applying weights to those

measures, and combining the set into a

single performance criterion. An appendix

provides a step by step procedure for

setting these criteria.

Measurements are made through the use of

simulated engagements and expert estimates.

A-56
'

I Topics Relevant

i to System Development I Topic!

land Evaluation Technology I Ho. ! ABSTRACT

4.1 Weighted variables are developed by having

experts rank each measure, and performing a

statistical regression on the set of

rankings. Another multiple regression

procedure is used to combine these measures

into a single index of performance for

comparison with pre-set criteria.

An appendix (K) provides a step by step

procedure for calculating actual

achievement and cost scores, and computing

the performance index.

6.1 To deal with the limitations *i this study,

a cross validation effort is recommended to

test the efficiency of the regression

equations used in calculating the index of

performance. Also situational exercises

should be used to validate the UPAM, and to

provide practical application guidelines.

A-57
I

Coburn, R. Human engineering guide to ship system development (NELC

Tech. Doc. 278). San Diego, CA: Naval Electronics Laboratory Center,

October 1973. (AD-772 535).

I Topics Relevant

I to System Development I Topic

land Evaluation Technology ! No. ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System DeMr.'tion

2.2 Mission Definition

2.3 Environment Definition

Z.ii General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

HMMFM

3.3 Performance

Requirements, Specific

3 H Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Compontnts

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoonendatlons

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 This document was prepared to assist Navy

and contractor personnel in planning,

managing and carrying out human engineering

programs to support development of ship

systems. Very little attention is devoted

to issues concerning measurement of system

performance or effectiveness.

Human engineering services and end products

relating to assessment of system

performance include:

(1) Man-Machine Concept Analyses —

Prediction of man-related aspects of

system performance for candidate or

selected system configurations.

(2) Man-Machine System Design —

Establishment of performance

specifications which set bounds on

man-machine system performance and

define what the system must do in

operational terms.

A-58
Cogan, E.A. If it exists, it can be measured - but how? In E.A, Cogan &

J.D. Lyons,Frameworks for measurement and quality control

(HumRRO-PP-16-72). Papers presented at New York University First National

Annual Training in Business and Industry Conference, New York City, March

1972. (AD-748 081).

Topics Relevant

! to System Development

land Evaluation Technology ABSTRACT

I M

.••

State of the Art Review

of the Process

t General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Mod I (CPM)

Contextual C aponents

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultlirate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

S-1» Performance

Criteria, Specific

3-5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.-.. Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recomendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potential»/

Priorities

6.3 Research Planning

1.1 In selecting or devising a measurement, it

is essential to decide or determine the

purpose of the measurement. In industry

the purpose translates to decisions that

management or personnel people must make.

The second element in defining measurement

concerns what is to be measured. In Job

performance evaluation there are several

categories of tests available, each

measuring different things:

(1) Natural observation.

(2) Job sample tests.

(3) Analytic tests.

(i«) Indirect testi.

(5) Rating scales.

Measurement effectiveness or validity is

best described by what one should consider

in dealing with it. These considerations

are in the form of the following questions:

(1) Accuracy - What are the tolerances of

the emerging numbers?

(2) Stability - If one retested later, how

similar would the measurement numbers

be to the first set?

(3) Pay-off - How much better are the

decisions reached using measurement

than those reached without such

Information?

A-59
Companion, M. A. and Corso, G. M. Task taxonomy: Two Ignored Issues. In A.

S. Neal 4 R. F. Palasek (Eds.), Proceedings of the Human Factors Society 21st

Annual Meeting. San Francisco, October 17-20, 1977.

I • Topics Relevant

I to System Development

land Evaluation Technology ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurement:

I.I System Taxonomy

Model (SIM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1» General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measure.'

3.3 Performance

Requirements, Specific

3.'! Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

IJ.3 Apparatus for Testing

M.I Personnel for Testing

1.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5-3 Findings Interpretation

5.k Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 One of the problems encountered in the

definition of a task can be linked to the

problems associated with the system level

to which the taxonomy is being applied.

Another problem frequently encountered in

defining a task is deciding who should

define the task, the investigator or the

operator: what the scientist says a person

i" doing may or may not conform to what the

person thinks he is doing.

1.2 Two issues often ignored when developing a

task taxonomy are (1) a set of criteria,

i.e., rules on which a Judgment can be

based for the evaluation of how well a task

taxonomy accomplishes the goals underlying

its development and (2) the relation

between taxenomic structure and empirical

data, i.e., laboratory and field data. An

efroctive task taxonomy should include the

following criteria:

(1) The taxonomy must simplify the

description of tasks in the system

oe^riuse the goal of any taxonomic

soVli5n?if is to make the subject matter

more manageable.

(2) The tc 'onomy should be generalizable.

If the taxonomy is system specific,

the effort necessary to develop it

might outweigh the benefit derived.

A-60
V

.••

^ .^

1 Topics Relevant 1

1 to System Development I Topic

land Evaluation Technology | No. ABSTRACT

Also, generalizability is congruent

with the necessary assumption that

activities have some common basis.

(3) The taxonomy must be compatible with

the terras used by others. Unless the

taxonomy is in a form that is

meaningful to those who will use it,

its application will be inefficient

and often ignored.

(4) The taxonomy raust be complete and

internally consistent, i.e., it raust

deal with all aspects of human

performance in the system without

logical error.

(5) The taxonomy must be compatible with

the theory or system to which it will

be applied.

(6) The taxonomy should help to predict

operator performance. This is

necessary to evaluate and compare

performance between operators on

different as well as identical tasks.

(7) The taxonomy must have some utility,

either practical or theoretical.

(8) The taxonomy must be cost effective.

It is possible that in many situations

the time and money required to develop

and implement a task taxonomy may add

to the overall cost of the system and

provide little increase in operating

efficiency.

(9) The taxonomy must provide a framework

around which all relevant data can be

integrated. Without this the taxonomy

is merely a verbal device with no ties

to reality and, therefore, no

applicability.

_ •

A-61
Companion, M. & Teichner, W. H. Application of task theory to task analysis;

Evaluation of validity and reliability using simple tasks (AFOSR-TR-77-1008).

Las Cruces, NM: New Mexico State Uiiversity, Human Performance Laboratory,

January 1977. (AD-A0n3 243).

Topics Relevant i I

I to System Development I Topic!

and Evaluation Technology 1 No. i ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STO)

1.3 Overall Conceptual

Process Model (CPU)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoonendations

1.3 The purpose of this study was to determine

whether inexperienced people can be trained

to apply Teichner's Theoretical Task

Concepts.

2.1 This research was intended to provide a

first evaluation of the reliability and

validity of a task analysis performed with

respect to Teichner's Theoretical Task

Taxonomy and instructional procedures for

training Individuals to perform the

analysis.

4.5 Problems performed on desk and pocket

calculators were developed so as to

represent theoretical tasks. Ten subjects

were instructed in the theoretical

concepts, and were then provided a partial

operational analysis of the task problem.

They were then required to complete the

operational task analysis and to transform

it into a theoretical task analysis.

5.2 Using the built-in operational and

theoretical steps as references, the

validity of the subject's procedures was

evaluated in terms of how closely the

analysis agreed with the references.

5.3 It appears that, with very little training,

people can comprehend the concepts and be

at least as proficient in the theoretical

analysis as they are at describing actual

operations. Considering that, and the

general level of performance, it is

concluded that the practicality of the

Further Research Areas

6.1 Hea*-rement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-62
I Topics Relevant !

I to System Development I Topic

!and Evaluation Technology I No. ABSTRACT

approach is supported, i.e., operational

task descriptions or task analysis, can be

translated correctly into the tasks of the

theory by uinimally trained observers.

* 6.2 It is suggested that this approach should

be extended to the evaluation of more

complex tasks.

»

•

-

A-63
Coiinelly, E.H., Bourne, F.J., Loental, D.G. & Knoop, P.A. Computer-aided

techniques for providing operator performance measures (AFHRL-TR-7^-87).

McLean, VA: Quest Research Corp., December 1971*. (AD-AC14 330).

Topics Relevant

I to System Development

land Evaluation Technology

r

Topic!

No. | ABSTRACT

State of the Art Review

of the Process

1.1 General System

MeasurementE

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.4 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 The problem was to develop and implement

standardized techniques for deriving and

validating measures of operator

performance. Traditional techniques involve

hand-selecting measures which appear to

have content validity, then testing the

measures against other validation criteria

using operator performance data. This

usually results in a resource-consuming

iterative research process that is often

unsuccessful, because:

(1) It is never known at the onset whether

or not the most useful measures have

been overlooked.

(2) The number and potential validity of

measures investigated are limited by

and vary with the researcher's

ingenuity and the time available for

the study.

(3) The research process and all

associated manual effort must be

repeated for each new measurement

task.

1.3 The approach was to develop and implement

computer-aided techniques for deriving and

validating operator performance measures.

A "universal" set of potential measures was

defined which possesses characteristics

encompassing many traditionally selected

measures. The set also inherently contains

a myriad of other measures whose

characteristics render them reasonable

candidates. Vectors were then identified

which constitute generators for the set of

measures (i.e., the vectors span the

defined measure apace). Computational

A-64
I Topics Relevant I

I to System Development I Topic I

and Evaluation Technology j No. I ABSTRACT

t

:

i

algorithms were developed which generate

and operate on the constituent vectors

using multiple regression techniques.

Several empirical validation methods were

developed for testing candidate measures

thereby generated. All techniques were

implemented in a computer-aided measurement

processor which: (1) accepts sample

performance data and various user inputs,

and (2) generates and tests candidate

measures, computes statistics for assessing

their validity likelihood, and prints

results for user analysis.

2.1 The sample system used for measurement was

a T-37B aircraft in flight maneuvers.

2.2 Five specific flight maneuvers were flown:

(1) Cloverleaf.

(2) Split S.

(3) Lazy 8.

(M) Normal Landing.

(5) Barrel Roll.

3.1 Measurable attributes for the flight

maneuver sample were pitch, roll, heading,

maneuver sector, airspeed.

3.2 The measures were in degrees for pitch,

roll and heading.

4.1 Regression analysis was used to generate

reference functions which are representative of excellent performance.

4.3 The test apparatus consisted of an

instrumented T-37B aircraft.

U.M Test subjects were instructor pilots who

purposely demonstrated both good and bad

maneuver performances.

A-65
! Topics Relevant I

! to System Development 1 Topic

land Evaluation Technology ! No. ABSTRACT

5.2 The developed measurement processor was

successfully implemented on a Sigma 5

computer. Demonstrations of the operation

of the software were performed using a

liuited amount of pilot performance data

recorded on a T-37B aircraft. The

processor performed necessdry data

smoothing, automatically segmented the

flight maneuvers for measurement, and

developed criterion functions from the

skilled operator data provided. Actual

generation and validation of measures was

not demonstrable due to nonavailability of

originally anticipated data. However,

correct software performance of all parts

of the processor was verified.

S.l The theoretical concepts and computational

techniques underlying the developed

measurement processor are unique and have

great potential for operator performance

measurement research. The applied concept

of developing a set of vectors which span a

conceived measure space and operating on it

with regression techniques to generate

candidate measures is itself suggestive of

a new and extremely powerful measurement

tool. The processor operation can be

largely independent of user intervention;

however, it is also capable of accepting

user inputs reflecting his knowledge about

specific measurement problems. It

represents a truly interactive research

system wherein user tasks as distinguished

from processor tasks are logically defined,

and the outcomes of each are integrated.

Evaluation of the adequacy of the spanned

measure set, the generating vectors, and

the computational mechanics for generating

and testing measures could not be performed

as originally planned due to non-technical

problems which prevented the collection of

required data. This was extremely

detrimental to the study because: (1) many

of the techniques could not even receive

preliminary test prior to their

A-66
4

I Topics Relevant

\ to System Development 1 Topic I

and Evaluation Technology i No. ! ABSTRACT

incorporation in the processor, and (2) the

contributions made by this study to the

general technology can only be suggested

instead of exemplified.

6.1 Follow-up research should include

derivation of the basis of the defined

measure set using the implemented processor

as an aid to empirical studies. This is,

in essence, the real crux of the operator

performance measurement problem.

. .

A-67
Connelly, E.M., Bourne, F.J., Loental, D.G., Migliaccio, J.S., Burchick, D.A.

& Knoop, P.A. Candidate T-3" pilot performance measures for five contact

maneuvers (AFHRL-TR-74-88).

(AD-A014 331).

McLean, VA: Quest Research Corp., December IB!1*.

Topics Relevant |

I to System Development I Topic

jand Evaluation Technology ! No. ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

1.2 Parameter Determinations

1.3 Apparatus for Testing

1.1 Personnel for Testing

1.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.1 Conclusions and

Recoonendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

2.1 The purpose of this study was to develop

candidate T-37 pilot performance measures,

for ultimate use in an advanced simulator.

2.2 Five undergraduate pilot training contact

maneuvers were selected for this

development: lazy 8, barrel roll, split S,

cloverleaf and landing.

3.1 The first step of the approach was to

analyze each maneuver, using techniques

from function and task analyses, in order

to identify candidate measures.

3.2 Several algorithmic measures were defined

which, collectively, support performance

assessment over all maneuver segments.

Content validity was then assured, and the

user specified measures were then computed

using specially developed software.

Empirical testing was to be conducted next

to establish criterion-related validity.

However, non-technical problems prevented

the data collection phase from being

completed.

4.1 Specific measurement formula combiner the

various measure types such as: continuous

difference error measures (used to identify

the deviation from a trajectory), threshold

error measures (used to identify violation

of a threshold), differential difference

error measures (used for time dependent

A-68
I Topics Relevant

i to System Development I Topic

land Evaluation Technology | No. ABSTRACT

changes during a maneuver), discrete task

error measures (used to identify the

accomplishment of a discrete event), sample

error measures (used in relation to a

series of measures during a specific

maneuver), and miscellaneous error

measures.

M.3 Computer programs were developed to:

(1) Smooth, print out, and plot data

recorded on-board a T-37B aircraft.

(2) Automatically detect task segment

boundaries.

(3) Compute criterion functions from

skilled performer's data.

(4) Compute measures specified at run-time

by the user.

(5) Perform and print results of several

empirical validation tests of the

candidate measures for subsequent

researcher analysis.

A-69
Connelly, E.M., Schuler, A.R. 4 Knoop, P.A. Study of adaptive mathematical

models for deriving automated pilot performance measurement techniques

{AFHRL-TR-69-7, vol. 1: Model Development). Falls Church, VA: Melpar,

October 1969. (AD-704 597).

Topics Relevant

I to System Development

land Evaluation Technology

1. State of the An Review

of the Process

1.1 General System

Measur'ments

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.H General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

4. Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

6. Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Prloritiea

6.3 Research Planning

ABSTRACT

1.3

2.3

Present methods of (trainee performance)

measurement consist almost exclusively of

ratings or Judgments made by skilled pilot

instructors/examiners. Automatic performance measurement requires measures and

criteria in precise, quantitative terms in

lieu of the more qualitative terms familiar

to human evaluators.

The approach used in this study is quite

different (from most performance

measurement research). Instead of first

deriving measures and criteria explicity

and molding them into an evaluation system,

the authors try first to derive a reliable

performance evaluation system and then

analyze the system to determine performance

measures and criteria required fo»- valid

and reliable performance measurement.

The evaluation system is derived by

developing and using adaptive mathematical

and computer models operating on

representative performance data

corresponding to known skill levels. The

models automatically adjust themselves and

their methods of data analysis until

capable of independently evaluating

performance. Development of this system of

adaptive models is the subject of this

report.

Automated performance evaluation is basic

to the concept of adaptive training. The

problem is to develop valid and reliable

pilot performance measures and criteria for

use in automatically and objectively

evaluating trainee performance (on a flight

simulator).

A-70
Topics Relevant

to System Development I Topic',

and Evaluation Technology j No. | ABSTRACT

2.U The characteristics of the adaptive

modeling problem are as follows:

(1) The requirement is to simulate/automate the results or effect the trainee

performance assessment process, not

necessarily to faithfully simulate the

way in which a human would execute the

process.

(2) The performance assessment process, as

performed by a humar, is not sufficiently well defined or understood to

allow its direct implementation on a

computer.

(3) Information does exist about the data

required by a human in order for him

P to perform the process.

(4) Some skilled humans perform the

process very well.

y 2.5 The approach described in the report

ß involves development of an adaptively -

programmed computer model which performs

the following functions iteratively:

(1) The program examines actual data,

consisting of pilot performance data

on some flight maneuver or task; many

sets of representative data are

needed.

(2) The program hypothesizes/approximates

the required "process-effect," i.e., a

score or rating for the pilot whose

data has Just been examined; this

requires a method of predicting a

score from actual flight data. The

study developed three computational

methods for doing this, each of which

attempts to predict score or a portion

thereof in a unique way.

A-71
i

I Topics Relevant

I to System Development iTopic

land Evaluation Technology j No. ABSTRACT

-

(3) The program compares its own derived

score with the "true" pilot score or

rating that had been produced for the

set of data by skilled human Judgment.

(4) As necessary, the program adapts

itself and its methods of data

analysis to improve its previous

approximation.

(5) As long as the program's derived

ratings do not correlate sufficiently

closely with the experts' ratings, the

program repeats the preceding steps

with a new set of data and continues

to adapt itself. But, once a

sufficiently close correlation is

achieved, the program shifts to a

"testing" mode, wherein it ceases

adaptation and produces scores for new

sets of data for purposes of

evaluation.

3.2 The three computational methods used by the

program to produce pilot scores/ratings are:

(1) The state transfer technique.

(2) The relative technique.

(3) Absolute techniques.

State transfer technique is based on the

assumption that performance skill may be

partly predictable by examining trends in

the performance data. Relative technique

assumes that performance skill may be

partly predictable by examining

relationships among separate variables in

the performance data. Absolute technique

assumes performance skill may be partly

predictable by examining specific

characteristics of performance as compared

with some absolute reference-performance.

A-72
I Topics Relevant

i to System Development I Topic

land Evaluation Technology | Ho. ABSTRACT

4.1 The results prodjced by each of the

computational methods consist of components

of the performance evaluation. These

components are routed through an

"adaptive-mix," program which computes a

single composite performance score.

5.4 A preliminary demonstration and evaluation

of this adaptive modeling system produced

encouraging results.

A-73
Connolly, E. M. & Sloan, N. A. Manned system design using operator measures

and criteria. In A. S. Neal 4 R. F. Palasek (Eds.), Proceedings of the Human

Factors Society 2l3t Annual Meeting. San Francisco, Octooer 17-20, 1977.

Topics Relevant

I to System Development

land Evaluation Technology

Topic

No. ABSTRACT

1. State oT the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Con extual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environaent Definition

2.1* General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, ultimate

3- Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.^ Performance

Criteria, Specific

3.5 Measurement Procedures

4. Planning Components

of the Process

4.1 Analytic Methods

<t.2 Parameter Determinations

1.3 Apparatus for Testing

*.*» Personnel for Testing

1.5 Test Plans

5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

6. Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 The designer of a man-machine system

typically performs his design task with

knowledge of system objectives, human

factors principles, and display and control

requirements. However, it is the human

operator who adapts his control rule (his

input/output control characteristics) so

that the overall system responses satisfied

(to the degree possible) his performance

criteria. The performance that is actually

achieved will be obtained in cooperation

with a system that has a good system design

- and in spite of a system having a poor

design.

The above argument suggests that the

designer should have available as a design

tool a means for estimating the operator's

performance criteria and his control

actions. The designer would like to know

which design features support performance

and which features degrade performance.

3.4 The distance between the intial and final

positions is approximately 30 miles and the

time allotted for the transit is 90

minutes.

M.3 The equipment used in the present

investigation included a surface ship

bridge console system and a CRT.

4.5 In this study, s-bjects acting as Officers

of the Deck (OOD) controlled a simulated

ship in a simulated environment. Their

task was to divert a ship transit from the

initial point to the terminal point within

a pre-specified time interval while

avoiding simulated contacts along the way.

A-74
Connelly, E.M., Zeskind, R.M., and Chubb, G.P. Development of a Continuous

Performance Measure for manual control (AMRI-TR-To-^). Vienna, VA: Omnemii,

Inc., April 1977. (AD-A041 676).

Topics Relevant I

I to System Development I Topic

land Evaluation Technology ! No. ABSTRACT

State of the Art Review

of the Process

1.1 General System

Mearurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.<i General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1* Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Component«

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Reconanendations

Further Research Areas

6.1 Measurement Systen

Limitations

6.2 Research Potent^Is/

Priorities

6.3 Research Planning

1.3 Past performance measurement of manual

control systems has used a single summary

measure to indicate performance of the total

control problem. While summary measures

give the total picture, they do not indicate

problems that arise during the control

problems.

2.2 If performance is monitored continuously

throughout the control problem, each control

action, discrete or continuous, can be

individually evaluated. The Continuous

Performance Measure (CPM) could be used to

increase the efficiency of experiments,

training, and design of manual control

systems. This report documented research

that deals with development of CPM for

flight control systems.

3.1 The desired CPM will provide the correct

motion indication of the aircraft at each

point of the mission segment, thus providing

a flight standard against which actual

flight performance can be compared.

Moreover, motion errors can be evaluated in

terms of their significance with respect to

the summary performance measure in question.

Optimal control theory can determine eacu

aircraft state in the mission segment ana

the optimal control and solution trajectories. The term "optimal solution trajectories" is equivalent to the aircraft

motion trajectories that minimize the

performance measure selected for that

particular mission segment. The term

"aircraft state" comprises those state

A-75
I Topics Relevant I

I to System Development ITopicl

land Evaluation Technology ! No. I ABSTRACT

■■ variables that describe the values for all

aircraft variables involving position and

rotation, including velocity.

3.2 Several model components were constructed in

order to determine what is "optimal":

(1) A representation of the current task

objectives arA weights to determine

their importance

(2) A system representation of what is to

be controlled, and

(3) A representation of limiting or

constraining factors on the system and

its controls.

3.3 A mission can be segmented so that flight

variables may be specified for each

segment. The total mission can be regarded

as a series of segments where the endflight conditions of one segment are the

beginning conditions for the following

segment. A Segmented Mission Model can

thus be constructed if mission segment

specifications are converted to a summary

measure, including flight constraints.

3.4 In order to convert mission specifications

into a summary measure, a cost index

function, or "penalty function," was

constructed which identifies:

(1) Deviation from the desired end state,

and

4.2

(2) Variable rates of change, control

actions, and deviations from reference

trajectories occurring along the

solution path.

The particular mission, in fact, identified

the parameter of an aircraft operating in a

Close Air Support Night Attack mode.

A-76
1 Topics Relevant

I to System Development I Topic I

iand Evaluation Technology ! No. I ABSTRACT

5.4 This look into CPM showed that summary

measures developed from mission segment

specifications can be converted into

performance measures, this was done by

using optimal control theory to either

linearize the aircraft equations, or

solving the optimal control for the

non-linear aircraft equations.

.•

A-77
Connelly, M.N. & Willis, J.E. The development of a Human Effectiveness

Function Allocation Methodology (HEFAM) (Research Memo. SRM 70-11). San

Diego, CA: Navy Personnel and Training Research Laboratory, October 1969.

(AD-699 173).

Topics Relevant I

I to System Development jTopic

land Evaluation Technology 1 No. ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measuremerts

1.2 System Taxonomy

Model (STM)

1.3 Overail Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 This report describes an effort to develop

a workable cost/effectiveness methodology

for man/machine function allocation. The

project consisted primarily of a literature

review and interviews with numerous

technical personnel working in the areas of

hunan factors and personnel research.

The Human Effectiveness Function Allocation

Methodology (HEFAM) was conceived as an

automated data storage and processing

system to be used many times during the

development cycle of new weapon and support

systems.

The research on HEFAM was directed toward

three major areas of system development:

(1) Initial Data Collection.

(a) Development of data source.

(b) Development of data collection

techniques.

(2) Effectivenesr. Quantification.

(a) Development of conceptual basis

for human effectiveness

quantification.

(b) Development of formulae for human

effectiveness quantification.

(c) Development of preliminary

methodology for human

effectiveness prediction.

A-78
i Topics Relevant

I to System Development I Topic 1

land Evaluation Technology I Ho. i ABSTRACT

(3) Overall Conceptualization of the HEFAM

System.

(a) Data bank system.

(b) HEFAM - user interface.

(c) Relationships between th HEFAM

and other data banks.

5.3 The findings of the study are as follows:

(1) There has been much effort devoted to

the problem of improving human

reliability and collecting human

reliability data, however, there has

been very little conceptualization of

the overall problem of quantifying

human effectiveness, especially in the

area of human effectiveness

prediction.

(2) There is a lack of development of the

state-of-the-art of quantifying human

performance effectiveness.

(3) Mathematical models for HEFAM were

determined, computer storage capacity

estimated and data collection methods

proposed.

(4) The type of data required and the

manner in which it is to be stored

were stated.

5.U The study's conclusions are as follows:

(1) Much more work effort and time will be

needed to develop the HEFAM system.

(2) The HEFAM system will consist of a

methodology, computational formulae,

and an automated data processing

system.

A-79
u

I Topics Relevant I

! to System Development !Topic

land Evaluation Technology ! No. ABSTRACT

(3) More sources of data are needed in

order to provide a large enough sample

of human performance to form a data

base for effectiveness predictions.

CO Simpler methods of data collection

than the Operational Sequence Diagram

(OSD) method are needed in order to

collect human performance data in a

timely manner.

(5) Computational and predictive formulae

for HEFAM must be developed further in

order to be used for actual

computation or prediction.

The following recommendations are made:

(1) It is recommended that the HEFAM

system be developed as rapidly as

possible. This will require more

effort than is currently being

expended.

(2) The future development of HEFAM should

include: the further development of

the conceptual bases of human

effectiveness quantification; the

further development of the HEFAM data

processing system; the collection and

utilization of human performance data;

the development of better data

collection methods; the further

development and testing of

computational and prediction formulae;

and the development of the HEFAM

prediction methodology.

A-80
Cunningham, J.B. A systems-resource approach for evaluating organizational

effectiveness. Human Relations. 1978, 31(7). 631-656.

I Topics Relevant I 1

to System Development I Topic I

Sand Evaluation Technology ! No. I ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.ii General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3-3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

4.2 Parameter Determinations

1.3 Apparatus for Testing

1.1 Personnel for Testing

1.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.1 Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.2 The basic systems-model, which is applied

by the author to evaluate organizational

effectiveness, deals primarily with

sub-system interrelationships. Basic to

the systems model is an analysis of

environmental inputs, methods by which the

inputs are transformed (throughputs), and

the end-products of this transformation

(outputs).

2.1 Characteristics of the systems-model are

physical and chemical laws that are

applicable to social organizations as

follows:

(1) Every system uses energy in a cyclical

way: the environmental product or

output becomes the energy source for

the subsequent activity cycle.

(2) Systems are separated from their

environments by boundaries; since

events are structured in a systematic

way in an organization, the boundaries

of the system are between events.

(3) Equifinality in open systems: a final

or specific end state can be reachad

by a diversity of inputs and varying

environmental and internal activities.

(4) Entropy: in nature, all organized

systems "wind down" or move toward

disorganization and/or death - this is

the second law of thermodynamics; in

open systems, however, negative

entropy allows the system to

A-81
Topics Relevant I

to System Development ITopicl

and Evaluation Technology ! No. ! ABSTRACT

k-'

temporarily circumvent entropy by

importing more environmental energy

than it expends.

(5) Equilibrium, or dynamic homeostasis:

systems adapt to change and attempt to

maintain a balance in their status

quo; the system will also attempt to

acquire a margin of safety in inputs

above and beyond what it needs for

mere survival.

(6) Feedback: an information input into

the system resulting from previous

outputs and their effect on the

system's environment.

2.2 The primary goal of the application of the

systems model was to apply it to real-world

modeling of organizational effectiveness:

specifically, seven local organizations

(city and county governments) in southern

California were evaluated.

.,

The effectiveness of these organizations

was studied in terms of several factors:

(1) The organization's ability to respond

to its external environment.

(2) The organization's ability to utilize

resources in producing outputs and

maintenance/restoration of the system.

(3) The organization's ability to bargain

and optimize its use of resources in

an environment with multiple

decision-makers, each with different

goals.

3.3 With regard to the first factor (responding

to the external environment), four

different computer simulation problems were

designed - - an airplane crash, an

earthquake, a flood, and ordinary

day-to-day problems - - which provided a

moans to assess the adequacy of various

organizational resources for coping with

>."-

A-82
u

i

I Topics Relevant I

1 to System Development I Topic I

!and Evaluation Technology ! No. I ABSTRACT

these disasters of various magnitudes

(including variations in population

density, residential structures, and

knowledge-levels).

The specific requirement of the first

factor evaluation was to measure the

organization's ability to achieve the

highest resource allocation for various

levels of damage.

The organization's response to its internal

environment - - its efficiency and

bargaining power were measured in

experimental simulations of each of the

seven local government organizations.

In terms of the third factor, a specific

allocation of resources was needed for a

given organizational resolution of a

problem; the decision to allocate X was

made on the basis of a probable payoff Y.

Since a decision-maker should logically

allocate resources to problems of the

highest payoff first, each problem was

measured in terms of its demand for X

amount of resources.

3.U The actual measurement of resource

availability for each environmental

possibility was computed by dividing the

total cost of damage by each subsystem's

reserve of resources.

This factor was measured by the following:

each problem is assessed in terms of time,

location, cost, and amount of information

given to the decision-makers. Each of

three organizational resources - vehicles,

equipment, and personnel - were then given

a value or a cost as an indication of its

value to the organization. The resource

value needed for the resolution of a given

problem could then be calculated.

A-83
LV

I Topics Relevant 1 |

I to System Development [TopicI

land Evaluation Technology I No. ! ABSTRACT

The value of resources theoretically

necessary to respond to problem j is:

sop

j

1

k

(P jk

ifV

P.. is the number of resources of type k

wnich are the standard operational response

to problem j.

rr. is the value of resource k.

SOP is the Standard Operational Response.

4.1 The data for the above were collected

through:

(1) Use of a panel of experts to predict

problems and events occurring with an

organization's external environment.

(2) Prediction ot' possible environments by

computer simulations.

(3) Accounting of resource availability

for each local governmental

department.

(4) Use of a gaming simulation experiment

measuring the decision-maker's

efficiency and bargaining capability.

5.2 The decision-maker's ability to correctly

interpret the external environment was

calculated through the use of F ratios

indicating the degree of difference among

the resource ratios (resource

available/possible damage levels) of the

various organizational subsystems. These

analytic techniques were also repeated for

factors 2 and 3 (organizational efficiency

and bargaining capability).

5.3 Private organizations were found to respond

at a much higher level than the

governmental organizations studied; this,

A-84
i Topics Relevant

to System Development iTopic I

and Evaluation Technology j No. I ABSTRACT

however, does not necessarily indicate

greater efficiency: private organizations

over-rasponded as a public relations

gambit, not just in response to the

"problem". There was also greater

variation among private organization

efficiencies than there were in the public

sector.

fi

A-85
Cunningham, R.P., Sheldon, M.S. 4 Zagorsk!, H.J. Project NORM; Pilot study

report (TM-2232/000/00). Santa Monica, CA: System Development Corp.,

February 1965. IAD-458 341).

Topics Relevant

I to System Development

land Evaluation Technology

Topic I

No. i ABSTRACT

1. State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requ i rement s, Spec i fic

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

4. Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

6. Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.2 Historically, there has been much

controversy concerning the measures used to

assess performance, some of which purport

to evaluate functional units of the system,

while others deal with subsystems and still

others attempt to assess the behavior of

the total system. Littlj is known about

the relationship among the various

measurements or their relevance as criteria

for making adequate Judgments regarding

training operations. Frequently, it has

been asserted that single performance

measures are Inadequate for making overall

evaluations of system effectiveness,

presumably because multiple factors are

involved in the determination of mission

success or failure. Combining measures

into overall indices has, so far, not

seemed to help much, probably because the

relationship between them is not clearly

understood. Thus, lumping them together

does not necessarily improve the quality of

system evaluation.

2.2 Project NORM (Normative Operations

Reporting Method) was a pilot study for a

subsequent comprehensive research effort

aimed at obtaining a better understanding

of air defense mission evaluation and

analysis.

3.1 A mission problem containing 37 fakers was

tested. A computer printout was obtained

and data relevant to 41 situational,

subsystem and system performance variables

were extracted. Those variables actually

considered in the study were:

A-86
I Topics Relevant

i to System Development iTopic

land Evaluation Technology | No. ABSTRACT

M.I

5.U

(1) Detection latency.

(2) Total track off time.

(3) Conmltment latency.

CO Distance interceptor - faker at first

pairing.

(5) Faker target life.

(6) Faker distance travelled.

(7) Faker distance to nearest ground

target.

Statistical treatment of the data was made

with the aim of demonstrating how prior

knowledge of the mission situation can be

used to predict expected performance for

the purpose of gauging crew progress.

This pilot study clearly demonstrated the

feasibility of addressing the criterion

development problem in a quantitative and

scientific fashion.

A-87
Defense Science Board. Report of task force on test and gvaluation.

Washington, DC: Office of the Director of Defense Research and Engineering,

April 197*.

Topics Relevant

to System Development I Topic

and Evaluation Technology ! No. ABSTRACT

3.

5.

State of the Art Review

of the Process

1. 1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3-3 Performance

Requirements, Specific

3.4 Performance

Criteria, Specific

3-5 Measurement Procedures

Planning Components

of the Process

«». 1 Analytic Methods

4.2 Parameter Determinctions

4.3 Apparatus for Testing

H . 4 Personnel for Testing

1.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoranendations

Further Research Areas

6.1 Measurment Systee

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 This report provides guidance on test and

evaluation (T4E) at two distinct levels.

At the most general level, this report

discusses a number of issues which are

appropriate for all weapon system acquisition

programs. In addition, a general checklist

of items is presented which is organized for

a rapid overall review of T&E aspects,

generally applicable to all systems development and deployment. A sample of the recommendations is included in this abstract.

2.3 Demonstration and acceptance tests, as well

as tests intended to evaluate performance

under operational conditions, should always

be conducted under conditions as close to

those anticipated in practice as possible.

On the other hand, test conditions during

development should be determined by the

primary objectives of that test, rather than

by more general considerations of realism,

etc. Whenever a non-tactical, non-operational

configuration is dictated by test requirements, the results of the tests should not

be challenged by the fact that that configuration was not tactical or operational.

2.6 By the end of the systems definition phase,

it should be made certain that test criteria

are established so that there is no question

as to what constitutes a test and what performance is to be obtained. A relationship

between the identified performance parameters

and the test results should be established

prior to the conduct of the test. Further,

the set of objectives for each of the tests

should be clearly related to the program

objective.

A-88
I Topics Relevant

1 to System Ddevelopment

land Evaluation Technology

Topic !

No. I ABSTRACT

Each performance characteristics specified

should be measurable through bench and

laboratory or proving grounu testing. The

^ V test design and the number of tests should

I

•

!

be adequate to provide results with confidence limits compatible with tne statements of desired characteristics. Testing

in advanced development should be planned

to explore performance characteristics over

a broad range of environments so as to provide insight into system performance over

the expected operational range and not just

at a single point.

3.5 A good test program makes provisions for

feedback of test results, during conduct

of the testing, so as to influence:

(1) Course of the T&E program (test

director, program manager).

(2) Tradeoff decisions between modifying

the system design and relaxing the

operational requirements (program

manager, operating/supporting

commands, HQ).

(3) Missions, employment doctrine, tactics

and constraints, tactical organization,

etc. (operating command, operational

units).

CO Parts provisioning.

When developing, testing and evaluating the

various subsystems (and systems) of nonexpendable weapon systems, each component of

the systems should be numbered and a performance history kept which allows an analysis

of that component's performance with respect

to reliability, maintainability, availability,

etc. An analysis of failure modes should be

made in advance so as to relate test results

to the operational capability of the system

when in a degraded condition.

A-89
i Topics Relevant

I to System Development

! and Evaluation Technology ABSTRACT

Techniques and system range instrumentation

should be developed tc provide the type of

data in the proper form to allow ecommic,

analytical, and mechanical simulation for

alternate scenarios and combinations.

'1.3 If there are government furnished equipmen*

(GFE) and other government commitments in

the proposed contract, be concerned about

the following:

(1) Can the gear with required performance

be available when required?

(2) Can government supported facilities

provide the assistance required at the

time needed? If not, is it reasonable

to construct the required facilities

(test range, instrumentation, building,

etc.)? If not, what alternatives are

available?

Whenever posoible the initial phase of

operational test and evaluation (I0T&E) of

a weapon system should be planned to include

other systems which must have a technical

interface with the new system. Thus, missiles

should be tested on most of the platforms for

which they are programmed. Interfaces

between system should receive special attention,

The manner in which T&E instrumentation is

used can be extremely important in determining

the realism possible in the OT&E phases. The

instrumentation package should be fixed early

in the design phase of the development; it is

difficult and costly to change thereafter.

For this reason, instrumentation requirements must be specified early in the program

and operational factors must be incorporated

early.

The applicability of existing test ranges and

the adequacy of facilities and instrumentation should be verified.

A-90
Topics Relevant

to System Development

and Evaluation Technology

'I.M Testers, evaluators and operators have

quite different backgrounds and needs

which affect the TiE of the weapon system.

Each has a different approach which has

merit and utility at almost all points in

the T&E program. A mix of these types is

needed throughout the program. Early in

the program, the lead emphasis should be

from the tester, shifting to the evaluator

and finally the operator, but at all times

all parties and their needs should be

coordinated.

Training plans and certification plans for

test personnel should be established early

in the Full-Scale Engineering Development

Phase. Errors by test personnel are usually

expensive and often cloud the reason for

test failures.

It is imperative that the Independent Test

Agency participate in all of the T&E phases

to ensure that the user needs are represented in the development of the system

concept and hardware. Initially, the Independent Test Agency should play an advisor

role during the feasibility and engineering

testing, and gradually take over leadership

in the conduct of the testing program as it

becomes more and more operational. This

should facilitate the necessary communication

and interaction between developing and user

commands.

The test director and/or key members of the

test planning group within the project office

should have significant T&E experience. If

the requisite experience does not exist at

the appropriate levels within the project

office, test plans may be based on too shallow

or too naive a conception of the role and

potential utility of the T&E process. All

too often, key test personnel are assigned

to T&E slots with little prior exposure to

T&E or its management, and with inadequately

experienced support as well. The test planning

group should have personnel experienced in

engineering testing, development testing and

A-91
I

Topics Relevant

to System Development

and Evaluation Technology

Topic I

No. : ABSTRACT

•.5

and operational testing. This experience

should be available very early and all

efforts should be made to encourage these

people to remain with the weapon system

project office through the T&E phases of

the program.

The planners and evaluators for the OTiE

of the production equipment can do a better

job if they are initially involved in

planning and conduct the I0T4E.

In the initial conduct of 0T4E, the participants should be given a period of time to

dry run the scenarios and to shake-down the

instrumentation and the overall operation

before key resources are expended in tests

for record. In a properly planned OT&E

program, the people will have completed

proper individual training on the new system but the operational organization will

not be able to conduct full unit training

until the hardware, software, and support

equipment are on hand. After the period

when the unit is qualified as being operationally ready, it would be ready for

assignment to OT&E testing.

Test conduct can be influenced by the actions

of the observers and umpires. These people

can provide important clues to the participants

of operational suitability testing and in that

way lesson the validity of the test. For

example, in situations where air/ground duels

are to be conducted, briefed observers who

look in the direction of the aircraft, might

inadvertently tip-off the direction of approach

to the ground party in the duel. Similarly,

concentrations of observers at a certain

location may clue the aircrews where to search

first for the ground targets.

Every test plan should include clear statements of:

(1) The overall purpose of the test.

(2) Critical issues with respect tc

operational requirements.

A-92
Topics Relevant ;

to System Development I Topic I

and Evaluation Technology Mo. ! ABSTRACT

(3) The major test objectives.

(4) The schedule of test milestones.

(5) The major resources required:

(a) Test environment, facilities and

instrumentation.

(b) Operational environment

(6) The organizations which will conduct

the test program.

(7) The analysis and evaluation approach.

Tests should:

(1) Have specific objectives.

(2) List in advance actions to be taken

as a consequence of the test results.

(3) Be instrumented to permit diagnosis of

the causes of lack of performance.

CO Not be repeated if failures occur,

without detailed analysis of the failure,

(5) Be rehearsed for each new phase of

testing.

The test schedule should:

(1) Allow for s sufficient time between the

planned end of demonstration testing

and major procurement decisions so that

there is a flexibility for modification

of plans which may be required during

the test phases of the program:

(a) The number of test items available

and the schedule interface with

other systems needed in the tests,

such as aircraft, electronics,

missiles, etc.

(b) Support required to assist in the

preparation, conduct of the tests,

and the analysis of t*»Rt results.

A-93
', Topics Relevant I ',

j to System Development I Topic !

!and Evaluation Technology ! No. i ABSTRACT

(2) Be adjusted to minimize the so-called

T&E gap caused by a lack of hardware.

Specifically, a test gap can result

if funds are not applied until the

results of I0T4E are known because of

the required lead time for production

planning, production facilities, and

tool and production hardware.

Budgeting documents should be regularly

reviewed to ensure that there are adequate

identified funds for testing, relative to

development and fabrication funds. Budgeting documents need careful scrutiny to

ensure that there are adequate contingency

funds to cover correction of difficulties

at a level which matches the Industry/

Government experience on such contracts.

(Testing for difficulty without sufficient

funding for proper correction results in

band aid approaches which ultimately require

correction at a later and more expensive

time period.)

The constraints to be placed on the tes'

because of the range and instrumentation

are of prime importance. As previously

stated, the test facilities and instrumentation requirements to conduct operational

tests should be identified, along with a

tentative schedule of test activities. The

applicability of existing test ranges and

the adequacy of current facilities and instrumentation should be verified. Insofar

as possible, alternative approaches (different ranges, etc.) and instrumentation

improvements needed should be specified. If

range and instrumentation factors are found

to cast significant doubt on the meaningfulness of the test data because of a lack of

operational realism, the steps necessary to

assure meaningful data should be previously

identified and planned.

The primary basis for the test sample size

is usually based on one or more of the

following:

(1) Analysis of test objectives.

A-94
Topics Relevant I

to System Development ! Topic

and Evaluation Technology ! No. ABSTRACT

.

(2) Statistical significance of test

results at some specified confidence

level.

(3) Availability of test vehicles, items,

etc.

CO Support resources or facilities

available.

(5) Time available for the test program.

V

A-95
Dieterly, D.L. The evaluation of training with specific emphasis on criteria

(AU-AFIT-SL-9-73). Wright-Patterson AFB, OH: Air Force Institute of

Technolo-'.y, School of Systems and Logistics, October 1973. (AD-771 009).

Topics Relevant I

to System Development [Topic

land Evaluation Technology ! No. ABSTRACT

1. State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.<t General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

l.k Performance

Criteria, Specific

3.5 Measurement Procedures

I, Planning Components

of the Process

4.1 Analytic Methods

%,t Parameter Determinations

1.3 Apparatus for Testing

«.« Personnel for Testing

1.5 Test Plans

Application Cocponents

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Tindings Interpretation

5.1 Conclusions and

Recooaendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 The purpose of this paper is to present a

descriptive model of the evaluative process

that has been synthesized from the concepts

presented in numerous papers. In addition

emphasis was placed on the criterion

problem.

Evaluation of training should be considered

in the initial developmental stages of

program design. The evaluation process

consists primarily of translating the

program objectives into quantifiable

measures which can be used to feed back

into the program to indicate needed

modifications or as the final information

used in making the decision as.to whether

the program should continue or not. The

major problem in training evaluation is the

translation of objectives into criteria.

The criterion is a measurable variable

which represents the dimension of interest

in the research study. In industrial

psychology the criterion is usually

predicted from knowledge of the predictor

variable measure. The greater the ability

of the predictor variable to predict or

capture the criterion variable the higher

the validity of the predictors.

Three criterion concepts are presented:

(1) Ultimate, intermediate, immediate.

(2) External, internal.

A-96
Topics Relevant

I to System Development !Topic

land Evaluation Technology ! No. ABSTRACT

(3) Summative, formative.

Each type in each concept is defined as

follows:

(1) Ultimate criteria are usually stated

in broad conceptual terms and

represent the total goal of a

particular action.

(2) Immediate criteria are some variable

measures which are obtainable and are

logically felt to represent the

ultimate criteria.

(3) Intermediate criteria are some

measurable variables that are

logically felt to be related to the

immediate variables and which are

obtainable prior to attaining the

immediate criteria.

(1) Internal criteria are considered

outcome measures linked directly to

the training content which are

assessments made during or ianediately

after the learning experience.

(5) External criteria are designed to

assess behavioral changes in the

organizational role of the individual.

(6) Formative criteria are those used to

evaluate segments of the training

program to make changes within the

program prior to its culmination.

(7) Summative criteria are concerned with

the final evaluation of the total

program.

Three dimensions of criteria are discussed:

(1) The static dimension which is the

usual type of criteria of production

or Job performance behavior.

A-97
Topics Relevant

i to System Development I Topic

jand Evaluation Technology I No. ABSTRACT

(2) The dynamic dimension whll;h Is the

change over time of job performance.

(3) The Individual difference dimension

which measures the change in the

individual over time.

In selecting the criterion measure to be

used in a research design there are four

basic considerations upon which to evaluate

the criterion:

(1) Relevance.

(2) Reliability.

(3) Discrimination.

(M) Practicality.

Criteria measures used in evaluation

usually fall into one of these categories:

(1) A test

(2) A rating

(3) A production measure

(M) Some archival record data

For training evaluation all internal

criteria measures are some form of test

with the objective of indicating the

Increase in knowledge resulting from

training. Rating scales are used to

measure total job performance but are not

too valid. Productive measures increase

validity but introduce social dynamics of

the job situation. Archival records offer

some of the most interesting possibilities

for unusual criteria but result in a high

degree of chaotic error and systematic

bias.

A-98
DiGialleonardo, F.R. 4 Barefoot, D.B. An approach for measuring benefit and

cost In management and Information systems (TR 75-21). San Diego, CA: Navy

Personnel Research and Development Center, October 1971. (AD-AOII 209).

Topics Relevant

i to System Development I Topic

land Evaluation Technology j Ho. ABSTRACT

St«te of the Art Nvlew

of the Process

1.1 General Systea

Neasureaents

1.2 System Taxonooy

Mode) CSTO)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Componentr

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.H General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Me-^ureient Procedures

1.1 The objective of this study was to support

the Manpower Requirements and Resources

Control System (HARRCS) by providing an

effective means of assessing benefits and

costs in management and Information

systems. The primary focus is on measuring

benefit of the information rather than

cost. The cost and benefit indices

obtained through this approach are being

used to identify development opportunities

and priorities and to permit comparisons

between present and proposed systems for

manpower planning in terms of possible

increase in effectiveness. Scales that are

based on this information benefit model

presented below make up part of a larger

instrument that is the primary vehicle for

HARRCS Phase I data collection. That

instrtment, the Manpower Personnel Planning

Questionnaire, is included in the report as

Appendix A.

The benefit model developed in this study

is as follows:

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Realized Value (Benefit) = P x R x U where,

P, the Potential Contribution, is a value

attached to the information on the basis of

some predetermined set of specifications

that the information should meet;

.•^

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recomnenda t i ons

Further Research Areas

6.1 Measurement System

Limitations

6.2 Rese irch P-^entials/

Priorities

6.3 Research Planning

R, the Received Value, is th. portion of

potential contribution thit is normally

received by users of the information;

U, the Utilization Value, is the portion of

the received value that users are normally

able to actually apply in performing their

functions.

A-99

£* J^-J^. -------- - •-•--- •- - v>^^> ^

.

- -
I Topics Relevant

I to System Development {Topic

and Evaluation Technology ! No. ABSTRACT

Values for P, R, and U are developed

through the questionnaire administered to

both producers and users of the

information. P is estimated on a scale of

0 to 4 while R and U are estimated

percentages. Although producer and user

estimates usually have some bias in them,

the two perspectives are complementary and

the technique gains much of its strength

from having both perspectives.

Preliminary analyses were performed using a

total of 145 observations that were

obtained by applying the data gathering

instrument in limited parts of the Navy's

manpower planning system. Because of data

constraints conventional Independence

assumptions of statistical techniques such

as regression and analysis of variance were

not satisfied. However, the basic

objective of this preliminary analysis was

to obtain an indication of how helpful the

hypothesized benefit factors might be.

Indications are that of the three factors,

Potential Contribution (P) is most strongly

related to overall benefit. The Received

Value (R) factor showed the next highest

regression coefficient with Utilization

Value (U) being least related.

It was concluded that the approach

presented in this report is viewed as a

significant advancement in dealing with the

problem of benefit measurement and analysis

In hunanistic systems. It is reconmended

that a programmatic approach to this

problem area, using the benefit factors

developed in this preliminary study be

employed.

A-100

» __- _ - jT

v •." s" -. . ." • .' -." - - s'v --.-. > ,yv. -. . > ...
DiGialleonardo, F.R., Barefoot, D.B., Blanco, T.A. Technique for Interactive

Systems Analysis (TISA) (NPRDC-TR-75-22). San Diego, CA: Navy Personnel

Research and Development Center, 197M. (AD-A013 223).

I Topics Relevant I

I to System Development I Topic

land Evaluation Technology i No. ! ABSTRACT

State of HM Art Review

of the Process

1.1 General Systen

Measureaents

System Taxonomy

Model (STM)

Overall Conceptual

Process Model (CPM)

1.2

1.3

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.U General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

«.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recomnendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Prioritiea

6.3 Research Planning

1.1 The field of systems analysis has

traditionally been devoted to obtaining the

necessary basis for design or redesign of

hardware systems. While this analytic

capibility is no less desirable for soft

systems, applications in that area have

been hampered by data deficiencies,

difficulties in system definition and the

specification of desired performance,

measurement problems, and the like. As a

result, many attempts to perform systems

analyses on organizations, for example,

have resulted in lengthy verbal products

too inescapably static to have a real-time

impact on system design.

1.3 In response to particular problems in the

field of system analysis, a Technique for

Interactive Systems Analysis (TISA) was

developed. TISA is a computerized

technique for conducting systems analysis

in a conversational mode from interactive

terminals. It uses networking algorithms

to access and structure system descriptive

data from computer files. Specifically,

TISA was developed in order to perform the

systems analysis necessary to:

(1) Assess the state of the current system

at the onset of a major development

effort (i.e., establish a

baseline).

(2) Reveal deficiencies in the Manpower

System and probable candidates for

further R 4 D.

A-101

> -^ -^ 1—I-J—-^- - - -- 1 - - - ——
Topics Relevant

to System Development iTopic

and Evaluation Technology I Wo. ABSTRACT

(3) Provide a convenient and effective

means for assessing the progress of

the development program at any point

in Its evaluation.

3.3 TISA was designed to meet certain

requirements of data manipulation. These

requirements included the following:

(1) The ability to structure the data in a

network flow context.

(2) The ability to structure the data in

an organizational context.

•

(3) Versatility of data access for direct

and selective analysis with a variety

of routines in an interactive mode.

(H) Speed, lucidity, and reliability in

the graphical depiction of any system

of communications as specified by a

selected data set.

5.^ The following recommendations are made for

TISA:

(1) The development of TISA into a

standard initial analysis approach for

soft system development programs.

(2) The use of TISA as a methodological

device in more basic areas of research

such as organizational behavior,

information processing, and human

factors.

A-102

1 1 1

.,-..-.■■■■•-- - .- • .- -.- _• .

• * ' " . *

kd . _ •
During, K.E., Hickok, C.W. Emerson, K.C. 4 Clement, W.F. Control-display

testing requirements study (AFFDL-TR-72-122). Cedar Rapids, IA: Collins

Radio Co., December 1972. (AD-759 539).

Topics Relevant

I to System Development !Topic!

land Evaluation Technology | Ho. j ABSTRACT

1. State of the Art Review

of the Process

1.1 General System

Messurenents

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Envlronaent Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, ultimate

3. Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3-3 Performance

Requirements, Specific

3.4 Performance

Criteria, Specific

3.5 Measurement Procedures

4. Planning Components

of the Process

1.1 Analytic Methods

4.2 Parameter Determinations

«t.3 Apparatus for Testing

4.4 Personnel for Testing

*.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recomnendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.3 The following steps represent the blocks In

a procedural block diagram culminating In

system and pilot-centered evaluation

criteria:

(1) Describe vehicle operational profile.

(2) Select outcomes of interest.

(3) Specify outcomes and pilot acceptance

in terms of critical limits of

pertinent variables In numerical terms.

(4) Determine system error and state

variable performance response to

Inputs.

(5) Determine outcome probabilities and

pilot acceptance probabilities.

(6) Define safety, operational capability

and pilot acceptance design qualities.

(7) Determine procedural variables.

(8) Determine task variables.

(9) Determine environmental variables.

(10) Define normal/degraded feed back

arrangements and control-display

mechanizations to perform functions.

Allocate functions to manual and/or

automatic systems.

A-103

;.-« .• ..«]. .. .. . .—-• - - - -^ - - - ----•-- - - - - . . ^ ^- ,.- ^- . •.- . - ^ --
j Topics Relevant

I to System Development I Topic!

land Evaluation Technology No. | ABSTRACT

(11) Identify system performance-centered

variables and physical

characteristics.

(12) Identify human operator-centered

variables.

2.1 The system used to develop control-display

testing requirements is a Microwave Landing

System (MLS). The MLS p-ovides increased

senior capability to support terminal area

approach and landing operations. Its

potential includes zero visibility

landings, maximum runway utilization, noise

abatement, ground fire avoidance and

reduced separation standards.

2.2 The mission to be evaluated is aircraft

approach and landing using MLS.

2.5 The ultimate performance requirement of the

MLS is to insure successful aircraft

landing.

2.6 The ultimate criteria of the MLS system is

that the aircraft must be within a

successful landing window defined by

dispersions at decision height and

reference position at touchdown.

3.1 Quantitative and qualitative performance

and measures and evaluation criteria are as

3.2 follows:

(1) System Performance (Evaluation

criteria are commensurate with metrics

and absolute in value).

(a) At Approach Window

. Location in state space with

respect to window boundaries

Probability of approach

success

A-104

. . . '.•-.•.-. ..-.. .• .• .•.-.•.• /vVv .• v f ... . .- ■■■■. . .- .• . .•..•.- .- . •.•.•.-.- .•

.1 M „',*i I" •!• I I ! Bfel .1,^,^ A,.l * J .-t . i - - - ^^y^^^!^^
Topics Relevant

i to System Development I Topic I

!and Evaluation Technology ! No. ! ABSTRACT

(b) At Touchdown

. Longitudinal and lateral

touchdown location »-ith

respect to runway

Sink rate

Sideslip

Heading

Pitch, and roll attitudes

Airspeed error

(c) Composite Measures

(2) Safety Measures

(a) Probabilities (Evaluation

criteria are coanensurate)

Successful landing

Successful missed approach

. Accident or incident

Margin (stall, performance,

etc.)

(b) Qualitative Assssments

(Evaluation criteria are relative

and subjective; the graceful

degradation hypothesis provides a

guide).

Missed approach procedures

Failure detection procedures

. Emergency takeover procedures

A-105

L-. -.•- • • .- - . , , ^ -•--•-. -• - ,-.-.... .- •,-. . . . ... .. ... . .. ..

• ••. •'.

. . • *

..•.. A ■■,'._ i^* ,_* M_- ^j" ±**
Topics Relevant I

I to System Development !Topic I

!and Evaluation Technology j No. ! ABSTRACT

(3) Pilot Performance and Acceptance

Measures

(a) Pilot Dynamic Behavior

(Evaluation criteria are relative)

. Describing functions (loops

closed and equalization

demanded; control display

associations and residual

cross-coupling; sensitivity

of stability, disturbance

regulation, and

coranand-fol lowing

performance to variations in

gain, time delay, and

equalization; the adaptive

feed back selection

hypothesis and successive

organization of perception

hypothesis provide guides).

Scanning activity

distributions (incoherence

in system performance caused

by scanning remnant; system

status monitoring threshold

for confidence and

decision-making; the display

arrangement hypothesis

provides a guide).

. Opinion ratings

(psychometric scales).

Workload and operability

assessment (excess control

capacity; auxiliary task

scores and loads;

psychophysiological

measurements; there is no

guide to evaluation other

than sensitivity and

relative differences.

(b) Pilot Acceptance of System

Performance

A-106

• -.'i i.la'Vi 1 . • . . . I'^A.V. -". i. J. i'. i" •. •» --'-•- - - . . ...-....._, .v. .-, . .V.V _ . .-•. .•.\. r.. .'.:.'.-..-.• - - * > -
Topics Relevant

! to System Development {Topic

land Evaluation Technology ! No. ABSTRACT

Attitude, attitude rate, and

load factors variances from

trimmed values (Evaluation

criteria are commensurate

and absolute, e.g.^

probabilities of exceeding

acceptable levels from

trimmed values).

Control displacement and

rate variances from trimmed

values (Evaluation criteria

are commensurate and

absolute, e.g., probabilities of exceeding maximum

authorities).

i

Response compatibility and

motion harmony-automatic and

flight director versus

manual control (Evaluation

criteria are relative to the

response and motion

attributes under manual

control).

»

Command consistency-flight

director versus Manual

control Evaluation criteria

are based on the consonance

between the spectral

distribution of status

variables in the director

command and the displayed

status variables

themselves).

. Qualitative assessmentspilot commentary (Evaluation

criteria are subjective and

relative).

1.1 The control-display design validation

process will be conducted In three phases:

(1) Theoretical analysis

A-107

te_

,V A f m' ."^ M-_ _* _" " ._^
1

Topics Relevant ! I

to System Development I Topic I

and Evaluation Technology j No. ! ABSTRACT

(2) Simulation

(3) Flight test

U.S During actual flight tests, phototheodolites may be used for tracking

simulated IFR approaches, but radar must be

used under actual IFR. Special flight

control systems will be required to allow

pilot performance measurement.

4.5 Flight test plans must define specific

tasks and responses as well as complex

sequences of tasks designed to provide

information on pilot understanding,

workload, tracking, monitoring, and

decision-making abilities.

In the area of tracking performance,

specific acquisition, approach, and missed

approach paths need to be defined, and data

on aircraft displacement and attitude

accumulated through the beam capture,

approach, and landing phases. Data from

flights will be used to form statistical

performance distribution curves for

comparative analyses. To help evaluate one

aspect of pilot workload, these same

flights can be used to obtain similar

distribution curves for wheel motion,

command nulling error, pilot's

eye-point-of-regard, and subjective

ratings. When combined with pilot

interview data, these data will form an

analytical basis for control-display system

evaluation.

In addition to verifying beam capture,

approach, and landing, special emphasis

will be required to exercise the new

capabilities of HLS coverage for guiding

and monitoring segmented and curved

approach paths and missed approaches.

Deliberate disturbances to test control

responses, distracting influences to test

pilot error sensitivity, deliberate

failures to test pilot reaction, and

extensive use of the real environment will

A-108

..--v^vWlV--.--. • - - •

—

•- .v

f« _•_ - -»

—
Topics Relevant I |

I to System Development I Topic I

land Evaluation Technology ! No. I ABSTRACT

be made to determine display capabilities.

If the TIFS aircraft can be made available,

simulated weather would be a possibility to

provide one-to-one correspondence of

vehicle and simulation. Further, multiple

vehicles can be simulated with the TIFS

vehicles. This would provide a very

flexible platform, requiring only one set

of instrumentation.

Use of the digital computer and flexible

displays may be strong contributors to the

hardware flexibility requirements. This

would allow study of data rate problems

but, more significantly, would provide a

capability of readily changing control-display parameters over a very wide range.

Such tools should certainly be considered

for curved path studies. Controlled inputs

allowing for correlation studies in the air

should be considered with the flexibility

provided by this advanced equipment to

measure pilot behavior.

Automatic data logging can be valuable as a

flight test technique. Failure information, data discrepancies, and complex path

geometries can be introduced and pilot

responses noted with experimental

certainty. Extensive use will be made of

the monitor pilot, voice recording, strip

chart recording, magnetic tape recording,

and controlled test stimuli.

5.1* A program plan for control-display testing

requirements for the new Microwave Landing

System can expeditiously and effectively be

implemented within the 5-year period. A

great deal of directly relevant, applied

research on landing systems has been

conducted by many agencies. Perhaps the

most notable example is the Air Force PIFAX

program. For well over a decade this and

corollary efforts have addressed key

control-display. Instrumentation,

operational, and pilot factor problems in

A-109

------ • - . * »
Topics Relevant

I to System Development I Topic

land Evaluation Technology ! No. ABSTRACT

approach and landing cperations. The

result is a large reservoir of

experimentally demonstrated concepts,

principles, and prototype instrumentation.

Also, the existence of various ILS-based

systems for almost three decades has

developed a broad appreciation for

desirable operational characteristics among

users.

It remains to condense, coalesce, and apply

these inputs into a meaningful and

comprehensive program plan. In this

section, an attempt was made to show how

theoretical analysis, simulation, and

flight test can be coordinated so that,

when all levels of analysis are completed,

the resulting control-display system design

will be validated as "best.**

A-110

^ i -^
: K" Dunlap, J.W. & Affinito, F.J. Development of methodology for measuring

effects of personal clothing and equipment on combat effectiveneas of

individual soldiers (Final Rep.of Phase 3). Darien, CT: Dunlap and Assoc.

Inc., December 1967. (AD-836 904).

I Topics Relevant

I to System Development iTopic

land Evaluation Technology j Wo. ABSTRACT

State of the Art Review

of the Process

1.1 General System

Heasureaents

1.2 System Taxonoay

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.M Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recomaendatlons

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

2.1 An Integrated Test Facility was designed

and constructed combining the individual

test courses developed in a previous phase

of this study. Requirements were developed

for an instrumentation system to measure

automatically the performance of test

participants and requirements and

specifications were developed for a

centralized, computerized, data logging

system to record, process and statistically

analyze performance data collected by the

system.

2.2 A field methodology was developed for

measuring the effects of experimental

clothing and equipment on the combat

effectiveness of individual infantrymen.

2.3 The Integrated Test Facility, located at

Camp Picket, Va, consisted of eight test

situations, five of which were developed

and tested in a previous phase of this

study.

3.2 Performance data were collected by various

types of electromechanical sensor devices

located on the various performance courses

of the Integrated Test Facility.

3.3 Sensors were activated by the activities of

the subjects causing input signals to be

sent to a centrally located data logging

facility.

*,1 The Integrated Test Facility consisted of

eight distinct courses representative of

important combat tasks. Estimates of the

A-lll

--••-•

»•:

—
Topics Relevant i

I to System Development I Topic!

and Evaluation Technology ! No. ! ABSTRACT

performance scores which each course

generated were collected as well as

estimates of the time required to complete

each course.

5.4 The tryout and evaluation which was

necessary to meet the last objective of

this phase, could not be accomplished due

to procurement difficulties in acquiring

the Data Logging System. The evaluation

and final refinement will be conducted upon

acquiring this system.

A-112
Dunlap and Associates, Inc. Performance measures for human factors engineering

evaluation of EARL equipment (SSD-332-618). Darien, CT: Dunlap and Assoc,

Inc., June 1966. (AD-653 627).

V

Topics Relevant

i to System Development {Topic

{and Evaluation Technology ! No. ABSTRACT

1. State of the Art Review

of the Process

1.1 General Systeii

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, UHimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Spec'fie

S.M Performance

Criteria, Specific

3.5 Measurement Procedures

4. Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Coaponents

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Reconriendations

Further Reaearch Areas

6.1 Measurement System

Limitation?

6.2 Research Poteitlals/

Priorities

6.3 Research Planning

2.1 The system evaluated was a 105 m> howitzer

battery consisting of six sections, each

with one howitzer and an eight-man team.

2.2 The tactical activity was an RSOP

(Reconnaissance, Selection and Occupation

of Position): 105 BDJ howitzer sections

conducting modified planned occupation of

an area. The occupation was preceded by a

recon party in order to determine firing

positions, installation sites, routes of

march, defensive plans, etc. Each section

was to complete four major actions:

uncoupling the howitzer, preparing for

actions, firing and march order.

2.3 Each team member was equipped with the

following: fatigue uniform, field jacket,

helmet with liner, M56 harness with ammo

pockets, canteen, entrenching tool, poncho,

M16 rifle and M17 field mask.

2.4 The field test had the following conditions:

(1) Moderately unpracticed teams in actual

training.

(2) Emplacement at an actual field site on

the firing range.

(3) Full tactical uniform, equipment and

live ammo.

(4) Complete emplacement, live firing and

march order sequence.

(5) Field training SOP.

A-113

.. .

. .-:-.v.

i
Topics Relevant

I to System Development I Topic

land Evaluation Technology j Wo. ABSTRACT

3.1 The characteristics to be measured were:

(1) Sequence of team subtasks.

(2) Number of visual, oral and manual team

coordination cues.

(3) Quality of performance.

(U) Time to perform.

(5) Density of team's activity.

(6) Number of critical incidents.

(7) Subjective rating of performance.

(8) Tactical accuracy.

(9) Total time of team performance.

(10) Comparative indications of

physiological energy expenditure.

3.2 The units of measurement were as follows:

(1) Number of scheduled and unscheduled

oral communications.

(2) Number of scheduled and unscheduled

visual communications.

(3) Number of errors by team and

individual.

(1) Number of unsafe conditions.

(5) Time data in minutes and seconds.

(6) Quality data was "good," "fair," or

"poor," based on the subjective

judgment of an experienced battery

officer.

(7) Heart rate of certain team members.

A-114

• .

- -• • • - •- -
Topics Relevant

I to System Development I Topic!

land Evaluation Technology ! Ho. ! ABSTRACT

3.5 Measurement was accomplished using a series

of methods:

(1) Visual recording by film and TV

camera.

(2) Oral recording by taping and pick-up

technicians.

(3) Observers for visual and oral

interactivity and team events.

M.1 The data analysis methods employed were:

(1) Means

(2) Standard deviations

(3) t ratios

(M) Regression analysis

(5) Sequence effects analysis

(6) After-task questionnaires

M.3 The test apparatus included film and TV

camera, tape recorders, still cameras,

telemetry charts (heart rate).

M.l Forty-eight test subjects were used (six

teams of eight men each). The chief of

section was either an E-5 or E-6. All

others were E-2 or E-3,s.

5.2 The resulting analysis pointed out variables relevant to small-group interaction

and performance which were able to discriminate differences between performance

of teams with and without the M-17

protective mask.

5.U The conclusions of this study can be

summarized as follows:

A-115

^—. M -- - - -

-.
I Topics Relevant

S to System Development iTopic I

and Evaluation Technology ! No. I ABSTRACT

(1) A simple measurement design is

inadequate for field testing of teams,

Observations must be structured to

detect separate pitterns of performance across multiple variables,

individual team members, and various

sub-tasks.

a

(2) Human observers at the field site

cannot alone record this wide range

of data at the pace of team performance. Data recording equipment,

particularly video-recorders, assure a

more adequate and objective data

collection.

(3) Performance of certain team tasks,

generally those requiring individual

initiative in coordinating activity,

may be significantly altered by the

wearing of protective masks. Other

tasks of a repetitive drill nature may

be much less affected. Even very

meticulous testing during field

training activity may fail (because of

the "slack time" and indirect

motivations inherent m the practice

environment) to reflect validly

potential decrements in a tactical

environment.

(M) For 105 mm howitzer crews

specifically, there will probably be

no consistently critical performance

problems while wearing protective

masks, although the team's normal

pattern of oral intercommunication

is definitely disrupted.

6.2 The results of this effort offer three

promising directions for further study in

the areas of field collection equipment,

team data analysis, and standard

performance testing.

4-111

- - - Ä llllHlll
Topics Relevant I

I to System Development lTopic I

land Evaluation Technology ! No. j ABSTRACT

.-.

(1) A system of team data reduction and

analysis in depth should be

sophisticated, so that large

quantities of data from mul' iple

trials over time could be methodically

built into a data base for each

variable, team member, and sub-task of

a standard test. Application of

automatic data processing and

mathematical modeling should be

investigated, and weak spots in the

initial design, such as the quality

and density of activity variables,

should be strengthened through quality

control and systems analysis

techniques.

(2) The generalizability of objective

| field-monitoring techniques, such as

video-recording, should be explored in

detail. Any field performance that is

broadly repeatable could become a

standard test, in effect, by visually

recording each trial and redirecting

the structuring control from the

participants to the observation

procedure: standardly structured

camera coverage, videotape editing and

playback, observer training, and video

viewing schedules by separate

variables and sub-tasks.

(3) Routine team testing should be

conducted using the improved data

collection devices and data analysis

techniques perfected in the study

described above. The current study

emphasized the detection of pattern

changes, not the significance or

tactical validity of all such changes.

In its field observations, this

objective was achieved. The promising

trends in variables from this small

number of trials should now be pursued

in a much larger "n" to provide the

data base for probable statistical

significance in several categories.

These extended performance tests, if

A-117

,■■•:<::-- • -.-.,::-- -.:-'^^-^ :.-.-:-•::-. v..:-:-^:-^,

^.•. . ._
Topics Relevant

I to System Development

land Evaluation Technology

i—r

ITopicI

! No. | ABSTRACT

done with chemical protective

equipment, could then possibly

establish reliable patterns of

difference valuable in improving

design of the equipment and in

predicting masked team behavior under

valid tactical conditions.

b

A-118 -•

Ä^ - - - • *ä •. ^-.

- • - ' UMii
Egbert, J.J. & Rau, J.G. Techniques for bounding and estimating measures of

effectiveness. Newport Beach, CA: Ultrasysteras, Inc., July 1973. (AD-772

550).

Topics Relevant

I to System Development 1 Topic

land Evaluation Technology | No. ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonamy

Model (STM)

1.3 Overall Conceptual

Prjcess Model (CPH)

Contextual Coaponents

of tne Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultiute

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Att-ibutes

3.2 Practical Attribute

Measures

3.3 Perfoinance

Requirements, Specific

3.4 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

»•.'I Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potential»/

Priorities

6.3 Research Planning

3.1 This study describes methods for the

formulation of measures of effectiveness

(MOE's) used in studies where variables are

not readily (if at all) measurable in the

real world. Within this report is a survey

(and listing) of both algebraic and

statistical techniques which are useful in

bounding and estimating MOE's. Bounding in

this context means confidence limits or

intervals.

3.2 The results of the survey are intended to

be illustrative, rather than exhaustive of

some of the types of MOE's in use. For

example, an MOE for Airborne Antisubmarine

Warfare (ASW) is the probability of

submarine detection by helicopter. These

MOE's are chosen from a spectrum of naval

warfare areas with particular emphasis

placed on antisubmarine warfare.

3.5 For each M0Cthe measurement techniques are

provided to determine the bounds of the MOE

values. These techniques are of various

forms such as linear-exponential product,

sum of exponentials, sum of products etc.

A-119

.-f »

-----

L" ^ • . , .->•.• v.-•.

. . . . ^
Ellis, R.H. Task Analysla Reduction Technique (TART) for the quantification of

human performance (Research Memo. SRM 71-7). San Diego, CA: Naval Personnel

and Training Research Laboratory, September 1970. (AD-711 807).

Topics Relevant

I to System Development

land Evaluation Technology ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Prooess Model (CPU)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.') General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.'' Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoasendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 This document describes a procedure for

collecting performance data and examining

man/machine interaction. This procedure,

called a Task Analysis Reduction Technique

(TART), allows for facilitation of human

performance quantification, clarification

of analysis and improved useability of the

data.

3.1 The TART focuses on the interface at the

task level of complexity. Analysis at the

task level is advantageous because one can,

according to one's needs, break the task

down into movements and actions for

detailed analysis or one can combine tasks

and analyze different configurations at a

higher functional level.

3.2 The example provided demonstrates the use

of TART by deriving measures for the air

detector/tracker (ADT) position on the

Antisubmarine Warfare Tactical Data System

(ASWTDS).

3.3 The TART, in general, can produce the

following types of data:

(1) Frequency — the absolute and relative

frequency for each of the tasks for

given periods of time under any type

of loading condition for any type of

operator.

(2) Time — mean and relative times for

each task, under conditions of varying

loads and operators.

A-120

^•^-^.,^-. ^^ ^f. ^ . \ ._ _ , •> , i ,. ^ >

iÜJ
Topics Relevant \

i to System Development 1 Topic 1

land Evaluation Technology j No. ! ABSTRACT

(3) Frequency/time comparisons —

comparisons between the absolute and

relative occurrence of each.

CO Sequential properties — the nature of

strategies employed by the various

operators in performing their various

functions.

:

3.1

3.5

(5) Error analysis — number and type of

deviations from expected procedures

and the effects of innovative actions.

Specific performance measures are presented

in the ASWTDS example.

The TART consists of a three-part procedure:

(1) Task analysis Operational Sequence

Diagrams (OSDs) are developed for the

tasks observed.

(2) A video recording of the action and

feedback channels of the machine is

obtained.

•

t —

(3) The video-recording of that activity

is reduced to a time-line sequence of

basic tasks, and an analysis of the

frequency, time and sequential

properties of that time-line is

performed.

5.4 Using the techniques described in this

report, it is possible to collect and

quantify useful human performance data. It

can be used to analyze a realistic

man/machine interface. An automated form

of the task analysis OSD is useful for

developing base-line performance

expectations, and the basic form of the

data collected can be useful types of

analyses.

It is recommended the TART system be

continued and onboard ship programs begun.

It was also suggested that this technique

be included in the repertoire of those

considering man/machine interface analysis,

and a data bank of empirical personnel

performance data begun.

A-121

- - - - -

.--. --'-V- . •.•>.--.•-.-•

- - - - - - - tämä

>.v.

Jw I

.-■■• ..--::..---....v ■■:.
Engel, J.D. An approach to standardizing human performance assessment

(Professional Paper 26-70). Presentation at the Planning Conference of

Standardization of Tasks and Measures for Human Factors Research, Texas

Technological University, Lubbock, Texas, March 1970. (AD-717 258).

Topics Relevant

i to System Development

land Evaluation Technology

Topic 1

No. I ABSTRACT

1. State of the Art Review

of the Process

1.1 General System

Measurements

1.2 Syster Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

k. Planning Components

of the Process

1.1 Analytic Methods

1.2 Parameter Determinations

1.3 Apparatus for Testing

l.l Personnel for Testing

4.5 Test Plans

5. Application Coaponents

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.1 Conclusions and

Rec ooaenda ti ons

1.1 The purpose of this paper is to consider

data concerning the relationship of a task

taxonomy and performance measurement

taxonomy. A task classification scheme was

used as follows:

(1) Learning identifications.

(2) Perceptual discriminations.

(3) Principles and relationships

comprehension.

(U) Procedural sequencing.

(5) Decision making.

(6) Perceptual motor skills.

The performance measurement classification

system is categorized on the basis of

remoteness from actual job performance.

The four major segments along this

continuum are:

(1) On-the-job measures.

(2) Work sample measures.

(3) Simulated-job measures.

(4) Correlated-job measures.

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-122

• 1 -- ad ^L" - -

..-.•. ^-^-^
Topics Relevant

I to System Development !Topic!

land Evaluation Technology I No. j ABSTRACT

2.1 The system studied was that of the general

vehicle mechanic for wheeled and tracked

army vehicles.

,. ,- 4.1 A four-day proficiency test consisting of

33 sample exercises was used. The test

included a diagnostic scoring procedure for

use in scoring men on quality of

performance. In addition a questionnaire

was used to obtain supplemental information

regarding the mechanic's experience. A

second phase of research dealt with a

comparision of two Job correlated measures

with the work sample via a paper and pencil

evaluation test.

4.4 The test subjects were 38 organizational

mechanics drawn from all organizational

maintenance units at Fort Knox. The

exercises we^e individually scored by

experienced mechanics trained in test

administration procedures.

5.2 The results indicated that the total test

appears to have a high degree of

reliability, indicating it should permit a

high degree of accuracy measurement when

used as a criterion in evaluating other

measurement techniques. The second phase

written test had a low degree of validity

when correlated with the work sample and as

a result should not be used in group or

individual measurement.

5.4 Further research should be conducted along

two parallel lines:

(1) Development and refinement of an

interim task classification system.

(2) Development and refinement of a

interim classification system for

human performance measures.

A-123

- - - - - • -

V-"-V.- .

......
Eppler, W.G. Analytical design of manned control systems (SUDAAR Ho. 280).

Stanford, CA: Stanford University, Center for Systems Research, May 1966.

t . Topics Relevant I

I to System Development jTopic

land Evaluation Technology I No. ABSTRACT

State of the Art Review 1 •1

of the Process

1.1 General Systea

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirecents, Specific

3.1( Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

1.2 Parameter Determinations

1.3 Apparatus for Testing

1.1 Personnel for Testing

1.5 Test Plans

This dissertation treats the problem of

designing high-performance, closed-loop

control systems which Include a human

operator. In particular It describes

research conducted to answer the following

rather general questions:

(1) How should the state of the system be

displayed to the operator and how

should his responses to the display be

processed before they are Input to the

system?

(2) In what way does the operator's

dynamic response limit the performance

of the overall system, and how does

this limitation depend on which of his

several possible outputs Is used for

controlling the system?

The results presented are applicable to the

design of a wide variety of manned control

systems.

-

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.1 Conclusions and

Recoonendatlons

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-124

.-- ••.- .•-

* • • » -

- .•.-.v-v/. - - . ^v,

-

w

-' - - ---■■■• -.-'.-.-
Erickson, R.A. Field evaluation of a visual detection model. In G. W. Levy

(Ed.), Symposium on applied model of man-machine systems performance

(NR69H-i)91). Colunbus OH: North American Aviation, November 1968. (AD-697

939).

i Topics Relevant i

I to System Development !Topic

land Evaluation Technology | No. ABSTRACT

'.N

1. State or the Art Review

of the Process

1.1 General System

Heasureaents

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPU)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2." General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Procets

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

H. Planning Components

of the Process

1.1 Analytic Methods

1.2 Parameter Determinations

4.3 Apparatus for Testing

H.n Personnel for Testing

1.5 Test Plans

5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.1 Conclusions and

Recooaendations

6. Further Research Areas

6.1 Measn'-ment System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

M.S

5.3

The report describes a 1962 field

experiment conducted to validate a

mathematical model of the visual detection

process. All observations were made by

pilots flying A-M aircraft above a

bulldozed strip in the desert. Ground

targets were a Sherman tank and a radar van

without the radar dish/antenna. Thus, the

visual search was in one dimension only;

the model was not capable of handling

2-dimensional search. Flights were

conducted at altitudes of 1000, 2500, and

1000 feet, at indicated airspeeds of 275,

270, and 265 knots, respectively.

Data showed that the model's predictions of

detection range are quite accurate, but the

prediction of recognition range is in error

by substantial amounts.

A-125

- - -

.v.- .--

■■ -

.•:•. v-v

- - -
Farina, A.J. & Wheaton

The task characteristic

Pittsburg, PA:

191).

R. Development of a taxonomy of human performance;

«pproach to performance prediction (Tech. Rep. 7).

Americar. Institutes for Research, February 1971. (AD-736

I Topics Relevant I

I to System Development {Topic

land Evaluation Technology I No. ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measureaients

1.2 System Taxonony

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual onents

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environaent Definition

2.4 General Constraints

2.5 Perfomance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoonendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Restarch Potentials/

Priorities

6.3 Research Planning

1.2 The present research described a series of

studies conducted to develop an instrument

in terms of which the stimulus, procedural

and response characteristics of tasks could

be described.

2.5 The basic steps in this research were to:

(a) develop descriptive characteristics of

tasks; (b) assess the reliability of rating

scales derived to measure these characteristics; and (c) determine if these characteristics represented correlations of

performance.

3.1 Major ccmponents of a task were identified

and treated as categories within which to

devise task characteristics or descriptions. Each characteristic was cast into a

rating scale format which presented a definition of the characteristic and provided a

seven-point scale with defined anchor and

mid-points, along with examples for each

point.

3.2 Nineteen scales were developed and

evaluated in a series of three reliability

studies. The following are the task

components and related characteristics (for

which the 19 scales were formed):

.\-126
Topics Relevant

I to System Development I Topic

iand Evaluation Technology j No. ABSTRACT

Component Task Characteristics

Goal Number of output units

Duration for which an output

unit is maintained

Number of elements per output

unit

Work load imposed by task goal

Difficulty of goal attainment

Response Precision

Rate

Simultaneity of responses

Amount of muscular effort

involved

Procedures Number of steps

Dependency among procedural

steps

Adherence to procedures

Procedural complexity

Stimulus Variability

Duration

Regularity of stimulus

occurrence

Stimulus- Degree of operator control

responses

Reaction time/feedback lag

relationship

Decision making

A-127

s

v"

* _ -- •- - - -

.• . -..•.V.^vV-.-. -.,

. * . . ' .

- - - - - - ^^ ^
1

I Topics Relevant i I

I to System Development [Topic!

!«nd Evaluation Technology ! No. j ABSTRACT

4.1 The paradigM used to determine-whether the

task characteristics were correlates of

performance upon which predictive relationships might be established was that of

•• post-diction".

4.2 Post-diction referred to the situation in

which performance measures were abstracted

from studies already existing in the

literature.

•

4.5 Subjects were supposed to rate descriptions

of the tasks used in these studies on task

characteristics scales and then these

ratings were to be subjected to multiple

regression analysis to establish the extent

to which they were related to the

performance in question.

5.1 Two such post-diction studies were

conducted.

i

5.2 In general, it was found that a subset of

scales having adequate reliability

consistently emerged in all three

reliability studies. The results of the

two post-diction studies were encouraging

in that significant multiple correlations

of .82 and .73 were obtained between task

characteristic ratings and the performance

measures.

5.3 It does appear possible to describe tasks

in terms of task-characteristic language

which is relatively free of the subjective

and indirect descriptions found in other

systems. Further, task characteristics may

represent correlates of performance.

•

A-128

^- '^'-

• • .

. . ,- ,• :■■■:
i

Featherstone, C.L. & Scaglione, R.J. A feasibility study for determining a

small arms measure of effectiveness for handling characteristics (Master's

Thesis). Monterey, CA: Naval Postgraduate School, September 1975- (AD-B008

586L).

I Topics Relevant !

I to System Development iTopic

land Evaluation Technology i Ho. ABSTRACT

State or the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.U General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

|.« Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

«.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recomnendations

1.3 The results of a feasibility study for

determining a small arms measure of

effectiveness for handling characteristics

were presented in this report.

2.1 The system being measured was the small

arms user, and .45 and .38 caliber pistols,

2.2 The handling characteristics of these

weapons were selected for evaluation.

2.3 The tests were conducted on a typical

pistol range common to most Army

installations.

2.5 The ultimate performance requirement is

that the small arms user survives in a

combat or other situation and that his

weapons are utilized effectively.

2.6 The ability to shift a weapon rapidly and

accurately from one target to another is a

critical factor in the use of small arms

weapons.

3.1 The primary purpose of this study was to

measure an individual's performance while

he was accomplishing a series of movement

and firing tasks and determine whether the

resulting MOE's could discriminate between

the two systems.

Further Research Areas

6.1 Measurement System

Limitations

6.2 Re.earch Potentials/

Priorities

6.3 Research Planning

A-129

_-
Topics Relevant

I to System Development ITopic

land Evaluation Technology j No. ABSTRACT

3.2 The time it took for an individual to

accomplish these tasks without error was

recorded. In addition, an index of

difficulty for the performance of shooting

tasks was developed using information

theory.

4.1 Analytic methods were used to determine if

there is a statistically significant

difference between the mean times it takes

an individual to complete a set of

prearranged movement tasks with two

differently handling weapon systems.

M.B

1.»»

:

*».5

5.3

The weapons used in this experiuent were

the M1911A1 .15 caliber automatic pistol

and the Smit^ and Wesson Model 10-38

caliber revolver and appropriate

ammunition. Targets, timing devices, and

record sheets were utilized in this

experiment.

The subjects were selected from the

population of military personnel who, at a

minimum, would be required as part of their

combat equipment, to carry and be prepared

to use a sidearm. All participants were

volunteers, ages 28-37, all were officers,

time in military service ranged from 7-16

years. Range of experience with weapons in

question varied.

The weapon, task sequence, and subject

factor levels were set prior to conduct of

experiment. Four task sequences were

selected. Subjects were briefed prior to

the experiment and received written

instructions. Practice on both types of

weapons was permitted followed by actual

firing in the task sequence assigned. At

the end of the firing the subject filled

out an information sheet giving personal

data and weapon evaluation.

The conclusions were as follows: (a) There

was no statistical difference between

weapons used, (b) Three of the four task

A-130

--.

-.-.-.-« ,-,-.

■■

■■ • ---•^>-'-"-

.• Vx.-. '- ^--^ - -•••-v-v.
I Topics Relevant I

i to System Development I Topic

land Evaluation Technology j No. ABSTRACT

.• •.

-

sequences were different. Sequences 1 and

2 tested as having no significant

difference, (c) The average information

processing rates for each weapon were

different but were not tested for

statistical significance. The index of

difficulty measurement of the tasks was

shown to have a high correlation to the

time required by each weapon to perform the

tasks that had one or more movements

specifically designed into the tasks.

.'

•

j

.. -•.

A-131

•' •"-""."^"-'•" 0 V^ -**.■■ 'V'" ". ""» '-'•"-''•' V'-' -" V"^."-"- ••

I -
Fineberg, M.L., Meister, D., and Farrell, J.P. An assessment of the

navigation performance of Army aviation under Nap-of-the-Earth conditions

(Research Rep. 1195). Alexandria, VA: U.S. Army Research Institute for the

Behavioral and Social Sciences. (AD-A060 563).

I Topics Relevant

! to System Development

land Evaluation Technology ABSTRACT

1. State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

S.M Performance

Criteria, Specific

3.5 Measurement Procedures

l). Planning Components

of the Process

1.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

5. Application Coaponents

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

1.1

2.6

3.1

and

3.2

The basic objective

studies was to obtai

the Nap-of-the-Earth

skill level of Army

pilot experience and

training. Other obj

a baseline on pilot

and to develop a fie

to measure pilot per

for this series of

n empirical data on how

(N9E) navigational

aviators is affected by

two levels of

ectives were to define

navigation proficiency

Id research methodology

formance in NOE flight.

The dependent measures were measures of

navigational performance.

Two measures of mission success were used.

These were:

(1) A subjective mission success score -

the instructor pilot rated the subject's navigational effectiveness in

terms of three categories: complete

failure, partial success, and complete

success.

(2) An objective mission success score -

this measure was constructed to serve

as a component metric representative

of the subject's scores on four

individual measures: number of

initial points (beginning of a route)

missed, number of landing zones

missed, number of 250 meter excursions

from the course line, and number of

1,000 meter excursions.

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-132

••- - •• ' -" - ' - - . • -- --

- - - - - -. -•--- -.- -^ -w.- .-..- ^
Topics Relevant I I

to System Development I Topic I

and Evaluation Technology j No. ! ABSTRACT

4.3 Two UH-1H helicopters were employed in the

study over a test range near Troy, Alabama.

4.H The subjects in the study were 35 Array

helicopter pilots who were currently

proficient with the test copters and had

some exposure to NOE flight at either entry

or unit level. Four Army navigators

conducted the tests, two being highly

qualified NOE instructor pilots.

4.5 The navigators were assigned missions in

which designated landing zones had to be

found for simulated medical evacuations or

supply deliveries. All 35 aviators

navigated at least six NOE routes ranging

from 23 to 25 kilometers (km) in length.

Twenty-eight of the aviators were also

tested on aircraft control and the

n performance of various NOE maneuvers.

5.3 NOE navigation is a trainable skill.

Experienced aviators with training

performed better than experienced aviators

without training. In addition, recent

graduates with 15 hours of NOE training

performed better than experienced aviators

without training.

Specialized training in NOE navigation is

valuable. The instructor pilots indicated

that experience is more important in

aircraft control than in navigation. The

group, with an average of 1,380 flight

hours, controlled their aircraft better

than the group with 200 hours but this

difference in experience appeared to have

no effect on the navigation.

The field research measures and techniques

developed for these experiments seem wellsuited for NOE navigation research. The

Objective Mission Success Score (OMSS), a

composite performance measure indicating

the probability that the mission would be

successfully completed, proved useful and

has a .75 correlation with subjective

A-133

— '- - • - • •• ---•••v^-- - • - - .....
I Topics Relevant

1 to System Development I Topic I

iand Evaluation Technology i No. I ABSTRACT

ratings by expert NOE instructor pilots.

As expected, it was shown that NOE

navigation is an extremely difficult and

complex task. The OMSS' show that NOE

missions flown over the Fort Rucker terrain

resulted in an overall probability of

success of .63.

A-134

-- -
Finley, D.L. 4 Muckler, F.A. Human factors research and the development of a

manned systems applications science; The system sampling problem and a

solution. Northridge, CA: Manned Systems Sciences, Inc., July 1976. (AD-029

T7)

I Topics Relevant I I

to System Development I Topic I

and Evaluation Technology 1 No. I ABSTRACT

1. State 9f the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (MM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

4. Planning Components

of the Process

1.1 Analytic Methods

4.2 Parameter Determinations

1.3 Apparatus for Testing

*.t Personnel for Testing

4.5 Test Plans

5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.1 Conclusions and

Recommendations

6. Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.2 This report points out: the need to

identify and incorporate systems design and

operation parameters into research

programs; the nature of systems research

and the dimensional problem, and presents a

model to support the systems dimensionalization process, i.e., a systems taxonomy

model.

2.1 This study addresses the manned system and

the need to develop a body of knowledge on

the dimension of a system, as opposed to

such components as an individual operator

or a piece of equipment.

5.3 The beginnings of a systems taxonomy model

are presented. It consists of three major

levels: (a) system objectives; (b) system

functional purpose and (c) system characteristics (structural operator/equipment,

operating and support requirements). These

three levels are further defined by their

relationship to the nominal versus relative

levels of measurement.

Information is presented on how to use this

model and of the model level implications

and importance.

A-135

-

.vvV- .

- -- - -
Finley, D.L., Muckler, F.A., Gainer, CA. & Obennayer, R.W. An analysis ard

evaluation methodology for command and control; Final technical report.

Northridge, CA:

871).

Manned Systems Sciences, Inc., November 1975. (AD-A023

I Topics Relevant i I

I to System Development I Topic I

land Evaluation Technology i No. I ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performanc»!

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1» Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.2 The systems taxonomy model abstracted

documented from this article is shown on

the following page:

A-136

-
'

MEASUREMENT

LEVELS

SYSTEM TAXONOM1C

LEVELS

EXAMPLES OF POSSIBLE

TAXONOM1C CATEGORIES & DIMENSIONS j

LEVEL

ONE

Nominal Measurement SYSTEM OBJECTIVES

Production

. Supply

. Navigation

Air Traffic Control

> Health & Welfare

. Transportation

Maintenance

Weapons

Surveillance

Etc.

LEVEL

TWO

Nominal

___,_ — — — ..,._..— SYSTEM FUNCTIONAL PURPOSES

Nominal

• Indirect command'control/guidance

operations

. Relatively direct control navigation

operations

« Maintenance operations

• Data or materials processing

Relative

Relative

. Command

• Control

• Information

. Date

LEVEL

THREE

Relative Measurement

(Ordinal, Interval and

Ratio)

STRUCTURAL CHARACTERISTICS

• Organization and layout

• Size

• Level of automation

• Implementation capabilities

OPERATOR'EQUIPMENT

CHARACTERISTICS

• Human skills, equipment conditions

• Human abilities & IQs, equipment

capabilities

. Values

. Needs

OPERATING CHARACTERISTICS

• Inputs to operator

• Operator processing

• Operator outputs

• Units being dealt with by system

• Environment

. Feedback

SUPPORT REQUIREMENTS

CHARACTERISTICS

• Materials (including people)

• Maintenance (including people)

Prom: Füuey et al. (197S)

Systems Taxonomy Model

A-137

-^ *,.» -.» « . - - - - - i
Finley, D.L., Obermayer, R.W., Bertone, CM., Meister, D., and Muckler, F.A.

Human performance prediction In man-machine systems-Volume I-A technical

review (NASA CR-l6m). Canoga Park, CA: Tie Bunker-Ramo Corporation, August

1970.

Topics Relevant

I to System Development

land Evaluation Technology

Topic I

No. i ABSTRACT

1.1

1.2

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.'I Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing ^ ?

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5-4 Conclusions and

Recoanendations

Further Research Areas

6.1 Measure«ent Sy.em

' imitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

Over the pas three decades there has been

an increasing demand for quantitative

techniques of human performance prediction

in man-machine system tasks. A somewhat

bewildering variety of methods has evolved

to satisfy this need, ranging from specific

task simulation to classical tests of

fundamental human abilities.

From the existing literature, 75 behavioral

dimensions were defined and incorporated

into a Performance-Descriptor X

Physical-and-Interactional-Categories

Matrix. This is shown in Figure 1.

Implicit in the adoption of the dimensional

approach to human performance prediction

was the assumption that it would be

possible to denote a set of specific

procedures which would define a

comparatively objective mapping process

wherein the accuracy of the mapper would be

more a function of available knowledge than

of the goodness of his intuition.

(Mapping, as used here, refers to the a^

priori selection of those sociopsychological dimensions which would be

required to perform an operational task.)

The basic objective of this program was to

critically review tests and test techniques

for human performance prediction. Such a

review, however, is best facilitated by

conceptual and methodological criteria. At

a very basic level, four fundamental

questions were asked:

(1) To predict what?

A-138

liil ^. •-
iIi ii*

ilii 1 8> 1 II

a!

j«

11t«

iiii 1 f

5?Il

!1

11

klllllh

Hi| 1 III!

jllHllli

1 1 *8 l •* 1

If*i

\\sM

111

HI1 a

H

II

llummWat R

?1i

11 ill

ii

?! I

It1I siii i J j . 1 8»

Hi il 11 : i! I S s 88 « » r If 1

...K ...

ItS1

||

;s

-j

lit

.11 il

iiw

II1 '

li

vi5!s(11—

|!

HI

IM

l!

|l inf 7

\

l

|

\i

1

£2

8

llii

l|IHmil {ill

il

1»r II

«s1

-1—

lb

l"! 1

1! 81 ii13

— H—

Ml 1 c

....

^ \U iJ i f |l j lj|ii»iil |

mlt

ü

!l

--—

ii

liHI HI ili

1-i1

•m1 i1a1

im

1 ii l 1rii

j

1t I

Tl

• iiI|||

Figure 1. The PerCormance-DescriptorX

Physlcal-and-Interactional-Categories

Matrix (presented asa human-task

A-139 »apping guide).

"• •.•

••- ------

-

-. v.-.

•v% ",'' >.•- •* t • J- ','• •

•
1 Topics Relevant

I to System Development I Topic]

land Evaluation Technology j No. ! ABSTRACT

(2) Upon what dimensions and 'measures?

(3) With what tools?

(4) For what purposes?

Human performance prediction in man-machine

systems must be concerned with three levels

of measurement analysis:

(1) System requirements and appropriate

system performance measurement.

(2) Human operator task analysis and the

performance measures related to that

level.

(3) Basic behavioral dimensions involved

in human task performance.

Further, the precise interrelationships

between these levels should be quantified.

Tests must be related to human performance

dimensions found in human operator tasks

which are executed to help achieve system

performance criteria. For tests to be

meaningful in man-machine systems,

quantitative transformations must be

possible between levels. This requirev!

mapping operation turns out to be a

formidable technical challenge.

Both the questions and levels of analysis

can be combined into a single conceptual

structure, as shown in Figure 2. The

question of purpose is external to this

matrix, but each of the first three

questions can be asked at each of the three

levels. The addition of an analytic

requirement to interrelate these levels

results in a Generalized Methodological

Model which can be (1) used to evaluate the

existing literature and (2) form a framework of requirements for future test development. To be general, one must at least

postulate a hierarchical system structure

consisting of many levels of embedded

functional units, e.g., total system,

subsystems/modules/etc...components, action

elements. It may be uncommon to find a

A-140

.-•--- -- --- • - ^ - ^

.--.

- - i - i . .
<

I Topics Relevant

S to System Development 1 Topic I

land Evaluation Technology ! No. I ABSTRACT

•• r.

•

SYSIS4

Group and System

Level of Analycis

»- .•

SYSTEM - MW;

System-Task Relationship

and Individual

Level of Analysis_

MA:;

Test Battery

Developnent

J:

Figure 2« A generalized methodological model

for human performance prediction.

A-141

i • ---•-

-.

- - ^- A\i"

--

-.-. . . - •.». • ,

l
I Topics Relevant

to System Development I Topic I

and Evaluation Technology ! No. I ABSTRACT

[t

complex system with such a clear-cut

hierarchical structure; many system

functional units may depend upon, or

influjnce, many o* .ers. Consequently, the

system organization may require a complex

hlock diagram to display, but such a block

diagram will also be required for other

system engineering purposes.

2.2 The analysis was used in the detailed

application of the human performance

prediction methodology (developed from the

approach implied by Figure 1) to a specific

man-machine system activity: the celestial

and space-object radiometry experiments

conducted during the Gemini V and the

Gemini VII missions.

4.1 Several intensive analyses were performed

at several levels to provide specific

answers to the above basic questions. The

analytic outputs were in terms capable of

quantitative measure and the relationships

between system, system-man and the human

operator levels of critical performance

were identified with respect to these

terms.

6.1 Three major problems were found to dominate

the literature on human performance

prediction tests in man-machine system

performance; first, elementary and

essential rules in test development have

been frequently ignored; second, modern

techniques for the development of

cost-effective tests and test batteries

through utility analysis have not been

used; and third, the basic issues in test

validity have been avoided. So long as

this "strategy" continues, the literature

will be extremely suspect. However, all of

these difficulties can be resolved.

:-

Many methods of task analysis exist within

the literature, but a very „borough review

of these methods failed to reveal any

particular method of direct usefulness,

showed the lack of standardization in the

field, and suggested tnat a new attempt at

a basic taxonomy was in order.

A-142

-^ UM

."- .' .'"" ."- ."• ." ."*
I Topics Relevant

to System Development !Topic I

!and Evaluation Technology ! No. i ABSTRACT

; ..-• As a result of trying out three different

behavioral taxonomies, it was found that

the effectiveness and usefulness of a

taxonomy is a function of the following

factors:

(1) How appropriate the level of detail is

to the purpose of the taxonomy.

(2) How cleanly separated and appropriate

the categories are.

(3) How objectively and thoroughly the

categories are defined.

(1) In the case of the analytic behavioral

categories, how completely the

taxonomy covers the behavioral domain.

A-143

f,l

--.,.... .

-

. -. -.

- • •*»* .*
Fischl, H.A., Siegel, A.I., & Wolf, J.J. Application of a multiple task

Interactive model: Simulation of human performance in sonar maintenance. In

G. W. Levy Ctt.), Symposium on applied model of man-machine systems performance (NR69H-591). Columbus, OH: North American Aviation, November 1968.

TÄ^697 939).

Topics Relevant

! to System Development I Topic

land Evaluation Technology ! No. ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

I.I Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.K General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Coaponents

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

M Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

1.1 The paper describes an application of a

digital computer simulation model aimed at

predicting maintenance task performance in

a sonar system under development. The

model provides for simulation of one or two

operators in a man-equipment interface

situation.

2.1 The application described involved a

simulation of the coordinated activities of

a sonar technician and sonar supervisor in

performing two representative maintenance

tasks. The simulation examined the impact

of different skill levels and time

constraints on task performance.

4.2 To employ the model, the task to be

simulated must be broken down into subtasks

(e.g., "depress pushbutton," "set toggle

switch," etc.). Input parameters define

time constraints, probability of successful

execution, subtask sequence, rnd a variety

of other actions.

5.3 A number of recommendations for the sonar

system under development were made based on

the findings of the model application.

Application Coaponents

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoanendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

^.3 Research Planning A-144

-.-V-W.^ ii a*—_- I »
.•

Foley, J.P., Jr. Criterion referenced measures of technical proficiency in

maintenance activltlea (Final Rep.). Wright-Patterson, AFB, OH: Air Force

Hunan Resources Laboratory, Advanced Systems Division, October 1975. (AD-A016

420).

! Topics Relevant I

I to System Development iTopic

land Evaluation Technology j No. ABSTRACT

State of the Art Review

of the Process

1.1 General Systeai

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPU)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.'» General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3-3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

2.1 This study addressed the performance of

maintenance personnel.

2.2 The maintenance man's mission in a

man-machine system is to ensure that the

machine subsystem is in prime operating

condition when the mission is started.

3.1 A model battery of 48 criterion referenced

Job Task Performance Tests (JTPT) were used

in this study.

3.2 A scoring scheme for the measurement of the

task performance ability of maintenance

personnel was developed.

3.3 The following classes of Job activities

were considered: equipment checkout,

alignment/calibration, removal/replacement,

soldering, use of test equipment, and

troubleshooting.

3.U After considering product, process and time

as to their appropriateness for scoring

results for each activity, it was decided

that a test subject had not reached

criterion until he had produced a complete,

satisfactory product.

3.5 Many factors were considered including the

identification and clarification of tasks

to be measured, the hierarchical relationship of maintenance tasks, the most effective order of measurement and the ease of

test administration.

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-145

Mritfn ------------ •-•-'-•- --- i

. ' —

- i
Topics Relevant

I to System Development I Topic',

land Evaluation Technology i No. I ABSTRACT

4.1 The battery of tests included 48 tests, 81

problems and 133 scorable products. A

profile for displaying th' results of these

tasks which attaches meaningful information

to these numbers wa: developed. This

profile contained information regarding a

test subject's success on the full range of

tests including a subject's Job abilities,

strengths, and weaknesses.

1

5.3 Due to the unavailability of test subjects,

the tryout was not as extensive as planned.

The tryout did indicate that the tests as

developed are administratively feasible.

It is felt that their use would result in

further modifications and improvements.

A-146

.
Foley, J.P., Jr. Performance measurement of maintenance. Wright-Patterson

AFB, OH: Advanced Systems Division, Air Force Human Resources Laboratory,

1977. (AD-A053 ^75).

Topics Relevant

I to System Development I Topic I

and Evaluation Technology I No. ! ABSTRACT

m *

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (ST>1)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.« General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3-3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

R'-conmendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 This paper discusses the status of

performance measurement for maintenance and

describes how formal Job Task Performance

Tests (JTPTs) have been replaced by paper

and pencil theory and job knowledge tests.

The paper states that research has indicated

that these latter methods have proved

unsuccessful in terms of measuring the

ability to perform maintenance tasks.

The author describes the Air Force Human

Resources Laboratory (AFHRL) efforts to give

consideration to the man-machine interface

in performance measurement. One result of

this effort has been the articulation of a

structure for handling maintenance functions

and their complex relationships in a

systematic manner.

6.2 Several problems concerning the research,

development, and implementation of

performance measurement were discussed and

the paper ended with a proposal for future

research and development efforts based on

what has been accomplished. Five general

areas of consideration were recommended:

(1) Refinement of Model JTPT Battery

(Electronic Maintenance).

(2) Refinement of Symbolic Substitutes

(Electronic Maintenance).

(3) Development of Model JTPT Battery

(Mechanical Maintenance).

(4) Development of Symbolic Substitutes

(Mechanical Maintenance).

(5) Job Aptitude Test Research based on

results of JPPT.

A-147

^

-_"-•••."-•'. ".-V" • -'. -'. •'. -'

- - - - •- ^ ^ - ^
Ford, J.P., Harris, J.H. & Rondiac, P.F. Performance measures for-AIT armor

crewman (HumRR0-CR-D2-71*-?). Alexandria, VA: Human Resources Research

Organization, April ig?4«. (Ar)-A019 375).

i Topics Relevant

I to System Development {Topic

land Evaluation Technology I No. ABSTRACT

1. State of the Art Review

of the Process

1.1 General Systen

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedur««

1. Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

1.3 Apparatus for Testing

M.K Personnel for Testing

1.5 Test Plans

5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoonendations

6. Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 This report summarizes the procedures

followed in developing performance measures

for AIT (Advanced Individual Training)

Armor Crewmen.

The objective of the study was to establish

definitive performance measures for those

requirements considered relevant to job

entry and to present these measures in the

training objective format.

3.1 The performance measures to support

revisions in the Armor Crewmen AIT

program were developed in three phases.

(1) Development of a tasklist.

(2) Written objectives and tests.

(3) Selected tasks for training.

In the first phase a working committee

(representatives from the Training Center,

HumRRO and the Armor School) developed a

task inventory. Following this activity

each task was coded to reflect level of

mastery that a man should achieve prior to

entering an Armor unit. In phase 2

training objectives were developed. The

goal of these objectives was to define

entry level mastery for each task. The

objectives included conditions for task

performance and standards to evaluate

proficiency of task performance. The third

phase was to identify those tasks that were

appropriate and feasible for AIT.

A-148

.*v:-v '"*"-* -''•-"- -•- -' *

:

" ' - ' -
I Topics Relevant I

I to System Development iTopic

land Evaluation Technology | No.

v

3.2

In instances where time standards were not

available in published doctrine, they were

recommended by subject matter experts.

When selecting the tests to be used for

training, they were submitted to expert

reviewers to assure accuracy of

checkpoints, appropriateness of time limit:

and feasibility of administering tests.

The majority of the tasks involved an

observable procedure and most of the

standards are tests of the process in

performing the tasks. A time limit is

included on the assumption that time to

perform a task is an integral component of

mastery.

i

.- •_

A-149

, -»-.v- .>. / .' ii
Geddie, J.C. Profiling the characteristics of the developmental test

participant (Tech. Memo. 31-76). Aberdeen Proving Ground, MD: U.S. Army

Hunan Engineering Laboratory, October 1976. (AD-A031 563).

I Topics Relevant

! to System Development

land Evaluation Technology

1

!Topic

i No. ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.'I Performance

Criteria, Specific

3-5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

It.2 Parameter Determinations

1.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoonendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 This report discusses methods to control

the variance contributed by the human

operator which Influence the total system

performance.

1.3 It is proposed that human operators of the

system which is being evaluated be selected

according to criterion which eliminates

personnel who fall at extreme ends of the

distribution of values of relevant

characteristics.

Selection criteria should include:

(1) Military Occupational Specialty (MOS),

complete with skill level suffix.

(2) Physical dimensions.

(3) Sensory acuity.

When selection has taken place, a more

detailed description of his/her

characteristics should be obtained as

follows:

(1) Actual value for the participant on

each of the selection criterion

iieasurements.

(2) Scores from aptitude and MOS tests.

(3) Experience (in months) in the relevant

MOS.

(M) Total service time.

(5) Other related training, results of

tests, level of performance attained.

A-150

-.■■.-.

.- - -"• . • ."• ."• . •.-,'

!..• . ' ._• ^- I • ^- ^- i- .
\ \*

i Topics Relevant

I to System Development {Topic

land Evaluation Technology ! No. ABSTRACT

(6) Education.

It is noted that this procedure is not a

radical change from current procedures. It

suggests that some consistency be imposed

in the selection process.

5.1* It is felt that a long-run benefit from the

proposed approach would be the development

of performance based selection criteria.

If a data base is built, it should be

por.sible to develop a profile of levels of

the characteristics which are required by

similar tasks in other systems.

..-

A-151

-•- '-•.-. -. .••_ v". /., , .«-, J
Geer, C.W. Navy manager's guide for the test and evaluation sections of

MIL-H--46855 (DW-IOOOo^). Seattle, WA: Boeing Aerospace Company,

Logistics Support and Services, June 30, 1977. (AD-A0M5 098).

I- Topics Relevant |

I to System Development I Topic

jand Evaluation Technology ! No. ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1) Performance

Criteria, Specific

3-5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

1.2 Parameter Determinations

1.3 Apparatus for Testing

1.1 Personnel for Testing

1.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.1 Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potential«/

Priorities

6.3 Research Planning

2.1 Two general types of test and evaluation

methodologies were considered:

(1) Development Test and Evaluation (DT4E)

and

(2) Operational Test and Evaluation

(0T4E). A third category. Production

Acceptance Test and Evaluation (PAT&E)

consisted of testing production items

to demonstrate that contract requirements of the system were met.

2.2 HFE TiE (Human Factors Engineering - Test

and Evaluation) existed to

(1) Demonstrate system, equipment, and

facility design conformance, to human

engineering design criteria.

(2) Determine man's contribution to

performance requirements.

(3) Quantify man-machine interactive

measures of system.

(4) Detect undesirable design on

procedural features of system.

Through these steps, this guide was

designed to match particular techniques to

particular applications, and describes how

to utilize the techniques.

3.1 The "real world" requirements that the HFE

needs to verify in order to optimize the

man-machine interface were technical

.

P:

.•

A-152

.-.•. •.• _. --..',•^ ^ •_ -. LV •.--.'•.- •L-V'.^S."
I Topics Relevant I

I to System Development !Topic!

and Evaluation Technology | No. ! ABSTRACT

requirements of human performance, design

criteria, safety, training, personnel

skill/quantity, technical publications, and

life support criteria; decision-making

structures, data inputs, timing, level of

detail, applications.

4.1 T4E techniques consisted of direct manual,

system measurement, indirect manual,

automatic recording, physiological, and

simulation.

A-153

., .^.• -.-:.' - . • . ', •". •> . -", --, • .• .
Geer, C.W. User's guide for the test and evaluation sections of H1L-H-H6855

(D194-1006-1). Seattle, WA: Boeing Aerospace Company, Logistics

Support and Services, June 30, 1977. (AD-AOUS 097).

Topics Relevant I

to System Development I Topic

and Evaluation Technology ! No. ABSTRACT

1. State of the Art Review

of the Process

1.1 General System

Measurements

1.2 Systec Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

See abstract of similar document: Geer,

C.W. Navy manager's guide for the test and

evaluation sections of MIL-H-46855

(019^-10006-2). Seattle, Washington:

Boeing Aerospace Company, Logistics Support

and Services, June 30, 1977.

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3- Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

1. Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.H Personnel for Testing

4.5 Test Plans

5. Application Components

of the Process

5.1 Test Execution

i.l Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoomendati ons

6. Further Research Areas

6.1 Measurement System

Limitations

6.2 Research PotentiaTa/

Priorities

6.3 Research Planning

A-154
v

George, C.E. 4 Dudek, R.A. Performance, recovery and tran-machlne effectiveness; Final report on a basic research program under project THEMIS (Tech.

Memo. 9-71»). Lubbock, TX: Texas Tech University, April 1974. (AD-777 797).

i Topics Relevant I

I to System Development I Topic

land Evaluation Technology ! No. ABSTRACT

1. State of the Art Review

of the Process

w :•

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environnent Definition

2.« General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Cooponents

of the Procass

3-1 Practical Measurable

Attributes

3-2 Practical Attribute

Measures

3-3 Performance

Requirements, Specific

I.« Performance

Criteria, Specific

3-5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Coaiponents

of the Process

5.1 Test Execution

5.2 Data 'nalysis

5.3 Findings Interpretation

5.4 Conclusions and

Recomaendatlons

1.1 This report is a final summary of many

individual and interdisciplinary group

studies conducted as a part of the total

project. Each study had the goal of the

project, performance and recovery of man

within a work system, as a basic direction.

In many work oriented groups it is required

that missions be fulfilled while in, or

Just after being in, environments such as

noise, vibration, or unfavorable climatic

conditions. All of those factors affect

human performance drastically. Other

variables which affect human performance

include task demands, motivational level,

type of organization, and nutritional

status, as well as others. It was the

purpose of this research program to

generate basic data concerning human

performance and recovery under the

conditions mentioned above. Experiments

were conducted to study the effects of

these variables singly and in combination

on human performance and recovv y.

All experimentation was conducted within

the framework of a task model as follows:

(1) Items Measured (Dependent Variables):

a. Performance Responses - Latency,

accuracy, length of time,

Increments and decrements, etc.

Further Research Areas

6.1 Measurement System

Limitations

6.2 Reaearch Potentials/

Priorities

6.3 Research Planning

A-155

• ..•-.*..•
Topics Relevant

to System Development I Topic

and Evaluation T^ohnology j No. ABSTRACT

b. Physiological Responses - Oxygen

consumption, GSR heart rate, skin

and/or core temperature, urine

analysis, etc.

c. Training Responses - Subjective

states or moods, attitude,

learning, etc.

(2) Measurement Conditions (Independent

Variables):

a. Task Demands - Amount of work,

complexity, difficulty, work-rest

cycles (duration), and ambiguity.

b. Level of Motivation -

Interpersonal response traits,

drive state, reinforcement value,

amount of deprivation,

instructional set, etc.

c. Nutritional History - Meal

spacing, intake, social class,

ethnic group membership.

d. Environments - Effective

temperature, vibration, normal,

lighting conditions, etc.

e. Work Systems Setting -

Individual, team, crew.

(3) Tasks Included:

a. Man as a Machine

b. Man as a Machine User

c. Man as a Machine Controller

d. Man as a Machine Servant

e. Man as a Social Interactor

In addition to the task model, research was

guided by the military Interest in

continuous operations. The research team

emphasized experimentation with relatively

long-term repeated measurement designs.

A-156
Gephart, L.S. & Balachandran, V. A stochastic system effectiveness simulation

odel (Tech. Rep. UDRI-TR-70-12). Dayton, OH: University of Dayton-Research

Institute, June 1969. (AD-865 ^9).

i_ Topics Relevant I

I to System Development I Topic!

jand Evaluation Technology I No. j ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 Syste« Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model I)

Contextual Component

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3. ^ Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5-3 Findings Interpretation

5.4 Conclusions and

Recoanendati ons

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 The worth of any system Is directly

relatable to the confidence the user has In

its ability to perform its designated

tasks. Systems effectiveness and its

fiscal corollary, cost effectiveness,

consltltute the most important area of

concern to research and development

management. As a result the Weapon System

Effectiveness Industry Advisory Committee

(WSEIAC) was established to develop a

technique to apprise management of current

and predicted system effectiveness at all

phases of sj stem life. The technique known

as the WSEIAC model was developed for this

purpose.

This report augments the basis WSEIAC model

development in three areas. One of these

areas is the partitioning of system

capability into hardware and personnel

adequacy.

System effectiveness may be defined as the

probability that the man-machine complex

system will successfully meet an

operational demand and fulfill the

predetermined mission objective within a

given mission time when operated under

stated conditions.

A-157

•. . •. • ,
Gex, R.C. Personnel subsystem testing and evaluation for missiles and space

systems; An annotated bibliography (SB-61-21). Sunnyvale, CA: Lockheed

Aircraft Corp., Missiles and Space Division, April 1961.

I Topics Relevant |

i to System Development I Topic

jand Evaluation Technology I No. ABSTRACT

1. State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (ST*)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.14 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1) Performance

Criteria, Specific

3.5 Measurement Procedures

4. Planning Components

of the Process

4.1 Analytic Methods

it.2 Parameter Determinations

4.3 Apparatus for Testing

1.4 Personnel fo." Testing

4.5 Test Plans

1.1 In the face of radical restrictions to

traditional psychometric approaches,

progress has been made in personnel

measurement programs in connection with

weapon systems developed by all three major

branches of the military. It was felt that

this program could not be found in single

sources and therefore this literature

search was designed to provide an annotated

bibliography in four major areas:

(1) Performance evaluation and unit

proficiency.

(2) Personnel requirements.

(3) Training and training equipment.

(U) Human engineering.

5.4 Although many of the reports are concerned

with equipment design, training development,

and generation of personnel requirements

information, it is very likely that the

rationales and criteria adopted for such

activities can provide guidelines for the

subsequent test and evaluation. It is

believed that this bibliography will be

useful to personnel subsystem specialists

in particular.

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Rec omenda 11 ons

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-158

. . . .
Goldbeck, H.A., Wright, K.A. i Fowler, R.L. Operator performance and panel

layout for discontinuous tasks (AMRL-TR-70-137). Palo Alto, CA: Philco-Ford

Corp., WLD Division - Human Engineering Section, March 1971. (AD-727 791).

I Topics Relevant I

to System Development I Topic

and Evaluation Technology I Wo. ABSTRACT

.^ -'

1. State of the Art Review

uf the Process

1.1 General System

Measurements

1.2 System Taxorooy

Model (STK)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environaent Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1» Performance

Criteria, Specific

3.5 Measurement Procedures

4. Planning Components

of the Process

1.1 Analytic Methods

4.2 Parameter Determinations

1.3 This is a follow-on to a previous investigation of panel layout. The purpose of this

study is to determine whether or not previous findings generalize to discontinuous

action sequences.

2.1 The system under study is a panel layout,

controls, display hardware and operators.

2.2 The arrangement or layout of controls and

display should optimize operator

performance.

4.1 Analyses of variance were conducted on the

two major dependent variables, total operating time per trial, and total errors of

omission and commission per trial.

4.3 The basic apparatus consisted of four major

components: a subject's console; an

experimenter's console; a diode matrix;

recording and auxiliary equipment.

4.1 Subjects were 200 male college students

between the ages of 18 and 26 years, who

met certain physical and scholastic

criteria. All subjects were paid for their

participation.

n 3 Apparatus for Testing

4 H Personnel for Testing

4 5 Test Plans

Application Components

of the Process

5 1 Test Execution

5 2 Data Analysis

5 3 Findings Interpretation

5 4 Conclusions and

Recoonendations

Further Research Areas

6 1 Measurement System

Limitations

6 2 Research Potentials/

Priorities

6 3 Research Planning

A-159

•- -.
I Topics Relevant | \

I to System Development I Topic I

land Evaluation Technology | No. ! ABSTRACT

-

1.5 The experimental design was formed by

combining factorially three experimental

task operations with four principles to

produce 12 panels or groups of 10 subjects

each. The principles applied were:

Sequencing (SEQ), Functional Grouping (FG),

Location by Frequency (FREQ) or Location by

Importance (IMP).

The action sequence was made discontinuous

by dividing the dominant action path of the

continuous study into twelve coherent

segments with each segment beginning with a

display and ending with a control action

followed by a delay period.

5.3 The major finding of this study is that the

overall superiority of the Sequencing

Principle has been confirmed using a

discontinuous presentation of the

operator's task. It is concluded that the

most powerful tool available to the control

panel designer is optimum application of

the Sequencing Technique.

A-160
Greening, C.P. Unfinished business in the utility of visual detection models.

In G.W. Levy (Ed.), Symposium on applied model of man-machine systems performance (NR69H-591). Columbus, OH: North American Aviation, November

TAD=697 939).

15 per:

19687

! Topics Relevant

I to System Development

land Evaluation Technology

iTopic

I No. ABSTRACT

1. State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, ultimate

3. Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.4 Performance

Criteria, Specific

3.5 Measurement Procedures

>>. Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

6. Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.3 The paper describes a visual detection

model that has been subjected to a rather

thorough validation study. The model

provides estimates of target acquisition

probability as a function of range. These

estimates are used as a major input to an

overall system effectiveness model. The

visual detection model is a "glimpse

model", i.e., the cumulative probability of

target acquisition is obtained as the

product of a series of single glimpse

probabilities.

3.5 Validation of the model was obtained by

comparing the model's predictions for 3^

selected targets with the results obtained

in an experiment in which a number of

observers viewed a motion picture presentation of a flight over those targets. The

overall product-moment correlation between

predicted and experimental median ranges

was +.53.

M.2 Certain "unfinished business" problems

remain with each of the parameters involved

in the model. These parameters are:

(1) The probability of looking at the

target.

(2) Sensor resolution.

(3) Inherent object/background contrast.

(1) Sky/ground luminance ratio.

A-161

^ . • . • .
i Topics Relevant

I to System Development I Topic I

and Evaluation Technology ! No. ! ABSTRACT

(5) Aircraft altitude above the target.

(6) "Visihility" or meteorological range.

(7) Threshold contrast.

(8) Target dimensions.

(9) Number of required resolution

elements.

6.1 A number of parameters currently are

"missing" from the model, including:

(1) Level of confidence [of the observer],

(2) Briefing and reference materials.

(3) Search techniques.

At present it is not known how these

parameters should be handled.

A-162
Grunzke, P.M. Evaluation of the Automated Adaptive Flight Training System's

air-to-air intercept performance measurement (AFHRL-TR-78-23)• Williams AFB,

AZ: Flying Training Division, July 1978. (AD-A060 320).

I Topics Relevant I

to System Development I Topic I

land Evaluation Technology i No. I ABSTRACT

State of the Art Reviev.

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Envirunment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Perfornance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1» Performance

Criteria, Specific

3-5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendat i ons

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.3 The objectives of this study were to assess

the effectiveness of a performance measurement package and to evaluate the efficiency

of operational test and evaluation procedures developed as a data gathering tool

for performance measurement. An operational

test and evaluation was conducted on the

capabilities of the Automated Adaptive

Flight Training System (AAFTS) system prior

to this study. The data gathered during

the course of that effort provided the data

source for this evaluation.

2.1 The AAFTS was Installed on the F-4E #15

Weapon System Training Set (WSTS). The

subject of this evaluation was the aircrew

skills in Air-to-Air Intercepts (AAI).

2.2 The AAI program is broken into nine modes:

(1) Single turn attacks

(2) Increasing/decreasing aspect attacks

(3) Stern conversion attacks

(4) Stern conversion ID passes

(5) Snap-ups

(6) Single turn attacks with stern

conversion reattacks

(7) No lock-on attacks

(8) No Ground Control Intercept (GCI)

attacks

• -

A-163

- - - . - 
I Topics Relevant ! I

I to System Development !Topic!

land Evaluation Technology i No. ! ABSTRACT

1

L*

L"

•

(9) Tactical intercepts

Each of the above modes has a number of

steps within it. The steps were designed

' o present progressively more difficult

intercepts as a function of the values of

certain variables.

3.1 In the AAI environment, variables often

measured the geometry of attack, missile

launch parameters, and other variables

deemed important to an intercept. The

AAFTS device used a scoring system of this

nature. A set of 28 dependent variables

were determined by the Operational Training

Development Team (OTDT). These variables

included measures of how closely the crews

stayed to assigned airspeed; altitudes and

headings; how accurately the crew launched

missiles; whether or not correct missile

launching control procedures were followed/

accomplished; and other variables specific

to particular types of AAIs.

3.2 All variables had acceptable bandwidths of

performance that the crew had to maintain

in order to accumulate scoring points.

While the AAFTS has a 28 variable standard

format, only these variables relevant to

the particular engagement underway were

scored by the AAFTS system.

U.I T-statistics were used to test differences

between the total scores (a summation of

all the variables measured for each

engagement completed) achieved by the crews

for each mode. They were also used to test

differences between two types of crews and

on a variable-by-variable basis across

modes wherever the particular variable was

measured. The AAFTS device had specific

point values assigned to performance error

tolerance limits for each variable. An

equal interval scoring technique was also

employed to more accurately satisfy

assumptions on which the statistic is based

(i.e., a variable with scored values of 10,

8, 6, 3, 0 was arbitrarily rescaled to 4,

A-164
i Topics Relevant I

I to System Development I Topic I

land Evaluation Technology j No. I ABSTRACT

3, 2, 1, 0) and then between-groups

differences were examined by mode and by

variable.

M.B The AAFTS was a parasitic computer-based

Instructional device Interfaced with the

F-ME WSTS 115. The system can automatically score advancing aircrews through

various programs Including a 116-step AAI

program.

M.4 Seven operationally qualified crews and

five student crews participated In the

study.

4.5 Training received by both crews was

constrained by Tactical Air Command

training requirements and 0T4E limitations.

The operational crews received experience

in flying the F-4B WSTS #15 and the student

crews were instructed using the AAFTS

training device. Crews flew and were scored

on nine different types of air-to-air

Intercepts that were programmed into the

AAFTS device. The crews flew the simulator

under different conditions—the operational

crews were adaptlvely scheduled by the

AAFTS as often as possible while the

student crews flew what was dictated by the

instructor.

5.3 The data revealed significant differences

favoring the operational crews in two types

of attacks, single turns and stern

conversions. On a variable-by-variable

basis, there were three significant

differences between operational and student

crews, two of which indicated superior

performance by student crews.

5.4 The data warranted the following

conclusions:

(1) The AAFTS as an Instructional tool has

the potential to provide standardized

and objectively scored training/

evaluation sceneries for aircrews.

A-165

/ '
•

I Topics Relevant | |

to System Development {TopicI

and Evaluation Technology ! No. ! ABSTRACT

(2) The performance measurement package

performed adequately as an

informational feedback tool but

requires more research to select and

validate variables that discriminate

better between aircrew skill levels.

Failure to discriminate could have

occurred because the variables and

their weights were based on the

opinions of subject matter experts and

not on empirical evidence.

(3) Use of the OT&E procedures was

minimally effective for a performance

measurement but served well as a means

for acquiring aircrew subjective

impressions on the systems' overall

trailing/evaluation potential.

A-166
Gustafson, H.W. The measurement of proficiency of Air Force maintenance

personnel (ATPTRC SM 7-1957) Denver, CO: Air Force Personnel and Training

Research Center, Air Research and Development Command, Lowry AFB, 29 July

'967. (AD-SMI 711).

Topics Relevant

I to System Development

land Evaluation Technology ABSTRACT

IV'

1. State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.11 Performance

Criteria, Specific

3.S Measurement Procedures

4. Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

6. Further Research Areas

6.1 Measure: en System

Limitations

6.2 Research Pot mtials/

Priorities

6.3 Research Flawing

1.3 The purpose of this paper is to describe

some of the deficiencies in Air Force

proficiency assessment and to indicate

broadly how these deficiencies can be

corrected. In addition, suggestions are

given for promising immediate research and

development efforts.

2.1 Air Force maintenance personnel and their

training was the system addressed.

2.2 The training must accomplish the job of

making the maximum number of personnel

adequately proficient during the training

period.

2.M The Air Force competes with industry for

the services of trained personnel and is

not usually able to retain its personnel

longer than a four-year tour. There is a

need to train the maintenance personnel at

a great variety of tasks, some of which

require high levels of skill in a short

period of time without further education,

cross-training, or re-training.

2.5 The researchers showed that there are no

suitable over-all objective measures of

performance and there was no ultimate

criteria against which training standards

and intermediate measures of achievement

can be evaluated.

5.1 It was concluded that the many deficiencies

in current proficiency measurement of

maintenance personnel can be corrected

through a vigorous program of development

A-167
I Topics Relevant

I to System Development {TopicI

land Evaluation Technology ! No. I ABSTRACT

and implementation. However,_such a

program must have, as an integral aspect,

the definition of training and job

performance standards ^or each Air Force

maintenance position. Only whtn

performance is assessed against such

standards can all of the requirements for

proficiency measurement be met.

It is recommended that the definition of

training and job performance standards for

selected critical maintenance positions be

undertaken in association with the

development of performance measures in

order to serve as a model which can be

applied to other Air Force maintenance

positions.

6.2 Recommendations were made for future

research. They included:

(1) Research to refine principles anJ

techniques for assessing individual

performance in trouble: ooting and

other complex tasks.

(2) Research be sustained on improving

maintenance records and supervisers'

ratings as criteria of performance on

the job.

(3) A practical, procedural handbook be

developed which can be followed in

assessing performance capabilities.

(U) That the preceding efforts be aimed at

specifications for the development of

proficiency measures for inclusion in

weapon system development contracts.

A-168
Haight, F.A. Indirect r.ethoda for ncaaurlng exposure factors as related to the

Incidence of aotor vehicle traffic accidents (DOT-HS-800-601). State College,

PA: Pennsylvania State University, September 1971. (PB-205 031).

I Topics Relevant

I to System Development iTopic

land Evaluation Technology ! Ho. ABSTRACT

1. State of the Art Review

of the Process

1.1 General Systeu

Measurements

1.2 System Taxonomy

Model (STM1

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1) Performance

Criteria, Specific

3.5 Measurement Procedures

H. Planning Components

of the Process

1.1 Analytic Mtahods

1.2 Parameter Determinations

4.3 apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

5. Application Coaponents

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5-4 Conclusions and

Recomnendatj ons

6. Further Research Areas

6.1 Measurenet.t System

Limitations

6.2 Research Potentials/

Priorities

Ö.3 Research Planning

1.3 This article dealt with exposure of

categories of driver-vehicles to motrr

vehicle traffic (exposure in the narrow

sense) ur to all external hazards (exposure

ii the wide sense).

3.5 A nvrJer of different models for the

measurement of exposure were considered.

5.3 The Koornstra method showed sufficient

agreement with real data to justify further

exploration.

5.4 The next step would be validation through

use of other categorization, other time

periods and other locations than used.

A-169
Hakansan, N.H.

(RM-5238-PR).

9*2).

An adaptive method of test selection In systec development

Santa Monica, CA: The RAND Corporation, April 1967. (AD-653

I Topics Relevant

I to System Development {Topic

land Evaluation Technology ! Wo. ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.H General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.li Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

1.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Coaponents

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

Further Research Areas

6.1 Measurement Sysvem

Limitations

6.2 Research Poteotia^s/

Priorities

6.3 Research Planning

1.3 In this report an adaptive model of the

development process in weapons systems is

presented which enables decision-makers to

determine which tests, if any, should be

performed at a given stage and what

corrective actions, if any, should be taken

when test results are known.

The final checkout p!;ase of the system is

considered first. A model developed for

the purpose of determining the optimal

troubleshooting procedure when an operating

system fails is an appropriate abstraction

of this phase. The model is then

generalized to include the case in which

the object tested is destroyed. In

addition, sufficient conditions for the

optimality of the test procedures are

given.

Next, the problem of test selection In

early stages of development is superimposed

on the final checkout phase. A dynamic

programming formulation is given which

enables one to determine which test, if

any, should be performed at any given stage

and what corrective action, if any, should

be taken. As a by-product, means for

determining net benefits of a given test at

a given stage is obtained.

The situation in which there is serial

dependence among certain test Information

messages is examined. Finally, it is shown

that a moael of parallel research and

development efforts may be viewed as a

special case of the model developed in this

paper.

A-170
i Topics Relevant

I to System Development iTopic

land Evaluation Technology ! No. ABSTRACT

5.M While this model is presented in terms of

relatively simple systems and tests, it is

capable of handling systems and tests of a

highly complex nature.

UA-171
Hall, D.H. Confidence llmlta for measures of effectiveness in weapon aystcm

analysis (NWC TP 5437). China Lake, CA: Naval Weapons Center, May 1973.

(AD-762 U01).

I Topics Relevant I

I to System Development I Topic

land Evaluation Technology | No. ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Ana' tic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3-5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.3 This report justifies the use of

established statistical methods to obtain

confidence intervals for measures of weapon

system effectiveness.

2.1 These methods apply to effectiveness

measures that are generated by sampling

mathematically simulated weapon/target

encounters.

4.1 There are two general sources of

unreliability in computer simulations. (1)

the computer program does not model the

real world as closely as it should and (2)

the number of encounters simulated is

insufficient to yield statistically

significant conclusions. The first sources

of unreliability can be revealed and

remedied by a model validation procedure.

The second source of unreliability can be

reduced by standard statistical methods by

determining the proper sample size.

It is demonstrated in this report that

relatively straightforward statistical

techniques can be used to generate

confidence limits. Methods applicable for

use with continuously distributed functions

and with zero-or-one measures of

effectiveness are briefly described with

detailed illustrative examples.

A-172
Hammell, T.J., Gaateyer, C.E. & Pesch, A.J. Advanced officer tactics training

device needa and performance meaaurement technique (NAVTRAEQUIPCEN72-C-0053-1,

Vol. 1). Groton, CT: General Dynamics/Electric Boat Division, November 1973.

(AD-922 929L).

Topics Relevant I

to System Development ITopic I

and Evaluation Technology I No. i ABSTRACT

ftate of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STK)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.'I Performance

Criteria, Specific

3.5 Measurement Procedures

Hanning Components

of the Process

4.1 Analytic Methods

k.Z Parameter Determinations

•1.3 Apparatus for Testing

IM Personnel for Testing

1.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recooaendations

1.1 This study was conducted to determine the

advanced tactics training device needs for

submarine officers and to develop a

technique for measurement of tactical

training performance.

2.5 The determination of the requirements for

the training and devices wa? made through

the use of operational task analysis data,

direct observation and discussions with

qualified Navy personnel concerning the

existing tactics training program and

devices.

3.5 Recommended tactical team trainer

modification consisted of a diagnostic

feedback display, a sophisticated

instructor's console, a knowledgeable

opponent capability, automatic performance

measures, a standard training computer, and

a trainee performance library.

4.1 A performance measurement technique was

based on a mathematical weapons system

effectiveness model to measure trainee

tactical performance.

5.1 Application of the measurement technique to

applied tactical scenario examples

demonstrated its potential for performance

evaluation.

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-173
Hanifan, D.T. & Knowles, W.B. Human performance in system effectiveness

modeling; Isaueh, approaches and critique. La Jolla, CA: Dunlap and Assoc,

Inc., December 1968.

Topics Relevant

I to System Development

land Evaluation Technology

i

IToplcl

i No. i ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Mode] (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.t General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.'» Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

H.l Analytic Methods

4.2 Parameter Determinations

1.1 This report is concerned with human factors

engineering methodologies. It is

particularly concerned with quantitative

methods apolicable to system engineering

efforts.

The role of human factors engineering In

each phase of system development is

summarized. The methods used by human

factors personnel in performing the

activities are largely heuristic. They

include both quantitative and

non-quantitative techniques. The variety

of non-quantitative techniques is

illustrated in the following table.

Quantitative techniques come from both the

physical and behavioral sciences.

«t.3 Apparatus for "esting

4.4 Personnel for Testing

4.5 Test Plans

Appl ication Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoonendations

Further Research Areas

6.1 Measurement System

Limitationt;

6.2 Re»'arch Potentials/

Priorities

6.3 Research Planning

A-174
Topics Relevant 1—r

to System Development I Topic I

and Evaluation Technology ! No. I ABSTRACT

Illustrative Non-Quantitative Techniques in Human Factors Engineering

NAPRATIVE

Position Description

Job Description

Task Analysis

TABULAR GRAPHIC

Task Equipment Analysis Operational Sequence

Diagram

Task Analysis of

Procedures

Maintenance Sequence

Diagram

LISTS AND RATING SCALES

Equipment Evaluation

Guide

Human Engineering

Checklist

Position-Equipment-Task Time Line Charts

Summary

Task Analysis Worksheet System Analysis and

Integration Technique

System Requirements and Integration Task Index

Capability Analysis

Maintenance and Handling Mission Profiles

Analysis

Physical Demands Form Operational and Support Link Analysis

Data Collection Sheet

Job Psychograph Performance Observation Function Flow Diagram

Record Form

Work Characteristic Form Performance Factor Form Multiple Process Chart

Man-Machine Chart

Task Analysis for Error Information-DecisionIdentification Action Chart

A-175
i Topics Relevant

I to System Development

!and Evaluation Technology ABSTRACT

In defining an approach to the problems of

incorporating human factors into system

effectiveness modeling three basic issues

should be considered:

(1) The ways in which models are used

(2) The kinds of human engineering data

that are available,

(3) The ways in which people are used in

systems.

A representative sample of some of the more

important types of models are discussed.

These include:

(1) On-line simulations.

(2) Control theory models.

(3) Process models.

(4) Task analysis models.

(5) Human reliability models.

(6) Operability index.

(7) Technique for human-error rate

prediction (THERP).

(8) Technique for establishing personnel

performance standards (TEPPS).

(9) "Logical man".

(10) Special models.

A-176
Hansen, D.N., Johnson, B.F., Fagan, R.L., Tarn, P.L., & Dick, W. Computer-based

adaptive testing models for the Air Force technical training environment Phase

I: Development of a computerized medsurement system for Air Force technical

training (AFHRL-TR-74-M 3). Tallahassee, FL: Florida State University, July

1974. (AD-785 1^2).

- >

Topics Relevant | I

1 to System Development |Topic 1

land Evaluation Technology I No. | ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.t Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Coaponents

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoosendations

1.1 The purpose of this study was to explore

the utility, from a psychometric and cost

effectiveness standpoint, of a computerized

adaptive measurement system in the Air

Force technical training environment. This

phase of the study was designed to produce

an operational system ready to test

technical training students adaptively.

3.1 For the two blocks of instruction, a task

analysis was performed and appropriate

measurement items selected.

3.2 A review of the literature indicated that

two testing techniques showed promise:

flexilevel testing and hierarchical

testing. These procedures were modified by

adopting a two-stage approach whereby a

student would be branded into the testing

net according to a regression estimate of

his predicted score.

3.5 Two courses were selected to implement

these procedures: one was selected for

implementation of the hierarchical testing

and the other for flexilevel testing.

1.1 The selected measurement items were

incorporated into a computer program for

adaptive testing. The testing procedures

were programed in the TUTOR language

supported by the PLATO system at the

University of Illinois.

Farther Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-177
I Topics Relevant !

I to System Development ITopicl

land Evaluation Technology ! No. ! ABSTRACT

I

5.M Preliminary conclusions indicated that

adaptive testing offers the potential for

time savings up to 501. It was found that a

feasible computer system to drive the

adaptive testing strategies could be

relatively easily developed. The file

handling and report generating capabilities

of the PLATO system were found to require

considerable ingenuity in programming.

6.2 Three studies were designed to evaluate the

adaptive testing approach: (a) a study to

test and validate flexilevel testing, (b) a

study to test and evaluate hierarchical

testing, and (c) a study to explore testing

of the examinee in the criterion zone.

w

*

A-178
[>

Hansen, D.H., Ross, S. & Harris, D.A. Flexilevel adaptive testing paradigm;

Validation In technical training (AFHRL-TR-77-35-1-). Memphis, TN: Memphis

State University, July 1977.

I Topics Relevant I

I to System Development !Topic

land Evaluation Ttchnology ! No. ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.') General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

i|. 1 Analytic Methods

U.2 Parameter Determinations

1.3 Apparatus for Testing

k.U Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recomnendatlons

1.3 In this report, means to empirically assess

a computer1 zed adaptive testing model in an

ongoing technical training system were

presented.

2.1 The system studied was an Inventory

Management/Materiel Facilities Training

Course for Air Force personnel.

3.1 The achievements of students in one section

of the course were measured.

3.2 An adaptive testing model was utilized.

This model was a modification of Lord's

(1971) flexilevel paradigm and consisted of

(a) the sequencing of test items in a

difficulty hierarchy, (b) adaptive entry

into the test by students at a difficulty

level appropriate to their predicted score,

and (c) systematic movements of students up

and down the hierarchy based upon their

performance on preceding items.

U.4 Four hundred and forty four airmen

participated in this study. The student

population selected was considered fairly

homogeneous in characteristics pertaining

to age, educational background, career

goals, and military experience. The

typical student was male (759 m, 259 f) an

average age of 20, high school graduate

with less than one year Air Force

experience.

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-179
j Topics Relevant

I to System Development iTopic

{and Evaluation Technology ! No. ABSTRACT

4.5 Preparation activities included meetings

with course instructors for briefings.

Students were administered reading aptitude

tests to provide predictor performance

scores. Test items were administered

separately with rate of presentation

determined by the student.

::

5.3 The results of this testing procedure

revealed a high positive part-whole

correlation between flexilevel and total

test scores. Internal consistency indices

were essentially equivalent. The

flexilevel test required nine fewer items

than the entire test, yielding a length

reduction of 39.551 and time savings of over

18%. The interpretation of findings

stresses the potential benefits of adaptive

testing in terms of significant time

reductions and maintenance of high

standards of test validity.

A-180
•y

: ;

Harrison, W.L. A theoretical basis for the concept of effectiveness (Master's

Thesis). Monterey, CA: Naval Post Graduate School, October 1966. (AD-807

386).

I Topics Relevant I I

I to System Development iTopid

iand Evaluation Technology j No. i ABSTRACT

1. State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.^ General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

4. Planning Components

of the Process

1.1 Analytic Methods

1.2 Parameter Determinations

1.3 Apparatus frr Testing

t.'J Personnel for Testing

4.5 Test Plans

5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.** Conclusions and

Recoanendati ons

6. Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 The effectiveness of a system depends on

its mission. With a narrow system and

mission definition it is possible to

measure the system's ability to achieve

that mission directly through analytical

models or experimental testing. A broader

definition reduces the problem of

sub-optimization created by the narrow

definitions; however, valid analytical

expression is now threatened. Therefore,

the basic consideration in a system

definition is "optimization'' of the

measurability - sub-optimization

relationship.

1.2 The lowest level of measurable characteristics are those numerous and distinct

attributes of a system contributing to its

effectiveness, for example, speed and

burntime. The second level set of

characteristics represents a manner of

describing the system's effectiveness

through a smaller and more general set of

parameters. The elements of this higher

level are combinations of basic attributes

that describe a slightly more general

system characteristic, for example, the

range of the missile (as a function of the

above two lower level attributes).

1.3 From the definition of a system to a

quantitative criterion of its value, the

following steps are Identified:

A-181
I Topics Relevant

i to System Development

land Evaluation Technology ABSTRACT

r.

(1) Define the mission as broadly as

possible, being consistent with some

concept of how its ability to achiev I

the former can be expressed

quantitatively.

(2) The system designed to accomplish the

mission should be explicitly defined

to some "boundary." The latter must

separate the system from its

environment; contributions from other

elements or systems are Incidental.

(3) A criterion for Judging the value of a

system must be formulated and/or,

CO A method of optimizing the design or

choice of syst-m devised.

(5) Based on the method of optimization

chosen, certain types of measurements

must be obtained for a complete set of

characteristics at the highest level

possible.

(6) A method of expressing the

effectiveness of a system as a

function of the elements in a set must

be designed, or if this measure oan't

be obtained with the desired

confidence of correctness then,

(7) The mission should be redefined such

that effectiveness can be mc-e

confidently expressed.

•

A-182
Harry, H.P. RANN utilization experience. Case study number 10.

Effectiveness measurement methods (NSF-RA-G-75-038). Washington, D.C.:

Urban Institute, 1975. (PB-2U7 254).

Topics Relevant

i to System Development

land Evaluation Technology ABSTRACT

State of the Art Review 1•3

oi the Process

1.1 General Systen

Measurements

1.2 System Taxonomy

Model (STM) 2.1

1.3 Overall Conceptual

Process Model (CPU)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environaent Definition 3.1

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate 3.2

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.4 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing 5.1

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recooaendations

This project was designed to provide

substantive information on the effects of

government services in the community.

Nine service areas were chosen for

measurement. These were crime control,

recreation, library, fire protection,

transportation, solid waste, water supply,

waste water, and citizens' complaints.

For each of the above service areas, the

project staff developed 15 to 20 new

effectiveness measures.

Examples of these effectiveness measures

were: the clearance rate of arrests

through court system, the percentage of

individuals, by age groups, using a city

recreation facility, circulation per capita

of library materials, number and rate of

injuries and deaths due to fire, and

subjective measurements of passenger

comfort on local transportation. Other

measures included inspection results and

response times. No procedures are

described for deriving these effectiveness

measures.

Two cities (St. Petersburg, PL and

Nashville, TN) were chosen as the

experimental sites for this study. Data

were collected by appropriately designed

surveys of the user or providers of these

services. A computer program for

organizing and analyzing the data was

developed.

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-183
Topics Relevant i I

I to System Development I Topic I

land Evaluation Technology ! No. I ABSTRACT

5.4 It was concluded that this study provided

the tools and an implementation system that

would afford local officials significantly

improved information on their services.

Specifically, the measures were intended to

help identify current problem areas,

indicate trends, inequities in service

areas, and provide a data base for use in

comparison research between cities. Cities

throughout the nation are adopting the

techniques derived from this project and

the techniques are readily modifiable for

almost any local government.

A-184
. . -

Helm, W.R. Human engineering design deficiencies; A comparative analysis of

the P-3 and S-3 aircraft (NATC-SY-41R-76). Patuxent River, MD: Naval Air

Test Center, March 1976. (AD-B010 910).

I Topics Relevant I

I to System Development iTopic

land Evaluation Technology i Wo. ABSTRACT

v

V'

1. State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environnent Definition

2.'( General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.^ Performance

Criteria, Specific

3.5 Measurement Procedures

4. Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoaoiendatlons

6. Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.3 Engineering design deficiencies of two

aircraft were examined to determine

deficiencies of a human engineering nature.

3.1 The following human engineering

deficiencies were reported: erroneous,

ill-placed, inaccessible, inconsistent,

incompatible, missing, too complex,

unclear, unneeded, and unreadable. In

order to classify these deficiencies in

terras of system hardware components, the

following categories were adopted:

display, control, workspace, lighting, life

support, escape egress, seats, parachutes,

canopy, and weapons system.

3.2 The human engineering was considered

deficient when it specifically violated the

large list of display design principles, or

the list of control design principles, which

might lead to display or control induced

operator errors.

4.1 The critical incident technique was used to

catalogue, describe and analyze the human

engineering design deficiencies.

5.3 The results indicate that there were

substantial numbers of human engineering

design deficiencies in relation to total

engineering deficiencies. The results

suggest that, although the airframes of the

two aircraft are dissimilar, there was a

basic similarity in the number and types of

human engineering design deficiencies.

A-185
I Topics Relevant

i to System Development

land Evaluation Technology

Topic|

No. | ABSTRACT

5.H Since display and control deficiencies

accounted for approximately three quarters

of the human engineering d-jsign deficiencies. It could be advantageous to both the

design and evaluation personnel to concentrate more of their efforts in these two

areas.

•

A-186
Helm, W.R. Human factors test and evaluation. Functional Description

Inventory as a test and evaluation tool, development and Initial validation

study (HATC-SY-77R-75- Vol. 1). Patuxent River, MD: Naval Air Test Cei.ter,

September 1975. (AD-B007 M63L).

I Topics Relevant I

! to System Development I Topic

land Evaluation Technology ! Wo. ABSTRACT

1. State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3- Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3-3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

k. Planning Components

of the Process

4.1 Analytic Methods

1.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoanendations

1.3 The purpose of this report is to describe

and present a new method developed to

evaluate the operational functions

performed by crewoembers.

2.1 This study addressed the human factors

aspects of aircraft man-machine systems, in

particular the S-3A aircraft and crew.

3.1 A series of investigations were conducted

analyzing the operational functions of

crewmembers. Roles, duties and tasks

performed by each crewroember were

determined. Crewmember judgments were

compiled to establish relative importance

to mission success of these roles, duties

and tasks, in addition to the frequency of

performance on a typical mission, the

training necessary to insure effective

performance, and how effective the system

was in accomplishing these operational

functions.

3.2 The technique described above results in a

Functional Description Inventory (FDD.

The FDI has the potential as a tool for

providing quantitative assessment of the

human factors aspects of aircraft

man-machine systems.

5.4 It is recommended that the development of

the FDI technique be validated with fleet

training crews, etc., through comparative

analysis of Board of Inspection Survey

(BIS) Yellow Sheet deficiencies.

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning A-187

/

/r
Henderson, R.L. 4 Burg, A. The role of vision *r\6 audition In truck and bus

driving (Final Rep). Santa Monica, CA: System Development Corp., December

1973. (PB-230 776).

I Topics Relevant

I to System Development

land Evaluation Technology

I Topic

I No. ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Cooponents

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.« General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.H Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1

«.2

«t.3

4.4

4.5

Analytic Methods

Parameter Determinations

Apparatus for Testing

Personnel for Testing

Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclisions and

Recommendations

Further Research Armas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

2.1 This study addressed the commercial car. ier

driver, the accident record, and the

driving task.

2.5 The safe operation of buses and trucks was

the ultimate performance requirement.

2.6 The performance of the system in terms of

safety could be determined by the number of

accidents in which a driver is Involved

compared to test scores.

3.1 The objective of the study was to identify,

through analytic techniques, the basic

visual and auditory requirements of the

commercial carrier driver. A review of the

literature was conducted, and a detailed,

systematic examination of the driving task

was undertaken, including interviews with

drivers and direct observation of the

driving task. Inputs from these sources

determined the importance of each visual

function, and determined the auditory

standards to be tested.

3.2 A device was designed and constructed that

provided the capability of testing

performance on visual performance

parameters. Basic hearing loss data was

obtained by means of a standard audlometrlc

technique.

4.1 Tests were administered to a sample of

ooomercial carrier drivers whose

performance on the various tests was

compared against past accident history.

Multiple regression analyses and a

graphical analysis technique were used in

data analysis.

A-188
I Topics Relevant

i to System Development

land Evaluation Technology ABSTRACT

U.U 236 subjects were recruited from a large

trucking firm and two large bus companies

also participated in this study.

4.5 To evaluate experimentally both the results

of the analytical effort and the test

device, the entire battery of visual and

auditory tests were administered to the

subjects along with a questionnaire to

derive biographical and driving pattern

data. Driving record information was

obtained from company files for each driver

tested, including total number of accidents

on file, number of "responsible" accidents

on file, number of months covered by the

files and total number of accidents and

"responsible" accidents for the last 36

months.

5.3 The new measures of visual performance were

shown experimentally to relate to past

driving records. However the limited size

of the sample prevents generalization to

the entire population. It is recommended

that additional data be collected to

increase sample size. If findings can be

verified the probability is high that new

visual qualification standards can be

generated, resulting in a more valid

screening procedure. The results of the

analysis of auditory data are not so

positive. Assuming the results are not

antifactual, hearing loss standards

presently allowed are not related to

accident involvement. The question of

whether standards are too strict cannot be

answered experimentally.

A-189
Hicks, J.A. A methodology for conducting human factors evaluations of vehicles

in operational field tests (Draft). Fort Hood, TX: U.S. Army Research

Institute for the Behavioral and Social Sciences, Fort Hood Field Unit, June

1977.

I Topics Relevant I I

1 to System Development I Topic I

and Evaluation Technology | No. i ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

4.2 Parameter Determinations

14.3 Apparatus for Testing

H.H Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

ReconDendations

1.3 The purpose of this research was to develop

a standardized methodology for conducting

human factors evaluation of trucks and

similar vehicles in operational field

tests.

2.1 The system is composed of trucks and

similar vehicles within the context of

operational field tests.

3.1 Eighty-five hunan factor characteristics

relevant to vehicle design and evaluation

were selected for measurement and they were

organized into six principal categories:

Driver compartment, visibility, control and

control operations, instruments, handling

characteristics, and ride characteristics.

3.2 The method for measurement of these

characteristics focused on the assessment

of users' judgements of the vehicles being

evaluated.

3.5 The methodology used was the Human Factor's

Vehicle Evaluation Instrument (HFVEI).

Upon completion of the test drive, the

driver was Interviewed and used a six point

rating scale - extremely acceptable to

extremely unacceptable. In addition, the

drivers were required to rate the relative

importance (weight) of each of the 85

characteristics.

4.1 The data were analyzed using Analysis of

Variance (ANOVA) and post-hoc multiple

comparison techniques.

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning A-190
I Topics Relevant

I to System Development iTopic

land Evaluation Technology j No. ABSTRACT

U.2 Test parameters included the elimination of

those drivers who could not meet minimum

performance standards; the training of

interviewers, the familiarization of

drivers to procedure; the establishment of

a test course with a variety of representational tasks over terrain appropriate for

vehicles. The order of driving the course

was counterbalanced and the driver interviewed while still in the driving seat.

M.3 This evaluation compared a non-standard

3-1/2-ton cargo truck with both a standard

U.S. Army 2-1/2-ton truck and a 5-ton cargo

truck. The test took place around a

four-mile test course.

•

I -

U.M Twenty-nine licensed U.S. Army truck

drivers were trained to drive all three

types of vehicles.

5.3 The data analyses revealed that the drivers

judged the 3-1/2 ton and 5-ton vehicles to

be significantly better than the 2-1/2-ton

vehicle from a human factor's standpoint.

There were no significant differences

between the 3-1/2-ton and 5-ton trucks.

5.I The evaluation methodology should be of use

to all agencies conducting user tests of

trucks and similar vehicles.

A-191
Highsmith, R.C. Proposed measure of effectiveness for human resoirrce

availability periods and their impact upon unit operational readiness.

Monterey, CA: Nava. Postgraduate School, December 1976. (AD-A036 028)

I Topics Relevant

I to System Development

land Evaluation Technology ABSTRACT

5.

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.'! General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requlreiients, Specific

3.'* Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.1* Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5-1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

RecofflBendatlons

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

2.1 The following aspects of commands were

measured:

(1) Command Climate: Communication Flow,

Decision-Making, Motivation, Human

Resource Emphasis, Lower Level

Influence.

(2) Supervisory Leadership: Support,

Teamwork, Goal Emphasis, Work

Facilitation.

(3) Peer Leadership: Support, Teamwork,

Work Facilitation, Problem Solving.

(U) Work Group Processes: Work Group

Coordination, Work Group Readiness,

Work Group Discipline.

(5) Satisfaction

(6) Integration of Men and Mission

(7) Training

(8) General

(9) Equal Opportunity

(10) Drug Abuse

(11) Alcoholism Prevention

(12) Community Interrelationships

A-192
I Topics Relevant

i to System Development

!and Evaluation Technology ABSTRACT

2.3 The purpose of this study was to develop a

methodology for evaluating the impact of

the Human Resource Management Support

System (HRMSS) in those commands where it

is utilized. Its primary mission was to

assist those commands with trained

specialists in order to provide assistance

in increasing the overall performance of

its personnel toward goal attainment and

quality performance.

2.^ Several limitations characterized this

study:

(1) Data samples used to test the two

hypotheses were necessarily smaller

than those required for rigorous

statistical analysis.

(2) Data used to test both hypotheses were

from different units (test sites).

3.1 This study centered around two hypotheses:

(1) There will be no significant

difference in response between the

Human Resources Management (HRM)

speciclists and the commanding

officers regarding how well the

various Human Resources Availability

(HRAV) activities were accomplished.

(2) Comparison of units participating in

an HRAV with control units will

statistically differ in a significant

manner.

I -

3.2 Two questionnaires were designed to test

the first hypothesis. One tapped HRM

specialists regarding their assessment of

their own performance; also, the commanding

officer of the unit receiving the services

was asked the same set of questions. AB513

The second hypothesi'; would lead one to

expect some degree of positive change in

unit performance as measured by recognized

A-193
I Topics Relevant

! to System Development

!and Evaluation Technology ABSTRACT

performance criteria. This was measured by

comparison of unit performance in the area

of interest with the HRAV being the

independent variable.

4.1 A Likert-type scale was administered to

relevant respondents and data were analyzed

through the use of chi-square tests of

significance and regression equations.

5.3 It was concluded that the various HRAV

activities had merit to the commanding

officers and that the HRM Support Team

perform those activities at an acceptable

level of effectiveness.

A-194
'.",

Hill, J.W. & Eddowes, E.E. Further development of automated GAT-1 performance

eaaures (AFHRL-TR-73-72). Menlo Park, CA: Stanford Research Institute, May

T9W. rAD-783 2U0).

Topics Relevant I

to System Development 1 Topic

and Evaluation Technology j No. ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measureaents

1.2 Systea Taxonoay

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, ultimate

Analytic Components

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Require«-nts, Specific

3.1 Periui_'ce

Criteria, Specific

3-5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

1.2 Parameter Determinations

1.3 Apparatus for Testing

1.1 Personnel for Testing

1.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.1 Conclusions and

Recoonendations

Further Research Areas

6.1 Measurement Systea

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.3 This report describes a systematic,

statistically-directed search for automated

flight measurements that correlate with

pilot proficiency.

3.1 In experiment 1, four different flight

tasks of about 10 minutes long were

measured. In experiment 2, six additional

tasks were measured.

3.3 Pilots' performance during various

simulated flight tasks was required for

this evaluation.

3.5 Three hundred and twenty six measurements

over the four task series in the first

experiment and 2436 measurements in the 2nd

experiment were made. Many measurements

had the same name but were measured in

different tasks. A numbering system was

used to identify each of them.

4.1 Tables of more than 400 important

measurements were developed with group

means and standard deviations and further

cross-tabulations showing which tasks and

families of measurements best discriminate

among pilots. Three different statistical

methods were used to select a set of

measurements from experiment 1 and combine

them into two canonical variables, each a

linear weighted combination of the

measurements in the set, to discriminate

optimally among the three groups of

subjects. Applying the canonical variables

to the repeated measurements of Experiment

2 allowed several deductions abo"t the best

selection procedures to be made.

A-195

• *.- - "J- J-""'-"• -• -' .'• .'

— —-
I Topics Relevant T

I to System Development I Topic

land Evaluation Technology ! No. ABSTRACT

•

U.3 This approach utilized a GAT-T trainer, a

Line-8 computer system and pilots.

Vsl Subjects (3C in each experiment) were

selected on the basis of their flying

experience. One group had less than 10

hours experience, the second grcjp had

25-50 hours, and the third group had more

than 100 hours of experience. The majority

had had some experience in the GAT-1

trainer. Some subjects volunteered, others

were paid.

4.5 Before each subject was asked to perform

the series of maneuvers, a warm-up period

was allowed to familiarize pilots with the

GAT-1 and its flight characteristics. The

warm-up period varied depending on the

skills of the pilots. In experiment 1 the

major tasks were:

(1) Roll and pitch tracking.

(2) Roll and pitch tracking with power

changes.

(3) Flight profile.

(4) ILS landing approach.

In Experiment 2, the following tasks were

added:

(1) Roll tracking.

(2) Roll, pitch and yaw tracking.

(3) Reduced bandwidth roll tracking.

(1) Reduced competence roll tracking.

(5) Ground reference turning maneuver.

(6) Altitude position tracking.

5.1 The results show that there is little

difficulty in obtaining measurements that

correlate significantly with experience.

A-196

''-• -•
Holshouaer, E.L. Guide to human factors engineering General Purpose Test

Planning (GPTP) (TP-77-1M). Point Mugu, CA: Pacific Missile Test Center, June

1977 (AD B022 013L).

Topics Relevant

I to System Development

land Evaluation Technology ABSTRACT

1. State of the Art Review

of the Process

1.1 General Systea

Measurements

1.2 Systen Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

?.l System Definition

2.2 Mission Definition

2.3 Environment Definition

2.'4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Crifria, Ultimate

3. Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3-3 Performance

Requirements, Specific

3.'* Performance

Criteria, Specific

3.5 Measurement Procedures

li. Planning Components

of the Process

4.1 Analytic Methods

k.2 Parameter Determinations

1.3 Apparatus for Testing

1,1 Personnel for Testing

4.5 Test Plans

5- Application Coaponents

of the Process

5.1 Test Execution

5.2 Data Analysis

5-3 Findings Interpretation

5.4 Conclusions and

Recoosendatlons

6. Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

2.1 The purpose of this document was to provide

and the Navy with a generalizable Test and

2.2 Evaluation (T4E; planning scheme which can

be more easily reviewed and evaluated in

addition to providing a basis for

implementation monitoring and systematic

data analysis. In brief, the beneralized

T4E plan describes the details that the

Human Factors Engineer (HFE) should

consider when preparing or implementing a

plan for a specific weapon system. The

essential elements of this system covered

planning, conducting, reporting, and

correcting; the system becomes more

detailed as the tasks and sub-tasks become

smaller in scope.

3.1 The characteristics of the system consisted

of the following: review of test plan

requirements; review general and/or

detailed specifications; review previous

test data; prepare general test plan;

detail the test plan; negotiate and/or

revise test plan; commence testing within

given constraints; document human factors

deficiencies; prepare the human factors

input to project office T4E final report;

prepare separate report for the Feedback

Loop Action Generation (FLAG) information

system; negotiate for follow-on human

factors effort; participate in product

improvement and deficiency correcting

programs.

A-197

-. - - -

> v. • ."- ."- A >.>

- - - - - - - - - - > - . . . .
Topics Relevant

to System Development

and Evaluation Technology

Topic

No. ABSTRACT

3.2 Examples of how the above system components

we"e quantified are given in detail.

Examples of these sub-routines, in

respective order, are as follows:

(1) Determination of phase and scope of

T4E, determination of scheduling.

(2) Budget constraints, and manpower

requirements.

(3) Review of weapon system documents for

relevance to human factors.

(M) Review analytical efforts, simulations, modeling, etc., conducted

during concept validation.

(5) List areas to be investigated.

(6) Determine safety factors to be

incorporated into the aircraft/weapon

system.

Possible areas to be investigated may

consist of such factors as crew station

geometry, anthropometries, and controls/

displays; moreover, questionaire checklist

tests must be specified and instrumentation

requirements listed. Project office

personnel should be consulted regarding

test parameters and data collection

methods; finally, the report itself follows

a standard format which meets the

requirements of the FLAG system.

A-198

— - -• ---- ---- ----•-- - --

- •- - -. . -."«i*".
House, E.R. Assumptions underlying evaluation model:

1978, 7(3), 4-12.

Educational Researcher,

Topics Relevant i

i to System Development iTopic

land Evaluation Technology I Ho. ABSTRACT

1. State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPU)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

1.1 The basic theme of this paper is that all

the evaluation models are based on

variations in assumptions of liberal

ideology, the conceptions of liberal

democracy. The models differ from one

another as the basic assumptions vary.

The paper describes some evaluation models

and explains their premises. It is part of

a larger work which attempts to present a

comprehensive analysis of evaluation.

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.4 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-199

- - - - - -^-
Hyatt, C.S. & DeBerg, O.H. A scoring system for the quantitative evaluation

of pilot performance during Microwave Landing System (MLS) approaches

(ASD-TR-17). Wrlght-PatUrson AFB, OH: Aeronautical Systems Division (AFSC),

August 1975. (AD-A025 782).

Topics Relevant I

I to System Development I Topic

land Evaluation Technology ! No. ABSTRACT

1. State of the Art Review

of the Process

1.1 General System

Measurements

1.2 Systec Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPU)

2. Contextual Conponer.ts

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.^ General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

J.) Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.H Performance

Criteria, Specific

3.5 Measurement Procedures

'1. Planning Components

of the Process

1.1 Analytic Methods

1.2 Parameter Determinations

•».3 Apparatus for Testing

4.«I Personnel for Testing

4.5 Test Plans

5- Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoanendations

2.1 This evaluation addressed the Microwave

Landing System (MLS) Including aircraft and

pilots.

2.2 A number of landing approach patterns with

multiple aircraft were selected for this

study.

2.4 It was noted that aircraft using the same

airspace are likely to have a wide spectrum

of interpretative capabilities. The

constraints for use of this equipment are

weight, space, capital and maintenance

costs.

2.5 The ultimate requirement is that the system

provide accurate information regarding an

aircraft's position during a landing

approach.

3.1 Deviation in position and speed from

planned glide path were measured.

3.2 Errors across track, along track, above and

below glide path and speed along path were

determined.

4.1 The descent rates were compared with the

criteria and the difference was defined as

error. Error values were modified to

relate to their size and significance. The

modified error value was weighted for each

parameter to reflect its true influence

relative to other parameters. The modified

and weighted error values were combined

into score(s).

Furtner Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentiels/

Priorities

6.3 Research Planning

A-200

v -••>" ^ .jf«-• .-

.v.V '..-wW -.-

_
Topics Relevant I

to System Development lTopic!

and Evaluation Technology ! No. ! ABSTRACT

U.2 A limited number of typical approaches and

rates of descent were selected for this

analysis.

5.M This scoring system works and is being used

for other MLS systems.

r

A-201

- '* J'_«"_« «" . . •• . ,

- -

, - .
Hunter, B. Report on the final acceptance test for CTS (Computerized Training

System). Alexandria, VA: Human Resources Research Organization, August 1976.

(AD-A033 100).

Topics Relevant

I to System Development 1 Topic I

land Evaluation Technology ! No. I ABSTRACT

State of the Art Review

of the Process

1.1 General "ystem

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Coaponents

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environnent Definition

2.<l General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3-3 Performance

Requirements, Specific

3.'! Performance

Criteria, Specific

3.5 Measurement ProcMures

Planning Componentof the Process

«.1 Analytic Methods

k.2 Parameter Determinations

I1.3 Apparatus for Testing

*.i» Personnel for Testing

«1.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.I Conclusions and

Recoonendatlons

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.3

2.1

3.3

M

The Phase III Computerized Training System

(CTS) Final Acceptance Test Plan was

conducted in five levels that corresponded

to five test objectives:

Level I:

Tests.

Verification of Phase II Software

Level II: Reliability.

Level III: Verification of Phase II Class

I Language Tests.

Level IV: Response Time.

Level V: Full Operational Load.

The CTS, a full 128-termlnal system, was

under investigation in the present study.

The primary objectives of the test were to

determine whether the full 128-terminal

system will operate under operational live

user conditions and meet contract

requirements for reliability and response

time.

As regards reliability the system must

achieve a minimum of 951 under specified

conditions. Up-time refers to the

capability of the hardware and software to

perform basic tasks of system Initiation,

program compilation and execution, job and

file management, and program library

maintenance.

A-202

- - • '.vV-

- 

.
1 Topics Relevant I

[ to System Development 'TopicI

and Evaluation Technology ! No. I ABSTRACT

E; ?

Regarding response time, performance

specifications for the 128-system state

that there shall be a 90t probability that

the response time will be 2 seconds or

less.

5.1» Since the system did not meet reliability

objectives (plus had an insufficient number

of response time recordings for a

representative sample), it is recommended

that the system not be accepted in its

present state. Following debugging of the

present problems, further retesting of the

system for overall performance is

recommended.

N "-

A-203

i.V.i . '- ^- *'- '- < '-
Hutchins, C.W. A Computer Aided Function Allocation and Evaluation System

(CAFES). Proceedings of the Human Factors Society Annual Meeting, Huntsville,

AL, October 15-17, 197^.

I' Topics Relevant I

I to System Development I Topic I

land Evaluation Technology ! No. I ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.? System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.'! General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultiaate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

i|.2 Parameter Determinations

4.3 Apparatus for Testing

H.U Personnel for Testing

i|.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.t Conclusions and

Recomnendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.3 The Computer Aided Function Allocation and

Evaluation System (CAFES) is a support tool

for Human Factor Engineers (HFE) conducting

man-machine research, requirements

analysis, design, test, and training and

maintenance systems development. The CAFES

program's principle objective is to

facilitate application of essential

elements of human factors technology in

systems development using automatic data

processing techniques in order to analyze

and evaluate crew subsystem performance as

it affects total systems effectiveness.

2.1 The CAFES was being developed to provide

the HFE with an Integrated jystem of

computer models which progress from early

concept formulation phase through crew

station design to the final test and

evaluation of the completed system.

2.2 The CAFES objective was to allow the HFE to

treat in a comprehensive way all parameters

to be considered In the designing of a

man-machine Interface of advanced Navy

systems.

3.1 The CAFES models were, in fact, composed of

six integrated sub-models:

(1) A Data Management System (DMS) which

provides baseline data for all the

models.

A-204

i^i^^a^afcaAaiaiaii • • -
I Topics Relevant I

to System Development I Topic I

land Evaluation Technology I No. ! ABSTRACT

(2) A Function Allocation Model (FAM),

which is a collection of computerized

algorithms that helps derive and

process various alternatives for

allocating functions to operator(3)

and machines.

(3) A Workload Assessment Model (WAM)

which considers the human performance

aspect of man-machine function

allocation schemes on a time and

cumulative tasks basis to determine if

man can perform all of the tasks

derived from the functions that

compose one or more missions. The

model also determines those periods

when man is overloaded in terms of

time availability versus time required

to do all tasks.

(4) A Computer-Aided Design (CAD) which

will furnish aids to the designer in

producing crew station configurations

that are consistent with mission

requirements.

(5) A Crew Station Geometry Evaluation

(CGE) which evaluates the physical

geometry of a crew station.

(6) A Human Operator Simulator (HOS) which

provides a generalized model of a

seated human being in a task

environment with limitations on time

and goals.

3.3 The specific performance requirements of

the above models were as follows:

(1) DMS: its major functions are data

input and storage, file modification,

CAFES Executive, error diagnostics,

report generation.

(2) FAM: predicts overall system

effectiveness and generates crew

operational procedures for detailed

evaluation of promising allocation

A-205

Ml - - •

LN-'.WV-'. .,•. W,
I Topics Relevant I

I to System Development I Topic I

land Evaluation Technology ! No, ABSTRACT

candidates.

(3) WAM: crew workload is analyzed on the

basis of how workload varies in each

performance "channel" (vision, hands,

feet, etc.).

CO CAD: assists in crew station geometry

development, design analysis, and

design drawings.

(5) CGE: functions includes man-model

simulation (computes body

configurations approximating human

movements during task taxonomy),

conducts interference analysis

(identifies visual or physical

interference for various crew members

executing a task sequence), evaluates

military standard and specification

compliance.

(6) HOS: serves to simulate human

operator performance in aircraft

systems, interfaces directly with the

hardware simulator, provides simulated

human performance output, and

re-examines human performance workload

and task duration in an operating

environment.

5.^ Computer techniques can only assist the

HFE, not substitute for his judgment. The

computer can assist the engineer in design

and analysis tasks by recalling/processing

data and performing simple logic

operations. The responsibility of the HFE

remains with interpretation of effects from

system requirements, performance criteria,

and trade study results in defining system

specifications.

v

A-206

-\-

- - - - . , , , . . . . . . .
Irish, P.A., Grunzke, P.M., Gray, T.H. 4 Waters, B.K. The effects of system

and environmental factors upor experlerced pilot performance In the Advanced

Simulator for Pilot Training (AFHRL-TR-77-13). Williams AFB, AZ: Air Force

Human Resources Laboratory, Flying Training Division, April 1977.

I Topics Relevant

I to System Development

land Evaluation Technology ABSTRACT

1. State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.ll General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

k. Planning Components

of the Process

1.1 Analytic Methods

1.2 Parameter Determinations

1.] Apparatus for Testing

l.t Personnel for Testing

*.5 Test Plans

5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

6. Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

2.1 The system being studied was the Advanced

Simulator for Pilot Training (ASPT)

configured as the T-37 aircraft.

2.2 The mission included five specific flight

maneuvers: takeoff, GCA, 360 degree

overhead patterns, aileron rolls, and slow

flights.

2.3 The environmental conditions in this study

included three levels of wind, three levels

of turbulence, and two levels of

ceiling/visibility.

2.4 The constraints on the system were those of

the ASPT which include six degrees of

freedom motion platforms providing

approximately three feet of vertical travel

and four feet of horizontal travel.

Displacement capabilities included: pitch

- 20 degrees to 30 degrees; roll 22

degrees; yaw 32 degrees. The pneumatic

G-seat provided more continuous cues than

the motion platform in response to

requirements for each particular maneuver.

The visual system was comprised of seven

36-inch monochromatic CRT's giving the

pilot 110 degrees to -U0 degrees verteal

cuing and 150 degrees horizontal.

3.1 The dependent variables used in this study

and were derived from the ASPT Automated

3.2 Performance Measurement System which is a

criterion-referenced approach to

measurement. Because most skillful

piloting involves the attempt to maintain

or change to specified flight parameters

?

A-207

• • - -- - - - -------
I Topics Relevant

I to System Development

and Evaluation Technology i

• r

i i

iToplcl

No. ! ABSTRACT

criteria, deviations from these deslr' i

parameters pro" ides the method of

quantitative objective performance

measurement. Some examples of the

dependent variables (measures) reported In

this study Include:

(1) Deviations from prescribed flight

parameters.

(2) Power applied to control surfaces.

(3) Scores formulated from multiple

factors.

3.5 Each subject flew one profile 72 times and

the other 27 times during the course of the

study. The profiles were randomly ordered

for all subjects. Each session was begun

with Instructions provided by a computer

driven word generator. Each maneuver was

begun on command and completed when

selected criteria were satisfied. At the

completion of each maneuver within the

profile, the console operator entered

comments on any system malfunction or

errors experienced during the maneuver.

All other data values were recorded by the

ASPT computer at an Iteration rate of 3.75

to 15 times per second.

4.1 A multlvarlate analysis of variance was

used for data analyses. The statistic used

in determining significant effects was the

Wilks Lambda.

4.2 The Independent variables of the test were

as follows:

(1) Wind (0, 12, 24 knots from 60°)

(2) Turbulence (none, light, moderate)

(3) Celling/visibility (clear and minimum

(200 ft/1/2 mile)).

A-208

—^

, . :'. ^ . .-.•:. > ^.-_.-^ . _ .- - - . - - - - ,- ,- . - - -

, • , * -
I Topics Relevant I

! to System Development I Topic

land Evaluation Technology I No. ABSTRACT

M.S The primary test apparatus was the

six-degree-of-freedom Advanced Simulator

for Pilot T-aining configured as a T-37

aircraft cockpit.

U.U T'-e test subjects were three T-37

instructor pilots whose flying time ranged

from 550 to 900 hours.

•

' .5 Two separate experimental designs were

used. The first was structured to evaluate

main, first-order interaction, and

second-order interaction effects of all six

independent variables. The second design

was a partially compounded factorial which

used four independent variables, each with

three levels.

5.1 Subjects were scheduled on a day-to-day

basis depending on ASPT system

availability.

5.4 In this study, each of the system configuration variables produced significant effects. The dependent variables used to

measure performance in this study showed

that manipulation of the three environmental variable combinations produced changes

in system-oriented dependent variables.

Similarly, changes in the pilot input

variable were concomitant with simulator

configuration changes.

A-209

M

>* I 0- •"• -"• p <> -- •

- - . ^" v" -_• -.- - . -. J
Jahns, D.W. A concept of operator worklcaJ In manual vehicle operations.

Meckenheim, BRD: Forschungsinstitut für Anthropotechnik, December 1973.

Topics Relevant I

! to System Development iTopic

land Evaluation Technology ! No. ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.U General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

1.1 Personnel for Testing

».5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recomuendations

1,1 This report represents an initial attempt

to scope, through a literature review, the

complexity of developing a conceptual

structure (model) of operator workload.

The ultimate goal is to develop a

quantitative index of operator performance

for any point in time during a given

vehicle operation. The operator's role is

basically that of a data transmission and

processing link between displays and

controls of any vehicle. The operator's

basic functions are sensing, identifying,

and interpreting.

Workload is the extent to which an operator

is occupied by a task. It can be divided

into three functionally relatable

attributes: input load, operator effort,

and work result.

Four categories of workload research were

identified during the literature review:

(1) Time-and-motion studies.

(2) Information processing status.

(3) Operator activation-level studies.

(U) Equipment design - implicit studies.

The report describes each in detail and

presents their advantages and disadvantages.

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-210

: y. T - • - ^ - • - . - - - - - - 1- ...--. . *.
Topics Relevant

to System Development

and Evaluation Technology

1 Topic I

I No. I ABSTRACT

rl - The report concludes from the literature

survey that, In general, the approach to

workload research has been either too

molecular or too molar to provide the broad

spectrum of data required during various

phases of crew system design and evaluation. A number of potentially useful

techniques are availaole to partially

provide meaningful, quantitative answers on

many parameters Influencing operator effort. These techniques need to be systematically evaluated and integrated in the

specific context of design requirements.

Because of the complexity of factors

affecting human behavior and our limited

knowledge regarding operator capabilities

and limitations, no absolute measure of

workload exists.

A-211

: >

•. - •. ". '. " •% '- . • •

-" ---••• - *
1

Jahns, D.W. & Katz, R. Preliminary concepts for Computer Aided Function

Allocation Evaluation System (CAFES) (NADC-72106-CS). Seattle, WA: The

Boeing Company, Aerospace Group, May 1972. (AD-902 670L)

I Topics Relevant

I to System Development

land Evaluation Technology ABSTRACT

State of the Art Review

of the Process

1.1 General Systen

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPU)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environnent Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

H.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Reconnendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.3 This study describes a comprehensive crew

system development tool called CAFES

(Computer Aided Function Allocation and

Evaluation System). The CAFES concept is a

set of submodels working in conjunction

with a data/information management system.

Each submodel may be used individually or

in combination with one or more of the

others depending upon the data requirements

(and system definition). The modular

nature of CAFES provides flexibility for

integrating the parameter in the definition

and evaluation of the human role in naval

aircraft systems. The mission of CAFES is

a reduction in performance time and

increased quantification for the following

human factor tasks:

(1) Perform and evaluate man-machine

function allocations.

(2) Develop design concepts/requirements

for each crew work station.

(3) Identify potential human factors

engineering risks.

The following are descriptions of the

various submodels which make up CAFES:

(1) The Functional Allocation Model (FAM)

Is a collection of computerized

algorithms that will, in conjunction

with the Data Management System (DMS)

and crew system designer, derive and

process various alternatives for

allocating functions to operator(s)

and machines. The FAM will identify

and organize system function to an

A-212

', . . *■■• .-'-• •* •-"• " - • ,■■,••.•".-.• .-".-.•.- .•-■■-•.• • • -•-•• -••-•..•.•--.•.•. . •.

L.-k.^.,.....'.•..•... •b.»l-k i.-. . ..« ,. ... , - - - - - iifca^^^i
i Topics Relevant I

! to System Development I Topic

land Evaluation Technology i No. ABSTRACT

aliocable level, identify and rank

order function allocation schemes, and

will provide detailed communication

with the available system definition

at any point in the development cycle.

(2) The Workload Assessement Model (WAM)

considers the human performance

aspects of man-machine function

allocation schemes to determine

whether man can perform all of the

tasks derived from the allocable

functions that make up one or more

missions. The submodel uses a

timeline of mission duration and

determines those periods when man is

overloaded, necessitating either a

reallocation of that function to

machine or additional crew or a

modification of the system

requirement.

(3) The Computer-Aided Design (CAD)

furnishes aids with which the designer

can develop crewstation configurations

compatible with the men who will work

in them, consistent with the missions

to be performed, and constrained by

military design standards and

specifications as well as technical

and cost considerations.

(U) The Crewstation Geometry Evaluation

(CGE) program is a standardized method

for evaluating the physical geometry

of a crewstation. It evaluates the

physical compatibility of a seated

crewmember of any size with any

crewstation as soon as a design

concept is available.

(5) The Human Operator Simulator (HOS) is

a generalized model of a seated human

being in a time-based, goal-oriented,

task-processing environment. It is

used in parallel with the CGE submodel

in the CAFES system to evaluate the

A-213

• i i ...., . ^. i mm <-Ai >
I Topics Relevant

I to System Development

land Evaluation Technology ABSTRACT

i

function allocation as well as the

adequacy of the crewst?tion design.

It determines operator workload as

well as procedure (task) duration,

movement distance, and sequence

statistics going from one control to

the next.

I

(6) The Tradeoff Analysis Routine (TAR) is

used after the CGE and HOS models have

been run and modification to the

weapon system are necessary from a

geometric or workload point of view.

The crew systems designer must decide

whether it would be most effective to

modify the design to eliminate the

problem or to reallocate one or more

functions to eliminate the problem.

i

A-214

i . aJ

-.-.-* - -

' - - - •--•- —- ..,
Janes, L.R. Criterion models and construct validity for criteria (72-32). San

Diego, CA: Navy Medical Neuropsychiatric Research Unit, August 1972. (AD-770

M12).

I Topics Relevant I

I to System Development ITopic

land Evaluation Technology I Wo. ABSTRACT

1. State of the Art Review

of the Process

1.1 General System

Measuraaents

1.2 System Taxonomy

Model (SIM;

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.'4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3.1 Piactical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3-5 Measurement Procedures

1.1 Methods pertaining to the measurement of

criteria and ascertaining underlying

criterion constructs were reviewed. Three

criterion measurement models, i.e., the

ultimate criterion model, the multiple

criterion model, and a "general" criterion

model for the determinants of managerial

effectiveness, were examined and attempts

made toward integration. The three models

were compared to a formal construct

validation model, and strengths and

weaknesses in both the constructs provided

by each of the criterion models and

construct validation procedures were

discussed. An Integrated multiple and

general criterion model and construct

validation procedures more extensive than

the multitrait-multimethod matrix were

recommended for future criterion research.

Planning Components

of the Process

1.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-215

- -

. - - . • * • .

- • liHnliitif ' •-•-•-
Jaschen, D.G. Mechanized Infantry Combat Vehicle, XH723, system operational

climatic test (TDP-OT-67A). Falls Church, VA: U.S. Army Operational Test and

Evaluation Agency, Test Design Division, 1975. (AD- B008 212L).

Topics Relevant

! to System Development I Topic I

land Evaluation Technology ! No. ! ABSTRACT

«

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Coaponert

of the Process

3.1 Practica' *»wa_ ^ble

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.'! Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

RecooDendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 This report describes a plan for testing

the Mechanized Infantry Combat Vehicle

(MICV).

2.1 The MICV system is a companion to the main

battle tank and is composed of four parts:

the vehicle itself, the primary armament,

the secondary armament, and the

supplementary armament.

2.2 The vehicle's ultimate functions are as

follows: mounted fighting capability for

one mechanized infantry squad; swim and

cross-country mobility; full protection of

the infantry squad from small arms; etc.

2.1 The MICV system is to be tested under

certain climate conditions mainly

characterized by "winter thaw."

2.5 The primary role of the MICV lies in its

offensive weapons capability.

2.6 In this role, the MICV is expected to

improve mobility, protection, firepower,

survivability, reliability, crew comfort,

etc.

3.1 Characteristics of the system which will be

measured include: mission performance,

survivability, availability, training, and

logistics.

3.2 Measurement of the above may be performed

by measuring the speed, range, and

maneuverability degradations caused by

snow, mud, and frozen ruts (for "mission

performance"); ability of MICV to be

A-216

-. .V

- - -

. - w -
i Topics Relevant

I to System Development

land Evaluation Technology

; Si

.

effectively camouflaged (.for survivability); ability of crew to perform

maintenance and service of the MICV (for

availability); techniques of vehicle

operation (for training), etc.

H.2 Test parameters include day and night

conditions and live fire conditions.

Organization, doctrine, training, and

limited logistical support will be factors

held constant; terrain and weather will be

uncontrolled.

4.1 Data collected will be analyzed and

reported in relative terms (not quantitative). RAM data will be qualitative;

reliability will be limited to a consolidation of data regarding malfunctions which

occur, impact of malfunctions, and maintenance problems associated with them.

Availability will be measured through

records on uptime and downtime of system.

Maintainability will be observed under

winter thaw conditions and will be

established through observations of the

test experience.

4.4 Participants will perform mechanized

infantry squad and scout team tactics as

prescribed.

A-217

v •-.

- • --^^ - - - - - -- -•— —---^
Johnson, E.M. 4 Baker, J.D. Field testing:

Human Factors, 1971, 16(3), 203-214.

The delicate compromise.

I• Topics Relevant

; to System Development

! and Evaluation Technology ABSTRACT

1. State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requi'enents, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3-3 Performance

Requirements, Specific

3.t Performance

Criteria, Specific

3.5 Measurement Procedures

1. Planning Components

of the Process

1.1 Analytic Methods

1.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoanendati ons

S. Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 This paper selectively compares human

factors field testing with laboratory experimentation. The basic steps of research

design provide a point of departure to

illustrate the differences between laboratory and field research. Differences in

problem recognition are described. It is

noted that the laboratory researcher often

uncovers a research problem by exhaustive

examination of the literature. In contrast,

a problem for the field researcher typically

originates with questions from a sponsor/

user group. A table is presented which

charts the stages of a system's life and

ties thrse stages to the associated events

that aid problem recognition.

It is evident that the precision of the

evaluation data is a direct function of the

stage of life at which the problem is

identified.

M.I Problem recognition is only the first link

in the chain of thought necessary for

planning a successful research study. The

next step of the experimental method should

follow in a standard order.

The differences between laboratory and field

testing in the basic steps of a research

design are presented on the following page.

5.1 While it can be noted that there are great

similarities between laboratory experimentation and field research, many subtle, but

critical, differences exist in the substance.

Field testing is not a simple extension of

the laboratory into an operational setting.

A-218

.-.•-> .•■■

- -

/ aai - v- - .

- -- - -

. •-. -.; --^

■■•"• •-
Major Differences in Hypotheses

Characteristic

Type of Study

Laboratory Field

Sources

UKi i valion

Definition

Nunber

Type of Test

Purpose

Literature search

Theory/prior data

Deductive

Precise

Few

Null hypothesis

levelop further research/theory

Colleaguer and Institutional reports

System model

Inductive

Vague

Many

Positive hypotheri«

Answer a question

Major Differences in Independent Variables

Charjcterirtic

Type of Study

Laboratory Field

Selection

NiaiMr

Fela-. 'ween

Dtfinitlon

Durability

Ad hoc; expenmente- determ.ned

Few

Orthogonal

Precise

High

Post hoc; system deterrrined

Many

Nonorthogonal

Vague (easily confused with dependent vanat.er

Low-obsolescent

Major Differences in Dependent Variables

Characteristic

Type of Study

Laboratory Fiela

Stlfei

NtMUTM

t»tinl\

^ielaticr.ship Between

Type of Measure

Sensitivity

Desirable Sensitivity

Ad hoc; experimenter determined

Human performance

Precise

Few

Independent

Quantitative

Linked to independent variable

Statistical

Post hoc; syste." determinea

Human performance/system performance

Vague (easily confused with independent varlatler:

Many

Unknown intercorrelation

Often qualitative

Linked to system performance

Practical

Major Differences in Control

Characteristic

Type of Study

Laboratory Field

Experimental Error

Experimental Controls

Protocols

Intrusive Factors

Physical Environment

Time

Scale (Logistics)

V.j'.at.e. Driver. By

Replications

Matched groLps/control groups

Well defined

"Controlled"

Controlled

"One trial"

Usually one group

Vague

Uncontrolled

Natural

Broken into short segments (trials) Continuous-mission/scenario shift time

Snail Large

Experimenter System

Major Differences in Subjects

Characteristic

Type of Study

Laboratory Field

Source

Motivation

Attitude

"College Sophomore"

Implicit

Positive

User Population

Explicit

Neutral (or negative)

Characteristic

Frequ^cy of Report

Audience

Dissemination

Ethics

Result

Major Differences in Output/Reports

Type of Study

Laboratory Field

Low—determined by quality

One—professional conounity

Wide—professional journal

Individual

"Publish or perish"

High—determined by cost

Many: user/sponsor; management; professional

community

Narrow—institutional report

Organizational

•Publish and perish"

A-21P

t . ^-

-'. •'.. --. -"<•". -

m -

• .• . .•• v-v
Topics Relevant I

to System Development I Topic

and Evaluation Technology! No. ABSTRACT

6.1

Techniques, procedures and research

strategies diTer. However, a technology

is evolving to bridge the gap between the

two research domains. Recognizing the

problem and describing its dimensions does

not solve it. There are few reports of

attempts to validate laboratory findings

in the field. Some indicate success, others

indicate failure.

Limitations in the methods and data of

human factors tend to prevent the accurate

prediction of human performance in operational settings. What still is required

is an approach to research design which

will, in fact, coordinate the laboratory

and the field.

A-220

- > - " - -

—— -- .- -. .- — . , . • -^
Kagerer, R.L. & Weiss, E.C. Development of a checklist and guidebook for human

factors evaluation of general eouipnent (Final Rep.). Alexandria, VA: The

Matrix Corporation, January 1968. (AD-827 808).

I Topics Relevant

i to System Development I Topic

land Evaluation Technology I No. ABSTRACT

1. State or the Art Review 1.1

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3-3 Performance

Requirements, Specific

3.4 Performance

Criteria, Specific

3-5 Measurement Procedures

t. Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

This report describes the development of

two docunents: "Human Factors Evaluation

Data for General Equipment" (HEDGE) and

"Guidebook Supplement." These products

were designed to provide test engineers

with a human factors evaluation guide to

potential human factors problems for the

various types of equipment tested and a

methodology for conducting these evaluations. These docunents were designed

specifically for use by personnel with

engineering qualifications who have had

little or no exposure to human factors

evaluation techniques.

The work was conducted in three phases.

Phase I consisted of defining requirements

for human factors testing and analyzing

current test and evaluation practices. A

classification system for test items and

test functions and an exhaustive listing of

the requirements and constraints for the

contents and format of the human factors

guide were developed.

Phase II developed the format and content

of the guide. The HEDGE document consists

of four major sections: (1) how to use

HEDGE, (2) an index and associated

definitions, (3) detailed data sheets, and

(4) appendix.

The detailed data sheets are the heart of

the strategy proposed. Since the major

emphasis in human factors evaluation is the

nature of the interaction between man

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-221

- - -• - - - -t^. — - - - -
I Topics Relevant I

I to System Development !Topic

land Evaluation Technology | No. ABSTRACT

and the item, these sheets attempt to

detail the man/item tasks that must be

perlormed on all (or most) items in r given

subclass for a specific test function.

The second part of these data sheets is a

detailed design consideration sheet. These

pages contain the critical human factors

considerations in the test and evaluation

of the class of items for which they were

prepared.

Phase III of this project was devoted to

the evaluation and revision of the

products. This evaluation involved on-site

use and evaluation by test engineers and

the conduct of field tests.

Based on these user trials it was determined that these documents are appropriate

and adequate for human engineering evaluations of general military equipment.

A-222
Kaplan, J.D., Crooks, W.H., Boylan, R.J. & Sanders, M.S. Human Rssources Test

and Evaluation System (HRTES) (PDR-1057-78-11). Woodland Hills, CA:

P^rceptronics, November 1978.

I Topics Relevant

I to System Development

land Evaluation Technology

State of the Art Review

of the Process

1.1 General Syste«

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPU)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1 General Constraints

2.5 l rfc-Bance

R i/iirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Perfomiance

Requirements, Specific

3.1 Perfonaance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendati ons

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research potentials/

Priorities

6.3 Research Planning

1.1 The Human Resources Test and Evaluation

System (HRTES) is a systematic and integrative approach to planning and conducting

evaluations of human contributions to system performance. It encompasses a set of

procedures which will assure that human

resources (1) are properly included in a

system design, and (2) are adequately

assessed and evaluated during Operational

Test and Evaluation (OT & E).

1.2 The HRTES philosophy is that understanding

of missions is basic to the measurement of

systems in operational tests. There must

be a logical link between the missions to

be performed and the selected measures of

performance. The first step in the HRTES

procedure to accomplish this linking is to

define systems according to their generic

class(es). Each generic class is defined

by general functional and hardware

similarities. Systems belonging to the

same generic class have certain missions in

common while having other missions specific

to themselve.». individually.

1.3 The generic classification or indexing of

the system is the first step in the HRTES

process. ' The subsequent steps can be

listed as follows: (2) assignment of

missions, (3) specification of system

performance issues, (M) identification of

human performance functions and human

performance measures, (5) Identification of

test conditions, (6) specification of human

resources issues and human resources

measures, (7) operational testing, (8)

evaluation of OT results, and (9) diagnosis

of performance inadequacies.

A-223

L •- - . ". '>&.

------
1

I Topics Relevant I I

I to System Development I Topic I

land Evaluation Technology ! Mo. I ABSTRACT

2.1 The prototype HRTES handbook Hsts eleven

(11) generic classes of Army Systems, VIZ,

Armored Vehicles; Anti-Armor Weapons; Tube

Artillery; Missile Artillery; Aviation; Air

Defense Systems; Point Target Weapons; Area

Weapons; Electronic Warfare Systems;

Electronics Associated with Other Systems;

Engineering Systems.

2.2 Following generic classification of the

system, the analyst using the HRTES

procedure identifies and assesses the

relative importance of the various missions

of the system. Knowledge of the missions

allows the analyst to specify the system

performance issues of interest.

2.^ Specific test conditions (or constraints)

that may affect or further define the

system performance issues must be

identified.

3.1 Human performance functions that derive

from the system performance issues and

human resource issues that contribute to

human performance constitute the measurable

attributes of concern to the HRTES

procedure.

»

A-224

L'- ^'- *. • «---».- ^•. « '«, »* ^ ,'. t,^ •..'. -". i'- i'^ '- V- '. ^

>

«^^^^ ^'^1-'«.' .*^' .. MV >V
Karuah, A.D. Benchmark analysis of time-sharing systems (SP-331

»?). Santa

Monica. CA: System Development Corporation, June 1969. (AD-689 781)

Topics Relevant |

I to System Development ITopic

land Evaluation Technology I No.

State of the Art Review

of the Process

1.1 General Systee

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Per romance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3 •I Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

' rlorltles

6.3 Research Planning

2.2 A benchmark is a routine which is run on a

number of different computer configurations

to obtain comparative throughput

performance figures regarding the abilities

of the various configurations to handle

some specific application. A benchmark

problem is a selected portion of an entire

Job that should be representative of that

Job.

2.3 The benchmark programs measured system

performance under the following

environmental conditions:

(1) Stand-alone - only one object program

makes demands upon a system's

resources.

(2) Benchmark - the seven programs are

considered users, and run

simultaneously as a simulation of a

typical user population to assess

system changes In a real-world

environment.

(3) Real-world - one benchmark program is

run as a pseudo-user when the system

is operating near its rated capacity

under conditions of the real-world.

2.4 Planning and estimating the system's

workload was difficult due to a variety of

demands and the random user times.

Establishing equivalent environments within

the computer systems being evaluated was

the major constraint of benchmarks.

A-225

1 -.-- - - - ^

— * ^0 L^J^L T a 1 a ' '
I Topics Relevant I I

I to System Development I Topic|

land Evaluation Technology I No. ! ABSTRACT

3.1 Because of the factors stated-above, the

behavior of certain functional variables of

a system was measured; a functional

variable included software, hardware, and

.;• user components that support a particular

activity. The particular variables

measured were:

- swap activity - I/O activity

- compute activity - page activity

- interactive - resource

activity allocation

- user population

3.2 The following were measures of the stimuli:

For compute activity, the number of loops

is the measure of the amount of CPU time

allocated to the program by the scheduling

algorithm. The interactive activity used

the average number of messages printed per

minute as its measurement. I/O activity

used the average number of records read

from the disc per minute as its

measurement. No direct measure of swap

activity was made. Instead, the obtained

results from the small and large versions

of the same program were compared with the

measurement of the increase in system

overhead due to the increase in swap times.

The metric used in evaluating the effect of

page management schemes was the compute

activity and interactive activity measures.

An increase in these values implied a more

effective page management scheme. For

resource allocation activity, a measure of

the overall effect of the resource

management algorithms may be determined by

running an environment consisting of only

the benchmark programs. The effect of

different algorithms can be determined by

changes in the measures printed by each

benchmark program.

The measurement of compute activity was the

maximum throughput that any compute-bound

Job could obtain. The measurement of

response time was the minimum (best) time

that any terminal bound job could attain.

A-226

'

v

'

1-'"-- -''-""• -"'-"• '•'-'-' -' -'-- '-'-•«-' -> -• -•' -'• -• - • -•* - ' -'• - -'' -'• -•' - ' - -'- -• -' -*' -" -^ -'> --^ - - »-' -*-^-^ .!J-^-J -V .1 J-, .
--

Karush, A.D. Two approaches for measuring the performance of time-sharing

systems (SP-336U). Santa Monica, CA: System Development Corporation, May

mT. (AD-691 366).

I Topics Relevant |

I to System Development I Topic

land Evaluation Technology ! Ho. ABSTRACT

.

•

*

State cf the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.t Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

1.2 Parameter Determinations

1.3 Apparatus for Testing

1.1 Personnel for Testing

1.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.1 Conclusions and

Recoonendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 The paper discusses two significantly

different approaches for measuring the

performance of time-sharing systems. The

"analytical" approach involves the

insertion of probes into the system to

allow measurement and recording of the

system's most subtle behavior. In the

"stimulus" approach the system is

conceptualized as a "black box" containing

a limited number of known functions. This

measurement technique involves applying a

controlled set of stimuli to the black box

in order to activate its function and then

observing the results.

3.1 The analytical approach can have several

and applications. In program and hardware

3.2 analysis, the stochastic behavior exhibited

by the system and its interpretation in

terms of the executive or supervisor

function may be elucidated. In system

analysis the degree and manner in which the

man-machine software complex has affected

its environment in terms of costs, efficiency and security can be determined.

There are two paths that can be followed in

system research: (1) modeling or simulation; and (2) controlled experimentation

upon the time-sharing itself. The stimulus

approach can be applied to several applications. It can be used to indicate the

degree of throughput and response time

degradation under varying system loads; it

can indicate the degree to which the system

load varies; it can "take a reading" of the

system with respect to the volume of

service being requested from it; it can Lean

A-227

-V..N -A— -V•V V^V-V-VKV-rv«^ ..-..

M^mtatete*.
I Topics Relevant I I

I to System Development I Topic!

iand Evaluation Technology | No. I ABSTRACT

experimental test bed for examining the

effects of different scheduling algorithms;

and it can examine the effects of changes

to functional variables.

3.5 The analytical approach has several

practical measurement options: (a)

sampling measurement - this technique can

provide a frequency distribution which

describes the activity of the program; (b)

trace measurement - here the interest is

often focused upon the sequential behavior

of some portion of the system; (c)

accounting measurement - this is a

frequently required measure of a

time-sharing system. It is a summary of

resource utilization such as volume of file

storage, size of programs, etc.; (d)

logical measurement - the purpose of which

is to select for further evaluation a

subset of a data base as a function of the

content, description, or environment of

that data base; (e) playback measurement -

involves recording all the input to the

system or subsystem together with time

lags, and then rerunning the component of

interest with the recorded input substituted for the real input.

The stimulus approach also has several

practical measurement options depending

upon the environment in which the system

runs: (a) stand-alone environment - the

system Is viewed as a batch processor with

a benchmark program as the only program

running which provides the best measure of

throughput and response time; (b) benchmark

environment - in this situation all

benchmark programs are run simultaneously

simulating a "typical** user population; (c)

real world environment - this technique

Involves loading the system with an almost

full complement of users and running a

benchmark program simulating a user who has

a constant and known demand for service.

Several runs are needed to establish norms.

A-228

- - . - . ■■- . , _ , . ..... .-:•.:-. /vh
•

I Topics Relevant

I to System Development

land Evaluation Technology

•

.

-

i

5.1 Both the "analytic" and "stimulus"

approaches were used in SDC's ADEPT time

sharing system.

5.3 The development costs for the stimulus

technique were much lower. The analytical

technique provided data on the more subtle

behavior and complex interactions and

provided measures of performance on any

portion of the system. The stimulus

approach produced gross data on the

behavior of the system in terms cf

throughput and response time. The analytic

measurement required extensive offline

reduction and analysis before the data were

meaningful. The measurements for the

stimulus techniques were simple and

immediately obtainable.

S.M A number of areas for further development

for both approaches were mentioned.

A-229

••--*-*-' • - - - •- - *m . -..-- - - " -
Kelley, C.R. Hunan operator models for manual control. In G. W.-Levy (Ed.),

Symposium on applied model of man-machine systems performance (NR69H-591).

Columbus, OH: North American Aviation, November 1968. (AD-697 939).

Topics Relevant I

I to System Development {Topic

land Evaluation Technology ! No. ABSTRACT

1. State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.,t Performance

Criteria, Specific

3.5 Measurement Procedures

4. Planning Components

of the Process

1.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

1.1 Personnel for Testing

1.5 Test Plans

5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analyses

5.3 Findings Interpretation

5.1 Conclusions and

Recoamendations

1.3 Models are constructed for various purposes

and with various goals in mind. Four

general functions of models are:

(1) To formalize scientific theory.

(2) To improve man-machine theory.

(3) To improve automatic devices.

(U) To measure "hidden" processes.

2.6 Some models are built solely to replace roan

in specific tasks, and should be Judged

solely on how well they perform these

tasks, regardless of whether they perform

them like the man does. But the more

oonnon purpose of the model is to help

understand human performance. In that

case, four criteria for Judging the worth

of a model are proposed:

(1) Accuracy—Does the model's performance

of a task produce a response record

closely resembling that of a human

operator?

(2) Verifiability—Are the processes that

underlie the model's performance

similar to u.ose that underlie human

performance?

(3) Generality—Is the model applicable to

a large range of tasks, or only to a

few?

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Plann in«

A-230

. ^- -v^v- t*ata*-*._«•_* .'^ "_« v_k P_Ä .' MMM
Topics Relevant

I to System Development ITopic!

land Evaluation Technology i No. ! ABSTRACT

(4) Parsimony—Is the model unnecessarily

complex?

3.5 Th; author reviews five classes of human

operator models in more or less historical

sequence:

(1) Continuous transfer and describing

function models.

(2) Operator intermittency:

models.

Sampled data

I

(3) Operator non-linearities: Some

partial models.

(4) Computer analogs.

(5) Predicted and preview models.

6.1 The point is made that human performance is

not linear, and may be poorly represented

by linear control-theory models, except for

certain fairly simple or restricted tasks.

Also, human control is exercised not on the

basis of present error, but rather on the

basis of predicted future error.

A-231

• . * , - »"• .•.*«- -"-••'• •.".•*" .-k'-*»VOA> F .•*-«-.' •••"-- ."L**'^'*". " •»j^ .^ - • .'•
Kerner, H. and Beyerle, W.

modeling (V-PMS). Vienna:

University.

A PMS level language for performance evaluation

Institut fuer Digitale Anlagen Technical

Topics Relevant

to System Development ITopicl

and Evaluation Technology ! No. i ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.4 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recomnendations

1.1 A comparison of Register Transfer (RT)

level modeling and V-PMS is quite

indicative. While RT-ipodeling approaches

a restricted goal, viz. a hardware

structure capable of performing a few

hundred algorithms, a V-PMS level model has

a complete computer system as its target,

i.e. a composite of hardware structures of

the RT-level complexity cooperating under

the control of an operating system in the

execution of a load.

In order to construct a language suitable

for describing the hardware as part of a

total PMS-level model for performance

evaluation, the original form of PMS was

substantially changed by providing an

expanded set of building blocks with

corresponding definitions. This language,

with its clearly defined functions and

performance data, its unambiguous communication blocks and rules for interconnections,

provides a human reader with a clear understanding of the performance of components

and of their internal communication within

the computer system, and links the hardware

part to a model of an operating system (to

be supplied at a later time). For computer

readable system specification a syntax for

connecting the above symbolic components is

proposed. A description of the CDC Cyber

7U/CDC 6600 system examplifies the use of

the proposed language and its merits for

building performance evaluation models.

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potent!»Is/

Priorities

6.3 Research Planning

A-232

- - - - - i - - ' i .—.—i fc - iü •- - - _ .i^
Kiraly, R.J., Babinsky, A.D. & Powell, J.D. Aircrew oxygen aystem development;

Man-in-the-loop test report (NASA CR-73395). Cleveland, Ohio, TRW,Inc., July

1970.

I Topics Relevant I

! to System Development I Topic

land Evaluation Technology 1 No. ABSTRACT

1. State of the Art Review

of the Process

1.1 Cene-al System

Me»su. eaents

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

Z.H General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

cf the Process

3-1 Practical Measurable

Attributes

3-2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

Z.H Performance

Criteria, Specific

3-5 Measurement Procedures

4. Planning Components

of the Process

4.1 Analytic Methods

1.2 Parameter Determinations

4.3 Apparatus for Testing

I.D Personnel for Testing

1.5 Test Plans

5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recomnendations

1.1 This report discusses the test of a

"man-in-the-loop" program conducted with an

aircrew oxygen system.

2.1 The system was the NAOS Flight Breadboard

System which uses electrochemical generator

and carbon dioxide removal. Human and

animal test subjects participated in this

study.

2.2 The mission of this system was to provide

aircrews with a safe, reliable, and compact

oxygen system.

2.3 Two experiments, one involving animals and

one involving human test subjects, were

conducted in a laboratory situation.

2.5 The ultimate performance requirement of

this system was that aircrews receive the

life support necessary during flight.

2.6 The object of the study was to provide a

safe, reliable, compact system which would

replace present systems minimizing the need

for ground support facilities and reduced

time and effort for servicing.

3.1 Two experiments were conducted in this

study. The first experiment was conducted

to measure the effects of oxygen generated

by the Flight Breadboard System on lung

tissue of small animals. The second

experiment involving human test subjects

concerned the physiological effects and

comfort of the equipment.

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-233

. ^ %

:

-- 

^_, . . —
Topics Relevant

I to Systen. Development i Topic

land Evaluation Technology ! No. ABSTRACT

3.3 The specific performance requirement of

this system was that the system meet safety

and comfort limits.

3*4 Subjective judgments of comfort were

obtained when evaluating different masks

and suspension systems. Determination of

the physiological aspects were obtained by

physical examination of the test subjects

and based on expert Judgments.

U.3 The animal tests were conducted in a

plexiglass enclosure through which

rebreather gases were calculated. Human

tests utilized aviation oxygen masks and

the Flight Breadboard System.

4.4 Hamsters and mice participated in the

animal experiment. Four members of the

project team volunteered for the human

tests.

U.5 The animals received a single acute

exposure of 3.5 hours duration and a

chronic exposure of 5.5 hours/day for ten

consecutive days of rebreather gases. The

animals were sacrificed and lab examinations conducted on lung tissue.

In the human tests, two series were

conducted. In the first test, subjects

experienced the system operated with and

without safety presssure to determine

comfort levels and possible physiological

damage to respiratory systems. The second

test was to determine relative comfort of

alternative equipment. In both tests

carbon dioxide levels and oxygen levels

were monitored. Mask leakage measurements

were also made in relation to the employment of safety pressure and comfort levels.

5.4 The system operated successfully in most

areas of expected performance. However,

mask leakage problems oust be solved if the

full advantage of closed-loop oxygen

systems is to be realized. It is

recommended that future testing make

provisions for a higher oxygen rate to be

used.

A-234
Klein R. Integrated operational test and analysis procedures for small arms

weapon systems evaluation. Manuscript in preparation. (Undated)

.

I Topics Relevant

to System Development

and Evaluation Technology

3-

State of the Art Review

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requir^nents, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

1.1

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM) 2 .1

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components 2 .2

of the Process

2.1 System Definition

2.2 Mission Definition 2 .3 2.3 Environment Definition

2.U General Constraints

2.5 Performance

Requirements, Ultimate 2 14

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

2.5

2.6

3.1

1.2 Parameter Determinations

4.3 Apparatus for Testing

1.1 Personnel for Testing

1.5 Test Plans

Application Components 3.2

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation M

5.1 Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Prioritits

6.3 Research Planning

This report addresses the integrated

operational test and analysis procedures

for small arms weapon system evaluation.

The system w..3 composed of infantrymen,

their weapons, and equipment.

The mission was the use of small arms in

combat situations by infantrymen.

Simulated combat conditions were utilized

in this study.

The ability to duplicate combat actions and

tasks in a test facility affected the

validity of test results.

The ultimate performance requirement of

this system is mission accomplishment in

which the mission is to close with and

defeat the enemy.

One of the primary measures of success of a

combat mission was described as the number

of enemy casualties.

The performance of infantrymen using small

arms weapons was evaluated. The criteria

used were accuracy, sustainability,

responsiveness, and reliability.

The number of hits was the measure of

effectiveness of this system.

The criteria for the evaluation of weapon

system performance fell Into four

categories:

A-235

*^~- '------

.'..• ^ # -' '.*•..
\ Topics Relevant

to System Development I Topic I

and Evaluation Technology | No. i ABSTRACT

(1) Accuracy (number of hitsl, hit

probability, first round hit

probability, engagement hit

probability, distribution of near

misses.

(2) Sustainability (number of hits per

round and hits per basic load).

(3) Responsiveness (time to fire first

round, time to first hit, time betwe^

rounds, time to shift fire).

(1) Reliability (number of malfunctions,

numbers of rounds between

malfunctions, and time to clear

malfunctions).

3.5 For the purpose of this study, the missioi

accomplishment measure was equal to the

number of target hits.

1.1 Sample size should be based on the specif

test criteria and selection of the appropi

ate MOE. The technique for performing th

primary analysis is a 3x2x2 factorial exp

iment. The factors were facilities, weap

ons and modes of fire. A linear model cai

be developed and an analysis of variance

can be performed.

4.2 The sample of test soldiers should be

representative of the infantry as a whole

in terms of age, rank, experience and

physical attributes. Weapcn should be

assigned on the basis of a performance

measure. Training was found to be a

substantial source of bias. An accelerat(

training class was held to familiarize th(

subject with new weapons. Results showed

bias favoring a standard weapon but time

limitation prevented elimination of this

bias. A balanced experimental design was

implemented. The test facility was

designed to duplicate as closely as

possible the "real-world" combat-situatioi

A-236
I Topics Relevant I

I to System Development I Topic!

iand Evaluation Technology ! Ho. j ABSTRACT

M.H Ninety-six infantrymen/in 96 rifle systems

participated in this experiment.

U.5 The components of the test plan were:

selection of subjects, determination of

sample size, weapon assignment, training of

subjects, scheduling, test facility

determinants, test implementation, and data

analysis.

I _

A-237

-\

. -.- .-..--•..--". .--v . . ... :-v-:•• .-•.- . .-.•-v-v-y. ••;•".

- - - - . .
Klein, R. D. & Thomas, C. B. The development of combat related measures for

aaall arms evaluation. Fort Benning, GA: US Army Infantry Board, June 1969.

(AD-713 552).

I Topics Relevant I

I to System Development I Topic

land Evaluation Technology ! No. ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.<l General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.'» Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 This paper discusses the development of

combat related measures for small arms

evaluation.

2.1 The system studied included the infantry

soldier, his weapons, and equipment.

2.2 The mission of the system was the most

effective use of men, equipment, and

weapons in a combat situation.

2.6 The specific requirements of the system was

the achievement of a "hit" during a

quick-fire engagement in the shortest

period of time.

3.1 A list of 26 separate combat-actions was

prepared. Secondly, a list of tasks

normally accomplished by the infantryman

when executing the combat actions was

developed. It was determined that three

basic tactical situations (attack,

quickfire, and defense) would accommodate

all these actions and tasks.

3.2 Twenty-six measures of effectiveness were

developed. These included time to first

round, time between trigger pulls,

distribution of near misses, time to shift

fire, and hits per pound expressed as a

percent of a soldier's basic load.

3.5 Several measurement procedures were used:

the time to fire first roi -d measures, the

time it takes for a soldier carrying his

rifle at ready position to engage a

surprise target, and the time between

trigger pulls which is defined as an

A-238

- - - - --

.- •.

- - -• - -

.--.V...-.V.

-- -
I Topics Relevant I

1 to System Development !Tople|

land Evaluation Technology i Wo. ABSTRACT

t_

indicator of the soldier's ability to

absorb the recoil, acquire the target,

obtain a new sight-picture, point the

weapon and squeeze the trigger. An

effectiveness scale of five levels was

developed.

1,1 The firing engagements were categorized

according to the degrees of effectiveness

as defined by whether or not the firer

achieved a hit. Those engagements which

required multiple trigger pulls and failed

to achieve a hit were defined as least

effective. Those which resulted in a hit,

whether single or multiple trigger pulls

were required, were designated as the most

effective. Time-of-first round, target

range, time between bursts, time between

trigger pulls, number of rounds to first

hit, burst size, and burst hit probability

were presented graphically, and provided a

comparison between the weapons being

tested.

4.3 The test equipment were two different

automatic rifles, tested in a simulated

combat-firing facility.

5.3 It was determined that the potential of one

weapon was greater than the other being

tested. Two findings were considered

important: (1) the service test must

determine the weapons systems optimum

operating mode to yield complete information on weapon potential and (2) training

procedures are related directly to weapon

performance and therefore should not be

considered separate entities.

5.4 It is felt that this technique of equating

a specific measure to real world effectiveness represents an advance in military test

procedures, but is limited in that the

"real world" is still a simulated combat

firing facility. However, with these

limitations the facility is a dynamic test

environment and brings into play many of

the influencing variables common to the

A-239

1 V

" • * ' > ' • S '

-• • - - --- - J^j. "W , "k V V ." V v

^.

-.
1

I Topics Relevant

i to System Development

land Evaluation Technology

i

iTopic

I No. ABSTRACT

combat environment and provides an

improvement in operational testing. The

ultimate value to be realized from this

methodology is how well these statistical

measures can be weighed in this

"real-world" atmosphere.

A-240

.^

. - - - - •. -.

---•- *- "- * -«'

- - - "- N

- . - »• - ^_ AH ^J-^.
.•-

Kleinman, D.L., & Baron, S. Manned vehicles systems analysis by means of

odern control theory (NASA CR-1753). Cambridge, MA: Bolt, Beranek and

Newman, Inc., June 1971.

I Topics Relevant

! to System Development iTopic I

land Evaluation Technology | No. I ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.'! General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.'! Performance

Criteria, Specific

3-5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

9.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recomsendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 Optimal control and estimation theory may

be used to develop a model of human

response in manual tracking tasks. The

tasks considered were those in which the

controlled element was linear and the

system was disturbed by a white noise

input. The model included representations

of human limitations and a cascade

combination of a Kaiman filter, a

least-mean-squared predictor, and a set of

optimal feedback gains as compensating

elements. An "optimal scanning mechanism"

was also added to the model to account for

situations where the human operator must

visually scan several instrunents in order

to achieve his control objectives.

The use of the model in predicting task

performance, controller describing

functions, and power spectra was

demonstrated. The model was then validated

by comparing model results with experimental data from three simple, but classical,

manual control tasks.

Sensitivity studies of simple manual

control tasks (i.e., single input, single

display indicator) were conducted. The

measures examined incuded:

(1) Neuromotor time constant.

(2) Time delays.

(3) Observation noise.

CO Motor noise.

A-241

• -• -.:•.-

^--^■■^vvv^vy^^::^ .^.

.•.--,

.- v.- -. .-- V

v
I Topics Relevant I I

I to System Development !Topic I

and Evaluation Technology j No. I ABSTRACT

A prediction of pilot performance in a

hovering task was also attempted.

Agreement between measured and predicted

quantities was obtained, demonstrating the

value and potential of the optimization

approach to manned-vehicle systems

analysis.

A-242

t^£*:£ •: ;-.•-:-. :•. :v.v;'-:v:-.:. .
Knoop, P. A. Survey of human operator modeling techniques for measurement

applications (AFHRL-TR-78-35). Wright-Patterson AFB, OH: Air Force Human

Resources Laboratory, Advanced Systems Division, July 1978.

Topics Relevant I

I to System Development I Topic

land Evaluation Technology ! Mo. ABSTRACT

•;

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPU)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

<t.2 Parameter Determinations

4.3 Apparatus for Testing

4.'» Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoonendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1,1 The purpose of this study was to review

existing human modeling techniques and

evaluate their potential utility for

performance measurement applications. It

appeared that existing performance

measurement techniques do not have the

capability to support the type of flight

simulation research that entails accounting

for the perception and utilization of

various cues.

Since model validity Is particularly

Important In the case of the envisioned

measurement applications, the first task

was to Identify the major human operator

characteristics that ought to be accounted

for.

A review of work dating from 19^ was

conducted and this material was categorized

Into six types: (1) describing functions,

(2) optimal control model, (3) discrete

and finite state methods, (1) adaptive

techniques, (5) preview models, (6) other

nonlinear approaches. A survey of models

in each category was made by reviewing the

literature and summarizing various modeling

studies. Models in each category were

evaluated based on the extent to which they

represented the Identified htman operator

characteristics as well as other aspects of

their general validity for performance

measurement applications.

5.4 Results show that none of the models

Implemented more than a few of the operator

characteristics; many are based on

assumptions which are unacceptable for

A-243

.--. .:.: .-

•-> .•.
I Topics Relevant

i to System Development I Topic

land Evaluation Technology j No. ABSTRACT

measurement applications and others have

not been developed far enough to justify

their use as a po.nt of departure for

measurements.

:

It is concluded that existing models are

not sufficiently representative of known

characteristics to be useful for general

applications in performance measurements.

A-244

A\VC .'.-.. • . -". "' -". . • • . -. •

-• .-".-."."- . '.-.
Knowles, W.B., Burger, W.J., Mitchell, M.B., Hanifan, D.T. & Wulfeck, J.W.

Models, measures, and judgments in system design. Human Factors, 1969, 11,

577-590

• r.

Topics Relevant | |

I to System Development !Topic I

land Evaluation Technology I No. I ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environaent Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

1.1 Current practice of system design commonly

requires the use of analytical models.

While such tools are finding increasingly

wide application and acceptance, their use

is often limited by the lack of data in

forms suitable for modeling effects.

Overall, there can be little doubt that the

design of systems would be greatly

facilitated if tasks and equipment could be

treated within the same conceptual scheme.

Unfortunately, techniques originally

developed to describe and predict human

performance for that purpose have fallen

short of the goals.

6.3 The following approach is suggested for

studying system design:

(1) Identify a practical situation where

suitable populations of expert Judges

and tasks can be obtained, and where

estimates and actual measures of task

performance can be obtained.

(2) Conduct a study to identify the

relevant dimensions along which to

categorize tasks and Judges, and to

obtain correlations between estimated

and actual performance.

(3) Develop means for structuring the

presentation of stimulus materials,

i.e., task descriptions, so that the

relevant aspects of the tasks are

suitably displayed.

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-245

/vjifev..\ •-. •,
I Topics Relevant

i to System Development {Topic

land Evaluation Technology ! No. ABSTRACT

(1) Develop means for classifying judges

and for selecting "valid" individual

Judges.

(5) Conduct a study to revalidate the

estimates obtained with the improved

methods.

(6) Develop category scaling procedures to

be used generally in evaluating

equipment-oriented tasks, and develop

a set of standard stimulus materials

to be used in differentiating among

prospective judges.

•

. >.

A-246

•- .- .•-.'-. • . - .s .--• . .--.--. .- -.- •.- -.- - _ • -.- o .
Kreifeldt, J. Analysis of predictor model. Unpublished Preliminary Data,

Massachusetts Institute of Technology, }95li.

Topics Relevant I

I to System Development I Topic

land Evaluation Technology ! No. ABSTRACT

SUte of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environnent Definition

2.'* General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.^ Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

<t.2 Parameter Determinations

4.3 Apparatus for Testing

lt.lt Personnel for Testing

11.5 Test Plans

Appl ication Coaponents

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.11 Conclusions and

Recooaendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 This memorandum discusses an analysis of a

predictor model which realistically

accounts for Inputs which have spatial as

well as time dependence. An analysis is

presented which attempts to abstract the

real world to some degree.

In this outline a model of a human operator

controlling a vehicle was analyzed. The

model attempts to account for the fact that

in many situations (i.e., driving) the

operator has an input which is not a single

point in time but an input which has

spatial as well as time features, that is,

he can look at the road ahead.

The sampling theorem in spatial coordinates

was Invoked in order to treat the

time-space input as k discrete Inputs to

the operator simultaneously available. The

model then states that the operator runs

some sort of thought experiment in which

position is extrapolated and future error

is computed if the same control signal is

maintained. This computed and weighted

predicted error forms the basis for the

operator's control action.

The model performs the thought experiment

by computing from vehicle initial

conditions and command signal what the

errors will be at the previewed points.

These individual errors are simultaneously

A-247

. • - .
I Topics Relevant

I to System Development

land Evaluation Technology

ITopicI

1 No. i ABSTRACT

computed and weighted in a length of time

requiring % seconds.

The transformed impulse response for this

model was derived and composed of discrete

and continuous elements.

This impulse response was specifically

evaluated for a first-order vehicle and two

input points. It was seen that if stable,

it eventually reaches a reference step

height input.

A-248

.•-.'.-J*'.-

' -• -»
Kurke, M.I. Operational concept analysis and sources of field data.

Human Factors. 1965, (7), 537-5^.

Topics Relevant

I to System Development

land Evaluation Technology

i

! Topic

! No. ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.H General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3-3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3-5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoanendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 The human factors approach to an operational

analysis should be commensurate with that

of the operations research approach. One

approach to this methodology is described

in this paper. A hypothetical example (i.e.,

a rock) is given in which the essential

elements of analysis are presented in order

to determine whether an item of materiel

should be included in an Army inventory.

2.1 The five major functions described are:

firepower, mobility, combat service support,

intelligence and reconnaissance, command,

control and communications.

2.2 The functions are analyzed into subfunctions

which are relevant until a level of description is reached that can be measured operationally. For example, firepower can be a

measure of hit probability as a function of

range. In addition, if is determined from

an analysis of the subfunction, mobility,

that the ability to throw the hypothetical

weapon (i.e., the rock) while running is a

very important operational capability, then

this ability would be an attribute measured

in terms of accuracy when using protective

equipment versus not using protective

equipment.

3-1 The essential elements of the analysis

include: effectiveness in attaining objectives, effectiveness when chemical agent

protective equipment is being used, training

requirements, and operating costs.

••1 The field data collected for military OR

studies come from three sources: field exercises, troop tests, and field experiment".

A-249

-.v .-.•:
Topics Relevant I

to System Development I Topic

and Evaluation Technology ! No. ABSTRACT

The characteristics of these three sources

of field data for operational concept and

evaluation have been compared with the

field data sources for systems analysis

and evaluation.

The three sources of field data are defined

as follows:

(1) Field Exercise—an exercise conducted

in the field, under simulated war

conditions, in which troops and armaments of one side are actually present,

while the other side may be imaginary

or in outline.

(2) Field Experiment—an investigation to

experiment with or evaluate new or

revised doctrine and organizations,

and new, modified or current materiel

in order to develop combat capabilities

(3) Combat Development Troop Tests—a field

investigation designed to test the

ability of a prototype organizational

structure to follow a specific doctrine

using specific equipment to complete a

specific mission and/or to test the

concept of operations as limited by the

structure and functions of a prototype

organization.

A table is presented which compares the

three sources of army field data in terms

of methods, kinds of analyses, reliability

of data and types and usefulness of results.

In a study of 32 tests and experiments published in 1963, it was noted that only the

field experiments and troop tests provided

systematic data.

The field experiments tended to gather base

data, compare systems and collect information about specific variables. However,

although more than half of the conclusions

in these experiments were drawn from objective data, the balance supplemented their

results with subjective data.

A-25Ü

._- . .
Topics Relevant I

to System Development ! Topic

and Evaluation Technology ! No. ABSTRACT

Some troop tests were developed to collect

base data about a system but only one out

of eight compared systems. Generally, in

these reports, test objectives were not

clearly stated. Few of these tests investigated specific variables although all investigated the total system. All troop

test conclusions were based on subjective

data. More than half of these troop tests

were uncontrolled both statistically and

experimentally and none presented anything

more sophisticated than a measure of

control tendency.

The test plan for functional analysis would

include:

(1) Development of a frame of reference

for operations.

(2) Determination of the functions or subfunctions relevant to describing

effectiveness. In this analysis, some

screening or weighting of functions in

terms of their relative importance may

be performed.

(3) Development of operational criteria

based on the determination of relevant

functions.

(U) Refinement of the essential elements of

the analysis in terms of specific performance criteria.

(5) Determination of the appropriate operational criteria, their measures and

the correct mode of collecting the

information.

(5) Determination of appropriate data

collection techniques, i.e., field

experiment supplemented y computer

simulations.

A-251

.-.•.•.•.

v •>>> »^ .-.-•.

•.--".-
Kvalseth, T.O. A decision-theoretic model of the sampling behavior of the

human process monitor. IEEE Transactions on Systems, Man, and Cybernetics,

November 1977 SMC-7(11). '

Topics Relevant |

I to System Development |Topic I

land Evaluation Technology ! No. | ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.H General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3-3 Performance

Requirements, Specific

l.t Performance

Criteria, Specific

3.5 Measurement Procedures

1.3 The present document presents a theoretical

model for the characterization of the

sampling behavior of an optimal process

monitor. The dynamic decision model was

based on the proposition that an ideal

monitor adopts a sampling strategy that, at

any point, maximizes some expected value

(worth or utility) for the monitoring of a

single signal with a fixed cost of sampling

and of the signal exceeding certain

threshold units. The present model was

shown to lead to the strategy that sampling

should occur when the conditional

probability of the signal going beyond the

threshold units exceeds the ratio of the

sampling cost to the threshold cost.

Planning Components

of the Process

«.1 Analytic Methods

4.2 Parameter Determinations

I.] Apparatus for Testing

l.t Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-252

.v:v\
Labor/Management Task Force on Rail Transportation. Terminal performance

easureaent systems. (DOT-FR-M-3003). Washington, DC, May 1975. (PB-252

225).

Topics Relevant I

to System Development I Topic

and Evaluation Technology | No. ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Mo*^ :'>M)

Contextual Cooponei s

of the Process

2.1

2.2

2.3

2.«

2.5

2.6

System Definition

Mission Definition

Environnent Definition

General Constraints

Performance

Requirements, Ultimate

Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.>i Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5-3 Findings Interpretation

5.4 Conclusions and

Recoonendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Reserrch Potentials/

Priorities

6.3 Research Planning

2.1 The St. Louis [Railroad] Terminal Project

is an experimental program seeking

ultimately to produce a more saleable

transportation service and greater profit

to the railroads. The experimental

procedure places heavy emphasis on

quantitative measurement of the impact of

operational changes. To determine the best

method for measuring performance of the

terminal as a whole, a seminar on terminal

performance measurement systems was

conducted by the St. Louis Project Team.

2.2 Current terminal performance measurement

systems were found to be designed to

support one or more of the following

functions:

(1) Evaluate performance and trigger the

planning process to develop changes

that will produce improved

performance.

(2) Evaluate experimental changes in

operations to determine the actual

improvement in performance.

(3) Monitor the operations to provide

information that results in corrective

action to prevent a deterioration in

performance.

(4) Assess the performance of the managers

responsible for the operations.

U uL A-253

-S -N >> .- v.vvv*-v-v. .•-.•.-. -.-.
'

Topics Relevant

to System Development I Topic

and Evaluation Technology ! No. ABSTRACT

3.1 Observations of the various terminal

performance measurement systems indicated

that:

(1) All measurements systems use car

movement data collected by real-time

[automated] systems, with no

additional burden placed on clerical

forces.

(2) Speed of car movements is mersured by

the time cars spend under a terminal

superintendent's responsibility.

(3) Reliability of car movements is

measured by comparing elapsed time

with a standard.

CO Most systems measure the car time

costs of car movements.

(5) The systems currently in use do not

measure all of the variables with

which terminal management must deal.

73

A-254

r. < . . • . •. . - • .-.•.•'.•".-''.- .•'. .•".-.-".-. ,'.- . . .".-'.-
Larson, O.A., Sander, S.I. 4 Steinemann, J.H. Survey of unit performance

effectiveness measures (TRT^-II). San Diego, CA: Navy Personnel Research and

Development Center, January 1971». (AD-77M 919).

Topics Relevant I

I to System Development I Topic

land Evaluation Technology I No. ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Coaponents

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environnent Definition

2.4 General Constraints

2.5 Perforaance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recomnendations

Further Research Areas

6.1 Heasureaent System

Limitations

6.2 Research Potential* '

Priorities

6.3 Research Planning

1.1 A survey to determine the state of the art

of performance assessment systems and

methodologies was conducted to support a

Marine Corps iequirement for improved

measures of performance effectiveness to be

used in conjunction with the Tactical

Warfare Analysis and Evaluation System

(TWAES).

In addition to a literature search,

researchers observed field exercises and

examined related U.S. Army systems.

A number of areas were identified as

needing support efforts including timely

reporting and assessment of events by

umpire personnel.

S.M It was concluded that the literature does

not contain substantive research studies

defining human performance assessment

methods and criteria suitable for direct

application to TWAES. Studies were found,

however, that will be of use in initiating

and structuring the research in this area.

Organizational, situational and evaluative

factors all hold promise as potential

sources of performance effectiveness

measures along with the use of Delphi-type

approaches for the development of interim

methods and criteria.

A-255

•j ->
Lawrence, D.H. The evaluation of training and transfer programs in terms of

efficiency measures. The Journal of Psychology, 1954, ^6, 367-382. (AD-87

172).

I Topics Relevant

I to System Development

land Evaluation Technology

Topic!

No. I ABSTRACT

.--•

State or the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Pe. Tormance

Criteria, Specific

3-5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

1.2 Parameter Determinations

1.3 Apparatus for Testing

1.1 Personnel for Testing

1.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.1 Conclusions and

Recoonendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 This paper discusses the decision making

involved in designing training programs.

It is noted that programs are designed to

produce the desired level of proficiency

with the minimum outlay of time, energy and

cost. However, it is theorized here that

the implications of such a principle are

not explicitly formulated as they bear on

the efficacy of the decisions or on the

research needed as a basis for such

decision.

This paper attempts to remedy part of this

deficiency by means of a preliminary,

simplified simulation of the training and

transfer problem in terms of the principle

of minimization of expenditure.

Simplified assumptions are made about cost

functions and learning curves involved in

training to permit a clear expression of

the methodology involved. Any use of this

methodology in other specific training

situations may involve a different set of

assumptions.

A mathematical formula is presented which

demonstrates a relationship between the

cost and the stage of training. Phis

states that, in general cost is universally

related to the level of proficiency

achieved at any given time. Factors

influencing the cost of training an

individual are considered as well as the

average cost for group training.

The paper discusses ways in which training

costs can be reduced by means of improved

A-256

. .•.-.-
I Topics Relevant I

I to System Development I Topic!

and Evaluation Technology | No. i ABSTRACT

training techniques, selection of

individuals to be trained, and by the

utilization of the transfer of training

procedures.

It is pointed out that standardized

training programs which bring all

individuals to a level of proficiency can

result in overtraining for some. Cost

benefits of considering individual

differences by modifying training programs

are discussed.

The final section of this paper deals with

the employment of transfer of training

situations as a means of reducing training

costs. The assumption underlying the

transfer paradigm is that an individual can

be trained to a desired level of performance easier and with less cost if he is

first trained on a preliminary task.

A-257

----- -
Levy, G.W. Criteria for selection and application of models. In G.W. Levy

(Ed.), Symposium on applied model of man-machine systems performance

(NR69H-591J. Columbus, OH: North American Aviation, November 19>8. (AD-697

939).

I Topics Relevant !

i to System Development I Topic

land Evaluation Technology I No. ABSTRACT

1. State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3-5 Measurement Procedures

4. Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

5. Application Coaponents

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

6. Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.3 This paper presents some th jghts on

considerations for selecting and applying

man-machine system models. The author

classifies these considerations as:

(1) Those relating to the model itself,

i.e., its output, input, and form.

(2) Those relating to the precision and

validity of the model predictions.

3.2 Concerning the model's output, consideration should be given to whether the model

predicts human performance or system

performance. System performance measures

involve parameters of both human and

machine and even their interaction. Such

measures imply a system effectiveness mode

of thought, a concern with overall system

performance rather than human performance

per se. However, it becomes necessary to

separate man and machine performance

explicitly and to develop human performance

submodels that include system-oriented

human performance measures. Examples of

these may include mean time to repair,

slant range at target identification, etc.

4.2 As regards model input, an initial

consideration should be the number of

variables involved. Often, too many

variables are specified, either because an

attempt is made to achieve an unrealistic

predictive precision, or because it isn't

possible to separate meaningful variables

A-258
•

I Topics Relevant I

I to System Development I Topic!

land Evaluation Technology i No. I ABSTRACT

from the great number of possible

variables. On the other hand, meaningful

variables sometimes are overlooked because

they haven't been studied empirically.

Operator motivation is an example of a

variable that most models disregard. A

second consideration related to input is

the existence of the necessary data.

Thirdly, model parameters may also be

treated as needed input for application of

the model: the significant consideration

is whether these parameters are fixed

beforehand or estimated from the data.

Concerning the form of the model, the first

issue is whether it is an empirical or a

theoretical model. Theoretical models are

obtained by a hypothetical-deductive

procedure, empirical model equations are

obtained by curve fitting. Theoretical

models provide a guide to research and a

progranmaticity which empirical models

lack, but theoretical models involve

assumptions which may or may not have been

tested. Another issue concerning form is

"what kind of man is being modeled." Is it

an individual with individualistic

parameters or is it the famous average man?

Is he trained or untrained? Few models

handle the problems of individual

differences and training.

5.3 Concerning the precision or accuracy

required of a model, this is generally

regarded to be a function of the stage of

the system life cycle in which the model is

being used. However, it is believed that

the same levels of precision are required

in the initital stages of design as in

later applications. In order to be truly

useful, applied models must consider the

relevant interactions of design parameters

with difficulty of conditions and must

possess a sufficient degree of accuracy.

Unfortunately, the criterion of how much

accuracy has never been spelled out.

A-259

,1,1
•

1

I Topics Relevant

I to System Development I Topic|

!and Evaluation Technology I No. | ABSTRACT

6,1 Turning to validity, it is noted that the

vast majority of man-machine systsm models

have never been validated. It is suggested

that a research design termed the "field

laboratory validation design" is very useful

in developing and validating applied

models. This design calls for collection

of performance data and input data in field

situations, with the input data recorded

for use in laboratory studies aimed at

model development. The models in turn are

validated by comparing their outputs with

the pre-collected field performance data.

J

A-260
Lindsey, M.B. A human performance Investigation In a NAVORD weapon system -

Part 2 (Report 27-31»9). Annapolis, MD: Naval Ship Research and Development

Center, Propulsion and Auxiliary Systems Dept., March 1974. (AD-918 202L)

I Topics Relevant

! to System Development iTopic

land Evaluation Technology ! No. ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.'* Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

U.2 Parameter Determinations

4.3 Apparatus for Testing

H.H Personnel for Testing

«1.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.t Conclusions and

Recooaendations

Further Research Areas

6 I Measurement System

Limitations

6.2 Research Potentials/

I •lorlties

6.3 R search Planning

1.1 This study is part II of an investigation

regarding integration of ordnance systems

personnel functions and the design of

man-made interfaces. A specific

modification of a methodology was developed

to yield quantitative data for the purpose

of determining the performance levels of

maintainers/operators in ASROC

(Antisubmarine Rocket) launching groups.

1.3 If "performance" means demonstrable

operational capability, then a system's

performance is taken as the total behavior

of its man and machine module components

interacting over a specified period of time

in the environment for which it was

designed; its reliability is the

probability of performance.

3.1 Previous researchers have categorized the

major variables that affect system

performance as:

(1) Personnel

(2) Equipment

(3) Procedure

(4) Environment

(5) Administration

(6) Interaction

4.1 The statistical methods used to determine

the performance levels were calculations of

A-261
•

ft

! Topics Relevant I

I to System Development I Topic

land Evaluation Technology I No. ABSTRACT

the mean (including the grand mean), the

standard deviation, and the range.

M.3 A job performance questionnaire was

developed for the job activities as a

cross-section of ASROC; the work done by

Siegel on multidimensional scaling was

found to be applicable with only minor

revisions made (definitions appropriate to

ASROC were adjusted accordingly.)

.

A-262
LTV Aerospace Corporation. Shuttle active thermal control system. Vol. 3;

Modular radiator system test data correlation with thermal model (Rep. No.

T169-28, NASA CR-M0269). Dallas, TX: LTV Aerospace Corp., Vought Systems

Division, December 1973.

Topics Relevant

1 to System Development 1 Topic I

and Evaluation Technology j No. I ABSTRACT

State or the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

4.2 Parameter Determinations

1.3 Apparatus for Testing

4.1 Personnel for Testing

4.5 Test Plans

Appl ication Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoonendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Frloritl«»

6.3 Research Pia. ing

1.3 Accurate predictions of the space shuttle

radiator system performance is of prime

importance in the design and development of

the heat rejection system. Due to the

large size of the radiator system it is

impractical to determine the optimum radiator system by test. An accurate model is

needed to study parametrically all design

variables and insure optimum radiator

performance. This volume presents results

of an analysis to correlate the system

thermal model with test data.

2.1 The modular radiator system submitted to

testing consisted of modular test panels,

each made up of 12 tubes arranged in a "U"

shaped pattern. The innermost tube is the

prime tube. The other 11 tubes comprise

the main system. Heat rejection is

regulated by controlling the flow split

through the main system and the prime tube.

At high heat loads, approximately 99X of

the flow is routed to the main tubes;

minimum heat rejection occurs when

approximately 99% of the flow is routed to

the prime tube.

2.4 The multi-panel configuration proposed for

use on the shuttle requires that the model

predict interaction between the panels;

thus, dictating a separate model for each

panel.

A-263
1 Topics Relevant 1

! to System Development I Topic I

land Evaluation Technology ! No. ! ABSTRACT

3.1 The primary objective of the thermal model

is to provide a tool for performance

predictions of the radiator system under

the design conditions of maximum and

minimum heat rejection. A single tube is

used to model the bank of 11 main tubes.

The resulting model thus consists of two

separate models - one for the mair system

for maximum heat rejection conditions, one

for the prime system for minimum heat

rejection. There are no thermal

connections between the two systems.

5.2 Comparisons were made between test results

and (model) predictions for selected test

points. The model predictions agreed well

with the test data. Excellent agreement

was achieved for the high heat load

conditions. Good performance predictions

also were produced under minimum load

design conditions, although these correlations generally were not a? good as were

those for high load.

In summary, a thermal model of the proposed

space shuttle modular radiator system has

been developed and verified by comparison

to thermal vacuum test data. Application

of the test panel modeling techniques to

the (actual) flight panel should prr ide an

accurate model for the radiator system

performance evaluation.

A-264
r.:

Lyons, J.D. Measuring effectiveness: Quality control of training. In E.A.

Cogan 4 J.D. Lyons Frameworks for measurement and quality control.

(HumRRO-PP 16-72). Papers presented at New York University First National

Annual Training in Business and Industry Conference, New York City, March

1972. (AD-748 081).

I Topics Relevant I

I to System Development I Topic

land Evaluation Technology I No. ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM'

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.4 Performance

Criteria, Specific

3.5 Measurement Procedures

Plannirg Components

of the Process

1.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusion- and

Recomnendatlons

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Reaaarch Planning

1.1 The process of developing the raw material

of human potential deserves a system of

quality control at least as carefully

devel^c-a as that applied to the

manufacturing process. The quality control

system must accomplish four major

objeci ives:

(1) Quality assurance.

(2) Control of student progress.

(3) Training program improvement.

(4) Training system diagnosis and change.

Quality assurance requires that

specifications be delineated in terms of

operational requirements and these

requirements be reflected in end-of-course

proficiency measures.

Control of student progress is accomplished

by developing a means of selecting and

organizing the learning experiences to

facilitate achievement of objectives. This

can somecimes be achieved by dividing the

training program into modules each with its

sub-objectives.

A systematic quality control process must

identify weaknesses and strengths in the

program by assessing and diagnosing the

performance of the trainee.

A-265
.] I Topics Relevant I I

I to System Development I Topic!

r land Evaluation Technology ! No. ! ABSTRACT

I

.< The essential elements of a training

quality cortrol system

are:

(1) Training objectives (performance

requirements).

(2) Proficiency and diagnostic measures.

(3) Data reduction and analyses.

(4) Procedures for decision and corrective

actions.

(5) Communication procedures.

(6) Managerial support.

A-266
t

Markel, G.A. Toward a general methodology for systems evaluation (352-R-13).

State College, PA: HRB-Singer, Inc., July 1965. (AD-619 373).

Topics Relevant I

I to System Development iTopic

land Evaluation Technology | No. ABSTRACT

-

s -

State of the Art Review

of the Process

1.1 General Systen

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Ccoponents

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.t General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Plann:ng Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5-3 Findings Interpretation

5.4 Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 This report considers some of the issues

involved in a general theory of systems

evaluation. It suggests an organized and

comprehensive framework for approaching any

systems evaluation.

Definitions of systems are considered and

characterized as an assembly of

interrelated elements acting as an integral

ensemble in performing some functional

objective.

The report examines some of the issues

Involved in realization of practical

evaluations. It discusses the need to

define the problem — why is the evaluation

being performed, which parts of systems

concepts are involved, etc. It is felt

that breaking down the problem into

smaller, more tractable problems

facilitates a workable approach to any

evaluation.

Fundamental considerations such as criteria

selection are discussed. The report

stresses the need to select the criteria

with great care. The underlying eubstance

of the evaluation process is measurement

and the key to successful measurement and

evaluation is to be found in criteria

selection.

It is suggested that the development of

general theory of systems evaluation can be

approached by identifying and defining

these elements which can provide a basis

for the overall evaluation of any system.

These elements may be described in three

A-267
_ ,- Topics Relevant

to System Development I Topic'

and Evaluation Technology | No. I ABSTRACT

broad areas of primary concern for systems

in general:

(1) Systems structure.

(2) Systems operations.

(3) Systems performance.

It is felt that a valid approach is to

employ a modular concept of systems

evaluation addressing each basic factor as

a separate module. Each application uses

only the modules most appropriate to that

system and the results obtained from the

various modules are integrated

subjectively.

Different modeling concepts are discussed

in the last section of this report. It is

felt that network models are best suited

for structural aspects, hierarchical tree

models are best for studying aggregation of

operational aspects, and black box models

provide the best perspective for overall

performance appraisal.

A-268
-

Mason, A.K. & Rigney, J.W. Toward a general characterization of electronic

troubleshooting. In R.E. Blanchard & D.H. Harris (Eds.), Man-machine

effectiveness analysis. Papers presented at The Human Factors Symposium,

University of California at Los Angeles, June 15, 1967. (AD-735 718).

I Topics Relevant

i to System Development

land Evaluation Technology

I Topic

I No. ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.'( General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.'* Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

i|.2 Parameter Determinations

4.3 Apparatus for Testing

4.1) Personnel for Testing

1.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.1 Conclusions and

Recommendations

1.3 Several factors underly the desire to

formulate a troubleshooting processor model

accommodating a broad spectrum of possible

specific procedures: (1) such a model

would unify the alternative ways of

characterizing and explaining human

technician troubleshooting behavior; (2) it

would serve as a vehicle for formulating

the cost effectivneess structure associated

with troubleshooting electronic equipment.

3.2 Ideally, a characterization of the

electronic troubleshooting process would be

sensitive to the degree of automation in

the troubleshooting tasks. Thus, measures

of effectiveness for particular equipment

could be generated across the spectrum of

alternative troubleshooting processors

(from the automated to the manual), and

improve the sensitivity of the

"maintainability" jomponent of cost

effectiveness models.

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-269
Matheny, W.G., Patterson. G.W. & Evans, G.I. Human factors In field testing

(LS-FR-71-1). Hurst, TX: Life Sciences. Inc., December 1971. (AD-7U1 216).

Topics Relevant I

I to System Development {Topic

land Evaluation Technology i No. ABSTRACT

3.1

State of the Art Review 1.3

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual 2 1

Process Model (CPU)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment, Definition

2.H General C istraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific 3,2

3.') Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Cooponer'-s

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5-4 Conclusions and

Reconnendalions

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

The aim of this study was the development

and test of methods for evaluating operator

performance during field test of complex

man-machine systems.

Several Navy aircraft systems were examined

in detail/and two systems were selected for

detailed study — the A-7A and the P-3C

anti-submarine warfare systems. The major

portion of the work was carried out using

the P-3C.

Tests were conducted to determine the

feasibility of identifying operator

performance measurement points within the

system. Both an Operational Sequence

Diagram (OSD) and a Mission Time Line (MTL)

were developed for the Tactical

Coordination station within the context of

a standard Evaluation Mission.

The appropriate scales of measurement for

various task types are as follows:

(1) Absolute deviation from standard

percent of time out of design limits

(continuous control tasks).

(2) Percent of incorrect responses

(discrete control tasks).

(3) Percent settings not on design setting

or percent outside design limits

(pointer/symbol positioning tasks).

(U) Percent of decisions agreeing with

Judges* established decisions

(technical decision tasks).

A-270
I Topics Relevant

I to System Development 1 Topic|

land Evaluation Technology | No. j ABSTRACT

4.5 The Mission Time Line (MTL) was used to

test the feasibility of obtaining estimates

of Tactical Coordinates Workload in the

early stages of work with the P-3C. The

feasibility of the method is demonstrated

in this report, but the procedures utilized

are not since the estimate of workload from

the MTL is more properly a part of the

system development phase.

5.1 Recommendations are made with regard to the

need for assigning personnel with authority

and means for accomplishing human factors

requirements.

(1) Detailed listing of operator tasks and

OSD's should be developed and kept

current throughout the development and

tests of the systems. Manual performances and tactical decision points

should be identified early along with

scales of measurement and performance

criteria.

(2) The evaluator must be oriented toward

obtaining reliable measures under test

conditions. The evaluator should

strive towards the most representative

conditions possible while maintaining

good test procedure.

(3) Reliable performance data must be

obtained by which the criterion

requirement for successful performance

can be compared in order to reach an

evaluative Judgement as to the

adequacy of design.

(M) The determination of whether an

operator can successfully accomplish a

sequence of tasks in the time

available in the actual mission uust

be made under conditions as nearly

representative of the operational

minimum as possible.

A-271
I

I Topics Relevant I

I to System Development I Topic I

land Evaluation Technology I No. | ABSTRACT

(5) It is important that the "evaluator

set-up observational and report

procedures to determine the degree to

which deviations from design

procedures occur through use of

earlier learned habits.

•

A-272
Matthews, E.P. A visual target acquisition model. In G.W. Levy (Ed.),

Symposium on applied model of man-machine systems performance (NR69H-591).

Columbus, OH: North American Aviation, November 1968. (AD-697 939).

Topics Relevant

I to System Development

land Evaluation Technology

i?

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.K General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.11 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

I Topic

I No. ABSTRACT

1.3 The paper describes a particular visual

target acquisition model originally

developed for a particular air-to-ground

application, but subsequently extended to

other air-to-ground and 9ir-to-air

applications. The basic building block of

this model (and all other visual

acquisition models) is the capability of

the human eye. Separate modifying branches

are added to this basic block to increase

the model's versatility.

3.1 The model treats acquisition data as a

function of slant range to target. Four

types of sequential acquisition events are

included in the model:

(1) Detection—When the observer becomes

aware of something in the field of

view.

(2) Orientation—Enough of the target's

outline is seen to distinguish between

its longest and shortest dimensions.

(3) Recognition—The outline is seen with

sufficient clarity to establish the

general classification of the target,

e.g., tank, truck, parked aircraft,

etc.

(4) Identification—Sufficient detail to

establish the particular kind of tank,

truck, or whatever.

5.3 Verification of this model using actual

test data have been very good.

A-273
McCalpin, J.P. Development of a methodology for obtaining human performance

reliability data for a machine gun system (Tech Memo. 6-7^). Baltimore, MD:

AAI Corporation, February 1971* (AD-7T7 020).

I Topics Relevant

I to System Development

land Evaluation Technology ABSTRACT

3.

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Proc.ss

2.1 Syst'.-m Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.'t Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recounendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Reaearch Potentials/

Priorities

6.3 Research Planning

2.2 The purpose of this study was the

development of the general procedures

necessary to obtain human performance data

which will satisfy a prior model that

Includes human performance data In models

of Infantry weapon system reliability.

3.1 The following procedures were followed In

order to Implement the study: (a)

development of a taxonomy for weapon-system

tasks performed by the operator; (b)

selection of an Infantry weapon system for

study; (c) analysis of malntanence tasks to

Insure said tasks were consistent with

taxonomlc terms; (d) development of

categories for human performance errors

that relate to hardware failures; (e)

propose a study to collect objective data

on human performance.

4.1 The following methods were used to

Implement the study.

(1) A behavioral taxonomy was developed

which divided behavior into process,

activities, and specific behavior.

Process referred to groups of behavior

associated with perception, judgment,

comnunication and muscular activities;

the activities column classified each

of these processes; the "specific

behavior" column showed the

system-oriented behavior associated

with each category of each process.

A-274
Topics Relevant i

to System Development iTopic I

and Evaluation Technology I Mo. 1 ABSTRACT

(2) The task taxonomy was developed from

the behavior taxonomy. Hardware

elements that comprise most gun

systems were identified; row headings

representing the hardware elements

were matched with column headings

representing the terms in the behavior

taxonomy; the cell entries represent

performance criteria.

(3) Human performance requirements and

human error categories were developed

and correlated such that the cnticality of human performance error was

established in terms of its probable

effect on the weapon.

5.1 It was concluded that human performance

data can be incorporated into the models

currently available that seek to input

human engineering factors into system

reliability models.

I _

A-275

'
McDonnell Douglas Astronautics Company, Lastern Division, Optimized cost/

performance design methodology. Vol. 3-CQncept analysis and model development, Book 1-Concept Analysis (MDC-E0005). September "1969.

I Topics Relevant

I to System Development

jand Evaluation Technology

Topic I

No. ! ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1) General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoonendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 The overall purpose of this study was to

provide a method of using cost as a basic

design parameter in identifying and

defining more economical space transportation systems. This volume reports on Task

6 which sought to determine economically

optimum design and operational

philosophies.

2.1 Using alternatives defined in previous

tasks, systems for resupply of an orbiting

space station were synthesized. These

systems included both ballistic and lifting

entry vehicles with reuse concepts.

4.1 These systems were analyzed using the

Economic Optimization Model to determine

the effect of various system parameters on

total program cost for a fixed program

size.

4.2 Cargo weight per launch and certain

operational modes were optimized.

Sensitivity of program cost to other

operational modes, crew size, cargo

density, number of launches, subsystem

elements/return time/orbit inclination, and

number of landing sites were also

evaluated.

A-276
-

.

McDonnell Douglas Astronautics Company, Eastern Division, Optimized cost/

performance design methodology, Vol. 3-Concept analysis and model developent, Bok 2-Model Formulation (MrC-E0005). September 1969.

Topics Relevant I

i to System Development I Topic

land Evaluation Technology j No. ABSTRACT

•

-

.-

1. State of the Art Review 1• 1

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.'4 Performance

Criteria, Specific

3-5 Measurement Procedures

i). Planning Components

of the Process

1.1 Analytic Methods

4.2 Parameter Determinations

1.3 Apparatus for Testing

H.i* Personnel for Testing

4.5 Test Plans

5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoonendatlons

6. Further Research Areas

6.1 Measurement System

Limitations

6.2 «esearch Potentials/

Priorities

6.3 Research Planning

This report presents a model for relating

various operating and design parameters to

cost and by using search and optimization

techniques, finds the least cost system for

the specified parameters. This model was

developed for two types of spacecraft -

ballistic and those with bodies having low

lift-drag ratios. There are three major

modules to this model. The first is the

main module which is the executive control

logic, the second is the size module which

translates performance and operational

requirements into a vehicle description and

weight statement. The third module is the

cost module which develops the total

program costs from the data supplied by the

other two modules.

A-277
McGrath, J.E., Nordlie, P.G. 4 Vaughan, W.S. A systematic framework for

comparison of system research methods (HRS-TN-59/7-SM). Arlington, VA:

Human Sciences Research, Inc., November 1959. (AÜ-229 923).

I Topics Relevant

1 to System Development iTopic!

and Evaluation Technology ! No. I ABSTRACT

State of the Art Review

of the Process

y. 1 General System

Measurements

1.2 System Taxonomy

Moael (STM)

1.3 Overall Conceptual

P-ocesf Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.U General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.^ Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5-2 Data Analysis

5.3 Findings Interpretation

5-4 Conclusions and

Recomaendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.3 System research has four stages or purposes

with three kinds of research functions.

The four stages are: delineation of system

requirements, delineation of design

consequences of requirements, system

development and Integration, and system

evaluation. The three types or research

functions are: development of models,

collection of research Information, and

synthesis of Information.

Two general types of models were

distinguished: design and research models.

Successive cycles of development of models,

and collection of Information and synthesis

of Information serve to reduce uncertainty

in the formulation of research problems and

to delimit a reduced matrix of potential

design solutions from which an optimal

design is to be selected.

The Information gathering and synthesis

function includes: determination of

relevant variables, determination of the

range of values of these variables, and the

determination of the interaction of the

variables. The information gathering and

synthesis methods differ to the extent to

which they permit, require or prohibit the

occurrence of values of relevant variables;

and, thus, the amount of information which

potentially can be obtained from them. The

efficiency of a given system research

method can be assessed in terms of tne

total amount of accountable information

about the system research problem which its

application can provide for a given level

of research effort.

A-278
McKendry, J.M. & Harris«!, P.C. Assessing human factors requirements in the

test and evöluatlon stage of systems development, Volume II (NDGU-öS). State

College, PA: HRB-Singer Inc.. June 1964. (AD-603 30*0.

I Topics Relevant i

I to System Development ITopiel

iand Evaluation Technology I No. ! ABSTRACT

SUte of the Art Beview

of the Process

1.1 General Systeo

Measureaents

1.2 System Taxonomy

Model (SIM)

1.3 Overall Conceptual

Process Model (CPU)

Contextual Components

of the Process

2.1 Systen Definition

2.2 Mission Definition

2.3 Environnent Definition

2.H General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3-3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

1.2 Parameter Determinations

1.3 Apparatus for Testing

1.1 Personnel for Testing

1.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Rec onnenda 11 ons

Further Research Areas

6.1 Measurement System

Llmllatlons

t.d Research Potentials/

Priorities

6.3 Research Planning

1.1 A project officer's primary responsibility

is to determine whether or not the system

undergoing an operational test can

accomplish its assigned mission under

realistic fleet conditions. An important

aspect of system performance to be

considered is the field of human factors,

that is, human performance in relation to

system performance.

An Operational Evaluation (OPEVAL) is the

test and analysis of a weapon system,

support system, component, or equipment

conducted by the Operational Test and

Evaluation Force, under service operation

conditions insofar as practicable, to

determine the ability of a system,

component, or equipment to meet specified

operational performance requirements and/or

to establish suitability for service use.

When appropriate, an operational evaluation

may be ordered solely for the development

of basic tactical doctrine, training

procedures, and requirements for training

aids and/or countermeasures.

1.3 The OPEVAL has five major phases:

(1) Preparation and initial planning.

(2) Devising and writing the test plan.

(3) Conducting the test.

(4) Evaluation of data from the test.

A-279
Topics Relevant

to System Development I Topic

and Evaluation Technology ! No. ABSTRACT

(5) Derivation of conclusions and

preparation of the final report.

a.M The 'ollowing were identified as OPEVAL

considerations in human factors testing:

(1) While human factors are important,

they are only one of a number of

important factors that need to be

considered by the project officer.

(2) It is unreasonable to ask that the

project officer become a skilled human

factors specialist. He cannot trace

all of the implications of what he

observes, nor should he be expected to

comprehensively examine exactly how

some of the important human effects he

notes can be brought under control.

(3) Most test environments do not have a

large enough sample of men for the

project officer to get a clear

indication of how all of the human

factors problems arise. He cannot

duplicate the fleet personnel problem

on a single ship.

(4) Many human factors tests are extremely

time-consuming and expensive to run

because all of the situational

conditions must be repeated exactly,

with different men working within the

system.

(5) Practical aborts can be expected which

will complicate and sometimes negate

the project officer's attempts to

complete a test of anything, including

human factors.

(6) The most efficient use of the project

officer's time would be to concentrate

on the most important human factors

effects and to gather data on these

effects. Additional associated

A-280
I Topics Relevant

I to System Development iTopic I

iand Evaluation Technology i No. I ABSTRACT

problems must, by necessity, remain

the province of the human factors

specialist who can assemble large

amounts of dat?, carefully study the

personnel situation in general, and

draw needed conclusions.

U.3 The instruments used to record human

performance data included an operations

events recorder (the model in this study

was a 20 pen Esterline Angus), recycling

timers, tape recorders, counters (both

manual and automatic), and function

recorders.

I

A-281
McRuer, D., Jex, H.R., Clement, W.F. & Graham, D. A systems analysis theory

for displays In manual control (STI Tech Rep. 163-1). Hawthorne, CA: Systems

Technology, Inc., June 1968. (AD-675 983).

Topics Relevant

1 to System Development

land Evaluation Technology

[Topic

! No. ABSTRACT

State of the Art Review

of the Process

1 1 General System

Measurements

1 2 System Taxonomy

Model (STM)

1 3 Overall Conceptual

Process Model (CPM)

Contextual Coaponents

of the Process

2 1 System Definition

2 2 Mission Definition

2 3 Environaent Definition

2 K G«neral Constraints

2 5 Perforaance

Requireaents, ultimate

2 6 Performance

Criteria, Ultimate

Analytic Conponents

0 ' the Process

3 1 Practical Measurable

Attributes

3 1 Practical Attribute

Measures

3 3 Performance

Requireaents, Specific

3 k Performance

Criteria, Specific

3 5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Coaponents

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Reeomtendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.3 This report presents the basic structure of

a comprehensive theory for the development

of control displays for pilots of manually

controlled vehicles. This theoretical

framework provides a paradigm for control

display development, a rational basis for

experimental programs, a theoretical

foundation for analyzing the comparative

merits or problems of new operational

display systems, and for the generation of

new display concepts.

The theory combines several manual control

developments in its structure:

(1) The vehicle dynamics, environmental

disturbances, command structure, and

mission criteria derived in terms of

meaningful servo analysis parameters.

(2) The "best" or "alternative best"

feedbacks for the pilot are derived

using the "multiloop feedback

selection hypothesis," which includes

the human operator's describing

functions, remnant, and subjective

preferences.

(3) Quantitative evaluation of the system

performance measures, information

bandwidths, and stability margins made

by systems analysis techniques.

(4) The required display resolution,

scaling, scanning pattern and rates,

and workload margins are derived,

based on pilot monitoring and scanning

A-282
I Topics Relevant i I

i to System Development !Topic I

iand Evaluation Technology j No. j ABSTRACT

models in terms of meaningingful

sampling analysis parameters.

(5) Preferential display arrangements are

predicted from scanning pattern and

workload measures.

(6) Progression and regression of the

level of pilot behavior (e.g., during

training, transfer, stress, or

equipment failure) are treated by the

successive organization of perception

theory of manual control skill

development.

The theory can be utilized for the

following:

(1) Exposure of potential problem areas

and directions for improvement at an

early enough stage to minimize detail

design risks and costs.

(2) Prediction of the best display

scaling, filtering, and equalization

parameters (e.g., "quickening").

(3) Analysis of display/pilot/vehicle

system instability under instrument

flight conditions (e.g., flight path

oscillations under Instrument Landing

System (XLS) guidance).

(4) Selection of optimum feedbacks and

their gains for integrated displays.

(5) Specification of display instrument

dynamic range, bandwidth, and

tolerable dynamic lags.

(6) Estimation of functional limitations

on existing instruments as applied to

new missions, vehicles, and tasks.

A-283
I Topics Relevant

! to System Development

!«nd Evaluation Technology ABSTRACT

(7) Optimization of the location of

related instruments for specific

mission phases.

(8) Rapid post-mortem investigatiori of

anomalies in measured display system

performance.

(9) Analytical evolution of relevant

parameters and their interactions, as

a guide to the experimental design of

display simulations and flight tests.

(10) Correlation and unification of the

results of numerous ad hoc display

tests performed in the past.

(11) Exposure and prediction of solutions

to new problem areas in integrated

displays, terrain-following displays,

etc.

(12) Evolution of display simulator

functional requirements (e.g.,

instrument servo lags, cathode ray

tube capabilities).

(13) Interpretation of experimental

findings in an analytical manner to

permit their extrapolation to actual

flight and future problems.

(14) Establishment of a rational basis for

cardinal elements on contact analog

displays.

(15) Evolution of displays to resist

disorientation and to achieve optimum

head-up display arrangements.

(16) Synthesis of improved blind-landing

and terrain-following displays.

(17) Specification of the best training and

utilization procedures to enhance

learning and skill transfer.

An example is presented of the application

of the theory to the ma'-ially controlled

ILS approach of a large Jet aircraft.

A-284
<•

Heister, D. A pragmatic approach to the prediction of operational

performance. In R.E. Bianchard & D.H. Harris (Eds.), Man-machine

effectiveness analysis. Papers presented at The Human Factors Symposium,

University of California at Los Angeles, June 15, 1967. (AD-735 718).

I Topics Relevant

i to System Development I Topic I

land Evaluation Technology ! No. I ABSTRACT

State of the Art Review

of the Process

1.1 General Systen

Heasureaents

1.2 System Taxonotny

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Coaponents

of the Process

2.1 Systea Definition

2.? Misslüi Definition

2.3 Environaent Definition

2.<i General Constraints

2.5 Performance

Dequlreaents, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Cooponents

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recooaenda11 ons

Further Research Areas

6.1 Measureaent System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

i:3

2.1

3.1

The pragmatic approach to predicting

operational performance has the following

characteristics: (Da conscious attempt

to avoid mathematical models and

theoretical processes and, instead, to

extrapolate predictive indices directly

from empirical data; (2) an emphasis on

data, not theory; (3) a focus on certain

parameters that are assumed to be important

to operator performance. Logically and

heuristically, it can be assumed that a

restricted number of parameters account for

the greatest part of the operator's

performance; if human performance were

affected equally by all possible factors,

it would be infinitely variable and

unpredictable.

The important parameters tend to be task

oriented or at least related to operational

system requirements. The parameters

selected for predicting operational

performance obviously define what data are

needed.

In predicting performance, it is also

necessary to account for the fact that more

than one task may be performed concurrently

by the same operator. Each of two

concurrent tasks has its own important

parameters, for predicting the task's

individual performance; the concurrency of

the two tasks is another important

parameter for predicting the total

performance.

The parameters considered to be important.

In general, are the following:

A-285
I Topics Relevant

! to System Development I Topic

land Evaluation Technology j No. ABSTRACT

3.3

(1) The number, organization^ and

utilization sequence of controls and

displays involved in the task.

(2) The display exposure time.

(3) Display visibility.

(4) The nature of the stimulus displayed

(i.e., structured or unstructured).

(5) The number of visual stimuli.

(6) The operator function (i.e., the type

of response required of the operator:

e.g., discrete control response,

continuous control response,

simple monitoring, detection,

discrimination, tracking, stimulus

identification, information

extraction, decision-making, etc.)

(7) Stimulus movement.

(8) The characteristics of the

control-display coordination.

(9) The amount of information the operator

must handle.

(10) The feedback (type, amount) provided

to the operator following his actions.

Other important parameters that relate to

task requirements include:

(1) The performance accuracy required of

the operator.

(2) The operator loading (pacing, or time

constraints).

A-286
! Topics Relevant

i to System Development

land Evaluation Technology ABSTRACT

1.2 Theoretically, data on these parameters can

be obtained under either experimental

conditions or the actual operational

environment. The author doubts that

experimental work will be able to supply

the necessary infor-aation, given the poor

record of human factors research in

supplying relevant data stores, and given

the focus of most research on operationally

meaningless, artificial, unrealistic tasks.

The difficulty of data collection in the

operational environment is one of setting

up conditions which isolate the parameters

of interest: the parameters usually exist

only in interaction under operational

conditions. The solution is to identify

operational conditions which display

combinations of parameters of interest. By

locating and measuring different parameter

combinations, comparing results, and

ascribing differences to variations in the

parameters between the two combinations,

the individual parameter effects can be

isolated.

A-287
Meister, D. Assumptions underlying test and evaluation strategies. San

Diego, CA: Navy Personnel Research and Development Center. (Speech,

undated).

Topics Relevant i |

1 to System Development iTopic]

land Evaluation Technology ! No. i ABSTRACT

State of the Art Review

of the Process

1.1 General Systec

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environaent Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

4.2 Parameter Determinations

<i.3 Apparatus for Testing

1.4 Personnel for Testing

1.5 Test Plans

Application Coaponents

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Re c onnen da ti ons

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 Five assumptions are discussed at length.

They are summarized below:

(1) Human performance can be measured in a

system work context.

(2) Since human performance occurs as part

of system operations, it must be

measured in a system-related way.

(3) Maximum simulation fidelity produces

more valid tests and evaluations than

lesser amounts of simulation fidelity

to the operational system and environment in which the system being tested

will perform.

(4) Test and Evaluation (TE) is important

only during a specific period of

system development.

(5) It is possible to conduct TE without

specifying in advance human performance criterion values. The primary

question asked in any TE situation is:

Is personnel performance satisfying

overall system requirements?

It should be noted that assumptions 3 to 5

are either elaborated on and/or refuted by

the author.

2.1 Test and evaluation, as the author

described it, is termed OST (Operational

System Testing) in which the new system is

a prototype placed in a functional setting

that is or resembles the environment it

will ultimately be exposed to.

A-288
Meister, D. Assumptions underlying the human reliability model. In G. W.

Levy (Ed.), Symposium on applied codel of man-machine systems performance

(MR69t-591). Columbus, OH: North American Aviation, November 1968. (AD-697

939).

Topics Relevant |

I to System Development !Topic

land Evaluation Technology j No. ABSTRACT

-

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.'I Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Coaponenta

of the Process

5.1 Test Execution

5.2 3ata Analysis

5.3 Findings Interpretation

5*4 Conclusions and

Recoomendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentiala/

Priorities

6.3 Research Planning

1.1 The author points out that any measurements

of system reliability or system

effectiveness which do not include indices

of human performance must necessarily

produce an erroneous estimate of that

system's reliability or effectiveness. The

goal of the human reliability methodology

is to quantify human performance in a

man-machine context.

2.1* The author notes that these procedures

derive by analogy from an equipment

reliability orientation; the human

component is viewed as a "black box", and

human performance is conceptualized in

terms of a single probabilistic measure,

i.e., task completion or task failure.

This concept also implicitly assumes that

each stimulus condition presented to the

subject increases the potential for error

and reduces the probability of successful

task completion.

Another assumption of the model is that the

individual probabilities of performance for

the behavioral units summate mutiplicatlvely; however, when combining task

probabilities to derive higher function

probabilities, it is unreasonable to assume

independence of those tasks implicit in

multiplicative summation.

The model also makes the overly siirplistic

assumption that error is equivalent to

failure to complete the task; however, it

is perfectly obvious that a task might be

completed successfully even if several

A-289
Topics Relevant

to System Development !Topic

and Evaluation Technology ! No. ABSTRACT

errors are made during its pecformance

because the human operator often can

correct his errors.

The model also has the inadequacy of

dealing only with individual operator

behavior; it forms predictions of group

performance through the straight

miltiplicative process, and this is

manifestly inappropriate.

The model also assumes no change in the

behavioral unit probabilities as a function

of repeated performance, and takes no

account of individual variations in skill.

3.5 The major steps in the application of the

human reliability model are as follows:

(1) Analyze system operations into

discrete units of behavior to which

predictive data can be applied.

(2) Determine the parameters affecting

each task behavioral unit.

(3) Assign probability values based on

historical data to each behavioral

unit.

(i») Combine the individual unit probabilities together to form task

probabilities, function probabilities,

etc.

5.4 In the summary, it is noted that many of

the assumptions in this model are

consciously incorrect, and therefore it is

questioned whether this knowingly

inadequate model still has value. The

answer is yes, and that it has two types of

values: First, it stimulates conceptual

activity by the investigators and raises

scientifically important questions which

might not have been raised without the

model. Second, it has heuristic value, in

A-290
I Topics Relevant

I to System Development

!and Evaluation Technology

Topic

No. ABSTRACT

that is reduces the amount of uncertainty

that we have.

6.1 The paper discusses the human reliability

model primarily as a means of illustrating

certain characteristics of behavioral

models in general and certain characteristics of model-makers themselves. The

author's personal view is that a model is

effective to the extent that it helps

either to gather data and/or to explain

those data. As a corollary, he states that

any behavioral model which is not concerned

with real-world data (as opposed to laboratory data) is not useful. However, he

observes that behavioral models characteristically have employed laboratory data and

ignored or been unable to handle natural

event data. Models thus are painfully data

limited - - their characteristics and their

basic assumptions often are determined by

the limitations of the data input to them.

These data limitations often result from

the way in which we scientists have been

taught to perform our studies and by the

biases we knowingly or unknowingly insert

into these studies.

The human reliability model is cited as a

case in point. The author asserts that

this model's assumptions derive from the

unsystematic manner in which the model's

input data were secured, and he points out

that at least in part, these assumptions

demonstrably are not in accord with

empirical reality.

A-291

i -. • - L^J - - - .^^_^_-

. -^—^^.^^.^.^^^^ riiaA . rita 1 ......
Meister, D. A systematic approach to human factors measurement, San Diego, CA:

Navy Personnel Research and Development Center, October 1978.

I Topics Relevant I I

I to System Development iTopicI

land Evaluation Technology i No. I ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measureaents

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

3.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3-5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recomendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorit^s

6.3 Research Planning

1.1 The concept of "system" is the foundation

of human factors studies which seek to

measure factors that affect personnel

performance in manned systems. In

measuring any system, one must keep in mind

certain universal properties of all

systems: they are organized

hierarchically, have a purpose, are

interdependent, and contain their own

standards for measurement, evaluation, and

feedback since a purposive system

establishes, ipso facto, its own standards.

1.3 Looking at operational manned systems in

general, one can list several aspects of

the measurement process that are common to

any analysis:

(1) Assessment of the impact of system

parameters on personnel performance.

(2) Assessment of the impact of human

factors on system outputs.

(3) Specification of the "mission

scenario" of the system (initial

stimulus to end-point).

(4) Replication (validation) of the

research study under identical or

simulated conditions.

A-292

^w ^ - . - iä AM M
I Topics Relevant

i to System Development 1 Topic I

!and Evaluation Technology i No. I ABSTRACT

All system-relevant factors must be

included in any measurement situation to

ensure that those variables chosen

represent the operational system. There

are two factors involved: (1) all

variables that would be expected to affect

system output in the operational environment must be included; (2) all interactions

in the system representation must also be

included.

2.1 The type of measurement system focused on

here was called PSM or Personnel Subsystem

Measurement: measurement of personnel

performing a task or job within the actual

work, or system, environment.

•

2.2 PSM's purpose was to determine the

feasibility of a research approach; choose

an optimal alternative; determine

performance capability; resolve personnel

subsystem dysfunctions; contribute needed

research.

i

2.5 A comprehensive use of PSM has the

potential to answer the following questions

pertaining to four separate stages: system

development, training program development,

system operation, and system maintenance.

(1) System Development:

a. Do personnel have the capability

to perform certain tasks at a

specified level?

b. Is there an optimally efficient

interface between system design

and personnel?

•

c. Regarding personnel performance,

two or more system configurations

is most effective?

A-293

. ■■ .-.• . , . . . . - -■■-- -- - . -,_- .. - - /•:.vi>>^ "•-•"-'
I Topics Relevant

1 to System Development {TopicI

land Evaluation Technology ! No. ! ABSTRACT

d. Does the prototype or

developmental system fulfill

system requirements from the

personnel pers active?

e. How does one develop a system by

using human factors inputs? What

relationship exists between

system characteristics and

operator performance?

(2) Training;

9

a. Has training been adequate?

b. Is there transfer of performance

from the simulated learning

environment to the real,

operational environment?

c. Does training in one mode compare

with training using another mode?

d. How closely must the training

environment match the operational

one?

(3) System Operations:

a. Do system personnel perform

according to requirements?

b. Is the system ready to operate as

required?

c. How can system verification

problems be solved?

d. How do new and old system

configurations compare?

(4) System Maintenance:

a. How do technicians perform and

resolve equipment malfunctions,

i.e., "diagnostic maintenance"?

A-294

-^.- V-v^--.-'^-'.--^-'.-^ :.T:. v.--^-. -c.-. .r; .,• ... .....r.V^^^^.^r.-.v^/....-.
I Topics Relevant I

to System Development I Topic!

land Evaluation Technology ! No. I ABSTRACT

b. How efficient is that

maintenance?

3.1 In using measures to evaluate syste Ti

performance, PSM's approach is to measure

the individual, team, subsystem, and system

levels where appropriate. PSM emphasizes

normative data, which are primarily

descriptive.

Two types of normative data were used in

PSM evaluation:

(1) Data describing systems and their

interelationships.

•

i

(2) Data describing personnel task

performance.

M.I Data are analyzed by correlational analysis

in PSM under operational conditions; in a

laboratory experiment, testing the

significance of differences is the most

common technique.

1.2 The parameters of a given PSM evaluation

involved system and personnel

specification. With regard to personnel

parameters, the following must be defined:

characteristics of equipment, job,

individual aptitude, skill, experience, and

motivation.

A-295

• :•:•:•:•:•:-:-•::•

. ^i. i

.^-v.
Heister, D. Human factors In operational system testing; A manual of

procedures (NPRDC SR 78-8). San Diego, CA: Navy Personnel Research and

Development Center, April 1978.

I Topics Relevant I

I to System Development I Topic I

iand Evaluation Technology i Wo. | ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3-3 Performance

Requirements, Specific

3.t Performance

Criteria, Specific

3-5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

4.2 Parameter Determinations

1.3 Apparatus for Testing

4.1 Personnel for Testing

4.5 Test Plans

1.3 This handbook has been developed to enable

Marine Corps test personnel to evaluate a

new system for its human factors adequacy.

The term "human factors" refers to the

entire complex of elements that affect

personnel performance.

This handbook provides information on the

Personnel Performance Test Plan, including

its purpose, a step-by-step description of

its various sections and requirements for

completing those sections, a model for

illustrating the material to be included in

such a plan and a procedure for developing

quantitative personnel performance

criteria.

The next section covers the selection and

development of measures and measurement

methods followed by a human engineering

checklist of procedures. The handbook also

describe "Self Report Rating Scales" and

Interview questions. Test procedures are

covered and there is also an introduction

to statistical methodology. The last

sections contain a "Personnel Performance

Test Planner's Checklist" and a Personnel

Performance Test Report. The final section

gives references which might be useful to

the user of this handbook.

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-296

A at *_ ^ V p - * « ^ - » ^ - - ■■

» „ .. ^» .A L.> .- »i - •-•»-*• •* . v» r

-"""• %.'•• •/-.- •.

- - ---
Meister, D. & Rabideau, G.F. Human factors evaluation in system development,

New York: John Wiley & Sons, 1965.

I Topics Relevant

I to System Development

land Evaluation Technology ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Envlronaent Definition

2.<t General Constraints

2.5 Performance

Reciulrements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

U.2 Parameter Determinations

1 3 Apparatus for Testing

1 »* Personnel for Testing

« 5 Test Plans

Application Components

of the Process

5 1 Test Execution

5 2 Data Analysis

5 3 Findings Interpretation

5 1 Conclusions and

Recooaendations

Further Research Areas

6 1 Measurement System

Limitations

6 2 Research Potentials/

Priorities

6 3 Research Planning

1.1 The practice of human factors in the

relationship between the capabilities and

limitations of men and the characteristics

of machines involves the following:

(1) Performance of a functional analysis

of system requirements.

(2) Development of personnel selection and

training criteria.

(3) Design and evaluation of manually

operated control and crew personal

equipment.

(U) Study of the environmental factors

affecting human performance.

(5) Participation in system performance

tests.

(6) Surveillance of production facilities

and applied research.

Whenever there are design alternatives,

hunan factors evaluation is essential to

eliminate the systems' susceptibility to

human error.

1.2 It is difficult to compare the man-machine

behavior of one system with that of another

because the terms of that behavior are not

yet operationally definable.

> - A

A-297

' V
i Topics Relevant I I

I to System Development |Topic I

!and Evaluation Technology j No. ABSTRACT

A crucial characteristic of the system is

its purposiveness (goal-directed behavior).

Goals can be divided into two groups:

primary (mission-oriented) and secondary

(supporting). Primary goals seek -o

accomplish the system mission and direct

the performance of all mission-related

system activities. Secondary goals seek to

maintain the integrity of the system until

the mission has been accomplished.

2.2 The mission described the man-machine

activities performed to accomplish the

primary system goals. Unless it is framed

in terms of mission goals, system behavior

becomes extremely difficult to explain or

understand because purpose is the single

factor which unifies a great variety of

disparate system behavior.

V

.

A-298

! l" - I . ^ ^ . ^-i-^- . - - . - . .-., .-»..\--...- - -,^imi^«a
Melching, W.H. A concept of the role of nan In automated systems

(Professional Paper 1M-68). Presented at the Southwestern Psychological

Association Annual Heating, New Orleans, LA, April 1968 (AD-671 128).

I Topics Relevant I

I to System Development {Topic

land Evaluation Technology ! No. ABSTRACT

State of the Art Review

of the Process

1.1 General Systen

Neasurenents

I.I System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process llodel (CPU)

Contextual Conponents

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environaent Definition

2.<4 General Constraints

2.5 Performance

Requirements, ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3-3 Performance

Requirements, Specific

3.1* Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

H.2 Parameter Determinations

4.3 Apparatus for Testing

4.1 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recooaenda11 ons

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 A problem to systems designers is the

allocation of functions between man and

machine. This paper reports an attempt to

isolate and identify factors pertinent to

making allocation decisions. In an

analysis of the functions and missions of

several automated systems, the following

factors were shown to be highly relevant to

allocation decisions:

(1) State of the art.

(2) Cost of automation.

(3) Space and weight constraints.

(4) Lead time.

(5) Difficulty to program.

(6) Han's role in automated systems.

The designer of an automated system needs a

clear-cut conception of the general role of

man in such systems. In effect, the

designer needs a conception of what man's

role should be before he can decide what it

will be.

A-299

*M '.•'.- .- . '.- .-

-vy.
Meyer, R.P., Laveson, J.I. & Pape, G.L. Development and application of a task

taxonomy for tactical flying (AFHRL-TP-78-12, Vol. 1). St. Louis, MO: Design

Plus, September 1978 (AD-061 387).

Topics Relevant I

to System Development ITopic

and Evaluation Technology i Wo. ABSTRACT

1 •.

Sttte of the Art Review

of the Process

1.1 General System

Measureaents

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPU)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3-3 Performance

Requirements, Specific

3.4 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recomaendations

2.1 The primary purpose of this document is the

development of a "surface task analysis"

based on a sample of sixteen tactical

maneuvers found in tactical flying (seven

air-to-ground maneuvers and nine

air-to-air).

2.3 The authors chose the F-ME aircraft as a

representative tactical craft, and based

their analyses on pilot performance in the

F-HE.

2.5 Ultimately, the use of a surface task

analysis will allow a complete description

of a flying task or maneuver, step-by-step.

2.6 By focusing on the pilot's aircraft control

behavior, this study offers a data system

from which improved flying training

concepts and methods can be derived. Using

this system (surface task analysis),

training developers can determine and

substantiate the content of training

programs. Training programs can also be

identified and alternative solutions

formulated using this system.

3.1 The surface analysis technique, by definition, measures the following: (a) cues, or

inputs which the pilot '•eceived from his

flying environment to perform a certain

task; (b) mental action, which processed

the cues, and (c) motor actions, or outputs

in the form of movements of the aircraft

flight controls.

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-300

iÄiiii^ ^iiMiäüii^är^1"

. . .•. •
! Topics Relevant !

I to System Development iTopic

land Evaluation Technology ! Wo. ABSTRACT

3.2 The cues were divided into four categories:

visual, aural, control, and motion cues.

Visual cues included outside or

environment-generated cues, such as the

horizon or cloud formation; inside, or

aircraft-generated cues, includes

Information obtained from flight

instrumentation or radar. Aural cues

include such items as communication,

weapons tones, slipstream, etc. Control

cues involved aircraft flight controls,

which were either (a) dynamic tactual,

e.g., aileron, stabilizer, rudder,

throttle, or (b) discrete tactual, e.g.,

toggle and rotary switches, etc. Finally,

motion cues consisted of stimuli which were

sensed by body receptors such as or -

G-force, vibration, etc.

M

V.

3.3 Rules and procedures were developed for the

specific application of the cues categories

in performing a surface task analysis.

Generally speaking, each surface analysis

must identify the following: (a) the

aircraft type involved; (b) the maneuver(s)

involved and weapons delivery; (c) whether

the maneuver environment is a range- or

tactically-oriented one; (d) the flight

path(s) of the aircraft; (e) the starting

situation of the aircraft and the specified

task goal. A diagram of the maneuver is

then prepared, and for each element of the

flight sequence, a chart is filled in with

Information on cues, mental action, and

motor action.

4.1 The data base for the entire taxonomy

classification consisted of interviews with

pilots regarding sixteen representative

tasks, both air-to-ground and air-to-air.

There are, therefore, sixteen separate

surface task analyses. Detailed analyses,

or classifications of flight maneuvers were

given to complete Volume I of the study.

Volume II details the rationale and methods

used to formulate a taxonomic structure for

tactical flying tasks.

A-301

- - - - I
Meyer, R.P., Laveson, J.I. & Pape, G.L. Development and application of a

task taxonomy for tactical flying (AFHRL-TR-78-42. Vol 2). St. Louis, MO:

Des'.gn Plus, September 1978. (AD-061 388).

Topics Relevant

I to System Development I Topic

land Evaluation Technology ! No. ABSTRACT

1. State of the Art Review

of the Process

1.1 General System

Heasureaents

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPU)

2. Contextual Components

of the Process

2.1 System Definition

2.i Mission Definition

2.3 Envlronaent Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

i(. Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoonendatlons

6. Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Prioritlas

6.3 Research Planning

1.3 A taxonomy of tactical flying skills was

developed as a user-oriented skill-task

analysis system for practical application

in solving Tactical Air Command (TAC)

continuation training problems and for a

behavioral data base for skill maintenance

and reacquisition training research and

development. Sixteen representative

tactical air-to-air and air-to-surface

maneuvers were analyzed and classified

within the system with provision for later

expansion. A classification system was

developed to accomraodate the complexities

of tactical flying. A data system was

organized with sufficient flexibility to

objectively address many areas of tactical

flying. The taxonomy system also included

methodology for addressing on-going

training problems and requirements.

A-302

•-•- -- - - -

- - •

•v- ^■■. ^ V
/-

Meyer, R.P., Laveson, J.I. & Pape, G.L. Development and application of a task

taxonomy for tactical flying (AFHRL-TR-78-U2, Vol. 3). St. Louis, MO: Design

Plus, September 1978. TAÜ^AOöl 478).

»v

Topics Relevant

I to System Development [TopicI

and Evaluation Technology i Wo. I ABSTRACT

State of the Art Review 1.2

of the Process

1.1 General System

Heasureaents

I.I System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.<l General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

A taxonomy of tactical flying skills was

developed as a user oriented skill-task

analyses system for practical application

in solving Tactical Air Command (TAC)

continuation training problems and for a

behavioral data base for skill maintenance

and reacquisition training research and

development. Sixteen representative

tactical air-to-air and air-to-surface

maneuvers were analyzed and classified

within the system with provisions for later

expansion.

3- Analytic Components

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3-3 Performance

Requirements, Specific

3.'t Performance

Criteria, Specific

3.5 Measurement Procedures

H. Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recomendations

6. Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-303

- * "^ ^ ^ ^ ^ ^ ^-~A^-.-^-~-^-^- ^ . - -

- - *

- - • - -^ • - • - -> -• - -- - - -
a

Miles, J.L. Some recent trends in human factors testing (Tech. Memo. 22-76),

Abeerdeen Proving Ground, MD: U.S. Army Human Engineering Laboratory. June

Wb. (AD-B012 110).

I Topics Relevant I

I to System Development I Topic

land Evaluation Technology | Wo. ABSTRACT

{

SUte of the Art Review

of the Procera

1.1 General System

Measurements

1.2 System Taxonooy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.14 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.'t Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

«.1 Analytic Methods

H.2 Parameter Determinations

4.3 Apparatus for Testing

4.>i Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Reconxnendatlons

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 The cost-effectiveness of human factors

(HF) testing in any material development

project depends in large part, on the uses

to which the test data will be put.

Whereas the human engineer contributes data

in terms of human performance or effects of

specific variables, the systems analyst

wants the question "So what" explained in

numbers. AMSAA (Army Material Systems

Analysis Activity) has stated that the

"soldier is part of the system" and human

factors date, should be analyzed "not as a

separate additional activity, but as an

integral part of our evaluation of each

system". The HF data therefore, need to

fit into five categories of interest to the

systems analyst: effectiveness, reliability, availability, maintainability, and

training.

3.1 The normal performance measures used to

determine figures for the above five

categories were time and error.

A-304
Miller, J.G. Living systems. New York: McGraw-Hill, 1978.

I Topics Relevant

1 to System Development

i and Evaluation Technology

i i

ITopid

! No. ; ABSTRACT

State of the Art Review 1 .2

of the Process

1.1 General Tystem

Measurements

1.2 System Taxonomy

Moäel (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.H General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria. Ultimate

The author examines all biological and

social systems and divides them into seven

hierarchical levels: cells; organs

(composed of cells); organisms (independent

life forms); groups (families, committees,

etc.); organizations (communities,

universities); societies or nations; and

supranational systems. Each of these

levels carries on both matter-energy and

information-processing functions for a

total of 19 critical subsystems.

The author defines 13 distinct concepts

which must be understood in analyzing any

living system at any level:

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

U.2 Parameter Determinations

4.3 Apparatus for Testing

U.k Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recomnendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

(1) Space and Time: all concrete systems,

living and non-living, exist in

physical space; this latter space acts

in various ways to partially constrain

and determine the system under

consideration. For example, residents

of a neighborhood who are physically

closer in space to each other will

interact with a greater rate of

frequency; alternatively, the number

of nucleotide bases - which are

configurations in space - that a DNA

molecule contains will determine the

amount of information (bits) that it

can contain.

Examples of conceptual space which can

be found in all living and nonliving

systems include: (1) Lewin's "life

space," which is the perceived environment that immediately surrounds an

Individual; and (2) Warner's "socialspace'1 which separates, say, an

upper-lower class person from a

lower-middle class person.

A-305

- -^ >:- -

,••.-.•.

—^ - - • - -
I Topics Relevant I I

1 to System Development I Topic!

land Evaluation Technology ! Wo. I ABSTRACT

The concept of time has been

acceptably quantified through everyday

use of seconds, minutes, etc., to

measure durations speeds,

accelerations, etc. However, the

general theory of relativity makes it

clear that for large systems, e.g.,

astronomical systems, time cannot be

measured on any absolute scale.

(2) Matter and Energy: Hatter, defined as

anything which has mass and occupies

physical space may have several forms

of energy, defined in physics as the

ability to do work: (a) Kinetic

energy, when mass is moving and

creates a force on other matter; (b)

potential energy, referring to a mass'

position in a gravitational field, and

(c) rest mass energy, which results

from the conversion of matter into

energy. Matter and energy are

equivalent (E=MC ); all living systems

require specific types of

matter-energy.

(3) Information: here defined as " . . .

the degrees of freedom that exist in a

given situation to choose among

signals, symbols, messages, or

patterns to be transmitted." The term

"marker" is used to denote an

observable unit or change of

matter-energy whose patterns indicate

informational symbols (e.g., cuneiform

writing, punched cards, pulses on a

telegraph wire); due to advances in

technology, the matter-energy costs

of storing and transmitting these

markers has decreased, while

information-processing efficiency has

increased by decreasing the mass of

these markers: cuneiform tables

carried 10" bits of information per

gram; paper with typed messages 10

bits per gram; electronic magnetic

tape storage carries 10 bits per

gram. The point to be made for

A-306

- ,

- -

- - . - " - .

. -',--

i i -^ ^ •----- -• - -

^L_^_

- • - - -
\ Topics Relevant

I to System Development I Topic!

land Evaluation Technology I No. I ABSTRACT

systems analysis is that there are

significant matter-energy constraints

upon information-processing of all

living systems as a result of the

composition of their "maricers."

(U) System: the most general definition

is '' . . .a set of interacting units

with relationships among them."

Systems may be open (boundaries may be

crossed by information or

matter-energy transmissions), closed,

nonliving, living, totipotential (can

carry out all critical subsystem

processes necessary for life),

partipotential, fully functioning, or

partially functioning. Each type of

system responds to varying types of

measurement.

(5) Structure: "... the arrangement of

its subsystems and components in

three-dimensional space at a given

moment in time."

(6) Process: changes in the matter-energy

or information in a system is called a

process. A process is reversible if

it remains the same regardless of the

time dimension. Process includes the

function of a system and its history.

(7) Type: refers to similar

characteristics among a number of

living systems.

(8) Level: level refers to the particular

position of a system within a

hierarchy, e.g., groups are composed

of organisms, organizations are

composed of groups, etc.

(9) Echelon: this refers to the various

levels of components in a system or

subsystem. Certain decisions are made

by one component of a subsystem and

others by another, the two components

being either at the sane or different

A-307

. ä£bi --.--v.

^' «fcafa
[ Topics Relevant !

1 to System Development I Topic!

land Evaluation Technology ! No. ! ABSTRACT

'

[

echelons of the subsystem^

(10) Suprasystem: this refers to the next

highe • system in which the system in

question is a subcomponent or part.

The itnnediate environment of a system

should be distinguished from the

suprasystem: the latter will affect

both the system's environment and the

system itself.

(11) Subsystem and component: subsystems

are distinct structures which carry

out distinct processes within a

system. The distinct, structural

units themselves are called the

components. Within the level of

organizations, the concept of role

would be a component; however, not

only will the role affect the

organization, but the nature of the

individual who fills that role will

have an impact as well.

(12) Transmissions in Concrete Systems:

Transmissions may be of matter,

energy, or information, and can be

analytically distinguished in terms of

(a) inputs across system boundaries,

(b) internal processes within a

system, and (c) system outputs. The

"template" of a system is its original

genetic or information input that

programs the system's structure and

process.

(13) Steady State: all living systems

maintain steady states (homeostasis)

among their variables, keeping a

balance not only with intra-system

variables, but with their environments

and suprasystems as well. The

"LeChaterlier principle" is

particularly apropos to the steady

state concept, and states that a

stable system un^er stress will move

in that direction which minimizes the

effect of stress (a compensatory force

A-308

v

-

v

v

■■.... .. .^.::. ...-,
I Topics Relevant

I to System Development [TopicI

and Evaluation Technology i No. I ABSTRACT

will be exerted to oppose the stress,

usually accompanied by changes in

related system variables that are not

directly affectfd by the stress).

Cybernetics, the study of methods 01

feedoack control, is an important part

of steady state theory.

A-309

- -

. • -".-

«- - •. »-

Mill *

„i*-^-;
Miller, J.R. Assessing alternative tranaportatlon systems (RM-5865-D0T).

Santa Monica, CA: The RAND Corporation, April 1969. (PB-185 167).

1

I Topics Relevant

I to System Development

land Evaluation Technology

i

jTopic

I No. ABSTRACT .

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.'I Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

ii.2 Parameter Determinations

1.3 Apparatus for Testing

H.k Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 This paper addresses the problems of

assessing worth among alternative

transportation systems. It assumes that a

decision context has been specified and

that a fixed set of discrete alternatives

has been produced.

2.5 The performance requirements are that the

criteria are complete, mutually exclusive,

of major significance and free of worth

interdependence.

2.6 The first step of the assessment procedure

is to define explicitly what is desired in

the way of performance for each

alternative.

3.3 Having established a list of overall

objectives, the second step is to generate

a hierarchial structure of successively

more specific performance criteria. This

involves breaking down or subdividing

higher level criteria into one or more

lower-level criteria alleged to be included

within the meaning thereof.

3.4 The third step is to select a physical

performance measure for each lowest-level

criterion. Performance measures describe

what an alternative can deliver, while

performance criteria state what the

decision-maker desires. The purpose of

selecting performance measures is to

establish concrete connections between

desires and deliverable performance from

real alternatives.

3.5 In the fourth step specific worth

relationships are mapped out between each

.»

A-310

- -.-- . : •:.-.:•.:.-
I Topics Relevant I

I to System Development !Topic

land Evaluation Technology j No. ABSTRACT

. -

lowest-level criterion and its related

performance measure. This is imoleraented

by defining scoring functions which assign

a unique numerical worth score to every

possible value of a performance measure.

The worth scores are then combined to

arrive at an overall index of each

alternative's worth. This is accomplished

by defining a weighting function. An

additive function with constant trade-off

weights can be adopted for this purpose.

The sixth step is to validate both the

scoring functions and the weighting

functions against whatever alternatives

have been produced. This means computing

an overall worth index for each alternative

and Judging the results for reasonableness.

Results generated during early passes

through the procedure are usually

unreasonable in some respect. Selective

revisions by use of the following might be

needed: adding criteria, further defining

or subdividing existing criteria,

re-calibrating scoring functions and

re-adjusting weights.

4.1 The final process is to assess each

alternative by trading off each's overall

worth by the following considerations:

risk, required resource expenditures,

temporal changes in objectives, aspiration

levels and tastes, and different and

possibly conflicting points of view among

diverse interest groups.

5.1 An experimental test of the procedure was

included in the report.

A-311

.

- . . A^VvIvIv ^ ^^ «. • - .»-•.-'» ..
Mills, R.G. & Hatfield, S.A. Sequential task performance; Task module

relationships, reliabilities, and times (AMRL-TR-72-56). Wright-Patterson

AFB, OH: Aerospace Medical Research Laboratory, 197^. (AD-787 322).

Topics Relevant

I to System Development

land Evaluation Technology ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurenents

1.2 System Taxonomy

Model (UN)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.3 Attempts to quantify human performance in

man-made systems are generally of a system

reduction form; reductionism means that a

system is analyzed into modular task

elements. Following this analysis, an

estimate of HPR (Human Performance

Reliability) and the performance time

associated with each module is obtained

with their mathematical synthesis provided

as an estimate of the complete system's

performance.

The problems underlying this procedure

include assumptions of normal

distributions, of a single distribution

1 derlying task-module performance time,

. f of interactions among task modules.

U.1 The statistical methods used to analyze the

data included calculation of the mean,

median, standard deviation, t-tests, and

Duncan's Multiple Range Test.

M.3 The apparatus for this study included a

display/control device, two books of Z

tables and a device for displaying single

lines of a computer print-out on a

trial-by-trial basis.

The Z-values were obtained by assessing the

tables using the displayed X and Y values

as table coordinates. The display/control

device consisted of five analog meters and

five digital read-out modules; one meter

was designated the "Primary Meter," the

remaining four meters displayed extraneous

information, while three of the five

A-312

iteteM* "^Ä • - - -
Topics Relevant

to System Development I Topic I

and Evaluation Tecnnolosy I No. ABSTRACT

i.

i

digital modules were collectively labeled

"Y".

5.3 The following results were indicated: (1)

the normality assumption for distribution

of task time is inappropriate; (2) the

rules for combining task times are

satisfactory if the underlying distribution

of task times is known; (3) HPR is affected

severely by combining tasks; and (U) any

model for estimating HPR will require

parameters to account for task combining

and difficulty.

KM -

A-313

iaä^ v-

- - -

^

i II

•. --

i.i
Mitchell, H.B. & Blarchard, R.E. The allocation of System Effectiveness

Requirements for man-machine effectiveness analysis. In R.E. Blanchard &

D.H. Harris (Eds.), Man-machine effectiveness analysis. Papers presented at

The Human Factors Symposium, University of California at Los Angeles, June 15,

1967. (AD-735 718).

Topics Relevant

i to System Development I Topic

land Evaluation Technology I Wo. ABSTRACT

State of the Art Review

of the Process

1.1 General System

Heasurenents

1.2 System Taxonomy

Model (ST>1)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.3 Test Plans

Application Components

of the Process

5.1 Test Execution

|.l Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recomendations

1.1 Allocation of System Effctiveness Requirements (SER's) is the process of determining

how the total system's effectiveness requirements distribute among the system's

constituent man-machine functional

units/states. To develop a procedure for

effectiveness requirements allocation,

guidelines must be generated for:

(1) Specifying the system effectiveness

requirement along all its dimensions.

(2) Partitioning the system into segments

and states.

(3) Characterizing and specifying input

data.

(1) Relating the SER to system segments

consistent with the input data.

2.1 The concept of requirements allocation

implies a multiplicity of contributors

to the meeting of those requirements.

"Contributors" generally fall into one of

two categories of verbal description: (1)

activities, or (2) system states.

The author found that analysts who are

activity-oriented tend to be more

stimulus-bound and less free from

pre-conceived notions than those who are

state-oriented. Specification of system

states tends less to imply transitlonary

methods for achieving those states and

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentiala/

Priorities

6.3 Research Planning A-314

- -. . .

.••: -•

. .-.-.-. -•.•.-.

1 liai .. r .._ «.• . .^ÄMMMM :.\., _•..,
! Topics Relevant

I to System Development

land Evaluation Technology ABSTRACT

tends to lead to a creative, open-minded

approach to analysis-both for new designs

and for evaluation of existing systems.

2.2 It is necessary that valid system

effectiveness requirements exist and are

derived from mission analyses, and that the

system is partitioned into manageable units

for evaluation of their contribution to

system performance. There still remains

the need for relevant data and procedural

rules for systematically applying those

data to enable allocation of the given SERs

among the component units.

i

-

t

'

2.5 A stipulated value for the SER is established to define an acceptable level of

performance with respect to system

objectives. Effectiveness requirements may

take the form of a single value, several

values or an interval. These values

represent levels of effectiveness which are

acceptable under specified operating or

environmental conditions. When more than

one effectiveness dimension is needed to

reflect the system objective adequately,

the SER may be represented as an index

resulting from the mathematical combination

of values on several effectiveness

dimensions.

For allocation of SERs, mission analyses

must have been directed toward defining

requirements appropriate for effectiveness

analyses. Values along all relevant

dimensions must emerge as an end product.

Currently such end products are sorely

lacking due to the intuitive approach to

design for meeting imprecisely defined

system objectives. SERs are rarely

specified, either because (1) they had not

bctjn considered, or (2) the researchers

don't wish to face the fact that serious

objectives may not always be reached, or

(3) they are unwilling to record

fallibility.

A-315

—i— ' ttm ^* •• - - - - - ^ :-m •..■■■
! Topics Relevant

I to System Development I Topic

land Evaluation Technology ! No. ABSTRACT

3.1 Quantification of effectiveness requires

identification of one or mere measurement

dimensions. Most freeuently used

measurement dimensions are accuracy, time,

quantity, and rate, constrained by cost

limitations. Effectiveness dimensions must

be related as directly as possible to

stated system objectives.

5.3 The allocation of SERs must provide a set

of performance requirements or standards at

a level sufficiently elemental to

facilitate:

(1) Trade-off studies.

(2) Relative appraisal of various system

design concepts.

(3) Absolute evaluation of given design

concepts.

(4) Absolute evaluation of a given design

against the SERs established for the

system.

A-316

-•-•-•-• — - — - - - - - : . ^
Mitchell, M.B., Smith, R.L. & Blanchard, R.E. Test appliration of TEPPS on a

Navy CIC aubayatero. Santa Monica, CA: Dunlap and Assoc, Inc., August 1967.

(AD-821 57"!).

Topics Relevartt I

j to System Development I Topic 1

land Evaluation Technology I Ho. I ABSTRACT

State of the Art Review

of the Process

1.1 General System

Neasureaents

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environaent Definition

2." Ger.-ral Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1) Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

it. 2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 The Technique for Establishing Personnel

Performance Standards (TEPPS) provides for

establishing performance standards that are

defined as criterion measures along such

effectiveness dimensions as probability of

accomplishment and time to perform

definable man-involved operations. The

technique involves analysis of a system

into describable and useful units of

personnel behavior. Once that has been

accomplished, system effectiveness

requirements are allocated along the units,

and the resulting standards can be used to

evaluate the efficacy and efficiency of a

particular system design relative to stated

system effectiveness requirements.

To achieve those objectives, application of

TEPPS involves the following general

procedures:

(1) Collect and organize pertinent data.

(2) Construct the Graphic State Sequence

Model (GSSM) relating system required

states to one another and to

man-machine activities.

(3) Relate probability and time to achieve

system output to probability and times

to accomplish defined man-machine

activities:

OR

(1) Employ TEPPS computer progr am for

deriving standards by allocating

A-317

• •-.-• ^---" V - ' V V - v .-" - "

-'-'-'-'- • - - - I •—. * '
Topics Relevant

! to System Development

land Evaluation Technology ABSTRACT

effectiveness requirements to

man-machine activities:

(2) Compare performance standards with

human capability values.

(3) Evaluate differences and determine

optimal corrective action.

^

2.1 A field test of the TEPPS was performed on

a simulated, conventional radar detection

and record-keeping subsystem of an anti-air

warfare system.

3.1 Data was collected by observation,

documentation and interviews at a training

center, and a GSSM was developed.

3.2 Since the program required human capability

estimates, subjective data were collected

from 16 expert judges on 29 unique

activities identified in the GSSM.

fi

-

3.1* Indices of probability of accomplishment

and performance time estimates were derived

from those data and were included as part

of the input to TEPPS computer program,

along with the coded GSSM and the imposed

effectiveness requirements. TEPPS computer

program was run and probability and time

standards were established under all modes

of system operation.

•

•

A-318

- L -^-,

-

a

—* *-^.»--> M
Montgomery, D.C., Callahan, L.G. * Wadsworth, H.M. Application of declslon/rlak analysis in operational tests and evaluation. Atlanta, GA: Georgia

Institute of Technology, The Sch )ol of Industrial and Systems Engineering,

September 1975. (AD-A024 205).

Topics Relevant I

I to System Development I Topic

land Evaluation Technology ! No. ABSTRACT

State of the Art Review

of the Process

1.1 General System

Heasureaents

1.? System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1 General Constrain's

2.5 Performance

Requirements, AtiUtt

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

1.1

4.2

«.3

«.I

t.5

Analytic Methods

Parameter Determinations

Apparatus for Testing

Personnel for Testing

Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoanendat i ons

Further Resea.-ch Areas

6.1 "jasurement System

limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1,1 The objectives of the project were to

develop a methodology with a set of

procedures for applying decision/risk

analysis to the design of operational tests

and the analysis of operational test

results. An operational test was defined

as that test and evaluation conducted to

estimate the prospective system's utility,

operational effectiveness, and operational

suitability. One of the objectives of

operational testing is an independent

evaluation of competing systems resulting

in some statement of relative attributes

and preference.

3.5 The project contains a review of four

analytical procedures for risk assessment.

Risk analysis can be viewed as the process

of combining the risk assessment with

alternative courses of action in an

iterative cycle.

(1) "An application of Hultlvariate

Discriminant Analysis and

Classification Procedures to Risk

Assessment in Operational Testing.n

This research developed a methodology

for determining an index useful in the

assessment of risk in operational

testing. The risk assessment problem

examined is that of preference

statements regarding systems. To

evalute the competing systems,

multivariate distributions of each

system and the overlap of these

distributions is used to determine the

index of risk. The index of risk is a

A-319

-te. -• 1

JS JS - - .
I Topics Relevant

! to System Development I Topic

land Evaluation Technology ! No. ABSTRACT

measure of similiarlty of the

competing systems.

(2) "An Application of Simulation

Networking Techniques in Operational

Test Design and Evaluation." Operational tests contain several activities, subtests, or subprograms. Each

of these activities or subtests has

related functional values. These

values may be deterministic,

stochastic, or some mathematical

transformation of a value computed in

an earlier acLivity. These activities

lead to milestones or events and the

outcome of the operational test can be

represented by a set of successful and

a set of unsuccessful events. This

set of conditions describes a

stochastic network. Of the network

analysis tools, network simulation

affords the greatest versatility and

flexibility in modeling this set of

conditions. Of the family ol network

simulation programs two programs have

evolved as useful analysis tools to

assess risk, SOLVNET and VERT. The

objective of this research is to

investigate the use of these network

simulation programs in the design of

operational tests and the analysis of

operational test data.

(3) "An Application of Bayesian Analysis in

Determining Appropriate Sample Sizes

for Use in U.S. Army Operational

Tests." The research was devoted to

modifying the Bayesian Techniques

associated with determining the

minimum sample size required to

construct interval estimates of the

true mean of an experimental or

sampling process which is modeled by a

normal distribution with unknown

parameters. The procedure considers

only the case where the prior

information can be represented by a

A-320

, . ^^^^^^^^^^_ -..-.-.
I Topics Relevant

i to System Development

land Evaluation Technology ABSTRACT

normal distribution with known mean

and known variance.

(4) "Finding a Mii.imuro Risk Path through a

Network U5iiig Resource Allocation

Techhiques." The objective of this

research was to develop an

optimization method for network risk

analysis where a resource constraint

is present. (This thesis was

unfinished and only the methodology

outline was presented.)

A-321

. • . -. . -

- - - - - - - - - -
Mumford, J. & Smith, J.P. The development of perfonmmce criteria, for

turret mechanica (HumRRO Research Memo.). Alexandria, VA: Human Resource

Research Organization, July 1961. (AD-477 647).

•

I • Topics Relevant

i to System Development

land Evaluation Technology ABSTRACT

State of the Art Review

of the Process

2.1

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM) 3.4

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components 4.5

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Envlromnent Definition

2.t General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1t Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysi5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

The armor turret mechanic and his equipment

in the training environment was the subject

of this study.

This study attempts to develop performance

criteria for turret mechanics.

The first step is to collect information on

the task at the organizational level. This

was accomplished by studying job descriptions and interviewing consultants

knowleag»able in the field.

Tasks selected, in a large measure,

reflected juo^ment of personnel consulted.

Exercises and tests were developed and

administered to subjects. A scoring system

was developed whion enabled the tests to

distinguish degrees of adequacy or

inadequacy of performance. On the basis of

this testing, the exercises were revised

and a new draft prepared.

A-322

>•.•-.•»--. .••. .•.•. . •. •. •. . • >». •. •.> .^j. - .-> > .-.--. -> •. -• .

v

. •,- f> ^. . •. -. . •,

- - ,- _. ... . . .
Nelson, R.T. A research methodology for studying complex service systems

(Working Paper No. 139). Los Angeles, CA: University of California, Western

Management Science Institute, July 1968. (AD-673 233).

I Topics Relevant I

I to System Development I Topic

land Evaluation Technology ! Ho. ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Moüel (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.14 Perforaance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

1.2 Parameter Determinations

1.3 Apparatus for Testing

1.1 Personnel for Testing

1.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.1 Conclusions and

Recoonendatlons

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 The purpose of this study Is to propose a

research methodology which considers the

physical, environmental, and decisionmaking aspects of complex service systems

as a total system and to present a methodology in generalized concepts and terminology

to emphasize the potential application to

all systems.

A schematic diagram is presented to structure the author's view of the physical

service system, Intra-systero decision

making and the environment.

An illustration of the methodology proposed

is also presented In this report. The

example given is a simple production/inspection system producing a single product to

inventory.

The outline and description of the proposed

research methodology Is presented below:

METHODOLOGY OUTLINE

(1) Identification and Modeling

a. Modeling of the physical system.

b. Characterization of the system

decision process.

c. Modeling of the system criterion

function.

d. Modeling of the environmental

response process.

A-323

• • • - - - - - - ,----.

. • •-.

_ • _ - - t__ <
! Topics Relevant

i to System Development !Topic I

land Evaluation Technology | No. j ABSTRACT

(2) Experimentation

a. Input-output analysis.

b. Equilibrium analysis.

c. System performance evaluation.

d. Variation of system-environment

decision variables.

e. Variation of system control

variables.

f. Variation of system design

variables.

(3) Implementation

An illustration of the process is presented

which includes continuing observation of

the operating system and its environment,

and modification of the system model as

well as continuing experimentation of the

decision algorithm.

It is expected that field studies may

present far more of a challenge because of

the identification, modeling and data

requirements which will arise in actual

situations. The methodological plan

proposed here focuses attention on these

requirements as an integral part of systems

analysis.

A-324

•-.-•.

J

----• - • - k^fltMb
Obermayer, R.W. & Vreuls, D. Combat-ready crew performance measurement system;

Phase I measurement requirements (AFHRL-TR-T^-loe-II). Northridge, CA: Manned

Systems Sciences, Inc., December 1974. (AD-B005 518L).

I Topics Relevant I

! to System Development I Topic

land Evaluation Technology ! No. ABSTRACT

1. State of the Art Review

of the Process

1.1 General System

Heasurenents

1.2 System Taxonoay

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.<< General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

1.3 In an effort to improve training performance Information, this study Is directed

towards a systematic definition of performance and development of methods for measurement. The primary goal Is to provide usable measurement tools for attacking problems related to combat-crew training. This

report covers the first phase - - definition of requirements for information based

on data collection surveys to six selected

combat-crew training sites.

2.1 The systems addressed were the A-7, B-52,

C-130, C-mi, F-l, and F-106 weapon systems, their crews, and their training.

;

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

lt.1 Analytic Methods

4.2 Parameter Detennlnatlons

1.3 Apparatus for Testing

ll.lt Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoonendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

2.6 The performance of trainees during and at

the end of training is the measure by which

these training programs can be Judged.

3.1 Instructional system development requires

that performance standards are identified

so that the most efficient approach is used

to train for the needed skills and knowledge to the desired level of performance.

The first phase of the study consisted of

three tasks: (1) Data collection; (2)

Analysis of common measurement

requirements; (3) Dimensions of measurement

modularity.

A-325

^* _»

« ' • - '

- - . .

.-r.
Topics Relevant

! to System Development {Topic

land Evaluation Technology I No. ABSTRACT

(1) Data Collection All existing

subjective and objective methods of

measuring performance during and at

the end of training were examined.

Researchers attempted to:

(a) Properly consider measurements in

the context of combat-crew

training.

(b) Assess the measurements already

included as well as potential

measurement.

(c) Assess the constraints placed by

the combat-crew training

environment on feasible usable

measurement systems,

(2) Training Measurement - - At each site,

information was collected with respect

to:

(a) The training sequence.

(b) Points where measurement exists.

(c) Measurement possibilities.

(d) Feasibility of research

measurement.

(3)

(e) Specific new measurement

development.

Six quite different aircraft were

included in the sample for measurement

analysis and an attempt was made to

determine whether a simple practical

measurement system for all

applications is possible.

Prototype Measurement - - As a natural

extension of the considerations of

measurement commonality, examples of

the Information required for training

were developed. These measurement

requirements are extensive and

A-326

t--'- MM

-' .* -" .•

_,_--- - -- - -. - ^ . .
Topics Relevant I

i to System Development 1 Topic

land Evaluation Technology I No.

complex. However, this detail is

probably needed to support the

training process.

5.*» This study is continued in Phases II and

III.

A-327

L .- .

•.-- " v" o -"V - --••• . •. . • . •. • . - - •. %".•*. .*.

mi* ..t-^- !. .... . . .jfJ
Obermayer, R.W. & Vreuls, D. Combat-ready crew performance measurement system;

Phase II measurement system requirement (AFHRL-TR-T^-IOS-III). Northridge, CA;

Manned Systems Sciences, Inc., December IQ?'*. (AD-B005 519).

I Topics Relevant I

I to System Development I Topic I

land Evaluation Technology ! No. ! ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurement?

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.'! General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.M Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

*i.? Parameter Determinations

1.3 This report describes the second phase of a

study directed towards a systematic

definition of performance and development

of methods for measurement in an effort to

improve training performance information.

This phase deals with the requirements for

a measurement system to process the

measurement which has been dictated by the

previous reports.

3.2 Research procedures were developed to

indicate the operation in which a

measurement system is to serve as a tool in

achieving research goals, and methods of

measurement processing were determined to

investigate the nature of data processing

associated with training research

measurement. In addition, system criteria

to guide design tradeoffs were addressed

and preliminary systems analyses conducted

to establish measurement system

requirements which follow rather directly

from the system criteria. In the latter

effort, measurement parameters were

identified and the approximate number of

measurement parameters for each flight

phase were presented.

1 3 Apparatus for Testing

u 4 Personnel for Testing

4 5 Test Plans

Application Components

of the Process

5 1 Test Execution

5 2 Data Analysis

5 3 Findings Interpretation

5 4 Conclusions and

Recoomendations

Further Research Areas

6 1 Measurement System

Limitations

6 2 Research Potentials/

Priorities

6 3 Research Planning

A-328

-

•->- --- _te* —
Obermayer, R.W. & Vreuls, D. Combat-ready crew performance measurement

jystem; Phase IIIA crew performarce measureroert (AFHRL-TR-7i*-108-IV).

Northridg», CA: Manned Systems Sciences, Inc., December ig?1*. (AD-B005 520L).

Topics Relevant I

I to System Development ITopic

land Evaluation Technology 1 No. ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

?,3 Environment Definition

2.1) General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

1.2 Parameter Determinations

1.3 Apparatus for Testing

1.1 Personnel for Testing

1.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.1 Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.3 This report describes Phase IIIA of an

effort to improve training performance

Information. This study is directed

towards a systematic definition of

performance and development of methods for

measurement. The primary goal of the study

is to provide usable measurement tools for

attacking problems related to combat-crew

training. In accordance with the initial

definition of this study, emphasis was

placed on pilot performance, but it was

soon recognized that avoiding the

contribution of other crew members and the

interaction between crew members, had

serious consequences. Therefore,

additional tasks were undertaken.

2.1 The systems addressed in this phase are the

F4E and the C1M1A weapon systems and their

combat crew training units.

2.1 After the requirements for pilot

proficiency measurements were established,

time was given to the consideration of the

variety of possible systems and to the

constraints imposed. A conceptual design

had been formulated, consisting of feasible

alternatives, to indicate the type of

information possible, the places where such

information would be useful, and the

possible ways such information could be

collected. It was concluded that

measurement of crew/system performance (as

opposed to pilot only) was important to a

thorough description of performance for

certain tasks and missions. Consequently,

the additional task of defining crew

A-329

——.

:- -•-

.......u.. - a --- - - - -- --
Topics Relevant I

! to System Development I Topic!

and Evaluation Technology ! No. I ABSTRACT

!

proficiency measurement requirements was

undertaken and these requirements will be

incorporated into the conceptual design.

1.5 This phase addressed the problem of crew

performance as opposed to pilot

performance. A survey was conducted to

thoroughly define requirements in the

measurement-system to sense crew

interaction. The survey was conducted at

nine sites, two of which were visited

specifically for the collection of

information on crew performar.ee.

5.4 One of the main difficulties in performing

detailed analyses of crew interaction is

the lack of specificity in stating requirements for crew relationships. Since crew

interaction techniques are non-standard,

and specific techniques are non-trained,

measurement of crew interaction cannot be

explicitly defined in most cases. A

further difficulty is the degree to which

the performance of each crew member is

dependent upon the performance of others.

It was concluded in this phase of the study

that better methods of crew interaction

must be discovered and developed.

A-330

-v ;.--^ - .- ^-^'.-r^-. ^y
'. v,

Obermayer, R.W., Vreuls, D. & Conway, E.J. Combat-ready crew performance

measurement system; Phase IIIC design studies (AFHRL-TR-T^-IOS-VI).

Horthridge, CA: Manned Systens Sciences, Inc., December ig?1». (AD-B005

521L).

Topics Relevant |

I to System Development iTcplc

land Evaluation Technology | No. ABSTRACT

-.

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

. Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Misslo.-i Definition

2.3 Environaent Definition

2.H General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

1.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

Further Research Areas

6.• Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.3 This report covers a further phase In the

research for the Improvement of eombat-rrew

training. This effort deals with design

studies to determine desirable system

features to meet the research needs

documented In the earlier reports of this

sequence.

An analysis of factors to be considered In

training measurement system design Is

presented and the nature of tradeoffs for

each system criterion established are

indicated.

3.1 A number of analyses were performed in this

program to guide design decisions. Eight

primary analytic steps were:

(1) Determine Measurement Needs

(2) Identify Physical Parameters for

Measurement

(3) Develop Automated Measure Descriptions

(4) Develop Manual Measure Descriptions

(5) Determine Where to Obtain Information

(6) Determine Data Processing Needs

(7) Determine if Visual Information is

Sufficiently Accurate

(8) Determine Cost Tradeoff Data

A-331

- - » - - ^ - - ^_ 
i Topics Relevant

! to System Development

land Evaluation Technology ABSTRACT

A measurement example was produced as a

product of visits to combat-crew training

units to attempt to express the main items

of information relevant to training. Using

this as a stimulus, preliminary measurement

definitions were made along with assumed

techniques for computation, leading to

identification of a required set of

parameters for measurement generation.

These analyses were begun in earlier

program phases, and subsequently were

revised and extended.

Continuing from the basic analyses,

specific computational algorithms were

chosen for both automatic and manual modes

of measurement, forming an initial software

specification. The next steps in the

sequence attempt to determine best methods

for sensing the needed information and the

nature of appropriate data processing

equipment. As ' ideo or photographic means

of sensing information must be considered,

it follows that a minimum resolution for

such devices must be specified to ensure

that the desired data are sufficiently

legible. Lastly, data are collected

relating to cost and personnel requirements

to permit tradeoff analyses between

alternative measurement system candidates.

The design analyses resulted in tradeoff

comparisons at two levels: (1)

comparison of competing data sources, i.e.,

audio, X-Y, video/photo, and

instrumentation (digital recording), and

(2) comparison of systems built around only

video/photo sensors and only digital

recording. Tradeoff comparisons at the

first level reveal the rule of alternative

data sources, while second-level

comparisons establish cost-effective system

combinations.

A-332

-■■■.

- • -• - - - ------•---'- ' -
Topics Relevant I

1 to System Development 1 Topic!

and Evaluation Technology j No. j ABSTRACT

5.1 A clear uncomplicated choice is not

possible between video/photo and digital

recording approaches to measurement system

design, but if s ich a choice must be made,

video/photo recording will bj chosen for

cost, information provided, flexibility and

ease of use.

However, a hybrid system, combining the

advantages of both, is preferable to either

type of recording alone. Due primarily to

cost, the bulk of measurement parameters

would be derived from a video/phc ,o system,

and the remainder with a small digital

recording capability. It would be

desirable for the major components of a

hybrid system to have a stand-alone

capability of modest means and power for

all combat-crew training measurement when

used together. Auditory data recording

should be incorporated together with the

option for merging data with that from

ground-tracking radar. All data recording

must include provision for synchronization

with other data sources.

A-333

'
Obermayer, R.W., Vruels, D., Muckler, F.A. & Conway, E.J. Combat-ready crew

performance measurement system; Phase HID specifications and implvmentation

plan. (AFHRL-TR-T^-loe-VII). Northridge, CA: Manned Systems Sciences, Inc.,

December 197^. (AD-B005 5221).

Topics Relevant

I to System Development

land Evaluation Technology ABSTRACT

State of the Art Review

of the Process

1.1 General Systen

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measi 'es

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Reconnendations

1.3 Phase HID of this study, designed to

improve training performance information,

presents specifications and an implementation plan for the performance

measurement system recommended as a result

of this program. The report covers the

acquisition and processing of data

personnel and facilities required, and

describes the implementation plan.

••3 The appendix of this report provided a

tentative survey and specification of

equipment and instrumentation required for

the implementation of the performance

measurement program. There are three major

sections to this appendix. Section one

addresses the monitoring and data

collection equipment anticipated for

installation on the test aircraft(s). The

second section discusses the post-flight

ground debriefing facilities and associated

equipment. The last section describes the

data processing facilities and equipment

requirements anticipated for detailed

analysis and evaluation of collected data.

I.'* An analysis of personnel requirements was

conducted in Phase IIIC of this study.

Further clarification is presented here

with a reassessment of the total personnel

requirements of the performance measurement

system.

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-334
1 Topics Relevant

I to System Development I Topic I

and Evaluation Technology ! No. I ABSTRACT

B^ U.5 The implementation plan recormnended was

based on Air Force Systems Command Manual

AFSCM 375-5. There are five major steps

required to implement this plan:

. . -

(1) Selection of a system integration

contractor.

(2) Completion of preliminary detail

system and subsystem design.

(3) Selection of the final system design

with appropriate testing.

(4) Procurement of system hardware and

system integration.

(5) Completion of final system tests

resulting in system turnover to the

Air Force.

A-335
O'Connor, M.F. & Buede, D.M. The application of decision analytic techniques

to the test and evaluation phase of the acquisition of a major air system

(TR-77-3). HcLean, VA:

WD.

Decisions and Designs, Inc., April 1977. (AD-A040

Topics Relevant

I to System Development ITopicl

land Evaluation Technology ! No. ! ABSTRACT

2.

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, ultimate

2.6 Performance

Criteria, ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

«.1

4.2

4.3

4.4

4.5

1.1 Currently the procedure for evaluating the

suitability of the air system from a human

engineering standpoint consists of test

pilot inspections of the air system. The

deficiencies are noted, the report is

written, and the defect put into one of

three categories. However, time is

important and so a method is needed to

communicate the information quicker with a

prioritization of corrections as well.

1.2 The answer to the questions of time and

prioritization is a computerized system

containing the test and evaluation

information appropriately prioritized. One

task to this goal is the development of a

hierarchical evaluation structure which

relates all the test and evaluation

Information to the missions of the F18 (the

aircraft system under consideration).

A diagram of that hierarchy is presented in

the report.

Analytic Methods

Parameter Determinations

Apparatus for Testing

Personnel for Testing

Test Plans

Application Coaponents

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoomendatlons

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-336
Pew, R.W., Baron, S., Feehrer, C.E. & Miller, D.C. Critical review and

analysis of performance models applicable to man-machine systems evaluation

(BBN Rep. No. 3416). Cambridge, HA: Bolt Beranek and Newman, Inc., March

1977. (AD-A038 597).

i Topics Relevant

I to System Development I Topic

land Evaluation Technology I No. ABSTRACT

-5.

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.'! General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3-2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

«.1 Analytic Methods

«.2 Parameter Determinations

*.3 Apparatus for Testing

4.4 Personnel for Testing

».5 Test Plans

Application Coaponents

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.1 Conclusions and

Recoomendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.3 The present document deals with the attempt

to design a guidebook for the evaluation of

nan-machine systems. The researchers feel

that advocating the rise of the information

processing data and modeling literature as

« base presupposes that models will be

developed with the most elemental components of performance and building from

there.

6.1 The limitations of models that were derived

by human information processing specialists

are:

(1) The models tend to be compartmentalized by the very fact that most of

them deal with particular stages of information processing rather than being

integrative of human performance in

general.

(2) For the most part, human information

processing models deal with the

average performance of well-motivated,

highly practiced Individuals, under

relatively ideal conditions. There

are many hypotheses but few data and

virtually no models in the information

processing literature on how human

performance capacities change under

stress, under reduced motivation,

before practice has stabilized

performance, when interacting in

groups or on the range, or characteristics of individual differences.

A-337
I Topics Relevant

I to System Development I Topic

jand Evaluation Technology I No. I ABSTRACT

6.2 Recomnendations for further research and

development of large scale systems

modelling efforts include the following:

(1) Development of a test bed to evaluate

alternative model formulations of

common task environments and to

conduct empirical validation studies

to compare model predictions with

actual human peformance.

(2) Methodological research on:

a. Implications of combining

sub-task or information

processing component models on

system performance in the

aggregate.

b. Validation of large scale

simulation models.

•

c. Development of guidelines for the

acceptable number of free

parameters in useful predictive

models.

(3) Further model development in topical

areas of high priority for representation of command and control systems.

(1) Advancing to state-of-the-art with

respect to the specific modeling

approaches discussed in the body of

the report.

A-338
.--

Phatak, A.V. Improvement in weapon system effectiveness by application of

identification methods for determining human operator performance decrements

under stress conditions (AMRL-TR-73-38). Palo Alto, CA: Systems Control,

Inc.. December 1973. (AD-773 856).

I Topics Relevant I

I to System Development !Topic

land Evaluation Technology ! No. ABSTRACT

>

-

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environnent Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1* Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

*.'( Personnel for Testing

4.5 Test Flans

Application Coatponents

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoomendatlons

1.3 The problem studied in this report is the

development of realistic models for weapon

system controllers that can be used to

predict the effectiveness of manned weapon

systems under stress conditions.

2.1 The system chosen for this study was the

pilot of a high performance aircraft.

2.2 The aircraft missions were engagements in

either precision weapon delivery or

air-to-air combat.

3.1 The task chosen for this experiment was a

single axis stable tracking task whose

dynamics were representative of a high

performance aircraft pitch axis control

system. However, two other compensatory

tracking tasks were also employed, but only

the task described above is considered in

this report.

4.1 Two types of models were considered: (1)

the input-output stochastic linear

state-variable models (equivalent to

describing function models when process

noise =_ 0), and (2) the optimal control

model developed by Kleinman, et al. The

maximum liklihood identification technique

was used in estimating model parameters

from input-output data.

Further Research Areas

6.1 Measurement System

LimlUtions

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-339
I

Topics Relevant

to System Development iTopic I

md Evaluation Technology ! No. ABSTRACT

M.S The subjects performed the tasks at sea

level followed by the same task at a

simulated altitude of either 12,000 or

20,000 feet. Each run of this experiment

consisted of two tracking periods. Each

period was preceded by one minute of

pre-breathing at the indicated altitude

followed by one minute of tracking.

The order of presentation of the simulated

altitudes and tasks was randomized in order

to minimize the effects of learning and

anticipation of experimental factors.

5.3 Results showed that the identification

algorithm was highly successful in

identifying the parameters of the stochastic state variable models. However,

difficulties were encountered in applying

the technique to identifying parameters of

the optimal control model. It appears that

the unsatisfactory identi; Ication results

with the optimal control model are due,

first, to the over parameterization of the

model structure and second, because of the

finite data and specific input-output data.

Inferences regarding performance decrements

due to stress were precluded because of

insufficient data base for this study.

A-340
Phatak, A., Weinert, H. & Segall, I. Identification of the optimal control

odel for the human operator (AMRL-TR-7M-79). Palo Alto, CA: Systems

Control, Inc., May 1975^ TÄD-AOOS 956)

T I Topics Relevant

I to System Development

and Evaluation Technology

ITopic!

i No. I ABSTRACT

1.

State of the Art Review ] ty

of the Process

1.1 General Syste«

Heasureoents

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Coaponents

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environaent Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Perfonoance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

<l.2 Parameter Determinations

^.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

In this report, the optimal control model

is analyzed from the system identification

point of view to determine which parameters

can be identified. As a result, a procedure for the identification of the optimal

control model parameters from measured

experimental data has been developed. This

procedure is validated by application to

experimental human operator data provided

by Aerospace Medical Research Laboratory.

In addition, a systematic approach is

presented towards the development of a

metric for measuring and ranking system

difficulty as experienced by human

operators in terms of system properties

such as the degree of controllability and

the degree of observability.

Application Coaponents

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.1 Conclusions and

Rec omtenda t i on s

Further Research Are»4

6.1 Measurement Syatea

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-341
Philco-Ford Corporation Human engineering design check list (WDL-TR 1968A).

Palo Alto, CA: Philco-Ford Corporation, May 196M. (AD-8?9 1»26).

Topics Relevant

! to System Development

land Evaluation Technology ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

S.* Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

t.2 Parameter Determinations

1.3 Apparatus for Testing

t.l Personnel for Testing

1.5 Test Plans

1.3 This report presents a checklist to be used

for human engineering acceptance testing

and design verification. It is also to be

used as a human engineering guide.

This checklist is a device by which human

engineering design criteria and standards,

as set forth in MIL STD-803A-1, can be

verified during acceptance testing and

checkout.

The human engineering design is verified

(a) in plant during equipment acceptance

testing and (b) in-the-field during

subsystem and system checkout.

Plant acceptance testing is composed of two

parts - - visual inspection and functional

test. Field (on-site) checkout has five

parts: completion of checks to verify

compliance with MIL-STD-803A-1 on equipment

acceptance-tested in-plant; inspection and

test of all control/display equipment;

monitoring of subsystem functional tests;

monitoring of system functional tests

during fly-bys and during system checkout

with the checkout-subsystem; analysis,

evaluation, and documentation of

verification results.

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.H Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-342
-•

Polak, H.T., Robertson, J.C. & Yuan, L.S. Systems Effectiveness Analyzer

(AMRL-TR-73-113). Sudbury, MA: Raytheon Company, Simulation and Surveillance

Systems, February 1974. (AD-781 12H).

Topics Relevant

I to System Development !Topic

land Evaluation Technology I No. ABSTRACT

4.

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STO)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.4 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recorauendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 The System Effectiveness Analyzer (SEA) is

a weapon system measuring system which is

based primarily upon the utilization of

existing equipment (controllers and peripherals) supplemented by the additional software required to enhance existing capabilities to evaluate simulated weapon systems.

2.1 The SEA System has two major functions:

(1) to checkout, control, monitor, and

perform statistical analyses associated

with tracking simulators; (2) to provide an

estimation of weapon system effectiveness.

2.5 There were several design features that

were considered imperative to the development of the SEA measurement system: input

flexibility, operator input options, simulator interface, turn-around-time minimization, and expansion.

3.5 The weapons system includes the human

tracker as its principal sensor of target

motion. The metrics used to evaluate

weapon system performance were the weapon's

round-by-round probability of kill and the

resulting target's probability of survival.

Angular error statistics are used to

evaluate tracker performance. One SEA

simulation has been implemented. The

simulated weapon system is an AAA battery

using manned optical sights. The target

being engaged is an aircraft deploying

optical countermeasures. The SEA commands

the tracking simulator's display of the

sights picture, both with regard to target

A-343
Topics Relevant

i to System Development

land Evaluation Technology ABSTRACT

r.

position and counter-measure stress level.

Outputs from the simulator include the

subject's control movements and the gun

trigger state. These outputs are used by

the SEA, in conjunction with the target

trajectory, to update the sights picture,

to accumulate the statistical measures of

the tracker's performance, and to compile

the data on which to base the effectiveness

analysis of the AAA system. The collected

data are essentially that which would have

been determined from an actual AAA system.

The Lead Angle Computer directs the gun to

the predicted position of the aircraft.

The mean miss distance of each shell fired

is estimated and error sources appropriate

to the gun system are used to find the

distribution of shot relative to the target

aircraft. Weapon lethality and target

vulnerability data are then used in conJunction with the shell mean miss distance

and scatter area to determine the kill

probability of each shell. The probability

that the aircraft survives the mission is

derived after each round is fired.

-

-.

-

A-344
Popham, W.J. & Husek, T.R. Implications of criterion-referenced measurement.

Journal of Educational Measurement, 1969, 6 (1), 1-9.

Topics Relevant

I to System Development [Topic

jind Evaluation Technology I Wo, ABSTRACT

1. State of the Art Review

of the Process

1.1 General Systaa

Measurements

1.2 System Taxonooy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

<t. Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Teat Plans

5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

6. Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 For several years, measurement and instructional specialists have been drawing distinctions between norm-referenced and

criterion-referenced approaches to measurement. A norm-referenced test identifes an

individual's performance in regards to the

performance of others; a criterion-referenced test identifies performance in regard

to an established standard of performance.

This report examines the implications of

these two approaches to measurement, particularly criterion-referenced measurement,

with respect to variability, item construction, reliability, validity, item

analysis, reporting and interpretation.

A-345
Potempa, K.W. A catalog of human-factors techniques for testing new

systems (AFHRL-TR-68-15). Wright-Patterson AFB, OH: Air Force Human

Resources Laboratory, February 1969. (AD-85^ 1*82).

I Topics Relevant ',

I to System Development ', Topic !

land Evaluation Technology I No. ! ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.t General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

0.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Coaponents

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoonendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 This report contains 16 human factors

testing techniques contributed by a number

of Air Force and contractor personnel.

While limited in number, it covers a broad

segment of the human factors testing

spectrum. This catalog was developed as a

prototype to determine its usefulness to

human factors personnel as a reference

source for human factors testing devices

and techniques.

The following information is generally

provided on each technique.

(1) Purpose

(2) Author

(3) Published references

CO Description of technique, including

how it is utilized, at what phase in

the system it is most useful, and

what kind of equipment is needed

(5) The advantages and disadvantages of

the technique

(6) The limitations, validity and reliability of the technique

The human factors techniques included in

this report and a description of their purpose are presented below.

(1) Miniature Simulation

To determine the maintainability of

an aircraft by evaluating the design

of the aircraft and its associated

aerospace ground equipment through

the use of a miniature scale model.

A-346
Topics Relevant I

I to System Development I Topic

! and Evaluation Technology ! No. ABSTR: CT

(2) Automated Readability Index

To determine reading difficul :y level

derived from ratios representing word

difficulty and sentence difficulty.

(3) Flight Simulator and Associated

Simulation Complex

To provide engineering data in the

area of guidance and control, navigation, human factors, target acquisition,

trajectory error analysis, sensor and

display development, and flying qualities.

m) Terrain Visibility Definition

To determine the in-flight vertical and

horizontal visual angles. This data

car then be used to determine the visual

area available to aircrews with the aircraft in the horizontal flight attitude.

(5) Definition of Functional Eye Position

To determine the eye position of pilots

seated in a cockpit in their normal

flight posture. This technique can be

used to generate a three-dimensional

eye ellipse with subjects seated either

in a cockpit mockup or in an actual

aircraft.

(6) Definition of Dynamic Body Positions

To locate various body components while

a man is riding a captive ejection seat.

(7) Personnel Activity Analysis Radio

System (i'AfiRS)

To collect job activity information,

via radio, on a number of operational

or maintenance personnel while they

are working dispersed over a wide area.

(6) Work Station Analyzer

To determine the viewing angles and

distances of display panels.

A-347
Topics Relevant !

to System Development '.Topic

and Evaluation Technology! No. ABSTRACT

(9) Secondary Task Monitoring

To determine performance decrement

under dynamic conditions on a flight

simulator.

(10) Operator Analysis in Command and

Control Systems

To analyze those operator performance

sequences which are judged to be the

most difficult to perform without

error.

(11) Human Factors Test Planning

To describe the development of a test

plan for field evaluation. It includes the development of test objectives and determination of data

requirements to plan a test program

which gives a well balanced coverage

of the areas of interest.

V

(12) Induced Error Technique for Evaluating

Command and Control Systems

To determine the criticality of various

problems that may arise under operational

wOnditions. Time to correct errors and

number of times the operator fails to

correct errors are determined.

(13) System Load Mission Plan

To determine efficiency of human and

system performance under heavy load

conditicns and to determine if the

combination of operators and equipment

specified can successfully handle the

maximum system load specified by the

operational requirements. An extensive

list of the performance measures utilized

in each of the operational analyses is

presented in this report.

A-348

t
Topics Relevant I

I to System Development ! Topic

land Evaluation Technology! No. ABSTRACT

(l^) Programmed System-Performance

Measuring Equipment

To provide an objective means of

evaluating performance on simulators

and avionics equipment. The equipment used was the Digital System

Synthesizer, a punched paper tape

recorder, a computer and necessary

software.

(15) DACOLS

To describe an automated method of

recording large amounts of time and

event data in a format which can be

readily reduced and analyzed by computer processing.

(16) Open-Ended Maintenance Questionnaire

To present a questionnaire designed to

collect information on maintenance

activities performed during the test

of new systems. The questionnaire is

of the open-ended type and was compiled

by taking those questions judged to be

the best from a large number which have

been previously used in system evaluation. Subjective judgment was the only

criteria used in selecting these questions and the questionnaire has not been

valided in a system test situation.

However, it is hoped that it may prove

useful in reducing duplication of effort.

A-349
Prltsker, A.A., Wortman, D.B., Seum, C.S., Chubb, G.P. & Seifert, D.J. SAINT:

Vol. 1. Systems Analysis of Integrated Networks of Tasks (AMRL-TR-73-126).

West Lafayette, IN: Purdue University, School of Industrial Engineering,

April ^9^^. (AD-A014 8M3).

Topics Relevant

I to System Development

land Evaluation Technology ABSTRACT

1. State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. "ontextual Components

jf the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3- Analytic Components

of the Process

3. I Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3-3 Performance

Requirements, Specific

3.^ Performance

Criteria, Specific

3.5 Measurement Procedures

l|. Planning Components

of the Process

4.1 Analytic Methods

<t.2 Parameter Determinations

i|.3 Apparatus for Testing

4.H Personnel for Testing

4.5 Test Plans

5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

6. Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potential«/

Priorities

6.3 Research Planning

1.1 SAINT (Systems Analysis of Integrated

Networks of Tasks) provides a graphic

symbol set for diagraimoing event sequences.

SAINT is a combination of a set of network

symbols (modeling vehicle) and simulation

(analysis technique). A topdown analysis

is employed such that the system is defined

i.i terms of missions with the output of

SAINT consisting of task and mission

performance estimates.

2.2 A mission was defined as a network of tasks

performed by a crew of operators having a

complement of equipment in the face of

environmental factors.

2.3 Workload stress is the time pressure

imposed on an operator by a discrepancy

between the amount of work to be done and

the time remaining for doing it. The

effects of this stress are reflected in the

operator's task duration and task success.

3.1 The set of problems focuses on task allocation, operator workload, and environmental

Stressors. It is conjectured that network

concepts and symbols can be developed that

will permit the modeling of one or more

operators performing an assigned set of

tasks within the context of a specific

mission and the operating environment for a

mission. Once the network concepts and

symbols are designed, a simulation program

can be developed for analyzing mission

performance as a function of operator and

environmental variables.

A-350
I Topics Relevant I I

I to System Development iTopic!

and Evaluation Technology I No. ! ABSTRACT

6.2 The following points are proposed:

(1) Verification of the factors and

relations included in the

characterization of task performance.

(2) Development of new concepts in order

to model tasks that require continuous

monitoring, queuing and resource

allocation.

(3) Extend the treatment of task type and

the method by which operators are

assigned to tasks.

A-351
Quinn, J.L. Research and development appraisal and evaluation (SLTR5-70).

Wright-Patterson AFB, OH: Air University, School of Systems and '.ogistics,

November 1970. (AD-876 006).

Topics Relevant

i to System Development {Topic

{and Evaluation Technology ! Wo. ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.'» Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

H.2 Parameter Determinations

4.3 Apparatus for Testing

1.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recomaendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 The particular method or combination of

methods employed in an evaluation must be

fitted to the given situation in light of

such variables as the level of research and

the stage of the development-cycle being

evaluated, along with a host of other

situatlonal factors. By using qualitative

means to assist in formulating quantitative

Judgments, management-decision makers can

predict, measure, and evaluate research and

development efforts and results.

Methods of evaluating research potential,

performance, and effectiveness may be

quantitative, qualitative, or combinations

of both. Quantitative methods employ

mathematical equations or models that are

coupled with dollar evaluation to arrive at

a figure of the research's effectiveness.

Quantitative methods evaluate the research

via the profits created and the improved or

cost-reduced products, techniques, processes, and materials. Qualitative evaluation is subjective Judgment, that is, the

composite Judgment of qualified and responsible management and research personnel

arrived at through logical, not mathematical, procedures and devices. Qualitative

methods are applied to the quality of

research results, the degree of research

efficiency, the results of long-range

research, and the intangible products of

research such as publications.

A-352
I Topics Relevant |

i to System Development 1 Topic I

!and Evaluation Technology i No. I ABSTRACT

1.2 A basic criterion for categorizing the

level of research effort is the degree of

uncertainty of research results. The four

classifictions of research are pure, basic,

applied, and developmental with pure

research at one end of the spectrum of

uncertainty and developmental research at

the other. The principal implication of

the spectrum of uncertainty on the evaluation of research effort is that the result

of research at the less uncertain end of

the spectrum may be predicted and measured

with some degree of confidence whereas

those at the upper end are more difficult

to predict and measure.

A-353
.-

Ramsay, J.O. Some atatlstical considerations In multldlmenaional scaling

(RB-66-26). Princeton, NJ: Educational Testing Service, June 1966. (AD-U89

591).

Topics Relevant

I to System Development

land Evaluation Technology ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (UN)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.,< Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 Like most statistical methods, multidimensional scaling Is merely a procedure for

converting the numerical data resulting

from an experiment into a more interpretable form. As with most statistical

methods, it is the hope of the experimenter

that not too much of the information available in the sample will be lost during the

course of analyses. The problem of minimizing the loss of information, therefore,

is really two problems - one of estimation

and the other of a scaling procedure.

The second problem has received considerable attention, while the first has

received virtually none. The purpose of

this study is to show the importance of the

estimation problem and to make some

appropriate comments.

A-354
V.

Rankln, G.L. An application of fault tree analysis to operational testing

(Master's Thesis). Georgia Institute of Technology, 1975.

Topics Relevant

! to System Development

land Evaluation Technology ABSTRACT

1. State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (Sm)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.4 Performance

Criteria, Specific

3-5 Measurement Procedures

4. Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 The purpose of this study is to determine

the use of fault tree analysis in operational test planning for military systems. The

research attempted to demonstrate how the

logic of fault trees could be used in an

area other than safety and reliability. In

this case it was in test design.

3.5 A detailed discussion was given as to the

procedures used in fault tree analysis:

fault tree terminology and symbology,

system definition, fault tree construction

and qualitative analysis, the identification and ranking of components and subsystems through fault tree analysis permits

direction of tests towards the weak links

in the system. System weakness and the

causes of important failures can be

explored in testing. Fault tree analysis

increases system understanding. It provides a vehicle to explore system alternatives. Structural analysis frees the

method from reliability data which may not

be available due to the nature of the

system or the equipment state of the art.

Fault tree analysis is a binary modeling

scheme which ignores partial failures. It

is also situation specific, that is, each

tree is constructed only about the failure

of interest. Hence, it can explore only

one critical issue for each model

developed.

An example was directed toward the operational testing of automated command and

control systems for use in the U.S. Army

division.

A-355
Rankine, R.R. The effects of aircraft dynamics and pilot performance on

tactical weapon delivery accuracy (UCLA-ENG-7085). Los Angeles, CA:

University of California, School of Engineering and Applied Science, November

1970. (AD-728 324).

Topics Relevant

I to System Development

land Evaluation Technology ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environaent Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.4 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions snd

Recommendations

further Research Aruas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 This report presents a model of piloted

weapon delivery through the application of

mathematical models of the human operator's

performance characteristics to the dynamic

description of the combined control-display-vehicle system. By doing this an

understanding of the interaction and relative importance of the various elements of

the system were obtained.

A statistical model of the propagation of

these pilot-induced errors into impact

error is then developed by considering each

of the pilot inputs to be a random

variable. A method for including the

effect of pilot compensation for an

observed error in one of the variables with

an intentional deviation in another is also

introduced. An analytical model of the

human pilot is used to estimate the

tracking error from the controlled-element

dynamics and the turbulence environment.

Typical pilot performance in controlling

the other task variables is estimated

empirically from piloted simulation

results. Once the system is modeled in

this manner, the effect of display,

computational, and dynamic changes on

weapon delivery accuracy can be readily

determined. A reiterative design approach

is used to improve the system using impact

point accuracy as a figure of merit; thus,

the effects of changes in displayed

information, of computational aids provided

to the pilot, and of varying degrees of

control system augmentation can all be

compared on a common scale.

A-356
I Topics Relevant

! to System Development {Topic

land Evaluation Technology I No. ABSTRACT

The approach is applied to a gun-firing

pass of a typical tactical fighter aircraft

equipped with a conventional fixed optical

sight. A sequence of design improvements

is proposed which are predicted analytically to improve impact accuracy. The proposed changes are then tested in a complete, six-degree-of-freedom, piloted simulation in order to validate the modeling

technique and determine the extent to which

predicted improvements materialize. The

simulation discloses that tracking error is

the principal contributor to strafing

inaccuracy and that poor tracking capability in either the longitudinal or lateral

axis will degrade the accuracy in both

axes. A five-fold increase in single

shot/pass hit probability is demonstrated

which can be primarily attributed to a 551

reduction in lateral tracking error. The

mathematical model of the air-to-ground

gun-firing pass is revised to incorporate

this coupling effect and other effects

noted in the piloted simulation.

A-357
Rasch, W. H., Jr. Guidelines for making tradeoffs;

Technical Performance Measurement. Ft. Belvoir, VA:

Management College. 1973. (AD-A015 256).

The special -role of

Defense Systems

Topics Relevant

i to System Development ITopic

land Evaluation Technology ! No. ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Coaponents

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environnent Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.i| Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Coaponents

of the Process

1.1 Analytic Methods

l|.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Coaponents

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.3 Generally speaking, this document describes

the trade-offs made during the various

phases of acquisition (e.g., of naval

vessels), explains TPM (Technical

Performance Measurement), and provides

guidelines for using the TPM outputs in

trade-off analyses.

2.1 The purpose of this system for making

trade-offs was to specify those performance

measurement factors relevant to ship

acquisition and to specify standards for

using TPM outputs to make the necessary

trade-offs for ship design decisions. The

system allowed for the determination of

which trade-offs were possible during each

phase of ship acquisition and gave a

"user's guide" for the process of TPM.

2.2 The primary goal of the trade-off TPM

system was the specification of the

performance/schedule/cost parameters that

eventually determine the system's

effectiveness and/or the success

probability of the mission.

3.1 The trade-off problem was attacked througli

several stages of analysis:

(1) The Validation Stage, or System

Contract Design (i.e., do individual

systems fulfill requirements?)

(2) Detail Design and Construction:

factors such as cost and delay, legal

approval, ongoing modification, etc.,

A-358
1 Topics Relevant

I to System Development 1 Topic I

land Evaluation Technology ! No. I ABSTRACT

must be considered during the

blueprinting of the ship design.

(3) Deployment and Disposal: technological

and "threat" changes should be

considered in an evaluation of the

ship's optimum capability; the ship

itself, as a total system, has a much

longer life than its subsystems.

3.3 Specifically, TPM was used to predict the

degree of actual or potential achievement

of certain technical objectives of a system

or subsystem, and causally analyzes the

variance between achievement and goals.

3.^ To quantify technical performance, the

following steps must be taken:

(1) Identify performance variables

(inputs) crucial for success and

relate them, by means of equations, to

design variables or outputs.

(2) Through questioning of design

personnel, subjective probability

distributions can be made for the

design variables.

•

(3) To estimate the probability of

obtaining desired performance or

meeting certain technical objectives,

appropriate techniques, e.g.,

simulation, can be used.

(4) Changes in the probability of

achieving target goals in performance

should be monitored.

M.5 In order to implement the TPM program, the

following elements play a determining role:

Parameter Selection and Documentation of

Detail; Construction of TPM models;

Profiling Parameters; Planning the TPM;

Assessing Organizational Participation;

Preparation of Reports and Formats; Data

Analysis and Predictions.

A-359
Rau, J.G. Meaaurea of effectiveness handbook.

Inc., August 1971. (AD-A021 461).

Irvine, CA: Ultrasystems,

I • Topics Relevant

1 to System Development I Topic

and Evaluation Technology ! No. ABSTRACT

1. State of the Art Review

of the Process

1.1 General Systeo

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.4 Performance

Criteria, Specific

3.5 Measurement Procedures

4. Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoonendatlons

6. Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.3 This document presents a summary of

measures of effectiveness used by the

Navy's Operational Test and Evaluation

Force (OPTEVFOR).

2.1 Several platforms, systems and subsystems

were considered.

2.2 The minimal operational situation was

described for each system or subsystem

discussed.

3.1 A reference source was provided based on a

review of OPTEVFOR reports, of measures of

effectiveness used in Naval warfare and

previous OPTEVFOR projects.

3.4 The specific criterion performance

requirements were presented followed by

specific measures of effectiveness for each

system.

6.1 The scope of this handbook is limited to

effectiveness measures only. Material

reliability and human factors measures are

not included. The handbook is for guidance

only and the coverage is not intended to be

complete.

A-360
Ray, E.C., Hiltz, H.A. & Spurway, D.T. Test design plan for Defense Satellite

Communications System III (DSCS III) initial Operational Test and Evaluation

(IQTtE). U.S. Army Operational Test and Evaluation Agency. Test Design

Division, 1979. (AD-B0M1 308L).

I Topics Relevant I I

1 to System Development iTopic I

land Evaluation Technology ! No. | ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPU)

Contextual Component?

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.11 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1) Performance

Criteria, Specific

3-5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

4.2 Parameter Determinations

1.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

Further Resea-ch Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

2.1 The Defense Satellite Communications System

Phase III (DSCS III) consisted of four

operational satellites that were

synchronized with the earth's movements,

and two spare satellites, positioned

equidistant at equal i.^ervals from the

earth's equator. Several types of "Earth

Terminals" (ETs) al-n comprised the system

at various locations around the world, with

four to eight SCCE facilities (Satellite

Configuration Control Elements).

2.2 The primary "mission" of the DSCS was an

increased communications capability,

particularly an improved ability to operate

in an Electronic Warfare (EW) environment.

2.5 The ultimate performance requirement of the

test design plan was the collection and

analysis of data regarding operational

effectiveness, vulnerability of the system

in an EW environment, assessment of the

operational, maintenance, and logistic

support concepts of the system, RAM data

(reliability, availability, and

maintainability).

3.1 Operational issues Included the areas of

mission performance, vulnerability,

training, organization, RAM, and

safety/human factors engineering.

3.2 These will be measured through such indices

as Bit Error Rate (BER), Test Tone-to-Noise

Ratio (TTNR), and carrier-power-to-noise

spectral density ratio. These measurements

A-361
I Topics Relevant

I to System Development I Topic!

and Evaluation Technology ! No. ! ABSTRACT

are used to determine if a link (circuits)

will pass data traffic.

4.2 Test factors were identified which wiuld

probably influence the operational

effectiveness of DSCS III; examples of

those factors which are varied included

communications access, user community,

antenna coverage, satellite configuration,

etc.

U.I There are four test conditions which

required action from the satellite control

system: a "normal" mode, "degraded" mode,

"hostile" electronic warfar" environmental

mode, and subnetwork control mode.

-]

General data collection involved

information on performance of the oystem

with regard to:

(1) Reception of telecommunications

service order from Defense

Communication Agency Operation Center.

(2) Performance of network analysis using

resource allocation software in

network computer.

(3) Instruction of satellite controller to

command necessary communications

payload change.

(1) Instruction of earth terminal

operators to establish link with

parameters as specified.

A-362
Repperger, D.W., Smiles, K.A., Neff, J.A. 4 Suraners, W.C. A feature selection

approach in evaluation of parameter changes on the human operator under

thermal stress. Ergonomics, 1978, 2U1), 35-^8.

Topics Relevant

i to System Development I Topic

land Evaluation Technology i No. ABSTRACT

State of the Art Review

of the Process

1.1 General Systec

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

J.H Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

4.2 Parameter Determinations

1.3 Apparatus for Testing

1.1 Personnel for Testing

1.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.1 Conclusior . and

Recommendations

1.3 A feature selection approach to define

changes in parameters of the human operator

is used in this study.

2.1 The human operator was involved in a

closed-loop tracking task simulating a

flight in a very unstable aircraft or a

very stable aircraft.

2.3 The experiment was conducted in a

laboratory situation under thermal stress

conditions. The subjects were exposed to

an environment of 50 C dry bulb and 30 C

wet bulb to simulate a pilot forced to sit

prior to take off in a heat-soaked

aircraft.

1.1 By choosing a class of features which

characterize the general shape of the human

operator's transfer function, significant

changes in these characteristics were found

to occur under thermal stress conditions.

An F-Ratio Test was used with an analysis

of variance to test the significance of the

change in those features which describe the

input/output characteristics of the human.

4.2 The human's parameters are assumed to be

characterized by second-order transfer

functions, and features of the transfer

function are chosen which describe its

general shape.

1.4 Four healthy and highly trained male

subjects were used in this experiment.

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-363
1

I Topics Relevant

I to System Development

jand Evaluation Technology

i

ITopic

! No. ABSTRACT

4.5 These subjects were exposed for 1 hour to a

simulated heat-soaked aircraft environment.

They performed a single dimension compensatory tracking task for b minutes

duration, separated by 5 minutes of rest.

The tasks represented flying a very

unstable aircraft cr a very stable aircraft

under vertical wind buffet.

•>.

5.3

E

Each subject participated in six experimental conditions, three control runs and

three exposures to the heat-loading

environment. During the experiment the

subject maintained one of three conditions

of water-electrolyte balance. He either

drank nothing or replaced weight losses

with water or NaCl solution. The subject

urinated and blood samples were drawn

periodically. Mean skin temperature,

rectal temperature, weight loss, heart

rate, air temperature, water temperatures,

and humidity were recorded along with

tracking performance parameters.

The results of the experiment indicated

that the effects of thermal stress (and the

type of electrolyte used in the recovery

period) will significantly change the

input/output characteristics of the human

operator.

A-364
v.

Rhoads, D.W. In-flight evaluation of your cockpit controller configurations in

a variable stability airplane {AFTDL-TR-70-95). Buffalo, NY: Cornell

Aer )nautical Laboratory, Inc., September 1970. (AD-876 589).

I Topics Relevant I

! to System Development [Topic

iand Evaluation Technology ! No. ABSTRACT

F.' •>

State of the Art Review

of the Process

1.1 General Systeo

Measurements

1.2 Systen Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPU)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environaent Definition

2.U General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

l.1* Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5-3 Findings Interpretation

5.4 Conclusions and

Recoooendatlons

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

2.1 This report described an in-flight

evaluation of four cockpit-controller

configurations in a variable stability

airplane with pilot.

2.2 Evaluation was based on three tasks: up

and away cruise condition maneuvers, low

level terrain following simulated altitude,

approach landing, and take-off.

2.3 Each task was performed under conditions of

different simulated static and dynamic

characteristics of a B-1 type airplane.

3.1 Qualitative and quantitative data were

obtained consisting of pilot inflight

comments and pilot ratings of handling

characteristics and tracking error.

3.5 The pilot inflight comments were responses

to a prepared comment guide and the last

Cooper-Harper handling rating scale was

utilized

4.1 Much use of simple average numerical rating

data were used in pilot rating analysis;

average of replication, average of pilots,

etc. In the tracking tasks, raw data were

oscillograph recorded. A limited amount of

raw data were reduced to a variance

spectral density form. This form indicates

variance as a function of frequency and

shows the variance present in any frequency

bandwidth throughout the spectrum (and

hence a frequency bandwidth for maximum

variance) and the overall variance for the

paramenters of the maneuvers.

A-365
I

i Topics Relevant !

I to System Development !Topic

land Evaluation Technology ' No. ABSTRACT

1.3 A conventional wheel and colunn and three

new concepts: a circumferential wheel and

colunn; a circuLferential wheel and column

with hand controller; and a duel side arn

configuration were evaluated using a CAL

B-26 variable stability airplane of the B-1

type.

M.I Four pilots two USAF and two CAL

participated in these tests.

1.5 The pilots flew all four controller configurations four times each through each of the

three tasks.

.

i

5.3 Results of the analysis indicated that all

three new concepts would be accepted by

pilots of large airplanes with only a

nominal associated learning period and that

all three new concepts are preferable to

the conventional equipment. Despite some

detailed design deficiencies, the dual side

arm configuration was most preferred.

A-366
Rigby, L.V. The Sandia Human Error Rate Bank (SHERB). In R.E. Blanchard &

D.H. Harris (Eds.), Man-machine effectiveness analysis. Papers presented at

The Human Factors Symposium, University of California at Los Angeles,

June 15, 1967. (AD-735 718).

•

I Topics Relevant I

I to System Development !Topic

land Evaluation Technology | No. ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Corceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.<t General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

l.k Perforaamce

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoonendatlons

Further Research Areas

6.1 Measurement System

Ualtatlons

6.2 Research Potentials^

Priorities

6.3 Research Planning

1.1 The Sandia Human Error Rate Bank (SHERB) is

an evolving file of Human Error Rates (HER)

on specific tasks associated with component

assembly, equipment installation, system

operation, and maintenance. It is intended

to serve as a data base to allow analysts

to estimate the probability of human error

and its impact on system performance.

3.1 If human error is defined as any variant of

human performance that reduces the

probability of system or mission success,

then failures due to human error can be

treated very similarly to component

failures, i.e., human errors can be

predicted as a probabilistic function of

the variables determining or influencing

that human performance related to system

performance.

6.1 Development of an accurate data base of

human error rates is impeded by the

following factors: (1) Accidents and

mission failures resulting from human error

are not reported with the same regularity

and accuracy as equipment failures; there

is a lack of good feedback data. (2) The

data that are available vary widely in

terminology, manner of development, and

level of reporting; there is a lack of

standardization.

A-367
Rouse, W.B. Estimation and control theory: Application to modeling human

behavior. Human Factors. 1977, ^.(M), 315-329.

Topics Relevant

1 to System Development iTopic I

land Evaluation Technology ! No. i ABSTRACT

SUte of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.U General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Coopone its

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Coaponents

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Rec oomend a ti ons

Further Research Areas

6.1 Measurement System

Limitation,;

6.2 Re-^arch Potentials/

Priorities

6.3 Research Planning

1.1 The methodology of estimation and control

theory is considered in terms of response,

stability, estimation, and conv-rol of

linear dynamic systems. Within the context

of discrete time systems, multi-input,

multi-output, nth-order linear systems are

discussed, and general results for optimal

estimation, optimal control, and other

topics are presented. The application of

these results to modeling human behavior is

considered with special emphasis on

man-machine system models.

A-368
Sauer, D.W., Campbell, W.B., Potter, N.R. 4 Askren, W.B. Human resource

factors and performance reletionships in nuclear missile handling tasks

(AFHRL-TR-76-85, AFWL-TR-76-301). Dayton,

Inc., 1977. (AD-A042 601).

OH: Systems Research Laboratories,

! Topics Relevant

! to System Development

land Evaluation Technology

i.

2.

3-

State of the Art levlew

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

?.1 System Definition

2.C Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.U Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

U.1 Analytic Methods

H.2 Parameter Determinations

4.3 Apparatus for Testing

IM Personnel for Testing

1.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.1 Conclusions and

Recomnendati ons

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

2.2 The primary goal of this study was to

assess, through quantification, the

relationships between human resource

factors and nuclear system sefety, and to

use these data to simulate nuclear system

maintainance operations.

3.1 Thirteen factors considered relevant to

nuclear safety maintenance operations were

identified: motivation, group cohesiveness, behavioral/emotional stability,

fatigue, leadership, organizational

structure, task complexity, written

manuals, job skill proficiency aptitude,

training, work experience, and environmental conditions.

3.2 Each of these thirteen factors were

quantified for computer simulation use;

measures of quantification included

psychological tests, supervisor ratings,

and biographic data. These were

consolidated into one questionnaire and

administered to each missile technician.

M.I Data were analyzed through a computer

simulation model. The Short-Range Attack

Missile (SRAM) network was fed into the

computer with each task assigned four

descriptors: time, crew type, hazard

value, and maintenance task category.

Equations describing functions between the

h"man resource factors, time, and hazard

were correlated by the System Analysis of

Integrated Networks of Tasks (SAINT)

computer simulation model and cycled about

A-369
«

Topics Relevant i

! to System Development I Topic

land Evaluation Technology ! No.

500 times to approximate how a crew

possessing given characteristics of

motivation, work experience, fatigue, a.id

training under given environmental

conditions would perform.

4.2 Test parameters were described by the

and maintenance and ground handling tasks

U.3 involving the SRAM system. Tasks were

described by time required for completion,

career field involved in the task, degree

of hazard inherent in the task, and task

classification by either transport,

checkout, or assemble/disassemble.

5.3 Results of supervisor rankings of technicic?ns indicated that those techniciars who

had more experience, higher aptitude

levels, were more stable emotionally, had

less fatigue, and greater morale were more

satisfied with their jobs. The above

factors were also positively correlated

with accuracy and task performance.

A-370
Schaeffer, K.H., Fink, J.B., Rappaport, M., Wainstein, L. 4 Erickson, C.J. The

knowledgeable analyst; An approach to structuring man-machine systems (SRI

No. IMU-35M6). Henlo Park, CA: Stanford Research Institute, February 1963.

(AD-297 «32).

Topics Relevant

I to System Development

!and Evaluation Technology ABSTRACT

1. State of the Art Review

of the Process

1.1 General Systea

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.X Performance

Criteria, Specific

3.5 Measurement Procedures

4. Planning Components

of the Process

4.1

4.2

X.3

«.X

X.5

1.1 The objectives of syaLems analysis are to

analyze specific systems in order to solve

predesignated problems. These objectives

are at variance with those of the empirical

sciences, which attempt to discover general

laws. Since the two objectives differ, the

method of systems analysis differs from the

method of science. In an attempt to evolve

a general method for systems analysis, the

matrix-network approach for the analysis of

complex man-machine systems is presented.

This approach consists of seven steps which

show how a system can be structured and how

mathematical models of systems aspects can

be incorporated into the over-all analysis.

However, some of these steps involve,

besides formal rules, the judgment of a

knowledgeable analyst. To delve deeper

into this judgment function, various

logical, methodological, and psychological

aspects concerning this function are

discussed by different authors. On the

basis of these discussions the principal

author develops requirements which must be

met by successful approaches to the

structuring of complex systems.

Analytic Methods

Parameter Determinations

Apparatus for Testing

Personnel for Testing

Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.X Conclusions and

Recoamendations

Further Research Areas

6.1 Meisurement System

Limltallons

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-371
Self, H.C. Performance measures, observer selection, and reconnaisaance/strike

effectiveness (AMRL-TR-72-86). Wright-Patterson AFB, OH: Aerospace Medical

Research Laboratory, November 1972. (AD-770 6M7).

Topics Relevant

I to System Development

land Evaluation Technology ABSTRACT

1. State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPU)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3- Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1) Performance

Criteria, Specific

3.5 Measurement Procedures

4. Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

5- Application Components

Of the Process

5-1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Reconniendations

1.1 In a reconnaissance strike or reconnaissance/strike system there is a complex of

equipment and men. One would assume that

maximum effectiveness of the man-machine

system would be obtained by utilizing the

best men and the best machines; cnis

reasoning, however, conflicts with problems

of expenses, unavailability, and the inability to determine the best men for the job.

Characteristics of the men, the equipment,

and the mission interact such that

variations in equipment and/or situation

will have an effect on the men in the

system. In other words, a system approach

is essential when dealing with a system.

3.1 The performance measures in this study were

percent detection, accuracy, and reaction

time or slant range at detection.

3.3 The ideal observer would detect all of the

targets (tD 100}), mistake no targets

(accuracy = 100%), and would detect and

recognize targets the instant that their

images appeared on the display (reaction

time = 0 seconds).

3.U Mission-relevant performance criteria

Included the following:

(1) The number and percentage of targets

that are detected. An ideal observer

would detect all targets that were

displayed with some minimum image

quality.

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-372
I Topics Relevant

I to System Development jTopic

!and Evaluation Technology ! No. ABSTRACT

(2) An ideal observer would make no

mistakes in designating targets as

such.

(3) Targets should be adequately

recognizable at long slant range.

'

A-373
Shapero, A. & Bates, C., Jr. A method for performing human engineering

analysis of weapon systems (WADC Tech. Rep. 59-784). Wright-Patterson AFB,

OH: Wright Air Development Center, Aerospace Medical Laboratory, September

1959.

Topics Relevant

I to System Development

land Evaluation Technology ABSTRACT

1. State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1* General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3. «I Performance

Criteria, Specific

3.5 Measurement Procedures

*. Planning Components

of the Process

4.1 Analytic Methods

«1.2 Parameter Determinations

«(.3 Apparatus for Testing

«.«» Personnel for Testing

«1.5 Test Plans

5. Application Components

of the Process

5.1 Test Execution

5.2 Data analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recooaendttions

6. Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.3 A weapon System Analysis and Integration

Model (SAIM) has been developed that includes the system's human elements and

that can be employed as an aid in the

analysis, synthesis, evaluation, planning,

and management control of weapon systems.

A diagrammatic matrix is used to structure

the model. The system components (subsystems) are represented in the same way as

the system. The matrix can be described

as a two-dimensional square. The headings

of the rows and columns are symmetrical

and consist of the system elements. The

orientation of the matrix is such that

the elements appearing as headings of rows

are considered as affecting those appearing

as headings of columns. The matrix includes

system determinants, system and subsystem

components, and system integration.

SAIM is designed to indicate interactions

and to provide a means for describing the

interactions at a given time. The effects

of developmental changes that occur can be

taken into account by updating the matrix.

2.1 The SAIM is a descriptive matrix model that

classifies the elements of a weapon system

into those determining the nature and form

of the system, those comprising the parts

of the system, and those integrating the

parts of the system. The approach used in

SAIM predicates that any system and any of

its subsystems have the same kinds of conceptual elements and SAIM employs a scheme

for classifying these elements that is

applicable to any system or any level.

A-374
Topics Relevant I

to System Development I Topic

and Evaluation Technology ! No. ABSTRACT

In SAIM, system components are described

in terms of mechanism modules, human

operator modules and facility modules.

This modular treatment affords flexibility

since it allows each subsystem to be

handled in a building-block fashion, thus

making it easy to describe complex functions that combine many items, and to

extend or to modify the model as the system

itself is modified or extended. System

integrators are the elements that link the

system's components in a "nonhardware" or

abstract way.

2.2 In this model, the mission statement is

always framed in general terms, enabling it

to remain unchanged while permitting the

incorporation of changes in performance

requirements as imposed by technical, budget

or other changes in the weapon system.

2.^4 Limits on the system are imposed by the

state-of-the-art, by nature, and by the

agencies that have cognizance over the

system. The constraints imposed by nature

or society relate to the physical environment and/or human and material resources.

Constraints imposed by cognizant agencies

include those that relate to funds and

developmental time.

2.5 The performance requirements detail the

system's mission into a set of goals, standards and objectives. Performance requirements are usually categorized into operational

and support requirements. Operational requirements might include such factors as kill

probability, range, speed, etc. Support

requirements could include readiness, maintenance and servicing, and handling.

A-375
Sheldon, M.S. & Zagorsk!, H.J. Man-machine system evaluation - The Normative

Operations Reporting Method. In R.C. Blanchard & D.H. Harris (Eds.),

Man-machine effectiveness analysis. Papers presented at The Human Factors

Symposium, University of California at Los Angeles, June 15. 1967.

(AD-735 718).

Topics Relevant I

! to System Development !Topic

land Evaluation Technology ! No.

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Coaponer.ts

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environnent Definition

2.« General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.i< Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

a 3 Apparatus for Testing

K 1 Personnel for Testing

4 b Test Plans

Application Components

of the Process

5 1 Test Execution

5 2 Data Analysis

5 3 Findings Interpretation

5 4 Conclusions and

RecomDendations

Further Research Areas

6 1 Measurement System

Limitations

6 2 Research Potentials/

Priorities

6 3 Research Planning

1.1 It is increasingly evident that man-machine

system evaluation needs techniques that are

radically different from traditional

methods. The authors call this work area

systemetrics. The overall purpose of the

systemetric model is to develop a

methodology permitting evaluation of

man-machine performance to be based on a

series of flexible standards reflecting the

difficulty of the mission, in direct

contradistinction to the absolute standards

approach. The systemetric approach

requires intimate familiarity with the

system.

1.3 One example of the kind of work that can be

done is the Normative Operations Reporting

Method (NORM). This method includes no

exotic breakthrough; its creativity lies in

its unique combination and application of

assessment techniques that are well known.

The NORM methodology features a set of

flexible standards that are adjusted

according to the relative difficulty of the

mission.

2.1 NORM was applied for field evaluation of

the Semi-Automatic Ground Environment

(SAGE) system. In dealing with any

man-machine system, one must ask, "What is

the system trying to accomplish?" and

"What available data will adequately

reflect system performance?" Answers to

these questions will lead to development of

suitable criterion measures. The quality

of the criteria is the single most

important element in determining the

A-376
.V

Topics Relevant

I to System Development I Topic I

land Evaluation Technology i No. I ABSTRACT

meaningfulness of the evaluation. The

overall objective of the SAGE system was

stated as "neutralize as many invaders as

possible a: quickly as possible as far out

as possible.**

2.2 Once the criterion measures have been

defined, the developers of the systemetric

model must try to account for the portion

of performance variance that is

attributable to mission difficulty. To do

this, one must determine the mission

characteristics that are most likely to

contribute to mission difficulty. For the

SAGE system, the pertinent groups of

mission difficulty variables were found to

be (1) radar variables, (2) invader

variables, and (3) operational environment

variables.

3.2 The SAGE system overall objective was

translated into three quantifiable

criterion measures: (1) percentage fakers

killed, (2) faker life time in system's air

space, (3) depth of penetration.

The above three measures were supplemented

by other measures concerning explicit

system functions, including (1) detection

latency, (2) unassociated time

non-conformance of display symbology and

raw video, (3) interception time, and CO

tactical action latency.

M.I To develop more comprehensive criteria of

effectiveness, the performance measues are

factor-analyzed using the principal

components methods. The first two factors

that emerge are termed "Weapon Performance"

(defined by the measures: Tactical Action

Latency; Interception Time; Depth of

Penetration) and "Air Surveillance

Performance" (defined by five different

measures of air surveillance). These two

factors account for about 76% of the

observed variation in performance. They

•re more reliable than any of the

individual measures, and they have

intrinsic face validity.

A-377

')

I

L
I Topics Relevant

\ to System Development [Topici

land Evaluation Technology ! Wo. I ABSTRACT

n

r

5.1 The performance measures are collected from

simulated air defense missions performed at

SAGE direction centers. The data are

obtained from operational recording tapes

containing a history of all relevant

activities taking place during a mission.

5.2 Initial statistical procedures in NORM

focus on basic correlations between each

criterion measure and each mission

difficulty variable, and on the relative

independence of the candidate predictor

variables. Following this, multiple

regressions are run for each measure using

selected sets of mission difficulty

variables as independent variables.

Overall, about 501 of the variance of

criterion performance is accounted by the

presently available mission difficulty

variables.

5.3 The criterion research in SAGE has resulted

in relevant, quantifiable measures of

system and functional performance. The

creation of these measures has led to a

meaningful procedure for evaluating

man-machine performance at the direction

center level.

A-378
Siegel, A.I. Human performance reliability—Its measurement and impact on

system reliability. ASME Publication. 78-DE-17, New York: The American

Society of Mechanical Engineers, 1973.

Topics Relevant |

I to System Development I Topic

land Evaluation Technology i No. ABSTRACT

1. State of the Art Heview 1 •1

of th« Process

1.1 General SystM

Measureaenls

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environaent Definition

2.1 General Constraints

2.5 Perforaance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3- Analytic Components

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3-3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

4. Planning Components

of the Process

1.1 Analytic Methods

1.2 Parameter Determinations

1.3 Apparatus for Testing

1.1 Personnel for Testing

1.5 Test Plans

This paper discusses methods for measuring

human performance reliability and methods

for integrating human performance reliability with equipment reliability to derive a

measure of total system reliability.

Emphasis was placed on a computer simulation model. The data presented show the

model to possess validity and utility.

The model in its current form is basically

a sequential processor which incorporates

human, equipment, and environmental factors. Three categories of independent

variables are considered:

(1) Personnel (quantity, category types,

goals, physical properties, performance attributes).

(2) Mission (composition, duration,

environment, elements/tasks).

(3) Equipment (quantity, capability,

performance/status).

The dependent variables are the Output

Measurements (mission effectiveness, time

utilization, personnel, and report

frequency).

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recooaendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-379
Siegel, A.I., and Federman, P.J. Development of performance evaluative

measures; Investigation into and application of a fleet post trai^ng

performance evaluation system. (Rep. No. 7071-2). Wayne, PA: Applied

Psychological Service, Inc., September. 1970. (AD-713 192).

I Topics Relevant I I

! to System Development {Topic

!and Evaluation Technology ! No. ABSTRACT

Stat« of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

0.2 Parameter Determinations

1.3 Apparatus for Testing

*.1 Personnel for Testing

1.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.1 Conclusions and

Recomendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.3 The purpose of this study was to develop

and demonstrate an economical and practical

method for measuring technical proficiency.

2.1 Of particular interest was the readiness of

Naval electronically-oriented technical

personnel (reserve and regular) for

completing their assigned mission.

3.1 The study attempted to answer such

questions as: What is the current level of

effectiveness of maintenance personnel in a

given rating, ship or squadron; why is it

low or high; what specific job skills need

improvement? Also, an attempt was made to

determine the need for additional training

and to compare the effectiveness levels

between various ratings, ships, and

squadrons.

3.2 The method relied largely on a personnel

reliability index modeled after an

equipment reliability index. Specifically,

the personnel reliability index was based

on the compounding of probability of

successful performance values for each of

eight factorially derived electronic Job

dimensions. A second instrument was also

administered. This instrument was based on

a Guttman-scaled checklist and yields an

absolute measure of performance.

A-380
I Topics Relevant

I to System Development I Topic I

land Evaluation Technology ! No. I ABSTRACT

5.4 It was held that the methods and techniques

employed a series of psychometric and

measurement criteria. Statistically

significant differences were evidenced

among the derived indices for the Naval

ratings and job factors involved but not

among the ships and squadrons sampled.

6.2 The results and experience of the study

were interpreted as indicating that the

methods and data treatment techniques

employed possess sufficient merit for their

purpose to warrant trial use on a larger

scale.

A-381

.' --.
Siegel, A.I., Pfeiffer, M.G. & Worms, T.M. Development and evaluation of a

content analytic approach In Army field system data organization (ARI Tech,

Rep. TR-77-A11). Wayne, PA: Applied Psychological Services, Inc., August

1977. (AD-A042 075).

Topics Relevant I

I to System Development {Topic

land Evaluation Technology ! No. ABSTRACT

1. State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.<4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria. Ultimate

3- Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

4. Planning Components

of the Process

4.1 Analytic Methods

I.I Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Rec ommenda 11ons

6. Further Research Areas

0.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.2 The logic, methods, and results of a study

into the derivation of a content analytic

approach to Army field system data organization are described. The first steps of the

program involved a linguistic analysis of a

set of battlefield messages and multidimensional scaling analysis of a sample of

messages representing all information

classes yielded by the linguistic analysis.

Four multidimensional scaling analyses were

completed. Each was based on the linguistic similarity perceptions of one of four

experienced battlefield analysts. High

agreement was found among the factorial

structures yielded by the data of each

battlefield analyst. Accordingly, an overall analysis was completed and 15 factors

were identified as representing the perceptual substrate of the Army field information linguistic system. On the basis of

the derived factors, a battlefield language

taxonomy was developed. The taxonomy was

tested in two separate field oriented experiments.

The results of these experiments indicated

that intelligence analysts can classify

messages reliably within the taxonomy and

that they can reliably use the taxonomy for

inquiry purposes. Moreover, adequate

subjective confidence was expressed by the

analysts in the use of the system.

Finally, a computer system for automatic

classification of battlefield messages is

presented.

A-382
Siegel, A.I., Platzer, H.L. 4 Lanterman, R.S. Techniques for evaluating

operator loading in man-machine systems. Wayne, PA: Applied Psychological

Services, Inc., March 1967. {AD-810 787).

Topics Relevant I

I to System Development I Topic

land Evaluation Technology ! No. ABSTRACT

1. State of th« Art Review

of th« Process

1.1 Cenerel System

Neasureaents

1.2 Systen Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1) General Constraints

2.5 Performance

Requirements, Ultimai*

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.H Performance

Criteria, Specific

3.5 Measurement Procedures

1.1

2.1

3.1

Planning Components

of the Process

1.1 Analytic Methods

4.2 Parameter Determinations

3.2

«.1

1.3 Apparatus for Testing

«.« Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.« Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potential«/

friorltlea

6.3 Research Planning

This study Investigated the utility of the

human transfer function as an Indicator of

operator status.

An analysis of previously acquired data

from an experiment into the human transfer

function was conducted. In that

experiment, three subjects performed a

sequence of tasks, including a compensatory

tracking task, that kept them occupied

overnight. The results of the study were

not in accordance with the pre-experlmental

hypothesis, in that the two indicators

derived were not found to decrease

uniformly as the period of sleep

deprivation increased. The present set of

studies expanded on these previous data by

presenting a reanalysis of the previous

data and by describing the rationale,

method, and results of a new study into the

effects of operator overload on the

transfer functions.

The study addressed operator loading in

man-machine systems.

Measurements of tracking records were

obtained.

These measurements were made to the nearest

quarter millimeter.

Data for three minutes of each selected

hourly sample were transcribed at 0.10

second intervals. This yielded 1800 data

points for each input and 1800 for each

output position signal sample. The first,

second, fourth and sixth hours of tracking

A-383
I Topics Relevant

I to System Development

and Evaluation Technology

i i

ITopicl

I No. i ABSTRACT

a

. •

performance were selected for analysis.

Frequency analysis was employed in data

analysis.

4.3 An oscilloscope display and a

tracking-control were used in this

experiment. Programming and recording

equipment were also utilized.

4.4 Four male college students were subjects in

this study. They were paid for their

participation.

4.5 The experimental subjects tracked

continuously for eight hours between 8 am

and 4 pm. No breaks were allowed. Samples

of their performance were recorded for the

last five minutes of each hour. The

subjects were unaware how much, if any of

their performance was being recorded.

Transcription of the data involved

measuring the displacement of the input and

output signals as recorded on the ink

writing oscillograph.

5.3 The results of this study were interpreted

to suggest that the spectral analytic

technique possesses considerable potential

as an objective, behavioral, diagnostic

method for on-line assessment of operator

status in man-machine systems involving

perceptual motor behavior.

'

A-384
Siegel, A.I., Schultz, D.G. & Federoan, P. Post-training performance criterion

development and application; A matrix method for the evaluation of training.

Wayne, PA: Applied Psychological Services, Inc., January 1961. (AD-251 082).

Topics Relevant

I to System Development

land Evaluation Technology

State of th« Art Review 1. 3

of the Process

1.1 General System

Neasurenents

1.2 System Taxonomy 2. i

Model (STM)

1.3 Overall Conceptual

Process Model (CPM) 2. 2

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition 3 1

2.k General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

S.il Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

3.2

3.3

3.U

This report describes a matrix method for

the evaluation of training.

Training programs in general were the

object of the evaluation.

The mission of the training program was

described as the preparation of students

for the jobs performed after training.

"Suitability" for the job was used as a

basis for training evaluation. "Suitability" was defined as the training graduates'

ability to do the tasks involved in the

job.

Matrix solutions were described which yield

three indices, each reflecting a different

aspect of the comparison between the skills

of the trained man and the job's requirements.

The training program is deemed effective if

the graduate carries out the duties of his

job proficiently.

Specific measures of job performance can

indicate the effectiveness of a training

program.

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Pr'orlties

6.3 Research Planning

A-385
I Topics Relevant

! to System Development {Topic

land Evaluation Technology ! No. ! ABSTRACT

ti

3.5 In this study a specific evaluative scheme

was developed for summarizing suitability

and the application of the scheme to data

previously collected on technicians in four

Naval ratings. The method for accomplishing these tasks is the Technical

Behavior Checklist (TBCL). These checklists are comprehensive detailed compilations of the tasks performed.

4.1 Various responses to the TBCL by the

supervisor indicating the level of

proficiency of the ratee were assigned

different score values. The average

proficiency level of each task was computed

- - determined by the mean of these scores.

An index of importance of each task was

developed from the TBCL and the supervisor.

These importance ratings were weighted and

the task's importance for classification in

the matrix was a combination of its

frequency of performance and its

criticality to the mission. From this

information three indices were developed.

(1) The training index; (2) an "overtraining" index; (3) an "undertraining"

index. The training index reflects how

closely the average graduate fits the model

of high proficiency on relatively important

tasks and low proficiency on relatively

unimportant tasks. The overtraining index

indicated the extent to which graduates

were highly proficient even on unimportant

tasks. Important tasks that technicians

performed with relatively low proficiency

constituted the undertraining index.

U.I The subjects of this study were:

(1) Aviation Machinist Mates (Turbo Jet)

(2) Air Controlman (Tower)

(3) Parachute Rigger (Survivalman)

(4) Aviation Electronics Technician (Radar)

"

A-386
i Topics Relevant

i to System Development I Topic

!and Evaluation Technology ! No.

M.5 In a previous study a Technical Behavior

Checklist was developed for four naval

ratings. This checklist was a detailed

comprehensive checklist of the tasks

performed in that rating. For this study a

supervisor was asked to indicate the

proficiency level of the man he was rating

in terms of how much supervision and the

number of checkoi;fs required.

5.3 The technicians in the four ratings studied

were found generally to be suitably

trained.

5.4 The matrix method is feasible and simple to

use provided task proficiency measures and

an index of task importance are available.

The use of this method provides meaningful

results. Its use should provide an

increased understanding of the effects of

training and is one basis for changing

training emphasis.

A-387

- .-•.

.-

.-... .. . . . . .•

. - . - • - .

. • ":-. MA^
Siegel, A.I. & Wolf, J. J. Computer simulation: Savior, sanctuary, or

silliness? In 0. W. Levy (Ed.), Sympoaium on applied model of man-maohine

systems performance (NR69H-591). Columbus, OH: North American Aviation,

"November 1968. (AD-697 939).

Topics Relevant

! to System Development

land Evaluation Technology ABSTRACT

5.

State of the Art Review 1*3

of the Process

1.1 General System

Measurenents

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPU)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.'4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

I. It Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

«.1 Analytic Methods

1.2 Parameter Determinations

4.3 Apparatus for Testing

1.4 Personnel for Testing

1.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5-3 Findings Interpretation

5.'I Conclusions and

Recommendations

The paper discusses advantages and

disadvantages of stochastically oriented

digital computer simulation models and

their applications to predicting the

effects of th** human on the output of

complex systems. The advantages of

simulation modeling are given as follows:

(1) It necessitates a rigorous,

insight-providing analysis of the

human and machine elements and their

interactions.

(2) It provides a means of representing

the complex human organism to a

degreee of realism not available

through deterministic mathematical

formulations.

(3) It can cost less than either physical

simulation or laboratory

experimentation.

(U) It allows the testing of hypothetical

systems and of hypothetical operating

procedures.

(5) It can show the effects of

interactions on system function, e.g.,

the interaction between operator skill

and workday length.

(6) It can be useful in establishing

training requirements and areas for

training emphasis.

Further Research Areas

6.1 Measurement Systen

Liaitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-388

. . -^ -J. .... ifaeAMAateteAeteM . ■■-
»

I Topics Relevant

I to System Development I Topic I

land Evaluation Technology ! No. I ABSTRACT

v V,

(7) It enables us to deal with a complexity that cannot be managed by other

techniques.

It is also pointed out that there are

adverse aspects to digital simulation which

could serve to limit its potential. These

include:

(1) There is a tendency to confuse the

model on which the simulation is based

with a theory, and hence to employ

inappropriate standards and criteria

to judge the model and the simulation.

(2) Digital modeling depends on the data

and methods of a number of

disciplines, and this can lead to

complexity and confusion.

(3) The necessary data may be unavailable

or of poor quality (garbage in,

garbage out).

(4) There are equally damaging but

opposite points of view that impede

serious research by computer

simulation; one is that attempts to

simulate man via a computer are

antireligious or at least degrading;

the other is that a computer's results

are automatically sacred and of

unquestionable precision.

(5) The argument exists that behavioral

science does not possess powerful

enough theory for model building.

5.1 It is concluded that digital simulation is

an emergency tool for the behavioral

sciences.

A-389

^ .- ."-. • - - - m—m

-. Mü* . •I»iV>V'»aba.
Siegel, A.I., Wolf, J.J.4 Lanterman, M.R. A model for predicting Integrated

man-machine system reliability: Model logic and description. Wayne, PA:

Applied Psychological Services, Inc., November 197^. (AD-A009 81M).

I Topics Relevant

! to System Development

!and Evaluation Technology

Topic

No. ABSTRACT

State tf the irt Review

of the IMMM

1.1 Genet il Sytetc

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Coaponents

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.U General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Anal ytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

1.4 Performance

Criteria, Specific

3-5 Measurement Procedures

Planning Coaponents

of the Process

1.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

1.1 The integration of Human Reliability (HR)

and Equipment Reliability (ER) data into a

single comprehensive model for predicting

System Reliability (SR) has been one of the

major concerns of Navy system planners.

1.3 The present study attempted to:

(1) Extend and strengthen a previously

developed model for simulating the

acts and behaviors of the operators of

an intermediate size system to include

a greater number of options to the

model user.

(2) Evolve the model into one which

produces reliability oriented metrics

for both humans and equipment on both

an event and overall system level.

(3) Conduct an initial series of

sensitivity tests relating the new

variables and parameters to those

already present in the model.

3.1 The variables selected for measurement in

the present document included:

(1) Physical capability -working pace

(2) Competence - level of

Application Components aspiration

of the Process

5.1 Test Execution

5.2 Data Analysis (3) Fatigue - psychological

5.3 Findings Interpretation stress

5.4 Conclusions and

Recoonendations

(4) Physical a confidence

Further Research Areas

6.1 Measurement System incompatibility

Limitations

6.2 Research Potentials/ H, 1 Data were analyzed, in part. via the mean

Priorities

6.3 Research Planning and standard deviation

A-390

_ ' »^ •».".• »J" «^

- - - - - I - -

. <-.
I»..

I".

Simon, C.W. Advanced methodologies study program (Final Rep.),

CA: Hughes Aircraft Company, June 1975. (AD-A021 099).

Culver City,

I Topics Relevant

I to System Development

land Evaluation Technology

1. Stat« of the Art Review 1 • 3

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.« General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3- Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.'* Performance

Criteria, Specific

3-5 Measurement Procedures

U. Planning Components

of the Process

K.I Analytic Method.«

4.2 Parameter Determinations

4.3 Apparatus for Testing

• .I) Personnel for Testing

4.5 Test Plans

5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recomnendatlons

6. Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potential«/

Priorities

6.3 Research Planning

Many of the problems associated with the

low productivity and high costs of human

factors engineering research could be

resolved if more variables were studied in

a single experiment, particularly if these

variables were the ones having the greatest

effect on the performance of the task under

investigation. Acceptance of this

conclusion resulted in a formulation of a

two-stage approach to experimentation:

first, to identify the most important

variables and second, to relate them

functionally to performance. A suitable

screening methodology is selected for this

process with blocking techniques useful

both as a means of economizing when

collecting data and of reducing scores of

irrelevant variance that cause shift in

performance as a function of time.

A-391

:«.-

•^/^v.v^.v.'- ..-/ -:..v.--•;■■•. .-:v:v;v: ^-•- * - -' ^ - - - •-'- - --
Simon, D.E. et al. Standard procedures for Air Force Operational Test and

Evaluation (RADC-TR-74-270, Final Rep., Vol. 1). Albuquerque, NM:

Braddock, Dunn & McDonald, Inc., October ^9^^. (AD-BOOG 365).

Topics Relevant

I to System Development

land Evaluation Technology

State of th« Art Review 1 .3

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environaent Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.4 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

1.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoonendatlons

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

The report provides an overall skeletal

structure of the Operational Test and

Evaluation (OT4E) process. From the

structure, a step-by-step procedure is

derived. Steps can be summarized as

follows:

(1) Review of documentation.

(2) Formulation of test objectives.

(3) Selection of applicable test concept

(I) Measures of effectiveness (MOE).

(5) Test design.

(6) Simulations.

(7) Data.

(8) Range instrumentation.

(9) Test plan.

(10) Conduct of test.

(II) Data analysis.

(12) Conclusions and recommendations.

(13) Test report.

A-392

.■■■/-.•

___
Topics Relevant

I to System Development iTopic I

land Evaluation Technology ! No. ! ABSTRACT

2.1 Review of documentation relevant to a given

system to be tested is important because

the test managers need to have a clear

understanding of the critical issues, data

requirements, and test objectives, and reed

to be familiar with the system, how it

operates and its history in previous tests.

2.1 Selection of a test concept is a trade-off

between the magnitude and depth of testing

and the available resources. Factors

affecting the trade-off may include:

(1) Time.

(2) Availability of the (test) range.

(3) Availability of personnel.

(U) Availability of equipment.

(5) Risk of overrunning time due to test

complexity.

(6) Data quality and quantity.

(7) Cost.

3.1 0T4E objectives are formulated to answer

critical questions and to provide a basis

for making decisions affecting development,

production, support, employment of a system.

Typical OT4E objectives are:

(1) To observe the degradation.

(2) To collect data on...

(3) To evaluate ability to meet the

requirement...

3.2 Measure of effectiveness of an item is a

parameter which evalates the extent of the

adequacy of the item to accomplish an

intended mission under specific conditions.

An MOE is a function of Availability (A),

A-393
I Topics Relevant

I to System Development

land Evaluation Technology ABSTRACT

Dependability (D), and Capability (C).

MOE's are expressed as probabilities since

A, D, C are probabilities. Good MOE's:

(1) Should be sensitive to all variables

affecting the item.

(2) Should be precisely defined.

(3) Should not be overly broad.

iH) Should be mutually exclusive.

(5) Should have exhaustive inputs.

(6) Should be relevant to the mission.

(7) Should have inputs relevant to the

design issues.

(8) Should be expressed in terms

meaningful to the decision maker.

(9) Should have inputs that are measurable.

(U) And its inputs should be quantifiable

it at all possible.

A-394

.» - > ... . ■■■ - - - Ml...!.

-- ~-* - - *-• -• - —•- •-•-
,-.

Sjogre.-i. D.D. Measurement techniques in evaluation.

Research, (undated), MO(2), 301-320.

Review of Educational

I Topics Relevant

i to System Development

land Evaluation Technology

i. State of the Art Review

of the Process

1.1 General System

Heasureaents

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environaent Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Rec oonenda t i ons

6. Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

5.

1.1 This article deals with educational

evaluation and the implementation of an

input-process-outcome evaluation plan. For

many years evaluation has been concerned

with determining whether specified

objectives were attained but current

evaluation models focus on a larger number

of phenomena. Therefore evaluation

theorists Indicate that evaluation should

attend to outcomes other than specified

objectives, to inputs or antecedent

conditions, and to processes or

transactions.

A-395

> ^ > . . . ^-.-.:.——-.. ... _._..._ ,

.
Smith, R.L., Westland, R.A. & Blanchard, R.E. Technique for Establishing

Personnel Performance Standards-TEPPS (PTB-70-5, Vol. 1). Santa Monica, CA;

Integrated Sciences Corp., December 1969. (AD-704 103).

I Topics Relevant !

I to System Development iTopic

land Evaluation Technology ! No. ABSTRACT

State of the Art Review

of the Process

1.1 General System

HeasureMnts

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.H General Constraints

2.5 Perforaance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

i.lt Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Rec oomenda 11 ons

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 The cost and complexity of modern military

systems have given rise to the development

of a technology that is referred to as

system and cost effectiveness analysis.

Until recently, however, only limited

effort has been directed toward the

specification of the contribution made by

system personnel as well as human error in

a system's overall effectiveness.

TEPPS (Technique for Establishing Personnel

Performance Standards) was designed to meet

two primary objectives:

(1) To derive specific personnel

performance standards with definable

relations to system effectiveness

requirements.

(2) To determine the Influence on system

effectiveness of performance levels

that deviate from established

performance standards.

2.1 A system was a set of personnel-equipment

functional units whose collective purpose

is to achieve a particular goal. Differentiating between systems and subsystems is

arbitrarily based since most systems can be

defined as subsystems when referenced to

larger overall systems of which they are a

part. What is important is the relationship of a given system's goals with respect

to those of another. Thus, the effectiveness of a given system should be evaluated

with respect to the parent system. A

system can be assumed to have been 1001

effective if it performed up to its maximum

capability, regardless of

A-396

- ^' - ^ • - -' ----- - - •-•••
I Topics Relevant I

I to System Development 1 Topic I

land Evaluation Technology i No. I ABSTRACT

'

whether or not it was subsequently destroyed. Thus capability is an important

factor contributing to establishing system

effectiveness requirements and the evaluation of system effectiveness.

I

I -

,

A-397

--'•-•' iii'tV i J- i . i m • - - - AM - • • .-.. ^ ^-. -• - - -- ^
Smith, R.L., Westland, H.A. & Blanchard, R.E. Technique for Establishing

Personnel Performance Standards-TEPPS (PTB-70-5, Vol 2). Santa Monica, CA:

Integrated Sciences Corp., December 1969. (AD-70U 104).

I . Topics Relevant |

I to System Development I Topic

land Evaluation Technology ! No. ! ABSTRACT

.V 1. State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 rnvironaent Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1) Performance

Criteria, Specific

3.5 Measurement Procedures

4. Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

6. Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 Through the years, technological advances

have provided military systems with

increased capability. Planners have

subsequently devised techniques or tools to

resolve problems more quickly; however,

many such tools are individualistic in that

they have been independently developed and

employed by individuals through their own

unique experiences. For that reason TEPPS

(Technique for Establishing Personnel

Performance Standards) was developed.

Overall, TEPPS is a set of procedures for

gathering information about a system and

for developing a qualitative and/or

quantitative model of that system. The

utility and accuracy of TEPPS depends on

such information as system descriptive

data, operational requirements data, and

human capability data.

1.3 A graphic representation of the major steps

in applying TEPPS is presented in Figure 1.

2.1 The system employed was an Anti-Air Warfare

Combat Information Center (CIC) typically

employed on a picket ship involved in fleet

defense. The CIC system is composed of

three functions:

(1) The gathering and display of target

information.

(2) Actions of the command officer

(evaluator) on that information.

(3) The Air Intercept Controller's (AIC)

vectoring of a Combat Air Patrol (CAP)

fighter.

A-398

: - —•-•-

•.'.•« - *

—-

.' M
. -

t

i .'

Collect

system data 1

\

i

Probabilities

and time

Prepare estimates

GSSM

i

) i

Develop 1

IOTA'S \-

Computer

program

f 1

Perform qualitative

and/or semiqualitative

analyses

Construct

MSSM

i \

Allocate

standardsl

Computer

program

i

Perform

quantitative

analyses

. ;

Figure 1. Representation of Major Steps

in Applying TEPPS

A-399

-*• •** •». .*•' .*■■ -"> ,*• • h*« *-' *•''. ■■" -'t-" •»* -*

-• ' -' - -' - •--• ,-.-..-. v,v,.--.
Snode, A.E., Gruber, A. & Ely, J.H. The measurement of advanced flight

vehicle crew proficiency in synthetic ground environments (Tech. Rep. MRLTDR-62-2). Darien, CT: Dunlap and Associates, Inc., February 1962.

i Topics Relevant i i

I to System Development ! Topic I

!and Evaluation Technology ! No. i ABSTRACT

1. State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environnent Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3-5 Measurement Procedures

I. Planning Components

of the Process

4.1 Analytic Methods

H.2 Parameter Determinations

1.3 Apparatus for Testing

M.l Personnel for Testing

1.5 Test Plans

5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.1 Conclusions and

Rec onxnenda t i ons

6. Further Research Areas

6.1 Measurement System

Li ml ta1. Ions

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 The purpose of this report is to present

a systematic treatment of the major considerations in the measurement of advanced

flight vehicle crew proficiency in synthetic ground environments.

Measurement in the training context serves

two main functions. One is prediction of

how well an individual or team can be expected to perform under some specified conditions anticipated in the future. The

second is measurement of present knowledge

or level of proficiency in some area or

task.

2.2 It is always a rfifficult problem to define

mission purpose. Recognizing that a precise definition will seldom or ever be

spelled out, some statement of mission

purpose must nevertheless be available.

Such a statement represents a starting point

insofar as determining the criteria for

measurement and for suggesting what specific

tasks are most critical.

2.6 Seldom can one obtain direct measures of the

ultimate criteria for a system. It is

usually necessary to select some actual

criterion which in practice is an approximation of the ultimate one. In general, a

good criterion is one that is both reliable

and relevant. Reliability implies that

proficient or successful performance will

not vary widely because of chance factors.

Relevancy refers to the validity of the

actual criteria. The actual criterion is

relevant to the extent that it approximates

the ultimate criterion.

A-400

. ••

- - -

• .•»'-••.'..••". .

ta*^i
j Topics Relevant

1 to System Development

land Evaluation Technology

i i

I Topic i

'. No. i ABSTRACT

The multidimensional nature of proficient

performance will often require that several ;riteria, all of which are relevant

for a particular activity, be used. In

such cases, it may be desirable to combine

these several criteria into a single comprehensive one. Combining subcriteria into

a single composite will usually involve

assigning relative weights to the individual

criterion. These weights must often be

determined on the basis of expert opinion

since it is seldom possible to determine

statistically the intercorrelation or overlap between subcriteria. A measured relationship between the actual and ultimate

criteria is even more rare. Subcriteria

should be weighted according to their relevance, and those which overlap factors in

another subcriteria should receive a low

weight. Subcriteria which are more reliable

should be weighted more than those subject

to error.

3.2 Both overall and diagnostic measures of

performance are required in any systematic

effort. Overall measures refer to global

indices of behavior associated with large

aspects of performance such as occur in

mission segments and complete missions.

Diagnostic measures are quite specific,

identifying certain aspects or elements of

a job or performance in specific skill areas.

In any extensive evaluation effort, subjective and objective measures might well be

used. Generally, objective measures permit

measurement relatively independent of the

observer and with a high degree of reliaability. However, insistence on complete

objectivity tends to result in omission of

a variety of critical job components because

of the inability to measure them objectively.

Subjective techniques also have limitations.

The evaluation, generally, is dependent upon

the characteristics of the observer and

agreement between independent observers is

not high. The use of such observers can

introduce a set of biases into the

observations.

A-401

> _- ,•. .•:.:•••:.•: . . .y-YZv.: •
I Topics Relevant

! to System Development

'and Evaluation Technology ABSTRACT

•

Much of what is stated about combining

criteria also applies to combining performance measures. However, in the latter

case it is usually possible to determine

the interrelationship between each measure

and the criterion. Such information is

important in developing a single combined

proficiency score.

A single overall proficiency score has the

desirable characteristic of providing one

overall proficiency index. However, a

point to consider in determining whether

or not to combine several performance

measures is that a single index will no

longer reflect performance of the individual measures. Where at least a minimum

amount of proficiency with respect to a

particular measure is critical to an overall activity, it may be more appropriate

to treat the several measures in terms of

a profile. This will preserve the individual measures while allowing for a

simultaneous viewing of the measured results,

3.5 It is most important that any system for

structuring measurement of proficiency requires as an integral part a systematic

classification of behavior which can be

present within the system. Underlying the

development of this report is the assumption

that behavior can be analyzed in terms of

basic components, that these components can

be conceptually identified in a way that is

convenient and agreeable to people and that

there are specific measurement techniques

which appropriately go with various behavioral components. The behavior classification is operationally defined and structured

in a way meaningful for measurement. Four

levels of job analysis are presented under

which can be subsumed the varieties of operational events and behaviors predicted for

the system. The scheme is able to accommodate both diagnostic measures relating to

elemental tasks as well as to more global

measurements relating to overall system

performance. The classes are as follows:

A-402

. . -. •,.»• - -

" -.
Topics Relevant

I To System Development

! and Evaluation Technology ABSTRACT

Level I Elemental tasks

Level II Complex tasks involving

composite learning activities

Level III Mission segments

Level IV Combined mission segments

The classification begins with the simplest

level of analysis and proceeds to the most

complex and is so structured that each

succeeding level of complexity is made up

of recognizable behavioral units of the

proceeding level.

A sequence of logical steps is presented

concerned with setting up, obtaining and

reporting proficiency measures of individuals

and crews. With these procedures, the user

is provided guidance for the design of a

measurement system.

Step 1 - Conduct a system and job analysis

Step 2 - Identify important and critical

tasks

Step 3 - Determine performance requirements

for important tasks

Step U - Select and obtain measurements

appropriate for the behavior to be

evaluated

Step 5 - Make decisions on recording the

measurement data and combining the

results

A-403

--• i
Spencer, G. Mar-machine Simulation - the PIMO application. In R.E. Blanchard

& D.H. Harris (Eds.), Man-machine effectiveness analysis. Papers presented

at The Human Factors Symposium, University of California at Los Angeles,

June 15, 1967. (AD-735 718).

I Topics Relevant

I to System Development

land Evaluation Technology

i

ITopic

! No. ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Mode: (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.U General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.*! Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

H.l Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

1.1 Personnel for Testing

«.5 Test Plans

Application Components

or the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.1 Conclusions and

Recommendations

1.1 PIMO is an acronym for a project entitled

Presentation of Information for Maintenance

and Operation. The purpose of the project

is to develop a new, improved approach for

presenting the technical data used by Air

Force maintenance technicians. For test

purposes, the project focused on

maintenance operations for the C-ltlA Jet

cargo aircraft.

2.1 The staff realized that improvements in

maintenance information presentation had to

be expressed in terms of the object system,

i.e., the C-141A. The value of changes in

performance at the support level must be

evaluated in terms of object system

performance and/or cost.

4.1 Some means had to be devised which would

relate changes in maintenance performance

to changes in C-IMIA effectiveness. The

means employed was the Aircraft Maintenance

and Effectiveness Simulation model (AMES).

The AMES model was constructed to include

measures of functional reliability and

alternative personnel utilization, and was

used to establish payoffs in terms of

increased aircraft utilization and cost

savings which could be compared to the cost

of maintenance information system

improvements.

Further Research Areas

6.1 Measurement System

Limitation'

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-404

. ' - - • " .'

M.\m-J.-*. •_ v ^v^:/y^--•:• •.-:•.:;•:;•:•. •:•: •:-
Spyker, D.A., Stackhouse, S.P., Khalafalla, A.S. & McLane, R.C. Development of

techniques for measuring pilot workload (NASA CR-1888). Roseville, MN:

Honeywell,Inc., November 1971.

Topics Relevant i \

I to System Development I Topic I

iand Evaluation Technology I No. ! ABSTRACT

State of the Art Review

of the Process

1.1 General System

Neasureaents

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Coep^rents

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3-3 Performance

Requirements, Specific

3.4 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

i*. i Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoonendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentiale/

Priorities

6.3 Research Planning

1.3 The goal of this study was to provide

objective, quantitative methods of

measuring pilot workload.

2.1 The system being measured was a pilot and a

simulated tracking task.

2.3 The experiment took place in ? laboratory

situation and physical, psychological, and

environmental conditions were kept as

constant as possible.

3.1 The attributes measured were physiological

and performance dimensions over a range of

visual motor tracking tasks, and the

subject's reserve capacity.

3.2 Electrophysiological measurements were made

and a sensitive, nonloading measure of

reserve capacity was determined.

3.5 A Measured Workload Index CMWI) and a

Physiological Workload Index (PWI) were

extracted. A workload index based on the

pilot's physiological response to a

simulated tracking task was evolved.

Important steps in this approach included:

(1) Validation of a sensitive, nonloading

secondary task.

(2) Collection of physiological and performance data over a range (easy to

land) visual motor tracking tasks.

(3) Extraction of any potentially meaningful features from the analog

physiological data.

A-405

-•-. ,.-.-- .- .•

-t^- , . . v ^ 1 • - v - -
I Topics Relevant |

I to System Development I Topic I

land Evaluation Technology ! No. ! ABSTRACT

CO Normalization of the features.

(5) Selection of the "best" subset of

these features.

(6) Simultaneous computation of the

workload index and the best linear

predictor from the subset of features.

(7) Validation of this predictor.

The entire study was structured to provide:

(1) A sensitive, non-loading measure of

reserve capacity.

(2) An unencumbering reliable measure of

the psychological state. An important

measure of the success of this study

was the degree to which the MWI and

the PWI agreed across the randomly

presented 243 - four-minute trials.

Three direct measures of reserve capacity

were provided:

(1) Miss Rate - - % of error in responding

to secondary task.

(2) Response time - - average time from

secondary task stimulus onset to

response.

(3) Subjective Rating pilots'

evaluation of task difficulty. AB137.

5.4 The salient features of this study include:

(1) A simple, sensitive, nonloading

secondary task.

(2) A subjective rating which agrees with

other secondary task measures but with

less intersubject variance.

(3) A multichannel physiological

monitoring and recording system.

A-406

- n-. -^ - - - , ,

. .- .-.-,•-•.•

r. <i. - .

-' -" - - ' - ' - ' - '
Topics Relevant

to System Levelopnent I Topic I

land Evaluation Technology ! Wo. ! ABSTRACT

CO Automatic feature extraction software

which transforms the analog data base

Into meaningful features.

(5) Very good separation results using a

pattern recognition system, assuming

the data to represent a two-class

problem.

(6) Use of simultaneous least-squares

prediction to arrive at statistically

significant, validated, workload index

and the physiological features which

best predict it.

«

A-407 v(-

... .... -_^ •Ä^l-
Swink, J.R., Butler, E.A., Lankford, H.E., Miller, R.M., Watkins, H. & Waag.

W.L. Definition of requirements for a performance measurement aystem for C-5

aircrew members (AFHRL-TR-TS-S'*). San Diego, CA: Logicon Inc., October 1978.

I - Topics Relevant I I

I to System Development I Topic I

land Evaluation Technology ! No. ! ABSTRACT

State of the Art Review 1.3

of the Process

1.1 General System

Measurements 2.1 1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPU) 2.2

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environaent Definition

2.t General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.k Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

3.1

3.2

3.5

This study identifies and defines C-5

aircrew tasks and performance.

The system addressed was the C-5 aircraft

and aircrew.

The mission definition was the effective

operation of the aircraft on a typical

mission

Aircrew p -formance was selected for

measurement.

A candidate set of performance measures was

developed to assess aircrew proficiency.

This was accomplished

( p Determination and segmentation of a

representative mission profile.

(2; The determination of aircrew

activities within each segment.

(3) The determination of

mission-essential/critical aircrew

tasks and duties amongst these aircrew

activities.

The C-5 inflight performance measurement

can be obtained through several methods.

It appears that the C-5 MADAR system which

routinely monitors approximately 19 parameters and has the ability to interrogate a

number of key switch positions will provide

a feasible basis for interfacing the

inflight measurement system with the C-5

components.

A-408

- -- 

- - - ^ ^
Topics Relevant

! to System Development I Topic I

land Evaluation Technology j Ho. i ABSTRACT

1.3 A C-5 simulater was used in this study.

^.5 In Phase I candidate performance measures

were developed. The source of this

information included documentation,

interviews, and dialogue with operationally

qualified aircrews. A special purpose

evaluation sortie for the C-3 simulater was

developed.

In Phase II efforts were developed toward

the definition of several alternative

configurations to meet the performance

measure requirements and provide MAC with

capabilities of conducting the evaluation.

A review of existing system, documentation,

and interviewing techniques accomplished

this task.

In Phase III, the functional and

engineering requirements for the C-5

performance measurement system were

described.

A-409

- - -• - . . ..-. . -- ,.-- .J^..... . ^^^
Taylor, E.N. & Tillman, B. Human factors engineering study of two ball port

designs for IFV. Unpublished manuscript, October 1977.

Topics Relevant

! to System Development

!and Evaluation Technology ABSTRACT

State of the Art Review 1.3

of the Process

1.1 General System

Measureisen ts 2.1 I.I System Taxonomy

Model (STM;

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requiremenvs, Specific

3.4 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

«.1 Analytic Methods

4.2 Parameter Determinations

2.2

2,6

3.2

I.I

1.9

'1.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

Purther Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

This study was undertaken to compare two

ball port designs.

The current firing port weapon (XM231)/ball

port design for the Infantry Fighting

Vehicle (IFV) was compared to a new design.

Data were collected on the removal and

installation of the firing port weapons.

Critical to acceptance of the proposed

design was its effect on the time required

to dismount as part of rapid egress from

the vehicle. In addition, the weapon

retention characteristics were considered.

Time required for removal and installation

of the firing port weapon was measured.

Five subjects and one test coordinator

participated in the experiment.

Each subject was trained to install and

remove the weapon on each configuration.

The seat chosen for the experiment was at

the worst possible angle for the tasks.

Each subject performed six trials in

removing and installing the weapon in each

design configuration. Time measurements

were obtained by means of a stopwatch.

A-410

- - - - • m. A-^ .... . - . . - .

••;y^-->>^v\:- :v
I Topics Relevant I I

I to System Development 1 Topic 1

land Evaluation Technology ! No. I ABSTRACT

5.^ Average time for removal of the firing port

weapon was I.M seconds for the current

design-used snap ring, 3.5 seconds for the

current design-new snap ring, and 3*1

seconds for the proposed design-. While the

current design-used snap ring is approximately 2 seconds faster for weapon removal

than either of the other two configurations, its condition (state of fatigue of

the snap-ring) was probably unsafe.

Average times for installation of the

firing port weapon were 8.4 seconds, 6,2

seconds, and 3.8 seconds with the current

design-used snap ring, current design-new

snap ring and the proposed design,

respectively. While installation time is

less critical than removal time it is clear

that the proposed design provides an

advantage for ingress and rapid firing of

the weapons mounted in the ramp.

A-411

- • - -• - - - - g .

-.

—fc-- - - -
The Bunker-Ramo Corp. Proposal for a study of hiwcan factors field

performance measurement methodology (Proposal No. 5655-022-5U1). Canoga

Park, CA: The Bunker-Rarao Corp., Defense Systems Division, June 1965.

Topics Relevant

! to System Development [Topic

land Evaluation Technology ! Wo. ABSTRACT

1. State or the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (SIM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3- Analytic Components

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3-5 Measurement Procedures

t. Plannl Components

of the Process

1.1 Analytic Methods

i|.2 Parameter Determinations

1.1 This proposal describes a study to

investigate and develop a methodology for

human factors evaluation of equipment using

field performance measures at the

small-group level. The theoretical

framework around which this proposed study

is oriented involves the determination of

relationships among subtasks and between

subtask outputs and the terminal task

output. It is conceptualized that

measurement criteria should be developed

for those subtasks and outputs which are

most highly correlated with the terminal

task output. Additional inputs to the

measurement should involve consideration of

the stress imposed on task performance by

various operational conditions. Equipment

operability would be defined in terms of

resistance to the performance deg adation

caused by these stress conditions.

1.3 The proposed process for the development of

"alid field performance measures is as

rtlloua!

1.3 Apparatus for Testing

ft.« Personnel for Testing

I.I Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.« Conclusions and

Recoonendations

Further Research Areas

6.1 Measurement System

Limitations

b.t Research Potentials/

Priorities

6.3 Research Planning

v

follows:

(1) Select tasks which manifest a range of

behaviors from complex to simple;

which are related in the performance

of some total system function; which

are exposed normally to a variety of

operational conditions.

(2) Analyze tasks to describe the task

hierarchy and show interrelationships

among tasks; to describe the

behavioral functions which implement

performance; to determine the points

at which load conditions arise for

A-412

• - - 1 •--1 -

-- --h _-. . ,• . - _ v-'-^
Topics Relevant —

to System Development I Topic!

and Evaluation Technology ! No. ! ABSTRACT

I

B \

I

personnel; to indicate contingent

events which may affect task performance; to indicate the probable effect

on task performance of introducing

chemical agent protective equipment;

to describe the nature of stimulus

inputs, terminal outputs and potential

methods of task measurement.

(3) Develop and administer a task

performance characteristics scaling

test which would depend on the

experience of skilled military

personnel to discriminate between the

correlated factors involved in

selected task performance.

CO Develop predictive criteria from the

I test data to be validated in field

exercises.

(5) Conduct the field exercises; record

and analyze the field data to

determine the correlation between test

and field measurements.

A-413
Tien, J.M. Toward a systematic approac

Transactions on Systems, Man, and Cyben.

^-515.

o program evaluation design. IEEE

_iics, September 1979, SMC-9(9).

Topics Relevant

I to System Development

land Evaluation Technology ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1J Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Reconnendatlons

1.2 In this paper, the broad field of program

evaluation, specifically evaluation of

programs in the public sector, is reviewed.

Attempts are made to synthesize and

systematize the steps necessary to develop

valid and comprehensive evaluation designs.

First, a design framework is identified

which links program characteristics to

design elements through an expanded set of

threats to validity. Second, the various

design elements are grouped into five

systematically convenient components,

including test hypothesis, selection

scheme, measures framework, measurement

methods, and analytic techniques. Third,

it is proposed that the different types of

evaluation can be contained in an

evaluation taxonomy composed of eight

measures-related classifications.

It is noted that there were many ways of

classifying a program evaluation effort:

by subject matter of the evaluation; by the

purpose of the evaluation; by the

methodology employed in the evaluation; or

by some other criterion. A number of the

most commonly used evaluation categorizations is listed in terms of the subject

matter or purpose of the evaluation.

The authors state that this paper should be

regarded as only an initial step towards a

systematic approach to program evaluation

design while a basis for further discussion

is provided.

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-414

-• - - - - L- ^^^j

-'- '

.» ,. .,.,•. '_..

. fc,-.-.'.'-,---..

Jllljjtlj ^ > - « ~ «
Timson, S.F. Measuremgnt of technical performance in weapon system development

programs; A subjective probability approach (RM-5207-ARPA). Santa Monica,

CA: The RAND Corporation, December 1968. (AD-681 771).

Topics Relevant I |

| to System Development I Topic!

land Evaluation Technology ! No. I ABSTRACT

State of the Art IteTlew

of the Process

1.1 General Systen

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Cooponents

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environaent Definition

2.M General Constraints

2.5 Perforaance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3 2 Practical Attribute

Measures

3 3 Performance

Requirements, Specific

3 1 Performance

Criteria, Specific

3 5 Measurement Procedures

Planning Components

of the Process

t 1 Analytic Methods

1 ? Parameter Determinations

14 3 Apparatus for Testing

H u Personnel for Testing

H

At

5

)P1

Test Plans

icatlon Coaponents

of the Process

5 1 Test Execution

5 2 Data Analysis

5 3 Findings Interpretation

5 1 Conclusions and

Recooaendatlons

Further Research Areas

6 1 Measurement System

Limitations

6 2 Research Potentials/

Prlorit'a»

6 3 Research Planning

1.1 There is an effort underway to develop the

framework of a procedure for the collection

and analyses of data on uncertainty and

progress regarding technical performance in

weapon systems development. The data

collected are concerned with uncertainty

about the characteristics of the component

parts of the total system. These data are

combined using systems design relationships

to determine the uncertainty about the

performance of the total system.

1.3 The steps undertaken to determine the

uncertainty of a total system's performance

are:

(1) Find design equations that relate the

subsystem properties to the total

system performance.

(2) Determine the subjective probabilities

for the subsystem and the component

properties that influence the total

system performance.

(3) Utilize the Monte Carlo procedu-es to

generate probability distributions for

the system performance characteristics.

(4) Compute the statistical measures of

the system performance probability

distribution.

(5) Compare the statistical measures for

the different time periods to obtain

indications of progress.

A-415

- - - - -

» -* » » - -

--: •- • L". .^ .• .N .•• .•• .•- .• -• ."-. - . -."• -

N

-'• •• -"' -

v

>" -*•>">- »"• ."• •"• -"->'• -"• «•• .

v
r

! Topics Relevant I I

i to System Development I Topic I

land Evaluation Technology ! No. ! ABSTRACT

3.1 The performance characteristics critical to

the aircraft's mission capability are

speed, altitude, range, and endurance.

B

U.I The computational methods used to analyze

the data were the measures of central

tendency (mean, median, mode) and the

measures of dispersion (range, standard

deviation, variance).

i

'

,

%1

.

A-416

- - - - -
Topoiller, D.A. Mathematical models of human performance in man-machine

systems (AMRL-TR-68-22). Wright-Patterson AFB, OH: Aerospace Meiical

Research Laboratories, May 1968. (AD-673 3W.

Topics Relevant

I to System Development

land Evaluation Technology

i i

ITopicI

i No. i ABSTRACT

State of the Art Review 1.3

of the Process

1.1 General System

Measurements

1.2 Systeir Taxonomy

Model (STO)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.ll General Constraints

2.5 Performance

Requirements. Ultimate

2.6 Performance

Criteria, Ultima it

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

«.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

H.H Personnel for Testing

H.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoonendati ons

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

This report describes three research

approaches to the problem of mathematically

representing human performance parameters

in weapon, maintenance, and command and

contrui systems. In the first approach,

twenty operations research analyses and

models of military systems were examined to

determine if the models included human

factors parameters and to what extent they

were sensitive to variations in these

parameters. Although many of the functions

of the systems modeled were performed by

humans, human performance parameters were

not, in general, sufficiently defined to

permit mathematical or empirical manipulation within a man-machine simulation framework. In the second approach, an attempt

was made to establish predictive relationships, based on regression and factor

analysis techniques, between human engineering design parameters and those criteria of

systems effectiveness, such as maintenance

task time, that can be transformed into a

more molar index - system downtime. The

human engineering predictor-parameters

accounted for 50% of the criterion variance. In the third approach, a series of

experiments involving real-time simulation

of a command and control system was conducted to determine if, and how, a computer

might aid diagnostic performance (in tactical decision making) in threat evaluations.

The system output or criterion of effectiveness was the degree to which the system

assesses the true state of threat. With

computer aiding, correct decisions

increased by 13t.

A-417

•_' »^' «. *

-- •- 1

:: -:.:•;-;-. -v; -:• v-y-:-:-:- ;.- O >; >.: -..-: : ii&ii
Topniller, D. A. The role of applied man-machine models. In G. W, Levy,

(Ed.), Symposium on applied model of man-machine systems performance

(NR69H-59iK Columbus. OH: North American Aviation, November 1968. (AD-697

939).

Topics Relevant

i to System Development

land Evaluation Technology ABSTRACT

State of the Art Review

of the Process

1.1 General System

Heasureaents

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2. 1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3-5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

1.2 Parameter Determinations

1.3 Apparatus for Testing

1.1 Personnel for Testing

1.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.1 Conclusions and

Recomenda ti ons

1.1 Three stages can be identified in the

evolution of human engineering:

(1) Operationally-oriented design criteria

development (or the "knobs-and-dials"

stage).

(2) Design criteria formalization (publication of Human Engineering (HE)

specifications, standards, handbooks,

etc.).

(3) Systems effectiveness modeling (or

"systems-oriented human engineering").

1.3 Systems effectiveness modeling is concerned

with quantifying the impact that HE design

has on systems parameters. It faces the

question: if one maximizes performance for

a particular Dian-machine interface, what

impact will this have on overall system

performance?

Three major categories of man-machine

models are identified:

(1) Behavioristic Models.

(2) Adaptations of Engineering Models.

(3) Systems Economic Models.

Behavioristic models derive from or

parallel the stimulus - organismic -

response (S-O-R) paradigm of experimental

psychology. These models include:

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

?

A-418

- - - - ^ - I - - - — - - ^ . -

*-•

•- -

- . . . .... . .,. . ... . .
Topics Relevant

to System Development I Topic 1

and Evaluation Technology i Ho. ! ABSTRACT

(1) A "sensing or "input" portion.

(2) A "processing" portion.

(3) An "output portion."

Adaptations of engineering models basically

form two classes: Servotheoretic Models

and Information Theoretic Models. Systems

Economic Models are divided into two

classes: Operations Research Models and

Systems Effectiveness Models.

2.6 The major parameters of system effectiveness have been defined as availability,

capability and dependability. Availability

is equivalent to the system's readiness to

perform its mission. Dependability is

equivalent to a measure of system condition

at points during the mission. Capability

is a measure of the system's ability to

achieve the mission objectives. These

parameters are criteria of system performance which require measurement and prediction. A paradigm or quantifiable framework

is required which permits assessment of

man's performance and contribution to these

criteria - parameters.

3.5 The following are goals of applied

man-machine models. Such models should:

(1) Describe and quantify the functions of

both man and machine.

(2) Allocate functions for most efficient

utilization of man's capabilities and

machine's capabilities.

(3) Offer design alternatives with

functional tradeoffs between design

parameters and human performance.

(U) Predict influence and variations in

functional considerations on systems

criteria.

A-419

^ - ^ * - -- - . ^ ' - - - - - -

. . _ •

_^_
Topics Relevant

to System Development I Topic!

and Evaluation Technology ! No. ! ABSTRACT

(5) Be capable of sensitivity analysis.

(6) Be capable of assimilating new human

performance data.

(7) Be appropriate to the level of systems

description.

(8) Human performance measurement should

be standardized as equivalent to

engineering parameter measures.

A-420

- > - . - .__, . . - i . --^-.•- •-'-..-.-- .■■.:.— . .--.. ... . -•.. ^ --_•. . . ^-J-.,.
'.*•

Tranby, E.D. Advanced surface ship weapon systems test and evaluation

(NSWSES-TP-60). Port Hueneme, CA: Naval Ship Weapon Systems Engineering

Station, January 1976. (AD-390 826).

Topics Relevant

to System Development iTopic 1

and Evaluation Technology ! Wo. j ABSTRACT

State of the Art Review 1.3

of the Process

1.1 General Systea

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPU)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.'! General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.t Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

0.2 Parameter Determinations

1.3 Apparatus for Testing

H.H Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoamendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

The purpose of this study was to produce a

test and evaluation guide for an advance

surface ship weapon system. This guide is

intended to be used by personnel well

qualified operationally and technically,

but who have limited test and evaluation

experience.

The guide provides an overall skeletal

structure of events that take place in T

and E from the issuance of project

assignment to composition of the final

report. This structure provides a

step-by-step procedure that aids in

organizing the total effort.

Standardized formats with instructions

provide guidance in key steps such as

writing the test plan and producing final

test report. Checklists of important

actions are presented and reference lists

have been provided.

The structure of the docunent is heavily

influenced by the process the test team

experiences.

Numerous examples are used to illustrate

the collection, verification, reduction,

and control of data. A standardized test

plan format is presented with instructions

for use and follow-up. A standardized

format for test procedures is also

supplied.

The use of simulators for physical testing

and as a diagnostic tool is discussed.

The processes of analysis ar*. set down in a

step-by-step procedure. A standardized

test report format is presented and a

glossary is provided.

A-421

• ^_^_ ^ - -- .. ... -- - -- v - -

.1.-- -../.•.. A_ ^ ^ - i.
Turner, C.R. & Bard, J.F. Tactical AWACS measures of effectiveness

(ESD-TR-72-1U2). Bedford, MA: The MITRE Corp., April 1972. (AD-7'42 233).

I . Topics Relevant I

I to System Development [Topic

land Evaluation Technology ! No. ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual ComponenU

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

1.2 Parameter Determinations

I.I Apparatus for Testing

1.1 Personnel for Testing

1.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.1 Conclusions and

Recommenda tions

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

2.1 This report discussed the Airborne Warning

and Control System (AWACS).

2.2 The utility of AWACS when employed as an

element of the Tactical Air Control System

(TAGS) was examined and nine generic

tactical air missions were selected for

this evaluation.

2.5 The ultimate requirement of the AWACS was

to kill hostiles and destroy targets.

3.1 AWACS' MOE's were divided into five

distinct categories: (1) Reaction (receive

a mission, request a message, process data,

communicate response); (2) Surveillance

(detect, identify and track aircraft); (3)

Command (allocation of available

resources); (M) Control (the control of

friendly aircraft) and (5) Communications

(on-board communications capability).

3.2 Reaction is measured in units of time and

surveillance ability in terms of numbers of

friendly/hostile aircraft detected,

identified, tracked, etc., per unit of

time. Coramand is measured by the ability of

AWACS to allocate resources in terms of

number and percent of sorties scrambled,

immediate response requests accomodated,

and sorties diverted. Control reflects the

number and percent of friendly aircraft

under control per unit of time per sortie.

No MOE was assigned to the Communication

category - - this function is implicit in

all the reaction time and command MOE's and

in all but one of the control MOE's.

A-422

MM. • . .• .•.•

—
! Topics Relevant I I

1 to System Development 1 Topici

land Evaluation Technology ! Ho. I ABSTRACT

3.3 The AWACS' contribution in terms of

surveillance, command, control, and

communications capabilities was the

specific area of interest.

3.5 A methodology and criteria were established

for assessing the system's capability using

system-level measures of effectiveness. A

set of MOE's was established for both

levels (the AWACS system and the tactical

mission). To obtain a standard for

comparison the scenerio under consideration

was analyzed both with and without AWACS.

The incremental differences in tactical

mission MOE's combined with AWACS system

MOE's provided the insight into the

effectiveness of AWACS.

S.M It was felt that by following the

methodology developed in this report, the

specific contributions of the AWACS will be

clearly measured and established. Subsequent application of the tactical mission

level MOE's will yield additional insight

into the operational significance of the

AWACS' contribution to the success of the

tactical mission.

It is stated, however, that no single

unique MOE can be generated to measure

either AWACS' effectiveness or tactical

mission success.

A-423
Uhlaner, J.E. Human performance, jobs, and systems psychology-the systems

measurement bed (Tech. Rep. S-2). Arlington, VA: Behavior and Systems

Research Laboratory, October 1970. (AD-716 3H6).

I . Topics Relevant

I to System Development 1 Topic I

land Evaluation Technology ! No. I ABSTRACT

SUt« of the Art Review 1 • 1

of the Process

1.1 General System

Measurenents

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3-3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Ccaiponents

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoonendatlons

Further Research Areas

6.1 Measurement Syster.

LiKltatlons

*.2 Research Potentials/

Priorities

6.3 Research Planning

The systems measurement bed is a means of

focusing step-by-step on the human performance aspects of the system to be enhanced

and identifying the interrelationships of

the hunan factor system variables in order

to determine productivity under varying

conditions.

The job of leader must take into consideration the influence of the situation and

styles of behavior on job performance. How

a leader goes about carrying out the

mission objective is directed, in part, by

his particular style of behavior, value

system, and the situation.

1.2 A taxonomy of jobs containing cognitive

variance (responses more objectively

characterized as right or wrong) and

noncognitive variance (responses less

objectively characterized as desirable or

undesirable) were determined from this

study. The system measurement bed assists

the researcher in dealing with the

different measurement characteristics of

the two classes of jobs.

1.3 The establishment of a systems measurement

bed calls for a great deal of subject

expertise: situations have to be designed,

scenarios written, measurement strategies

devised, and computer programs prepared.

Appropriate experienced personnel must be

identified to serve as subjects. Lastly,

all these concepts, materials, and

procedures have to be built into a

logistically feasible space where relevant

factors and criteria can be incorporated.

A-424

••

:

-- - • -

'. ^- .•. 1 - _ _• » » 1 - - - - -' 1 1
Uhlaner, J.E. & Drucker, A.J. Military research on performance criteria;

A change of emphasis. Human Factors, 1980, 22(2), 131-139.

I Topics Relevant I I

I to System Development iTopic I

land Evaluation Technology i No. I ABSTRACT

-.

State of the Art Review 1 • 1

of the Process

1.1 General Syste*

Heasureoents

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPU)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment. Definition

2.1| General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of th« Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3-3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3-5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

0.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

This paper discusses trends in the

development and use of performance

measures. It is noted that Army research

to develop and to predict criteria of human

performance has attempted to achieve

greater relevance between performance

measures and Job tasks.

The paper discusses the various performance

criteria utilized in the past to predict

individual effectiveness; these criteria

include grades, ratings, and performance

tests. It describes the more recently

developed measures of unit effectiveness

and measures dealing with human factors

problems encountered in systems analysis.

Some of the measures mentioned are the

"Skill Qualification Test" (SQT), devised

as a performance - based measure in one of

the Army's new tactical training systems.

This method provides two-sided free play

exercises under simulated battlefield

conditions and includes a specific set of

operations for observing and recording

actions that operational personnel have

agreed upon as relevant to mission success.

Other Army measurement systems described

include the Organizational Effectiveness

(0E) program, using a Work Environmental

Questionnaire (WEQ) for diagnostic

purposes, and a field method for evaluating

the performance of Army helicopter pilots.

The paper goes on to describe the System

Measurement Bed (SMB) research approach

associated with the concept of how section

A-425

<.!- _»

.Ut.
I Topics Relevant I

I to System Development !Topic I

land Evaluation Technology ! No. I ABSTRACT

training and work environment interact to

influence the performance of the individual

on the job.

The paper concludes by noting that there

are exciting possibilities in the future of

human performance systems research and that

analytic techniques in this area have the

potential to impact a wide range of areas

beyond the man-machine systems in the Army.

B

A-426

— -' • - - ^ - -- - -
Ultrasystems, Inc. A study of measures of effectiveness used in naval analysis

studies (Vol. 1 Summary). Newport Beach, CA: Ultrasystems, Inc., October

1972. (AD-912 W3).

Topics Relevant

to System Development [Topic!

ind Evaluation Technology ! No. I ABSTRACT

1. State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

4. Planning Components

of the Process

K.I Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

5. Application Coaponents

of the Process

5.1 Test Execution

5.2 Data Analysis

5-3 Findings Interpretation

5.4 Conclusions and

Recoenendations

6. Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.3 This study is concerned with the collection

and comparison of measures of effectiveness

(MOE's) used in naval studies and analyses.

Volume I is the final report of the study

and includes the results of examining 213

Navy studies of system effectiveness

covering virtually all aspects of naval

warfare.

The results of this study ire presented in

several different forms. First, a data

base, utilizing two types of formats - -

Study Review Suranary or MOE Review, was

established to present in summary form the

effectiveness profile of each study

examined. This profile presents an outline

of the military situation addressed,

variables and qualitative factors

considered, and the special assumptions and

limitations in HOE formulation and

development. In addition, a general

summary of measures of effectiveness used

in naval warfare is presented.

2.1 Of the studies examined, the ASW area

accounted for 37%, the attack area

accounted for 23X, and the anti-air warfare

area accounted for 9% of the warfare areas

considered. The remaining 31* consisted of

mining and mine countermeasures,

surveillance, strategic systems, electronic

warfare, amphibious assault,

communications, command and control,

navigation, special warfare,

reconnaissance/intelligence, logistics, and

ship support.

A-427

.v.v AJ-VV • A.V

-- • •
1

i Topics Relevant

I to System Development [TopicI

and Evaluation Technology ! No. | ABSTRACT

3.1 Analysis was conducted on the types of

variables used in mathematical formulation

and development. The results showed that

nearly Mfi of all independent variables

considered were associated with the

particular interests of the author. Only

19X of all independent variables related to

the threat or target, and slightly less

than 3% related to the physical environment.

In Appendix E of this report an index of

the measures of effectiveness used in these

studies was presented. The index described

the system and its function, the situation

and the criterion for success, and lists

the measures of effectiveness considered.

6.1 Several limitations on system measurement

in these studies were noted. Briefly

stated they are:

(1) The criterion for success is seldom

explicitly stated.

(2) There exists more than one way of

quantifying how well the criterion for

success is met.

I

(3) For each possible mission title there

is more than one way of defining the

mission.

(1) The rationale for MOE selection is not

always presented.

(5) Physical environment aspects appear to

be generally ignored or casually

treated in effectiveness studies.

(6) It appears that there are cases where

the variable" selected for model

formulation are not readily (if at

all) measureable in the real world.

(7) In general, the MOE's used are those

that are readily obtained via model

development.

A-428

----'•

« " . * - - . - » , ^ i

,-..v..-,-.•_-. .
I Topics Relevant

I to System Development I Topic I

land Evaluation Technology i No. ! ABSTRACT

(8) Very seldom, when more than one MOE is

identified, is a ranking of importance

performed or combined measure

developed and used.

(9) Expected value type MOE's are most

prevalent in force level studies

whereas probability type MOE's are

most prevalent in subsystem level

studies.

(10) On the average, over twice as many

independent variables occur in ti.t

friendly force category than in the

threat and target categories combined.

(11) As the study level increases, from

subsystem to system to force level,

the percentage of independent

variables in the friendly force

category decreases and the percentage

of independent variables in the

friendly force interaction with threat

or target category increases.

(12) It is not easy to compare similar

effectiveness studies.

A-429
.

Ultrasystems, Inc. A study of measures of effectiveness used in naval analysis

studies (Vol. 2 Study Review, Summaries, Part 1). Newport Beach, CA:

Ultrasystems, Inc., October 1972. (AD-912 MUU).

I Topics Relevant

I to System Development

land Evaluation Technolofey

Topicl

No. 1 ABSTRACT

1.

C

•

i

Stat« of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environaent Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Co«ponents

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

1.1

4.2

4.3

4.4

4.5

1.1

2.2

Analytic Methods

Parameter Determinations

Apparatus for Testing

Personnel for Testing

Test Plans

5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoonendations

6. Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

This document contains annotated

bibliographies. The following is a list of

the 10 specific categories or headings plus

the number of bibliographies each contains:

(1) Airborne ASW 17

(2) Airborne AAW I

(3) Airborne Attack 1U

(4) Environmental Systems 1

(5) Mining 2

(6) Mine Counter Measures 3

(7) Ocean Surveillance 3

(8) Submarine ASW 18

(9) Submarine Attack 4

(10) Surface ASW 15

The following is a representative sample of

missions:

(1) Submarine search

(2) Sonobuoy barrier patrol

(3) Barrier placement/patrol

(4) Surface ship defense

(5) Air strike

(6) Close air support

(7) Surveillance of ocean area

(8) Bathythermograph maneuver

(9) Mine clearance

(10) Contact prosecution

(11) Search and destroy

(12) Submarine attack on convoy

(13) Escort/screen

A-430

•
I Topics Relevant I

\ to System Development I Topic

land Evaluation Technology ! No. ABSTRACT

3.2 The following is a representative sample of

practical attribute measures:

(1) Probability of submarine detection,

localization, and kill.

(2) Ratio of the incremental improvement

in accomplishing the mission to the

incremental monetary cost of such an

improvement.

(3) Detection range of raid relative to

the vital area center (CVA) for a

given intercept range.

CO Expected number of targets destroyed

in a given period of time.

(5) Difference in fuel consumption due to

the bathythermograph maneuver.

(6) Total force level required to clear a

given area in a given time.

(7) Expected number of ships hit.

(8) Elapsed time to target detection.

(9) Maximum exposure time of the submarine.

3.3 The following is a representative sample of

the criteria for success:

(1) Detection of submarine.

(2) Suppression of submarine activity.

(3) Destruction of target.

(4) Successful attack capability.

(5) Low cost measurement of the vertical

ocean temperature profile.

(6) Survival of aircraft and planting of

mines.

A-431
-.

I Topics Relevant

! to System Development

{and Evaluation Technology ABSTRACT

•

a

(7) Clearance of minefield

(8) Surveillance and establishment of the

track of ships at sea.

(9) Preparation for attack in the least

possible time without being

counter-detected.

(10) Insurance of the safe passage of

convoys, strike groups, and amphibious

forces in the presence of hostile

submarines.

:< A-4Z2
:;:

•-•

Ultrasystems, Inc. A study of mc-asures of effectiveness used in naval analysis

studies (Vol. 3 Study Review Summaries, Part 2). Newport Beach, CA:

Ultrasystems, Inc., October 1972. (AD-912 «US).

I Topics Relevant

I to System Development

land Evaluation Technology ABSTRACT

State of the Art Review

of the Process

1.1 General Systea

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

See Ultrasystems, Inc. A study of measures

of effectiveness used in naval analysis

studies (Vol 1 Sunmary). Newport Beach,

CA: Ultrasystems, Inc., October 1972.

(AD-912- 413)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoonendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Rese.irch Potentials/

Priorities

6.3 Research Planning

A^33
Ultrasystems, Inc. A study of measures of effectiveness used in naval analysis

studies (Vol. M MOE Reviews). Newport Beach, CA: Ultrasystems, Inc., October

1972. (AD-912 M46).

k Topics Relevant

to System Development

and Evaluation Technology ABSTRACT

State of th« Art Re M

of the Process

1.1 General Systeo

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

See Ultrasystems, Inc. A study of measures

of effectiveness used in naval analysis

studies (Vol 1 Summary). Newport Beach,

CA: Ultrasystems, Inc., October 1972.

(AD-912- M3)

•

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.H General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

E.

Analytic Components

of the Process

2.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.'* Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

k.2 Parameter Determinations

1.3 Apparatus for Testing

<l.i( Personnel for Testing

4.5 Test Plans

•

Application Conponents

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-434
U.S. Army Armor School. Tank platoon: Organization for combat and techniques

of movement (TC 17-15-3). Fort Knox, KY: U.S. Army Armor School, April 1975

(Pamphlet).

--

Topics Relevant

to System Development 1 Topic I

and Evaluation Technology ! No. I ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.'> General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.t Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for- Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recomnendations

further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Prloritie*

6.3 Research Planning

1.1 The U.S. Array must learn to fight

outnunbered and win. The priorities to

these goals include the following:

(1) Suppressive fires against enemy air

defense systems, especially

radar-directed systems, to permit

scout and attack helicopters to

operate more effectively.

(2) Attack and counterattack on reverse

slopes to protect attacking forces

from long range enemy observation and

fire.

(3) Operating in darkness or other

conditions of reduced visibility to

reduce range and accuracy of enemy

observation and fire.

(U) Precision, discipline, speed, and

security in the directing and

reporting of the battle in order to

win the battle quickly, unimpeded by

enemy countermeasures.

(5) Adequacy of stowed loads of

ammunition and fuel; speed,

responsiveness, and security of

resupply systems to reduce the need to

resupply, but to ensure prompt

resupply when it is needed.

(6) Detection and identification of the

enemy at maximum possible distances

from the friendly main body to prevent

engagement of the main body under

adverse conditions—when it is

A-435
I Topics Relevant

I to System Development I Topic!

land Evaluation Technology ! No. I ABSTRACT

v

unwarned, poorly deployed, not poised

to fight.

(7) Firing fast first in a tank-antitank

battle, especially at targets that can

shoot back. In a tank duel, accuracy

is important—firing first is more

important.

(8) Control and distribution of tank antitank fires to kill targets rapidly and

save ammunition for engaging the next

attacking echelon.

(9) Battlefield movement only along

covered avenues-making maximum use of

terrain to evade enemy long range

observation and fire.

(10) Suppressive fires delivered from

overwatching positions to reduce the

chance that maneuvering friendly

forces can be seen and t igaged by the

enemy.

2.3 The assault takes place from the last

available cover and carries onto the

objective. The assault is normally

initiated from a wedge; however, terrain

and individual tank actions will govern the

actual alignment.

A-436
U.S. Army, Army Materiel Command. Engineering design handbook; Recoilless

rifle weapon systems. Alexandria, VA, 15 January 1976. (AD-A023 513).

Topics Relevant i 1

I to System Development ITopicl

and Evaluation Technology ! Wo. I ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2-3 Environment Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

0.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoonendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

2.1 This handbook was an exposition of proven

methods and materials for the engineering

design of the recoilless rifle weapon

systems.

2.2 The purpose of this handbook was to guide

the engineer, the mature practitioner as

well as the novice, directly to project

goals. It provided a comprehensive summary

of the available relevant technology and

the system engineering rationale.

3.1 The basic input requirement to the weapon

system was the kill probability for a

particular target and specified range.

The kill probability, in turn, places

requirements on the hit probability and

fire power of the weapon being designed.

As these requirements are traced further

through the system, it is found that all

components of the rifle are affected. The

result of this interaction is a system

weight for a given terminal ballistic

requirement.

3.2 Kill probability was defined as the product

of hit probability of a kill given a hit.

From the definition of target vulnerable

area, conditional kill probability can be

expressed as the ratio of the vulnerable

area to the presented area.

Hit probability was defined as the

probability of a hit or hits on a target

occurring out of a given number of rounds

fired at a target. For a specified target

and weapon system, the hit probability

depends only on the overall weapon

dispersion. The principal sources of these

dispersions or firing errors are range

estimation aiming, muzzle velocity

variation, system Jump and cant, crosswind

and the fire control equipment.

A-437
U.S. Army Combat Developments Command. Communlcatlons-electronlcs-TS (CE-75)

phase 1-field Army (No. 6492, Vol. 9). Fort Monmouth, NJ: U.S. Army Combat

Developments Command, Communications-Electronics Agency, September 1968.

(AD-883 611).

I Topics Relevant I

I to System Development 1 Topic I

land Evaluation Technology ! No. I ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.4 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

<i.2 Parameter Determinations

4.3 Apparatus for Testing

1.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoonendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.3 In the CE-75 man-machine interface investigation, the following procedures were undertaken:

(1) Creation of a data base

(2) Analysis of systems concepts

(3) Expansion of a data base

(4) Examination of suitable mode s

(5) Review of the personnel subsystem

(6) Critical incident analysis

(7) Measurement of man's contribution to

system effectiveness

2.1 A system definition is a conceptual framework for attacking problems. In its broadest terms a system was comprised of hardware, facilities, logistic support, and the

trained manpower required for operation in

a particular environment. For CE-75, the

MMI system included the entire collection

of men, facilities, and equipment in the

CE-75 Field Army Tactical Communications

System. This system also included the area

and command systems.

A-438
V

: ;

E

I Topics Relevant

I to System Development [TopicI

land Evaluation Technology ! No. ! ABSTRACT

2.2 In a broad sense, the mission of the CE-75

communications system was to provide a

means for the timely transfer of meaningful

and significant information from action

officer to action officer. In a more

specific sense, what was considered were

the estimates of how much time can be

allowed for the information transfer

process, message priorities, and a delineation of the organization to be served.

Additional elements of the mission were:

(1) How much information and of what kind

need be transferred between

individuals?

(2) What mode should be used for

transmission of this information?

•

2.4 A man-machine interface is the boundary at

which a man and a machine interface in

order to achieve a system objective. The

extent of this boundary was constrained by

three factors:

(1) Tasks required of both the man and the

machine to attain the system

objective.

(2) Capabilities and limitations of the

machine.

(3) System objectives as affected by

environment, personnel policies, and

equipment use.

A-439
U.S. Army Human Engineering Laboratories. Standardization of tasks and

measures for human factors research. Proceedings of a Conference at

Texas Tech University. Lubbock, TX, 18-19 March 1970. (AD-TI'J 669).

Topics Relevant i i

to System Development 1 Topic!

and Evaluation Technology I No. ! ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

«.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoomendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 This conference provided a forum for

discussion of the question: "Is standardization necessary in human factors?" The

papers presented, although they did not

completely answer this question, did give

some guidance to research groups regarding

the necessity for insuring that human

factors research is mission oriented and

relevant to the needs of its sponsora.

There was considerable discussion with

regard to a proposal that a data bank be

established for use by human factors researchers. It was pointed out that a data

bank could supply copies of specific

tables, figures and formulae directly to the

requester which would reduce the amount of

duplicated effort presently expended by

researchers. Some expressed the fear that

such a system would seduce the user into

weighting poorly collected, unreliable data

equally with carefully generated reliable

data. The rebuttal to this point was that

experienced investigators could generally

evaluate a set of data in terms of its

"reasonableness" and source. It was the

consensus that the data bank concept could

be of great value although there was some

concern for the potential misuse of such a

system.

It was noted that the value of a data retrieval system would be enhanced if one

could query the system about performance on

specified tasks under specified conditions.

However, there was some reservation with

regard to the practicality of standardizing

tasks and/or variables. The comment was

made that tasks might be defined in terms of

human functioning compartmentalized by the

muscle groups involved rather than apparatus

used.

A-440
Topics Relevant I

to System Development ! Topic

and Evaluation Technology ! V.o. ABSTRACT

Another point of view was presented to the

effect that standardization of tasks may

be less important than specification of

conditions under which the performance data

is collected. The major concern throughout

the discussion was for end-goal, or mission

completion, requirements in research. This

was considered by most participants to be

more important than standardization.

There was lengthy discussion of work measurement problems and considerable dissatisfaction was expressed with currently available

measures of human work.

There was also discussion of field experimentation versus laboratory experimentation

and the need for research on mission-oriented

tasks. It was stressed that the operational

environment must be kept in mind when planning

human factors experiments. The need for a

systems approach which would include the

study of social and physical environments

in mission-oriented situations was also noted.

A-441
U.S. Army Infantry Board. Infantry weapons test methodology study. Vol. 1;

Small arms test methodology (USAIB-3^19-F-Vol. 1). Fort Benning, GA; U.S.

Army Infantry Board, November 1971. (AD-890 998L).

Topics Relevant

i to System Development

land Evaluation Technology ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.') Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

4.2 Parameter Determinations

1.3 Apparatus for Testing

•.* Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.3 This volume sunma'-izes progress made in the

area of small arms weapon system test

methodology.

2.1 In this phase of the study test facilities

were constructed for the evaluation of

small arms weapons utilized by the

infantry.

2.3 The object of the study was to simulate as

nearly as possible actual combat

conditions.

3.1 Several categories of HOE's were established: accuracy, responsiveness, sustainability, reliability, portability, and compatibility, and signature effects. These

characteristics are measured under attack

and quickfire and defense modes.

3.2 Quantitative data were used to answer

questions concerning the weapons system

performance.

3.5 Instrumentation was developed and data

processing and computer equipment installed

at the test facilities. The goal was to

use more than one test soldier at a time

and yet be able to measure individual

performance.

A-442
I Topics Relevant

I to System Development 1 Topic I

land Evaluation Technology j No. I ABSTRACT

1.1 Multivariate analysis is us.d to measure

more than one measure of effectiveness. and

takes into consideration the- variability of

sample size and the interrelationships

among the various measures. It permits the

observation of statistical weighting

factors and is considered the key to the

operational weighting of factors which

influence the combat environment.

Regression analysis using miss distance

data was used for comparison of different

types of rifles.

4.5 Three facilities are operational: attack,

defense, and quickfire. In this report, no

specific test plans were presented.

5.3 This report discusses in general terms the

operational capabilities of the test

facilities. Data are presented in various

appendices (not in our files).

A-443

.■►.
U.S. Army, Materiel Testing Directorate. U,S. Army test and evaluation

command development test II (EP)—System test operations procedure "Test,

Measurement, and Diagnostic Equipment (system peculiar)." Aberdeen Proving

Ground, MD: U.S. Army Human Engineering Labo-atory, 7 May }97>*. (AD-781 9^i)

1 Topics Relevant

i to System Development

land Evaluation Technology ABSTRACT

1. State of the Art Review

of the Process

1.1 Genenl System

Measurements

1.2 Systeit Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1) Ctne-al Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3.1 Practical Measrrable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3-1* Performance

Criteria, Specific

3.5 Measurement Procedures

4. Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

6. Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.3 This report describes a methodology for

evaluating system peculiar Test,

Measurement, and Diagnostic Equipment

(TMDE), including physical and operational

characteristics. Procedures are provided

for initial inspection, physical

characteristics, safety, performance,

extreme environments, r-f interference,

reliability, Maintenance, and human

factors. Supplementary instructions are

provided for identifying the test item,

documenting test criteria, developing

performance tests, environmental tests,

test plan organization, and maintenance

evaluation of the TMDE.

A-444

•
s

.

•

U.S. Army Operational Test and Evaluation Agency. Mechanized Infantry Combat

Vehicle (MICV), XM723, operational climatic test/force, development test and

experimentation (Draft). Falls Church, VA: U.S. Army Operational Test and

Evaluation Agency, March 1976.

I Topics Relevant

! to System Development

{and Evaluation Technology

[Topic

I No. ABSTRACT

1. State of the Art

of the Process

Review

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.i( General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3-3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

It.y Analytic Methods

4.2 Parameter Determinations

<l.3 Apparatus for Testing

H.H Personnel for Testing

1.5 Test Plans

Application Coaiponenta

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.1 Conclusions and

Recoonendatlons

2.1 The MICV (Mechanized Infantry Combat

Vehicle) was a lightly armored vehicle

designed to provide rapid cross-country

mobility, large volumes oP fire power from

orr'nic weapons, armored protection to a

( >i ' equipped rifle squad, and communicat ons between all elements of the unit.

It was designed to enable the rifle squad

to fight effectively in both mounted and

dismounted roles in offensive, defensive,

or retrograde formations.

2.3 The tests on the MICV were performed under

climatic conditions best described as

European winter thaw.

2.4 During a specific phase (i.e., fording, in

which is measured vehicle egress and

ingress ability at water crossings), time

constraints did not permit waiting for the

desired winter conditions. The intent was

to specifically address wetness effect on

system functions under winter thaw

conditions.

M.M People who participated in the present

study included six scout drivers, six tract

commanders, nine tank drivers, and nine

tank commanders. Infantry players received

both individual and squad training in their

duties in the MICV. The gunner training

developed the gunners to perform at a

predetermined safe level of proficiency,

though not at the level to perform combat

duty.

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

PrioritlM

6.3 Research Planning

A-445
U.S. Army Test and Evaluation Conmand. Materiel test procedure 3-3-521

Benning, GA: U.S. Army Infantry Board, May 1970. (AD-871 788).

Fort

I Topics Relevant

I to System Development

land Evaluation Technology ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Cooponei ts

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.? Performance

Requirements, Specific

3.|< Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

it.2 Parameter Determinations

1.3 Apparatus for Testing

l.t Personnel for Testing

1.5 Test Plans

Application Cocponents

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.'I Conclusions and

Recoomendations

2.1 Human factors engineering (HFE) is the application of scientific principles concerning human physical and psychological characteristics to the design of equipment, thereby increasing speed and precision of operations providing maximum maintenance efficiency, reducing fatigue, and simplifying

operations. HFE requires the consideration

of human characteristics such as separate

anthropometries, intellectual abilities,

sensory capacities, mobility, muscle

strength, basic skills, and the capacity to

learn new skills.

2.3 The study was conducted under a simulated

battle environment with the physical and

environmental conditions duplicating those

to be found in the equipment's future use.

3.1 Battlefield Mobility Within the

limitations imposed by the nature of the

equipment, physical design should provide

integral features and/or special provisions

to facilitate lifting and carrying by the

individual soldier with minimum loss of

efficiency. During conduct of common MTP

3-3-502, battlefield mobility, measurements

and observations will be made to determine

HFE design characteristics which enhance or

unduly limit the man-portability and

man-transportability of the equipment.

Some subjects for HFE consideration are

listed below. These will be explored as

appropriate for the particular test item:

Further Research Areas

6.1 Measurement Systea

Llnltations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-446
1 Topics Relevant

I to System Development 1 Topic

land Evaluation Technology ! No. ; ABSTRACT

(1) Weight, dimensions, and configurations

with respect to anthropometric characteristics of representative test

soldiers.

v (2) Method of carry and means to secure

during marches.

(3) Location, design, and texture of

gripping areas and carrying handles.

CO Location, configuration, and surface

texture of areas which, although not

specifically intended for the purpose,

may be used for lifting and carrying

heavy items.

•••

(5) Mutual interference with individual

equipment and clothing.

(6) Human limitations to endure strain,

fatigue, and discomfort while carrying.

(7) Effect on combat readiness of the

soldier.

(8) Handling characteristics during combat

movements which require the soldier to

run, jump, hit the ground rapidly,

roll, and assume various firing

positions.

(9) Capabilities and limitations for

carrying over all types of terrain.

(10) Ease of carry and freedom from interference while carrying in air and

ground vehicles.

(11) Ease of carry and delivery by

individual parachutists.

(12) Human capabilities and limitations

with respect to distance of carry.

(13) Weight, configuration, center of

balance, and load distribution of

items which are transported by two or

more men.

A-447
I Topics Relevant

S to System Development 1 Topic

land Evaluation Technology ! No. I ABSTRACT

(14) Compatibility of design for use with

standard individual load carrying

equipment.

(15) Packaging, protection, or other

preparation required to ready the item

for field transport.

1.3 Photographs, motion pictures, videotapes,

and fast frame photographs were used as

aids evaluating human engineering aspects

of the service test.

A-448
r>.

U.S. Army Test and Evaluation Command. Materiel test procedure 6-2-50?.

(Electronics Proving Ground). Fort Benning, GA: U.S. Army Infantry Board,

August 1969. (AD-720 976).

Topics Relevant

i to System Development I Topic I

land Evaluation Technology ! No. I ABSTRACT

s

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPU)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.U Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoonendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 Human engineering is defined as the design

of equpment, man-machine systems, and human

tasks for the most effective human accomplishment of the job. Such engineering

requires consideration of human characteristics such as anthropometries, intellectual

abilities, sensory capacities, mobility,

muscle strength, basic skills, and the

capacity for learning new skills. In

military human factors engineering, the

designer must consider human limitations

imposed by the environmental conditions

typical of military situations of use where

the operator is often working under stress

and fatigue.

1.3 The subtests given in this Materiel Test

Procedure (MTP) are performed on a

selective basis as required for a specific

item of equipment. The eight subtests and

their objectives are as follows:

(1) Control-Display Relationships - The

objective of this subtest is to

determine the degree to which the test

item design contributes to ease of

operation through incorporation of

preferred display and associated

control location relative to each

other and to operational

characteristics.

(2) Visual Displays - The objective of

this subtest is to determine the

suitability of visual displays

relative to type, size, location,

readability, consistency, and

operational characteristics.

A-449
1 Topics Relevant

I to System Development I Topic I

land Evaluation Technology I No. I ABSTRACT

(3) Auditory Warning Devices - The

objective of this subtest is to

determine the suitability of auditory

warning devices relative to human

factors aspects and operational

characteristics.

(M) Controls - The objective of this

subtest is to determine the

suitability of controls relative to

type, size, application, location,

coding, consistency, and operational

characteristics.

(5) Labeling - The objective of this

subt-est is to determine the suitability, readability, and consistency of

labeling used for critical markings,

identification, and instructions.

'.-

a

(6) Workspace Design and Layout - The

objective of this subtest is to

determine the suitability of workspace

relative to location, size,

accessibility, and configuration.

(7) Operator Comfort and Lack of Interference - The objective of this subtest is to determine if operator comfort and lack of interference aspects

are satisfactory.

(8) Special Observational Tests - The

objective of this subtest is to

determine the cause of special human

factors engineering problem areas

noted during some phase of equipment

operations or testing.

A-450
U.S. Amy Test and Evaluation Comnand. Materiel test procedure 10-2-505

(General equipment test activity). Fort Benning, GA: U.S. Army Infantry

Board, September 1971. (AD-729 855).

Topics Relevant

i to System Development

!and Evaluation Technology

.

State of th* Art Review

of the Process

1.1 General System

Heasurements

1.2 System Taxonomy

Model (SIM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.4 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 Experience has shown that the incorporation

of human engineering design criteria,

principles, and practices has improved

mission success through the integration of

the human into the system, subsystem,

equipment, or facility. Furthermore, the

cost effectiveness ratio has generally

improved as a result of the application of

optimun man-item design through increased

simplicity, improved safety of operations,

and reduced training and maintenance

requirements.

3.1 The attributes that were measured in this

study were noise, visibility, thermal

considerations, pressure considerations,

ventilation, vibration, and radiation.

4.3 The equipment that was required in this

study is as follows:

(1) Linear measuring devices, tape

measures, rules, etc.

(2) Scales, balances, and/or other

weighing devices.

(3) Temperature sensors.

(1) Pressure Sensors.

(5) Octave-band filter set.

(6) General purpose sound level meters.

(7) Light meters.

(8) Vibrometer.

A-451
I Topics Relevant I 1

I to System Development iTopic I

land Evaluation Technology ! No. ! ABSTRACT

.•-

(9) Radiation measuring equipment (as

applicable).

(10) Photographic equipiment.

(11) Gas sample set.

(12) Accoustical chamber with sound

pressure apparatus.

A-452
£

L

U.S. Department of the Army. Army training and evaluation program for

mechanized infantry battalion and combined arms task force (ARTEP 7-^5).

Washington, DC: Headquarters, Dept. of the Army, September 1975.

Topics Relevant

I to System Development

land Evaluation Technology ABSTRACT

6.

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STO)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.4 Performance

Criteria Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Pot:.itlals/

Priorities

6.3 Research Planning

1.1 This ARTEP serves as a basis for developing

a mechanized infantry battalion's training

and evaluation program. The program is

designed:

(1) To evaluate the ability of the

battalion to serve as the nucleus of a

combined arms task force performing

specified missions under simulated

combat conditions.

(2) To provide a guide of training

objectives by specifying minimum

standards of performance for

combat-critical missions and tasks.

(3) To evaluate the eff'ciency and

effectiveness of past training of all

echelons of the battalion from

crew/squad through battalion/task

force.

(4) To provide an assessment of future

training needs.

A-453
U.S. Department of the Army. Skill Qualification Test notice for MOS 11B SQT

3 infantryman (No. 7-11B3-N). Washington, DC: Headquarters, Dept. of the

Army, June 1976. (Pamphlet).

I Topics Relevant

I to System Development

land Evaluation Technology ABSTRACT

-

State or the Art Review

of the Process

1.1 General System

MeasL-rements

1.2 System Taxonomy

Model (STK)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.11 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Pr4 jrltlea

6.3 Research Planning

1.1 The Skill Qualification Test (SQT) tests a

soldler s ability to do those tasks in his

Military Occupational Specialty (MOS) that

are most important to his survival in

combat, to the accomplishment of his job,

and to his unit's mission. The SQT may

contain up to three parts: the written

component, the hands-on component, and the

performance certification component.

3.1 A soldier's ability to perform a task is

measured by the SQT.

A-454
.

U.S. Department of the Army. Tank gunnery (Field Manual 17-12). Washington,

DC: Headquarters, Dept. of the Army, March 1977.

1 Topics Relevant

I to System Development [TopicI

!and Evaluation Technology | No. ! ABSTRACT

1. State of the irt Review

of the J^ocess

1.1 General System

Heasureaents

1.2 System Taxonomy

Model (STM)

I.| Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3- Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

1. Planning Components

of the Process

t.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recomsendations

6. Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

2.1 The system under discussion in this manual

is tank gunnery and includes task

descriptions for all crew members—drivers,

loaders, gunners, and tank commanders.

2.2 The mission of this system was to score a

first-round target hit in the minimum

possible time.

3.1 Individual tasks for crew members were

presented. These tasks apply to nearly all

series and models of tanks.

3.2 Crew standards were presented for each

series and model of tank. Performance

standards included: time to fire, range,

degrees off-target for daylight hours and

darkness under artificial illumination.

Skill tests and scoring sheets for

performance tests are included in this

manual.

A-455
U.S. Department of the Army, Office of the Chief of Staff, Tank Forces

Management Group. Tank weapon system management. Washington, D.C.

i Topics Relevant

I to System Development I Topic

{and Evaluation Technology ! No. ABSTRACT

State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.IJ General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.') Performance

Criteria, Specific

3.5 Measurement Procedures.

Planning Components

of the Process

4.1 Analytic Methods

<l.2 Parameter Determinations

i|.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoosendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

2.1 The various factors that go into the

optimization of tank weaponry effectiveness

were broken down into subsystems and

analyzed regarding the impact of each

subsystem - and subsystem integration - on

the combat capability of the total tank

weapons system. Subsystems were defined

into personnel, logistics, training,

development, and management components and

are examined in turn.

3.1 The characteristics of each subsystem

to be measured were as follows:

(1) Personnel Management System,

consisting of training, distribution,

sustainment, separation, and

management components.

(2) Logistics System, consisting of

fixing, arming, fueling, and managing.

(3) Training, consisting of entry-level

individual training, collective and

combined arms training, reserve

component, training management, tank

distribution/procurement requirements,

and organization/doctrine.

(4) Tank Forces Management System,

including science and technology base,

tank system development, and resource

allocation.

3.2 Units of measurement, of course, differ

between subsystems. The personnel

subsystem consisted of processes directed

toward the procurement, training,

utilization, separation, development, and

motivation of military personnel. The

training requirements of the personnel

A-456
Topics Relevant I

to System Development I Topic!

and Evaluation Technology i Mo. ! ABSTRACT

anagement system are separated into basic

combat training, advanced individual

training, and one-station unit training.

The distribution aspect involving the

proper mix of personnel to fill field

requirements was assessed by:

(1) Properly identified and timely

submitted requirements by units.

(2) Accurate authorir.ations and personnel

inventory data bases.

(3) Support at all command levels through

the reduction of diversions.

The sustainment component of the Personnel

Management System (to a^v.lop and maintain

the career force to beat meet the readiness

requirement) was assessed through the

Enlisted Force Management Plan (EFMP),

which provides both qualitative and

quantitative goals for the Army for the

period FY 73 - FY 82.

The logistics subsystem, which seeks to

optimize the interaction of functional

equipment, trained personnel, and

responsive support, was assessed through

the:

(1) Fixing or supply and maintenance

system (wholesale and retail logistics

doctrine and procedures).

(2) Arming requirements for the tank

force, e.g., ammunition resupply

rates, transportation requirements for

bulk cargo from entry points to

ultimate user.

(3) Fueling supply, e.g., what

organizations at what level fulfill

this function and what report/request

procedures exist to support it.

A-457
Topics Relevant

I to System Development {Topic!

land Evaluation Technology ! No. 1 ABSTRACT

5.3

-

Assessment of the training subsystem is

made through evaluation of the Army's

service schools and training centers,

training materials and extension courses,

operational unit-field training, and

simulated combat training.

Finally, the tank forces management system

is based on an analysis of system

characteristics, levels of decision,

material acquisition analysis, and

operational and development testing.

The findings of this analytical report have

direct relevance for the primary mission of

the Army Tank Development Program:

equipping armor units with the best

available tank in sufficient numbers to

counter the threat:

(1) There currently exists no viable tank

development/procurement strategy in

the Army.

(2) There is no single coordinating agency

within the U.S. Army Material

Readiness and Development Command

(DARCOM) in charge of integrating all

tank relate^ science and

technology-based programs.

(3) Documents in current use do not

identify user tank development

priorities.

(M) Control of tank programs i»

fragmented.

(5) Tank project managers fail to initiate

early planning in the development

cycle.

(6) Responsible Army staff agrncies are

not reviewing logistic plans prinr to

program milestones.

A-458
I

Van Acker, A. & Wohl, J.G. Modeling the sonar operator's detection process:

A progress report. In G. W. Levy (Ed.), Symposium on applied model of

aan-aachine systems perforruance (NR69H-591). Columbus, OH: North American

Aviation, November 1968. (AD-697 939).

I Topics Relevant !

I to System Development I Topic

i and Evaluation Technology ! No. ABSTRACT

1. State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Componencs

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.'< Performance

Criteria, Specific

3.5 Measurement Vrocedures

4. Planning Components

of the Process

1.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

5. Application Cocponents

of the Process

5.1 Test Execution

5.2 Data Analys s

5.3 Findings Interpretation

5.4 Conclusions and

Reconmendations

6. Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.3 To assist in developing sonar operator

performance standards, models representing

different aspects of an operator's

activities are being developed and

implemented as computer programs. This

paper discusses the search and detection

performance model. The overall simulation

model includes three submodels:

(1) The geometric submodel (which controls

movement of the target submarine).

(2) The physical world submodel

(controlling sonar signals,

reverberat-ons, noise, etc.).

(3) The decision-maker submodel (which

simulates the operator's search and

detection processes and procedures).

2.M The decision-maker submodel involves

certain basic assumptions:

(1) The operator fixates on the CRT at a

series of points in consecutive

sequence, and a target can be seen

only when fixating.

(2) A specified scanning policy is

followed in regard to fixation.

(3) Only targets (not noise) will cause

interruption of the scan policy.

A-459 ) n
Topics Relevant

to System Development !Topic I

and Evaluation Technology j No. i ABSTRACT

•

(4) Probability of glimpsing a target is a

function of retinal position and

brightness contrast ratio.

(5) Probability of glimpsing a target also

is a function of an internal criterion

used by the operator (which depends on

his conservatism, short-term memory,

data handling capacity, the noise

level on the PPI, as well as some

psychophysical limitations).

(6) The decision maker decides that a

target is present if his last glimpse

of the target is foveal and if he has

glimpsed it a specified number of

times out of the last group of pings

(called the detection announcement

factor).

4.2 As variable inputs, the decision-maker

submodel has the following:

(1) A scanning policy.

(2) Fixation time.

(3) Glimpse criterion factor (the

operator's internal criterion).

(4) Detection announcement factor.

5.3 Early results with the submodel indicated

that fixation time is not a highly critical

factor to detection range, and that various

regular search patterns produced little

statistical differences. However, the

sonar performance was extremely sensitive

to the operator's internal glimpse

criterion, and also to the detection

announcement factor.

6.1 The basic detection model Involves only the

video sonar scan and applies only to a

small proportion of the situations

encountered in practice (viz., isothermal

water and non-maneuvering targets). Work

is underway to extend the model and remove

these constraints.

A-460
s

Von Winterfeldt, D. An overview. Integration and evaluation of utility

theory for decision analysis (SSR1 Research Rep. 75-9). Los Angeles, CA:

University of Southern California, Special Science Research Institute, August

1975. (AD-A021 497).

T

•-:

Topics Relevant

! to System Development iTopic

land Evaluation Technology ! No. ABSTRACT

1. State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1-3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.<l General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

i.t Performance

Criteria, Specific

3.5 Measurement Procedures

4. Planning Components

of the Process

1.1 Analytic Methods

1.2 Parameter Determinations

«.3 Apparatus for Testing

I.« Personnel for Testing

«.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.* Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

2.1 Utility theory is a division of measurement

theory that assigns numbers to objects by

indexing them to a decision-maker's values.

Utility theory, however, unlike measurement

theory in general - differs by measuring

objects of cost or value, relates objects

by means of preferences, while operations

Involving these objects are either missing

or are made through surrogate objects.

2.2 The goal of this analysis of utility theory

is in its application to certain decision

situations that may be classified according

to the following factors:

(1) Static versus dynamic decision

environment.

(2) Single decision maker versus multiple

decision makers.

(3) Single aspect choice entity versus

multiple aspect choice entity.

3.1 Specifically, five model classes of utility

theory are discussed: "weak orders",

"difference measurement", "bisymmetric

measurement", "conjoint measurement", and

"expected utility measurement".

3.2 The weak order measurement model is applied

in cases of multiple affected individuals

and assumes transitivity of preferences.

A-461
y. Topics Relevant

I to System Development I Topic I

land Evaluation Technology i No. ABSTRACT

Difference models compare the relative

differences of the strength of preference

between pairs of choice entities, and uses

"operational surrogates".

The idea behind bisymraetric measurement is

to find an element "c" that bisects "a" and

"b", i.e., the objective is that "c" has

the average utility of "a" and "b"

numerically speaking.

Conjoint measurement models are used for

measuring utilities over several choice

entities that take on different value

configurations.

Expected utility theory makes three common

assumptions about preferences among risky

choice entities:

(1) Risky alternatives can be ordered

transitively.

(2) If events have common outcomes, then

preferences among risky alternatives

should be independent of those events.

(3) There are equivalents for all possible

uncertain entities.

5.M Decision analysts may improve their

model-building by using utility theory to

define problem structure, model forms a.id

possible errors in models. However,

utility theory is not appropriate for scale

construction.

A-462
: :

Vreuls, D., Wooldridge, A.L., Obermayer, R.W., Johnson, R.H. & Norman, D.A.

Development and evaluation of trainee performance measures In an automated

Instrument flight maneuvers trainer (NAVTRAEQUIPCEN 74-C-0063-1). Westlake

Village, CA: Canyon Research Group, Inc., October 1975. (AD-AOaU 517).

I Topics Relevant

I to System Development

land Evaluation Technology ABSTRACT

1. State of the Art Review

of the Process

1.1 (ieneral System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3-3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

1. Planning Components

of the Process

1.1 Analytic Methods

1.2 Parameter Determinations

1.3 Apparatus for Testing

1.1 Personnel for Testing

1.5 Test Plans

5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 findings Interpretation

5.1 Conclusions and

Recommendations

6. Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Prlorltlaa

6.3 Research Planning

2.1

2.2

2.4

3.1

and

3.2

This system Is an automated flight training

system consisting of an Instrument flight

maneuvers flight simulator modified to

operate with three measurement systems.

The system was configured as a high

performance fighter aircraft (F-UE) capable

of four Instrument flight maneuvers. The

flight conditions selected are straight and

level flight, standard rate climbs and

descents, level turns and climbing and

descending turns.

The system consists of an F-4E cockpit

mounted on a four-degree of freedom motion

platform. It contains all controls and

displays of the jet fighter cockpit except

radio navigation, communications and

weapons control.

The following characteristics and units of

measure were used In this study:

PARAMETER UNITS

(1) System Clock Count

(2) Elevator Stick Force Pounds

(3) Elevator Stick Dis- Inches

placement

(1) Angle of Attack Units

A-463
I Topics Relevant

! to System Development I Topic

land Evaluation Technology I Ho. ABSTRACT

(5) Pitch Attitude Degrees

(6) Climb/Descent Rate Feet per Min

(7) Altitude Feet

(8) Right Throttle Displacement

Degrees

(9) Airspeed Knots

(10) Aileron Stick Force Pounds

(11) Aileron Stick Displacement

Inches

(12) Roll Attitude Degrees

(13) Turn Rate Degrees per

Second

(lU) Heading Degrees

(15) Rudder Pedal Force Pounds

(16) Rudder Pedal Displacement

Inches

(17) Sideslip Degrees

(18) Turbulent Air

Intensity

Arbitrary Units

3.5

Initially, 16 of these measures were

selected for use. Correlation analysis of

redundant information reduced the 16 to 12.

Multiple discriminant analysis produced

nine measures which could be weighted and

sunned into a single score.

All data were recorded on magnetic tape at

the rate of five times-per-second in real

time for each full training run.

A-464
i Topics Relevant

! to System Development

land Evaluation Technology ABSTRACT

M.1 Correlation and multiple discriminant

analyses were used on the collected

measurement data.

M.2 The controlled test parameters were air

turbulence set at "light" for this test,

aircraft weight was either light or heavy

and a center-of-gravity shift of 29.0 to

30.2 percent mean aerodynamic chord.

4.3 The test equipment was the Training Device

Computer System (TRADEC) located at the

Naval Training Equipment Center, Orlando,

Florida. It was configured as a fixed wing

aircraft (F-UE) on a four-degree of freedom

motion platform.

U.i» The test subjects were twelve low-time

civilian student and private pilots.

5.1 The major conclusion and recommendation of

the study was that the discriminant model

should be applied to the problem of

specifying measures for future flight

training systems. In addition, a series of

individual conclusions and recommendations

were presented.

A-465
Waag, W.L., Eddowes, E.E., Fuller, J.H. 4 Fuller, R.R. ASUPT automated

objective performance measurement system (AFHRL-TR-75-3). Williams AFB, AZ:

Air Force Human Resources Laboratory, Flying Training Division, March

1.975. (AD-A014 799).

I Topics Relevant

! to System Development

land Evaluation Technology ABSTRACT

1. State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

4. Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recomaendations

6. Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.3 The implementation of the measurement

system requires:

(1) Definition of criterion objectiveT in

terms of a candidate set of simulated

parameters.

(2) Evaluation of the proposed set of

measures for the purpose of validation

and simplification.

(3) Specification of criterion performance

by requiring experienced instructor

pilots to fly the particular maneuver.

(U) Collection of normative data using

students as they progress through the

training program.

2.1 The Advanced Simulation in Undergraduate

Pilot Training (ASUPT) facility is designed

to be a research device capable of

providing answers regarding the hardware

design and effective use of flight

simulators.

2.U A criterion referenced approach to

measurement system development was pursued

under the following constraints:

(1) Measures will assess the degree to

which the criterion objectives are

met.

1

A-466
v

.

Topics Relevant

| to System Development I Topic!

land Evaluation Technology ! No. ! ABSTRACT

(2) Measures will reflect only the most

salient characteristics of

performance.

(3) Measures will be meaningful and

interpretable to the user — the

student and instructor pilot.

(M) Measures will be generated on a

real-time basis so that feedback is

imnediate.

3.1 Those attributes measured in this study

were:

Altitude

Airspeed

Heading

Stick movement

Throttle movement

Elevator stick force

Pitch rate

Pitch acceleration

Roll rate

Roll acceleration

Vertical velocity

Vertical acceleration

4.1 The statistics in this study were

calculations of the mean, mean root square,

and the standard deviation.

A-467
Weapon System Effectiveness Industry Advisory Committee. Final report of

task group II. Vol. 1 Prediction-measurement; Summary, conclusions and

recommendations (AFSC-TR-65-2). 1965. (AD-USS 454).

Topics Relevant |

I to System Development I Topic I

land Evaluation Technology j No. I ABSTRACT

State of the Art Review

of the Process

1.1 General Systec

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.k Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Coaponents

of the Process

5.1 Test Execution

5-2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recomendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 The purpose of the Weapon System

Effectiveness Industry Advisory Committee

was to provide technical guidance and

assistance to Air Force System Command in

the development of a technique to apprise

management of current and predicted weapon

system effectiveness at all phases of

weapon system life.

1.3 It was noted that system effectiveness

evaluation/prediction can be reduced to an

ordered set of tasks as follows:

(1) Mission Definition

a. Functional description of

purpose

b. Quantitative requirements

(2) System Description

a. General configuration

b. Block diagram

c. Time line analysis

(3) Specification of Figures of Merit

CO Identification of Accountable Factors

a. Level of accountability

b. Personnel characteristics

o. Procedure characteristics

A-468
Topics Relevant ! I

I to System Development 1 Topic I

land Evaluation Technology ! No. I ABSTRACT

2.1

3.1

d. Hardware characteristics

e. Logistics

f. Data constraints

(5) Model Construction

a. Assumptions

b. Delineation of possible mission

outcome

c. Delineation of significant system

states

(6) Data Acquisition

a. Specification of data elements

b. Specification of test methodology

c. Specification of data reporting

system

(7) Parameter Estimation

(8) Model Exercise

a. Numerical evaluation of

effectiveness and its factors

b. Comparative system analysis

c. Parameter variation study

The examples of Volumes II and III adhere

rather closely to this analysis of the

steps required to achieve a system

sffectiveness evaluation/prediction.

The life cycle of a system was divided into

the conceptual, definition, acquisition,

and operational phases.

System effectiveness was defined as the

function of a system's availability,

dependability, and capability.

A-469
•

I Topics Relevant I

to System Development I Topic I

and Evaluation Technology ! No. I ABSTRACT

U.I

Availability was a measure of the system

condition at the start of the mission. It

was a function of the relationships among

hardware, personnel and procedures.

Dependability was a measure of the system

condition at one or more points during the

mission given the system condition(s) at

the start of said mission.

Capability was a measure of the ability of

the system to achieve the mission

objectives, given the system condition(s)

during the mission. Capability accounts

for the performance spectrum of a system.

No specific statistics were given.

However, the procedure went as follows:

effectiveness was the product of

availability, dependability and capability.

To provide for the treament of the various

conditions, these three variables were

expressed as a vector or matrix. In highly

complex systems, when matrices were

impractical, an analog or digital computer

simulatioii was employed.

A-470
Weapon System Effectiveness Industry Advisory Committee. Final report of task

group II. Vol. 2 Prediction-measurement; Concepts, task analysis, principles

of model construction (AFSC-TR-65-2), 1965. (AD-158 U55).

Topics Relevant

I to System Development

land Evaluation Technology ABSTRACT

SUte of the Art Review

of the Process

1.1 General Systetc

Heasureaents

1.2 System Taxonomy

Mode: (STW)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Coaponents

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Crlttria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.*! Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

«1.1 Analytic Methods

«.2 Parameter Determinations

«•3 Apparatus for Testing

«.4 Personnel for Testing

I.I Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoomendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Prioritiea

6.3 Research Planning

1.3 This document is part of the final report

of the Weapon System Effectiveness Industry

Advisory Committee (WSEIAC). The purpose

of the Weapon System Effectiveness Industry

Advisory Committee was to provide technical

guidance and assistance to air force system

command in the development of a technique

to apprise management of current predicted

weapon system effectiveness at all phases

of weapon system life. Four systems were

illustrated including an airborne avionics

system, an intercontinental ballistic

missile system, a long range radar

surveillance system, and a spacecraft

system. The four phases of system life

(conceptual, definition., acquisition,

operation) were discusaed as well as the

eight tasks used to evaluate system

effectiveness.

(1) Mission definition

(2) System description

(3) Specification of figure(s) of merit

(U) Identification of accountable factors

(5) Model construction

(6) Data acquisition

(7) Parameter estimation

(8) Model exercise

A-471
Topics Relevant

I to System Development I Topic I

!and Evaluation Technology j No. I ABSTRACT

2.1 The long range radar surveillance system

consisted of two transmitters in parallel,

an antenna, a receiver, a display and

synchronizer, and an operator.

3.3 The system should detect target aircraft

above the horizon line of sight at ranges

up to 200 miles and, while the target is

within this maximum range, track it in

range and azimuth within admissible error.

t

A-472
Weapon System Effectiveness Industry Advisory Committee. Final report of

task group II. Vol. 3 Prediction-measurement; Technical supplement

(AD-U58 U56).

task group II. Vol.

(AFSC-TR-65-2). 19^

Topics Relevant

I to System Development !Topic

land Evaluation Technology ! Wo. ABSTRACT

i.

2.

3.

Statt of the Art Review

of the Process

1.1 General Systea

Heasurenents

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CTM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.14 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.4 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

1.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

1.1 The purpose of the Weapon System Effectiveness Industry Advisory Committee was to

provide technical guidance and assistance

to air force system command in the

development of a technique to apprise

management of current and predicted weapon

system effectiveness at all phases of

weapon system life. The objective of Task

Group II was to review existing documents

and recommend uniform methods and

procedures to be applied in predicting and

measuring systems effectiveness during all

phases of a weapon system program.

2.1 This report is concerned primarily with

four examples of effectiveness evaluation,

involving the following systems:

(1) Avionics system in a tactical fighter

- bomber.

(2) Squadron of ICBMs.

(3) Fixed radar surveillance and threat

evaluation system.

(4) A spacecraft system.

Each system is evaluated at a different

phase of development.

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

RecooBendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentiale/

Priorities

6.3 Research Planning

A-473
Topics Relevant

I to System Development I Topic I

!and Evaluation Technology ! No. j ABSTRACT

3.1 The specific basic analytical model

proposed by Task Group II is, symbolically.

E = A[D]C

Where E = system effectiveness

A Availability

[D] = Dependability

C = Capability

The basic model is not restrictive;

variations on the basic model are

illustrated in four examples.

A-474
Weinstock, G.D., Douglas, M. 4 Blom, B. Development of criteria and measures

of effectiveness for U.S. Army tactical communications systems (Tech. Rep.

ECOM-5012-1). Paramus, N.J.: Communications Systems, Inc., May 1969. (AD-881

1U5).

Topics Relevant

to System Development

and Evaluation Technology

Topic

No. ABSTRACT

Statt of th« Art Review 2.1

of the Process

1.1 General System

Heasureaents 2.2 1.2 System Taxonorcy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM) 2.5

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition 2.6

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process 3.1

3.1 Practical Measurable

Attributes

3.2 Practical Attribute 3.2 Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific 3.3

3.5 Measurement Procedures

Planning Components

of the Process

«.1 Analytic Methods

1.2 Parameter Determinations «.1

4.3 Apparatus for Testing

1.1 Personnel for Testing

«.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Rec oonenda 11ons

This study addressed competing

communications systems.

The purpose of these systems is to transfer

information between two separate locations.

The requirement is that the information is

received and that the information reaches

its destination.

The ultimate criteria are that the

information is understood within

"acceptable boundaries of quality or error

rate" and that It reaches its destination

in a timely fashion.

The measure of benefit has two dimensions

— information and time.

Information flow and information capacity

can be measured mathematically.

Specific performance requirements of this

system include: transportability,

mobility, capacity, quality of service,

survivability, vulnerability.

An integrated system effectiveness model

was developed in this study which is

capable of providing a single explicit

measure of system effectiveness when

evaluating competing communication systems.

This model is based on statistical analysis

and the probability of successful

communications derived from queuing

relationships.

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-475
I Topics Relevant i I

1 to System Development [Topic!

land Evaluation Technology ! No. ! ABSTRACT

^.4 The approach was broken down into four

tasks:

(1) Development of a comprehensive listing

of criteria and associated performance

factors.

(2) Establishment of the format for three

matrices: (a) assignment of mission to

conflict intensity by frequency of

occurrence and relating allocation of

resources to conflict intensity; (b)

identification of tactical functions;

and (c) relating quantitative

communications requirements by

communication nets to tactical

functions.

(3) Development of quantitative relationships between the criteria and

measures of effectiveness.

(4) Use of the above items to show the

procedure for measuring effectiveness

of the proposed system.

5.4 The conclusion of this report states that

it is desirable at this time to test the

model and evaluation concept by means of a

not too complex test problem. It is

believed that a test problem will establish

confidence in the model and provide

insights into the model's operation.

A-476
Weisz, J.D. System analysis; Manpower resources/system design Integration

ethodology (Tech. Note 9-68). Aberdeen Proving Ground, MD: Human

Engineering Laboratories, August 1968. (AD-675 ' SD.

Topics Relevant 1

! to System Development 1 Topic 1

land Evaluation Technology ! Ho. ! ABSTRACT

State of the Art Hevlew

of the Process

1.1 General Syatea

Heasureaents

1.2 Systen Taxonomy

Model (SIM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Cooponents

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environaent Definition

2.1) General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

Planning Components

of the Process

O Analytic Methods

k.2 Parameter Determinations

1.3 Apparatus for Testing

1.« Personnel for Testing

^.5 Test Plans

Application Coaponents

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.» Conclusions and

Recommendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Frioritles

6.3 Research Planning

1,1 General methodology is introduced which,

when used appropriately by decision makers,

may assist them in deciding which system

concepts among several to approve for

further development. This technical note

attempts to establish a general framework

around which manpower factors can be

effectively introduced into system analyses

studies.

It is stated that with expert inputs from

specialists in training, selection and

utilization of personnel, and human

performance, a human factors team should be

able to provide an estimate of which

concept from the human factors viewpoint

will best fulfill system performance

requirements at least total cost.

It is suggested that to successfully

complete the manpower factors part of the

systems analysis, a constantly updated data

bank of manpower resources should be

available. The data bank would include

training time and cost data on all past

fielded systems, and a skill-level breakout

of the present Army performance data in the

areas of vision, tasks, etc.

Experts in the area of training and job

analysis should, with this Information, be

able to estimate the skills required for

each concept being considered. If the

estimates are detailed enough, sufficient

sensitivity should exist to differentiate

among the concepts being considered.

A-477
i Topics Relevant

I to System Development

!and Evaluation Technology ABSTRACT

In addition, appropriate data in the data

bank would enable estimates to be performed

as to whether a person can or cannot

perform the functions required in each

concept.

Following the initial system-concept

comparison, trade-off studies could be

conducted. The human factors team could

assist in deriving these data in various

forms: cost, estimated amount of down time,

system performance change and personnel

skill-requirements change.

[

o A-478
Wellman, L.N. & Neill, W.H. The development of performance measurement

standards for the USAF base level supply system (SLSR-12-72B, Master's

Thesis). Wright-Patterson AFB, OH: Air Force Institute of Technology,

September 1972. (AD-750 J12).

I Topics Relevant !

I to System Development I Topic I

land Evaluation Technology i No. I ABSTRACT

State of the Art Review

of the Process

1.1 General Syste«

Measurements

1.2 Systei Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPU)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Enviroment Definition

2.<i General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Perfonnance

Criteria, Ultiute

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.4 Performance

Criteria, Specific

3.5 Measureaient Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Re c ommenda ti ons

2.1 The system addressed in this thesis is the

Standard Base Supply System and its

associated personnel.

2.2 The Base Supply System provides support to

aircraft maintenance.

2.5 The ultimate performance requirement of

this system is that the base supply be

controlled in the most cost-effective

manner to ensure that resources are used

efficiently and effectively.

2.6 The current system has no set of standard

measurement areas to indicate how well the

system is supporting its objectives.

3.1 In order to determine which attributes

should be measured, an open-ended

questionnaire, accompanied by a letter of

explanation of the research project, was

sent to the selected "experts." The

response from this questionnaire resulted

in the development of a "follow-on"

questionnaire. From this information a

consensus of opinion on the measurable

attributes of importance was obtained.

3.5 The measurement tool utilized in this study

was the Delphi technique. To determine a

consensus, the mode response was chosen as

It is the response which occurs most often.

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-479
Topics Relevant

! to System Development

land Evaluation Technology ABSTRACT

M.I The Delphi technique was utilized to induce

opinion convergence through a sequence of

questionnaires using controlled feedback to

the participants.

4.M It was determined that an appropriate

sample size for the study would be 35. Two

groups were selected composed of supply

experts and experts from the maintenance

career field. The following criteria were

used as a basis for selection:

a. Stature of the individual in his

career area.

b. Breadth of his experience.

c. Degree of varied assignments with

different major commands.

d. Level of assignment from base, major

commands and Air Force levels.

5.U A consensus was obtained on seventeen

measured areas from the panel of experts.

This initial research indicates that the

Delphi technique can be effectively used

for design of some elements of a management

control system, but there are limitations

which should be considered.

A-480
«-•

Wherry, R.J. The development of sophisticated models of man-machine system

performance. In G.W. Levy (Ed.), Symposium on applied model of man-machine

systems performance (HR69H-591). Columbus, OK: North American Aviation,

November 1968. (AD-697 *39).

I Topics Relevant

I to System Development

land Evaluation Technology ABSTRACT

1. State of the Art Review

of the Process

1.1 General Systeo

Measureaents

1.2 System Taxonomy

Model (STP)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Coaponents

or the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environaent Definition

2.4 General Constraints

2.5 Perforaance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

4. Planning Components

of the Process

4.1 Analytic Methods

I.I Parameter Determinations

4.3 Apparatus for Testin«

4.4 Personnel for Testing

4.5 Test Plans

.5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoomendations

6. Further Research Areas

6.1 Measurement Syste--

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

1.1 This paper is concerned with the

development of models which will aid in the

design, development, test and evaluation of

military systems, equipment, and

facilities. Too often, design approaches

that are taken seem to be based on

"clinical" or intuitive judgments without

formally stated assumptions. Whenever

assumptions are not formally stated there

is an increased chance that erroneous

assumptions will not be recognized as such.

There is a need, as model builders, to

formalize assumptions to be sure that they

are not inconsistent.

1.3 This paper examines the feasibility of far

more sophisticated digital computer

programs [models] than currently exist. It

is possible to distinguish several classes

of models which would be extremely useful

if they (a) existed and (b) worked. These

include:

(1) Instrument design/evaluation programs.

(2) Hunan operator simulator programs.

(3) Optimization programs.

(M) Training requirements programs.

3.1 Major subsections of human operator

simulators are identified:

A^81
I Topics Relevant I

i to System Development I Topic

land Evaluation Technology ! No. ABSTRACT

(1) Individual Differences—The model

should have the option of reading in a

profile of abilities so that a system

could be tested .0 determine its

sensitivity to various operator

states.

(2) Procedures Procedures for the

simulated operator must be oriented to

the "perceptual" level and stated in

terms of desires rather than

imperatives. The model can never

positively require [or guarantee] that

the operator will absorb any

particular information or make any

particular control manipulation. The

locus for decisions to actions resides

in the operator, not in external

commands or procedures.

4.3 The use of a Human Operator Procedures

(HOPROC) language for Human Operator

Simulator Programs is discussed.

rI

A-482
Williams, G.S. A methodology to establish the criticality of attributes in

operational tests (Master's Thesis). Georgia Institute of Technology, 1975.

(AD-A02M 199).

Topics Relevant

1 to System Development

land Evaluation Technology iTopic

No. ABSTRACT

P

State of the Art Review 1 .1

of the Process

1.1 General Systen

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Perfomance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, ^r-1 Tic

3.5 Measurement Procedures

Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Appl ication Components

of the Process

5.1 Teat Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoaendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

This thesis addresses a method which

provides a basis for the selection of

critical attributes which best discriminate

between acceptable and unacceptable systems

in order to facilitate the selection of

measures of effectiveness.

The author states that current test

structure in operational testing is not

amenable to the standard application of

multivariate statistics. The methodology

developed in this thesis encompasses a

means to combine results from past tests

with subjective information to determine,

the relationship, in terms of covariances

between two attributes. This information

is incorporated with subjectively obtained

acceptable and unacceptable mean vectors in

stepwise discriminant analysis.

It is concluded that multivariate

techniques may be a valuable aid in

determining which attributes contribute

more in distinguishing between successful

and unsuccessful systems. It is also

concluded that current test design for

operational testing can be modified to

facilitate a broader use of multivariate

statistical analysis techniques. This

modification should permit the correlation

among attributes to be rOjectively

determined and the marginal normality of

observations for each attribute to be

validated.

A-483

.-.

. .... -' ^"

. .- . .i.-...•->-.....-•-.-•
Williams, H.L. Dependent models for estimating human performance

reliability. In R.E. Blanchard, & D.H. Harris, (Eds.), Man-Liachine

eff ctiveness analysis. Papers presented at The Human Factors Symposium.

University of California at Los Angeles, June 15, 1967. (AD-735 718).

I Topics Relevant

! to System Development

land Evaluation Technology

Topic!

No. I ABSTRACT

1. State of the Art Review

of th« Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environnent Definition

2.H General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3- Analytic Components

of the Process

3-1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

Z.k Performance

Criteria, Specific

3.5 Measurement Procedures

4. Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoonendatlons

1.1 Tasks performed by human operators,

technicians, and ground crews in assembly,

test, and handling frequently have a

significant <»;'fect on the efficiency of a

weapon system. In assessing system and

design feasibility, one must also determine

the reliability of human performance.

1.3 Methods developed for estimating human

performance reliability require that the

human task be divided into discrete steps,

and that a probability model be fitted to

the task. Probability values of successful

performance are estimated for each step in

the model, and the success probability for

the total task is then computed.

If the task's discrete steps are

independent, human performance reliability

can be estimated without undue difficulty.

Data stores exist which provide marginal

probabilities for independent task steps

(e.g., American Institute for Research data

store). One finds that the great majority

of operational procedures tasks encountered

break down into dependent steps. The

combination of dependent relationships

usually is unique and the analyst finds

that neither data nor procedures for

estimating the dependent probabilities are

available to him. One must conclude that

the conditional probabilities of a model

composed of dependent events will not be

found in a data store.

Further Research Areas

6.1 Measurement S; stem

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

A-484

•. -

------ - - - -.^-. - ^^v..;,:. »-• - - -
I Topics Relevant

! to System Development

land Evaluation Technology ABSTRACT

6.2 The problem of estimating conditional

probabilities of dependent task steps can

only be solved by developing transition

models that make the transition from

the marginal probabilities of the data

store to the conditional probabilities of

the dependent relationships. Although much

work remains to be done, one can determine

the general form of the transition models

by using the techniques of experimental

design and analysis. Two major problems

that must be solved before there will be

significant progress in developing

transition models are (1) factors

responsible for dependent relationships

among task steps must be fully identified,

and (2) effects of dependent relationships

must be determined.

A-485

- . . . -^ • •-- ^_^ aSL^ ^-.
Williams, R.L., Long, T.W. & Windholz, W.M. A standardized munitions

effectiveness methodology {K-75-57 R). Colorado Springs, CO: Kaman

Sciences Corporation, July 1975. (AD-B009 483).

Topics Relevant

I to System Development I Topic I

land Evaluation Technology ! No. I ABSTRACT

1. State of the Art Review

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

4. Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recoamendations

6. Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Priorities

6.3 Research Planning

2.1 This report develops a practical munitions

effectiveness methodology relevant to five

types of munitions: artillery, tank

ammunition, rockets, mortars and mines.

Since no formalized procedure exists for

making comparative munition effectiveness

analyses, the authors here attempt to

standardize the effectiveness evaluation

procedure.

2.2 The ultimate goal here is development of a

standardized formula applicable to munitions effectiveness analyses in general,

based on three general components: definition of weapon purpose, identification and

measurement of performance and "pain"

(costs of achieving a given level of performance), and determination of the exchange relationships among quantities

represented by the above concepts.

2.4 The effectiveness methodology developed is

applicable to the five classes of weapons

mentioned above, applies in all types of

weather and in all terrains.

2.5 The ultimate requirement of this munitions

effectiveness model is the determination of

the expected fraction of the target

destroyed, which is the formal definition

of munitions effectiveness (E).

A-486

- -- - - - - - - ^ - - . - , -- .
. .-,

Topics Relevant I

I to System Development 1 Topic 1

land Evaluation Technology I No. i ABSTRACT

2.6 For a single round, munitions effectiveness

is defined by the equation E = ARE(c), where

3.1

and

3.2

t

.

-

A

R

E(c)

Availability

Reliability and

Effective coverage AB158

Attributes of the munitions effectiveness

concept and their operationalization, where

quantifiable, are as follows:

(1) Safety: concerned with personnel

safety and the possibility of

premature detonations.

(2) Reliability: assessment of constituent elements of the munition in terms

of performance, both human and weapon;

also includes concepts of success and

failure.

(3) Availability: includes logistic

integrity of personnel - weapons

system; manufacture, availability of

raw materials, availability of plant

capacity, shipping, storage, and

transport to the unit o*" utilization,

and ability of weapon to survive troop

handling; also Includes concepts of

dependability, durability, ease of use

and training.

(4) Lethality: capability, arming and

delivery accuracy, kill mechanisms of

overpressure, shock, penetration or

fragmentation, etc.

..

(5) Range: treated by assessing overall

effectiveness as a summation of

effectiveness at specific targets at

different ranges weighted by the

probability of frequency of

occurrence.

(6) Flexibility: the degree to which a

design can be modified for utilization

in different procedures.

A-487

.v.-..

------ ^ M
Topics Relevant

! to System Development

land Evaluation Technology ABSTRACT

(7) Countermeasures: enemy tactics which

affect target definition. Examples

include armor plating, reenforced

bunkers, etc.

(8) Utilization Rate: rate of fire.

(9) Weight: of weapon.

(10) Firing Doctrine and Tactics:

effectiveness of individual

subcomponents of man-weapon system.

(11) Complexity: simplicity and ease of

use are adversely affected by

complexity.

(12) Training and Documentation: the

likelihood of successful performance

of necessary tasks, degree of

training, etc., i.e., the human

element.

(13) Cost: referred to as the "pain."

U.I A sample effectiveness evaluation for the

Lightweight Company Mortar System is given:

the specific steps involved in the

evaluation are as follows:

(1) Step 1:

158

Define Munition Purpose.

(2) Step 2: Define the Mission: general

description, functional examination

(success criteria) for each

component/module/assembly.

(3) Step 3: Create the Availability/Reliability Event Sequence and General

Organizational chart.

(4) Step 4: Determine Numerical Estimates

for Events/Components (Reliability

Allocation and Prediction).

A^88

^-^iÄ ••'-••••-- - ------
:: I Topics Relevant

i to System Development I Topic!

and Evaluation Technology ! No. I ABSTRACT

(5) Step 5: Calculate Availability/Reliability Estimates.

(6) Step 6: Obtain Target and Lethality

Data.

(7) Step 7: Exercise Effectiveness

Computer Model.

.

A-489

vv-v-.. M
Willis, J.E. Feasibility of the development and utilization of personnel

performance effectiveness measures for man/machine function allocation

decision (Research Memo. SRM 68-7). San Diego, CA: U.S. Naval Personnel

Research Activity, October 1967. (AD-660 003).

Topics Relevant i i

i to Sy«tsiu Development I Topic

land Evaluation Technology ! No. ABSTRACT

1. Stat« of the Art Rtvlew

of th« Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STW)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environaent Definition

2.U General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3.5 Measurement Procedures

I. Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

1.3 Apparatus for Testing

H.H Personnel for Testing

4.5 Test Plans

5. Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recommendations

1.1 The general goal of this study is to

provide a methodology which will enable

cognizant persons to obtain quantitative

information on personrel effectiveness and

relative costs.

2.1 The system under discussion in this report

is the human component of a system who

performs operator or maintenance functions.

2.2 The mission of the human component in this

system is that his/her function is

performed adequately and in such a way that

it will always lead towards mission

accomplishment.

1.1 As there exists no body of quantitative

evidence about the performance effectiveness of personnel in present systems, it is

suggested that as a first step it would be

appropriate to collect data to be used as a

basis for predictions. Samples should be

selected which will generalize to entire

classes of populations.

4.2 It is suggested that the following

parameters should be used to describe the

operating conditions for which task

accomplishment must be predicted:

(1) Number

(2) Sequence

(3) Response pacing (operator pacing)

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potential!/

Priorities

6.3 Research Planning

A-490

. . L' t_" L " ^'
Topics Relevant

I to System Development

land Evaluation Technology

r»

(4) Feedback

(5) Required Accuracy

(6) Display exposure time

(7) Display visibility

(8) Type of stimulus in a display

(9) The number of stimulus dimensions

(10) The number of visual stimuli displayed

(11) Operator function

(12) Stimulus movement

(13) Requirements for control - display

coordination.

14.5 The plan suggested for research is:

(1) Select parameters and start

observation on simulated system.

a. Contact system designers regarding useful content and format of

data and reported PPE indices.

b. Utilize the automated OSD.

(2) Test and refine parameters.

(3) Contacts to determine how methodology

might be implemented.

(4) Develop automated system for handling

data.

6.2 It is recommended that the program for

development of a Navy PPE prediction

methodology be implemented and that a "boot

strap" operation be started to begin the

development of a data bank on personnel

performance.

i

A-491

• .• .-.•.
Wortman, D.B., Hixson, A.F. & Jorgensen, C.C. A SAINT model of the AN/TSQ-73

guided missile air defense system (Research Memo. 79). W. Lafayette, IN:

Prltsker and Assoo., Inc., January 1979.

I Topics Relevant I I

! to System Development !Topic

land Evaluation Technology ! No. ABSTRACT

1. State of the Art Review 2.1

of the Process

1.1 General System

Measurements

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.1 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3- Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.1 Performance

Criteria, Specific

3-5 Measurement Procedures

1. Planning Components

of the Process

1.1 Analytic Methods

1.2 Parameter Determinations

1.3 Apparatus for Testing

1.1 Personnel for Testing

1.5 Test Plans

5- Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.1 Conclusions and

Recommendations

6. Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potentials/

Prl'- itles

6.3 Research Planning

The AN/TSQ-73 Missile Minder was a

lightweight mobile automatic data

processing command and coordination system

for NIKE-Hercules and Hawk Army Air Defense

units. The AN/TSQ-73 integrates radar and

identification of friend or foe (IFF) data

from vocal and remote radars for console

display.

The SAINT model of the AN/TSQ-73 system was

designed to simulate the tasks performed by

a single operator/repairman involved in

monitoring and operating the AN/TSQ-73

display console during a simulated mission.

The SAINT model was comprised of four

submodels: operator control, aircraft

control, fire unit control, and systems

control. These four submodels operated

relatively independently of each other.

A-492

V. -•. .'.Vi
Wortnan, D.B., Sigal, C.E., Pritsker, A.A. & Seifert, D.J. New SAINT

concepts and the SAINT II almulation program (AMRL-TR-?'*-! 19)

IN: Pritsker and Assoc, Inc., April 1975. (AD-A01K 811).

W. Lafayette,

\ Topics Relevant I

\ to System Development ITopicI

land Evaluation Technology ! No. ! ABSTRACT

'

1. State of the Art Review

of the Process

1.1 General System

Neasurenents

1.2 System Taxonomy

Model (STM)

1.3 Overall Conceptual

Process Model (CPM)

2. Contextual Components

of the Process

2.1 System Definition

2.2 Mission Definition

2.3 Environment Definition

2.4 General Constraints

2.5 Performance

Requirements, Ultimate

2.6 Performance

Criteria, Ultimate

3. Analytic Components

of the Process

3.1 Practical Measurable

Attributes

3.2 Practical Attribute

Measures

3.3 Performance

Requirements, Specific

3.4 Performance

Criteria, Specific

3.5 Measurement Procedures

4. Planning Components

of the Process

4.1 Analytic Methods

4.2 Parameter Determinations

4.3 Apparatus for Testing

4.4 Personnel for Testing

4.5 Test Plans

Application Components

of the Process

5.1 Test Execution

5.2 Data Analysis

5.3 Findings Interpretation

5.4 Conclusions and

Recomendations

Further Research Areas

6.1 Measurement System

Limitations

6.2 Research Potential«/

Priorities

6.3 Research Planning

1.3 This report details the development of an

integrated package of computer routines

designed to aid the system design engineer

in determining the impact cf the human

operator on system performance. The

objective of SAINT is to provide the

necessary tools and conceptual framework to

develop simulation of complex man-machine

systems. SAINT enables the designer to

imput a description of the activities which

the human operator must perform in the

course of a mission. These activities are

represented in a task network framework in

which task performance descriptions as well

as precedence relations among tasks are

defined. SAINT performs an analysis of the

task sequence and provides information

concerning operator workload, task

completion times and other performance

measures.

The SAINT II package (an expansion of SAINT

I) has techniques which enable the user to

model continuously changing variables such

as aircraft position in space, engine

temperature, fuel consumption, etc.

Additional refinements have been devoted to

the incorporation of the capability to

modify operator and system characteristics

as a function of mission contingencies and

external events.

2.5 Each operator involved in a mission

simulation with SAINT II has certain

attributes which must be assigned to him.

Using input data, the program sets seven

operator attributes into a packet, although

A-493

_. ^ ^ *

r

- ^- - * j.. 1 -^ - - ^_i ^_»^ . .-.

•-''•'•-

-> -vfj^--.::- >:;-: :
Topics Relevant 1

I to System Development I Topic

land Evaluation Technology ! No. ABSTRACT

5.1

i

additional attributes can be included. The

seven standard attributes are:

(1) Speed factor.

(2) Accuracy factor.

(3) Stress threshold.

(1) Goal gradient.

(5) Time available to complete mission.

(6) The next intermediate stress tasks.

(7) Time available to reach the next

intermediate stress task.

A test simulation of aircraft refueling was

presented as an example. In the example,

the receiver and tanker are initially

flying at the same velocities.

Perturbations of the tanker's velocity are

incorporated in the model and represent

environmental disturbances (turbulence).

The objective of this simulation was to

determine how well the receiver pilot is

able to maintain his refueling position in

the face of these disturbances and the

prescribed control strategy.

A-494

'-'"*^***''-'"- • - - - - - ----- ,- ' ^- ."^ *- -*- -•" • '-• -• -*"- '-• '-•. .- - .- .• . .-.-.- . .•
Li

-

APPENDIX B

BIBLIOGRAPHY

(without annotations)

B-l

- - i •

'.•. . -• '.« '.» ' » •
AC Spark Plug Division, General Motors Corporation. Inertial guidance system

107A-2—Category II, Personnel Subsystem Test and Evaluation (PSTE) and Maintenance, Logistics, Reliability and Readiness (MLRR) test and evauation—Objective achievement status report (AF 04 (694)-177 AFBM Exhibit 60-20A). Milwaukee.

WI. 15 February 1964. (AD-829 749).

Akashi, H. & Mahmood, S. Performance of human operators under variouj system

parameters (NASA CR-6725T National Aeronautics and Space Administration,

(undated).

Analytics, Inc. Measuring the performance of operational decision aids (N00014-

75-C-0600, Final Rep. 1161-B). Willow Grove, PA: Analytics, Inc., April 1976.

(AD-AQ24 795).

Anderson, R. Measures of aircraft effectiveness (AOS-TR-73-5). Kirkland AFB,

NM: Office of the Assistant for Study Support (OAS), May 1973. (AD-913 306).

Anderson, W.H. Development of performance measures for organizational level

aviation maintenance managers. NTIS Weekly Government Abstracts, October 17,

1977. (Abstract).

Andrews, L.B. Detailed Test Plan (DTP) for the Initial Operational Test and

Evaluation (lOTAETfOT-II) of the AN/TPS-59 radar set (C 0041-0-07-1). Quantico,

VA: Commanding General, Marine Corps Development and Education Command,

August 1977. (AD-B020 848L).

Baker, J.D. Quantitative modelling of human performance in information systems.

Ergonomics. 1970, 13(6), 645-664.

Basso, G.L. A methodology for measurement of vehicle parameters used in

dynamic studies. Ottawa, Canada: National Aeronautical Establishment, July 1973.

Beau, J.F. Management of the human element in the physics of failure. Paper

presented at The Third Annual Symposium on The Physics of Failure in Electronics,

Chicago, IL, September 29, 1964. (AD-812 593).

Berson, B.L. & Crooks, W.H. Guide for obtaining and analyzing human performance

data in a materiel development project (Tech. Memo. 29-76). Woodlands Hills, CA:

Perceptronics, Inc., September 1976.

Blanchard, R.E. & Smith R.L. Field test of a Technique for Establishing Personnel

Performance requirements. 1969 Annals of Assurance Sciences, 271-272. New

York: Gordon and Breach Science Publishers, July 1969.

Bioom, R.F., Pepler, R.D., Schimenz, M.V. & Lenzycki, H.P. IFV/CFV personnel

selection analysis (Army Research Institute Research Note 80-41). Darien, CT:

Dunlap and Associates, Inc., July 1979.

Boehm.B.W. Computer systems analysis methodology: Studies in measuring, evaluating, and simulating computer systems (R-520-NASA). Santa Monica, CA: The

RAND Corporation, September 1970.

B-2

-•>:

- . -- - -

::::^^- - - -v ----:.
".:•

Bohdanovych, V.l., Kulyk, V.T. & Uyor.iov, A.I. (Development and practical

applications of systems analysis.) Vilosofs'ka Dumka, 1973, 1, 10-16. (Arlington,

VA- Joint Publications Research Service. 1973. JPRS 58935).

Bond, N.A. & Rigney, J.W. Measurement of training outcomes (Tech. Rep. 66).

Los Angeles, CA: University of Southern California, Behavioral Technology

Laboratories, June 1970. (AD-711 302).

Bovaird, R.L. & Zagor, H.I. A systems approach to predicting and measuring

Polaris fire control system operational availah:lity (RM 59TMP-57). Santa

Barbara, CA: General Electric Company, Technical Military Planning Operation,

December 1959. (AD-901 773).

Boycan, G.G. & Warnick, W.L. Training requirements for the armor crewman and

reconnaissance specialist Advanced Individual Training programs (HumRRO-CRD^-72-7). Alexandria, VA: Human Resources Research Organization, November

1972. (AD-759 569).

Breaux, R. Training characteristics of the automated adaptive Ground Controlled

Approach Radar Controller Training System (GCA-CTS) (NAVTPAEQUIPCEN-TN-52).

Orlando. FL: Naval Training Device "Cenier, July 1976.

Brown. D. Statistical guide for CDEC experimental design (Rev. Preliminary Draft).

BDM Scientific Support Laboratory. October 1976.

Brown, J.D. Field evaluation of M48A5 tank roduct improvements. Fort Knox. KY:

t U.S. Armor and Engineer Board, 14 January 1977. (AD-B016 139).

Buckley, E.P.. Goldberg, B., Rood, R., Hamilton, H. & Champion F. Development

of a perfor nance criterion for Enroute Air Traffic Control personnel research

through air traffic control simulation: Experiment I—parallel form development

(FAA-RD-75-186. Interim Rep.). Atlantic City, NJ: Federal Aviation Administration.

National Aviation Facilities Exferimental Center. February 1976. (AD-A023 411).

Bürgin. G.H. & Fogel. L.J. Air-to-air combat tactics synthesis and analysis program

based on an adaptive maneuvering logic. Journal of Cybernetics. 1972, 2(4). 60-68.

Burington. R.S. Concerning the reliability and effectiveness of weapon systems and

their measurement (R-14-6t Washington. DC: Bureau of Naval Weapons. Office of

Chief Mathematician. April 1961.

Campbell. S.C.. Feddern, J. & Graham, G. A-6E Systems Approach to Training:

Phase I (NAVTRAEQUIPCEN-75-C-0099-1). Bethpage, NY: Grumman Aerospace

Corporation, February 1977. (AD-A037 468).

Cheikin. G. STINGER human factors engineering final report (Tech. Memo. 25-76).

Aberdeen Proving Ground, MD: U.S. Army Human Engineering Laboratory, July

1976. (AD-B014 866).

•.

Chaney, F.B. & Thresh, J.L. Diagnosis and conection of quality problems: A

human factors approach. Paper presented at the ASQC Seminar on Product

Quality Audit, Milwaukee, WI, April 19-20, 1968. (AD-855 919).

B-3
Chapanis, A. Relevance of physiological and psychological criteria to man-machine

systems: The present state of the art. Ergonomics, 1970, 13, ?37-346. (AD-751 344).

Chasteen, C.L. IOT4E of an AWADS radar imagery recorder (MAC Project 74C110U). Eglin AFB, FL: Military Airlift Command, Ooerating Location F, May 1975.

(AD-3003 562).

Chop, A. Capability measures for system effectiveness (RADC-TR-72-26, Final Tech.

Rep.). Sunnyvale, CA: Lockheed Missiles and Space Company, February 1972.

(AD-892 8G3).

Churchman, C.W. Systems analysis and organization theory: A critique (Internal

Working Paper No. ZY. Berkeley, CA: University of California, Space Sciences

Laboratory, June 1971.

City of Reading. Systems analysis completion report. Vol. 2: Systems analysis

methodology (USAC-RPAO-005). Reading, PA: City of Reading, September 1971.

(PB-208 500-2).

Clovis, E.R. & Muller, T.H. Development of procedures for evaluating unit performance (TRA-75/009, Final Rep., Vol. 1). Monterey, CA: Litton Mellonics

Defense Sciences Laboratories, March 1975.

Coburn, R. Human engineering guide to ship system development (NELC Tech.

Doc. 278). San Diego, CA: Naval Electronics Laboratory Center, October 1973.

(AD-772 535).

Cogan, E.A. If it exists, it can be measured—but how? In E.A. Cogan & J.D.

Lyons, Frameworks for measurement and quality control (HumRRO-PP-16-72).

Papers presented at New York University First National Annual Training in

Business and Industry Conference, New York City, March 1972. (AD-748 081).

Companion, M.A. & Corso, M. Task taxonomy: Two ignored issues. In A.S.

Neal & R.F. Palasek (Eds.), Proceedings of the Human Factors Society 21st

Annual Meeting, San Francisco, October 17-20, 1977.

Companion, M. & Teichner, W.H. Application of task theory to task analysis:

Evaluation of validity and reliability using simple tasks (AFOSR-TR-77-1008).

Las Cruces, NM: New Mexico State University, Human Performance Laboratory,

January 1977. (AD-043 243).

Connelly, E.M., Bourne, F.J., Loental, D.G. & Knoop, P.A. Computer-aided

techniques for providing operator performance measures (AFHRL-TR-74-87).

McLean, VA: Quest Research Corporation, December 1974. (AD-A014 330).

Connelly, E.M., Bourne, F.J., Loental, D.G., Migliaccio, J.S., Burchick, D.A.

& Knoop, P.A. Candidate T-37 pilot performance measures for five contact

maneuvers (AFHRL-TR-74-88). McLean, VA: Quest Research Corporation,

December 1974. (AD-014 331),

B-4
K

Connelly, E.M., Schuler, A.R. & Knoop, P.A. Study of adaptive mathematical

models for deriving automated pilot performance measurement techniques

(AFHRL-TR-69-7). Vol. 1: Model Development). Falls Church, VA: Melpar,

October 1969. (AD-704 597).

Connelly, E.M. & Sloan, N.A. Manned system design using operator measures

and criteria. In A.S. Neal & R.F. Palasek (Eds.), Proceedings of the Human

Factors Society 21stAnnual Meeting. San Francisco, October 17-20, 1977.

Connelly, E.M., Zeskind, R.M. & Chubb, G.P. Development of a Continuous

Performance Measure for manual control (AMRL-TR-76-24). Vienna, VA:

Omnemii, Inc., April 1977. (AD-A041 676).

Connelly, M.N. & Willis, J.E. The development of a Human Effectiveness

Function Allocation Methodology (HEFAM) (Research Memo. SRM 70-11).

San Diego, CA: Navy Personr.el and Training Research Laboratory, October

1969. (AD-699 173).

Cunningham, J.B. A systems-resource approach for evaluating organizational

effectiveness. Human Relations, 1978, 31(7), 631-656.

Cunningham, R.P., Sheldon, M.S. & Zagorski, H.J. Project NORM: Pilot study

report (TM-2232/000/0G). Santa Monica, CA: System Development Corporation,

February 1965. (AD-458 341).

Defense Science Board. Report of task force on test and evaluation. Washington,

DC: Office of the Director of Defense Research and Engineering, April 1974.

Dieterly, D.L. The evaluation of training with specific emphasis on criteria

(AU-AFIT-SL-9-73X Wright-Patterson AFB, OH: Air Force Institute of Technology,

School of Systems and Logistics, October 1973. (AD-771 009).

DiGialleonardo, F.R. & Barefoot, D.B. An approach for measuring benefit and

cost in management and information systems"TTr (TR 75-21). San Diego, CA: Navy

Personnel Research and Development Center, October 1974. (AD-A014 209).

DiGialleonardo, F.R., Barefoot, D.G. & Blanco, T.A. Technique for Interactive

Systems Analysis (T1SA) (NPRDC-TR-75-22). San Diego, CA: Navy Personnel

Research and Development Center, 1974. (AD-A013 223).

Duning, K.E. Hickok, C.W., Emerson, K.C. & Clement, W.F. Control-display

testing requirements study (AFFDL-TR-72-122). Cedar Rapids, IA: Collins

Radio Company, December 1972. (AD-759 539).

Dunlap, J.W. & Affinito, F.J. Development of methodology for measuring

effects of personal clothing and equipment on combat effectiveness of individual

soldiers (Final Rep, of Phase 3). Danen, CT: Dunlap and Associates, Inc.,

December 1967. (AD-836 904).

Dunlap and Associates. Inc. Performance measures for human factors engineering of EARL equipment (SSD-332-618). Derien, CT: Dunlap and Associates,

Inc., June 1966. (AD-653 627).

B-5
Egbert, J.J. & Rau, J.G. Techniques for bounding and estimating measures of

effectiveness. Newport Beach, CA: Ultrasystems,Inc., July 1973" (AD-772 550).

Ellis, R.H. Task Analysis Reduction Technique (TART) for the quantification of

human performance (Research Memo. SRM 71-7). San Diego, CA: Naval Personnel

and Training Research Laboratory, September 1970 (AD-711 807).

Engel, J.D. An approach to standardizing human performance assessment (Professional Paper 26-70). Presentation at the Planning Conference of Standardization

of Tasks and Measures for Human Factors Research. Texas Technological University, Lubbock TX: March 1970. (AD-717 258).

Eppler, W.G. Analytical design of manned control systems (SUDAAR No. 280).

Stanford, CA: Stanford University, Center for Systems Research, May 1966.

Erickson, R.A. Field evaluation of a visual detection model. In G.W. Levy (Ed.),

Symposium on applied model of man-machine systems performance (NR69H-591).

Columbus, OH: North American Aviation, November 1968. (AD-697 939).

Farina, A.J. & Wheaton, G.R. Development of a taxonomy of human performance:

The task characteristics approach to performance prediction (Tech. Rep. 7).

Pittsburgh, PA: American Institutes for Research, February 1971. (AD-736 191).

Featherstone, C.L. & Scaglione, R.J. A feasibility study for determining a small

arms measure of effectiveness for handling characteristics (Master's Thesis).

Monterey, CA: Naval Postgraduate School, September 1975. (AD-B008 586L).

Fineberg, M.L., Meister, D. & Farrell, J.P. An assessment of the navigation

performance of Army aviation under Nap-of-the-Earth conditions (Research Rep.

1195). Alexandria, VA: U.S. Army Research Institute for the Behavioral and

Social Sciences. (AD-A060 563).

Finley, D.L. & Muckler, F.A. Human factors research and the development of

a manned systems applications science: The system sampling problem and a

solution. Northridge, CA: Manned Systems Sciences, Inc., July 1976. (AD-029 417).

Finley, D.L., Muckler, F.A., Gainer, CA. & Obermayer, R.W. An analysis and

evaluation methodology for command and control: Final technical report.

Northridge, CA: Manned Systems Sciences, Inc., November 1975. fÄD-A023 871).

Finley, D.L., Obermayer, R.W., Bertone, CM., Meister, D. & Muckler, F.A.

Human performance prediction in man-machine systems—Volume I—A technical

review (N70-21907; NASA CR-1614). Canoga Park, CA: The Bunker-Ramo

Corporation, August 1970.

Fischl, M.A., Siegel. A.I. & Wolf, J.J. Application of a multiple task interactive

model: Simulation of human performance in sonar maintenance. In G.W. Levy

(Ed.), Symposium on applied model of man-machine systems performance

(NR69H-591). Columbus, OH: North American Aviation, November 1968.

(AD-697 939).

B-6

.\..
Foley, J.P., Jr. Criterion referenced measures of technical proficiency in maintenance activities (Final Rep.). Wright-Patterson AFB, OH: Air Force Human

Resources Laboratory, Advanced Systems Division, October 1975. (AD-A016 420).

Foley, J.P., Jr. Performance measurement of maintenance. Wright-Patterson

AFB, OH: Advanced Systems Division, Air Force Human Resources Laboratory,

1P77. (AD-A053 475).

Ford, J.P., Harris, J.H. & Rondiac, P.F. Performance measures for AIT armor

crewman (HumRRO-CR-D2-74-2). Alexandria, VA: Human Resources Research

Organization, April 1974. (AD-A019 375).

Geddie, J.C. Profiling the characteristics of the developmental test participant

(Tech. Memo. 31-76). Aberdeen Proving Ground, MD: U.S. Army Human Engineering Laboratory, October 1976. (AD-A031 563).

Geer, C.W. Navy manager's guide for the test and evaluation sections of M1L-H46855 (D194-10006-2). Seattle, WA: Boeing Aerospace Company, Logistics Support

and Services, June 30, 1977. (AD-A045 098).

B Geer, C.W. User's guide for the test and evaluation .sections of MIL-H-46855

(D194-10006-iT Seattle, WA: Boeing Aerospace Company, Logistics Support and

Services, June 30, 1977. (AD-A045 097).

George, C.E. & Dudek, R.A. Performance, recovery and man-machine effectiveness: Final report on a basic research program under project THEMIS (Tech.

Memo. 9-74). Lubbock, TX: Texas Technical University, April 1974. (AD-777 797).

Gephart, L.S. & Balachandran, V. A stochastic system effectiveness simulation

model (Tech. Rep. UDRI-TR-70-12). Dayton, OH: University of Dayton-Research

Institute, June 1969. (AD-865 449).

*- Gex, R.C. Personnel subsystem testing and evaluation for missiles and space

systems: An annotated bibliography (SB-61-21). Sunnyvale, CA: Lockheed Aircraft

-'.- Corporation, Missiles and Space Division, April 1961.

Goldbeck, R.A., Wright, K.A. & Fowler, R.L. Operator performance and panel

layout for discontinuous tasks (AMRL-TR-70-137X Palo Alto, CA: Philco-Ford

Corporation, WLD Division - Human Engineering Section, March 1971. (AD-727 791).

Greening, C.P. Unfinished business in the utility of visual detection models. In

G.W. Levy (Ed.), Symposium on applied model of man-machine systems performance (NR69H-59l"t. Columbus, OH: North American Aviation, November 1968.

(AD-697 939).

Grunzke, P. M. Evaluation of the automated adaptive flight training system's airto-air intercept performance measurement (AFHRL-TR-78-23). Williams AFB,

AZ: Flying Training Division, July 1978. (AD-A060 320).

Gustafson, H.W. The measurement of proficiency of Air Force maintenance personnel (ATPTRC SM 7-1957). Denver, CO: Air" Force Personnel and Training

Research Center, Air Research and Development Command, Lowry AFB, 29 July

1967. (AD-841 711).

B-7

.-v.- .•
Height, F.A. Indirect methods for measuring exposure factors as related to the

incidence of motor vehicle traffic accidents (DOT-hS-800-601). State College,

PA: Pennsylvania State University, September 1971. (PB-205 031).

Hakanson, N.H. An adaptive method of test selection in system development

(RM-5238-PR). Santc Monica, CA: The RAND Corporation, April 1967. (AD-653-942).

Hall, D.H. Confidence limits for measures of effectiveness in weapon system

analysis (NWC TP 5437). ChTna Lake, CA: Naval Weapons Center, May 1973.

(AD-762 401).

Hammeil, T.J., Gasteyer, C.E. & Pesch, A.J. Advanced officer tactics training

device needs and performance measurement technique (NAVTRAEQUIPCEN 72-C0053-1, Vol. 1). Groton, CT: General Dynamics/Electric Boat Division, November

1973. (AD-922 929L).

Hanifan, D.T. & Knowles, W.B. Human performance in system effectiveness

modeling: Issues, approaches and critique. LaJolla, CA: Dunlap and Associates,

Inc., December 1968.

Hansen, D.N., Johnson, B.F., Pagan, R.L., Tam P. & Dick, W. Computer-based

adaptive testing models for the Air Force technical training environment Phase I:

Development or a computerized measurement system for Air Force technical

training (AFHRL-TR-74-48). Tallahassee, PL: Florida State University, July 1974.

(AD-785 142).

Hansen, D.N., Ross, S. & Harris, D.A. Flexilevel adaptive testing paradigm:

Validation in technical training (AFHRL-TR-77-35-1). Memphis, TN: Memphis

State University, July 1977.

Harrison, W.L. A theoretical basis for the concept of effectiveness (Master's

Thesis). Monterey, CA: Naval Post Graduate School, October 1966. (AD-807-386).

Harry, H.P. RANN utilization experience. Case study number 10. Effectiveness

measurement methods (NSF-RA-G-75-038). Washington, DC: Urban Institute, 1975.

(PB-247 2J4).

Helm, W.R. Human engineering design deficiencies: A comparative analysis of

the P-3 and S-3 aircraft (NATC-SY-41R-76). Patuxent River. MD: Naval Air

Test Center, March 1976. (AD-B010 940).

Helm, W.R. Human factors test and evaluation. Functional Description Inventory

as a test and evaluation tool, development and initial validation study (NATC-SY77R-75, Vol. 1). Patuxent River, MD: Naval Air Test Center, September 1975.

(AD-B007 463L).

Henderson, R.L. & Burg, A. The role of vision and audition in truck and bus

driving (Final Rep.). Santa Monica, CA: System Development Corporation,

December 1973. (PB-230 776).

Hicks, J.A. A methodology for conducting human factors evaluations of vehicles

in operational field tests (Draft). Fort Hood, TX: U.S. Army Research Institute

for the Behavioral and Social Sciences, Fort Hood Field Unit, June 1977.

B-8
L.

I

I

Highsmith, R.C. Proposed measure of effectiveness for human resource

availability periods and their impact upon unit operational readiness. Monterey,

CA: Naval Post Graduate School, December 1976. ^AD-A036 028).

Hill, J.W. & Eddowes, E.E. Further development of automated GAT-1 performance measures (AFHRL-TR-73-72). Menlo Park, CA. Stanford Research

Institute, May 1974. vAD-783 240).

Holshouser, E.L. Guide to human factors engineering General Purpose Test

Planning GPTP) (TP-77-14). Point Mugu, CA: Pacific Missile Test Center.

June 1977. (Äb-B022 013L).

House, E.R. Assumptions underlying evaluation models. Educational Researcher,

1978, 7(3), 4-12.

Hyatt, C.J. & DeBerg, O.H. A scoring system for the quantitative evaluation

of pilot performance during Microwave Lancng System (MLS) approaches

ASD-TR-17). Wright-Patterson AFB, OH: Aeronautical Systems Division (AFSC),

August 1975. (AD-A025 782).

Hunter, B. Report on the final acceptance test for CTS (Computerized Training

System). Alexandria, VA: Human Resources Research Organization, August 1976.

(AD-A033 400).

Hutchins, C.W. A Computer Aided Function Allocation and Evaluation System

(CAFES). Proceedings of the Human Factors Society Annual Meeting, Huntsville,

AL: October 15-17, 1974.

Irish, P.A., Grunzke, P.M., Gray, T.H. & Waters, B.K. The effects of system

and environmental factors upon experienced pilot performance in the Advanced

Simulator for Pilot Training (AFHRL-TR-77-13). Williams AFE, AZ: Air Force

Human Resources Laboratory, Flying Training Division, April 1977.

Jahns, D.W. A concept of operator workload in manual vehicle operations.

Meckenheim, BRD: Forschungsinstitut für Anthropotechnik, December 1973.

Jahns, D.W. & Katz, R. Preliminary concepts for Computer Aided Function

Allocation Evaluation System (CAFES) (NADC-72106-CS). Seattle, WA: The

Boeing Company, Aerospace Group, May 1972. (AD-902 670L).

James, L.P Criterion models and construct validity for criteria (72-32). San

Diego, CA: Navy Medical Neuropsychiatric Research Unit, August 1972.

(AD-770 412).

Jaschen, D.G. Mechanized Infantry Combat Vehicle. XM723, system operational

climatic test (TDr-OT-67A). Falls Church, VA: U.S. Army Operational Test and

Evaluation Agency, Test Design Division, 1975. (AD-B008 212L).

Johnson, E.M. & Baker, J.D. Field testing: The delicate compromise. Human

Factors. 1974. 16(3). 203-214.

Kagerer, R.L. & Weiss, E.G. Development of a checklist and guidebook for

human factors evaluation of general equipment (Final Rep.). Alexandria. VA:

The Matrix Corporation. Janiwry 1968. (AD-827 808).

B-9

"•-• '--- - ..-.•..... . -^ni
Kaplan, J.D., Crooks, W.H., Boylan, R.J. & Sanders, M.S. Human Resources Test

and Evaluation System (HRTES) (PDR-1057-78-111). Woodland Hills, CA:

Pereeptronics, November 1978.

Karush, A.D. Benchmark analysis of time-sharing systems (SP-3347). Santa

Monica, CA: System Development Corporation, June 1969. (AD-689 781).

karush, A.D. Two approaches for measuring the performance of time-sharing

systems (SP-3364)1 Santa Monica, CA: System Development Corporation, May

1969. (AD-691 366).

Kelley, C.R. Human operator models for manual control. In G.W. Levy (Ed.).

Symposium on applied model of man-machine systems performance (NR69H-591).

Columbus, OH: North American Aviation, November 1968. (AD-697 939).

Kerner, H. and Beyerle, W. A PMS level language for performance evaluation

modelling (V-PMS). Vienna: Institut fuer Digitale Anlagen Technical University.

Kiraly, R.J., Babinsky, A.D. & Powell, J.D. Aircrew oxygen system development:

Man-in-the-loop test report (N..SA CR-73395). Cleveland, OH: TRW, Inc., July

1970.

Klein, R. Integrated operational test and analysis procedures for small arms

weapon systems evaluation. Manuscript in preparation (undated).

Klein, R.D. & Thomas, C.B. The development of combat related measures for

small arms evaluation. Fort Benning, GA: U.S. Army Infantry Board, June 1969.

(AD-713 552).

Kleinman, D.L. & Baron, S. Manned vehicle systems analysis by means of modern

control theory (NASA CR-1753X Cambridge, MA: Bolt, Beranek and Newman,

Inc., June 1971.

Knoop, P.A. Survey of human operator modeling techniques for measurement

applications (AFHRL-TR-78-35). Wright-Patterson AFB, OH: Air Force Human

Resources Laboratory, Advanced Systems Division, July 1978.

Knowles, W.B., Burger, W.J., Mitchell, M.B., Hanifan, D.T. & Wulfeck, J.W.

Models, measures, and judgments in system design. Human Factors, 1969, 11,

577-590.

Kreifeldt, J. Analysis of predictor model. Unpublished Preliminary Data,

Massachusetts Institute of Technology, 1964.

Kurke, M.I. Operational concept analysis and sources of field data. Human

Factors, 1965, 1, 537-544. (AD-474 358).

Kvalseth, T.O. A decision-theoretic model of the sampling behavior of the human

process monitor. IEEE Transactions on Systems, Man, and Cybernetics, November

1977, SMC-7(11). '

B-10

v.v. s. .%%•..
>

i

■■-

Labor/Management Task Force on Rail Transportation. Terminal performance

measurement systems (DOT-FR-4-3003). Washington, DC, May 1975. (P^-252

225]:

Larson. O.A., Sander, S.I. & Steinemann, J.H. Survey of unit performance

effectiveness measures (TR74-11). San Diego, CA: Navy Personnel Research

and Development Center, January 1974. (Ab-774 ^19).

Lawrence, D.H. The evaluation of training and transfer programs in terms of

efficiency measures. The Journal of Psychology, 1954, 36, 367-382. (AD-87 172).

Levy, G.W. Criteria for selection and application of models. In G.W. Levy

(Ed.), Symposium on applied model of man-machine systems performance

(NR69H-591) Columbus, OH: North American Aviation, November 1968.

(AD-697-939).

Lindsey, M.B. A human performance investigation in a NAVORD weapon

system—Part 2 (Rep. 27-349). Annapolis, MD: Naval Ship Research and Development Center, Propulsion and Auxiliary Systems Department, March 1974. (AD-918

202L).

LTV Aerospace Corporation. Shuttle active thermal control system. Vol. 3:

Modular radiator system test data correlation with thermal model (Rep. No.

T169-28, NASACR-M0269). Dallas, TX: LTV Aerospace Corporation, Vought

Systems Division, December 1973.

Lyons, J.D. Measuring effectiveness: Quality control of training. In E.A

Cogan &. J.D. Lyons, Frameworks for measurement and quality control.

(HumRRO-PP-16-72). Papers presented at New York University First National

Annual Training in Business and Industry Conference, New York City, March

1972. (AD-748 081).

Markel, G.A. Toward a general methodology for systems evaluation (352-R-13).

State College, PA: HRB-Singer, Inc., July 1965. (AD-619 373).

Mason, A.K. & Rigney, J.W. Toward a general characterization of electronic

troubleshooting. In R.E. Blanchard & D.H. Harris (Eds.), Man-machine effectiveness analysis. Papers presented at The Human Factors Symposium, University of

California at' Los Angeles, June 15, 1967. (AD-735 718).

Matheny, W.G., Patterson, G.W. A Evans, G.I. Human factors in field testing

(LS-FR-71-1). Hurst, TX: Life Sciences, Inc., December 1971. (AD-741 216).

Matthews, E.P. A visual target acquisition model. In G.W. Levy (Ed.),

Symposium on applied model of man-machine systems performance (NR69H-591).

Columbus, OH: North American Aviation, November 1968. (AD-697 939).

McCalpin, J.P. Development of a methodology for obtaining human performance

reliability data for a machine gun system (Tech. Memo. 6-74). Baltimore, MD:

AAI Corporation, February 1974. (AD-777 020).

B-ll

.'.-.•
McDonnell Douglas Astronautics Company, Eastern Division. Optimized cost/

performance design methodology, Vol. 3—Concept analysis and model development, Book 1—Concept analysis (MDC-E0005). September 1969.

McDonnell Douglas Astronautics Company, Eastern Division. Optimized cost/

performance design methodology. Vol. 3—Concept analys»; and model development, Book 2—Model Formulation (MDC-E0005). September 1969.

McGrath, J.E., Nordlie, P.G. & Vaughan, W.S. A systematic framework for

comparison of system research methods (HRS-TN-59/7-SM). Arlington, VA:

Human Sciences Research, Inc., November 1959. (AD-229 923).

McKendry, J.M. & Harrison, P.C

test and evaluation stage of systems development. Volume II

College, PA: HRB-Singer, Inc., June 1964. (AD-603 304)

Assessing human factors requirements in the

(ND64-68). State

McRuer, D., Jex, H.R., Clement, W.F. & Graham, D. A systems analysis theory

f

or displays in manuel control (STI Tech. Rep. 163-1). Hawthorne, CA: Systems

Technology, Inc., June 1968. (AD-675 983).

Meister, D.A. A prapmatic approach to the prediction of operational performance. In R.E. Blanchard & D.H. Harris (Eds.), Man-machine effectiveness

analysis. Papers presented at The Human Factors Symposium, University of

California at Los Angeles, June 15, 1967. (AD-735 718).

Meister, D. Assumptions underlying test and evaluation strategies. San Diego,

CA: Navy Personnel Research and Development Center. (Speech, undated).

Meister, D. Assumptions underlying the human reliability model. In G.W.

Levy (Ed.), Symposium on applied model of man-machine systems performance

(NR69H-591). Columbus, OH: North American Aviation, November 1968.

(AD-697 939).

Meister, D. A systematic approach to human factors measurement. San Diego,

CA: Navy Personnel Research and Development Center, October 1978.

Meister, D. Human factors in operational system testing: A manual of procedures

(NPRDC SR 78-8). San Diego, CA: Navy Personnel Research and Development

Center, April 1978.

Meister, D. & Rabideau, G.F. Human factors evaluation in system development.

New York: John Wiley & Sons, 1965.

Melching, W.H. A concept of the role of man in automated systems (Professional

Paper 14-68). Presented at the Southwestern Psychological Association Annual

Meeting, New Orleans, LA. April 1968. (AD-671 128).

Meyer, R.P., Laveson, J.I. & Pape, G.L. Development and application of a task

taxonomy for tactical flying (AFHRL-TR-78-42, Vol. 1). St Louis, MO: Design

Plus, September 1978. (AD-061 387).

.

B-12

.

-.-.•-.-.-.,•.-. .-

. •. -.
Meyer, R.P., Laveson, J.I & Pape, G.L. Development and application of a task

taxonomy for tactical flying {AFHRL-TR-78-42, Vol. 2). St. Louis, MO: Design

Plus, September 1978. (AD-061 388).

Meyer, R.P., Laveson, J.I. & Pape, G.L. Development and application of a task

taxonomy for tactical flying (AFHRL-TR-78-42, Vol. 3). St. Louis, MO: Design

Plus, September 1978. fÄD-A061 478).

Miles, J.L. Some recent trends in human factors testing (Tech. Memo. 22-76).

Aberdeen Proving Ground, MD: U.S. Army Human Engineering Laboratory, June

1976. (AD-B012 110).

Miller, J.G. Living systems. New York: McGraw-Hill, 1978.

Miller, J.R. Assessing alternative transportation systems (RM-5865-DOT). Santa

Monica, CA: The RAND Corporation, April 1969. (PB-185 167).

Mills, R.G. & Hatfield, S.A. Sequential task performfmce: Task module relationships, reliabilities, and times (AMRL-TR-72-56)r_

Aerospace Medical Research Laboratory, 1974.

Wright-Patterson AFB, OH:

(AD-787 322).

Mitchell, M.S. & Blanchard, R.E. The allocation of Syste n Effectiveness Requirements for man-machine effectiveness analysis. In R.E. Blanchard & D.H. Harris

(Eds.). Man-machine effectiveness analysis. Papers presented at The Human

Factors Symposium, University of California at Los Angeles, June 15, 1967.

(AD-735 718).

Mitchell, M.B., Smith, R.L. & Blanchard, R.E. Test application of TEPPS on a

Navy CIC subsystem. Santa Monica, CA: Dunlap and Associates, Inc., August

1967. (AD-821 573).

Montgomery, D.C., Callahan, L.G. & Wadsworth, H.M. Application of decision/

risk analysis in operational tests and evaluation. Atlanta, GA: Georgia Institute

of Technology, The School of Industrial and Systems Engineering, September

1975. (AD-A024 205).

Mumford, J. & Smith J.P

mechanics

The development of performance criteria for turret

(HumRRO Research Memo.). Alexandria, VA: Human Resources

Research Organization, July 1961. (AD-477 647).

Nelson, R.T. A research methodology for studying complex service systems

(Working Paper No. 139). Los Angeles, CA: University of California, Western

Management Science Institute, July 1968. (AD-673 233).

Qbermayer, R.W. & Vreuls, L». Combat-ready crew performance measurement

system: Phase I measurement requirements (AFHRL-TR-74-108-11). Northridge,

CA: Manned Systems Sciences, Inc., December 1974. (AD-B005 518L).

Obermayer, R.W. 4 Vreuls, D. Combat-ready crew performance measurement

system: Phase 11 measurement system requirement (AFHRL-TR-74-108-III).

Northridge, CA: Manned Systems Sciences, Inc., December 1974. (AD-B005 519).

B-13

.-. - . J
Obermayer, R.W. & Vreuls, D. Combat-ready crew performance measurement

system: Phase II1A crew performance measurement (AFHRL-TR-740108-IV).

Northridge, CA: Manned Systems Sciences, Inc., December 1974. (AD-B005 520L).

Obermayer, R.W. Vreuls, D. & Ccnway, E.J. Combat-ready crew performance

measurement system: Phase IIIC design studies (AFHRL-TR-74-108-VI). Northridge,

CA: Fanned Systems Sciences, Inc., December 1974. (AD-B005 521L).

Obermayer, R.W., Vreuls, D., Muckler, F.A. & Conway, E.J. Combat-ready crew

performance measurement system: Phase HID specifications and implementation

plan (AFHRL-TR-74-108-VII). Northridge, CA: Manned Systems Sciences, Inc.,

December 1974. (AD-B005 522L).

O'Connor, M.F. & Buede, D.M. The application of decision analytic techniques

to the test and evaluation phase of the acquisition of a major air system-

(TR-77-3). McLean, VA: Decisions and Designs, Inc., April 1977. (AD-A040 691).

Pew, R.W., Baron, S., Feehrer, C.E. & Miller, D.C. Critical review and analysis

of performance models applicable to man-machine systems evaluation (BBN Rep.

No. 3446). Cambridge, MA: Bolt Beranek and Newman, Inc., March 1977.

(AD-A038 597).

Phatak, A.V. Improvement in weapon system effectiveness by application of

identification methods for determining human operator performance decrements

under stress conditions (AMRL-TR-73-38). Palo Alto, CA: Systems Control, Inc.,

December 1973. (AD-773 856).

Phatak, A., Weinert, H. & Segall, I. Identification of the optimal control model

for the human operator (AMRL-TR-74-79). Palo Alto, CA: Systems Control,

Inc., May 1975. (AD-A009 956).

Philco-Ford Corporation. Human engineering design check list (WDL-TR 1968A).

Palo Alto, CA: Philco-Ford Corporation, May 1964. (AD-829 426).

Polak, W.T., Robertson, J.C. & Yuan, L.S. Systems Effectiveness Analyzer

(AMRL-TR-73-113). Sudbury, MA: Raytheon Company, Simulation and Surveillance

Systems, February 1974. (AD-781 124).

Popham, W.J. & Husek, T.R. Implications of criterion-reference measurement.

Journal of Educational Measurement, 1969, 6(1), 1-9.

Potempa, K.W. A catalog of human-factors techniques for testing new systems

(AFHRL-TR-68-15): Wright-Patterson AFB, OH: Air Force Human Resources

Laboratory, February 1969. (AD-854 482).

Pritsker, A.A., Wortman, D.B., Seum, C.S., Chubb, G.P. & Seifert, D.J. SAINT:

Vol. 1. Systems Analysis of Integrated Network of Tasks (AMRL-TR-73-126).

West Lafayette, IN: Purdue University, School of Industrial Engineering, April

1974. (AD-A014 843).

Quinn, J.L. Research and development appraisal and evaluation (SLTR5-70).

Wright-Patterson AFB, OH: Air University, School ot Systems and Logistics,

November 1970. (AD-876 006).

Ramsay, J.O. Some statistical considerations in multidimensional scaling

(RB-66-26). Princeton, NJ: Educational Testing Service, June 1966. (AD-489 591).

B-14

. --
Rankin, G,L. An application of fault free analysis to operational testing

(Master's ThesislL Georgia Institute of Technology, 1975,

Rankine, R.R. The effects of aircraft dynamics and pilot performance on

tactical weapon delivery accuracy (UCLA-ENG-7085). Los Angeles, CA:

University or California, School of Engineering and Applied Science, November

1970. (AD-728 324).

Rasch, W.H., Jr. Guidelines for making tradeoffs: The special role of Technial

Performance Measurement. Fort Belvoir, VA: Defense Systems Management

College, 1973. (AD-A045 256).

Rau, J.G. Measures of effectivess handbook. Irvine, CA: Ultrasystems, Inc.

August 1974^ (AD-A021 461).

Ray, E.G., Hiltz, H.A. & Spurway, D.T. Test design plan for Defense Satellite

Communications System III (DSCS III) Initial Operational Test and Evaluation

(IGT&E^ U.S. Army Operational Test and Evaluation Agency, Test Design

Division, 1979. (AD-B041 308L).

,_ Repperger, D.W., Smiles, K.A., Neff, J.A. & Summers, W.C. A feature selection

approach in evaluation of parameter changes on the human operator under thermal

stress. Ergonomics, 1978, 21(1), 35-48.

Rhoads, D.W. Inflight evaluation of four cockpit controller configurations in a

variable stability airplane (AFFDL-TR-70-95). Buffalo, NY: Cornell Aeronautical

^ Laboratory, Inc., September 1970. (AD-876 589).

K

Rigby, L.V. The Sandia Human Error Rate Bank (SHERB). In R.E. Blanchard

& D.H. Harris (Eds.). Man-machine effectiveness analysis. Papers presented at

The Human Factors Symposium, University of California at Los Angeles, June 15,

1967. (AD-735 718).

Rouse, W.B. Estimation and control theory. Application to modeling human

behavior. Human Factors, 1977, 19(4), 315-329.

Sauer, D.W., Campbell, W.B., Potter, N.R. & Askren, W.B. Human resource

factors and performance relationships in nuclear missile handling tasks. (AFHRL-

_ TR-76-85, AFWL-TR^e-SOlX Dayton, OH: Systems Research Laboratories, Inc.,

1977. (AD-A042 604).

Schaeffer, K.H., Fink, J.B., Rappaport, M., Wainstein, L. & Erickson, C.J. The

knowledgeable analyst: An approach to structuring man-machine systems (SRI No.

IMU-3546). Menlo Park, CA: Stanford Research Institute, February 1963.

(AD-297 432).

Self, H.C. Performance measures, observer selection, and reconnaissance/strike

effectiveness (AMRL-TR-72-86). Wright-Patterson AFB, OH: Aerospace Medical

Research Laboratory, November 1972. (AD-770 647).

Shaptiro, A. & Bates, C, Jr. A method for performing human engineering

anaiysis of weapon sptems (WADC Tech. Rep. 59-784)7 Wright-r-atterson AFB,

OH: Wright Air Development Center, Aerospace Medical Laboratory, September

1959. (AD-255 920).

l B-15

.x»v. . -.-. .•.-.-.-. .-'.-•.-.
Sheldon, M.S. & Zagorski, H.J. Man-machine system evaluation—The Normative

Operations Reporting Method. In R.E. Blanchard & D.H. Harris (Eds.), Manmachine effectiveness analysis Papers presented at The Human Factors

Symposium, University of Call nia at Los Angeles, June 15, 1967. (AD-735 718).

Siegel, A.I. Human performance reliability—Its measurement and impact on

system reliability. ASME Publication, 78-DE-17, New York: The American

Society of Mechanical Engineers, 1978.

Siegel, A.I & Federman, P.J. Developrnent of performance evaluative measures:

Investigation into and application of a fleet post-training performance evaluation

system (Rep. No. 7071-2). Wayne, PA: Applied Psychological Services, Inc.,

September 1970. (AD-713 192).

Siegel, A.I., Pfeiffer, M.G. & Worms, T.M. Development and evaluation of a

content analytic approach in Army field system data organization (ARI Tech.

Rep. TR-77-A11). Wayne, PA: Applied Psychological Services, Inc., August 1977.

(AD-A042 075).

Siegel, A.I., Platzer, iLl* & Lanterman, R.S. Techniques for evaluating operator

loading in man-machine systems. Wayne, PA: Applied Psychological Services, Inc.,

March 1967. (AD-810 787).

Siegel, A.I., Schultz, D.G. & Federman, P. Post-training performance criterion

development and application: A matrix method for the evaluation of training.

Wayne, PA: Applied Psychological Services, Inc., January 196E (AD-251 082).

Siegel, A.I. & Wolf, J.J. Computer simulation: Savior, sanctuary or silliness?

In G.W. Levy (Ed.). Symposium on applied model of man-machine systems

performance (NR69H-591). Columbus, OH: North American Aviation, November

1968. (AD-697 939).

Siegel, A.I., Wolf, J.J. & Lanterman, M.R. A model for predicting integrated manmachine system reliability: Model logic and description. Wayne, PA: Applied

Psychological Services, Inc., November 1974. (AD-A009 814).

Simon, C.W. Advanced methodologies study program (Final Rep.). Culver City,

CA: Hughes Aircraft Company, June 1975. (AD-A021 099).

Simon, D.E. et al. Standard procedures for Air Force Operational Test and

Evaluation (RADC-TR-74-270), Final Rep. Vol. 1). Albuquerque, NM: Braddoek,

Dunn & McDonald, Inc., October 1974. (AD-B000 365).

Sjogren, D.D. Measurement techniques in evaluation. Review of Educational

Research, (undated), 40(2), 301-320.

Smith, R.L., Westland, R.A. & Blanchard, R.E. Technique for Establishing Personnel Performance Standards—TEPPS (PTB-70-5, Vol. 1). Santa Monica, CA:

Integrated Sciences Corporation, December 1969, (AD-704 103).

Smith, R.L., Westland, R.A. & Blanchard, R.E. Technique for Establishing Personnel Performance Standards—TEPPS (PTB-70-5, Vol. 2). Santa Monica, CA.

Integrated Sciences Corporation, December 1969. (AD-704 104).

B-16

•. %" . •..-.*%■■..-.. •• .-• . •.
Je

•••-.

.•; Smode, A.E., Gruber, A. & Ely, J.K. The measurement of advenced flight vehicle

crew proficiency in synthetic ground environments (Tech. Rep. MRL-TDR-62-2).

Dänen, CT: Dunlap and Associates, Inc., February 1962.

Spencer, G. Man-machine simulation—tiie P1MO application. In R.E. Blanchard

& D.H. Harris (Eds.), Man-machine effectiveness analysis. Papers presented at

The Human Factors Symposium, university of California at Los Angeles, June 15,

1967. (AD-735 718).

Spyker, D.A., Stackhouse, S.P., Khalafalla, A.S. & McLane, R.C. Development of

techniques for measuring pilot workload (NASA CR-1888). Roseville, MN: Honeywell,

Inc., November 1971.

-"•

;

v

Swink, J.R., Butler, E.A., Lankford. H.E., Miller, R.M., Watkins, H. & Waag, W.L.

Definition of requirements for a performance measurement system for C-5 aircrew members (AFHRL-TR-78-54). San Diego, CA: Logicon, Inc., October 1978.

Taylor, E.N. & Tillman, B. Human factors engineering study of two ball port

designs for IFV. Unpublished manuscript, October 1977.

The Bunker-Ramo Corporation. Proposal for a study of human factors field permance measurement methodology (Proposal No. 5655-022-5U1). Canoga Park, CA:

The Bunker-Ramo Corporation, Defense Systems Division, June 1965.

Tien, J.M. Tov crd a systematic approach to program evaluation design. IEEE

Transactions on Systems, Man, and Cybernetics, September 1979, SMC-9(9),

494-515.

Timson, S.F. Measurement of technical performance in weapon system development programs: A subjective probability approach (RM-5207-ARPA). Santa

V Monica, CA: The RAND Corporation, December 1968. (AD-681 771).

w

Topmiller, D.A. Mathematical models of human performance in man-machine

systems (AMRL-TR-68-22) Wright-Patterson AFB, OH: Aerospace Medical

JV Research Laboratories, May 1968. (AD-673 348).

'.- Topmiller, D.A. The role of applied man-machine models. In G.W. Levy (Ed.),

lC Symposium on applied model of man-machine systems performance (NR69H-591).

Columbus, OH: North American Aviation, November 1968. (AD-697 939).

Tranby, E.D. Advanced surftce ship weapon systems test and evaluation

(NSWSES-TP-60). Port Huentme, CA: Naval Ship Weapon Systems Engineering

Station, January 1976. (AD-390 826).

Turner, C.R. & Bard, J.F. Tactical AWACS measures of effectiveness (ESD-TR72-142). Bedford, MA: The MITRE Corporation, April 1972. (AD-742 233).

- .

Uhlaner, J.E. Human performance, jobs, and systems psychology—the systems

measurement bed (Tech. Rep. S-2). Arlington, VA: Behavior and Systems

Research Laboratory, October 1970. (AD-716 346).

Uhlaner, J.E. & Drucker, A.J. Mi'itary research on performance criteria: A

change of emphasis. Human Factors, 1980, 22(2), 131-139.

B-17

V-.--V-V-^v.i.->-:v:v^••;;->';•; •. •>^• •: .-; N--:- .^v-^v--•>^*i:-':••"-• "•;•."•:•
Ultrasystems, Inc. A study of measures of effectiveness used in naval analysis

studies (Vol. 1 Summary). Newport Beach, CA: Ultrasystems, Inc., October 1972.

TAD-912 443).

Ultrasystems, Inc. A study of nreasures of effectiveness used in naval analysis

studies (Vol. 2 Study Review Summaries, Part~iy Newport Beach, CA: Ultrasystems, Inc., October 1972. (AD-912 444).

Ultrasystems, Inc., A study of measures of effectiveness used in naval analysis

studies (Vol. 3 Study Review Summaries, Part 2). Newport Beach, CA: Ultrasystems. Inc., October 1972. (AD-912 445).

Ultrasystems, Inc. A study of measures of effectiveness used in naval analysis

studies (Vol. 4 MOE Reviews). Newport Beach, CA: Ultrasystems, Inc., October

1972. (AD-912 446).

U.S. Army Armor School. Tank platoon: Organization for combat and techniques

of movement (TC 17-15-3). Fort Knox, KY: U.S. Army Armor School, April

1975. (Pamphlet).

U.S. Army, Army Materiel Command. Engineering design handbook: Recoilless

rifle weapon systems. Alexandria, VA, 15 January 1976. (AD-A023 513).

U.S. Army Combat Developments Command. Communications-electronics-75

(CE-75) phase 1-field Army (No. 6492, Vol. 9). Fort Monmouth, NJ: U.S. Army

Combat Developments Command, Communications-Electronics Agency, September

1968. (AD-883 611).

U.S. Army Human Engineering Laboratories. Standardization of tasks and measures

for human factors research. Proceedings of a Conference at Texas Tech University,

Lubbock, TX, 18-19 March 1970. (AD-714 669).

U.S. Army Infantry Board. Infantry weapons test methodology study. Vol. 1:

Small arms test methodology (USAlB-3319-F-Vol. 1). Fort Benning, GA: U.S.

Army Infantry Board, November 1971. (AD-890 998L).

U.S. Army, Materiel Testing Directorate. U.S. Army test and evaluation command

development test II (EP)—System test operations procedure "Test, Measurement,

and Diagnostic Equipment (system peculiar)." Aberdeen Proving Ground, MD: U.S.

Army Human Engineering Laboratory, 7 May 1974. (AD-781 946).

U.S. Army Operational Test and Evaluation Agency. Mechanized Infantry Combat

Vehicle (MICV), XM723, operational climatic test/force, development test and

experimentation (Draft). Falls Church, VA: U.S. Army Operational Test and

Evaluation Agency, March 1976.

U.S. Army Test and Evaluation Command. Materiel test procedure 3-3-521.

Fort Benning, GA: U.S. Army Infantry Board, May 1970. (AD-871 788).

U.S. Army Test and Evaluation Command. Materiel test procedure 6-2-502.

(Electronics Proving Ground). Fort Beming, GA: U.S. Army Infantry Board,

August 1969. (AD-720 976).

B-18
i

f:

U.S. Army Test and Evaluation Command. Materiel test procedure 10-2-505.

(General Equipment Test Activity). Fort Benning, Ga. U.S. Army Infantry

Board, September 1971. (AD-729 855).

U.S. Department of the Army. Army training and evaluation program for

mechanized infantry battalion and combined arms task force (ARTEP 7-45).

Washington, DC: Headquarters, Department of the Army, September H75.

U.S. Department of the Army. Skill Qualification Test notice for MOS 118

SQT 3 infantryman (No. 7-llB3-N^ Washington, DC: Headquarters, Department of the Army, June 1976. (Pamphlet).

U.S. Department of the Army. Tank gunnery (Field Manual 17-12). Washington, DC: Headquarters, Department of the Army, March 1977.

U.S. Department of the Army, Office of the Chief of Staff, Tank Forces

Management Group. Tank weapon system management. Washington, D.C.

Van Acker, A. & Wohl, J.G. Modeling the sonar operator's detection process:

A progress report. In G.W. Levy (Ed.), Symposium on applied model of manmachine systems performance (NR69H-59lT~ Columbus, OH: North American

Aviation, November 1968. (AD-697 939).

von Winterfeldt, D. An overview, integration, and evaluation of utility theory

for decision analysis (SSR1 Research Rep. 75-9). Los Angeles, CA: University

of Southern California, Special Science Research Institute, August 1975.

(AD-A021 497).

Vreuls, D., Wooldridge, A.L., Obermayer, R.W., Johnson, R.M. & Norman, D.A.

Development and evaluation of trainee performance measures in an automated

instrument flight maneuvers trainer (NAVTRAEQUIPCEN 74-C-0063-1). Westlake

Village, CA: Canyon Research Group, Inc., October 1975. (AD-A024 517).

Waag, W.L., Eddowes, E.E., Fuller, J.H. & Fuller, R.R. ASUPT automated

objective performance measurement system (AFHRL-TR-75-3). Williams AFB,

AZ: Air Force Human Resources Laboratory, Flying Training Division, March

1975. (AD-A014 799).

Weapon System Effectiveness Industry Advisory Committee. Final report of

task group II. Vol. 1 Prediction-measurement: Summary, conclusions and

recommendations (AFSC-TR-65-2). 1965. (AD-458 454).

Weapon System Effectiveness Industry Advisory Committee. Final report of

task group II. Vol. 2 Prediction-measurement: Concepts, task analysis,

principles of model construction (AFSC-TR-65-2T1 1965. (ÄD-458 455).

Weapon System Effectiveness Industry Advisory Committee. Final report of

task group II. Vol. 3 Prediction-measurement: Technical supplement (AFSCTR-65-2). 1965. (AD-458 456).

Weinstock, G.D., Douglas, M. & Blom, B. Development of criteria and

measures of effectiveness for U.S. Army tactical communications sytems

(Tech. Rep. ECOW-5012-1). Paramus, NJ: Communications Systems, Inc.,

May 1969. (AD-881 145).

B-19

.-vV\-\-vVv>.:vv :-/;•-^^
Weisz, J.D. System analysis: Manpower resources/system design integration

methodology (Tech. Note 9-68). Aberdeen Proving Ground, MD: Human

Engineering Laboratories, August 1968. (AD-675 481).

Wellman, L.N. & Neill, W.H. The development of performance measurement

standards for the USAF base level supply system (SLSR-12-72B, Master's Thesis).

Wright-tatterson AFB, OH: Air Force Institute of Technology, September 1972.

(AD-750 912).

Wherry, R.J. The development of sophisticated models of man-machine system

performance. In G.W. Levy (Ed.), Symposium on applied model of man-machine

systems performance (NR69H-591). Columbus, OH: North American Aviation,

November 1968. (Äb-697 939).

Williams, G.S. A methodology to establish the criticality of attributes in

operational tests (Master's TThesis). Georgia Institute of Technology, 1975.

(AD-A024 199).

Williams, H.L. Dependent models for estimating human performance reliability

In R.E. Blanchard & D.H. Harris (Eds.). Man-machine effectiveness analysis.

Papers presented at The Human Factors Symposium, University of California at

Los Angeles, June 15, 1967. (AD-735 718).

Williams, R.L., Long, T.W. & Windho!?., W.M. A standardized munitions effectiveness methodology (K-75-57 R). Colorado Springs, CO: Kaman Sciences

Corporation, July 1975. (AD-B009 483).

Willis, J.E. Feasibility of the development und utilization of personnel performance effectiveness measures for man/machine function allocation decisions

(Research Memo. SRM 68-7). San Diego, CA: U.S. Naval Personnel Research

Activity, October 1967. (AD-660 003).

Wortman, D.B., Hixson, A.F. & Jorgenson, C.C. A SAINT model of the AN/

TSQ-73 guided missile air defense system (Research Memo. 79). W. Lffayette,

IN: Pritsker and Associates, Inc., January 1979.

Wortman, D.B., Sigal, C.E., Pritsker, A.S. & Seifert, D.J. New SAINT concepts

and the SAINT II simulation program (AMRL-TR-74-119). W. Lafayette, IN:

Pritsker and Associates, Inc., April 1975. (AD-A014 814).

B-20

;-:••:•• :•-; . ; .-... -•.-->::.-;... ;. :-: - >:•-•:.; . ;-:•.•:.•:-•-:--; .•:.- -.••^-^ -•--••••••.■■•^ .- -•-