.00

o

Tecknical Report 476

THE CONTRIBUTION OF HUMAN FACTORS

IN MILITARY SYSTEM DEVELOPMENT:

0METHODOLOGICAL CONSIDERATIONS

Harold E. Price, Marco Fiorello, John C. Lowry

M. Gregory Smith, and Jerry S. Kidd

-UBioTechnology, Inc.

Prepared for

DOD HUMAN FACTORS ENGINEERING (HF`E)

ECHNICAL ADVISORY GROUP (TAG)

Reproduced From

Best Available Copy ,A

U. S. Army

Research Instilute for the Behavioral and Sociali Sciences

0 THER SUPPORTING ORGANIZATIONS

July 1980

AvofOved for VVk00ubs~vco wono
U. S. ARMY RESEARCH INSTITUTE

FOR THE BEHAVIORAL AND SOCIAL SCIENCES

A Field Operating Agency under the Jurisolction of -the

Deputy Chief of Staff for 'Personnel

FRANKLIN A. HART

JOSEPH ZEIDNER Colonel, US Army

Technical1 Diector Comnuder

NOT1CES

OISTPISIUTION Primary distnbujtuon o# thus reot' hbs e.b N.Pe.adacrepnec

WAncer"40 distribution @4 repoirls to: -U. S. Arty Remeof Institute for the SahWAWrra NWso" Sema kises

A TTP: PIRf-I.P, 5001 Eisengtovia Avenue. Absemndrie. Virginim 2rd=3

FINA flIDOI~I~k:This report maey be deefroyed when it is noe bfflgerq tseed. Pte.. dos Rai ivrn it to

teU. S. Armyt Peuwgf In~titute for "the ftoreorl anpd Sm154 Sgerasif

~2LTh. tariderip ina MWt repor wre not to be cfeeruad as ran ffiedOsam ems v a smi tfme A VyWilb

w111098.1 me Oesagraate by jther 1hitheruled doewmrerats.
U-LNCLASSI FIED__________-

SECURITY CLASSIFICATION OF THIS FAGE (When Data Zwererd)

REPORTIONPAGEREAD) INSTRUCTIONS K1. T~APME~AION AGEBEFORE COMPLETING FORM

REPORT NUMBER 2.GOVT ACCESSION NO' 3-PEIPENT*S CATALOG NUMBER

Tehia Repor 476j

¶LE (and Subtitle) 5. TYPE OF REPORIT & PERIOD COVERED

T'HE gONTRIBUTION OF..UIMAN ZACTORS IN IFinal i~eportit

,.4ILITARY_,YSTEM-.DEVELOPMENT 401Jlyl8 Sp.w 

Methodological Considerations. ,6. PER .FORMIt4GQfj~f~tREPRN

(I Harold E. ricŽýýIarco Aiorello) MD 9037RC-053

I,~~~~ ~ ~ -M ~ Geor/Smt 1-b -,--lv 

NAM AND ADD49816 RESS 10. PROGRAM ELEMENT, PROJECT. TASK

AREA & WORK UNIT NUMBERS

BioTechnology, Inc. I

3027 Rosemary Lane (L 2QI62722A765

FallsChurch,_VA__22042 __

,11 CONTROLLING OFFICE NAME AND ADDRESS -CE@~'~!

*U.S. Armý ýkes'.arch Institute for ehe, 1 J'ilj je98O

*Behavioral-and Social Sciences

.50012 EiKsenliower Av- Alexandria. VA 22333 291

14, MONITORING AGENCY NAME & ADDRESS 15' I. SECURITY CLASS. (of thisareport)

(if different fromn Con troffing O)ffice')/

Unclassified

F115.. DECLAS5IFICATION/DOWNGRADiNG / SCHEDULE

'16 DýSTRIBUTION STATEMENT (of this Report)

Approved for open, release;'distribution unlimited

ii?, DiSTR IBUT ION STATE ME NT (of the abstract entered in Block 20. if different from Report)

[is. SUPPLEMENTARY NOTES

Acompanion research note 'eARl RN-80-23) is available through the Defense

Technical Information Center.' Title: "Department of Defense and Service Requremntsfor Human Factors R&D 'in the Military System Acquisition Process."

19. KEY WORDS lContinue on revers aide if neesaay and identify by block musiber)

Human Factors Human Engineering Mil -itary System Development System Metrics ( Cost-Benefit Analyzis

20. ATR ACT (Couutunue on reverse side if necressry and ideantify by block #WinbW)

This document reports On an, effort to determine (a) the conceptual basis

for human factors contributions to military system development and (b) the

feasibility of an impact assessm'ent methodology to mauetevleo

human factors R&D in system development. For these purposes*, two parallel

* analytic processes were implemented.

DOD FORM 1473 EDITION OF 1NOV"61 IOBSOLETE A IU. JAN £ai7P*m /We 010414W E C PS l
UNCLASSIFIFD

SECURITY CLASSIFICATION OF THIS PAGE (ften D•o Entered)

analytic process SOne provided the conceptual basis for human factors

in military system development. First, a rationale for human factors

contributions and products was prepared. This rationale was further

supported by policy documentation containing requirements for human factors

R&D. The second analytic process provided the basis for evaluating human

factors contributicns. A review of cost-benefit analysis techniques

applicable to human factors was performed together with a derivation of

measurement metrics. These efforts resulted in a framework for performance

of impact assessment and a 'determination that it is a feasible methodology

for application to evaluations of human factors contributions to military

system development.

This technical report provides recommendations for further refinement

and validation of impact assessment in the measurement of the value of

human factors efforts and products. Recommendations for developing human

factors, impact assessment handbooks are also provided.

/

UNCLASIFIE

*,1
Technical Report 476

THE CONTRIBUTION OF HUMAN FACTORS

IN MILITARY SYSTEM DEVELOPMENT:

METHODOLOGICAL CONSIDERATIONS

Harold E. Price. Marco Fiorello, John C. Lowry

M. Gregory Smith, and Jerry S. Kidd

BioTechnology, Inc.

Prepared for

DOD HUMAN FACTORS ENGINEERING (HFE)

TECHNICAL ADVISORY GROUP (TAG)

Submitted by:

Stanley M. Halpin, Acting Chief

HUMAN FACTORS TECHNICAL AREA

Approved by:

Edgar M. Johnson, Director

__ORjMANIZATIONS AND SYSTEMS

RESEARCH LABORATORY

U.S. ARMY RESEARCH INSTITUTE FOR THE BEHAVIORAL AND SOCIAL SCIENCES

5001 Eisenhower Avenue. Alexandria,'Virginia 22333

Office, Deputy Chief of Staff for Personnel

Department of the Army

July 1960

Arfm Proiect Number Systems Requirements and Operations

20162722A765

' 0 UW6li fle me: disaffmsim utm~Mi&u.

Mt
ARI Research, Reports and Technical *Reports are intended for sponsors of

R&D tasks and for other research and military agencies. Any findings ready

for implementation at the time of publication are presented in the last part

of the Brief. Upon completion of a major phasd of the task, formal recommendations for official action normally are conveyed to appropriate military

agencies by briefing or Disposition Form.
FOREWORD

The Human Factors Technical Area of the Army Research Institute (ARI) is

concerned with human resource demands of increasingly complex battlefield

systems used to acquire, transmit, process, disseminate, and utilize information. This increased complexity places great demands upon the operator interacting with the machine system. Research in this area focuses on human

performance problems related to interactions within command and control centers

as well as issues of system development. The research program includes both,

technology base and advanced development research as well as a limited amount

of technical advisory service (TAS) to Army agencies and activities. The

general purpose of TAS is to provide immediate consulting assistarce in meeting

short-term priority reqtirements.

One area of special interest involves the development of estimates for

the contributions of htr.an. factors in military system development. The

inquiry into the topic resulted from a tri-service committee decision to

investigate the possibility of providing system designers/managers with

evidence of the value of human factors to compare with other pertinent

information from engineers, operations research analysts and system analysts.

This initial report emphasizes the methodological considerations of such an

undertaking and creates a foundation for implementation of such an effort

by system personnel.

The following individuals contributed to this effort: Dr. Edgar M.

Johnson and Dr. Thomas M. Granda (ARI); Mr. Pau! Linton (Naval Air Development

Center); Mr. John L. Miles, Jr. (Human Engineering Laboratory); rr. Donald

A. Toomiller (USAF Aerospace Medical Research Laboratory); and Dr. Alfred

R. Fregly (Air Force Office of Scientific Research).

(JOS PH ZELI1I=

ec ical Director

A,. .
THE CONTRIBUTION OF HUMAN FACTORS IN MILITARY SYSTEM DEVELOPMENT;

METHODOLOGICAL CONSIDERATIONS.

BRIEF

Requirement:

To determine the conceptual basis for human factors contributions to military system acquisition and development. Given

this conceptual basis, to determine a feasible method for

evaluating the contribution of human factors.

Procedure:

Two parallel analytic processes were used to determine a

conceptual basis and feasible methodology for assessing the

contribution of human factors in system development.

e A first analytic process involved the development of

a rationale for human factors in system development,

followed by a determination of the existing basis for

human factors R&D (ranging from formal DOD requirements

to informal documentation). This culminated in a

determination of the conceptual basis for identifying

human factors contributions, through analysis of human

factors principal products, system-specific efforts,

and technology base.

* A second analytic process was undertaken to determine

a feasible method for evaluating human factors contributions, including the identification of metrics for

measuring the value of human factors. Concurrent with

this, a review of cost-benefit analysis techniques

was conducted. Out of these efforts, an impact assessment methodology emerged as the most feasible methodology

for measuring the value of human factors contributions

in military system development.
Product:

Several items worthy of note include: the developmant of a

conceptual basis in which specific human factors efforts and

products for each phase of system development were defined; a

preliminary set of measurement metrics were developed;, the

framework of an impact assessment methodology for evaluation of

human factors .R&D was developed; and an impact assessment

vocabulary hierarchy, tailored to human factors, was specified

(i.e., impact areas, metrics, and empirical measures).,

Utilization:.

The conceptual basis for human factors, in conjunction with

the impact assessment methodology, can be used to advantage by

practitioners who wish to determine the contributions of human

factors R&D to military system development. Although the

methodclogy requires additional refinement, and validation through

case examples, the present approach can be implemented.

Vill
CONTENTS

Page

Chapter 1 Introduction .............................. 1-1

Project Objectives . . . . . . . . . . .. . . . .-

The Boundaries of Human Factors . . . . . . . . ... 1-2

Definition of Human Factors . . . . . . . . . 1-2

Classification of Human Factors RDT&E . . . ... 1-4

Scope of Human Factors 1-4

The Integration of Human Factors and

Military System Development . . . . . . . . . . . 1-7

Overview of the Project . . . . . . . . . . . . . 1-9

Human Factors in Military Systems ............ ..... 1-12

The Measurement of Human Factors Contributions . . .. 1-13

The Concept of Impact Areas . .......... . . .......... 1-17

Summary and Synthesis . . . . 1-19

References . . . . . . . . . . . . . . . . . . . . 1-22

Chapter 2 A Rationale for the Value of Human Factors

in'System Development ............. 2-

Historical Background of Military Human Factors . . 2-2

Military System Complexity and, Human Factors . . . . 2-6

Human Error, Human Factors, and

System Effectiveness 2-10

Optimal Manned Systems--The Compatibility Factor . . . 2-18

Unique Human Capabilities 2-19

Human Limitations . . . . . . 2-20

Compatibility and User Acceptance , . . .... 2-21

Are There Opportunides for Human Factors

in System Development? . . . . ... . ........ 2-24

To We Use What We Know About Human Pactors? . .... 2-26

Are Human Factors Contributions Affordable? . . . . 2-32

Conclusion . . . . ... .* ** .* .* . 2-34

References . . . . . . . . . ... . . . . . . . . . .2-36

* ~ix
CONTENTS (continued)

Page

Chapter 3 Identification of the Contribution of

Human Factors in System Development ... ....... .. 3-1

Human Factors R&D: The Existing Basis

in System Acquisition . . . . . . . . . ......... 3-4

The Major System Acquisition Process ... ....... .. 3-4

Revised Department of Defense'

5000 Series Directives ........................... 3-8

Formal and Informal Requirements for

Human Factors R&D at the Service Level . . . . . . 3-12

Human Factors R&D: Specific Efforts

for Each System Development Phase ...... ........... .. 3-13

Mission Analysis Phase Human Factors ............ . . 3-21

Human Factors Efforts and System Development

Activities During Mission Analysis . ........ 3-22

Content of the Role of Man Statement . . . . . . . 3-27

Mission Analysis Phase: Example of

Human Factors Contributions . . . . ... . . . . . 3-30

Concept Development Phase Human Factors ... ......... . 3-33

Human Factors Efforts and 'System Development

Activities During Concept Development .... .. 3-33

Content of the Allocation of Functions to

Man Statement as Part of the. Decision

Coordinating Paper . . . . . . . . . . . . . . . . 3-37

Concept Development Phase: Exemple of

Human Factors .ontributions . . ... . . . . . . . . 3-40

Demonstration/Validation Phase Human Factors . . . . . 3-42

Human Factors Efforts and System Development

Activities During Demonstration/Validation .... 3-43

Content of the Task Analysis and Human

Engineering Requirements Product . . . . . . . . . 3-50

Demonstration/Validation Phase: Example of

Human Factors Contribution . . . . . . . . . . . . 3-55

Full-Scale Development Phase Human Factors . . . .. 3-58

Human Factors Efforts and System Development

Activities During Full-Scale Development . . . . . 3-59
CONTENTS (continued)

Page

Content of the Optimal Man-Machine

Interface Design . . . .......... . . . . . 3-63

Full-Scale Development Phase: Two Examples

of Human Factors Contributions . . . . . ..... 3-68

Production and Deployment Phase Human Factors . . . . 3-71

Human Factors Efforts and System Develonment

Activities During Production and Deployment . . . 3-72

References . . . . ... . . . . . . . . . . . . . . . . 3-76

Chapter 4 Human Factors RDT&E in the Technology Base

and the Contribution to System Development . 4-1

R&D Funding Categories .... . ... . . . . . . 4-4

Tracking Research to Utilization . . . ......... 4-5

Illustrations of Contributions from the Technology

Base to Specific System Development ... ....... . . . 4-8

References . . . . . . . . . . . . . . . . . . . . . . 4-20

Chapter 5 Deriving Metrics for Measuring the Value of

Human Factors in Military Systems Development.. 5-5

Background . . . . 5-2

Identifying and Defining Metrics ............ 5-4

Literature-Search and Data Base System . . . 5-6

Screening-and Selection Process 5-8

Disrivation of Metrics .. . . . . . . . . 5-8

Impact Areas, Metrics, and Related Measures . .... 5-13

Capability Impacts . 5-13

Cost Impacts . . . . . ........... 5-14

Compatibility Impacts' 5-16

Surmary . . . . . . . 5-21

References 5-22

Chapter 6 Methodology for Evaluating Human Factors

Impacts in Military Systems Development .... 6-1

Human Factors Impact Assessment:. Related Concepts,

Limitations, and Considerations • • .... . . . . . 6-2

X1
CONTENTS (continued)

Page

Methodological and Data Limitations . . . 6-5

Considerations for Human Factors

Practitioners . . . . . . . . . . . .. .. .... 6-8

Rigor vs. Broad-Based Analysis . ........ 6-11

iluman Factors Impact Assessment:

Conceptual Framework................. 6-13

Basic Framework and Steps. .... * . . . . .6-13

Techniques for Estimating Impacts:

Classification and Selection . ...... 6-32

Future Steps .. . . .. . . .. . . 6-38

References .. . . . . . . . . . . .6-40

Chapter 7 Conclusions and Recommendations ........ 7-1

Appendix A A Rationale and Precedent for Establishing

the Principal Human Factors Products

During System Development ......... . . A-1

Appendix B Derived Vocabulary List of

Human Factors Metrics ............. B-i

Appendix-C Examples of Human Factors Efforts Related,

to System Development Products and Metrics .. C-1
EXHIBITS

Exhibit Page

1-1 DOD Definitions.......................... ..... 1-3

1-2 Classification of Human Factors RDT&E ......... .. 1-5

1-3 Hypothetical Distribution of System

Development Outcomes With and Without

Optimum Human Factors Participation . . . . . . . . 1-7

1-4 Relationship of the Major Efforts of the Study. . . 1-10

1-5 Relationship of System Attributes to the

Choice of Impact Analysis Techniques ....... . 1-20

2-1 Development of MIL-STD-1472 . . . . . . . . . . . . 2-4

2-2 Cockpit Space and Information Requirements

for Several Navy Aircraft Weapon Systems . . . . . 2-8

2-3 Trend of Accident Rates for Typical Navy

Aircraft and the AV-8A Harrier ..... .......... 2-9

2-4 A Trident Missile Test at Cape Canaveral ..... 2-12

2-5 Effect of Human and Equipment Reliability

on System Reliability, RE x R, = RS ....... . 2-14

2-6 Typical Sources for Attribution of Human Error . . 2-15

2-7 Human Engineering Problems in Weapon Systems . . . 2-25

2-8 Human Factors Deficiencies Report for

Several Navy Aircraft Weapon Systems . . . . . .. 2-30

3 1 Principal Human Factors Products for

Major System Development Phases . . ... . . . . . . 3-3

3-2 'The Activities and Decision Points of OMB's

Circular No. 'A-109 Major System Acquisition Cycle . 3-5

3-3' A Comparison of Names Used for the

Major System Acquisition Model .. . . . . . . . . . 3-7

3-4 The Phases, of the Military's Major

System Acquisition Model . . . . . . . 3-9

3-5 Specific Human Factors Efforts and Products for

Each Military System Development Phase . . . . . . 3-15

xiii

, hr...'• ' ., . . . ...; . • " • . , .
EXHIBITS (_:ntinued)

Exhibit Page

3-6 Specific Human Factors Efforts During the

Mission Analysis Phase (0.) ....... ...... 3-23

3-7 Specific Human Factors Efforts During the

Concept Development Phbge (I.) ........... .3-34

3-8 Specific Human Factors Efforts During the

Demonstration/Validation Phase (2.) ?'......... 3-44

3-9 Specific Human Factors Efforts During the

Full-Scale Development Phase (3.) .. ......... 3-60

3-10 Specific Human Factors Efforts During

Production and Deployment (4.) ....... ...... ..... 3-73

4-1 Areas of Opportunity for Human Factors R&D

in the Technology Base to Contribute to

Military System Development .... . . . . . . 4-2

4-2 The Classification System for Human Factors R&D . . . 4-3

5-1 Hierarchical Relationship of Impact Areas,

Metrics, and Empirical Measures ...... ........... o.5-5

5-2 Sample of System-Related Terms and Associated

Dimensions (Unit of Measure) . . . . . . . . . ... .5-9

5-3 Preliminary Metrics Derived from the Literature'. . . 5-10

5-4 Sample Relationship Among Selected System

Related'Metrics and' Empirical Measures

(As Found in the Literature) . . . . . .. .... 5-11

5-5 System Life Cycle Cost Impact Area for a

Weapon System 5-12

6-1 Simple Illustration of an Impact

Assessment Matrix . . . . . . . . . . .. .... 6-4

6-2 Impact Assessment Framework . . . . .... ... . . 6-14

6-3 Goal Relevance Tree Hiezarchy of Goals -

Objectives - Criteria . . . . . . . . .. ... . 6-16

6-4 Input-Output Matrix for Problem Formulation . . .. 6-18

xiv

A n
EXHIBITS (continued)

Exhibit Page

6-5 Candidate Models and Techniques for

Human Factors R&D Cost Benefit/Impact

Assessments (sample only) . . ....... ............ . 6-34

6-6 Models and Research Techniques Useful in

Human Factors Analysis in Relation to the

Phases of System Development and Principal

Human Factors and Products . . ........ . . ... 6-39

C-I Illustrations of'Metrics Used to Link

Human Factors Products and Impact Areas .......... .. C-2

xv,
CHAPTER 1

INTRODUCTION

Project Objectives

In a sense, the present project represents an attempt to

confront an irony: while the sophistication of military systems

is increasing, the attention paid to the capabilities and limitations of those people who must operate and maintain them has

not been increasing. The arena in which this irony can be

most effectively confronted is the process of military system

development.

The overriding objective of the present project is to

enhance the value of all U.S. military systems. The route to

that objective that is paramount for the present effort is the

assurance that the human factors contribution to military system

development is timely, of an appropriate quantity, and, most

explicitly, that it is of high quality.

In another sense, then, we are concerned, with the quality

control of what is widely regarded as an essential ingredient

in the overall military system development effort. If we follow

our own precepts, however, we know that quality assurance depends

on accurate feedback and that feedback, in turn, depends on

evaluation.

Consequently, the key ingredient in achieving the broader

objective is the establishment of the means to evaluate the

contribution of human factors to military system development.

It is the construction of that key ingredient that has been the

immediate objective of the project.

1-1

a '' ' ',
Tne Boundaries of hLun FaLctors

In order to identify and measure the contribution of human

factors it is necessary to define human factors. It is also

appropriate not only to provide a aef.nition but also to provide

some classification of human factors R&D and a discussion of its

scope, for reasons that should become clear shortly. Collectively

these terms will be called the "Boundaries of Human Factors

Research and Development." Each of these boundaries will be

briefly described below.

Definition of Human Factors

A comprehensive definition of h-iualin f£ctors R&D that is used

by DOD and all services does not exist. This is not to suggest

that we do not already know essentially what human factors is,

but rather to suggest that we are not interested in a precise,

academic definition of human factors for this study.

For the purposes of this report, we know that human factors

is one of the four categories of people-related research funded

as part of the RDT&E brdget of the DOD. These four categories

have been defined by the Military Assistant for Training and

Personnel Technology in the Office of the Under Secretary of

Defense for, Research and Engineering in some recent briefing

materials. These definitions are provided in Exhibit 1-1.

Another DOD definition of human factors is contained in the

Technology Coordination Paper for FY 1978 (Department of. Defense,

1979):

Human factors technology is concerned with the

design, development, evaluation, and deployment

of manned systems so that human operators wculd

be able to operate and maintain military systems

at their optimum performance level. This includes

the systematic investigation of how the design of

a person's job and the tools that are provided

affect his capacity to do a job.
Exhibit 1-1

-DOD Definitions*

0 HUMAN FACTORSDevelopment of improved methods and technologies for the analysis, design.

and evaluation of equipment/systems for safer and more efficient operation

and maintenance.

* PERSONNEL & MANPOWERDevelopment of tachniquet/rlethods for utilizing available personnel resources

through improved aslection. job assignment, organizational analysis, and

management techniques to meet combat available and projected force needs.

* EDUCATION & TRAININGDevelopment of educational/training methods and media for managing,

designing. and evaluating new generation instructional systems for military

applications.

* SIMULATION & TRAINING DEVICESDevelopment of cost-effective training equipment and technology that

produce the needed performance for operation and maintenance of military

systems.

'This chart is from a brief provided by the Military Asuistant for Training and Personnel Technology

(OUSDR&E).

This DOD definition perhaps, defines the boundaries of human

factors in terms of its technical, domain; but metrics for determining the value of human factors and costs for assessing the

affordability of human factors need to be more adequately defined.

Concerning these two points, a few statements Can be made that

are useful in shaping this important definition:

* Metrics for measuring the value of human factors must

include measures of both system capability and cost.

We will also define the area of man-system compatibility

as.a category of metrics. Further, human factors efforts

on products must also relate to system performance.

1-3
* Affordability of human factors must be assessed not only

in terms of dollars spent but also in terms of cost

avoidance (through reduced selection, manpower, or

training requirements).

* User acceptance must be part of the value of human factors.

Changes in personnel attitude not only, contribute to

more effective use of the equipment or system, but may

have long-term effects on issues such as attrition and

retention..

In summary, there seems to be a consensus that human factors

includes effective integration of man's role and performance into

system operation and maintenance.

Classification of Human Factors RDT&E

Another boundary of human factors that is important is the

classification of the work. Again, DOD has standardized this

dimension for RDT&E. Exhibit 1-2 shows the three classifications

for human factors work and-gives som3 indication of what ix

included in each class- (Fiorello et al., 1979). A cursory

examination of these classes suggests that they may correspond

roughly to categories of R&D funding (6.1, 6.2, and 6.3), at

least with respect to the' technology base, but this 'possibility

must'be explored in more detail (see Chapter 4).

Scope of Uuman Factors

Perhap3 the most limiting boundary of human factors has been

the scope of its integration into the system development process.

In brief, there has been precious little utilization of human

factors in the early phases of system acquisition, particularly

in the Mission Analysis' and Concept Development Phases. There

has been more utilization in the Demonstration/Validation Phase;

1-4
and by far the g;reatest utili_,ti,;fý ~ cc,:uried in. the FullScale Development Pha~se, where thie LrL.uiticinal. human engineering

or man-machine interface design occýiii_-. £'crhaps, the reason for

the lack of human factors in the earlier pieases of systema development is the lack of recogniti,-ri tila( t:.Crte is botlh a product

and a payoff to be had during these ,eaziy T;hases.-

Classificationi of Ifuhi mti Vat , lo ; ME' 1

a Human Related Studies

IWhat are the capa.bilities and limitation~s of uperatc~rý &od rruaintainerl of ryftermsisubsystemns7

Emphasis is on increasing our state of knowledge concerning humans' operational- performance

Included are data. performance methods ielevailt to Ohsj!U.aracteristics. sensory and motor capa

bilities, and human information processing,

9 Human-Machine Refatd Studies

"N"o do we allocate fu.nctions, between people an~d squipment7" This is often termed "subsystem"

related 'because it concerns the design of a specific man-machine relationship. Included are efforts

dealing with conputer-aided methods for human engineering, workload measurement techniques.

designing for masintainability-, conirof and display design, and workspace layout.

* Human-Machirse-Misson Related Studies

"How'.we total configurations of people 'and equipment oonstruzted for maximum tactical and

strategic off oniventwl' This concerns the optimum combination of individual and team performance.

within the total operational system. This combination~ applies not only to major ground, sea, air

and subsurface systemns, but to the command and control of these systems as well

The principal. products truia eacti phase of systema acquilsition

should be a meaningful way to. represent. the scope of human factors

in a military system development. .These phase-products should

vary in specificity from the very conceptual requirement level to

the very detailed design level, just as is the caso with products

of engineering logistics, etc., during each phase. After a great

deal of analysis and synthesis (discussed primarily in Chapters 2,*

3, and 4), a set of human factors oroducts. has been defined. For

the purposes of this report, the principal human factors products

from each major phAse- of the system acquisition process-'are

identified below.
MAJOR PHASE

OF SYSTEM ACQUISITION PRINCIPAL HF R&D PRODUCT

Mission Analysis Phase - Development of the Role of Man as a part of a

Mission Element Needs Statement (MENS)

Concept Development Phase - Allocation of System Functions to Man as a

part of the Decision Coordinating Paper (DCP)

DemonstrationNalidation - Task Analysis and Determination of Human

Phase Engineering Requirements

Full-Scale Development Phase - Design of the Optimal Man-Machine Interfaces

A more definitive explanation of these products is offered

in Appendix A. Additionally, the payoff or value that can be

realized at each phase of system acquisition will be explored.

For the moment, the purpose of identifying the principal products

is to bound the scope of what we mean by human factors, and to

suggest that each product will require some costs and must yield

some benefits.

1-6
The Integration of Human Factors and

Military System Development

The problem of proper integration of the human factors

conL:ibution into the development of military systems can be

dramatized by asking the question, "What result would occur

if the development process involved zero human factors

participation?" A completely non-controversial answer to that

question can be cast in a statistical framework. That is, you

would get a distribution of outcomes. In fact, the best hypothesis would be that the distribution would be symmetrical and

near-Gaussian because of the multitude of influences at work.

Such a hypothetical distribution is presented in Exhibit 1-3

as the dashed-line cur. The solid-line curve representps the

characteristics of the distribution shift (again hypothetical)

when the human factorr contribution is introduced early and

continuously throughout the development process-. (One should

probably interpolate a family of distributions to represent

various degrees of human factors participation at less than

optimum levels.)

Exhibit 1-3

Hypothetical Distribution of Sygtem Development Outcomes

With and Without Optimum Human Factors Participation

ACCEPTANCE. SYSTEMS DEVELOPED WITHOUT

REJECTION LINE HUMAN FACTORS PARTICIPATION

, LopU - SYSTIEMS DEVELOPED WITH OPTIMUM

FREOENCYMAN FACTORS PARTICIPATION

.'SYSTEM EPFEctIVENEUcost RATIO

1-7
CarefUl examination of the distributional model presented

in Exhi.it 1-3 can help in the process of understanding various

aspects of the resistance to human factors participation on the

part of some managers of military system development efforts.

It is clear from the uiagram, for example, that some systems

could achieve acceptance (in the sense of going into full-scale

production) with little or no formal human factors participation.

Thus, if the program manager were "lucky," he could avoid the

cost of that contribution. In the same vein, there are-some

systems below the rejection line that did have optimum human

factors participation--substantiating the poiat that,such

participation does not guarantee success.

However, the diagram also reveals that a rational strategy

would be one which always incorporated human factors participation.

The program manager who does otherwise is simply playing against

the odds if the postulated relationships in Exhibit 1-3 are valid.

1

1-8
Overview of the Project

The present project constitutes very much of a team effort.

At the top, the planning, analysis, and review process was

carefully coordinated among representatives of the three services,

the COTR, and tne contractor. The project was scheduled into

three phases which were intentionally set up to be partially

iterative, but with a definite pattern of progression. At the

completion of each phase, a comprehensive review was made and

highly specific feedback and orientation was provided by the

review panel to the contractor. Between formal review sessions,

guidance was provided by the COTR and other advisors from the

HFE-TAG.

A useful overview of the component structure of the project

is provided by Exhibit 1-4. To a degree, each box corresponds

to a major effort of the prnject, but does not correspond to a

chapter in the report. Those boxes on the left side of the chart

represent the efforts to identify the contribution of human

factors in system development. The results of these efforts

are reported in Chapters 2, 3, and 4. Those boxes on the right

side of the chart represent the efforts to identify a framework

for evaluating human factors contributions. The results of

these efforts are reported in Chapters 5 and 6. The final box

corresponds to the final chapter in the report.

The main message that should be drawn from Exhibit, -4 is

*the intricacy of the relationship between the efforts and the

essential symmetry of that relationship. These attributes are,

at the same time, both the cause and the result of the teamwork

within the working level of the project.. Human factors content,

human factors research methods, military system development

procedures, program management practices, and the philosophy.

1-9
00

.3.

00
and methodology of cost-benefit analysis and related approaches

all h&- to be brought into the work effort and made to be

mutually constructive. The diversity is representee in the

diagram. The constructive aspect is symbolized by the symmetry

of the links.

Specific examples of particular applications of human

factors to particular systems and instances of various Attempts

to evaluate the impact of such applications are scattered

throughout the narrative, but examples of the linkage between

human factors products and impact analysis criteria as'mediated

by system metrics are also contained in an appendix (Appendix C).

.. 1-11
Human Factors in Military Systems

In Chapter 2, the question of the contribution of human

factors to the development.of military systems is considered

in historical context. The evaluation of the speciality is

shown to have been based on'tangible results. It is also

revealed that while the content of the contribution was often

based on quite rigorous scientific procedures, the evaluation

of the contribution, in the-sense of establishing human factors

as an essential ingredient in military system development, was

either neglected or handled by anecdotal evidence.

The anecdotal evidence is persuasive as far as it goes.

Moreover, the philosophical base upon which both human factors

specialists and system engineers and the other disciplines were

working tended to be an increasingly integrated one during the

period from the late 1940s to the late 1960s.

During this same period, the formalization of the role of

human kactors in military system development was accomplished.

The first generation of official directives has since been

revised many times, but the most recent versions contain clear

reiterations of the recognition of the requirement for contributions from human factors sources that was first officially

articulated in the 1950s. A summary of the formal basis for

human factors participation in military system development

is presented in the first section of Chapter 3. The formal

inetigations are seen to be linked to the structured chronology

of the 3ystem development process. This very structure sets

the stage for the further conceptualization of the mode of

contribution that is the subject of the balance of Chapter 3.

In these sections, the key message is that, the contribution of

human factors can be characterized as specific products, each

1-12
of which is tied to a stage in military system development.

Further, a simple model has been p repared which identifies

specific human factors efforts that make up the products of

each system development phase. Finally, in Chapter 4, there is

a brief treatment of how human factors R&E from the technology

base can also contribute to the principal human factors products

of each system development phase.

This conceptualization of human factors contributions

reinstates the condition of the tangible consequence that

characterized the early development of the speciality. As such,

the stage is further set for the possibility of a rigorous evaluation approach, one that represents a clearadvance over the

retrospective and anecdotal approaches of the past.

The Measurement of

Human Factors Contributions

The concetn for a feasible method to measure and evaluate

the human factors contribution to military system development did

not suddenly arise as part of an attempt to rescue human factors

from oblivion. The concern is primarily a manifestation of two

parallel trends in the much larger arena of public administration.

These trends can be briefly characterized as agrowing sense of

a need for-stricter accountability in the expenditure of governmental resources and the evolution of rationalistic procedures

for providing such accountability.

Military systems development is unquestionably an area of

expenditure of government resources. The major public policy

issues thatare associated with these processes have to do with,

allocation-of-resources decisions. Should public funds be'

invested in system X or system Y or system Z? Such allocation

decisions presumably' should be guided by analytic results in

the form of estimations of'relative returns on 'investment (ROI).

1-13 ,
Below the level of choice between X, Y, and Z are a series

of subordinate choices. Having decided on X as the best potential system,, what should be the proportional investment in.

technology A, B, and C? When either A, B, or C might make a

crucial contribution to the effectiveness of system X, what mix

will yield the best potential payoff?

At the most basic level, these questions are technical

questions for the political economist. And, indeed, it has

been from that source and from disciplines such as Operations

Research and others that have an intellectual affiliation with

political economics that the evolution of rationalistic procedures has come. A central contribution of this evolutionary

effort has been the formal methodologies under the rubric of

cost-benefit analysis. Consequently, the cost-benefit approach

is used as a starting point for the specific tailoring of a

methodology to meet the objectives of the present project.

As it turns out, the strict monetary criteria required in

formal cost-benefit anatlysis make it awkward to apply in its pure

form to our central problem. However, the basic logic of the

methodology and its inherent emphasis on quantification do provide a productive orientation. This orientation is reflected in

Chapters 5 and 6, which cover measurement metrics and methodology,

respectively.

The point is that the work reported here is well within a

conceptual movement that might, by this time, be called a tradition.

Of specific precedents, however, there have been precious

few. There are so few, in fact, that to achieve some perspective

on the present enterprise, it is necessary to examine another

parallel literature: that of evaluation in education and training.

1-14 "
The specific application of approaches resembling those of

cost-benefit analysis had its own broad history in the general

movement toward treating education in a more scientific way (see,

e.g., Campbell'and Stanley, 1963). The lift-off point came,

however, in the mid-to-late 1960s with the work of Suchman (1967)

and, more particularly, Stufflebeam (1968). The past decade has

witnessed the production of many hundreds of articles and reports,

most of which were focused on the actual evaluation of some particular educational or training program or a particular training

technology, and a few of which were focused on methodological

advances, as such, or what might be called the "management of

evaluation." An up-to-date example of the former is provided by

the recent works of Orlansky and String (1977 and 1979) in their

summary evaluations of the cost-effectiveness of flight simulators

and computer-based instruction in military training. An example

of the latter mode is provided by the work of Conley et al.

(1979), in their review of six major government-sponsored training

programs. Conley and his group, who work for the U.S. Office of

Personnel Management (formerly the Civil Service Commission),

were conceined with the very issues central to this effort: the

need for evaluation, its feasibility, and the procedures for

doing it in such a way that the outcome can be used to guide

management decisions.

The key point is that, again, a substantial precedent exists

for the general'approach and, in these instances, the approach

has been more or less successfully applied to an activity that

is considered to be somewhat "soft" in the sense of its being

exclusive of rigorous, quantitative measurement.

To return to the precious few direct antecedents, three

examples can be cited. Each is of quite a different sort.

1-15
One such case is the work of Geer (1979). In a sense, it

is the more remote precedent because the document as a whole

is devoted to the problem of how to conduct human engineering

analyses. Only a few pages (out of 220) are devoted to the

evaluation of the human engineering contribution (pp. 24-25).

The approach is built on a brief review uf the TMI-2 incident

and is affirmative and non-quantitative in character.

'A second case is represented by the work of Price et al.

(1979). In this case, evaluation is the central objective but

the substance to be evaluated is the human factors effort in

research rather than scientific system deveLipment. The approach

taken, however, is to relate research (technology base) outputs

to such achievements as cost avoidance in both military operations

and training. In short, the benefit side of the cost-benefit

ratio is stressed.

The third case is represented by a report compiled by the

BDM Corporation (1980) that focuses on the human factors aspects

of aircraft accidents. As such, the focus is much narrower than

that of the present effort. However, it is closely akin from a

methodological point of view, in that an attempt is mide to

evaluate the human factors contribution to aircraft accident

prevention using an analysis of return on investment.

In a sense, then, we have come full circle. The present

work, particularly as reflected by the contents of Chapters 5

and 6, represents an extension and focusing from three sources:

the general source of allocation analysis in the public policy

domain typified by the methodology' of formal cost-benefit

analysis; the parallel source exemplified by attempts to apply

such rigorous methods to the evaluation of the elusive'processes

of education and training; and the very restricted source of

specific attempts to evaluate human factors contributions through

a linkage to some aspect of system effectiveness.

1-16
Chapter 5 reveals how the focus can be realized by means

of quantitative measures that fit into the broader fra.-zwork

of system engineering. Chapter 6 describes how the measurement

operations can be made and how they can be interpreted through

the use of a conceptual model adapted from the basic structure,

of formal cost-benefit analysis.

The Concept of Impact Areas

A specific case is made in Chapter 6 that strict costbenefit procedures are not appropriate for the evaluation of

the contribution of human factors to military system development.

The central reason is that strict cost-benefit models require a

single ultimate criterion: monetary value. It turns out to be

not only awkward but occasionally ludicrous to reduce the human

* factor aspect to a dollar measure.

Consequently, a compromise was sought. *The goal became

that of deriving a methodology that would be as close to strict

cost-benefit methods as possible while covering the full scope

of the human factors contribution. The existing derivation of.

cost-benefit methods that met that goal was policy/impact

assessment.

The adoption of impact assessment as an exemplary procedure.

opened the door to another crucial adaptation,., That is, it was

discerned that the scope issue could best be met by adding a

criterion factor called compatibility to the basic two, already

labeled cost and capability.

Thus, a triad of criteria were &doptedz cost, capability,

and compatibility, and the members of the triad were designated

as impact areas to directly connote that the proposed methodology

was to be a version of policy/impact assessment.

1-17
The cost criterion is used in essentially the same way as it

is in the cost-benefit methodology. It is expressed in dollars

and pertains to the total life cycle costs of a system.

The capability criterion is very close to the benefit component of the cost-benefit -.iethodology except that it pertains

ultimately to system-mission performance and is not reduced to

dollar value in the impact assessment version of the methodology.

The compatibility criterion is uniquely responsive to the

substance of the human factors contribution. As a concept in

its own right, it links logically to the consensual goal of

human engineering, which is to achieve an optimum match among

human, machine, and mission.

Because compatibility is something of an innovation in the

lexicon of evaluation methodologies, it seems useful to give it

a little extra attention. Specifically, we can break it down

into its constituent parts: 'physical compatibility, physiological compatibility, and psychological compatibility. Physical

compatibility refers to the human as a physical object having

certain dimensions of size, weight, reach, etc. The design of

workplaces such as the cockpit of an aircraft must provide for

these physical attributes.

Physiological compatibility refers to the human as a functioning organism. Thus, metabolic processes such as respiration

must be taken into account in design. Also, factors such as

visual acuity under differing conditions of illumination are

physiologically based and the designer errs if he or she specifies

a display that cannot be read under operational conditions.
Psychological compatibility is the most complicated of the

constituents. It breaks down further into behavioral and attitudinal components. Behaviors relate primarily to established

habits such as the habit of turning a knob clockwise to increase

some effect and counterclockwise to decrease. Design should be

responsive, to such habits.

The attitudinal component is manifest mainly in the

phenomenon of user acceptance. Even under strict military

discipline, users can reject involvement with a particular

system., The reasons might not be entirely logical but can be

nonetheless powerful. The prediction of such reactions or even

their measurement after-the-fact goes well beyond conventional

engineering considerations but lies well within the province

of human engineering. It is in this domain that the need for

special observational methods and measurement techniques becomes

most obvious and where some of the particular justifications for

adaptations or extensions of strict cost-benefit models derive.

It is reconciling these variants with the mainstream of system.

engineering--bringing these aspects back into the family, so-to

speak--that constitutes one of the major contributions of the

present project.

Summary and Synthesis

The total effort involved in the present project has already

been revealed to be complex in the sense of being a composite of

several different topics and, orientations. This point is made

even more dramatically in Exhibit 1-5, which represents an attempt

to characterize the intended outcome in a composite summary format.

The central core of the representation is the system development process itself. It is shown as consisting of four phases

denoted by Roman numerals.

1-19
- ur

CcC

V--j

102
The lower portion of the diagram stresses the role of the

technology as provider of the problem-solving resources. In

particular, the engineering disciplines are specified, including

human factors.

In a similar way, the role of technology base activities is

shown: to support the evolution of the problem-sOlving resources.

The upper part of the diagram emphasizes the feedback mechanism introduced at the beginning of this chapter. The function

of the impact analysis, in the first instance, is to provide

techniques for data gathering. The convention used is intended

to show that different techniques can be used under different

circumstances and that some choice must be made.

The outcome of impact assessment is then shown to feed back

primarily to the problem-solving resources box because that box

is also the locus of the management decisions in military system

development. That box, indeed, is the ultinate target of the

project reported here.

"1-21
References

Campbell, D.T., & Stanley, J.C. Experimental and quasi-experimental

design for research in teaching. In N.L. Gage (Ed.), Handbook

of re8earch on teaching. Chicago: Rand McNally, 1963.

Conley, M., et al. (Workfozce Effectiveness and Development

Group). Report of the Training Evaluation Demonstration

Project: Issues ana recommendations. Washington, D.C.:

U.S. Office of Personnel Management, March 1979.

Erikson, R., Miles, J., & Secrist, G. Technology Coordination

Paper: Training and personnel technology. Department of

Defense, August 1978.

Fiorello, M., Jukes, A., Kidd, J., O'Connor, D., Price, A.,

Swezey, R., & White, T. DOD people-related RDT&E: Reference

document. WN ML 909-1. Logistics Management Institute,

Washington, D.C.,, March 1979.

Geer, C.W. Human engineering procedures guide. F33615-79-C-0520.

Aerospace Medical Research Laboratory, WPAFB, Ohio.. Boeing Aerospace Company, Seattle, Washington, December 1979.

Orlansky, J., & String, J. Cost-effectiveness of flight simulators

for military training. Vol. I: Use and effectiveness of

flight simulators. IDA Paper P-1275. Institute for Defense

Analysis. Office of the Director of Defense Research andEngineering, August 1977.

Orlansky, J.i & String, J. Cost-effectiveness of computer-based instruction in military training. IDA Paper P-1375.

Institute for Defense Analysis. Office of the Secretary of

Defense for Research and Engineering, April 1978.

Price, H.E., & Sands, F.F. (Eds.). The utilization of people- related RDT&E. Navy Personnel.Research and Development Center, San Diego, CA, April 1979.

Stufflebeam, D.L. Toward a science of educational evaluation.

Educational Technology, July 1968.

Suchman, E.A. EvaZuativte reeearch. New York: Russell Sage Foundation, 1967.

U.S. Air Force School of Aerospace Medicine. Definition of

investigation areas for human factors aspects of aircraft

accidents. F33615-78-C-0608. BDM Corporation, February

1980.

1-22
CHAPTER 2

A RATIONALE FOR THE VALUE OF HUMAN FACTCOS

IN SYSTEM DEVELOPMENT

The substance of this chapter includes a brief review of the

historical evolution of human factors 'in military system development and a compilation of the established arguments in favor of

the utilization of tfie human factors resource by program managers.

One conclusion is that those involved In one or another

aspect of military system development--particularly dur.ng the

past ten years or so--have allowed what should be a strong

collaboration-type relationship to take on some of the features

of an adversary relationship. The present project constitutes

an attempt at "rapprochement" by expanding upon the established

rationales for the use of human factors resources through the

elucidation of methods permitting a more explicitly objective

assessment of their value.

For those readers who are already convinced of the value'of

human factors in military system development, this chapter 'can

serve only as a quick refresher course with some special emphasis

on the constraints involved in achieving timely participation.

For the more skeptical, the chapter should reveal- that the effort

to promote human factors in military system development does

indeed have an established 'rationale that is substantial, even

though its powers of persuasion have not been overwhelming in.

recent times.
Historical Background of

Military Human Factors

The history of human factors in the military has been

reviewed many times (see Meister &' Rabideau, 1965; Christensen,

1976; and Chaikin, 1978) and will not be dwelt upon here.

Essentially', there is a consensus that the major requirement

for human factors contributions to system design occurred during

World War II and grew out of earlier work in aeromedical research,

industrial psychology, and industrial engineering. As expressed

by Meister and Rabideau (1965):

With World War II a new factor entered which had

tremendous consequences for human factors. This was

a period of increases in technological complexity,

involving such new systems as radar and sonar and

highly complicated fighter and bomber aircraft,

designed to be used in new environments and under

highly demanding conditions. These conditions, under

which the operator could not function as readily as

he had before, complicated medical, physiological,

and psychological requirements for design.

In the lore of human factors, the oral tradition asserts

that in the early stages of mobilization,' the Army in particular

had a surplus of psychologists. For lack of a betterassignment,

two or three of these individuals were given the "detail" of

reviewing a rash of P-47 accidents. Several of these aircraft

had crashed when the flaps had been lowered on takeoff when the

wheels were stili down. The problem turned out to be a confusion

of two controls (flaps and wheel retraction) by pilots who had

been trained on an aircraft in which the flap and wheel controls

were reversed from their location in the P-47 cockpit. The

ability of the psychologists to "solve" the problem convinced

some key officials that it would be a good idea for a psychologist

to look at all new-systems to ensure that no "traps" were included

in the design for the unwary operator.

2-2-'
Later, when the major electronic systems were being brought

into operation, human factors participation had become more or

less routine. The so-called "team approach" that included the

human factors scientist had become a basic rule in the minds *of

such system pioneers as Churchman, Ackoff, Arnoff, Roy, and Flood.

It is notable that at this still-early stage in the history

of the speciality one generic problem was characterized as system

complexity. We were already seeing systems with large numbers of

interactive components where the functional relationship between

such components was not self-evident to ordinary operators or

maintenance technicians and where many of the components were not

"familiar" to such personnel. The current situation with respect

to military systems is a direct extension and major expansion of

the trend that started at that time. Forty years later we are

still talking about the complexity of military systems and their

demands on the capabilities and limitations of people.

No more history need be provided here except to make two

further points. The first is that during the years from World

War II until today, an enormous amount of h an factors data has

been developed, much of which has been inco porated into handbooks, guidelines,'specifications, standard , directives, etc.

An excellent overview of this historical de elopment is provided

by Chaikin (1978), who, also chirts the deve opment of MIL-STD1472, Human Engineering Design Criteria for Military Systems,

Equipment, and' Facilities (see Exhibit 2-1) * The second point

*Along these lines, it might be useful to c nsiier the possibility

that the human factors vocation is to some extent a victim of its

own successes. In a sense, the early find ngs (and admonitions)

of human factors specialists have become a standard engineering

practice. If the technology had remained :he same or had evolved

only very slowly, the whole story would'have concluded with a

condition in which the human factors specialist has "worked

himself out of a job."

, •2-3
Exhibit 2-1

Development of MIL-STD-1472

WDT Exhibit 57-8 1 Aug 57

WDT Exhibit 57-8 1 Ma 58

AFEM Exhibit 57-SA 1 Nov 58

5 SMIL-STD-03 o 58 ASMA OD 844 25 Mar 60 EFK Nov , [ASMASTD4

1Oct6 1

ASMA STD-434A 24 Apr 63

MIL-sTD-803A1 27 Jan 64 MILORD-1248 20JAn6 6

MIL-STD-803A2 1Dec64 L= -1248 NI 30Aug66

MIL-STD-803A3 19 May 67

MIL-STD-1472 9 Feb 8 1

MIL.STD-1472A 15 May 70

MIL-STD-1472B 31 -Dec 74

MIL4ST-,14728 Ni 10 May 76

MIL-STD-1472B N2 10 May 78 J

2-4
is that despite this enormous accumulation of data and the

directives, standards, specifications, etc. for enforcing the

application of human factors in military systems, that portion

of human errors which generally can,'be attributed to deficiencies

in equipment system design is still a severe problem in the Army,

Navy, and Air Force today.

The issue of complexity and the problem of human error and

its relationship to system effectiveness will be discussed in

the next two sections. A separate section on optimal manned

systems and the compatibility factor will follow. The final

three sections will address the questions of opportunity, using

what we know, and the affordability of human factors.

2-5
Military System Complexity

and Human Factors

The notion that the complexity of military systems was

the driving force behind human factors (or human engineering)

literally coming to life in World War II is still with us.

For 30 or 40 years someone has been saying that the current or

abcut-to-be-introduced systems are complex and require special

consideration of human elements. During this same time period,

however, no one seems to have defined complexity, nor to have

quantified it. What is apparent is that systems are at least

different today from what they were 30 years ago, and that most

systems in the military are being rapidly replaced by newer,

high-technology versions. In any event, the complexity theme is

appropriate to establish the value of human factors in military

systems development.

Complexity (or some synonym) has been mentioned at all

levels of the Department of Defense recently. For example,

Dr. Harold Brown, the Secretary of Defense, in his annual

report to the Congress, FY 79, stated-the following (pg. 9):

Modernization, in some cases, has brought with it shorter mean-times to failure, longer repair times, and increased training requirements, as well

as greater sophistication and capability of equipment.

Inflation, increased pay, and the need to modernize

our forces have meant curtailed funds for operation

and maintenance.

Accordingly, we must keep up our training not

only because U.S. forces may be sent into action with

very little advance warning, but also because we rely increasingly on the sophistication of our equipment

to compensate for potential superiority in enemy

numbers. It is equally essential that our war reserve

stocks be maintained, mostly for our own needs, but

to some degree for Asian allies as well. At the same

time, we must raise the percentage of our equipment

that is combat-ready because, owing to unit costs,

we have less of it to bring to bear in an emergency.

2-6
To put the matter bluntly, unless we are prepared

to maintain these components of readiness, collective security and deterrence will be seriously

undermined .

Dr. Brown refers to the sophiatication of modern equipment

rather than complexity. He also talks about the need to raise

the percentage of equipment that is combat-ready; and while he

does not say so in the excerpt, all military system readiness

is measured in terms of Loth equipment readiness and personnel

readiness. Obviously, a piece of equipment that can perform its

mission, but that wilZ not because the operator cannot operate

or maintain it, is not combat-ready.

General William E. DuPuy, in a speech in 1977 describing

the Army training system, also deals with the complexity issue.

General DuPuy talks about problems converging on the Army and

makes the following statements:

The first thing that is converging is all that new equipment. The rate of introduction of new equipment will increase exponentially. The first thing that

goes like that is the amount of equipment that is

going to arrive in the Army between 1978 and 1985.

And the Army has to digest it. Traditionally armies

have a hard time digesting new things. We all do, especially organizations like armies. Anyway, that's

the first area of, convergency. You are going to be innudated with new tanks, new MICV's (Mechanized Infantry Combat Vehicles], new TACFIRE's (Tactical Fire Direction Systems], Battery Computer Systems, Patriots, ROLANDS, a whole new set of communications

equipment, a whole new set of electronic warfare

'equipment, and on and on . .

Complexity is another problem converging on the'Army. Every single new system being fielded is more complex than the one it replaces. This complexity is getting to be more of a problem than just operating and main- taining it. But the complexity of this new set of equipment raises, if you-will, integrated complexity.

2-7
The issues of introduction of new equipment and complexity

can be dramatized by examining Exhibits 2-2 and 2-3. These

exhibits are from recent briefings on the human factors program

at the Naval Air Development Center. Exhibit 2-2 indicates the

dual problem, of increasing information requirements for aircraft

operators and the decreasing available cockpit space for providing

displays or controls. As may be seen, the last weapon system on

that chart, the AV-8 (Harrier) V/STOL aircrdft, has approximately

one-third of the cockpit space that the F-4 aircraft has.

Exhibit 2.2

Cockpit Space and Information Requirements

for Several Navy Aircraft Weapon Systems"

Available Cockpit Space (in2 )

N 14/ INCREASING

S1300 ' INFORMATION

0 REQUIREMENTS

L 1111 AV-

"*Thls chart Is from a brief Iprovidud by the H.,uman Facor Ingineeringl Division of the Naa Air O~eslop~ment

2-8
Exhibit 2-3

Trend of Accident Rates for Typical Navy Aircraft and the AV-8A Harrier

20

18.02

0

8 15 TYPICAL TREND FOR CONVENTIONAL

NAVY AIRCRAFT L\l

CL

cA 10 PRESENT TREND

FOR AV-8A Iz 6.17

5 4.20 U S • 3.06 I

1 .6 3 .9 2

0.

1971 1972 1973 1974 1975 1976 1977

FISCAL YEAR

*This chert is from a brief provided by the Human Factors Engineering Division of the Nava Air Dewvaopment Center,

Warmlnstar, Pennsylvania.

Exhibit 2-3 shows a disconcerting trend, namely, that the

current V/STOL accident rate is increasing. This is contrary to

the experience typically encountered when new aircraft are introduced. Furthermore, "pilot factor" as a contributing cause seems

alarmingly high. With respect to this last point, the data shown

in Exhibit 2-3 represent 21 accidents, 16 of which occurred in

the V/STOL flight regime (i.e., conversion flight, landing, or

takeoff). Of these 16, 11 had pilot factor as a contributing

cause. It should also be added that the Naval Air Development

Center has since initiated a program to provide human factors

support early in the design of V/STOL aircraft.

2-9-
Human Error, Human Factors,

and System Effectiveness

The operators and maintainers of military systems do make

errors and many, if not most, of. these errors can be traced to

faulty design. In a classical study by Shapero et a]l. (1960),

a survey of nine Air Force missile systems showed that human

error contributed from 20-53% 'of system unreliability. These

percentages referred only to human errors during field exercises

with these systems, i.e., errors during the launch or relaunch

activities. The study did not attempt to go back into the life

history of each system and find human errors in the design and

fabrication of each. Swain (1964) commnents on the' Shapero report

in the following way:

Human errors have a greater effect on system reliability than many people realize . . . . In most

cases, it is more efficient to redesign procedures

and equipment which can minimize or even eliminate

certain types of human errors. Engineers can be

taught valuable design principles to minimize human

error, but engineers cannot take the place of a

human factors specialist.

Swain also emph~asizes the human engineering problem based on a

Sandia investigation of human error which analyzed a large number

of production defects at the plant of an AEC prime contractor.

It was found that 82% of the defects could be directly attributed

to human error.

Meister an Rabideau (1965) also discuss the-problem of

human error and develop the link between human-error and human

factors and system effectiveness. They quote some human error

percentages (pa e 15) from other sources which estimate:

that 40% of the problems uncovered in missile

testing derive from the human element. 63.6% of the

(shipboard) collisions, flooding and grounding could

be blamed on human error. Reports produced by the

United States Air Force indicate that human error

was responsible for 234 of 313 aircraft accidents

during 1971

2-10
More recently, Coburn (1973), in a paragraph describing

the benefits from systematic human engineering, again states

the problem of human error and its relationship to human factors

(human engineering):

The payoff in conducting a systematic human

engineering program is realized in improved system performance, reduced training cost, improved manpower

utilization, fewer errors and accidents, reduced

maintenance costs, higher probability of mission

success, and improved user acceptance. Without

applying a systematic human engineering program,

attainment of an effective ship system is fortuitous

and improbable.

Failure to apply systematic human engineering can be

costly--research indicates that typically up to 40%

of all ship system malfunctions are attributable to

human error.* Even increasing automation cf ship

systems does not eliminate the application of human

engineering programs, since man is still involved as

a user and maintainer.

To maximize the payoffs previously cited, human

engineering must be applied throughout the ship

system life cycle. It starts with inputs to planning documents and continues throughout concept formulation,

contract definition, engineering development and

production, test and evaluation, and finally fleet

operations.

*Pickrel, E.W., & McDonald, T.A. Quantification

of human performance in large complex systems.

Human Factors, 1964, 6, 647-662.

Finally, the effect of human error was dramatically illustrated

recently in Time magazine (January 8, 1979) by the photographs

shown in Exhibit 2-4 and the simple description noting human error,

which is included here verbatim.

2-11
Exhibit 24

A Trident Missile Test at Cape Canaveral

Photographs from TiME, January 8, 1979 by Mark/Norman Summner

A MISSILE'S UPS AND DOWNS

It was one of the most dramatic flights in missile

history, all recorded in these exclusive photographs

for TIME. The Trident had hardly left its launching

pad at Cape Canaveral when it started to wobble

wildly. About 500 ft. in the air, it suddenly made

A boomerang turn; then exploded and smashed to earth

125 yds. from its takeoff site. The fallen missile

burned fiercely for 10 min., sending a column of

white smoke soaring skyward, but no one was injured.

It was the third failure out of 17 Tridents tested

to date, and the cause was human error. Someone

included an extra step in the check list, which led

to the guidance system's shutting down a half-second

before ignition. The missile, which is designed for

submarine launching, was out of control from the

moment of takeoff. The Navy's calm term for the

Trident's destruction: "No test."

2-12
The point of the above discussion on human error is that

the real value of human factors is to reduce actual or potential

human error or increase human reliability. It is not just human

reliability but the consequences of human reliability on system

reliability that are important. The relative impact of human

error on system reliability has been shown graphically by Meister

and Rabideau (1965). They discuss this graph (Exhibit 2-5) in

their own words as follows:

shows the relationship between human reliability RH and equipment reliability RE and their

contribution to overall system reliability RS.

Thus, an RE of .85 coupled with an RH of .90

produces an RS of approximately .78. Lower the

human reliability to .30 with the same equipment reliability of .85, and system reliability now

becomes .25. It is apparent, therefore, that

anything which decreases RH must be a primary

concern of the human engineer. It is assumed

that much of this error results from inadequacies in system design which create, favorable conditions

for error occurrence.

Error potentiality resulting from inadequate design

can only be eliminated by systematic and continuing evaluation through the development of that design. If the human factors aspects of system performance

are not routinely evaluated, it is very likely that they will be overlooked, with the result that

the particular system function involved will be

developed inefficiently.

2-13
Exhibit,2.5

Effect of Human and Equipment Reliability

on System Reliability, RE x R11 = RS

Raft,- Re

Rags SY*mrbMy

Afrn 0"90

jr, a .3

As8= .70,

n- a 0 O

Nsa. 030

020 0.40 CAD 010 14O

All human error is not, of course, a function of poor human

factors. In a paper included in the Minutes of the 12th Training

and Personnel Technology Conference on' "Human Factors in-Weapon

System Test and Evaluation" (Taylor, 1978), John Miles of the

U.S. Army Human Engineering Laboratory developed a chart of

various. sources for attribution of human error. This chart has

been modified and expanded and is included as Exhibit 2-6. As.

can be seen, there are five basic sources which can account for

human error in military systems; and there are five types of

deficiencies in equipment system design which human factors

application should resolve or minimiz'e. It is also worthy of

note that the five typical sources of human error (the top of

the chart) may also be turned around and viewed as the five

'principal techniques for'achievinq performance competency in

2-14
lull -il I

I,'. lft"'4li

~I L: f 12 1 i

L01

2-

' 4-

I -0 U i g .

l~iiil "iii

1111.... ii
personnel. These five techniques for performance achievement are

discusseO by Price (1979). He points out that human engineering

(human factors) is seldom used early in system design to obtain

performance competency. Traditionally, performance competency

is obtained through selection and training; and, more recently,

emphasis is being placed on improved job performance aids. Still

more recently, motivation is receiving more emphasis. All of

these approaches have their place and their problems. In brief:

e Training is becoming extremely expensive and is "under

the gun" by DOD and Congress.

e Job performance aids are showing great promise, but have

yet to prove their cost effectiveness on a long-term

basis.

* The ability to select for aptitude is not a course that

is readily a-.ailable in the military any more, as is

pointed out by Rook (1965):

The more rational course is to use the capabilities of people as we find them and to

create situations in which the job at hand

can be done by the people we get, rather than

only by the people we wish we had.

motivation SFinally, as a way of reducing human error is

important; and it is also important for .the long-term

effects that can be achieved through job design (see

Price, 1979) which affect the total personnel system.

However, from a performance point of view, Rook has

pointed out that umotivational schemes have nearly always,

produced transient results in which maximum performance

increases are usually about 30% or less.* Rook also

discusses that a much greater potential for reducing

error is by modifying the performance situation (human

factors). He states that:

2-1 ',,
The amount which errors can be reduced and

quality improved by changing environmental

conditions is virtually unlimited--if your

money and time are also unlimited. By

pinpointing error-likely situations and

designing around them, almost any error

can be reduced to a tolerable level . . . . The most significant point to be made

about situational changes is that they are

relatively permanent.

"2-17
Optimal Manned Systems--

The Compatibility Factor

In the first chapter of this report we identified cost,

capability, and compatibility as the three impact areas that

should be used to assess the value of human factors in military

system development. The preceding section discussed human error

and system effectiveness wherein system effectiveness relates to

the impact areas of both capability and cost. This section will

acquaint the reader with the third impact area, compatibility,

via a discussion of optimal manned systems.

. . . where those systems which man himself plans, designs, and constructs are concerned, I submit to

you that there is no such thing as an 'unmanned

system.' It must be appreciated as axiomatic that

all such systems have a man or men somewhere in the

loop between planning, attempting, and replanning. 1"iether the question is one of foolproof assembly,' skillful maintenance, unerring operation, or parrying counteraction, man's performance in relation to the

equipment which is involved will decisively affect

the accomplishment of the 'system.'

The corollary to this axiom is that the impact of

man's characteristics must be taken into account

in the design of the equipment if the system is to

possess maximum probability of achieving the goal for which it was designed in the first place. (Flickinger and Hetherington, 1957)

The preceding quotation is an excerpt from a paper presentedby Brigadier General Don D. Flickinger at a symposium on human

factors in system engineering. The point is just as valid today

as it was when made over 20 years ago. Moreover, there is such

a thing as an "optimal manned system" in which the demands of the

operational system exploit man's unique capabilities and compensate for his limitations. in other words, an optimal manned

system is one in which man is' most compatible with his designed

2-18
role, functions, tasks, and system interfaces. This

"compatibility factor" and the notion of an optimal manned

system may be best understood by considering some of man's

characteristics in a system setting. Generalized statements

about human capabilities and limitations are of course subject

to the interpretation of specific systems and environments.

However, the generalities offered below serve to establish an

awareness of the compatibility factor.

Unique Human Capabilities

In complex systems, man makes the most significant contribution in situations where all of the performance alternatives

cannot be specified in advance and thus pre-programmed. Humans

will adapt to any changes in the system input and environment.

This characteristic is mandatory where the relations between

input and output may require restructuring in the course of

mission accomplishment and where all operations cannot be

reduced to logical, preset procedures.

Man makes possible a more diversified system mission.

He can translate uncertainty into probability and deal with

low probability/high value exigencies; and he can develop a

"behavioral strategy" when no optimum strategy can be specified.

His ability to perform a variety of functions and to utilize

alternatives means more is accomplished, including multiple

mission performance, recallable mission attempts, less vulnerable

mission accomplishment, and vehicles returned for re-use.

Man has the ability to make and report unique observations

and experiences, including observations on his own performance,

observations on system performance, observations of a scientific

nature, and incidental intelligence.

2-19
Man will enhance system reliability. Significant human

capabilities which may not be easily duplicated by a machine in

so small and reliable a package include:

'e Selection among alternative ways of achieving a mission;

9 Integrating a large amount of information gathered from

experience and bringing it to bear in a novel situation;

e Sensitivity to a wide range of stimulus patterns;

* Capability to detect signals through noise;

* Capability to act as an intermittent servo in the performance of a number of. different systems or equipment.

Human Limitations

Man comes in only one physical model and can only be integrated into the system concept as a physical whole, with certain

general characteristics of size, weight, shape, strength, etc.

Man has certain performance limitations such as sensitivity,

reaction time, number of information channels, rate of operation,.

environmental stress tolerance, etc.

There is a definite price to pay for maintaining reliable

performance potential in man, in terms of training, maintenance

of proficiency, manuals and other job guides, and human factors

design.

Man has life support needs. His performance deteriorates

rapidly when these physiological needs, such as nourishment,

environmental protection, sleep, comfort, and general health

maintenance are not satisfied.

2-20
/

Man has psychological needs. His performance usually

deteriorates over prolonged periods of high stress or nonactivity, and can change significantly as a result of such

psychological variables as motivation, frustration, conflict,

fear, etc.

Compatibility and User Acceptance

A review of the capabilities and limitations just stated

will reveal that compatibility is physical, physiological,

behavioral, and attitudinal. The physical and physiological

compatibility is basically obvious and non-controversial.

Man cannot be expected to perform if he cannot fit into a crew

compartment or he cannot reach the controls or he cannot breathe.

Behavioral data and man's sensory, perceptual, cognitive, and

motor capabilities have to some extent been used during system

development, at least with respect to human engineering of

man-machine interfaces. However, man's attitudinal system

(i.e., acceptance) has not been systematically included in

man-machine systems design. This is a serious error as a highly

motivated man can compensate to a considerable extent for poorly

designed equipment or he can get the best out of equipment he

likes. It is true that if a system is designed so that it is

easy for the man to grasp and manipulate the controls, and if

the displays are easy for him to perceive and understand, then,

certainly, the system will be more acceptable and utilization

will be enhanced. However, these traditional human engineering

efforts are not sufficient in themselves to account for total

acceptance. A man dissatisfied by a' particular system design

due to status, economic, or survival fears, or simply a desire

to operate the system manually because he enjoys it, may not

properly use equipment which has been designed to meet all other

criteria, He will reject, underuse, or misuse the system,

consciously or unconsciously. Consequently, system effectiveness

may suffer regardless of the inherent reiiability of the equipment

per se.

.2-21

__ -- - - - - - - - -
The user acceptance issue may be even more prominent in

advanced systems concepts incorporating extensive automation.

For example, one approach to achieving the long-standing

objective of highly reliable aircraft landing operation under

severely degraded visibility conditions lies in the increased

application of automatic flight control techniques. The development of highly reliable automatic control systems (such as

landing systems) by the best engineering talent available will

not solve all the problems associated with their effective

utilization and will, in fact, create new problems. For many

years to come, such complex systems will be man-machine systems

that, at a minimum, will require a man to initiate the machine

functions, monitor them, and decide when to disengage and override them. If all man-machine interfaces, including the user

acceptance, are not optimum, system effectiveness cannot be

optimum.

The principal attempt to optimize interfaces is through good

human engineering design. However, traditional human engineering--

usually performed after the system has been designed and the

breadboard equipment developed by engineers--has been applied as

if man were perfectly rational, and as if it were only necessary

to consider such aspects of man as his perceptual and motor

capabilities. In actual fact, however, it is equally, if not

more, important to consider man's potential attitudes toward the

system and to realize that these attitudes are influenced by his

fears, anxieties, aspirations, and social customs. If the user

acceptance is poor and performance is degraded, then coercion or

appeals to pride, team fellowship, or patriotism may serve as

poor seconds. On the other hand, if systems are designed

initially with high acceptability just as they are with high

reliability, then less human maintenance is required just as

less hardware maintenance is required, and an optimal manned

system is possible.

2-22
In summary, this section has briefly discussed the philosophy that all complex military systems are manned, that there

is such a thing as an optimal manned system, and that the

compatibility between the operational system demands and man's

unique capabilities and limitations--physical, physiological,

behavioral, and attitudinal--can impactsystem performance and

cost. For an optimal manned system to result from a system

development program, human factors considerations must be

an integral part of the acquisition and development process

throughout.

The first part of this chapter has attempted to provide

a simple rationale for the inclusion of human factors considerations (by human factors professionals) in system development,

the remaining sections of this chapter will address three major

implementation questions:

1. Are there opportunities for human factors in system

development?

2. Do we use what we know about human factors?

3. Are human factors contributions affordable?

* * 2-23
Are There Opportunities for Human Factors

in SystemDevelopment?

It has already been offered that human factors as a profession has data and methods to offer which will effectively

impact system cost, capability, and compatibility.' The question

remains as to how these prescriptions are to be influential.

One might even question the feasibility of achieving partial

implementation. The technical core of an answer to these

questions is presented in Chapter 3. However, it is a useful

transition to consider the opportunities for implementation at

this point from a preeminently human factors point of view.

Such a point of view can be simply summarized by a table taken

from Johnson and Baker (1974). It follows as Exhibit 2-7.

Basically, this chart simplifies the weapon system acquisition

process and identifies 'a number of human factors pr:3lems (or

requirements for human factors input) during the development of

a complex system. Chapter 3 provides the more formal and more

official requirements.

Given that there are opportunities, are human factors data

and methods used? 'That is the next question.

2-24
Exhibit 2-7

Human Engineering Problems in Weapon Systems

Stag of System Life Source of Problem Recognition

Preparation of System Requireents Review of requirements. Abstract and analytic, rather

than empirical. Criteria identification.

Conduct of Design (or Feasibility) Study Study of alternative approaches.

Preparation of outline of functions to be performed.

Delineation of data relevant to these functions.

Preparation of flow charts, or other detailed summarization, to describe the functions.

Performance of capabilities analysis. Stateof-the-artf

State-of-the-people determination.

Development Planning Allocation of functions within the defined system

boundaries (man can do better/computer can do

better determinations).

Linking together the functions in the system. Net work

determination.

Assignment of functions by type of individual involved,

Design of Development Model Preparation of performance des criptions; task analytic

job description.

Analysis of individual workloads.

Study of individual interactions.

Delineation of groups' personal space (by function).

Delineation of individual's workspace, layout within

group.

Determination of location of system components.

Study of alternatie personal space layouts.

Analysis Qt human information raqwremnents.

Analysis of human response requstements.

Design of systemn intefaces

* ~Determination of aim illery job supports.

Delineation of procedures

Study of equipent initegratn for simplification.

Evaluation of Prototype (OrseaboswtI Model Eveluation through mockups and, eventually, prototype

Production Model Evaluation of system (engIneering and procedures)

t~rem Johnsen 0 @1.r te4J

* 2-25
Do We Use What We Know

About Human Factors?

The true "golden era" of human factors occurred from the

early 1950s to the early 1960s when, among other events, there

were always four or five major electronic systems under development by the Air Force. From the mid-1960s to the mid-1970s,

however, the pattern has been one of decline and deterioration.

It is legitimate to ask why.

There appear to be many reasons why human factors contributions have been questioned and why the human factors knowledge

has not been applied to systems or accepted by system sponsors.

Some of these reasons appear to be organizational factors,

personnel factors, management factors, communication factors,

and a host of other factors which are not directly germane to

this project. Nevertheless, those reasons and others may derive

from the basic problem facing human factors in the military:

that the human factors researcher and practitioner are too

frequently called in after system design and development has

proceeded to a point where costly redesign and retrofit is

necessary to implement human factors recommendations. This is

the biggest complaint of researchers and practitioners who

believe they have something of value to offer. Therefore, it

seems worthwhile to examine in 'detail some aspects of the problem

of waiting too long to integrate human factors into the weapon

system acquisition process.

First of all, the question should be asked: Is this really

a problem? The results of a survey (undated) conducted by Meister

and received in July 1979 indicate that the problem still exists.

Meister surveyed the three major participants who determine how

much human factors R&D is done and how it is perceived.' These

participants are R&D laboratories, R&D contractors, and human

2-26
factors practitioners in the defense industry. Two excerpts

from Meister's paper that deal with the practitioners' answers,

and one from his Conclusions section serve, we believe, to make

the point that human factors is largely unaccepted at present

in system design and development:

Nor do design engineers tend to solicit the

assistance of practitioners. 76% of respondents

agreed with this statement. Again, there are

individual variations, special individuals and

special circumstances but the armed neutrality between designer and practitioner seems the same

as it was when it was described in 1967 (Meister

and Farr). A key element in securing designer

cooperation appears from respondents' comments

to be supportive management.. A number of factors

appear to explain the designer-practitioner relationship: the designer's wish to function

with complete autonomy; his view of HF requirements as more constraints he must put up with;

the HF group's reputation. It is helpful if the

HF group has sign-off on man-machine interface

drawings, but few groups have this sign-off.

Slightly more than half (57%) of practitioners

feel that there is still considerable resistance

on the part of designers to the inclusion of HF

inputs in design. The positive side is that

almost half (41%) do not agree with this notion.

It may be that these responses suggest that things

are improving somewhat, because in years past almost all practitioners would have given negative

answers. on this point. Some practitioners feel

that if behavioral inputs are reasonable, engineers will accept them. Unfortunately some HF inputs' are

inadequate and this creates resistance to or rather

avoidance of the inputs. Timing is all-important; inputs made after decisions have been reached by designers will be resisted.

This resistance may result in part from the fact

that engineers may find HF inputs to design insufficiently precise and quantitative. 72% of

the practitioners felt this to be the case. Some

pointed out that HF data must be, translated by practitioners into specific design terms or else

the input is-merely an additional burden to the

engineer. It is clear that there is continuing

2-27
and increasing pressure to justify the utility of

behavioral R&D. While this may not be unfortunate

in and of itself, it does lead to a number of

unfortunate results: faddism; impatience with studies

whose effects are slow to emerge; unwillingness to

invest research resources where results are risky.

A second point to be made concerning the lack of human

factors integration and application early in system design and

development can be drawn from a consideration of the military

test and evaluation program. In June 1978, the Office of the

Under Secretary of Defense held the 12th Training and Personnel

Technology Conference (TPTC) with the topic for review "Human

Factors in Weapon Systems Test and Evaluation (T&E).* The

minutes of this meeting are reported in Taylor (1978). Colonel

Henry L. Taylor, the Executive Secretary of the Conference, and

Dr. Jesse Orlansky, a consultant from the Institute for Defense

Analysis who provided written comments on the meeting, make some

interesting observations concerning test and evaluation and human

factors. In particular, the discussion centered on information

drawn from "The FY 79 Department of Defense Program for Research,

Development, and Acquisition," 1 February 1978, Chapter 9,, Test

and Evaluation. In the summiry of the minutes, Colonel Taylor

pointed out:

1. Sixty-one major programs will undergo T&E

in FY 1979, and DOD will monitor a total of

84 major weapon systems.

2. The budget request for T&E in FY 1979 is

$3,683 million for development, engineering

and testing, and $1,009 million for support, of ranges, test facilities, targets and joint tests, for a total of $4.7 billion for T&E in

FY 79.

3. The following areas are now being emphasized by the DOD T&E program:

* a. reduction of vulnerability of weapon

systems

2-28
b. reliability improvement of weapon systems

c. greater commonality and standardization of

weapon systems among military services and

with our European allies (e.g., HARM, STINGER,

TRITAC, JTIDS)

d. conduct operational,- test and development test

earlier in development cycle.

Joint Test and Evaluation (JT&Es) initiated in 1972

are used to evaluate the effectiveness of a weapon

system in its intended operational environment and

frequently uses the forces and systems from two or

more Services.

Dr. Orlansky, referring to the same topic, made the following

"Upassing comment":

. . . the magnitude of the T&E budget ($4.7 billion

in FY 79) does not, by itself, justify a larger or

smaller Human Factors budget. However, $4.7-billion

means that T&E is larger (more important?) than any

category of RDT&E (the largest is 6.4 Eng. Development $3.9 billion); larger than any Service RDT&E program (Air Force is $4.3 billion); larger than any RDT&E

authorization title (Naval Vessels is $4.7 billion);

etc., etc. The real thought is whether any nominal

Human Factors effort could produce more savings than

comparable efforts in other areas. Perhaps yes,.if

someone is willing to explore the possibilities.

With respect to the comments made by Colonel Taylor and

Dr. Orlansky, above, the point of concern is whethet or not early

integration of human factors in system development would preclude

some of the problems (and enormous costs) associated with later

test and evaluation.

Two final observations will be made concerning human factors

which are "too late and too little." First of all, even after

systems enter the operational forces, human factors problems still

exist which are reported as deficiencies and result in costly

2-29
redesign and retrofit. It is also imperative to note that these

deficiencies are not just gripes about sticky knobs, hard-to-read

displays, uncomfortable crew compartments, etc., that lead to

simple degraded performance in terms of time and error. Rather,

some of these deficiencies include loss of life and costly equipment. As an indication of the magnitude of these, deficienciesi

Exhibit 2-8 is from a 1978 briefing given by the Naval Air

Development Center on its human factors program. While these

deficiencies only relate to three weapon systems, the absolute

number of deficiencies is an indication of the magnitude of the

problem.

Exhibit 2-8'

Human Factors Deficiencies Report

for Several Navy Aircraft Weapon Systems*

300 - 2%4

LU

S114' 117

1I [CREW SIZE

F-14A P38/C S-M

AIRCRAFT

* This Isar from a brief provided by the Human Petor Engonrile Dwislon of th N". iir Ovekpmant Center,

2-30
The final problem observation is that in 1977 DOD announced

a tailoring process for specifications and standards in Directive

4120.2-1. As reported by Chaikin (1978), this directive; in brief,

permits: (1) selecting documents having potential application to

a specific procurement; (2) reviewing those potential documents to

satisfy only those clearly applicable to a contract; (3) imposing

only the minimum necessary requirements; and (4) examining the

surviving requirements to tailor or adjust the provisions so that

they support the particular system involved. As Chaikin further

observed, the same DOD Directive states "beneficial recommendations from prospective contractors 'shall be solicited to determine

whether additional cost-effective applications-and tailoring of

cited . . . standard . . . requirements can be accomplished or

cost-effective substitutions proposed."

This directive provides a loophole for avoiding human

factors; yet, it also provides a basis for insuring the -inclusion

of human factors if they can be established to be cost effective.

In summary, there appear to be several kinds of constraints

on the successful application of human factors to military systems

development. It should be re-stressed that there is responsibility on both sides--the human factors side and the program

management side--to achieve higher levels of cooperation. 'Most

crucial is the point that near-term costs (including some real,

conflicts between engineering criteria and human fa-tors criteria

in design work) can result in long-term gains that'are many

multiples of the near-term cost.

2ý-31
Are Human Factors Contributions Affordable?

We have tried to indicate by means of the preceding

discussion that there are at least four basic arguments for

the inclusion of human factors in military system development.

Each of these arguments has substantial historical roots so

that they are a part of a standard rationale. In brief, these

arguments are:

* Life cycle costing for personnel in a system can be

impressive. Personnel must be sustained physiologically

and psychologically. The cost of addressing personnel

performance through training, selection, technical

manuals, and other performance aids is expensive; and

the cost of personnel turnover is even more expensive.

* Complexity remains a growing problem that requires a

human factors contribution for its amelioration.

* Design deficiencies cause otherwise avoidable human

error--such errors are costly not only in terms of

lost lives and lost equipment but also in terms of

unfulfilled missions.

o It is possible to conceive and implement optimal

manned systems--systems that are designed to utilize

human capabilities-and minimize human limitations.

The response to these arguments should be:

o Human factors must lead to an increase in human

reliability and consequent increase in system

.reliability.

2-32
* Human factors must be applied as early as possible in

system development to achieve greatest cost-benefit.

The cost of redesign and retrofit of weapon systems

where human factors information has not been used may

far outweigh the costs for human factors earlier in

system design and development. However, this must be

viewed as a life cycle cost.

* Human factors value will derive not just from the

accommodation of human capabilities and limitations

(which would typically reduce performance time and

error), but also from better equipment utilization

because of improved attitudes. If user acceptance

is not considered in system design, the system may

be underused, misused, or not used at all--which

could result in mostly costs and no benefits.

* Effectively integrated human factors in systems will

reduce other costs such as those for training, selection,

and technical manuals.

9 Lack of human factors in systems will result in damage

to the equipitent and in hazards tO the user. In this

casei incorporating proper human factors results in

cost avoidance through avoidanre of damage to equipment

and harm to personnel.

• Human factors in system design and development will

contribute substantially to system maintainability.

In general, one may expect (1) time-to-maintain to go

down, (2) maintenance-induced failures to go down,

and, (3) spare parts consumption to go down.

2-33.
.* Human factors consideration based on the environment in

which weapon systems are to be operated and maintained

will insure that human performance is not seriously

degraded and, thus, that system performance will not be

seriously degraded.

* Human factors integrated into the entire weapons system

acquisition process and life cycle process has an

intrinsic value because it becomes part of the hardware

or system. Therefore, the effects of any human factors

stay with the equipment or military service and reduce

the cost of ownership. This is in contrast to training,

in which the' investment stays with the individual rather

than with the equipment.

Conclusion

It is important for policymakers and program managers to

realize that good human factors is not a case of just "proving

the obvious," i.e., that human factors is simply common sense.

In the 'past, a common sense approach has produced marginally

'acceptable system products (from a human factors point of

reference) based on the fact that the hardware and technology

associated with that hardware have' been around for some time.

Experience with it has produced a level of knowledge, one

might term "lessons learned"--which is really the common sense

to which we refer. In periods involving quantum leaps in

technology and hardware sophistication, as we have been experiencing for some time, this common sense breaks down due foremost

to the absence of "lessons learned" that comes with experience

with a technology. Human factors personnel have training and

experience to bring valuable knowledge and techniques to the

system development process. Human factors personnel have obtained

2-34
this knowledge due largely to dealing with gaps in technology

wherein common sense has broken down. In addition, operations,

analysis and research in fields such as system engineering,

aviation medicine, applied physiology, experimental psychology,

anthropometry, and sociology have contributed a great deal of

basic design data, which human factors personnel know where to

find•and how to interpret. Perhaps most important is the fact

that human factcrs personnel have the necessary motivation to

search for optimal solutions where man is involved.

2-35
References

Brown, H. Department of Defense annual report, fiscal year 1979.

Department of Defense, 1978.

Chaikin, G. Human engineering design criteria--The value of

obsolete standards and guides. Paper presented at the

Human Factors Society 22nd Annual Meeting, Redstone Arsenal,

AL, October 16-19, 1978.

Christensen, J.M. Ergonomics--Where have we been and where are

we going? Proceedings of the 6th Congress, the International

Ergonomics Association, University of Maryland, July 1976.

Coburn, R. Human engineering guide to ship system development:

Responsibilities, procedures, and methods for carrying out

human engineering programs (NELC/TD 278). San Diego, CA:

Naval Electronics Laboratory Center, October 1973.

Department of Defense. Specifications and Standards Application.

DOD Directive 4120.21, Washington, D.C., April 1977.

DePuy, W.E. The Army Training System Overview. Speech by Commanding General, Headquarters, TRADOC', 24 May 1977.

Flickinger, D.D., & Hetherington, A. Man--the Essential Factor

in Systems. Directorate of Human Factors, Headquarters,

Air Research and Development Command, Baltimore, MD,

December 1957.

Johnsoi, E.M., & Baker, J.D. Field testing: The delicate

c mpromise. Human Factors, 1974, 16(3), 203-214.

Meister, D. The influence of Government on human factors

rasear6h and development. Paper prepared for the U.S. Navy

P rsonnel Research and Development Center, San Diego, CA,

u da.ed.

Meister, D., & Rabideau, G.F. Human factors evaluation in system

deveZopment. New York: John Wiley & Sons, 1965;

A missile's ups and downs. Time, January 8, 1979.

Price, H.E. Job design as part of the Navy enlisted personnel

i dividualized career system. Paper presented at the Human

Factors Society Annual Meeting, Boston, MA, October 1979.

Rook, L.W. Motivation and human error (Sandia Corporation

e.chnical Memorandum No. SC-TM-65-135). Albuquerque, NM,

September 1965.

2-36

,,-.~.. - - .
Shapiro, A., et al. Human engineering testing and malfunction

data collection in weapon system test programs (WADD Technical Report 60-36). Dayton, OH: Wright Air Department Division, U.S. Air Force, February 1960.

Swain, A.D. Human factors in design of reliable systems. Paper presented at the Tenth National Symposium on Reliability and

Quality Control, Albuquerque, NM, January 7-9, 1964.

Taylor, H.L. Minutes of the Twelfth Training and Personnel

Technology Conference (TPTC): Human fac'ors in weapon

system test and evaluation. Office of the.Undersecretary of Defense, Washington, D.C., May 11, 1978.

"2-37
CHAPTER 3

IDENTIFICATION OF THE CONTRIBUTION OF

HUMAN FACTORS IN SYSTEM DEVELOPMENT

In order to identify the contributions of human factors in

system development it is necessary to understand both the major

system acquisition and development process, and the potential for

specific human factors contribution in each phase of this development process. Therefore, the first section of this chapter

summarizes: the major system acquisition process, requirement

at the DOD level for human factors R&D, and requirements at the

service level for human factors R&D. The material in this-section

may be familiar to some readers, and therefore is sufficient as

a refresher. However, for the less familiar reader, a separate

research note (ARI RN-80-23) is available which provides a review

of essential decision points, products, directives, and other

requirements that govern system development and enable human

factors R&D. This document may be obtained through DTIC.

The significant conclusion that emerges from this summary and

review is that there is an adequate and formal basis existing

for. integrating human factors R&D into military systems, but in

fact this is not being done.

The second section will delineate specific human factors

efforts in each system development phase and indicate how these

efforts contribute to the development of the principal human

factors product of that phase. The relationship between human

factors efforts/products and system development activities is

documented in the form of a graphic descriptive model.

The remaining five sections of this chapter describe in

detail the human factors efforts and system development activities,

the nature and content of the principal human factors product, and

an example of human factors contribution for each development

phase.

3-1
Prior to addressing these major topics of the chapter it

is necessary to reconfirm the notion of principal human factors

products in systems development. In Chapter 1 it was asserted

that there is indeed a principal human factors product that will

result from human factors efforts during each major phase of

system development.

Exhibit 3-1 delineates these products for each phase,

together with an indication of their potential effect on system

design. These products were essentially derived from several

studies or papers concerned with concepts or models for suggesting

what human factors efforts, decisions, and products should be

undertaken where in the system development phase. In general,

the principal human factors products identified in this report

represent a reasonable consensus of these other studies. Some

representative documents from which these products were developed

include Price (1962); Price, Smith, and Behan (1964); Erickson,

Miles', and Secrist (1978); Goclowski, King, and Ronco (1978); and

Baker, Johnson, Malone, and Malone (1979). The rationale and

precedent for these products is established in more detail in

Appendix A for the interested reader.

3-2
4 1 4) 4' 0 4 (A 41i `1'4 0 $4 01 9 41 '44 *-.4 4 U 4) .,q (a, .

W4 uI >4 00 >0 -.4

x o1'6. 0 -4 0 co io 4 V. 0 V0~ 4.44 a' V I 0 -4 c0-4 0' -H ý4 0 06 9: (D - 4 ja ý4 0 04 1 0 -. 41 o 01 

-A 4 O-& 4) .0 w 'o 0 6 N d '."' 4 1 401 0 to '01 '4.4 a6

(A v4 0H a6 W 4) r .4 to 0 0a-.

W )1 3, 34'1 &4 a') >4 6 -. 4 r.~I wn 4101)0 0i w (A n 4' AI in -H 41 00 I1 Ai 4~4 k : 0 @1 -4 (D ~ 14) 0 0 u 0

U) toag ii -A - -H~ 4' n~ e

_ l r- ah~ a r. to M j? 4 -

4'r. -= m u I

- ~-% -. - :5- -' -r " -,4

ex

0 41

oU

0~ 00-

0 r5x -4

0- 0(A .14 

1. 0 0 10u 

a #4 ra r-.4r

I'q 44 0 - 0 4)~ '440

JA 0 0 -W14

W ~ 4 !5 0 >

144 6j 0 04 0'44 14 W.

Z 0 jI 2.6-4 0 0%0 62 06' c-6 .

*4 0 (a 0 0004 - %

0' 06 IH 4 4 0

to 01 4.)r . .0 0 CC- aO 

0 >01 0a 03

to 4 0 8 4r o0 Aj u ~ W

IVin0

40 as

-~* '4

$4 -A aj 4' 4)4 U 0 0. 06 ( .1 -4 c0 -0 )

0 A rd0

(A : 0. N>

3-3
Human Factors R&D: The Existing Basis

in System Acquisition

This section of the chapter is a brief overview of the major

system acquisition/development process and the existing basis for

human factors R&D at the DOD level and at the service level. As

was mentioned previously, a more detailed analysis of this topic

is prcvided in a separate research note available through the DTIC.

The Major System Acquisition Process

OMB Circular No. A-109 (1976) establishes the guidelines and

policies for major governmental acquisitions. The circular

outlines the required sequence of activities through which the

proposed system must pass, and specifies the key decision points

at which the evolving system must gain approval before the

government will continue to fund a developing system or to

procure any new major system. DOD Directives 5000.1, 5000.2, and

5000.3 give the military services more detailed instructions

in implementing Circular A-109 for the acquisition of major

military systems. A general discussion of Circular No. A-109

will be followed by a short discussion of the directives.

OMB Circular No. A-109. The policies in Circular No. A-109

attempt to systematically integrate the various factors in system

development and to avoid past problems of cost overruns and

premature commitments to full-scale development ano-production.

To accomplish this, the circular outlines seven activities and

specifies four major decision points. Exhibit 3-2, adapted from

OFPP Pamphlet No. 1 (1976), shows these activities and decision

points. The boxes describe the types of activities involved and

the numbered circles indicate the major decision points. This

acquisition model requires an-identification of a need (Mission

3-4
Exhibit 3-2

The Activities and Decision Points of OMB's Circular

No. A-109 Major System Acquisition Cycle

EVALUATION AND

RECONCILIATION

OF NEEDS

IN CONTEXT

OF AGENCY MISSION.

RESOURCES AND

PRIORITIES

MISSION EXPLORATION

ANALYSIS OF ALTERNATIVE

SYSTEMS

DEPLOYMENT

AND 2

OPERATION

PRODUCTION DEMONSTATIONS

FULL SCALE

DEVELOPMENT,

TEST, AND

EVALUATION

3-5
Analysis), a comparison between the present technology status and

what is needed (Evaluation and Reconciliation of Needs), a

decision to continue or stop * , a study of the strengths and

weaknesses of alternative systems if the previous decision was to

continue (Exploration of Alternative Systems), a decision to

continue or stop , a demonstration of the c chosen system(s)

(Competitive Demonstration), a decision to stop or continue ;,J

the building and test of the complete system (Full-Scale Development), a decision to stop or continue ® , the production of the

system (Production), and field use of the system (Deployment and

Operation). The Department of Defense directives specify the

activities within the phases in more detail.

Department of Defense Directives. The Department of Defense

(DOD) Directives 5000.1, 5000.2, and 5000.3 give guidance in

implementing OMB Circular No. A-109 for military systems.

DOD Directive 5000.1 provides policy for acquisition of

major systems--those systems exceeding $75 million for research,

development, test and evaluation, or those systems exceeding

'$300 million for procurement. Directive 5000.2 supplements 5000.1

with policies and procedures for the DOD system acquisition

process. Directive 5000.3 gives guidance for military test

and evaluation. These three directives provide for five

events and describe the activities-in thephases between

those events. Those five events are identification of mission

needs, Milestone 0, Milestone 1,.Milestone 2, and Milestone 3.

Because the circulars, directives, military standards, and

service regulations used various names for the same phases, it

becama necessary to adopt standard phase and decision point names

to avoid 'confusibn when using information from the different

documents. In Exhibit 3-3, the major system acquisition cycle

3-6
ma'

pi moa

o 1W

.4~ -1 ;

IL U,12

41 a

-9-

I al 4
model of Circular A-109 is again shown. Below that model are

the standard names adopted for this report. Below the standard

names are other names co~mmonly used in various documents to refer

to the same phases.

The military major system acquisition modiel as described by

the directives is in Exhibit 3-4. The model spans. the time from

initial threat analysis to deployment of the system.-

Revised Department of Defense

5000 Series Directives

Revised Department of Defense 5000 series directives were

obtained just before production of the final technical. report.

These latest revisions do not affect the case that can be made

for human factors R&D requirements in the military acquisition

process. Rather than substantially changing the relationship of

these DOD documents to requirements for human factors R&D in

systems acquisition, they serve instead to effectively augment

this relationship. Selected excerpts with humnan factors R&D

implications are shown below to illustrate the characteristics

of the new directives.

* DODD 5000.1 Maj-orSystem Acquisition 19 March 1980

objectives'

*Integrate support, manpower, and related concerns into

the acquisition process.

Policy

Affordability. Affordability, a function of cost,

priority, and availability of fiscal and manpower

resources, shall be established and reviewed in the

context of the PPBS process . . ..

*3-8
us UU

44

- 0

-I- Uj

0 j

00

L; w

S Ua.

00

00

L -i J

U' A

> IL

W6 -

F3 9
DODD 5000.2 Major System Acquisition Procedures

19 March 1980

Design Considerations

Manpower and Training

(1) New systems shall be designed to minimize both

the numbers and the skill requirements of

people needed for operation and support,

consistent with system availability objectives.

Manpower and personnel factors, to include

numbers, occupations, and skill levels of

manpower rer'uired, shall be included as

considerations and constraints in system

design. Integration of manpower and personnel

considerations with the system shall start

with initial concept studies and shall be

refined'as the system progresses to form the

basis for crew station design, personnel

selection and training, training devices and

simulator design, and other planning related

to manpower and personnel.

(2) Where applicable, planning for training shall

consider provisions for unit conversion to

the fielded system and training of reserve

component personnel. Such planning shall

consider tradeoffs conducted among equipment

design, technical publications, formal

training, on-the-job training, unit training,

and training simulators and shall develop a

cost-effective plan for attaining and maintaining the personnel proficiency needed to

meet mission objectives.

(3) After Milestone 0, manpower requirements

shall be subjected to tradeoffs with system

characteristics and support concepts.'

Manpower goals and thresholds consistent

with projected activity levels, maintenance

demands, and support concepts shall be

identified by Milestone II. Tradeoffs for

maintenance effectiveness among manpower

(numbers, occupations, and skill levels),

support equipment, system design, and the

support structure shall'be conducted. The

manpower and training requirements to

support peacetime readiness objectives and

wartime employment shall be developed by

3-10
Milestone III. These requirements shall

be based upon considerations that include

available Operational Test and Evaluation

results and current field experiences with similar equipment.

Quality. A quality program shall be implemented

in accordance with the criteria and procedures set forth in DOD Directive 4155.1 (reference (j))

to ensure user satisfaction, mission and operational effectiveness, and conformance to

specified requirements.

DODD 5000.3 Test and Evaluation 26 December 1979

Policies and Responsibilities

Test and evaluation (T&E) shall begin as early

as possible and be conducted throughout thesystem acquisition process to assess and reduce

acquisition risks and to estimate the operational

effectiveness and operational suitability of the

system being developed. Meaningful critical

issues, test objectives,, and evaluation criteria

related to the satisfaction of mission needs shall

be established before tests begin.

Before-the Milestone III decision, adequate DT&E

shall be accomplished to ensure that engineering

is reasonably complete (including survivability/

vulnerability, compatibility, transportability,

interoperability, reliability, maintainability,

safety, human factors, and logistics supportability),

that all significant design problems have been

identified, and that solutions to these problems

are in hand.

Attachment I -,page 2 Definitions

Operational Suitability. The degree to which a

system can be satisfactorily placed in field use,

with consideration being given availability,

compatibility, transportability, interoperability,

reliability, warti.me usage rates, maintainability,

safety, human factors, manpower supportability,

logistic supportability, and training requirements.

3-11
As these quotes aptly illustrate, the requirement for human

factors has been reinforced with more direct and to-the-point

statements regarding the role of human factors in system

development.

With reference to DOD-level requirements for human factors

R&D, it should be stated that DOD maintains one human factors

specification (MIL-N-46855) and one human factors standard

(MIL-STD-1472), each containing extremely detailed human factors

requirementk

Formal and Ii ormal Requirements for

Human Factors R&D at the Service Level

Each service within the Department of Defense maintains its

own implementation procedures for human factors requirements.

These requirements, found in various service documents, may be

broken up into two categories. Human factors efforts stated as

requirements can be designated formal documents. Formal documents

consist of the service regulations and instructions (i.e., Army

Regulation AR 602-1; Air Force Regulation AFR 800-15; and

Department of the Navy instruction NAVMATINST 3900.9). On the

other hand, human factors-efforts described as recommendations

can be termed informal documents. Informal documents consist of.

the various service guidebooks, handbooks, manuals, etc., that

have been developed.

The following points. can be made about these service-level

documents.

* The trend in formal documents is to define requirements

and responsibilities for human factors without placing

constraints upon, the methodology, analysis, and data

characteristics used in research.

3-12
*The informal documents cover primarily those topics

not promulgated in the form~al requirements.

*The consistent thread running 'throughout the entire

service doctrine is the application of human factors

as a total system concept, encompassing earlier and

more detailed involvement in major military system

developments.

Greater detail and technical explanations of these formal and

informal service-level documents may be found in the aforementioned

research note. The following sections will serve to illuminate

specific human factors R&D efforts, for each system development

phase, which are considered ideal in any military system

.development.

Human Factors R&D: Specific Efforts

for Each System Development Phase

There is an ever-incr~easing disparity between the complexity,

and sophistication of modern military weapon systems and the,

capabilities of the military personnel.' The complexity and

sophistication of modern military hardware is increasing at

arate greater than ever before, while the capability of our

military personnel to operate and maintain these systems is,

by even the most optiristic accounts, just barely keepinc pace.

This hardware-personnel mismatch places a greater-than-e er

burden on human factors to contribute a favorable impact on

capability, cost, and compatibility.

Human factors provides information that not only affects

equipment design, but also helps determine the design of personnel selection, training, and organizational structures that

will make equipment more cost-effective within an operational

system. Above all, human factors is; concerned with the mission

3~-13
objectives of an equipment system, and with actions that will

assure the ability of people and machines to meet those missioi,

objectives.

Human factors should begin by analyzing mission scenarios

that are expected to be encountered in combat. Analysis is

performed to identify the critical roles men will play to succeed

with any particular mission. Those roles will be broken down

into functions, which can then be simulated or performed under

the expected operational conditions, so as to evaluate equipment

designs, determine manpower requirements, forecast training

requirements, and detect the organizationalstructures that will

be required to support the equipment. Alternatively, prototype

equipment and organizations can be tested empirically by field

trials, in which the intended user population attempts to perform

the required mission scenarios. Such tests have recently been

performed for several pieces of equipment during the operational

tests (OT-I, -II, or -III) required by the Life Cycle System

Management Model.ý They have provided valuable data about equipment and organizational design, in time to (1) preclude defects

that would have degraded missior performance, and (2) prevent the

need for expensive modifications once the equipment was fielded..

Finally, human factors identifies support requirements.

One of its important tasks is to assure that such needs as

maintenance equipment, training programs, and training devices

are ascertained soon enough, so that when the equipment is

delivered it can be fielded promptly as part of an integrated

man-machine system.

Exhibit 3-5 shows the major activities of the system

acquisition cycle on a time line from Mission Analysis Phase

to Deployment. Concurrent with the system acquisition time

3-14'
Exhibit 3-5

Specific Human Factors Efforts and Products for Each Military System Development Phase

INIVLDAUAY US& FA TRA V=[

AMS POLTICAL INFO. EWASLI

WIuMo Corpmw w

OWPING

#U'UCONITUIAONSUI

gAflMI[I comornomi NIMY)" GATIVIIAWECIS.

(MPE) ONTIPIED MWNIFPIGO OF SIMUAN 3VVSTE

aoiMTIP110

ANAcLow I II

IPLWU

ROLE OF

MAN DETERMINED

B ca------MILESTONdE 0

3-15.
Exhibit 3.5 (Continued)

Specific Human Factors Efforts and Products for Each Military System Development Phase

ROLE OF MAN DEFINED

MENS -MILE -MI-E STONE 0

APPLY CRITERIA IN THEIR ORDER

PROGRAM PRASSIGN 1 - ,, OF FUNCTION IMPORTANCE FOR EACH

MANAGER

"ORDER OF IMPORTANCE (OR CRITICAL) CRITERIA

* .. DEFINED FOR EACH FUNCTION

COMPARE HUMAN CAPASIUTIES

TO FUNCTION CRITERIA

STUDY

ALTERNATIVES

POSSIBLE MANNED

FUNCTIONS DETERMINED

DECISION COOROINATING EXPLORE POSSIBL

PAPE.R t'D(::M %"EPOEP•IE MANMACHINE COMBINATION$ (.,TO ACHIEVE FUNCTION

ENOPOINTr

(O) SARC I

' " ~ ~RIEcOMIIeNDATION"•o ,

DSI

ECONMIDR FUNCTIONS ALLOCATED TO MAN

I CANCEL- -------------- ... t. MLEONE I

APPROVE

3.... r16' |

Jr ' . ' . ... .,' 3-'16 , ' , .,
Exhibit 3-5 (Continued)

Specific Human Factors Efforts and Products for Each Military System Development Phase FUNCTIONS ALLOCATED

-------- MILESTONE 1

TIT& * TASK OPERATIONAL MDJITENANCE EVALUATION PEOUIREMFkT$ SEQUINCE REQUIREMENTS

MASTER PLAN ANALYSIS ANALYSIS ANALYSIS

UVALATE STA.N REQUIREMENTS hbOUINfN DEFINED LUA A iDTNIF PIED DIENTIFIED

CONSIDER

SUBIMIT MAINTENANCE

SIfETATION

PROPOTSALS 1"IOSOHIE

CONRAECTIOR CONCEPTCSENTANEC

CONDTUCT 002TSLCE POTOTYPE I

CONCEPTSONRO

FOR TESTING CONCIPT SELECTED

CONOUCT OT IJOTI CONSOLE CONCEPTS

€o¢ATERNACTVE

• "RCI- DoErIGN , 

! Sft I°

S', , RSCOMMI CATIONS

TIM TASK ANALYSIS AND

HUMAN INGOINI NER IQUIREIRMINTS

IGC00CTI -HP T&I

TASK AAiLYSIS t.mO MiUMAN

ENGINEERING REOUtflEMENTS VALIDATED -- ......--------------- W16 MILETOII 2

3-17
Exhibit 3-5 (Continued)

Specific Human Factors Efforts and Products for Each Military System Development Phase TAMK AI4ALVI1Z

A"WHEREOINEW&MY

--------- --------- M ILESTONE 2

WWome" He DOWN

PR0T0TWU EVALLUAMAS

PW TOTY.P...........................................OIAU A~O *OR TuOuGWiMPACE DGUSON

CONDUCT

@VALUATIDIW

............................ ...........

WOATI

L~A IIIO

DETAILED MAKMACHINE,

110 00 INTERFACE DESIGN VALIDATED

AODI NOWL-~--gLSO E

* . 3!3-18
S0- '

w D

0 *L

""*-3I.

22

-m - - - - --.

ow .* U19
line are the human factors efforts in each phase and their

expected outputs. The principal human factors product of each

phase is indicated by the bold type. Together, the series of

charts making up Exhibit 3-5 can'be considered a descriptive

model of human factors efforts/products in system development.

The material to follow discusses, for each phase of system

development, how human factors should be utilized as a system

progresses through each phase. The human factors efforts within

each phase will be described, the content of the principal human

factors product of each phase delineated, and examples presented.

The objectives of human factors R&D are to ensure that military

equipment and organizations are well-fitted to human users and

produce the maximum advantage in 'terms of the military's mission.

This model is a generalized one, synthesized from many

sources;* it is non- service-specific, and applicable to a wide

variety of systems. However, the model cannot and does not

represent a rigid process 'of development for all systems. As

with any model, there are qualifiers. While the order of humanfactors considerations should be maintained for most developing

systems, there most likely would be variations in which phase the

specific human factors considerations would be addressed.

Depending upon the complexity, operational environment, crew

role and size, number of manufacturers, etc. involved in developing

the system, the resolution of specific human factors efforts may

shift from one phase to another. In other cases, the developing

*Principal reference sources for development of the model,

depicted in Exhibit 3-5 were: Bakei, Johnson, Malone, & Malone,

1979; Coburn, 1973; Collins, McGuiness, Erl•:hman,,& Bryce,

1975; Geddie, 1979; Goclowski, King, Ronco, & Askren, 1978;

Kaplan & Crooks, 1980; Meister & Rabideau, 1965; Merriman, 19764>

MIL-H-46P55B; Price, Smith, & Behan, 1964; Price & Tabachnick,

1968; Van Cott & Kinkade. 1972.
process may be compressed or specific efforts not needed, both of

which could result in entire phases being eliminated. In reality,

many of the human factors efforts are iterative, and the complex

feedback and feedforward loops have been deleted in favor of a

perceptually and conceptually uncluttered model.

Mission Analysis Phase Human Factors

When a threat or technological breakthrough has been

identified and a decision made to propose a new system, the

various system developers should begin a coordinated effort to

clearly state the objectives and define the criteria of the

system. These would not be statements on how to accomplish the

mission, but rather on what is to be accomplished. These first

activities are extremely important, as the criteria will become

the standards for equent design and for test and evaluation.

Once the objectives :ave been identified and the criteria defined,

those other political, economic, and time constraints that bound

the system design must be identified.

The human factors output of this phase is the determinaticn

of the role that men will play in the new system. Will man be an

operator, maintainer, sensor, manager, •analyzer, decisionmaker,

information manager, back-up to equipment, or some mix of the

above? A very important decision is whether man will be local

or remote from the mission equipment. To define that role, all

functions that are needed to achJieve the mission objectives must

be specified first. To identify all functions, the operational

and environmental conditions under which the system i4 to operate

must be determined. For example, will the system operate in

temperature extremes, during day and night, in unusually rugged

conditions, or for unusually sustaired periods of time, etc.?

In addition, analyses of existing similar systems (if any) should

.3-21
help identify operational and environmental conditions as well as

other positive and negative aspects. For example, what was the

role of man in the predecessor or similar system(s)? What man

functions and man-machine functions have been successful and

unsuccessful? All of the information can be used in perforrxing a

functional analysis.

Performing trpdeoff studies with the major tactors

(e.g., logistics, maintenance, costs, advantages anC. disadvantages

of using man in alternative roles) should tesult int cost-effective

system configurations, given system constraints. Such human

factors analyses also lower the probability of major changes in

design downstream to accoxooda-e the idicsyncracies of man.

Human Factors Efforts and System

Development Activities During

Mission Analysis

This subsection provedes a description of the human factors

efforts and system deve^:pi.ent acti,'ities shown in Exhibit 3-6.

The descriptions are keyed' to the numeric co-le on the chart.

The initiation :rocess for s'.'e" ý-'Muisition is based,

first and foremost. upon the r,:?c:xon of a need for a system

to fulfill a to--be-specifieQ' r-.on. This is driven from one of

two sources:

(0.1) Threat AnaZyeie. A systematic means to assess enemy

capability in relation to one's own capability to wage war.

Formally defined, it is stated thus:

The process employs analytic techniques for developing plausible alternative representations of foreign environnments and capabilities. Threat analysis--

1.. Provides an assessment of foreign capabilities

in terms 'of combat material, employment doctrine,

environment and force structure.

_____ 3-22
Exhibit 3-6

Specific Human Factors Efforts During the Mission Analysis Phase (0.)

inLD IMLTAY 1OAAAVS

AN POLITCAI 0"P. tA -- ~ ~ 1

0 - V. 

01*MAO gAgLm

L~

ROEO

r"Ir MAN DETEMINa A 

0" - -- 

a~l) 1ONI06 19P3-23"LOSV W
2. Provides an assessment of the level of development which the economy, the technology, and

the military forces of a country have attained

or could attain.

3. Includes recasting existing intelligence

assessments and forecasts to provide statements

of the threat as it relates to a specific U.S.

research or combat development project.

(AR 3,81-1l)

(0.2) Identification of TechnoZogicaZ Breakthrough. In an

attempt to take advantage of the possibilities offered by newly

or nearly available technologies an effort must be initiated.

The following quote. tempers the application of new technology.

These technical and'scientific advances must be

evaluated within the framework of the military.

system developments to insure the proposed

development is relevant to the DOD's needs.

(Adapted from DA Pam 11-25)

(0.3) Misaion EVement Needs Statement (MENS). The MENS is

a DOD-level required product of the Mission Analysis Phase and is

aptly summarized by the following:

The MENS pro-i~des justification for initiation of

further system acquisitioning, programming, addresses

mission related deficiencies (e.g., cap&bilities

change due to: change in enemy threat system

obsolesences, new technology availability), provides guidance for system concepts and identifies

constraints.

(Adapted from SECNAVINST 5000.1A)

Requisite human factors inputs to the Missibn Analysis Phase

should be found in the MENS, which is the subject of the preceding

discussion. The discussion to follow is thesubstance of human

factors inputs, and culminates in the major human factors product

in the Mission Analysis Phase--the role of man in systems.

3-24
(0.31) Define System Objectives. Human factors inputs must

be general in nature by definition. Thus a general but valid

statement of the mission(s) and threat must be developed

(e.g., potential targets, enemy weapon capabilities).

(Adapted from MIL-STD-490)

The specification of mission requirements,

objectives and criteria include the determination

of the class system and hardware involved and a

statement of the activities assigned to a system

envisioned for a specific mission(s).

(Adapted from Kaplan & Crooks, 1980)

Once having defined system criteria, additional descriptions

must be prepared that provide a refinement of mission details.

These include determination of mission constraints and limitations

that impact the feasibility of continued system development.

(0.32) Determine Operational Conditions. This includes the

specification of two items: (1) system operation and (2) conditions of performance. Both are important to mission feasibility.

System operating characteristics and goals must be identified to determine a probability for mission success. Personnel,,

crew, and hardware characteristics during-performance of a

mission must be, defined to illustrate mission constraints that

impact mission success.

(0.33) Determine Environmental Conditions., Known system

characteristics, operating locus, and enemy countermeasures must

be identified to reveal limitations upon various approaches to

a mission. This includes a determination of terrain and climate/

weather conditions.

3-25

S, .f I
(0.34) Analyze Similar Systems. Since most new systems

rarely make use of technology never before experienced in any

total sense, the analysis should encompass previously encountered

strengths and weaknesses as well as lessons learned to avoid'

previous development pitfalls.

A process of synthesis must be utilized to incorporate the

findings of the various analyses previously conducted in order to

determine the overall feasibility of a system development program.

In addition tO system characteristics, this process should lead

ultimately to a defined role for man--the master product of human

factors in mission analysis (and the sole human factors reason

for participation of specialists at this stage).

(0.35) 'Conduct Function Analysis. Function analysis,

originally conceived as a process in system engineering to'select

functional categories for system performance (involving the techniques of functional flow diagramming and block diagramming), has

been extended to include the functions of man in the system,

withoui whom no system performance would be possible. This

analysis involves the se ection of manual, hardware, or automated

performance for each fun tion. The analysis stops short of

allocating specific functions to man or machine, but terminates

with data tantamount to uch a distinction. For human factors,

the interest was exclusi ely in assessing the capabilities of

humans in any specific s stem.

(Portions adapted from MNL-STD-490)

(0.36) Role of Man Defined. The culmination of the preceding activities is the role of man in systems (from a human

factors standpoint). Sirca this is a major product of mission

analysis, its principal omponents'have already been indicated

and will not be repeated again.

3-26
Content of the Role of Man Statement

A statement of the role of man as part ot the Mission

Element Needs Statement (MENS) should include the following

consideratior.s

Assumptions:

* A separate "role of man" analysis will be provided for

each alternative system concept selected.

e Human engineers will develop "role of man" concepts and

interact with mission analysis team in development of

MENS.

e "Role of man" components are listed according to probable

order of presentation in MENS (not according to their

development sequence).

Actions:

1. List effects envisioned for overall system as a result of

role of man devised for each alternative system concept

as configured (e.g., operability, maintainability, mission

effectiveness).

2. List effects envisioned for man's-role/personnel

subsystem as a consequence of each alternative system

concept as proposed (e.g., safety, 1habitability, user

acceptance).

.3. Determine location of man in system to perform

designated role.

"4. Specify advantages accorded man's role for each alternative concept (e.g., facilitate operation of system,

Sallowance for contingencies).

.3-27
5. Specify disadvantages accorded man's role for each.

alternative concept (e.g., manpower reserves consumption,

level of training requirements).

6. Determine required human performance, behaviors,

capabilities, and performance limits (e.g., sensing,

processing, information storage, decision making,

responding) identified for each functional category.

7. Determine personnel constraints impacting man's role for

each alternative system concept such as the following:

a. maximum and minimum numbers of personnel who can be

used in the system

b. types of personnel (e.g., skill level and aptitude)

available for system assignment

c. anthropometry of identified personnel population

(existing and projected)

d. user acceptance problems projected and their effects

e. effects of system and mission as configured on

personnel vulnerability (e.g., environmental hazards)

f. communication requirements and limits (system and

other personnel).

F Det ermine implications envisioned for each alternative

!ygtem concept upon requirements for:

a. training (e.g., level of training, trainability,

training support and facilities, training devices).

3-28
b. manpower (e.g., manpower levels, performance

availability)

c. life support

d. "-ilities" support (e.g., logistics, reliability,

maintainability)

e. social/organizational impact (e.g., MX basing).

9. Select contributions to function analysis in Mission

Analysis Phase:

a. identification of threat

b. need demonstration: new system or modification to

current system

c. requirement

d. mission

e. system objective definition (and required input/

output)

f. mission segment

g. scenario(s)

h. functional categories

i. functional flow and operational event sequences

j. system specification:

1, manual'

2. hardwired

3. automated: Facilitate system functioning

Override (bypass) system malfunctioning

Control systemgraceful degradation

Permit system, to operate.

3-29
10. List human factors characteristics that will facilitate

successful system development and mission success for

each alternative concept (design, development, testing,

production, deployment, and operation):

a. advancement in state-of-the-art human factors

technology

b. currently available human factors technology.

11. List impacts upon cost and system effectiveness for each

alternative concept in association with human factors

inputs:

a. R&D, training, personnel, manpower

b. mission success, vulnerability, survivability.

12. Prepare Human Factors R&D Program Plan tailored to each

alternative concept for balance of system life cycle.

Mission Analysis Phase: Example

of Human Factors Contributions

Both doctrine and recent history suggest that future

warfare will require a capability to fight at night, and during

operations sustained around the clock. The Soviets claim an

ability to fight at night. Aside from that threat, the importance

of night combat is obvious. Modern battle will move rapidly;

the unit that can move and fight at night may gain a permanent.

advantage. Furthermore,, weapons are so lethal that'maneuvering

at night may have great advantages in security. But unfortunately,

little is known about the capabilities and limitations of soldiers

during the night or in continuous operation. We lack knowledge

of the mission scenarios that would be involved and therefore of

how to train for battle at night. There is a requirement for

research on which to base appropriate training; the development

of tactical doctrine, and plans for unit rotation, performance

aids, and training devices.

3-i-30
An example is provided by the work of the 9th Infantry

Division at Fort Lewis, Washington, which was concerned with

improving the ability of individual soldiers to navigate at

night.

Army researchers began by analyzing the strategic goals

of night operations and continuing operations. From these goals

they identified specific tactical missions, analyzed the situations that would occur, and then determined the human behavioral

requirements for mission success. This produced a statement of

the behavioral requirements for mechanized infantry squads and

platoons in a combined arms defense against a deliberate breakthrough attack. A methodology was developed in which a number of

specific mission scenarios can be varied, so that the soldier's

tasks occur under a variety of conditions (differing levels of

light, fatigue, stress, out-of-phase daily rhythm). Tasks

critical to the enemy's defeat were evaluated ir terms of what

behaviors they required compared with what soldiers were actually

aole to perform. Areas in which soldiers fell short were identified and recommendations were made concerning changes to tactics,

equipment, and training.

Army researchers studied several basic issues, including the

ability of soldiers who are moving in vehicles to maintain their

orientation. It was found that even infantrymen on foot had a

poor sense of direction at night, and lost their navigational

reference. Those in personnel carriers, and the crews of armored

vehicles, were substantially poorer in this regard. This fact is

of special interest because current vehicles do not carry navigational aids. Study of individual differences found no way to

select people based on their backgrounds, but suggested that

actual experience in land navigation did improve both night and

day navigation. Further research is how in progress.

3-31
Meanwhile, light-attenuating devices (LADs) were identified

as a possible tool for training in nighttime operations.

Ordinary darkened lenses posed a hazard: They attenvate light

well only in the visible range, and transmit harmful infrared and

ultraviolet light. Since the pupil of the eye responds only to

visible light, the eyes of the subjects wearing ordinary goggles

would be dilated, making them particularly vulherable to damage.

By using lenses with an appropriate degree of attenuation, it is

possible to closely approximate visual conditions under different

degrees of moonlight or darkness. The Army evaluated the goggles

at Fort Rucker in flying training and at Fort Lewis in land

navigation. Test results suggested a satisfactory behavioral

approximation of darkness, and research continues concerning

their suitability for teaching various nighttime tasks.

The use of LADs has been demonstrated fcr the training of

tank drivers at Fort Knox, for infantry navigation at Fort Lewis,

and for nap-of-the-earth night helicopter flying at Fort Rucker.

Training of this kind, conducted by day, has great advantages in

safety. More training can be accomplished for the time invested

because supervisors and trainers can lirect the activity using

normal vision, and because logistics is not impeded by darkness.

The Basic Combat Training Group at Fort Jackson has implemented night rifle training using the LAD. Aside from the

general difficulty in seeing, soldiers tend to overestimate range

at night--an error which training can correct.

This research has provided a limited capability to train

infantrymen, truck drivers, and helicopter pilots in night operations. It provides an initial scientific understanding of

soldier performance at night, and contributes to building a human

technology data base to help develop methods and machinery to

counter a night fight threat and maximize our night fight

capabilites.

•.- 3-32
Concep4t Development Pbase Human Factors

In the Mission Analysis Phase, the system functions were

identified. In the Concept Development Phase, criteria for each

function are derived and rank-ordered or weighted in importance.

For example, in a particular function one must determine the

relative importance of accuracy, flexibility, and firing rate.

In one system a fast firing rate is most important; widle in

another accuracy may be ranked first.

After rank-ordering or weighing the criteria, the critetia

are compared to human capabilities. The comparison process

should suggest which functions should be allocated entirely to

machines, others entirely to man, and still others to some manmachine combination. Those man-machine functions should be

further studied to determine which combinations would produce

the most effective performance. It is equally poor to have a

man do a job at which a machine is better as it is to have a

machine do a job at which a man is better. The intent of the

man, machine, and man-machine function, allocation is to have the

most effective participation of man, given the system constraints.

Human Factors Efforts and System

Development Activities During

Concept Development

This subsection provides a description of the human factors

efforts and system development activities shown in Exhibit 3-7.

The descriptions are keyed to the numeric code on the chart.

The major products of the Mission Analysis Phase, which

includes the Role of.Man in Systems, are carried forward as input

to the Concept Development'Phase, wherein the initial allocation

of functions between men and machines occurs. Steps in system

development include the following in concept development.

3-33
Exhibit 3-7

Specific Human Factors Efforts During the Concept Dccvlopmcent Phase (0.)

ROLE OF MAN DEFINED

M-EN- ----- -MILESTONE 0

t 1.1 *. APPLY CRITERIA IN THEIR ORDER

ASSGN ASSIGOF FUNCTION IMPVTANCE FOR EACH

PIlOGRAM ,.

MANAGER•"-

RORER OF IMPORTANCE (OR CRITICAL) CRITERIA

.D OEFINED FOR EACH FUNCTION

, CIMPAAE HtUMAN CAPASILITIES

¶) To FUNCTION CRITERIA

STUDY

ALTERATIVIES

POSSIBLE MANNED "FUNCTIONS DETERMINED

1AE10ATtEXPLORE )POSSWILE MAN.

PABEfl 10C1.31 M. ACHINE COMBINATIOINS "CTO AHIEVE FUNCTION S"'.~F.NOPO)INT"

I W ARC I

RIIIcOMMENODATI

RWoNSIoeR FUNCTIONS ALLOCATED TO MAN

A CAt- . -- -- -MILESTONE .1

3-34
(1.21) Assign Program Mc!nager. Inherent With the designdtion of a system developm,':nt program manager is a cluster of

activities involved wits- mTrana, tiL of the develop;-ent life cycle

(e.g., Develcpment Plan, Decision Coordinating Paper, !-tc.).

(1.2) Stud:j AZternatives. This general title includes

the initiation of development of, and culminates with the verification of, the conceptual system(s). It involves the continued

refinement cf Mission Analysis Phase products, validation of the

same, and development, study, and approval of alternative system

concepts which are summarized in a Decision Coordinating Paper

(DCP).

The principal human factors activities in the Concept

Development Phase are directed toward the allocation of functions

to man and machine. The activities involved in arriving at this

product are the subject of, the following discussion.

(1.22) 'AppZ Criteria in Their Order of Importance for

Each Function. This is a two-part activity. It involves

(1) development of appropriate tradeoff criteria and (2) application of the tradeoff criteria to the allocation of functions

betweeh men and machines.

Examples of candidate criteria include:

e cost (procurement and operation)

. weight

d development time

* davelopment risk

* safety

e- maintainability

* system effectiveness prediction

e physical volume, size limits

* survivability.

3-35
And for hunan factors specifically:

"* human performance capabilities 'and limitations

"* machine performance capabilities and limitations

"* special case effects of automation upon system capability.

Application of criteria can involve manual or computer-aided

models, but must demonstrate:

* effects of the system upon human performance

9 effects of human performance upon system effectiveness

9 rationale of the decisions based upon criticality' of

alternative versions.

(Adapted from Coburn, 1973)

(1.22) Comparu Human Capabilities to Function Criteria.

After obtaining a preliminary allocation of functions to men

and machines, an assessment must be made of human capability to

perform effectively each function designated to man. This may

involve lessons learned and other data obtainable through analysis

of previous similar systems or through human performance reliability

simulation by means of a number of currently available models.

Whatever method is utilized, the result must be a rank order of

preferred candidate manned functions along with a rationale for

their choice.

(1.23) Explore Possible Man-Machine Combinations to Achieve

Function Endpoint. The final activity in the analytic process of

function allocation requires' the analysis of candidate allocation

versions (or man-machine combinations), utilizing models 'such as

those described above, but with the intent of assessing more

comprehensive performances (such as workload characteristics)

rather than individual functions. The method utilized should

provide a decision and associated rationale for human factors

choice of man-machine combinations.

3-36
(1.24) Function Allocation. The principal product of the

Concept Development Phase is the allocation of function to man

and machine. Thi& product is described in great detail elsewhere

and will not be repeated here.

(1.3) Decision Coordinating Paper (DCP). DCPs are documents that support, authorize, and promulgate decisions to

initiate development programs and establish appropriate Advanced/

Engineering Development Line items (OPNAVINST 5000.42A). They

present the rationale for starting, continuing, reorienting, or

stopping a major development program. DCPs address affordability

of a proposed system as well as other important factors

(e.g., threat, risks, acquisition cost, strategy, and performance

parameters for evaluation).

(Adapted from DA Pam 11-25)

(1.4) DSARC I/(S)SARC I. Defense System Acquisition

Review Council I/(Service) System Acquisition Review Council I

provide recommendations as to the status and readiness of each

major system under development to advance to subsequent phases

in its life cycle. They review such documents as the DCP in this

process. Fi al decisions are made by the Secretary of Defense or

his designee.

Content of the Allocation of Functions

to Man Statement as Part of the

Decision Coo.dinatinq Paper

A statement of the allocation of functions to man as part

of the DCP s ould include the following considerations:

Assumptions:

e The ollowing items will provide direct input to the

spec fication of the function allocation picess:

3-37
- Mission Element Needs Statement OMENS)

- mission scenarios

- functional flow block diagrams

- mission time lines.

o Function allocation will provide support to the proposed

system by illuminating the following criteria:

- system performance

- cost-effectiveness.

Both criteria have as a function human performance.

Human performance can be specified according to degree

of detail available about the system mission and environmental factors.

* Function allocation will detail functions involving both

operators and maintainers.

* The following general process is assumed, for the function

allocation process:

- identify and allocate tasks and functions to be

assigned to all personnel

"- identify required equipment

- evaluate selected man-machine combinations

- arrange tasks and functions to maximize mission

effectiveness and reliability.

Actions:

e This section is arranged according to a topical development sequence for function allocation (not development

sequence).

3-38
1. Specify human factors criteria selected for allocation

of functions (e.g., response time, error rate or human.

performance reliability, cost).

2. Specify other criteria selected for allocation of

functions (e.g., cost, personnel cost, required training,

weight, development time, development risk, safety,

maintainability, system effectiveness, physical volume

and size limits, and survivability).

3. List allocation of each function to:

a. one or more operators/maintainers

b. machine only (includes automation)

c. combination of man and machine

d. function currently not amenable to man or machine

performance.

4. Multiple operator/maintainer and man/machine functions

will include specification of the type of redundancy in

the task being proposed (e.g., parallel or sequential

mode, or hybrid of both).

5. Provide estimate of feasibility of performance for each

function allocated4 List the effect of different allocation versions upon mission success (e.g., probability).

Provide estimate of workload upon operators/maintainers

t as a result of each allocation version Cat least,

nominally). (At this level of development, workload

implies task difficulty and will include requirements

for: precision, concentration, criticality, mission

priority, and task continuity for operators/maintainers

involved in each manned function:) Account for effects

of user acceptance for each allocation version.

3-39
6. List human performance capabilities required of

operators/maintainers for each function involving man

and verify whether or not man can perform each in terms

of required physical and mental parameters over the

required time period and within the anticipated

environment.

7. Prepare rank orders for ca*ididate allocation Combinations according to criticality of functions. (Criteria'

for criticality will also be specified.)

8. List all bottlenec]k3, data overloads, acceptance

.problems, and other mission-critical faults that occur

as a consequence of each allocation version. Specify

the means by which each allocaticn version will relieve

them and/or how to modify the allocation version to

accommodate them.

9. Prepare a comparison matrix which exhibits all allocation versions versus the selection criteria (entries in

the matrix are estimates of absolute performance 'or rank

for each allocation version or each criterion measure).

10. List preferred manned functions as well as other

combinations or allocated, versions.

11. Provide a rationale' for th? preferred approach and

selection to justify the allocation.

Concept Development Phase:

Example of Human Factors'

Contributions

Navy fliers are faced With too many tasks requiring

the use of eyes and hands. Excezsive workload can have

serious adverse effects on mission effectiveness and safety.

3-40
The workload must be reduced if the performance potential. being

designed into new systems (and which the operator is now too busy

to fully utilize) is to be attained. New designs for airborne

systems reduce crewstation crowding but do nothing to reduce

workload problems because they still continue to rely solely on

visual and motor task performance by the operator. The research

discussed here is an effort aimed at achieving a technological

breakthrough.

A technological breakthrough is being achieved by developing

an alternative means of communicating with the aircraft system by

the spoken word.

In general, voice systems allow the operator to input data

or ask questions about the status of the system using conventional

speech, and to receive verbal status advisories or warnings as

well. This capability for full "interaction" by voice is called

a voice-interactive system (see figure).

PRMnMaS APMUCAIIWM

0 v cm OMM -. *TVimMme

SOfL' O VOMS

PO3-4U1gWA

peosucis M11OM S Ml

3-41
Voice systems can reduce workload and enhance the productivity of the operator in a var:iety of system applications.

Laboratory studies and operator estimates indicate that data can

be entered into onboard computers two to three times faster by

voice than by manual keyboard when tne operator is performing a

control (hand) or visual task at the same time. A voice system

fully integrated with other weapon subsystems can reduce time to

detect and respond to an emergency by 30% to 50%, depending on

the operator's involvement with other tasks. Such time savings

dŽuring c:ritical mission segments could yield dramatic returns in

improved mission performance.

Demonstration/Validation Phase Human Factors

In this phase, task analysis and operational sequence

analyses 3re conducted to determine what tasks are performed

in what .-. :dar to accomplish earth manned function. These analyses

specify• what information needs to be present and what types

of responses are Pacessaiy for each task. These analyses also

specify what skills and knowledge are required to perform the

task. Tha ragalts of the analyses are used to develop station

arrangement concepts.. The stations can represent one function of

the system or ai group of similar tasks from all the functions.

The work space is then. developed from station arrangements.,

There are a nuz'ber of techniques available to maximize the

efficiency of the work space and decrease operating errors.

The console concept is developed from the results of the

work space analysis, the information and response requirements,

and task clustering. The control-display analyses deal with how

the required information is to be presented and what types of

controls are best for the responses.

3-42
The human factors product of this phase specifies what

kinds and quality of human performance are required, and the

human engineering required for operators and maintainers,

including the information and response needs at each interface.

Human Factors Efforts and System

Development Activities During

Demonstration/Validation

This subsection provides a description of the human factors

efforts and system development activities shown in Exhibit 3-8.

The descriptions are keyed to the numeric codes on the chart.

The product of the Concept Development Phase' should be input

directly to human factors activities requisite to the Demonstration/Validation Phase. The product of this phase contains two

ccmponents overall which distinguish two elements: (1) Task

Analysis, and (2) Human Engineering Requirements. The results

of the former serve as major input to the latter developments.

(2.11 Test and EvaZuation Master PZan (TEMP). The TEMP

is the controlling document which derives test and evaluation

requirements for development test and evaluation and operational

test and evaluation. The TEMP identifies decision criteria and

funding constraints in support of the overall approved program

objectives. ' "

(Adapted from OPNAVINST' 5000.42A and SECNAVINST 5000.1A)

(2.2) Submit ProposaZa. Since development of a prototype

system is a major requirement of the Demonstration and Validation

Phase, a request for proposal (RFP) must be prepared, contractor

proposals written and 'submitted, and a winning contractor(s)

awarded the contract to develop the prototype. Requirements to

be issued in the RFP should be obtained from the TEMP, the DCP of

the previous phase, and previous activities' results.

3-43
Exlhibit 3-8

Specific Ituman - _ctors Fffort,, During the Dezonstration/Validation Phase (2.)

r I•, P.UNCTIONS ALLOCATED l MILE'STONE I

21'2.31 12.32 (2.3)

TEST a / TASK OPERATIONAL MAI NTENANCE

EVALUATION RIEOUIREMENTS SEQUENCE REQUIREMENTS

MASTER PeLAW" ANALYSIS ANALIYSIS ANALYSIS

DREMUIREMENTS

EVALUATI3N DESIGN REQUIREMENTS SEQUENCE DEFINED

* IOENTFIED IDENTIFIED * 2.34)

,2.2S IUSMIT *MAiNTENANCE

I':'OPOSALS * PHILOSOPHIES

CONTRACTOR ARRANGEMENT A SiEL.ECTEDO CONICEPTS MAINTENANCE CONCEPT

LT CHOSEN

CONSTRUCT CONCEPT SELECTED

PROTOTYPE 2.361

WORK SPACE

.o....... CONCEPTS

PROTOTYPES

FOR TESTING CONCEPT SELECTED

CONDUCT , CONSOLE CONCEPTS I hOT I

CONCEPT IELECTED

EVALUATIONS LEGEND. CONTROL . . DISP..Ay

CONCEPT QSEITERI0

CONCURRENT

EPOE

CONCE'? SELECTE

(2.39)

"RECOMMENDATION * ' CWN090R "Pt DESIGN

* ALTERNATIVES

~ARCII *DESIGN

48) SARCIl * : , RECOMMENDATIONS , Y (2.3.1 ,l

* 5IULATIOAIMOCKUP l VALUATIONS.

RSC lNOATIOiN • TASK ANALYSIS AND

• UANINGINIEERINGAEOUIElMETS •2.3111) • .".... IF (2.411

ICONDUCT!

TASK ANALYSIS AND HUFAN

ENGINEERING RECUIRE11ENTS VALIDATED 42,42 . •L.. .- '- . L---- MILfEroNEi2

A""OVE

.3-44
(2.3) Construct Prototype. The purpose of a prototyping

effort in the Demonstration/Validation Phase is to confirm

that the technology is feasible and that the design concept

has military utility against a stated military requirement.

Prototypes may also be fabricated for competitive evaluation to

select the best approach for further development. Human factors

activities relevant to the Demonstration/Validation Phase should

be conducted in association with the development of the prototype.

(Adapted from AR 70-1)

(2.31) Task Requirements Analysis. Requirements for human

performance are generally gained through the development of a

task analysis for a specific system in mind. Mlost task analysis

requirements are derived through a two-step process involving:

(1) subtask derivation and (2) skill and knowledge analysis.

Subtask derivation results in task descriptions, work designation

(operator/maintainer/support), task locus, and behavior and time/

sequence variables. Task skill and knowledge analysis results in

assignment of skill level to tasks/subtasks, military specialty

requirements, and necessary and/or special knowledge requirements.

(Adapted from VanCott & Kinkade, 1972)

(2.32) Operational Sequence Analysis. Dynamic analysis of

the operationsenvironment, such as that offered by operational

sequence diagramming (as opposed to static analysis such as'that

offered by task analysis), provides time-based data revealing

among other things operator workload requirements, performance

requirarments exceeding operator and equipment capabilities, and

sequences amenable'to translation into training and mission

scenario development.

(2.33) Maintenance Requirements Ana~yeie. A maintenance

task equipment requirementa analysis is conducted to define

3-45
requisite maintenance tasks, test equipment utilized and

procedures for their use, equipment to be maintained, and

malfunctions possible. This data is used to determine skill

level and knowledge requirements .or maintainers of system

components, and training requirements for maintenance (including

criteria and measures of performance). In addition, maintenance

support should be considered for inputs on maintenance of system

effectiveness.

(2.34) Consider Maintenance Philosop;7ies. Having obtained

detailed requirements for system maintenance, consideration

should be given to the choice of maintenance philosophies to be

implemented. These especially involve the input of human fectors

to agencies responsible for training and support of maintainers.

Recommendations for training requ-Lrements, trairing devices and

simulation requirements, and job aids and manuals should also be

developed.

The previous discussion constitutes what is roughly equivalent to the determination )f requirements for human performance

through human task analysis. This data (along with data prepared

in previous stages of system development) should be utilized in

the specification of human factors engineering requirements.

These requirements are then used to suppoit development of the

prototype system. The ensuing distussion details the general

plan for this process.

(z.35) Station Arrangement Concepta. Proceeding from a

general level of detail, a preliminary arrangement of personnel

and equipment within the workstation is made. Thi- process is

generally based on knowledge of information flow within the

station (as determined by such a technique a, information flow

charting) as well as knowledge of communication and operator

traffic flows (as determinea in link analysis, for example). t

-3-46
(2.36) Workspace Concepts. While the station arrangement

process described above is concerned with interactive performance

of men and machines, workspace concepts deal with environmental

and other associated aspects of the station and other locations,

involving even men in passive states with respect to the system.

Environmental effects deal with issues of climate and habitability.

Safety, personnel mobility, equipment space, and other associdted

requirements must be considered in relation to workspace conditions.

(2.37) Console C¢ncept.3. The above considerations dealt

with crews and/or nonspecific individual operator requirements.

Console concepts, on the other hand, involve primarily a specific

operator in relation to specific duties and equipment/components.

Considerationg that require examination for design implications

of individual operator consoles include operator visual, auditory,

manipulatory, and ambulatory requirements. Different standards

are applied to console concept selection involving stand-up and

sit-down operators.

(2.345) Control-Display Concepts. The final analysis in

relation to development of personnel work stations is the specification of control-display concepts. This paper-and-pencil

arrangement analysis of controls and displays on panels and

consoles should be based on an analysis of operator utilization

frequency, accuracy, sequence, etc. and the importance of these

displays and controls to controlling or monitoring system performance. Guidelines for these purposes include general topicr

such as the following:

1. Priorities for locating controls and displays

2. Spacing between controls and displays

3. Grouping controls and displays to either function or

sequence

4. Sequence of operation.

(Adapted from MIL-HDBK-759)'

3-47
The preceding discussion is clearly slanted toward the

.operations side of a system. Analysis of the maintenance portion

must also proceed through a process similar for operations.

Concepts to be included in this sort of maintenance analysis must

focus upon aspects of equipment and test equipment portability,

equipment and workspace accessibility,.and procedures associated

with maintenance actions (e.g., fault isolation, correction, and

prevention).

2.39 cside2-H:.ran _ Fac:ora Engineering (HFE) Design

ALternatii.,a. The concepts derived above must'now be grouped

together to evolve alternative requirements for human factors

design wherever .here is a man-machine interface. Analysis at

this stage must reconcila potential crew interaction problems

and individual workload capabilities and limitations. Each

alternative, as well as the one selected as optimal, should be

presented in the form of drawings, tabulations, and narratives.

(2.310) SimuZation/Mockup Evaluations. Tc begin verification of analyses performed up to this point, as well as to begin

HFE detail design based on actual studies of the man-machine

intarface, the use of' simulation, mockups, and their subsequent

evaluation must be performed. These activities should result in

determix:ing the efficacy of the HFE design alternative recommended, as opposed to the remaining alternatives. In addition,

development and refinement of specific HFE design parameters

should also proceed to the extent of confirming the validity of

HFE design alternatives--this can be done to the level of

drawings, tabulatio:.s, and narratives, or through providing data

-pertinent to thei. modification.

(2.311) Human Performanae and Human Engineering Requirements.

As stated previously, the major HFE product of the D.mcnatration/

Validation Phase is the ,pecification of human task analysis and

3-48
human factors engineering requirements. Since it is a major

product, it has been previously introduced as such and will not

be repeated here.

(2.4) Conduct DT I/OT I. Primary verification of the

feasibility of a prototype system to achieve stated requirements

is determined through development test (DT) and operational test

(OT). In the Demonstration/Validation Phase these are DT I/OT I.

DT I is c')nducted to demonstrate that technical

risks have been identified and that solutions

are in hand. Components, subsystems, brassboard

configuration or advanced development prototypes

are examined to evaluate the potential application

of technology and related design approaches prior.

to entry into full scale development.

(DA Pam 11-25)

OT I is conducted to determine military validity and worth

to the user. OT I estimates:

a. The Fotential of the new system in relocation to

existing capabilities

b. The relative merits of available competing prototypes/

systems from the aspects of military utility

c. The adequacy of the concepts for employment, support

ability organization, doctrinal, tactical', and training

requirements, and related critical issues.

(Adapted from DA Pam 11-25)

(2.42) Humn Fac tore TConduot TE. Human factors test and

evaluation (HP T&E) is'conducted in conjunction with OT I. The

purpcse of HF T&E for the overall system development is to

demonstrate that human performance technical risks have been

identified along with their solutions. 1HF T&E also attempts to

validate the human task analysis and the human factors engineering

requirements.

I
K. ;::) Cash Ana ysia and Eurnan Fact+o2 Engioneering. Require-

?6fZrs ;'al :.tea. The development of validated task analysis and

human factcrs enýinoering requirements will provide the next phase

of development with valid human factors data to firm up the

detail design wherever a man-machine interface is located.

r 7a t e PCI. 'The Decision Coordinating Paper (DCP)

is updated to include recommendations for further system development as well as designation of preferred' alternative designs

and rationales fqcr such choices.

'xC7 C.ARC I.< ,S) SA FO II. The purpose of DSARC II/

(S)SARC i1 is to evaluate the readiness of ti-e system development

program to enter full-scale development. Reviews are conducted

of the I)CP, among other documents. Approval by the DSARC and

(S... RC sets the stage for ,-cntinued development of the system.

Content of the Task Analysi's and

Hl:man I eerin g Pequirements Produ c t

A documented task analysis and statement of the system

human engineering requirements shall include the following

considerations:

Assumptions:

Th:ý fotlowinrg itetms will serve as input to the process of

detejr-ining humar performance and human factors engineering.

requirements:

* MENS

* DCP

* Products of function allocation.

Task analyt' techniques will be utilized to encompass'

pertinent aspects of operitions and maintenance for a

proposed system. kRcquirements for human factors engineering

will also "neompaq operations and maintenance.

Ji 3'-50

a,
Actions:

1. The principal product of the human task analysis portion

of this phase will be a completed task analytic package

(including static and dynamic aspects for all tasks).

Overall, the package will provide the following data:

a. tasks and task sequences required of operators and

maintainers

b. actual equipment employed

c. safety

d. maintenance.

* Techniques utilized to derive these data will include

procedures such as the following: Behavioral Task

Analysis, Operability/Maintainability Analysis. Hazard

Analysis, Workload Analysis, Task-Equipment Analysis,

Operational Sequence Diagrams, and Link Analysis.

2. The overall task analysis, including task descriptions,

will be presented in the form of flow diagrams, tabular

presentations, and narratives.

3. The human task analysis will commence with a sunmmary of

gross tasks. This summary will demonstrate the feasibility of achieving system performance requirements as

well as ensuring that human performance requirements do

not exceed capabilities. In addition, the effects upon

the following items will be described:

a. manning level

b. equipment procedures

c. requisite skills and training

d. communication requirements (between operators and

operators and the system)

e. logistics support.

• 3-51

I\

I _ ___
4. The human task analysis will specify tasks critical to

system performance as well as evidence to support its

criticality. These tasks will include but not be

limited to the following data:

a. information requirements by operators/maintainers

(including cues for task initiation)

b. information available to operators/maintainers

c.. evaluation process

d. decisi-ns reached after evaluation

e. action taken

f. body movement required by action taken

g. workspace envelope required by action taken

h. workspace available

i. location and condition of work environment

j. frequency and tolerance of action

k. time base

1. feedback, informing operators/maintainers of the

adequacy :f action taken

m. tools and equipment required

n. number of personnel, specialties, and experience

0. job aids )r references

p. communica ion required (including type'

q. hazards

r. interaction of multiple personnel

s. operational limits of personnel (performance)

t. operation .1 limits of machine and software.

* 3-52

ha ". 'i**..-
,p

5. The human ti-sk analysis package will provide the results

of an opere:.ility/maintainability workload analysis

(including i.he interaction of multiple personnel). The

operability analysis will detail the following:

a. design goal--quantity and quality of information

throughput

b. predict expected quantity and quality of throughput

operators should'expect

c. comparison of predicted with desired throughput and

resolution of differences.

The maintainability analysis will detail the following:

a. desighgbaIle"fc of-automated

maintenar' e

b. predict performance times for correction (including

identification, fault isolation, and correction) of

system malfunctions

c.. compare predicted maintenance with goal and resolve

differences.

6. Develop requirements for human factors engineering by,

analysis of effects of critical tasks upon system and

equipment performance, cost, periods of peak personnel

workload, conflict situations placing demands upon

personnel and equipment as well as requirements-not

previously apparent. In addition, life support characteristics will be detailed covering but not limited to

the following: noise, shock and vibration, temperature

extremes, atmospheric contamination, toxicity, electric

shock, mechanical hazards, electromagnetic and nuclear

radiation, explosion/fire, pressure and/or decompression.

3-53

joV

-, . . -
This analysis will also result in the prediction of the

probabilities for operator and maintainer error. Details

to be included in the error analysis are:

a. identification of the locus of errors

b. malfunction

c. extreme conditions and environments

d. effects of enemy action

e. recoianmendations for avoidance of design-induced

error

f. rating of error likelihood

g. rating of error criticality

h. estimate of seriousness of consequences to personnel

and/or equipment; and system, subsystem, and/or

component performance.

7. Additional requirements for human factors engineering

involved with development of procedural documents,

personnel planning, and system testing will be developed.

This data will be obtained from ai analysis resulting

from the compilation of task-related data into preliminary operator/maintainer procedurally oriented task

descriptions. (Especially important in this regard

would be the determination of system and personnel

performance time and accuracy requirements to be used

in system test'and evaluation. A sequential'analysis

of the operational sequence diagram would provide these

data on a dynamic basis suitable for this use.)

3-54
Demonstration/Validation Phase:

Example of Human Factors

Contribution

The crewstations of an aircraft must be usable by crewmembers who vary widely in physical size. If a significant

number of pilots cannot reach the critical controls, accidents

and injuries will increase. Between 15 and 20 aircraft mishaps

per year have been attributed to difficulties in reach, at a cost

of 20-30 million dollars in damage. The problem has become

serious enough that the Chief of Naval Operations recently asked

that aviators be matched with specific aircraft according to how

well they "fit" the physical dimensions of the cockpits and

controls. W-ile this approach is effective in reducing accidents,

it limits the use of the trained aircrew population and wastes

valuable training and retraining time. Pilot/cockpit size mismatches can usually be solved by early engineering design changes.

But to do so requires that cockpit geometry mismatches be detected

while the aircraft is still on the drawing board. To that end,

the Navy heeds a method of analysis that can compare and quantify

planned' cockpit geometry against the aircrew population at this

early design stage.

To meet the need for a method of comparison, NADC developed

the Crews tation Assessment of*Reach (CAR).model. The CAR model

is based on extensive prior research in industry and government.

This work has resulted in sophisticated cockpit geometry models

which can compare the physical dimensions of a specific operator

against the dimensions of a proposed crewstation. The CAR model

uses a condensed version of lhose earlier models to evaluate a

cockpit design against a statistical sample representing the

entire operator population. Thus, CAR is able to estimate, the

percentage of available aircrewmen who can operate a proposed

design and the percentage who will have difficulty in performing

any specific control action.

3-55
CAR is applied at the earliest possible stage of design,

using the initial C.rawings as an input. The model examines

hand and leg control positions, head/cancpy clearance, and seat

movement required to achieve over-the-nose vision. Where reach

or clearance problems are detected, CAR identifies the controls

involved. Because the computer program is "interactive," the

researchers can immediately evaluate alternative designs. A

large number of alternatives can be explored with a minimum

of time and cost, and acceptable solutions can be identified

promptly.

CAR uses a mathematical model of the human skeleton, consisting of the major body segments ("links") and the joints which

connect those links, with all their lengths, limits of mnvement,

and variations in dimension within the operator population (see

figure). It can quickly calculate how the skeletal model must

move to perform any specific action, undex various conditions of

harness restraint or requirements for hand action (e.g., grasp,

touch, manipulate).

--

C~Onaf LIM.

*3-56

• , . . , _M a,

MUK LM
CAR has been used in the design of three aircraft: the

Light Airborne Multi-Purpose System (LAMPS) MK-III (SH-3), the

F-18, and the AV-8B. For LAMPS and the F-18, changes were

recommended and were used in further development. For the

AV-iiB, studies are still in progress to correct some identified

problems.

Applied to the F-18 preliminary design, CAR revealed that

only 10% of the aviator population would be able to use all

critical controls. The seat, stick, and emergency controls were

therefore relocated, using CAR recommendations, to accommodate

nearly 100% of aviators. The engineering changes that were

required included major modifications of the aircraft structure.

CAR has been adopted for use elsewhere in the government

and in industry. Within the government, it has been modified by

NASA for Space Shuttle design. In industry, it has been used

in-house by McDonnell Douglas, Northrup, Sikorsky, the Clark

Equipment Company, and IBM.

Earlier methods of analyzing cockpit geometry required

laborious manual procedures, or eLse' computer models not suitable

for use in' early Stages of design. The results were often

expensive, late, imprecise, and hird to convey to design engineers.

At worst, problems remained undet cted until the aircraft were in

service, and, then often surfaced is accident data. CAR improves

the accuracy. of analysis while re lucing the time required from

more than two weeks to less than day, with a 90% decrease in

cost. CAR can be applied earlier in the design cycle than ever

before, and used interactively to findengineering answers and

test them ahead of time. It will produce aircraft which are more

mission-effective because they are better fitted to the aircrew.

3-57
N

For example, when used to guide design of the F-18, CAR made

it possible to correct on paper design deficiencies that would

have cost millions to change if not detected until construction

began. Savings through avoiding lost training time for pilots

who could not have safely used the initial design are estimated

at 10 to 40 million dollars per year.

Full-Scale Development Phase Human Factors

This phase should result in a firm and detailed man-machine

interface design. At the start of the phase, various man-machine

combinations should be considered that would satisfy the human

performance and human factors engineering requirements. The

testing of these combinations by simulation or mockups should

identify the most effective combinations. Simulation trials are

a good method for pinpointing peak personnel and equipment workloads, detecting probable human errors, identifying inefficient

interfaces, and determining if the design is appropriate for the

intended user.

The equipment must be designed to meet the physical and

cognitive needs of the intended user. Physically, the equipment

must be designed to permit effective movement, effective use of

knobs and controls, transporting of goods, effective use of arms

and legs, or whatever the tasks call for. Cognitively, the human

factors engineer is responsible for recommending designs that

are neither too difficultnor simplistic, but that, rather,

assign the proper amount of Cognitive workload to different

kinds of users. To have an effective system, the design must

differ for different types of users. One design would be appropriate if a position is expected to have high turnover in short

periods of time and the user is to be minimally trained and to

possess few skills, while another design would be appropriate for

a position with less turnover and occupied by a user with better

training, skills, knowledge., and ability to make complex decisions.

3-58
The man-machine tests and evaluations should cover every

detail of the design, from such things as effectiveness of a

type of information displayed at various times to control-display

compatibility, spacing of controls, shape of controls, sequence

of controls, anthropometry, and in general, all of those areas

in MIL-STD-1472B and MIL-H-46855B. The HF T&E in this phase

should result in a reliable man-machine design.

Human. Factors Efforts and System

Development Activities During

FFu-Scale Development

The major human factors products of the Demonstration/

Validation Phase, task analysis, and human engineering requirements serve as the basic data for the detailed human factors

design of the system wherever a man-machine interface occurs.

This subsection provides a description of the humanfactors

efforts and system development activities shown in Exhibit 3-9.

The descriptions are keyed to the numeric codes on the chart.

(3.1) Submit Proposals.' It is possible that in full-scale

development a different, contractor may be selected than the

contractor employed in the Demonstration and Validation Phase.

In addition, it-is also possible (albeit unlikely)'to continue

with competitive developments. For these reasons (although they

are somewhat rare in occurrence) the complete cycle involving

RFPs, proposals, and contract award is repeated.

3.2 Construct Prototype(s). When development prototypes.

are fabricated in full-scale development, the intent is to assure

that the engineering problems have been solved and to permit

thorough evaluation of the system. (This occurs prior to a

commitment to full-scale production or simultaneously with lowrate initial production.)

(AR 70-1)

3-59
Exhibit 3-9

Specific fhuman Factors, Efforts During the Full-Scale Development Phase (3.)

TASK ANALYSIS I AND HE REQUIREMENTS

-----------------------------------------------------------.--- Ml LESTONE 2

SUBMIT COSIE HT DESIG

PROPOSALS ALTERNATIVES

VýNTAACTOR DESIGN

SE L TE D RECOMMENDATICAdS

(3.21 13.22)

[ CONSTR:UCT SIMULATIONIMOCKUP

PROTOTYPEI. EVALUATIONS

PROTOTYPESSI fETAILED MNIICIll

r ORTESTNG .................... NTERFACE OESIGNf3.23)

(3.3)(3.31)

OT 111 OT it CONDUCT

AIF T&Z

EVALUATIONS

43.4)

UPDATE

DETAILED MAN-MACHINE

RECOMENDTIONINTERFACE DESIGN VALIDATED

43.32)

(3.0)

-DSARC 1111'_________

* ISISARC III 1 LEaEN O: 1

ONEITMEIRIOR

R4ECOMMENDATION V.CNURN

RECONSIDER

I ~ 'EL b-- MILESTONE 3

APPROVESYSTEM

.3-60
It has traditionally been during this stage of development

(prototyping in full-scale development) that the bulk of human

factors R&D has occurred.

(3.21) Consider HFE Design Alternatives. Before entering

full-scale development, a decision is required whether to

accept or modify the prototype system built and tested previously.

in the Demonstration/Validation Phase. For HFE proper, this

encompasses the man-machine interface. This is necessary when

any or all of the following events occur:

1. Contractor awarded full-scale development is different

from contractor during Demonstration/Validation Phase.

2. Deficiencies identified through the HFE portion of OT I

require modification to the system (this may also include

DT I/OT I findings at large as well).

3. Design requirements change development of hardware and

software components (e.g., to take advantage of newly

breaking technologies), thus forcing HFE to keep abreast

r* development.

Based on events occurring as illustrated above, HFE will provide

design recommendations.

(3.22) Simulation/Mockup Evaluationa. Should hardware and/

or software design requirements be modified resulting in changes

to the man-machine interface, new studies and analyses involving

simulation and/or mockups may become necessary to evaluate r.he

effects of change upon personnel (operators/maintainer.,) workload,

and assigned activities. Analysis may be required especially when

the effects of design changes are unknown.

'(3.23) Detailed Man-Machine Interface Deeign. Final HFE

design requirements should be prepared in the following formats:

3-61.
drawings, tabulations, and narratives. This will facilitate their

implementation in detail design of the full-scale development

prototype. HFE requirements include human eligineering principles

and criteria which offer assurance that the final product can be

efficiently, reliably, and safely operated and maintained.

Relevant locations for the application and human factors

engineering include the work environment, crewstation, and

facilities being designed for the system.

('Portions from MIL-H-46855)

The product of this phase is the design of an optimal manmachine L.Aterface. It is discussed elsewhere and will not be

repeated here.

(3.3) Conduct DT IH/OT II. Developmental Test II and

Orerational Test II (DT II/OT II).are required to determine

whether or not the full-scale development prototype is ready

for production.

DT II einsures that engineering is reasonably complete, that

all significant design problems have been identified, and that

solutions are in hand.'

OT II provides a valid estimate of expected system operational effectiveness and.suitabilityas determined through tests

involving the aid of operational and support personnel of the

type and qualifications of those' who are expected to use and

maintain the system when deployed.

(Adapted .from AR 70-10)

(3.31) Conduot HF T&E. Conjointly with'OT II, Human Factors

test and Evaluation of the full-scale development system w4 11 be

conducted to:

3-62
1. Assure fulfillment of the applicable requirements

2. Demonstrate conformance to HFE design criteria

.3. Determine whether undesirable design or procedural

features have been introduced.

(Adapted from MIL-H-46855)

(3.32) DetaiZed ran-Machine Interface Design VaZidated.

The results of an HF T&E should be validation and verification

of design requirements to provide an optimal man-machine

interface design.

(3.4) Upda t e DCP. The Decision Coordinating Oaper is

updated to include a current evaluation of the system. The

decision to proceed into full production must be based on this

DCP.

(3.5) DSAF%' III/(S)SARC III. Defense System Acquisition

Review Council III/(Service) System Acquisition Review Council

III purposes are to recommend to the Secretary of Defense

approval of production (or, possibly, low-rate initial production)

of a system. The DCP, among other documents, is reviewed during

this process.

(DA Pam 11-25)

Content of the Optimal Man-Machine

Interface Design

The optimal man-machine interface design recommendations

should include the following considerations:

Assumptions:

The following items will be regarded as inputs to the human

factors engineering design of the man-machine interface:

f- 3 ,
o Design criteria documents (e.g., MIL-STD-14721

s Performance specifications

* Drawings and data (e.g., functional flow diagrams,

schematic block diagrams, interface control drawings,

overall layout drawings)

* Human factors engineering input (e.g., task analysis)

converted to detail equipment design features..

"The following processes are considered characteristic of

this phase of system development:

o Human factors engineering studies, experiments, and

"laboratory tests (to resolve human factors and life

support issues)

o Mockups and models

o Dynamic simulation (necessary for detail design of

equipment requiring critical human performance)

o Human factors engineering contributions to detail

design

o Human factors engineering contributions to manpower,

personnel, and training issues as a consequence of

detail design

i Human factors contributions to test and evaluation.

Actions:

1. Effects of the working environment, including

habitability and operability, will-be preue-ted.'

These effects will cover tho following arcas: work

environment, crew stations, and facilities. The

incorporation of human factors into the detail

design of the above will be demonstrated by presenting

detail design drawings, specifications, etc. for the

following three conditionst- normal, unusual, emergency.

3-64
Topic's to receive coverage will include at least the

following:

a. atmospheric conditions

b. weather and climate

c. range of accelerative forces

d. acoustic noise, shock, and vibration

e. disorientation

f. accessibility

g. adequate visual, auditory, and physical links

h. adequate non-workspace areas'

i. psychophysical stress

j. fatigue

k. clothing and personal equipment

1. equipment handling

m. chemical, biological, electrical, electromagnetic,,

toxicological, and radiological effects

n. illumination

o. sustenance, storage, and refuse

p. safety protection.

2. The incorporation of human factors in detail design of

the crewstation layout/arrangement and of equipment

having an operator/maintainer interface will'be demonstrated. This will include'the presentation of drawings

illustrating the inclusion of human factors; for

example: panel layout drawingo, communication system

drawings, overall layout drawings, and control drawings.

The following additional items will be requisite to the'

demonstration of the inclusion of human Zactors in system

detail design:

3-65
a. ingress and egress to workspace and facilities

b. a list of panels, racks, controls, displays, and

indicators existing at the time of documentation

which have received human factors approval

c. rationale of human factors layout/arrangement,

detail design of crew station(s), and any equipment

having an operator/maintainer interface

d. a list of considerations used to arrive at design

decisions: results of studies, requirements based

on task analysis, mock-up tests, mock-up based

decisions, and simulations

e. a list and explanation for deviations from human

factors or design requirements to the man-machine

interface

f. sketches,, drawings, and photographs of required or

anticipated panel and rack arrangements or new

designs/design modifications

g. drawings or photographs of each crewstation design

showing locations of all crewstation panels in

relation to seat/operator position.

3. The inclusion of human factors in-design considerations

involving the interaction of maintenance technicians with

their respective equipment will be demonstrated. In

general, this will depict the' following steps/stages:

a. recognition of malfunctions (displays)

b. isolation of malfunctions (troubleshooting)

c. fault correction (access, removal and replacement,

repair)..

3-66
A human factors maintainability/accessibility design

analysis will be presented to include at least the

following:

a. preliminary drawings, sketches, or photographs

showing each equipment and location in relation to

surrounding equipment, passageways, and structures

(this includes ancillary equipment also)

b. rationale of human factors design of each item

requiring maintenance as well as presentation of

decisions used to drive the decision process

(e.g., MIL-STD-1472, results of s~tudies, simulation,

mockups)

c. incorporation of maintenance task analysis

d. descriptions to include but not be limited to the

following:

* physical size, purpose of support, and test

equipment required for maintenance

* maintenance procedures

* relation between accessibility and failure rate,

service frequency, calibration frequency, and

requirements for rapid maintenance

* methods used to determine accessibility for

maintenance

* anticipated maintenance and accessibility

problem areas.

4. Best available data on equipment operating procedures,

operational sequence diagrams, and task analysis will

be provided to organizations responsible for manpower

development.

3-67
5. A human factors test and evaluation plan will be

prepared to cover the following general concepts:

a. fulfillment of human factors requirements

b. conformance to human factors design criteria

c. quantitative measures of system performance

d. detection of undesirable design or procedural

features.

Full-Scale Development Phase:

Two Examples of Human Factors Contributions

The example that follows is actually a case of system

modification rather than system development, but the human

factors impact is conceptually the same.

The Strategic Air Command (SAC) is continually considering

proposed new hardware or modifications to existing hardware for.

improving offensive and defensive avionics equipment in the B-52

fleet. These new subsystems are expensive and must be evaluated

with respect to the degree of additional combat crew effectiveness

that can be realized. In short, the new equipment must be

operable in the 8-52 mission tinvironment by the current population.

of'electronic warfare officers and must result in improved mission

performance.

The Strategic Avionics Crew Station Design Evaluation

Facility (SACDEF), developed by the Human Engineering Division

at Wright-Patterson Air Force Base, is on-line to provide the

effective selection of offensive and defensive'avionics equipment

and integration for B-52 improvements. This capability involves

the quantification of Strategic Air Command crewmember actions in

conducting simulated Single Integrated Operations Plan (SlOP)

3-68
missions. It uses computer-integrated electronic warfare and

bombing/navigation systems simulators to evaluate performance

improvements obtained by means of proposed new equipments and

crew station reconfigurations.

Trained crewmembers' performance is monitored and statistical analyses of combat crew performance with the proposed

hardware are provided to SAC. Recommendatiods made by the human

factors specialists have been well received (87 of 93 original

recommendations were adopted in the Phase VI update of the B-52

fleet). Further, the Strategic Air Command has adopted the

policy that no new hardware will be installed on the B-52 fleet

until their trained crewmembers have participated in the evaluation studies conducted at Wright-Patterson AFB. The results of

this capability are not only used directly by the using command,

but aiso provide technology advances in systems effectiveness

modeling and simulation that will enhance man-machine integration

"try-before-buy" evaluation of new weapon systems.

The Aerospace Medical Research Laboratory is now developing

a comparable test and evaluation capability for navigator/radar

navigator functions in conjunction with the proposed update of

the B-52 aiionics systems in support of SAC RQC 75-6.

A final example follows which fully illustrates the impact

of huiman factors R&D upon design and development of militarysystems in 'the Full-Scale Development Phase (Gartner et, al.,

1958).

Good human engineering of equipment controls and displays

has &Long been a hallmark of human factors R&D. This is due to

the fact that well human engineered equipment design can have a

dramatic effect upon the operability of systems. Rarely, though,

3-69
has there been an opportunity to compare a "pre-human engineered"

design of an item of equipment with a human engineered version in

order to substantiate claims as to the value of this particular

human factors technology. One such opportunity did arise,

however, when an empirical experimental investigation between

two alternative designs of test equipment for complex naval mines

was performed. This project, which involved development of

design recommendations, detail design, fabrication:s experimental

investigation, and evaluation of test equipment items, was

embedded within a larger program to offer human engineering

support to test equipment designs as well as development of a

human engineering design guidebook for engineers.

As outlined above, the following approach was taken:

Two test equipment items were fabricated according to human

engineering recommendations and the same two items were fabricated according to their original designs. The original design

test equipment and the human engineered test equipment were then

empirically compared against each other with respect to criteria

of time and error. Results indicated that successful improvements in performance (i.e., use) of the human engineered test

equipment occurred with regard to reductions both in time to

task completion and reduced erxor likelihood. With practice,

performance on both test equipment designs converged. -However,

since it was known that utilization of the test. equipment was

to be too infrequent for users to sustain learning, the human

engineered version was considered necessary. Furthermore, it

was also felt that practice effects would operate for either

design version, with the expectation of greater influence on

operators using the human engineered equipment.

This research demonstrates the value of human factors R&D in

Navy mine test set design. Implementation of human factors

design recommendations (such as were developed for the 'test set

"3-70
equipment and documented in a guidebook for test equipment

development and design) in a systematic and standardized manner

could lead to greater user performance differences between the

human engineered and non-human engineered designs. This may be

due to overall familiarity with characteristic arrangements of

controls and displays. Standardization procedures in human

engineering design which utilize known population stereotypes

with regard to user expectations could result in dramatic

reductions in time to task completion and human error. These

goals are crucial to design of an optimal man-machine interface.

Production and Deployment Phase Human Factors

If human factors have been properly thought out and executed,

the system should be ready for production and deployment after

Milestone 3. However, there are provisions for additional

engineering and human factors testing (DT/OT III) if the tests

and evaluations in DT/OT II indicate problems. The additional

testing is conducted on the first few systems in the initial lowrate production. When additional testing indicates the problems

have been corrected, the system goes into full-scale production.

If any problems arise or if improvements seem prudent after

the system it fielded, the system acquisition model provides for

additional testing to recommend design changes. A good human

factors plan will keep these costly design changes to a minimum.

The scope of the present project is strictly with the system

development phases of the ac4uisition process and is not concerned

with production and deployment. However, to provide some closure

to the process, a brief description of production and deployment

follows. No human factors products are defined, and no examples

are included.

3"71
Human Factors Efforts and System

Development Activities During

Production and Deployment

This subsection provides brief descriptions of the general

human factors efforts and system production and deployment

activities shown in Exhibit 3-10. The descriptions are keyed to

the numeric codes on the chart.

(4.1) Initial Production. When it is decided to enter

production of a system, initial production items are generally

used for production tests and follow-on evaluations as necessary.

Generally, production is not suppressed to await completion of

follow-on evaluation (nor for that matter does deployment await

conclusion of this evaluation).

(Adapted from AR 1000-1)

(4.2) Conduct DT III/CT III. Development Test III/

Operational Test III are conducted to determine if production

units have the capabilities demonstrated in prototypes and are

operationally suitable and effective.

DT III is conducted on production prototypes or production

items delivered from either an initial or a pilot production run.

The purpose is to verify their adequacy and quality when they

are produced in quantity and according to production contract

specifications, using quantity production 'processes. This test

determines whether or not the transition from an engineering

development prototype to a production item has been made

successfully.

OT III is normally a test of initial production and has the

fundamental: purpose of providing data on the item or system in

order to estimate its operational suitability, verifying that all

testable critical issues have been resolved, and determining that

..all benefits and burdens of the item or system are identified.

(Adapted from AR 70-10)

3-72
z 2'

2*R

IL 0

*1

V z 2

w * W

* O w Lox *. 2- * L * 

S *

. *0(

IA**** j 0 - * -

3-73
Additional testing is implemented where required to resolve

(hopefully) residual problems.

(4.21) Study Problems. Any HFE problems identified

following Milestone 3 are studied to determine means to alleviate

them. In considering redesign, an analysis of the loss of

production time, increased costs due to redesign effort, and

production costs must be made in crder to realistically determine

what. human factors alternatives are feasible. Often, consideration is given to increased trainirg and/or personnel with higher

skill levels than was previously decided.

(4.22) Conduct HE T&E. Additional human factors test

and evaluation (HF T&E) may be ecessary to (1),determine the

efficacy of the proposed change (2) determine how well a

specific change has improved operation/maintenance of the system.

(4.23) New Configuration. A new man-machine interface

is configured as a result of human factors design changes.

Personnel and training requirements as well as system operability are often affected as a result of such changes, and may

necessitate further investigation.

(4.3) Full Production. Full-scale production will proceed

following approval based on findings of DT III/OT III.

(4.4) Deployment Preparation. Deployment of systems to

the field includes not only delivery and set-up of the new

system. It also requires fulfillment .of requirements found in

an initial operational capability, such as: -user unit is equipped

with production items that are deemed suitable, with unit

personnel that are adequately trained to operate, care for, and

maintain the item, and the unit has the capability to perform itz

assigned mission.

(Adapted from TRADOC Reg. 600-4)

.3-74
(4.5) Retrofit/Improvements. Based on prc.. !ems identified

in actual use. or change in doctrine, threat or mission product

improvements and/or retrofit programs may be needed to resolve

them,. This also requires the cognizance of, and often the analysis

by, HFE personnel to ensure that personnel and training requirements are covered as well as that system man-machine interface

optimization is maintained.

(4.51) Study Problems. See 4.21.

(4.52) Conduct HF T&E. See 4.22.

(4.53) New Configuration. See 4.23.

3-75
References

Baker, C.C., Johnson, J.H., Malone, M.T., & Malone, T.B. Human

factors engineering for Navy weapon system acquisition.

Naval Sea Systems Command, Washington, D.C., July 1979.

Coburn, R. Human engineering guide to ship system development:

Responsibilities, procedures, and methods for carrying out

human engineering programs. NELC/TD 278. San Diego, CA:

Naval Electronics Laboratory Center, October 1973.

Collins, J.J., McGuinness, J., Erlichman, J., & Boyce, S. Human

factors engineering integrated technology in total ship

planning and'acquisition: A first approximation. Prepared

under Contract N0024-75-C-6139 for Naval Sea Systems Command,

Wasa.ington, D.C., August 1975.

Department of the Air Force. Human factors engineering and

"management. '.FR 800-15. October 1974.

Department of the Army. Army research, development, and acquisition. AR 70-1. 1 February 1977.

Department of the Army. Test and evaluation during development

and acquisition of materiel. AR 70-10. 29 August 1975.

Department of the Army. Threat analysis. AR 381-11. 1 October

1974.

Department of the Army. Human factors engineering piogram.

AR 602-1. June 1976.

Department of the Army. Basic policies for systews acquisition.

AR 1000-1. 1 April 1978.

Department of the Army. Life cycle system managemen model 'or

Army systems. DA Pam 11-25. May 1975.

Department of Defense. Major systems acquisitions. DOD Directive

5000.1. January 1977.

Department of Defense. Major system acquisition process. DOD

Directive 5000.2. January 1977.

Department'of Defense. Test and evaluation. DOD Directive 5000.3.

April 1978.

Department of Defense. Major system acquisition.' D )D Directive

5000.1. March 1980.

"3-76

-X
Department of Defense. Major system acquisition procedures. DOD

Instruction 5000.2. March 1980.

Department of Defense. Test and evaluation. DOD Directive 5000.3.

December 1979.

Department of the Navy. Human factors. NAVMATINST 3900.9. Naval

Material Command, Washington, D.C., September 1970.

Erikson, R., Miles, J., & Secrist, G. Technology Coordination

Paper: Training and personnel technology. Department of

Defense, August 1978.

Gartner, W.B., Price, H.E., et al. An experimental evaluation of

the application of'human engineering principles to mine test

set design. PRA Report 58-2. Bureau of Ordnance, Department of the Navy, January 1958.'

Geddie, J.C. Human factors model. Personnel communication, HEL,

Aberdeen, MD, November 1979.

Goclowski, J.C., King, G.F., Ronco, P.G., & Askren, W.B.

Integration of human resource technologies in weapon system design: Process for the coordinated application of five

human resource technologies. AFHRL-TR-78-6(II). Wright- Patterson AFB, OH: Air Force Human Resources Laboratory,

March 1978.

Human engineering design criteria for military systems, equipment,

and faciJities. Military Specification MIL-STD-1472B, May 1978.

Human engineering requirements for military systems, equipment, and facilities. Military Specification MIL-H-46855B, 1979.

Kalan, J.D., & Crooks, W.H. A concept for developing human

performance specifications. TM-7-80. Aberdeen Proving Ground, MD: U.S; Army Human Engineering Laburatory, April'

1980.

Meister, D., & Rabideau, G.F. Human faotors evaluation in eystem deveZopment. New York: John Wiley & Sons, 1965.

Merriman, S.C. Human factors programs in Navy systems acquisition. PMC 76-.2. Defense Systems Management College, Ft. Belvoir, VA, November 1976.

MIL-HDBK-759. Military standardization handbook: Human factors

engineering design for Army materiel. Aberdeen Proving Cround, MDi U.S. Army Human Engineering Laboratory, March

1975.

3-77
MIL-STD-490. -Specification practices. Department of Defense,

30 October 1968.

Office of Management and Budget. Major system acquisitions.

Circular No. A-109. Executive Office of the President

(OMB), Washington, D.C., April 1976.

OPNAVINST 5000.42A. Weapon system selection and planning. Department of the Navy, 3 March 1976.

Price, H.E. Human engineering and system development. Presented

at the Product Assurance Symposium of the American Society

for Quality Control, San Fernando Valley Section, October

1962.

Price, H.E., Fiorello, M., Lowry, J.C., Smith, M.G., & Kidd, J.S.

Department of Defense and service requirements for human

factors R&D in the military system acquisition process.

ARI RN-80-23. U.S. Army Research Institute for the Behavioral

and Social Sciences, Alexandra, VA, July 1980.

Price, H.E., Smith, E.E., & Behan, R.A. Utilization of

acceptance data in a descriptive model for determining man's role in a system., NASA CR-95. National Aeronautics

and Space Administration, Washington, D.C., 1964.

Price, H.E., & Tabachnick, B.J. A descriptive model for determining optimal human performance in a system. Vol. III:

An approach for determining the optimal role of man and

allocation of functions in an aerospace system (including

technical appendices). NASA Ames Research Center, Moffett

Field, CA, March 1968.

SECNAVINST 5000.1A. System anquisition in the Department of the

Navy. Department of theANavy, 17 November 1978.

TRADOC Reg. 600-4. Integrated personnel support (IPS).

Headquarters, U.S. Army Training and Doctrine Command,

Ft. Monroe, VA, 1 June 1978.

Van Cott, H.P., & Kinkade, R.G. (Eds.). Human engineeriug guide

to equipment design. Washington, D.C.: U.S. Government

Printing Office, 1972.

.3-78
CHAPTER 4

HUMAN FACTORS RDT&E IN THE TECHNOLOGY BASE

AND THE CONTRIBUTION TO SYSTEM DEVELOPMENT

There has occasionally been some concern expressed thatI

Training and Personnel System Technology RDT&E is not fully

justified as being necessary to support specific system

development. This concern is not directed at human factors

in particular, but encompasses manpower and personiiel, education,

and training, and simulation and training devices. Establishment

of a clear correlation between the funding and performance of

technology base R&D and its utilization in specific systems

development is difficult, and is at any rate beyond the scope

of this project. However, in the human factors category it is

possible to discuss the potential for technology base RDT&E to

support specific system development by relating the technology

base R&D to the principal human factors products of each phase

of system development. In other words, the technology base R&D

in human factors should also be identified with:

1. Determining the role of man

2. Allocation of functions to man

3., Task analysis and human engineering requirements

4. Design of optimal man-machine interfaces.

By combining the system phase/human factors products with the

DOD classifications for human. factors into a matrix, areas of

oppodrtunity for human factors and system development have been

characterized. This matrix is shown as Exhibit 4-1. Exhibit 4-2

is included to remind the reader what kinds of research are

included in each classification. Each cell of this matrix

represents an area of opportunity for human factors to eventually

contribute to a specific system development.
1%

cr C UJ V)

0 WUc

- 6 z

ku 0

C. xE u ,

A 2

0 U.

or CCNr L Z0c/u 2

112 4

,34

z 4

LC2

U. j 

4 > 4 (a ZZ z

> 0 0 a 4

w- I- U

ma Z UI ~ j

4-2
Exhibit 4-2

The Classification System for Human Factors R&D

(Taken fiom the TCP for FY 1979, see Erickson, Miles, & Secrst, 1978)

Areas Examples

Human related Physical characteristics

Sensory capabilities

Information processing

Forezasting job requirements

Measures of effectiveness

Human-machine related Flight instrumentation

(subsystem oriented) Equipment layout

Maintenance

Workload assessment

Human-machino-mission related Strategic offense and defense command & control

Tactical offense and defense command & control

Command & control

Measures of. system effectiveness with inputs

Several things will influence whether the technology base

R&D in human factors does in fact contribute to the development

of a weapon system (or to the decision not to develop a system).'

We snail briefly address only two: (l) R&D funding categories,

and t2) the auditing method problem. These two were selected

because all human factors (in the technology base) is supported

by a particular category of funds, and because a cause-effect

relationship between research and utilization is a very difficult

thing to establish and measure.

4-3

* L.
R&D Funding Categories

Funding for all technology base areas in DOD is provided

within program elements in the budget which provide for funds

generally categorized as basic research (6.1 funds), exploratory

development (6.2 funds), or advanced development (6.3 funds).

DOD defines these funding categories as follows:

6.1 Basic Research--scientific study and experi- mentation directed toward increasing knowledge

and understanding in those fields of the sciences

related to long-term national security needs.

It provides fundamental knowledge for the solution

of identified military problems and furnishes part

of the base for subsequent exploratory and advanced

developments in defense-related technologies'and

new or improved military functional capabilities.

6.2 Exploratory Development--includes all effort

directed toward the solution of broadly defined

problems, short of major development programs,

with a view to developing and evaluating technical

feasibility.

6.3 AdvancedDevelopment--includes all projects

that have moved into the development of hardware

for test. The prime result of this type of

effort is.proof of design concept rather than

the development of hardware for service use.

Projects in this category have a potential military

application.

Advanced development is divided into two subcategories:

nonsystem advanced development (6.3A), addressing technological

option uncertainties; and systems advanced development (6.3B),

which is the design of items (usually hardware) for test or

experimentation.

It is very difficult to determine a clear point at which a

research and development activity moves from 6.1 to 6.2 to 6.3.

The distinction is often somewhat arbitrary. Some research

4-4
activities are in fact supported by more than one funding

category. Additionally, the individual services have different

management organizations and systems for administering technology

base funds. Finally, funding categories overlap in time; that

is, 6.1 does not end abruptly and 6.2 begin, but rather a project

funded under 6.1 may overlap in time with the same project being

supported by 6.2 funds. In short, it is not possible to precisely

and consistently identify funding categories and research progress

for human factors projects in the technology base.

Tracking Research to Utilization

There is no established method for trackinq or auditing

the results of technology base research to eventual utilization.

This is true not only in the human factors or the training and

personnel systems technology areas, but in other technologies

as ,;ell. Two studies will be briefly discussed to illustrate

this problem.

The first study was conducted by the General Accounting

Office (GAO) in 1977 and %as a review of human resources research

and development (now the training and personnel systems technology

area) in the Department of Defense. The excerpt below was taken

dizectly from the digest of that study.

Eight Defense research and development organizations Identified 374 reports on human resources research

and development published during calendar years 1973

through 1975 which were intended to support changes

to:

-- regulations, orders, doctrines, policies, or

manuals;

-- courses of instruction or training programs; or

-- equipment.

4-5

•j .
GAO then asked the intended users how the results

ý7ere used and any reasons for not using thenm and

found that:

3-- 6 percent of the reports were used,

-- 38 percent were not used, and

-- 6 percent were being considered for possible

use.

The GAO concluded that the Department of Defense could

improve utilization by more effective management. The authors

of the present report find no quarrel with this conclusion, but

do have some serious concerns about the method used to arrive

at the results. Those concerns will not be pursued here.

Nevertheless, a few points can be made about the review and

results as reported in the excerpt above. First of all, if

indeed 56% of the reports that were traced were used, this may

be a significant and positive finding. Other technology base

areas do not fare any better with the utilization of their

reports. A second point is that all research and development

does not find its way into eventual utilization. A very Valuable

payoff from research and development can be in the form of

negative findings, which stop the research itself or stop the

development of some system. Third, the GAO review was confined

to reports published from 1973 to 1975, and it is the opinion of

the present authors that some of the 38% not used will, eventually

find some utilization. One has to be more patient when

estaLlishing the relationship between research and utilization.

The second study was also condUcted in 1975, by the National

Heart and Lung Institute (NHLI) (see Comroe and Dripps, 1975),

and was concerned with the top 10 clinical advances in diagnosis,

prevention, and treatment of diseases of heart, blood vessels,

and lungs. This was a four-year study to analyze what knowledge

was required for the great &dvances since the early 1940s. 'Over

4-6
150 experts screened 4,000 scientific articles and then analyzed

529 of these that they considered to be essential for the top 10

clinical advances. Of the 529 key articles:

e 41% reported research that at the time was unrelated to

the later clinical advance (non-targeted research)

e 61.8% described basic research

9 21.2% were clinical investigations (targeted[research)

o 14% were concerned with the development of apparatus

techniques or procedures

* The key articles range in time over 200 years, with many

important ones being published as long as 75 years ago.

The important points about this study are fairly obvious. At the

time it was performed, 41% of the reported research was unrelated

to the problem it later helped to solve. And 21.2% of the

research, while clinical in nature, was unconcerned with the

fundamental issues. Also notable is the range of time (200 years)

that the eventually rel-zted research covered.

The NHLI investigators also point out that a major defect

in education and science is the perpetuation of the "one man

equals one discovery" myth (e.g., Marconi equals wireless;

Bell equals telephone).

Inferences about the technology base R&D efforts in human

factors that. are based on the NHLI study results are certainly

limited. Clearly, the quantitative results of one are not

applicable to the other; however, the-difficulty in auditing

or tracking cause-effect relationships is similar.

4--7

4.7 •
Illustrations of Contributions

from the Technology Base

to Specific System Developeýnt

To conclude the discussion of human factors R&D and tne

technology base, it is interesting to note that the value of the

research conducted in each cell of Exhibit 4-1 could be assessed

by the contribution that the research makes to specific system

develooments over the years. Again, no matter how interesting

it may be, measuring the value of this contribution is not within

the scope of this project. Nevertheless, It is meaningful to

qualitatively emphasize the relationship of human factors in the

4 technology base to system development products. To this end, we

will try to present a brief illustration for each cell in the

matrix. The illustrations which follow are keyed to the numbers

in the cells.

Block 1. Human related R&D in the Mission Analysis Phase:

measuring human tolerance to motion to assist ship design and

development.

deed. The Navy is investigating experimental designs of

surface effect ships (SES). A design constraint which requires

human factors R&D previous to any prototype development is

* determination of human stress tolerance to motion and associated

human performance capabilities while in mQderate and high sea

states.

Research. Research has begun on human tolerance to degrees

of SES motion, using a motion generator for simulation. In

addition, techniques for measuring complex human performance have

been developed to assess the ability of Navy personnel to perform

shipboard tasks during extended exposures to such motion.

Utilization. This human factors R&D will have direct impiications for the design of SES subsystems. It is also envisioned

to have an impact upon ship operations.

4-8
Block 2. Human-machine related R&D in the Mission Analysis

Phase verifying-warning system audibility.

Need. In the absence of quantitative data, the Air Force

was concerned with the adequacy of the current auditory warning

systems to offer advanced indication of a missile propulsion

system toxic propellant leak to operating personnel located in

missile silos. A portable vapor detector was used to detect such

leaks and to sound an auditory alarm. Concern was expressed over

the possibility that individuals working in silos with ambient

noise levels of 73 to 89 decibels might not be able to hear the

alarm. Consideration was being given to development of a new,

more elaborate warning system.

Research. In order to properly evaluate the requirement for

a new warning system, human factors R&D was needed to determine

the adequacy of the current system, before initiating development

of an essentially new, alternative system. Toward this end, a

field study of the actual system was performed utilizing operational personnel to report when they heard the alarm. Results

indicated that the personnel heard the alarm each time it was

sounded and made no false reports.

UtiZization. Consideration of possible new system development was abandoned. Cost estimates for development of the

alternative 'system for 500 silos ranged'from $250K to $l,000K.

Cost savings achieved through elimination of an unnecessary system

development program were due 'to a study costing aproximately

$1,000.

4-9
Block 3. Human-machine-mission related R&D in the Mission

Analysis Phase: Development of an automated command, control,

communication, and intelligence (C 31) system.

3 Need. In order to develop a computer-based C I system for

automated battlefield support; the A.rmy had a requirement for

human factors R&D to develop a data base covering mission related

human performance-and human-machine concepts. Due to unique

problems posed by such a system, data base concepts are required

which: (a) identify human user input and output capacities, and

(b) offer maximum real control over the battle.

Researck•. Simulations for automated C3 I were developed to

offer expected mission scenario simulation, identify the role of

man in automated C3 T systems, demonstrate operator task feasibility, and determine optimal design criteria for the man-machine

interface. Examples of results include:

* Guidance for input and display data

o Guidance for military terms abbreviation to reduce

workload and errors /

o Use of embedded-training

* Comparisons of data surmmary methods (e.g., graphic

displays)

e Tradeoff criteria between critical data retention and

expanded data retention

* Recommendations for data reduction and purging to

facilitate system performance.

UrtiZization. Data base information accrued from simulation

research on the automated C3I was furnished to Army developers to

aid in system development as concepts guidance and criteria for

automatedC 3 I systems. These systems are expected to facilitate

human ability' to govern c-nmbat on the ground.

4-10

_____ ____
Block 4. Human related R&D in the Concept Development

Phase: Visibility requirem3nts for underwater information

displays.

Need. Recent-advances in diving technology (e.g., freefloating manned submersible) have created a requirement for human

factors R&D on underwater vision related to display design.

These technology advances, tied to reduced visibility and lack of

a data base sufficient to guide underwater visual display design,

combine to threaten Navy diver mission success and life support.

Research. The research program that was initiated encompasses both display requirements and basic experiments on visual

performance underwater. Water turbidity simulation techniques

wer- developed to simulate harbor and oceanic waters. Since

numbex. reading and signal detection were identified as the

most critical Oisplay-oriented tasks, experiments based on

these pazameters were constructed that revealed the following:

brightness was the strongest legibility factor, green was more

legible than red, and only harbor turbidity had a major effect

on legibility.

UtiZization. This continuing line of research has resulted

in wide distribution of information to both research and fleet

operational communities, whose' responsibility it will be to

implement these findings into displays to be used in ambient

undersea environments such as instrumentation in neA and existing

diving systems. System-to-diver and diver-to-diver communication

will also be facilitated by these experimentally derived,

guidelines.

4-11
Block 5. .uxnan-machine related R&D in the Concept Development Phase: Selected problems- in armor operatioas and design.

.'.>il. A number of prcblen-ms with potential impacts on aror

operations and design have been identified Ly the Army'. Exaz.&les

are: concern with th•e effects of external envirormnental conditions

on the internal environrent of a buttoned-up tank, and concern

with the adequacy of current escape and evacuation systems.

-,-c,. in order to assess the environmental effects upon

tank woz'kspace, data on internal temperature and humidity were

obtained using a recording hygro-thermograph. "These data were

compared with comparable ,external conditions. Results showed

that temperature and relative humidity inside ai tank lag behind

the external conditiors by approximately three hours.

In addition, opinion data uere obtained from crewmen

concerning the adequacy of escape and evacuation systems and

potential design changes. One conclusion was that if a tank

wcre hit, the gunner will be the most vulnerable and would have

the greatest difficulty escaping. Also, revealed was that lifting

straps snould be auded to uniforms for evacuation of wounded,

and that eucape/evacuation training was extremely limited.

" ic"z-' n. This infdrmation could play a role in future

design o: ne.w tanks as well as in current training and operations.

Especially in t.-e case of tank crew escape/evacuation, design

recom',endatin,6s could have an impact.

4-A12
Block 6. Human-machine-mission related R&D in the Concept

Development Phase: Effects of operator interface on system

cost-effectiveness.

Need. The lack of availability of a rcliable technique to

aid in design tradeoff decisions at the man-machine interface of

system development has resulted in a Navy need for computer

models to simulate operator characteristics. In order to be

useful, such a model must be able to determine whether a proposed

or potentially modified system will result inma net gain in

effectiveness over cost, as well as to choose the most costeffective alternative means of achieving 'a specific performance

level, when the operator's job is considered.

Research. Human factors R&D has developed an Operator

Interface Cost Effectiveness Analysis model which is capable of

calculating the interactions between a human operator (including

control/display location, procedures, decisionmaking, observation,

recall, and physical movement), system hardware, and software,

as well as specific mission events. It has been applied to

evaluate alternate mission equipment configurations. For example,

two forward-locking infrared (FLIR) sensor configurations for the

P-3C surveillance aircraft were evaluated using this model.

Results showed that one configuration was superior due to:

* Less operator disruption in other tasks

* Less time to perform mission

. 25% less costly to operate.-

Utilization. As illustrated in the example provided, this

model promises substantially improved performance for Navy manned

systems as well as substantial cost savings by preventing'development of inferior hardware at the man-machine interfaces. It is

also being used in other system development programs for the Navy.

4-13
Block T. Human related R&D in the Demonstration and

Va•Idation ihase: Noise limits for Army materiel.

Xe•d. i-ntense sound accompanies many aspects of military

:cerations and training. Due to hearing loss which occurs as a

result of abusively loud noise and/or lack of effective hearing

protection devices, the services have a requirement to initiate

neaxýng protection and auditory research. This is especially so

fer desiin-related researqh with intent to reduce noise through

eviqn uidelines.

Feseazr2ý. The Army initiated research into noise effects,

limits, measurement, anti t ezting techniques as well as hearing

protecto6n. This began with an accumulation of existing noise

guid2,)incs and other information (such as industry standards

covering the topic) and continues today with research to fill

gaps in hearing technology. Continuing development of the auditory and noise data base has resulted in the initial and revised

publication of MIL-STD-1474, Noise Limits for Army Materiel.

Plans are currently underway to raise the standard to encompass

UUD-wide application.

S*t-Zzation. The use of this standard should result in

a signifIcant reduction in the present 40-50 million dollar

alnu•I expenJiture in hearing loss compen&ation paid by the VA

to military veterans. in addliion, this noise research has paid

off 4n'the dev.lopment uf a prototype high compliance idler for

tracked vehicles designed to, reduce noise emanating from.the

.axle and track location.

4-14
Block 8. Human-machine related R&D in the Demonstration and

Validation Phase: Human factors in redesign of a ground infantry

weapon system.

Need. The Army had a requirement for human factors R&D on

the DRAGON antitank missile system. Desired wenpon improvements

included means to increase target "hit rate," or accuracy, as well

as to make it more portable. Portability was important given

that-the weapon system is to be employed by ground infantry to

counter tank threats.

Reaearch. Human factors R&D was implemented to investigate

means to improve DRAGON missile system accuracy. These efforts

resulted in the redesign of portions of the weapon system. For

example, a lightweight tripod/viscously damped mount was developed

to replace the previous non-human engineered configuration. In

addition, redesign efforts resulted in a weapon system capable of

folding into a lightweight, compact package easily portable by

one individual.

Utilization. Field tests show that the human factors

redesigned DRAGON weapon system yields a 30% increase in hit eate

as compared to the previous design. The tripod modification

provides a nrecision tracking capability for a gunner that will

substantially improve his ability to 'hit distant moving targets.

Design improvements in portability and compactness will facilitate

transport of the weapon system..

4-15
Block 9. Human-machine-mission related R&D in the Demonstration and Validation Phase: V/STOL human factors planning.

Need. Navy interest in developing a viable V/STOZ aircraft

has resulted in a need for human factors R&D to support human

factors design of the aircraft. One reason for this is the

taxing workload demanded of pilots, creating a "pilot factor"

crucial in design and operation of this type of aircraft.

Contrary to expectation, the accident rate for this development

p:ogram was also increasing. Pilot factor contributed heavily

to this problem.

Research. In reaction to this state of affairs, the Navy

initiated a program to provide human factors R&D support to the

human factors design of the V/STOL aircraft. A major activity

has be~n the compilation of a data base of available documentation to support the program. The data base identified pilot

workload as a critical issue,. A "primer" was also developed

which introduced V/STOL technology and operation to human

factors personnel.

Utilization. This effort has contributed to the human

factors data base for V/STOL aircraft. In addition, human

factors personnel involved with V/STOL aircraft design were

Offered job performance aids. This line of research should

cont..ibute to an ultimate reduction in "pilot factor* accidents.

,4-16
Block 10. Human related R&D in the Full-Scale Development

Phase: Information display for landing signal officers (LSOs).

Need. A Landing Signal Officer (LSO),standing aboard an

aircraft carrier must guide a carrier pilot and aircraft into an

appropriate approach to a landing and then mak• a time-critical

decision (in seconds) as to the safety with which this rapidly

approaching aircraft may land on the carrier. These decisions

have, traditionally been based upon a limited assortment of visual

and auditory cues. In addition, approach and landing speeds are

high; and perceptual cue availability is adversely affected by

night and/or advers-e weather conditions. Due tc annual accident

rates associated with carrier landings, the Navy has a requirement to aid LSOs by developing supplemental information displays.

Research. Detailed investigations involving task analysis

were conducted of the requisite visual and auditory cues and

associated judgments made by LSOs when guiding the aircraft's

approach. The resulting information was incorporated into an

innovative, see-through, head-up display system.which provided

the LSO with these critical parameters, without interrupting his

visual tracking of the approaching aircraft.

UtiZization. Operational evaluations were conducted on the

display system by.actual LSOs. Results indicated overwhelming

approval by potential users. These users are confident that the

display system will facilitate safe and efficient landin• operations, thus reducing accidents, associated losses in equipment

and personnel, and reduced cbsts. The system is to enter

production and be fielded on all carriers.

4-1*7
Block 11. Human-machine related R&D in the Full-Scale

Development Phase: Man-machine integration technology.

Need. Due to advances in airframe, flight control, and

avionics technologies which promise to revolutionize aircraft

capabilities, the Air Force has a requirement for human factors

R&D to improve areas of man-machine integration technology such

as aircrew visibility. Additional areas concentrate around a

need to improve tactical aircraft cockpit, controls, and displays.

In addition, methods to measure pilot workload under different

configurations need to be identified.

Reee2rch. The teasibility of a voice activated switch of a

weapon systemi was demonstr-ated towards improvement of aircraft

cockpit controls and displays. These results led to a request

for voice controlled switching for the A-10 pilot during weapon,

delivery. In addition, an improved pilot/fire control interface

combining voice activated switching and he2met-mounted sight and

fire control status displays to facilitate continuous pilot outof-the-cockpit vision is being developed. Methods are being

developed to measure pilot workload under different cockpit

configurations. Part of this effort has produced a set of

symbols that present order-of-battle information to a pilot

rapidly and accurately. °

U'ti7-.zation. Improvement to aircraft cockpit, controls, and

displays and increased aata base development of pilot workload

capabilities-for various cockpit configurations has immediate

application as well ;s application to future systems.

4-18
Block 12. Human-machine-mission related R&D in the FullScale Development Phase: Human factors R&D support for FIREFINDER

radar.

Need. The Army has developed the FIREFINDER radar to pinpoint enemy indirect fire weapons. Mission suc--ess is contingent

upon a fast reaction time which makes human performance capability

at the man-machine interface critical. As a result, human factors

R&D was required to support system development.

Research. A task analysis for FIREFINDER was developed,

tested, and refined to keep up to date with system hardware

reconfigurations. Army human factors personnel coordinated with

FIREFINDER training simulator developers to support design of a

training effective simulation system. For excample, the simulator's

basic training effectiveness was confirmed using operational personnel in operational test and evaluation previous to delivery of

the final simulation system. Additional deficiencies were

identified and corrected at the contractor's plant.

Utilization. The FIREFINDER task analysis is in use by the

Army for developing operator/maintainer training courses. The

simulator now in use by the Army is a more effective trainer, with

the pre-identified deficiencies corrected-, than it might otherwise

have b)een.

4-19
References

Comroe, J.H., & Dripps, R.D. Top ten clinical advances. Contract

I-HO-I-2327. National Heart and Lung Institute,. 1975.

Erikson, R., Miles, J., & Secrist, G. Technology Coordinat--z:y

Paper: Training and personnel technology. Departl-'L. of

Defense, August 1978.

General Accounting Office. Human resources research and development results can be better managed. FPCD-77-43. April 1977.

4-20
CHAPTER 5

DERIVING METRICS FOR MEASURING THE VALUE OF

HUMAN FACTORS IN MILITARY SYSTEMS DEVELOPMENT

The previous chapter described a conceptual basis for

identifying the contribution of human factors in military

systems development. The major question to be addressed now

is: Can we measure that contribution? This question will be

answered by two successive discourses. The first, presented

-in this chapter, deals with metrics for describing human factors

value. The second, presented in the chapter to follow, develops

a methodology for measuring human factors value.

The objective of this chapter is to demonstrate that system

design and human factors criteria and terminology are compatible,

and that a vocabulary for human factors impact assessment can be

constructed from engineering and human factors by means of common

and complementary terms.

This chapter presents the results of a literature review

to compile terms useful for describing human factors R&D products

and impacts, compares those terms with conventional system

engineering and design terminology, and derives a preliminary

vocabulary to define human factors R&D impacts on military

systems.

5-1
Background

The importance of objective, quantitative data for decisionmaking in the system development process is apparent to anyone

involved. The presence of formal mathematical models or their

rear equivalent has permeated every level of system development,

from the engineering draftsman to the top levels of the Department

of Defense. New management techniques supported by quantitative

measurement have evolved rapidly over the past 40 years.

A parallel process has been occurring in the behavioral

sciences that undergird human factors applications. There has

been a consistent emphasis during the past 40 years on rigorous

measurement and, in effect, an attempt to emulate the physical

sciences with respect to precision.

While significant strides have been made (for example, in

scaling techniques) there is really no valid prospect that the

behavioral sciences will ever "catch up" to the physical sciences

in the matter of precision because of the inherent characteristics, such as high variability, in the phenomena of concern.

This circumstance generates a chronic problem for those

concerned with the contribution of human factors to military

systems development. In current parlance, the problem is the

synthesis of "soft" measures from the psycho-physiological-domain

of human factors with the "hard" measures presumably available to

systems engineers.

This study establishes a basis for an nperational synthesis

of those measures. The three essential components of this basis

are:

5-2

"a[
1. A formal relationship of explicit human factors products

with the system development process. Chapter 3 develops

the relationship of principal human factors products for

the major system development phases. The premise of the

relationship is that both human factors and systems

engineering are components of the system development

process.

2. A methodology with techniques that integrates "soft" and

"hard" measures. The methodology presented in Chapter 6

is designed for the formal treatment of quantitative and

qualitative impacts.

3. A set of metrics to describe human factor impacts and to

define parameters for the modeling process. This chapter

addresses the derivation of metrics for describing and

measuring human factors impacts on military systems

development.

5-3
4

Identifying and Defining Metrics

The discussion in previous chapters and in DOD documentation

leads to the assertion that any human factors or design engineering

change must ultimately be assessed in terms of its contribution

to the system-mission. At the system-mission level, three impact

areas have been identified: cost, capability, and compatibility.

These impact areas represent categories to aggregate or embed all

the effects on the military system of separate design and/or

operations support analyses. They are intended to represent the

"bottom-line" effects of system changes (or choices), and we use

them in this derivation of metrics as the most broad terms in a

measurement vocabulary hierarchy.

The three impact areas represent distinctive effects of a

change on the system-mission. However, the effect from a single

human factors or design engineering change will often be relatable

to several or all of the impact areas. Notionally, an improvement

in operator-system compatibility could enhance the mission-system

capability and also lead to fewer operator-induced repair actions,

thereby reducing repair costs.

The terms at the lowest level of our hierarchy represent

empirical measures. They are representative of the dependent

variables that human factors researchers have used for many years;

but such empirical measures are not always easily identifiable

with system effects. Therefore, it was decided that we need an

intermediate level in our measurement hierarchy that is acceptable

to both system developers and human factors professionals. The

intermediate level of terms is defined as metrics Exhibit 5-i

illustrates the three levels of our hierarchial m asurement

Vocabulary.

5-4

5 -I *
Exhillit 5-1

I icrarchical Relatiotiship oft" Impact Area-,,

Mletrics, and Empirical Measires

AGGREGATE

MISSIONSYSTEM LIFE CYCLE ICOMPATIB LITY CAPABILITY

IMPACT AREAS COST I

CREW RLALTYIAVAIL- USER

METRICS / ELI I IACCEPTANCE ABILITY e

EMPIRICAL TRAINING FAILURE TIME TO AMBIENT NOISE

MEASURES COSTS FREQUENCY REPAIR LEVEL I

SPECIFIC

The desirable characteristics of metrics have been determined

to be the following:

1. More specific than impact areas (i.e., the principal

focus or objective of an analysis).

2. Representative of the direct result of an analysis or

experiment.

3. Relevant to the human, factors principal products.

4. Compatible with the conventional terminology of system

engineering.

5. As mutually excluiive as possible.

5-5
Cnardcteristics I an- 2 imply that a micric is the output of a

•nadel ur experinn or some other anciysiv . In these cases, the

,ietri is az fcnction of some lower leel parameters or empirical

meýsures. Characteristic 3 implies that metrics must be relevant

to L,1e 41mn factors efforts. The metrics must be

I-nilaua to n.tan ractors researchers and practitioners and allow

for re~zist• interpretation of the c•ian:es under, consideration.

C:aracteristic 4 implies that measurements corL-ion to systems

engineering and numan tactors are to be represented by a single

ic. Thui Thus, systein engineering anl human factors considerations

are integrated within common metrics. This clearly has implicatioiis for the moceling of these integrating metrics, a subject

discussed eisettwnere in this report. Characteristic 5 is a

desirable property that,' if satisfied, implies that definitionaily

one metric does not ovcrlap with another. We recognize that this

characteri.t.ic is a particularly difficult one to satisfy, arid is

one that will only !e approximated by our preliminary vocabulary.

These characteristics were interpreted as selection criteria

in the derivatio-n of the preliminary vocabulary.

Literature Search and

Deta Base System

-We used an empirical approach to derive the metrics. It

entailed collating actual terminoloqies in the fields of human

factors research, human factors engineering, and systems research.

The literature review was followed by the development of a data

base tailored specifically to~human factors in military systems

development. Traditional publications for human factors research

provided a wealth of data on specific empirical measures and

metrics.. Computer printouts were obtained from the National

Technical Information Service (NTIS) and the Defense Technical

Information Center (DTIC). Documents containing extensive

5-L"
bibliograplnit-s wer,ý also :e~~for pertinent citations. In

additioii, an iiitensive effo-rt was ,inadt6 to identify- and obtain

pertinent governm~ent directives, instructions, standards, and

guidance _Jocx;xncnt6. This liter:ature review resulted in the

identification. of more than 350 relevant documrents. Subject

nkatter cateý.:r-es iK-.Cluded: h'ný_dnn factors engine~ ring; costs;

Mali~ary a-ysce.n Ilopmen t s/&cq u ;a*tions; test and evaluation;

-an-machi;.t studies; an~d syt~analysis, design, and develupment.

*The Pr~ducts oZ2 the 1literauure searches were used to

develop a tabs. The data base was the primary instrumnent

used to ar.1l,'zc: trne selected material and derive a set of

empirical mteasures and metrics. The following general docuxnentation Cse3~~swere acc-unulated for purposes of review

* and assessmient, and were put imW- the computer data base:

1. T,Žcni-.cal documzentatio:n- -including technical reports,

papers, memorandums, bibliographies, professional

jc-.r..al ai-ticles, an,,3 technical books.

2. Policy documentation--includi-ng Department of Defense

directives; instructions; pamphlets; military

specifications and 'standards; and individual service

instructions, regulations, and pamphlets.

2a.' Guidance documentation--including DOD and iiidividual service military handbooks, guidebooks,

manuals, and pamphlets.

3. Work Unit Summarics*(rDOD Form 1498).

4. iiiforinal d~ocum-entation-- including proceedings of va~rious

c~vc~.'fles nd meetings, unpublished literature, and

pcrsuc)nal testimoniies.
Screening and Selectior Process

The carlidate terms derived from the documents collected

.were sorted and grouped. The first screen applied was a frequency

of use check. Ail low frequency-count terms were reviewed for

usage; terms not exxplicitly described by the author were deleted.

The terms zcmaining after the first jcreening were then

defined using the following procedure:

1. Quantitati'e ameasures were defined according to (a) their

-it cf measure or dimension and (b) constraints (e.g.,

t ie period, events, cycles, etc.) and special circumstances for their use (if any), such as unique usages

(e.g.,. location).

2. QualitatIv". measures, subjectively determined, were

put through _n additional review. Checks were also

made to deter-mine if such measures had a quantitative

counterpart or could be, tied to some underlying dimension. Whenever characteristic constraints and special

circumstances for use of these measures were discussed

in the original document, this information was included

in the analysis.

3.' Following this procedure, the list was examined to

ensure that sufficient ihiformation was available about

each term to avoid ambiguity.

Derivation of Metrics

The iext step entailed the derivation of metrics. Criteria

reflecting the five metric characteristics were applied to the

list of' empiriLal measures. A partial list of candidate measures

5-8
and associated definitions is given in Exhibit 5-2 (the complete

list is in Appendix 5). A list of the derived metrics is given

in Exhibit 5-3. In each list measures and metrics common to

system engineering and human factors have been grouped as system

related, and those principally used in human factors grouped as

personnel related. The list of metrics is a preliminary one, and

in subsequent analyses and case studies it should be refined.

Each of the metrics is relatable to several empirical measures.

Exhibit 5-4 illustrates several of the functional relationships

observed in the literature reviewed.

Exhiilt 5-2

Samplc of Systrcm-Relat'd Terms and Associaled Dimnensions (Vnit of Nteasurc)

ACCESSIBILITY subjective: satisfactory/unsatisfactory ease of admission to various areas of a i item

ACCURACY probabil ity/frequency of documented error

CAPABILITY subjectivc: mission objective achievable given the

condition during the mission

COMPATIBILITY subjective: ability of items of equipment to

coexist (including effects of temperature and

moisture

CRITICALITY subjective: reintive degree of task importance fIo

mission succes

DURIAB;LITY probability: item will survive

"a) its projected fife

b) overnaul point

c) rebuild point

without a durability failure (failure that caulfs an item to be rebuilt or replaced

EASE OF USE subiective: tasks associated with simplicity, readability, etc.

FAILURE RATE/FREQUENCY 1) number of failed items

2) number of effects (out ot tolerances) per month,

5-9

- k ~ -
Exhibit 5-3

Prelimiinarv Me~trics Deried from tile Literaturv

SYSTEM-RELATED METRICS

AVAILABILITY

RELIABILITY

READINESS

DEPENDA81LITY

EFFECTIVENESS

MAINTAINABILITY

PERSONNEL ACCOMMODATION/ENHANCEME NT

DESIGN/PRODUCTION (PRODUCIBILITY)

SYSTEM RUGGEDNESS

OPERABILITY

PERSONNEL-RELATED METRICS

HUMAN PERFORMANCE'

SKILL, GENERAL

SKILL. MAINTENANCE

TASK LOADING

PHYSIOLOGYWPERCEPTICN

ENVIRONMENTAL FACTORS

OPERATIONS FACTORS

MOT IVATIONI/SOCIALIORGANI ZATION FACTORS

N6ote: This list ritprosients an initial attampt tc9 construct a let of metrics for

human factors and sys.orv engineering All the system-folsted metrics

ore common to engineering and human factors. and the personnelreitted metrics arei p~incipally related to human factors.

Because of the high saliency of the cost issue, the inherently,

quantitative propertie's of cost-assessment, and the relatively

low ambiguity associated withithe cost concept, the derivatioh'of

cost metrics and measures was more straightforward than for'the

other metrics.

-10
SapeR"JationslhiP SampleExhibit 5-4 Among Sejected SYsivin RCI~aed kletrie-s EmPiricj ea.,re alnd (As Foundr in tile Lit(.ratilre)

SYSTEM R ELATED

METRIC OEFI1 NED AS A FUNCTION OF EIMIIA

EIICDAL L 

E :MEASURES

M4EAN TIME ToREPI

MEAN TIME TO REPAIR (ACTUALJ

RELIA8 I~y OPERATIONAL)

ME-AN TIME ' UMSCHEOULTWEEN

MEAN TIME BETWEEN ANYMAINTENANCE

READNESSMEAN TIME

BETWEEN FAILURE

BETWEEN MAINEAC ACTION NEAC

EFFECIVENERE PAIR (FLIGONTUNE)

Ss 

SURVIVABILITY
The literature of system evaluation contains many detailed

cost structures. It was convenient to use the generic life cycle

cost structure developed by Fiorello and Betaque (1977). That

structure reflects current usage in the DSARC process, and is

presented in Exhibit 5-5.

Each of the impact areas is discussed in the next section.

Brief illustrations are given of the relationships among derived

empirical measures, metrics, and impact areas.

Exhibit 5-5

System Life Cyclc Cost Impact .Arca

for a Weapon System

SYSTEM LIFE CYCLE COSTS

100 RESEARCH AND DEVELOPMENT

"200 INVESTMENT

201 Weapon System Investment

202 Support Investment

300 OPERATING AND SUPPORT

301 Deployed Unit Operations 302 Below Depot Maintenance

303 Installations Support

304 Depot Maintenance

305 Depot Supply

306 Second Destination Transportation

307 Personnel Support and Training

308 Sustaining Investments

Source: Fsorello&'Uetaque. 1977.

1Ne1 : In the twminology used in thit MR0 ,rt life cycle cost Is the enPc seoe; the cost ca"Wories at the 100. 200, aWd 300

Wl00e11e aNOlOgoeu to cost metrics; &nd the Iome, leel oet

elanian we aenlogous to enmeirica meamares.

5-12

IL_ _
Impact Areas, Metrics, and Related Measures

Capability Impacts

The impact area called capability has a clear-cut tie to the

man-machine-mission performance of a new system.

There are many examples of the linkages between the measured

capability of the human and the ultimate performance of thesystem during the mission operations. Many different categories

of behavior have been examined in many different system contexts.

A particularly good illustration can be derived from the focus

on the human operator in a controlling or decision-making role in

command, control, communications, and intelligence (C 31) systems.

Specifically, the case involves radar signal processing. Radar

system functions (i.e., missions) include ground control of

tactical air strike operations, interceptor operations, and

return-to-base operations in adverse weather or poor visibility

conditions.

So-called "raw" radar returns are indiscriminate in the

sense that the indication of the position of one'aircraft on

the radar display looks just like-the-indication of every other

aircraft in the same coverage load. Under low traffic loads,

radar controllers are able to "keep track" of the identity of

the aircraft represented by a particular return "in their heads."

Under medium loads (approximately seven simultaneous ,"targets")

or higher,. the controller becomes prone to errors of identifi'-

cation, and system performance consequently-degrades rapidly..

During the early applications of radar, it was possible to

compensate for this capacity limitation by the use of manual plot

boards or plotting tables. The job of directing the movement of

interceptors was divided into three main requests. rhe radar

5-13
operator (1) reported "target" positions to a plotter (2) who

moved coded markers. The controller (3) made decisions based

on the representational display provided by the plotting board.

In the mid-1950s, it was conceived that it was possible

to put a computer between the radar signal and the operator.

The computer could be given the burden of "remembering" the

identity of a target and generating a display such that the

identity was reliably associated with position.. There followed

a major sequence of human factors contributions that: ýestablished precisely what the consequences were for overall system

effectiveness, specified how much information should be tied to

each target (e.g., altitude as well as identity), directed the

formatting and coding of the information, and defined backup

procedures in the event of a subsystem failure.

Empirical measurement in this case was carried out primarily

in the context of real-time simulation experiments. The specific

variables measured were the average delay in transit during

return-to-base operations and error counts, defined by the

instance of two' aircraft coming into a predetermined proximity

relationship.

The metric level in this case would be represented primarily

by effectiveness and operability. These, in turn, would feed

into the impact area of capability.

Cost Impacts

At the mission-systems level, the cost impact area is

defined by the life cycle cost of the system; in other words,

the total cumulative cost of bringing a system into being arid

using it over its operational life.

5-14ai "H"I i
Human factors engineering typically accounts for a small

fraction of the cumulative cost. The outlay of funds is used

to support the generation of human factors products, during

the design stages of system development, and for specialist

participation in test and evaluation.

The specific engineering inputs from human factors sources

may impact the development and production costs (both positively

and negatively). For example, it is possible that the layout-of

displays and the controls on an instrument panel that is optimum

from a human factors point of view can be more or less difficult

to fabricate during production or could require more or less

expensive components. However, it is actually more likely that

what turns out to be optimum from a hiuan factors viewpoint is

also optimum with respect to eccnomy of production. Moreover,

the possible additions to cost tend to be relatively insignificant when compared to some of the cost avoidance potential

inherent in successful human factors contributio.,s.

Any number of actual or hypothetical cases could be cited in

which human factors considerations contributed to a significant

reduction in life cycle costs. For example, the size of the

operational crew for any given weapon system is a question that

links human factors considerations to economic consequences. The

larger the crew, the higher the life-cycle costs will be; but an

arbitrarily small crew might not be able to handle peak work,

loads during crucial mission stages. Work load capacity limitations are the kind of specific products that can be generated

in a rigorous manner through the proper application of human

factors procedures. Such estimates should not be made by ruleof-thumb or guesswork.

5-15
The quality of performnance (inverse of error frequency)

on the part of crew members in different crew size and organizational arrangements can be measured empirically. The context

of measurement can vary from rough task simulation in laboratory

settings to observations in the field of actual operations.

The results of such observations can contribute to the overall

system design deliberations in several areas, but in the present

framework the critical linkage is to cost factor 301, costs

associated with deployed unit operations (see Exhibit 5-5).

Once that linkage has been made, the logic of aggregation to

overall life cycle cost is.straightforward.

The final point to be made is that the cost impact could be

forecast with little ambiguity once the crew size decision has

been made.

Compatibility Impacts

Compatibility is, in general, the most complicated of the

three impact areas. It is complicated because there are three

distinct but interrelated facets involved: physical, physiological, and psychological compatibility.

Physical Compatibility. Physical compatibility relates to

the human component as a physical object. The major resource for

the conduct of tests and analyses of physical compatibility is

anthropometric measurements. An example can be drawn from the

design of the cockpit ejection subsystem for a high-performance

jet interceptor aircraft. The problems involved in such a

design are many. The routine for ativation must be simple

because the pilot is under severr distracting stress when

activation is required. The ejection module must clear the

aircraft in such a way th, z there is no impact with the aircraft

structure and must stabilize so that parachute deployment is not

impaired.

i'+ +5-1,
/ °

/'

As was indicated in Chapter 2, cockpit size is shrinking as

a general trend. One ejection subsystem design recently proposed

met all the complex requirements except one. The trajectory of

the module was such that as the pilot cleared the cockpit the

first few inches of the toe portion of the flying boot came in

contact with the edge of the instrument panel. The impulse force

was sufficient to shear off the boot tip and the pilot's toes

with it.

In this case, as in others, it requires no elaborate computational model to conclude that a better, more comprehensive

application of anthropometric data would improve the design of

this subsystem. The consequences for system performance are

similarly clear.

Physiological Compatibility. It is convenient to use the

discussion of physiological compatibility to make the distinction

between capability and compatibility a bit clearer. This

distinction can be accomplished by considering the human factors

design parameters, of an armored personnel carrier. In such an

instance, it is useful to temporarily separate the functions of

the operator from those of the rider or passenger.

The mission of the system is to deliver the passengers in

an unimpaired condition 'to some locale under hostile conditions.

The key attribute of the passengers is that they are not in

control during the journey. ,In effect, they are passive cargo.

But there are many ways in which the design of the system can

be either compatible or incompatible with their physiological

characteristics.- For example, when traversing hostile territory

in a combat situation, the vehicle, will be "buttoned-upt " in the

sense that all hatches and doors will be closed. Such a closed

environment creates a potential limitation of effective fresh air

5-17
A.-,-

circulation. The vehicle itself is a potential source of toxic

gaseous air contaminants (e.g., carbon monoxide)' Hostile

action can generate other toxic airborne chemicals. In such a

case, the human factors engineering role would be to inventory

all possible sources of toxic contamination. The analysi.s would

include a full range of adverse circumstances that could arise

from the terrain, weather, and hostile action.

It is conceivable, but unlikely, that such a design review

would indicate a zero hazard potential in the "buttoned-up" mode.

If the hazard potential were present, then certain design options

such as compressor-powered ventilation systems could be evaluated.

That is, the cost and complexity of the ventilation process could

be weighed against the estimated probability of a hazard arising

in operations, the intensity of such a hazard, and the consequences

to the mission if the hazard were not eliminated.

During the design and prototype stages of system development,

the empirical level measures would come "from the book" in the

sense that the human vulnerabilities to toxic compounds are well

known and documented. Otber parametric considerations such as

the likelihood of inflammation of hydraulic fluid due to mishap

or enemy action would have to come from other members of the'

design team. During field trials of prototype models, however,

a reasonable empirical test could involve the actual sampling

and chemical analysis of the air in the interior of the vehicle

during combat exercises.

Finally, an analogous approach could be taken to a list

of other potential physiological hazards:, temperature extremes,

noise, vibration, and acceleration stresses. While such a

thorough analytic and empirical review might be considered

burdensome, the potential negative consequences of the delivery

of exhausted, disoriented, or incapacitated troops into a fire

zone are clearly worth the trouble to tvoid.

5-18
Psychologicai Comnpatibility. Psycholog.ical compatibility

has two component parts: behavioral and attitudinal. The question of behavioral compatibility can be illustrated easily by

an example of the use of technical documentation by maintenance

technicians. From a system development standpoint, the initial

empirical data needed would be in the form of a distribution

function of the reading skill levels. of the population of

assignable technicians.

The rather obvious human factors recommendation would be

to match the difficulty level, and in particular the vocabulary

of technical documents, to the relevant skill level of the user

population (i.e., the tenth percentile or the second stanine

level and above). During early system development, design

criteria could be met by the imposition of a "control" vocabulary

on the preparers of the technical documents. During later stages'

(validation and verification), empirical tests of the readability

could be conducted using representative technicians from the

prospective user population.

Increasing the compatikbilit'ý of technical documentation

can result in substantive imprpvements in weapon system cost

and availability. An example is the experience of the avionics

maintenance improvement team with the avionics suite on the

F-lllD. Re-test-OKs were averagiing over 44% and Wereereduced

by 15% due to improving the technical documentation.

Attitudinal compatibility is somewhat harder to illustrate,

and it is a matter of some dispute whether or not even S'trong

negative attitudes on the part of a system operator can be a

significant problem when that operator is under military discipline. However, in the recorded history of the interaction of

military personnel and their equipment there have been instances

'5-19
wherein system performance suffered because of the users' negative

attitudes. A modern example of user acceptance, or attitudinal

compatibility, can be drawn from an article that appeared in the

Armed Forces Jozirna! (April 1978). Because of its brevity, an

exact excerpt of the article is included below.

ARMY'S PIERRE: "FIX BAYONETS!"

DOESN'T COME ACROSS ON THE COMPUTER

It is easy, in tossing around the acronyms and

technical terms associated with developments in

command, control, and communications to lose sight

of their purpose: to help human beings perform

their tasks. Dr. Percy Pierre of the Army puts it

this way:

'We seem to have disguised equipment that

performs reasonably understandable functions

behind a variety of unpronouncable acronyms

and names (like digital multiplexer) that

are only meaningful to those familiar with

the echnology.'

Pierre, like so many others consulted for this

issue, cautions that voice transmissions will still

be required on the battlefield, no matter how far

into the future one projects, or how extensively

commands and information are converted into digital

forms. He told Congress recently, for instance,

that "Follow Me" or "Fix Bayonets" are commands

that do not convey the same impact when received

in a computer printout.

The human's desire to hear another human voice in a

time of crisis held back deployment of an advanced,

digital device a few years back. According to experts

at one of the leading electronics companies, their

engineers had developed a tiny two-way radio device

that could be used at rifle company and platoon level

to send and receive messages in formatted form. Su.- if a leader wanted to call for artillery fire mission,

he just punched in a code and the device sent it.

He received acknowledgement with a beep signal.

The device worked well, but it failed in field tests.

The troops didn't want a beep when they called for.

help--'.hey wanted to request the fire mission by

voice in order to express their urgency, and they

wanted a calm human voice telling them that the

rounds were on the way, not a beep. They distrusted

the beep, since it could have been caused by a

malfunction in the system.

5-20
Summvary

The principal message in all of the preceding examples has

been that it is possible to proceed bot~h upwards and downwards on

the "ladder" of measurement aggregation. Each type of military

system will involve different spscific measures and different

patterns of linkage from one level to another. However, it is

* demonstrable that the 'Linkage can be made. The analysis can be

focused at the level of discourse employed by syst ems designers

and program managers to assess the systems upon which they are

working. The procedure by which specific evaluation of the human

factors contribution can be made is discussed in the next chapter.

I 5-21
R'•ferences

Armed Forceas Journal International. Army's Pierre: "Fix

bayonets!" doesn't come across on the computer. Vol. 115(8),

April 1978.

Fiorello, M., & Betaque, N. Aircraft system operating and

support costs: Guidelines for analysis. Logistics

Management Institute, Washington, D.C., February 1977.

5-22
CHAPTER 6

METHODOLOGY FOR EVALUATING HUMAN FACTORS IMPACTS

IN MILITARY SYSTEMS fF'!ELOPMLNT

The objectives of this chapter are to provide-

* A systematic framework or methodology tl.at can be used

to measure the value of human factors in the development

of military Vystems

o A classification for different techniques or models that

can be used, within the above framework, to estimate or

reflect human factors related impacts on military systems.

The first section of this chapter., "Human Factors Impact

Assessment: Related Concepts, Limitations, and Considerations,"

provides a brief overview of the underlying principles and methodologies used to formulate the recommended framework. We utilize.

the concepts of cost-benefit analysis and policy/impact assessment

in our framework and attempt to tailor those methodologies to the

human factors setting. The second section, "Human Factors Impact

Assessment: Conceptual Framework," presents the set of steps

that comprise the recommended methodology. Each of the steps is

described, and, in several of the steps a notional, human factors

related problem is used as an illustration of the process. The

third section, "Techniques for Estimating Impacts: Classification

and Selection," discusses the step in the methodology that selects

* and adopts the techniques for estimating the associated costs and

impacts of the human factors related recommendations in military

system development. A basicclassification of .techniquos or models

is presented that can serve as a basi- for identifying their

characteristics, thus facilitating the selection/development of

the most effective model form for the particular human factors

impact under consideration.

S~6-1
Human Factors Impact Assessment:

Related Concepts, Limitations,

and Considerations

Our focus is on the assessment of human factors applications

in the context of military systems development. Throughout this

multi-phased development, there are opportunities for applying

human factors. The majcr areas of emphasis of thu human factors

R&D products were discussed in previous chapters and 'listed in

Exhibit 4-1. During thcse "windows"'of opportunity, one or

several human factors related actions can be considered. For

each of the candidate actions, or for a set of them in which the

individual actions are not separable, the quantitative and qualitative impacts must be estimated. The human factors quantitative

and qualitative impacts can then be compared to the estimated

impacts of other non-human factor actions (e.g., reliability

growth) to assist the decisionmaker in allocating resources.

The underlying concepts used to formulate this human factors

impact assessment methodology are cost-benefit analysis and

sjstem impact assessment, and their conceptual source, systems

analysis.

Cost-benefit analysis is ,a form of systems analysis. It is

a method for deriving relevant information about the desirable and'

undesirable effects, of projects or alternative actions under consideration. The approach is, in general, analytical; it entails

specifying objectives and alternative solutions and selecting the

preferred alternative based upon its relative cost-benefit rating.*

*Excellent discussions on the theory of cost-benefit analysis

can be found in Anderson and Settle (1977), Fisher, (1971), Quade (1975), Mischin (1976), and Prest and Tuvey (1965). Quade (1975) provides a 'very readable overview of the evolution of systems analysis techniques, including cost-effectiveness analysis followed by cost-benefit analyzis followed by impact assessment

analysis.

6-2
In theory, this is just what needs to be done with respect to

evaluating the contribution of human factors to system development.

A basic premise of classical cost-benefit theory is that all

costs and benefits are expressed in monetary units. In many

applications this requirement does not present difficulties.

However, in the human factors setting the process of interpreting

all the metrics and their impacts into dollars would often have

to be done in an arbitrary manner, and that could distort the

analysis. Presently there is no foolproof way to treat intangible,

distributed impacts in a strict cost-benefit monetary forecast

framework. In general, a strict cost-benefit approach for human

factors analysis will not be practicable. What appears more

feasible and useful is an extension of the cost-benefit approach

known as impact assessment.

For these and other reasons (e.g., inability to isolate

individual impacts), it may not be feasible to establish an

unambiguous ordinal ranking--let alone a cardinal ranking--of the

alternative actions in strictly dollar terms.. What is needed is

to extend the cost-1,enefit monetary metrics by presenting the

additional relevant *.npacts and metrics, such ai user acceptance,

in their natural (non-monetary) dimensions.* Techniques such as

system impact assessment do exactly that. The impact azsessment

matrix representation is a systematic array of the information,

including non-monetary measures, and useful comparisons can be:

made. A simple illustration of an impact assessment matrix is

shown in Exhibit 6-1. We note that in any instance in which the

qualitative metrics are non-discriminatory and the quantitative

metrics can be expressed in dollars, the impact assessment method

reduces to the classical cost-benefit framework.

*A good introduction to System impact assessment is provided by

Goeller (1976).

6-3
> _ u

00 <

t 50

0~

I. >

II

-( Z -> 0

-75

>2

1-44c - <l

0 4

z

0 w

lO!

u 5;:

6.-4
On the other hand, where the quantitative mietrics are

non-discriminiftory, dependence on the qualitative metrics is

necessary. Where this is Lhe case, various techniques can be

used to facilitate the comparisons. One of the simplest is to

use a color scoreboara approach in which the relative rdnkings

of the candidate actions are indicated by colors for each qualitative metric or measure.* Usually four colors are used (green

for best, blue for next best, red for worst, and yellow for all

the others). More complex techniques assign relative weights

reflecting importance to each qualitative metric or measure,

utilize ordinal scales for representing the relative ranking

of the actions, norrvalize the ordinal scales, and translate

the products of the relative weights of the metrics and their

normalized rankings into a quantitative rating. These and other

relevant techniques that can b,2 used to estimate the value of the

metrics in the cells of the matrix in Exhibit 6-1 are discussed

later in this chapter.

Methodological and Data Limitations

The impact assessment approach avoids a fundamental

limitation associated Aith a strict cost-benefit approach.

However, there are still a number of important limitations and

considerations that must be dealt with when attempting to assess

the potential impacts that could result from a human factors

related application. These limitations and considerations are

discussed next.

*The use of color acts as a reminder that the scale is ordinal

at best, and helps prevent the natural tendency to overlook the

limitations of the data. See Goeller (1976) for illustrations

of this approach for transportation system decisions. The color

scoreboard can easily incorporate the quantitative measures as

well.

(i-5
Isolating Human 1'a,_tors impacts. It is voriy di.fficult--

and irrcquently Aimp'ossible--to accurately isolate the individual

impaccýs fron, aggreg'aLed impacts when the humir, factor impacts

Snot. independent c:ýf ona another, oc when the- individual contributions to the overall, agqcegated impacts are not uniquely

measurable.

in such instances, aqgregating all the concurrent hiuman

f.Actors relatea actionis is called for . When, tl.,: imnpacts from

the aggregated human factor acti..ns cannot be distinguishod

accurately from the imrpacts of the non-human factors ac.tions,

ai. approxlimatc attribution of the total negative and positive

aTr~nacts on the milit,..ry system to the contributing, actions is

iecessary. A conceptu.Al Lasis for such attriL-ution-s can be

found, for example, Ain Saaty (1979) and Ostrofsky (1977).

Utilizing Solphisticated Techniques. Many of the models

that can incorporate intangipI e :mpacts are complex, difficult

to canti not. struiglitforward to understand. When a complex

proc).edutre is ri.-eaJ -:o assess the causal relationship(s) between

an action and an impdct,.it will often be necesaary to em'ploy

0;.auytizal specialists to apply the technique and interpret the

reý;ults. The resources necdcd to do tbe analysis are part of

tl.e- cos.--inipact assessment decisions.

Cowrpontint v, vsten Ipacts. Oftens the focus of the human'

fact-ors P.&D a 'ctivi1ty will tic on an individdal procedure or compon-z-nt and not -an entite systemi. When the proceditre or component

is chenged, as 4i consequence Xf the human factors related actions,,

'the impact should be rý.lated to-the system's mtission capability,

cost, or compatioility. It is often difficult to relate the

resullts of an analysis of a part to the whole. I .n many such

instancL-s, an opportunity cost argument for the !"freed" resources

o.L improved capability is the most appropriate explanation of the

iupact.
Tracki n Impacts from Phase to Phase. The conceptual

process envisaged cdlls for the consideration and assessment

of hu:man fLctors impacts throughout the development phases of

a military systtrm. There are officidal'- tour development phases

(as per OMB Circular A-109) including milestone 3--the production

decision. Each phase represents a window of onportunity for

human factors related actions. The impact assessment framework

is intended to be applied to the candidate actions within each

phase. In keeping with the baseline concept used in cost-benefit

analysis, the projected impacts are evaluated relative to a

specified baseline. When a design or procedure is changed,

the baseline for subsequent impact assessments is also changed.

Consequently, the baseline will be continually updated as changes

are introduced over the system development phases. Thus, an

impact forecasted in one phase will not necessarily be additive

with impacts forecasted (claimed) in earlier or subsequent phases.

Impacts forecasted should be presented and documented relative to

the baseline for the phase in which they are generated, and not

casually aggregated across ph.ases.

Differentiating R&D Funding Impacts. In general; it will

not be apparent how to relate in a quantitative and precise

manner the different R&D categories used*.to, fund human faclors

analyses to differences in the resulting impacts on the system

design. To the extent that the R&D budget categories and bhe

type of R&D activity are defined and applied in a consistent

manner, then a degree of differentiation will be feasible.

System vs. Non-System Specific Impacts. In general, it will

not be apparent how to estimate in a rigorous way the impa ts of

Is:--n factors research beyond a specific weapon systemt setling--

that is, to classes of equipment, or general military proc dures.

This is particularly true for "basic" research. (Note: TVis

6-7
1A 5 -1

problem~ could be an artifact of the budgeting procedures used in

DOD. A di.;tinctior. between human factors research (which is nonSys temn S;Iec-ific) and human factors engineerinq (whic' is system

specit ic) mijti.ewove the problem altoce'ther.)

1"X~ta L_;imiati~cnis. Two recent studies, Butler (1979) and:

Orlansky (1979) have observed thalt there i~s a lack of sufficient

J. togen~eous, longitudinal data to properly formulate, measure

and alidate the analYsis o-f impacts. (whether humani fact-ors

based or not) on military personnel performance an"' training..

This lizmitation raises th2 issue of feasibility for any approach

t~hat requircs extensive data or tb-at does not generate, store,

and ;Pveasu,'tý the reqiuirad data as part of its analytic design.

Risk i~rl Uncertainty. In goneral, the treatment of risk

and uncortainty in mnodels tiiat assec.s impacts is nott adequate.,

?rocedures do exist. IL. quantify and incorporate risk i_:1 cost

and cniLprojt,-c-ti-_ns. Sor±. for ex.mle Fihe (973) ,Beer.,

(195i ,Sobel (1965) , Murphy (1970) , arid Diene.emann, (1966).

_knpower Poi '. nalytic techniques tend to mask the

* riniita.ry mtarpowcer policy effects on candidate design changes

g enerated by R&D results or design variations. In general, a

si~mulati-or mcdel i '; re-quired to incorporate the impact changes

and marnDower policy requ.Lreme'nts in a consistent framework.

Suc. u1.) r of ten not applicable until the later stages of

the 5ýztcll cevelopm~eit. process.

Corvsicexations for 11om.an Factors Practitioners

An imnportant step in~ thc deve~lopmtent of ai methodology is

to determine the set of chatracteristics Ita potential users,

*desire for it. We have icie.ntiftled thrne basic user requirements

6-8
that must be accommodated: compatibility with the system design

process, compatibility with the human factors R&D process, and

practicality of the tools.

System Design/Development Context. To be effective, the'

human factor impact assessment process must be wompatible with

the system design and development process. The methodology must

facilitate the embedding of the human factors contributions and

products into the system design characteristics, as described in

Chapter 3, and into the system cost-performance measures.

A number of operational design tools used in cost-availability

analyses (see, for example, Forster, 1974;' Baran & Goclowski,

i978; Fabbro & Fiorello, 1977; and Fiorello & Betague, 1977)

provide frameworks to relate system parameters, such as reliability

and maintainability, to cost and performance in a causal manner.

Those tools also enable the design changes under consideration to

be ranked in terms of their contribution to cost reduction and/or

capability enhancement. Such tools provide a useful design

perspective that is relevant and necessary for the assessment of

human factors R&D in system design.

In addition to the design context, the methodology must be

compatible with the changing focus and characterization of the

design activity, as the design progresses in its development from

the conceptual through the operational stages. In the earliest

stages, broad macro issues are pertinent. In the latter stages,

more detailed equipment or micro issues are pertinent.

Compatibility with Human Factors R&D Issues. To be useful

in the human factor R&D process, the methodology must be relevant

and suitable to'that process. It must utilize familiar parameters

and allow for realistic interpretation of the impacts. Further,

6-9
specific tools must be capable of handling the multi•-objective

and multi-constraint settings typical of the hiuan factor and

manpower setting. This consideration, when, fully developed into

an operational screening criterion, will be useful to ideittify

andselect compatible models.

Praginatic Procedures. in addition to being compatible

with the system design/development process and the human factor

R&D perspective, the methodology must also be practical to use.

The trend in contemporary system analysis is to portray evaluations. in as rich a formulation as possible (see Quade, 1975;

and Goeller, 1976) to provide the decisionmaker with a full set

of relevant information. This is done by retaining the important

natural, multi-dimensional impacts and presenting them in the

evaluation. The advantage of this approach is that changes to

the system can be exa'.*ned not only in monetary terms but also

in terms of their other non-monetary impacts. The major drawback

is that these procedures are becoming more and more complex, and

they tend to require many explicit judgments for the various

multiple impacts. Ostrofsky's (1977) design morphology is

certainly a case in point, in that it provides a systematic

portrayal of pertinent design issues; it can accommodate human

factor parameters (through appropriate surrogates), but only at

the expense of extensive detail and complex computations.

As model complexity increases, so do the skill requirements,

the difficulty of interpretation, and the efforts to validate and

gain acceptability for the methods.

Also, a pragmatic methodology should be flexible enough to

use on a breadth of human factor related projects. It should be'

readily available and reasonably convenient to apply, have low

6-10
setup costs, and not require unrealistic data inputs. The intent

is to develop these requirements into discrete criteria for use

in the selection and application of candidate models.

Lastly, the methodology should help structure the relationship

between the human factors practitioner and decisionmakers--

particularly Program Managers. For example, anecdotal reports

suggest that there have been incidents in which disagreements

have-centered on such issues as whether a design decision could

be made on the basis of existing principles or some relatively

informal tests, as opposed to a series of'relatively elaborate

experiments. A good methodology would be one which helped the

parties to such divergent views reach an agreement without rancor.

In other words, the methodology would be perceived by all parties

of interest as providing a fair test of the time and resource

investment options in a.given design decision situation.

Rigor vs. Broad-Based Analysis

A fundamental issue, underlying many of the above points,

is whether the analysis should be relevant, rigorous, and

statistically complete, or primarily'relevant (descriptive and

broad). A rigcrous evaluation requires (a) formal problem

statements, (b) definition of the analysis and testing process

within a communicable model framework, tc) the capacity for'

replication by different analysts at different times, (d) evaluation designs dependent upon the use or availability of baseline

or control groups, and (e) that the number of observations and

the number of model relationships are both greater than the

number of test characteristics or variables of interest.

The notion of broad, relevant studies i's used here to imply

a broad-based analysis where the intent is to describe what has

taken place or is expected. to identify the predominant issues

6-11
in a certain setting, and to incorporate them. Many relevant

variables cannot be measured in a rigorous, quantifiable manner

(for example, user acceptance and vw.riations in skill-mix).

This dichotomy, although somewhat contrived, is pertinent

to the definition of the cost-benefit or impact analyses. This

is so because not all human factors issues or parameters can be

analyzed in a rigorous manner. This limitation on rigorous

analysis must be dealt with explicitly in a tradeoff decision

during the formulation step of the anaiysis.

Dealing with the above limitations in itself requires

management and analysis resources. It is important to recognize

what the relative estimated costs are of the evaluations and

the impacts. If the costs of the analyses are comparable to

the expected value of the impacts, then it is likely that, the

analysis as defined is inappropriate.

6-1
Human Factors Impact Assessment:

Conceptual Framework

Basic Framework and Steps

Exhibit 6-2 outlines the basic impact assessment framework.

The development and presentation of the analysit entails ten

steps or phases. The steps are presented in a logical sequence,

in three groups; but in any one analysis, as indicated by the

dotted lines, it may be necessary to repeat several steps in

different sequences to refine perceptions And assessments of

critical issues. Each step will be discussed in some detail

below.

1. Establishing the Problem,' Goals, and Criteria. The

objective of this step is to isolate the specific issues to

be analyzed, to bound the requirements, to specify the specific

goals and objectives, and to derive'the decision criteria.*

Fisher (1971), Quade (1975), and Goeller (1976). provide useful,

generic guidance for this step. Specifically, this step'defines

the content and purpose of the human factors product to be

developed. The principal human factors products are listed in

Exhibit 3-1, earlier, and Exhibit 6-6 at the end of this chapter.

This step is one that should be recognized as a variation

on the generic system analysis method. The rule is: look at

the ends first and work back from those ends. Ip human factors

terminology, we would'probably prefer the sequence: Goals,

Objectives, and Outcome Measures (or, for the latter, Dependent

Variables). However, the principle of going from the broad to

the narrow and the idea of a hierarchy that includes more

*In this discussion, the term goal represents an "end," objective

a "means" (that is, a specific accomplishment within an explicit

time or cost target), and criteria represent specific decision

conditions.

6-13

| Ii
Exhibit 6-2

Impact Assessment Framework

FORMULATING THE IMPACT ANALYSIS

1. Establishing the Problem, Goals, and Criteria

2. Defining the Alternative Solutions

3. Specifying the Baseline

4. Preparing the System Definition Statement

5. Selecting the Impact Areas, Metrics, and Performance Measures

! CONDUCTING THE ANALYSIS

I 6. Selecting/Constructing the Impact Assessment Model(s)

I 7. Collecting and Processing the Data

' 8. Setting the Conventions for the Analysis

9. Estimating and Evaluating the Impacts

PREPARING AND INTERPRETING THE, RESULTS

10. Pre-x.nting the Results, Associated Uncertainties, and Bounding Conditions

6-14

,.,. -- ~ -. - * - -\ - - -. ~-. .'.-a.- .
"variables" as that move is made is a common link. This

arrangement is illustrated in Exhibit 6-3 in a particular

cost-benefit impact assess.ent convention (Goeller, 1976;

Ustrofsky, 1977).

The following hypothetical example illustrates the use of

goals, objectives, and weights. Assume that the system under

consideration is proposed for the XYZ main battle tank. The

major goal is to achieve an armored fighting unit that could

defeat its hostile counterpart in certain tactical scenarios.

The objective, 011 could be that the frontal armor would hold

against 80% of main round hits (i.e., any grazing angle greater

than ±5o). The objective, 02, could be to achieve an average

first-round time advantage of 3 seconds. In this case, 02 could

receive an a priori value weight somewhat higher than 0,.

Criterion C2 1 (contributing to objective 02) could be a

maximum turret traverse rate of =>20 0 per second. Criterion C2 2

could be a maximum elevation/depression rate of =>450 a second.

In this case the criteria might be assigned equivalent value

weights.

Several attributes of the hierarchical setup should now

be' clearer. Specifically, as one moves down the structure,

the objective measurability improves. But more importantly,

the actual assumptions about performance are made very explicit.

That,'is, the design assumption clearly is that if a given

elevation/depression rate and a given• traverse rate are achieved,

a given first round time advantage will result. Not only is that

assumption measurable (e.g., by computer simulation), but the

tentative weight assignment is also similarly measurable.

Computer simulation would permit a whole range of permutations on

the traverse rates and elevation/depression rates to be explored,

6-15
Exhibit 6-3

Goal Relevance Tree Hierarchy of

Goals - Objectives - Criteria

Go (Wo) MAJOR GOAL

I I

O(WI) O(W 2 .' Om (WM) OBJECTIVES

C21 (W%1 ) CzM (W2) CRITERIA

RELATIVE WEIGHTS:

TOTAL VALUE - WO

OBJECTIVE WORTH - W1

CRITERIAWORTH = W11

6-16,

,.
and a very close approximation of the relative importance of

one to the other could be obtained. Moreover, the weights

could be revised as other kinds of testing were done.

A notable gap in the above synthetic scenario is the

lack. of explicit consideration of the human factors aspects

of sighting and firing the main armament. Fcr example, human

factors questions would arise about the compatibility of a

maximum 800 traverse rate with the human factors requircment

(hypothetical) to lock-on to a target on the first traverse

with no waver. Ruman factors engineering solutions based on

traverse deceleration rate damping, sight reticule size, etc.,

would need to be fitted into the goal and objective-attainment

relationship as constraints. The basic message here is that

it-might not pay to have a relatively high traverse rate, if it

led to an overswing of the turret 9 times out of 10 because the

rate/velocity dynamics were incompatible with normal human

(psychomotor) tracking capabilities.

The characteristics of the appropriate set of goals,

objectives, and criteria is critical to the effectiveness of

the analysis. Several useful discussions on this process are

provided by Fisher (1971), Quade (1975), and Ostrofsky (1977).

The'input-output matrix technique used by Ostrofsky (1977)

appears' to be a particularly useful way to structure this step.

An illustration of the matrix is shown in Exhibit G-4. The row

headings define the user and thesystem major phases, and the

column headings define the requirements and bounding or constraining conditions (e.g., resources). The row headings 'used

in Exhibit 4-1 could also be used. Ostrofsky has used this

format to formally incorporate human factors considerations

into the system design proces.

6-.17'
Exhibit 6-4

hnput-Citput Matrix for Problem Formulation

"Major System Inputs Outputs

Development Phases Intended Environ.mental Desired Undesired

Mission Analysis

Concept Development

Demonstrat~ons

and Validation

Full-Scaie Development

Production

(Swrue: Ostrofskv, 1977)

6-18
The output from this step is a problem statement, an inputoutput matrix for bounding the design-analysis problem, and a set

of weighted objectives and decision criteria to be used. The

problem statement is an issue that one or more human factors

related actions can help to resolve.

2. Definin the Alternative Solutions. The objective of

this step is to generate a set of explicit strategies or alternative solutions to resolve the problem or issue identified in

Step I. For example, within the human factors principal R&D

product--development of the role of man as a part of the

mission--alternative crew sizes, mission flexibility, and system

recoverability could be specific considerations. There are two

major ways this can be done. The first is to specify a set of

alternative design configurations or characteristics or process

changes at the subsystem, component, or function -level. The

second is to specify a criterion function (see Ostrofsky, 1977)

that incorporates the design parameters in a mathematical

function, and to exercise the function to determine the preferred

design or system specification. Either approach can be used.

The former is more comnmon and straightforward. The latter is

typically more rigorous and requires more definitive analysis.

Making the decision options explicit is a fundamental

Drinciple of systems analysis. We can illustrate this principle

Ln the context of using cost-benefit impact analysis to measure

the impact of human factors'.

Methodologies such as cost-benefit analysis are being used

Lncreasingly to support system design decisions and, to a lesser

legree, to Support the management decisions in system development.

rhe application illustrated here includes both types, but empha3izes the latter. Management decisions of special interest are

6-19

I -
'those concerning when particular inputs to the design' deliberations should be encouraged, and how much investment to make in

each potential source of such inputs.

For illustrative purposes, then, let us say that the range

of options available to the Project Manager with respect to when

to encourage human factors inputs is given an initial framework

by th.ý four design phases previously defined, i.e.:

e Mission Analysis (MA)'

& Concept Development (CD)

* System Demonstration/Validation (SD)

e Full-Scale Development (ED).

The main options, then, are:

1. None

2. MA only

.3. CD only

4. SD only

5. ED only

6. MA and CD

7. MA and SD

0

16. MA and CD ard SD and ED (all).

(In the higher-order options, the question of-relative degree of

input becomes a factor--but that factor overlaps with the allocation issue and adds a complication that is not' needed for this

illustration.) Thus, in this illustration there are 16 distinct

alternatives for when human factors inputs can be encouraged.

It is sufficient for thiz step simply to enumerate them.

6-20.
3. Specifying the Baseline. The objective of this step

is to define the status quo conditions relevant to the analysis;

namely, the baseline. Projected impacts are evaluated in terms

relative to a baseline. For each system development phase, a

systems bpselin, must be defined. Thus, if a human factors

action resulted in a design change in the demonstration/validation

phase, the baseline for the succeeding system development phase

would incorporate that change because it hal already been accomplished. Thus, the baseline is generally tied to a phase in the

development cycle.

The baseline provides a basis for the projection of future

conditions in which the human factor changes under consideration

are not developed and implemented. A baseline could be defined

for a set of human factor impacts when the individual impacts

cannot be isolated. However, it must always be defined so that

the impact areas and metrics under consideration are explicitly

identified.

The easiest way to understand this step is to make the

argument: 'each new system has a (more or less direct) precursor

system (or systems)'. • The baseline rests on the precursor or

composite family of precursors which we can call the reference

system. In most instances, the reference system will be the one

that would be used 'to perform the mission if the new' system were

not developed. For those analyses in which human factors are

emphasized, the mission compatibility criterion has a strong

old-new functional similarity aspect.

The following discussion illustrates the notion of the

mission/functional analysis in defining the baseline. There

are two analytic substeps in establishing the baseline for. system

design and cost projection purposes: Functional Differences and

ý6-21
Functional Deficiencies. The first entails the specification

of the reference system similar functions and any technological

differences between the reference system and the proposed new

system. For example, for the XYZ tank, the reference system

would be the operational MYY tank, and the technology differences

that impact on the man-machine functions could include those in

the main armament, armor metallurgy, turret stabilization, fire

control, and propulsion components. The functions of interest

are those needed to operate and maintain these components. rhe

product from this substep is a reasonably detailed functional

differentiation.

The second substep is a deficiency analysis of the reference

system. Again, it is functional deficiencies that count. For

example, was/is the reference system deficient in maneuverability?

In what specific ways? We also need to know what specific human

factors related deficiencies were brought to light during the

field use of the reference system. Possible source data for this

kind of deficiency identification could include the complaints of

operators and maintenance personnel. Observations of the actual

behavior of crews and maintenance units in action could also be

appropriate. Also, the human factors specialist could actually

go through dry runs of crucial segments of operational and/or

maintenance sequences. The product from this substep is a

definitive list of deficiencies. If value weights could be

assigned to each deficiency in a unambiguous manner, this could

also be useful.

The baseline is completed as a step in the overall methodology when the array of technological chdnges and reference

system deficiencies are put together in such a way as to give a

preliminary picture of the prospect of whether the technological

changes will tend to ameliorate or accentuate the deficiencies on

6-22
a one-by-one basis. Thus from the baseline we can geL a set of

assumptions that indicates what some of the major design problems

are going to be for the new system and, importantly, which are

likely to be human factors related.

4. Preparing the System Definition Statement. The objective of the system definition statement is to summarize concisely

all the essential information And assumptions about the subject

system that dre necessary to conduct the impact assessment.

kn important part of this definition is a historical record of

t.e evolution of the system's desigcjn and development, and the

corresponding impact and cost estimates. Though it will not be

possible in many instances to acgregate cost-benefit/impacts from

system development stage to stage, the definition statement can

provide selective evidencý_. of the role and contribution of human

factors R&D.

At a minimum, the system definition statement should contain

specifics on the following:

* Mission Profile (What is the system for?)

e System Performance and Operational Characteristics (What

are the syrtem capabilities?)

Acquisition Program Schedule (How is the system to be

procured?)

• Deployment (Peacetime and Wartime) Plan (How will the

system be utilized?)

* Support Concept (Initial and Mature) (How will the system

be 3upported and maintained?)

* Logistics Goals (What are the unique logistics related

goals, e.g., reliability?)

6-23
* Integrated Logistics and Training Considerations (How will

the operators and maintenance personnel be trained? How

will the required material be purchased, nanaged, etc.?)

* Human Factors Related Issues tWhat operation and maintenance considerations can affect the cost, capability, and

compatibility of the design?)

The first seven items are typically called for under current,

recommended major weapon system acquisition analysis guidelines.*

For these analyses, we have augimented those guidelines by adding

a separate disc;ission of human factors related issues that should.

be considered. These are issues that would be noted and discussed

in the human factors products (e.g., role of man) at the different

system development phases. The outcome of that consideration

and/or impact assessment should be reviewed throughout the system

development stages.

5. selecting the Impact Areas, Metrics, and Empirical

Measures. The objective of this step is to define the system's

life cycle cost, capability, and compatibility impacts, metrics,

and empirical measures for the goals and criteria identified in

Step 1. Some criteria may be included explicitly as cost or

empirical metrics, depending on their specificity, measurability,

and abstract properties.

Metrics and measures used to define the specific nature

and' focus of the human factors R&D impact must be tailored to

the phase of system development, the human factors product form,

*See, for example, DOD Directive 5000.1, Major System Acquisition;

5000.39, Acquisition and Management of Integrated Logistics

Support for Systems and Equipment; and DOD Instruction 5000.2,

Major System Acquisition Process.

6-24

""'* .~.... . ".- .. - . . . .,- ,- .6.'" .n , Sn J & VS S' | |' ! | | |
and the system-mission characterization. The impact area(s) and

associated component metrics and empirical measures comprise the

vocabulary to describe the effect of the human factors related

change(s).

The three generic impact areas--cost, capability, and

compatibility--have been introduced in previous chapters.

In Chapter 5, each of the impact areas was shown to be definable

in terms of a number of metrics, and the metrics were shown to

be functions of combinations of empirical measures. The generic

hierarchical relationship was illustrated in Exhibit 5-1.

Moreover, .he measures and metrics for capability and compatibility. in particular, reflect contemporary usage for describing

cause-effect relationships in both human factors R&D and system

engineering. In general, a human factors related change that

affects capability or compatibi 1

.ity will also affect cost.'

The set of vocabulary terms presented in Chapter 5 are from

our preliminary findings. They represent an initial step toward

the definition of a formal and stable set of terms to discuss,

model, and communicate the effects of human factors related

changes in military systems design and development. Each of

the impact areas and their component metrics and measures are

discussed briefly below.

0 Cost: For a weapon system specific setting, the cost

impact area is the life cycle cost of the system. A

candidate set of cost metrics, (major categories of costs

such as Operations and Support) and measures (cost elements such as Below Depot Maintenance) were presented in

Exhibit 5-5. If a military system, other than a weapon

3 such as a C I system, was the subject of the analysis,

it is likely that some different cost meas,7zs would be

6-25
required. The guiding criterion is: select the set of

cost metrics and measures that reflects the significant,

relevant costs effected by the human factors related

changes.

. Capability: For a weapon system specific setting, the

capability impact area is the mission worth of the system.

A preliminary, empirically derived set of capability

.metrics (e.g., availability, reliability) and measures

(e.g., mean-time-to-repair, mean-time-to-failure) were

presented in Exhibit 5-3. The particular combination of

measures used to functionally define a metric is dependent

upon the system or process being analyzed, and the various

ways the effect of the human factors changes can be

measured.

e Compatibility: For a weapon system specific setting,

the compatibility impact area is the physiological and

psychological suitability of the design. A background

discussion of compatibility metrics (e.g., user acceptance, motivation) and measures (e.g., temperature,

noise, vibration stress, altitude) is given in Chapter 5.

The underlying notion of the compratibility impact area

is that many human factor related effects are not easily

assessed using the-same quantitative metrics and measures

as for cost or capability., For example, reducing an

operator's stress is a substantive benefit, even though

its contribution to-enhanced system performance is not

directly quantifiable.'

The result of this step. is a specific set of vocabulary

terms to be used for describing the impacts, and in selecting/

constructing a model to estimate the values for the measures,

metrics, and ultimately their effects on the impact areas.

6-26

\,.
6. Selecting/Constructing the Impact Assessment Models.

The objective of this step is to derive or select appropriate

techniques or models that can provide both quantitative and

qualitative measures of the cost, capability, and compatibility

impacts expected from the application of the human factors

change.

In effect, one needs to relate the criteria from Step 1,

the information from Steps 2 to 4, and the impacts and rmietrics

from Step 5. Furthermore, that relationship must be relevant

to human factors R&D products and the system development process;

These relationships are tailored to and essentially define the

content of the human factors efforts and cells in Exhibfts 3-5

and 4-1, respectively.

A reasonable approach is to utilize Ostrofsky's (1977)

design methodology as a basic procedure, and to augment it with

other models that deal explicitly with life cycle cost and system

capability measures. (Examples of the latter are Goclowski,

1978; Forster, 1974; Fabbro & Fiorello, 1977; AF-Logistics

Support Cost Model, Design-to-Cost Model, and the Mission Success

Completion Probability Model.) In addition, there are several

techniques, other than Ostrofsky's, for evaluating and quantifying (imposing cardinal measures) on essentially qualitative,

ordinal measures. Examples are Gardiner (1979), Saaty (1979),

Quade (1975)', Hays (1975), Dalky (1969), and Linstone (1975).

Briefly, the sequence envisaged is as follows (Ostrof sky,

1977):

6-27
a. For the criteria defined in Step 1, specify the

underlying parameters. These parameters represent

the constituents of the criteria in a systems-component

sense. Each parameter is classified in terms of being:

- measured directly

- measured from a model

- included in other elements

- not measurable within existing resources.

b. Define submodels of the primitive, measurable elements

to define functionally the higher-level parameters.

c. Combine the submodels into an overall model to estimate

each criterion, and, in turn, an aggregate criteria

function for the overall goal.

While each of these steps is critical, it is most important

to understand the causal linkage between the elements, which can

be a mixture of qualitative and quantitative measures, the

parameter submodels, and,; in turn, the criterion function.

For a "hard" parameter such as reliability, the linkage between

it and cost and availability is rather well understoodi and many

acceptable models exist. For the "soft"' parameters such as user

acceptance,, the linkage is not nearly so clear. What is required

is a procedure that will handle both quantitative and qualitative

criteria (and their parameters and elements) in a systematic and

credible manner.

In summary, Step 6 puts all the information from Steps 1

to 5 into a formal setting with functional, causal relationships.

From the previous steps, we have:

,6-28
(Step 1)

* A set of goals, objectives, and criteria in a hierarchical

array.

(Step 2)

* A listing of (management) decision options.

(Step 3)

* A specification of the baseline in the form of an explicit

comparison between the reference system and the proposed

new system with respect to technological differences and

functional deficiencies in the reference system, and projected implications of such deficiene-ies.

'Step 4)

o In overall characterization of the proposed new system

and how it is to be operated and maintained.

(Step 5)

* A listing of critical metrics and empirical measures.

The model used to put these elements together can take a

numberof different forms, depending upon the system development

phase and problem setting. A discussion of model types and

selection criteria is given in the last section of this chapter.

We can now proceed to summarize the final four steps.

7. Collecting and Processing the Data. Given the specification of the impact areas, metrics, and the model fotm, this

step provides the required data to "drive" the model. Frequently,

týi: lack of data in sufficient quantity or detail will constrain

the nature and accuracy of the cost-benefit analysis.

6-29
8. Settina the Conventions for the An}vsit_. This step

specifies the conventions or ground rules used in arriving at

the cost, capability, and compatibility impact es3tixates.

Conventions for cost a;.d capability analysis shouhe cover.

a. Normative projections

b. Constant versus adjusted dollar cost estimates/

projections

c. Mature versus transient system characteristics

d. Personnel budget or economic costs

e. Capital investment leadtime considerations

f. Relevant, variable versus total costs

g. Uncertainty analysis (including technicaJ risk)

h. Presentation and documentation standards.

9. Estimating and Evaluating the Cost Benefits. This step

provides the output from the model and data prepared in Steps 7

and 8.

10. Presenting and Interpreting the Results. This step

entails preparing the presentation (including illustrations and

documentation of the results), identifying the requirements for

additional analysis, and specifying important issues that have

high degrees of uncertainty. An important part of the presentation is a description and quantitative portrayal of how the

change impacted the system design and its life cycle costs and

performance. Where feasible, the specific contribution of the

human factors change should be isolated. Often it mxuy not be

possible to isolate the-impact. In those instances, it may only

be reasonable to make the comparisons at the aggregate or systems

level (e.g., new vs. baseline), and to.infer the role ofthe

human factors impact. In addition to the standard tabular and

6-30
graphic presentation, the notion of color scoreboards, as used

by Goeller (1976) can be used to make and present comparisons of

alternatives.

6-31

I * *- * t- s..
Techniques for Estimating Impacts:

Classification and Selection

Step 6 of the impact assessment methodology is, in some

ways, the most crucial and the most complicated of the 10 steps.

It involves the selection of a model (or several models) for

conducting the particular analysis relative to the attributes of

a given system, the human factors issues; and the phase of system

development.

The selection process is complicated by the fact that there

is no one model or model type that is appropriate or best for

analyzing all the human factors issues throughout the system

development phases. Consequently, we cannot recommend any one

model for this step. Rather, we will discuss different model

types and some suitable selection criteria.

Because there are a number of models that can be used in

the above framework, it is useful to have, at least, a way to

classify model types in terms of their basic characteristics.

Our classification is a modification of the one used by Quade

(1975). The model classes are not mutually exclusive; that is,

a model assigned to one class can also have some of the characteristics of models in other classes. The class name indicates

the basic or principal characteristics of--and the means used

by--the modelWs) to analyze the issues under consideration.

We have identified seven model classes to catalog the types of

models that are potentially useful to human factors analysis.

1. Mathematical Models

2., Computer Simulation Models

3. Experiments

4. Operational Games

5. Surveys/Group Decisions

6. Verbal Models

S7. Physical Models.

6-32
A sample of some of the techniques that fall within the

above classes is provided in Exhibit 6-5. As this methodology

is developed and applied, additions and deletions to the sample

list will be made. Of the classes of models, thcse used in

category 1 (mathematical), category 2 (computer simulation),

category 3 (experimental), and category 7 (physical) appear to

be those used most frequently in human, factors analysis.

A suggested set of selection criteria that can be used

to identify the most suitable model to use in a given setting

includes:

1. Validity (Does the model reproduce or realistically

represent the functional relationships under

consideration?)

2. Relevance (Does the model deal explicitly with the

human factors issues'under analysis?)

3. Cost (Is the model very expensive to construct or use?)

4. Non-Trivial (Does the model provide substantive

insights into the process under analysis?)

5. Feasibility (Can the model be used? Are tie data

required available? Are the staff with the required

skills available? Is there sufficient time to use

the model?)

6. Reliability (Does the model give consisten results

under'different circumstances?),

7. Acceptability (Can the model results be co umunicated

successfully to the system development des gners and.

managers? Put another way, can or will th designer

use the model results?),

"6-33
Exhibit 6-5

Candidate Models and Techniques for Human Factors R&D

Cost Benefit/Impact Assessments (sample only)

Model Category Techniques

1. Mathematical Models Systems Analyses Techniques

The following is just a

small sample of mathematical - Systems Impact Assessment

model references: - Policy Analysis

See Goeller (1976), Hays, - Sensitivity Analysis

O'Cozmor, and Peterqon

(1975), Gardiner (1979), - A Fortiori Analysis

Quade (1975), Fisher (1971),

Mood (1974), Petruchell Statistical (data interpretation)

(1963) Hays and Wikler Techniques

(1970), Draper and Smith (1966), (1966), Baran Darane and and Gowskh Gorlowski - Correlations (cross-section)

(1978), Fabbro, Piorello, - Regressions (simple, multiple).

and Shaw (1977), Ostrofsky -A

(1977), Saaty (1979), Baker - Factor Analysis

and Pound (1964), Cetron, - Time-Series Analysis

Martino, and Roepcke (1967), - Pooled Cross-Section-Time-Series

Albnosta and Holzman' (1970),

Souder (1972), Fisher (1973), - Parametric Inferences and

Martin and Sharp (1973), Projections

Beers (1957), Dienemann - Multivariate Analysis ( 1966).

Military Operations Research Models

- Logistics Support Cost Model

(AF-LSC)

- Reliability, Maintainability,

and Availability Tradeoff

Models

Material Availability and

Resource Investment Models

Types of Design Aids

- Coordinated HLmwan 'Resources

Technoloqy (CHRT) Model

- Life Cycle Cost impact Model

.(LCCIM)

- Design Morphology (Ostrofsky,

1977)

6-34
.-..---.. ... .... .. • ' • _ . .. ........ .. . .• ... .... . .. . . ... . . - ' - ' -

Exhibit 6-5 (Continued)

Model Category Techniques

1. Mathematical Models Types of Designý Aids (Continued)

(Continued) - Pair-Wise Comparisons

- Maintenance, Reliability,

Diagnostic Accuracy,

- Availability, and Support

Cost Models

Types of Cost Models

- Planning Factor Models

- Detailed Engineering Estimates

Decision, Risk, and Utility Theory I - Decision Theory

- Risk/Uncertainty Analysis

- Project Scoring

- Utility Scales

- Relevance Tree Techniques

(Reverse Factor Analysis)

2. Computer Simulation Simulation-Process/E'entt Flow

Models - Monte-Carlo Techniques

(LCOM. CASES, etc.)

Mqck-ups (analogy)

3. Experimental Meth'ods Types of Experiments

See0 Davies', 1967 One-Shot Case Study (weakest but

most commonly used evaluation

design)

- One-Group, Pre-Test/Post-Test

Design (before vs. after)

The Static Group Comparison

(with vs. without)

-Pro-Test/Post-Test, Control

Group Design (before/after

and with/without)

6-35
Exhibit 6-5 (Continued)

Model Category Techniques

3. FExperimental Types of Experivents (Continued)

Methods (Continued) Solomon Four-Group Design

(controls both the experimental

effect and the possible interaction effects of the measuring

process itself)

- Post-Test Only, Control Group

Design

- Comparison of Alternative

Prograi Strategies (with

randoto assignment'N

- Non-Equivalent Coznparison Group

- Comparison of Alteriative

Program Strategies

- Time ,Series Design

- Multiple Time Series Design

4. Operational Games '4ming Techniques

- War Games

- Zero-Sun Games,

5. Surveys/Group Decision Nominal Group Techniques Models

Soam examples axe: - Highly Structured'

Dalky (1969),' Linstone - Hoircgeneous (e.g., only planners)

and Turof± (1975), Morris - Small Groups (approx. 8-l0)

(1977), Drown, Cochrans

and Dalky (1969). - Basic steps,

- silent generation

- rour4 robi;. presentation - clarification

- voting/ranking

- discussion of results

-. Interpr*tiva Structural Modeling

- Personal Znterviews

6-36
Exhibit 6-5 (Continued)

Model Category Techniques

5. Surveys/Group Decision Nominal Group Techniques

Models (Continued) (Continued)

-The Delphi Technique

- structured

- hierarchical

- quasi-anonymity

6. Verbal Models Scenario Building/Specification

Analogy Arguments Dialectics

7. Physical Models Types of Scheduling Aids

Some examples are:, - Program Evaluation and Review

Moder and Rogers (1968), Techniq% (PERT)

Miller (1963), Conway

(1967). - Critical Path Models (CPM)

- Functional Flow Diagrams

- Graphical Evaluation Review

Techniques (GERT)

- Queuing Models

- Programming Techniques (linear,

nonlinear)

Mock-ups '(physical)

6-37
As we gain experience applying the above criteria and using

the selected models, 'a more specific set of candidate techniques

can be provided. Conceptually, we need to identify certain

models for each of the cells in Exhibit 6-6 and indicate their

applicability in terms of the selection criteria. That information could be made available in a handbook that would be updated

on a regular basis.

Future Steps

The scope for this study effort includes: the development

of a conceptual basis that formally relates hmian factors

efforts and products to the major phases of a military system

develooment, the derivation of a set of metrics that can be

used to specify human factors impacts, and the formulation of a

methodology to quantify the impact(s) of human factors products

on the system design. This chapter accomplishes the third component (Chapter 5, the second, and Chapter 3, the first). It,

presents a methodology that prescribes a set of practical steps

tailored to the processes of human factors assessment. Much

remains to be done, however. To fully develop and understand

the impact quantification process, case studies are needed to

demonstrate the methodology and, especially, to select and use

pertinent models (Step 6). As experience is gained, the methodology can be refined and the actual results categorized for use

as future references. A useful'set of references would contain:

(a) a refined methodology, (b) case examples, and (c) data and

models.

6-38
'A__ _ _ _

0I-

*~5 l

72_ _ _ _ __ _ _

c x

0 '

6-3
/

References

Alboosta, C.A., & Holzman, A.G. Optimal funding of an R&D

project portfolio. Presented at the llth Institute of

Management Science Meeting, Los Angeles, CA, October 1970.

Alderson, R.C., & Sproull, W.C. Requirements analysis, need

forecasting, and technological planning using the Honeywell PATTERN technique. Technological Forecasting and Social

Change, 1972.

Anderson, L.G.,,& Settle, R.F. Benefit-cost analysis: A

practical guide. New York: Lexington Books, 1977.

Baker, N.R., & Pound, W.H. R&D project selection: Where we

stand. IEEE Transactions on Engineering Management, EM-ll,

December 1964.

Baran, H., & Goclowski, J. LCCIM: A model for analyzing the

impact of design on weapon system support requirements and

LCC,. Dynamics Research Corporation, 1978.

Baran, N., & Goclowski, J. Life cycle cost impact model.

Technical Report R-2624, Dynamics Research Corporation,

1978.

Beers, Y. Introduction to the theory of error (2nd ed.).

New York: Addison-Wesley, 1957.

Blalock, H.M., & Blalock, A.B. Methodology in social '-'search.

New York: McGraw-Hill, 1968.,

Brown, B., Cochran, S., & Dalky, N. The Delphi Method II:

Structure of experiments' (RM-5957-PR). The Rand-Corporation, Santa Monica, CA, 1969.

Brown, G. Smoothing, forecasting and prediction. New York:

Prentice-Hall, 1963.

Butler, R. Feasibility of a cost/benefit comparison of simulators

and traditional maintena~nce training methods (PR-i05)'. The

Assessment Group, Santa Monica, CA, 1979.

Cetron, M.J., Mostino, J., & Roepcke, L. The selection of R&D

program content - Survey of quantitative methods... IEEE

Transactions on Engineering Management, EM-14, March 1967.

Conway, W., et'al. Theory of scheduling. Reading, MA:

Addison-Wesley Publishing Company, 1967.

6-40

.- ~~~- A- ... . .
Dalky, N.C. The Delphi Method: An experimental study of group

opinion (RM-5888-PR)., The Rand Corporation, Santa Monica,

CA, 1969.

Davies, L. Design and analysis of industrial experimcnts.

New York: Harper Publishing Company, 1967.

Department of Defense. Major system acquisition. DOD Directive

5000.1, March 1980.

Department of Defense. Major system acquisition process. DOD

Instruction 5000.2, March 1980.

Department of Defense. Acquisition and management of integrated

logistics support for systems and equipment. DOD Directive

5000.39, December 1979.

Dieneman, P.F. Estimating cost uncertainty using Monte Carlo

techniques (RM-4854-PR)., The Rand Corporation, Santa Monica,

CA, 1966.

Draper, N.R., & Smith, H. Applied regression anaZysis. New York:

John Wiley & Sons, Inc., 1966.'

Economic analysis and program evaluation for resource management.

Army Regulation 11-18, Headquarters, Department of the Army,

January 1976.

Fabbro, R., Fiorello, M., & Shaw, M. Avionics design trade-offs

(working note, draft). Logistics Management Institute,

Washington, D.C., October 1977.

Fiorello, M., & Betaque, N. Aircraft system operating and

support costs: Guidelines for analysis (LMI Report 75-1/1).

Logistics Management Institute, Washington, D.C.,,March 1977.

Fioreilo, M., & Jaffin, S. Costs and benefits of federal automated data processing standards (DC 801). Logistics

'Management Institute, Washington, D.C., 1978.

Fisher, G.H. Cost considerations in systems anaZysis. New York:

American Elsevier, 1971.

Fisher, G.H. The nature of uncertainty (P-5133). The Rand

Corporation, November 1973.

Forster, J. Criteria for evaluating weapon system reliability,

availability and cost (Task 73-11). Logistics Management

Institute, Washington, D.C., March 1974.

6-41
Gardiner, P.C. Multi-attribute utility measure for logistics

decision making. Spectrum, Summer 1979.

Goeller, B. System impact assessment, (R-1446). The Rand

Corporation, 1976.

Hays, L.M., O'Conner, M.F., & Peterson, C.R. An application of

multi-attribute utility theory: Design to cost evaluation

of the U.S. Navy's electronic warfare systems. Decisions

and designs. October 1975.

Hays, W.L., & Winkler, R.L. Statistics: Probability, inference,

and decision. New York: Holt, Reinhard and Winston, 1970.

King, G.F., &-Askren, W.B. Human resources, logistics, and cost

factors in weapon system development (AFHRL-TR-79-28(1)).

AFSC,' Texas, September 1979.

King, J.L., & Schrems, E.L. Cost-benefit analysis in information

systems. Development and Operation Computing Surveys, 1978,

10(l), 19-34.

Kneppreth, N.P., Hoessel, W., Gustefson, D.H., & Johnson, E.M.

A strategy for selecting a worth assessment technique

(DAHCl9-78-G-0008). ARI-TR-280, February 1978.

Lindgren, B.W. Statistical theory. New York: Macmillan, 1962.

Linstone, H.Q., & Turoff, M. (Eds.). The Delphi Method:

Techniques and applications. Reading, MA: Addison-,

Wesley Publishing Company, 1975.

Lockett, A.G., & Gear, A.E. Representation and analysis of

multi-stage problems in R&D. Management Science, 1973,

19 (8).

Martin, W., & Sharp, J. Reverse factor analysis: A modification,

of relevance tree techniques. TechnoZogical Forecasting and ......

Social Change, 1973, 4.

Miller, R.W. Schedule, coato and profit control with PERT.

* New.York: McGraw-Hill,'1963.

Mischan, E.J. Cost-benefit analysis. New York: Prager

Publishers, 1976.

Moder, J.J., & Rodgers, E.G. Judgment estimates of the moments

of PERT type distributions. Management Science, 1968, 15(2).

6-42
Mood, A.M., et al. introduction tc the theory of statistics

(3rd ed.). New York: McGraw-Hill, 1974.

Morris, K.E. The application of the Delphi technique to the

development of productivity measures. Ohio State University

Research Foundation, Project No. 4236-Al, Columbus, OH, 1977.

Murphy, E.I. Statisticai methods of measuring uncertainty of

cost estimates. Redstone Arsenal, U.S. Army Missile Command,,

February 1970.

Office of Management and Budget. Major system acquisitions.

OMB Circular A-109, Executive Office of the President,

Washington, D.C., April 1976.

Orlansky J., & String, J. Cost-effectiveness of computer-based

insctrction in military training (IDA Paper P-1375).

Institute for Defense Analysis, Office of the Secretary

of Defense for Research Engineering, April 1979.

Ostrofsky, B. Design planning and development methodology.

New York: Prentice-Hall, 1977.

Papoulis, A. Probability, random variables; and stochastic

process. New York: McGraw-Hill, 1965.

Peters, R.A. Return on investment. AMACOM, American Management

Association, New York, 1974.

Petruchell, R.L. Some curve-fitting fundamentals (RM-5766-SA).

The Rand Corporation, Santa Monica, CA, December 1963.

Prest, A.R., & Turvey, R. Cost-benefit analysis: A survey.

The Economics Journal, 1965, 15(300).

Quade, E.S. Analysis for public decisions. New York American

Elsevier, 1975.

Saaty, J. Hierarchies and priorities--Eigenvalue analysis.

Draft for forthcoming publication, University of Pennsylvania,

1979.

Sobel, S. A computerized technique to express uncertainty in

advanced system cost estimates. The Mitre Corporation,

Bedford, MA, 1965.

Souder, W.E. Analytical effectiveness of mathematical models for

R&D project selection. Management Science, April 1973, 18.

6-43

.1~ *.-.
Souder, W.E. h scoring methodology for assessing the suitability of management science models. Management Science, June 1972,

18.

Souder, W.E. Comparative ar.alysis of R&D investment models.

AIEE Transactions, March 1972, 4.

Statement, A., Bennett, *., U Moore, W. Economic analysis handbook theory and application (OAD-CR-22).

Vol. I Army Investment Analysis Vol. II Concepts and Techniques Vol. III Guide for Reviewers of Economic Analysis "Vol. IV Case Studies.

General Research Corporation, November 1973.

Wagner, H. Principles of oper'ationis research. New York:

Prentice-Hall, 1969.

Whoiey, J.S. Federal evaluation policy. The Urban Institute,

Washington, D.C., 1975.

Wonnacott, H., & Wonnacott, R.J. Introduction to statistics.

New York: John Wiley & Sons, Inc., 1969.

6-44

-AAA
CHAPTER 7

CONCLUSIONS AND RECOMMENDATIONS

This study sets out to explore and define a conceptual basis

and methodology for identifying and measuring the contributions

of human factors to military system development.

Our major conclusions are:

* A conceptual basis for relating human factors contributions to system development can be defined. In

Chapter 4 we identify and relate the placement and

content of specific human factors efforts and products

with the development phases of a military system. Each

phase represents a window of opportunity for certain

human factors products.

e The human factors contribution is measurable. In

Chapter 5 we derive a preliminary (but useful) set'

of metrics for describing and Measuring human factors

impacts on military systems. Actually, a three-level

hierarchical vocabulary is presented: systems-mission

terms are at the top, for cost, capability, and

compatibility impacts; metrics for defining the primary

focus or results of design engineering and human factors

analyses are at Level 2; and empirical measures are,'at

the third and. lowest level. We also show that system

design and human factors criteria and terminology are

compatible by constructing a vocabulary from engineering

and human factors common and complementary terms.

7-1

,..
* A methodology for evaluating the impacts and metrics is

feasible. In Chapter 6 we build upon the basic concepts

of cost-benefit analysis with recent impact assessment

advances. The resulting conceptual framework can be used

to evaluate both quantitative and qualitative metrics

needed to represent human factors effects.

Several additional steps are required to validate these

conclusions. The main step is to actually exercise the proposed

methodology on some real-world cases.

In order to support such test applications, it would be most

advantageous to first, create two special data bases in the form

of computerized relational files. One of these files should be

an expanded and slightly restzuctU'red version of the file used in

the present effort to establish the metrics vocabulary. The new

version of this prototype file would support the elaboration and

selection of the specific models (techniques) to be used in

operational data collection and analysis. The other file would

consist of an inventory of cpisodes in prior and ongoing military,

systems development programs (or, ideally, complete chronologies

of work programs). The objective would be to provide the means

to pinpoint targets for the impact assessment process.

The experience gained and results from the case study applications of the methodology should be documented in a series of

Human Factors Impacts on System Development Handbooks, such as:

e Guidelines for analysis that would present:

- the conceptual basis for relating principal human

factors products and military system development

phases

- the impact assessment methodology

- the impact assessment vocabulary.

7-2
e Models and techniques that would catalog selected models

used to assess human factors related changes.

e Reference information that would present:

- a list of the controlling documentation

- cost and planning factors

- description of the relational data files (discussed

above).

a Issues and case studies that would document:

- current and emerging human factors/system design

problems

- the case studies.

7-3
kPPENDIX A

A RATIONALE AND PRECLDENT FOR ESTABLISHING THE

PRINCIPAL HUMAN FACTORS PRODUCTS

DURING SYSTEM DEVELOPMENT

The notion of principal human factors products resulting

from each phase of system development is a cornerstone of the

methodology developed in this report. The products of each

phase.were' identified in Chapter 1 as follows:

MAJOR PHASE

OF SYSTEM ACOUISITION PRINCIPAL HF R&D PRODUCT

Mission Analysis Phase - Deivelopment of the Role of Man as a part of a

Mission Element Needs Statement (MENS)

Concept Development Phase - Allocation of System Functions to Man as a

part of the Decision Coordinating Paper (DCP)

DemonstrationNalidation - Task Analysis and Determination of Human

Phase Engineering Requirements

Fuil-Scale Development Phase - Design of the Optimal Man-Machine Interfaces

In Chapter 3, the principal human factors products were explicitly

derived in terms of specific human factors efforts in each system

development phase. A recontmendation about the types of information included in documenting each product was also provided.

The purpose of this Appendix is to provide the underlying

rationale and precedent for establishing the principal human

factors products in this project. Each product will be discussed

separately.

A-l

J
Development of the Role of Man

Many approaches have been put forth for analyzing man's

capabilities and limitations with respect to his potential

role in system performance. Some approaches suggest that

man and machines should be compared for system performance,

while others suggest that man and machines are not comparable

but are complementary., Some suggest that man should be designed

into the system wherever possible; others suggest that man

should be designed out of the'system wherever possible. There

are numerous controversial issues concerning man's capabilities

and limitations for system performance. The philosophy

expressed here (adapted from Price & Tabachnick, 1968) is

il) that man has certain unique performance capabilities that

cannot be compared with machines; (2) that many system performance requirements can be obtained either by man, or man-machine

design solutions (man-rated), or can be obtained by machine

alcne (automatic); (3) that if man's inclusion in the system

is justified by his performance of mission-critical functions,

his utilitarian capability may be exploited for performance

of other system tasks which are not cost-effective to automate;

and (4) that man has certain unique limitations which require

some type of personnel support system to accomnmodate his

physical,. physiological, and psychological constraints wherever

he is used. This philosophy suggests that the potential role

of man in systems should be based ,on four questions:

1. Can man's unique capabilities be significant in the

attainment of the system goals? While it.is difficult,

to categorize man's unique capabilities, they seem to

lend themselves to two major groups, as follows:

A-2
a. Man has the ability to learn; that is, acquire new

kno-v ledge and skills. Man can learn by practice,

trial and error, or transfer of previous training.

This unique ability to learn and transfer that

learriing to another situation has important

implicationa for man's potential role in a system.

First, man can perform in many complex system

situations if those situations are merely similar to

the learning situation; second, man can accomplish

deliberate or insightful learning "on-the-job" if

the occasion calls for it.

b. Man has the capacity for creative cognition. This

unique capability is frequently referred to as the

ability to "think," but man's truly unique characteristic is that he is capable of insightful or

heuristic thinking. On this basis, it may be said

that man is unique in his ability to exercise

judgment in unstructured situations, or to form

concepts.

2. What system performance could be implemented by man?

This question is concerned with either operations

or maintenance performance which is basic or clearly

related to mission success, and may usually be

restricted to primary or critical performance

activities.

3. If a role for man is justified because of his unique

capabilities (question 1) or primary performance

activities (question 2), wlat other performance can be

A-3
assigned to him to take advantage of his utilitarian

capabilities? In other words, if man is justified in

the system for other reasons, then full advantage

should be taken of such things as his flexibility,

adaptability, and motor skills, to perform tasks

which may be very difficult or costly to automate.

4. Will man's unique limitations constrain his use in the

system? Together, these limitations comprise what we

have been calling "compatibility" impact areas. This

question must consider both system and individual

factors such as the following:

a. Man has certain physical characteristics of size,

weight, shape, strength, etc.

b. Man has physiological needs, such as air, nourishment, environmental protection, sleep, comfort, and

-general health maintenance.

c. Man has psychological needs. His performance can

change significantly as a result of such attitudinal

variables as motivation, frustrationf conflict,

fear, etc.

There are at least two other key issues which should be

considered in any discussion about the role of man. These ere

that (1) user acceptance factors are most critical and will have

a maximum effect on system effectiveness as a result of the

determination of maia's role; 'and (2) determination of man's

role also means a determination of whether man is local (in the

A-4
immediate mission environment) or remote (at so-e geographical or

physical location away from the immediate mission environment).

Man's Role'and Acceptance Problems

The morale of man is frequently lowered if he is not functioning at what for him is a high skill level. Aristotle

defined happiness as functioning at the highest level one is

capable of. More recently, Nissen (1954), in his paper on

motivation, stated that "capacity is its own motivation."

Consequently, if a man's system role does not permit hi,. to

exercise his capabilities or capacity, he will beI!ome frustrated

and lose motivation for performing his assigned and expected

system duties.

This problem of not being permitted to maintain and improve

a skill capability can become compounded by an expectancy.

Frequently, a man will be led to believe that he will function

and learn at a higher level than in fact he will on the job.

This false expectancy, often the result of overzealous recruitment, will increase the frustration due to non-use of complex

skills.

Unfortunately, little work has been done on acceptance

problems in complexsystems. However, soie work performed by

Price, Smith, and Behan (1964) on .1cceptance problems in automated landing systems for aircraft, and the study of morale

problems in the military unit,-revealed the following three

general principles regarding acceptance of the system role of

man.

1. Men are generally accepting of system roles which

give them an opportunity to exercise and, therefore,

maintain skills which they feel are important to

maintaining their position in the occupational and

social status system in which they are immersed.

A-5
2. Men are generally accepting ut system roles which

permit them to vary their procedures and the manner

of accomplishing their tasks, on their own initiative.

Roles that fail to permit man to vary his procedures

on his own are generally labeled mechanical.

3. Men are more accepting of roles which permit them to

learn. In a recent study (unfortunately, utilizing

a sample of only seventeen) a correlation of +.61

(statistically significant at the .01 level of confidence) was found between how much the men felt they

are learning on their job and their intentions to reenlist. (pp. 70-71)

In surutary, it may be said that men generally accept those

roles that have more responsibility, authority, and opportunity

to learn. This relates directly to the consideration of man's

role in the system as local or remote.

Local or Remote Roles for Men in Systems

Military systems may be frequently ccnceived in terms of

geographical or physical location of the performance units within

the total system configuration. For the present purposes, local

refers to the immediate mission environment in which the system

operates, and remote refers to some geographical or physical

location away from the immediate mission environment. A manned

bomber, for instance, has man-in a local role in the mission

environment; while a ballistic missile system has man in a remote

role from the immediate mission environment. Price, Smith, and

Behan (1964) have also discussed this issue, particularly concerning man's local role in a system; their reasoning is included

below.

When man is included locally inma system, one of

the usual reasons is to have him available to deal with

unusual and unforeseen events. It is man's recognized.

aptitude for reprogramming or redesigning his role on

the spot to deal with the unexpected that is so valuable,

because it will increase system reliability. However,

A-6

' A. , 
this is an aptitude of man, not a subsystem output achieved at no cost to the system or system designers.

Like all required outputs it is not free but requires

inputs. In order to be able to effectively redesign his

role, in, unusual or emnergency situations, two preconditions mist exist, and at this point in the design we

must determine if in fact they will exist. These

preconditions are as follows:

a. The man must understand the over-all funci ion

of the system, and more specifically the subsystem he is interfacing with, his role in it,

and how all automatic functions operate for

which he might have to provide total or partial

back up. Where man is not given adequate explanations as to how functions other than

his own are performed, particularly machine

functions, he will make up his own explanations,

as has been pointed out by Firstman and Jordan

(30). These explanations will more than likely

be incorrect and, therefore, not an effective

tool in an unforeseen situation.

b. Man must be proficient at rapidly solving new

and unforeseen problems in the subsystem

environment. It has been demonstrated that

this capability can be learned. Such capability

has been labeled learning how to learn, or more

simply as a learning set. Howeve,, this ability

can be created and maintained only by giving the

man the responsibility and freedom to continually

try out new tasks and methods. Obviously it is

not possible to produce the capability in man

to deal with unforeseen events by selection,

traditional training methods, or job guides.

Therefore, if man is to be placed in a system,

particularly locally, in order to increase system reliability by having him as an additive for dealing

with unforeseen events, it is essential to give him

as much responsibility and authority as is feasible.

Maximum responsibility and authority are necessary to

permit him to develop a learning se_ co that he will

have the capability to deal with unexpected events.

(pp. 27-28)

A-7

L ... ' * ,*. - I| | •
An interesting perspective cn the local versus remote issue

in the role of people and systems appeared in a recent issue

(April 1978) of the Armed Forces Journal. Because of its brevity,

the item is included below just as it appeared in the magazine.

DON'T BE HYPNOTIZED

BY THE FLASHING LIGHTS

A senior Army leader, who has fought and won in many

combat actions, raised a fundamental caution when

discussing the CP advances with AFJ. His concern:

that commanders will become so mesmerized with the

fantastic display devices and torrents of information

slicing into their command posts that they will stay

there rather than get out with the troops fighting

the battle. He says it's easy for them to convince

themselves that the CP--not the fiaht--is the proper

place for them. That is wrong, he says.

He was asked how to prevent the "CP syndrom [sic]."

He considers the solution fairly easy. Simply let

commanders take the command post with them, by using

micro-miniaturized terminals and display devices.

Then they can be in the fight, but still tap the

resources of the CP.

The particular item abcve is also of interest from the viewpoint

that we are experiencing an extraordinary growth in hardware and

computer technology, and this tecnnology growth will critically

impact the roles of men in future systems.. It is therefore more

important than ever that the human factor not become the forgotten

factor at this phase of system development.

To bring this discussion of the role of man to a close, it

seems reasonable to acknowledge come support of this concept from

other authors. Coburn (1973), in his document entitled "Human

Engineering Guide to Ship System Development," takes note of the

fact that the justification for developing any new Navy system is

not to produce hardware 'but rather to achieve some specific

operational capabiiity. Coburn further notes that these

capabilities are generally the objective of the General

A-6
Operational Requirement (GOR)'and. that GOR 43, "Personnel Logistics,"

discusses human engineering. While the GOR does not use the term

"man'3 role," it does imply a similar responsibility for human

factors engineering as indicated by the excerpt included below.

Human Factors-Engineering. This area is primarily

concerned with the implementation of human operator

considerations in the development, operations, and

maintenance of new and current organizations, weapons,

and support systems. The human operator is defined

in the broadest context, to include system managers, assigned leaders, operators, maintainers, and support

personnel. The requirement for successful integration

of people insists that qualitative'and quantitative,

elements of normallyfunctioning human capabilities,, within the constraints of people resource availability,

be the focal points 'around which organizations,

weapons, and support systems are designed.

Air Force Regulation 800-15, Human Factors Engineering and

Management, states that one of the major objectives of HFE is

to assure that "man's role in the system is' defined in order to

optimize his performance in relation to that specific system."

Finally, Melching (1968), in a paper titled "A Concept of

the Role of Man in Automated Systems," discusses the problem of

man's role in highly automated Army air defense weapon systems.

An excerpt of.particular pertinence to the topic at hand is

presented below.

It is suggested that man's role in automated systems,

although perhaps more subtle and more difficult to

assess, is comparable in significance, to the other

factors. Thus, for example, just as the designer of

a system cannot decide whether to automate a given.

set of functions until fie is satisfied that automation

is within the state of the art,,so should he not

attempt to assign functions to man until he has

arrived, at a satisfactory conception of the general

role of man in that system.

A-9

S•-. , - -'
In other words, to guide him in this thinking and

planning, the designer of an automated system needs

a clear-cut conception of the general role of man

in such systems. Without this basic guidance, the

functions that he allocates to man may reflect only

a sort of "fallout" from his attempts at automation,

rather than any careful premeditation on his part.

In short, rhe designer need3 a conception of what

man's role should be before he can decide what it

wilZibe.

In brief summary, it is the'opinion of the present Puthors

that determination of man's role during the mission analysis

phase of the systems development is a key human factors product

upon which hinges the critical impact of huMan factors with

respect to capability, cost, and compatibility.

The Allocation of System Functions to Man

In every military system development cycle there must be

decisions concerning (1) if man should be in the system or not,

and (2) if he is in the system, what he will do. If one is

concerned with obtaining optimal human performance in military

systems, then clearly, determining (1) whether man will have a

role, and (2) if so, what functions he will participate in are

two of the most important decisions in a system development cycle

that'bear upon this concern. Subsequent. activities in system

development that are concerned with task analysis, selection,

training, and human engineering of interfaces forman in the

system are consequences of the role and allocation of function

decisions and cannot make up for bad decisions in these two

areas.

Despite the importance of these two decision steps concerning

man's performance in the system to be 4eveloped, there has been a

general concern in the literature and among human factorsoprofes-

'sionals that allocation of function decisions is inadequate.

This is not a new concern.

A-1O
In a document entitled "Factors Affecting Degree of Automation

in Test and Checkout Equipment," which, among other things, reviews

the problems of allocation of function, Swain and Wohdl (1961)

assert:

A rather stark conclusion emerges: There is no

adequate systematic methodology in existence for

allocatini functions (in this case, test and -Feko funtions) between man and machine.

This lack, in fact, is probably the central problem

*in human factors engineering today . . . It is

interesting to note that ten years of research

and applicaticns experience have failed to bring

us closer to our goal than did the landmark article

by Fitts in 1951 (p. 9).

In an article entitled "Allocation of Functions Between

Man and Machines in Automated Systems," Jordan (1963) discusses

current problems and efforts to allocate functions between men

and machines, and arrives at a similar conclusion to that of

Swain and Wohl. Jordan's final conclusion is stated as follows:

"Herein lies the main future challenge to hmman factors

engineering." (p. 165)

Jordan also presents an analogy drawn from the physical

sciences and concerned with the concept of "ether," which:

. played a central role in physical thinking for over a century after having first been

introduced as a necessary medium for propagating

electromagnetic waves. But during all this time

all attempts to build and expand upon this concept

led to difficulties and contradictions. A century

of research on ether turned Out to be sterile in

that no significant advance was made during that time.

The conclusion which Jordan draws from this analogy is as

follows:

The lesson to be learned from this momentous episode is that when a scientific discipline finds itself

in a dead end, despite hard and diligent work, the

A-11
dead end should probably not be attributed to a

lack of knowledge of facts, but to the use of faulty

concepts which do not enable the discipline to order

the facts properly. Tne failure of human factor

engineering-to advance in the area of allocation of

functions seems to be such a situation .

During studies of highly automated Army Air Defense Systems,

Melching (1968) arrived at a similar concerr. over the allocation

of functions in systems, as indicated below.

With the aid of extremely capable electronic

computers, such systems are able to process vast

amounts of environmental and other data. The

capability of these systems is such that they

can, at least theoretically, conduct an entire

battle without the assistance of man.

Such a prospect is awesome, to say the least.

As a consequence, tlie builders and users of such

systems have shown a strong inclination to design

them so that a manual override of some sort is

possible. No one, it seems, is willing to let

the machine make all the decisions.

Manual -rride, however, is only one of several

issues t tend to arise in connection with highly

automzjid. systems. Numerous other questions also

appear. For example, in what specific ways and in

what circumstances would man be justified in intervening in the actions and decisions of an automated

system? Should man ever be given duties o ter than

that of system override? Should man ever ounction

in series with the machine component? Or hould• he

function only in a parallel backup fashionh

All these questions, of course-, are expres ions of

a problem that has long plagued system des gners---

that of allocation of functions between and

machine.

Men and machines are not competitors. Thig statement is

paramount when one is concerned with the allocation of a system

function for peýrformance by a man or a machine. It is equally

poor system design to have a man doing a machine's Job or a

machine doing a mans job.

A-12
Many system or function performance requirements may be

implemented by solutions involving man as part of the design

concept. Such solutions, wherein it is feasible to use man

as part of the solution, are called man-rated (froia Price &

Tabachnick, 1968). Thus,' man-rated performance is any performance that can be obtained with man as part of the design

solution. The essential question of function allocation is

what functions are man-rated and what is the most advantageous

extent of man's participation.

The range of human participation in potential man-rated

solutions may actually be a continuum; however, for the sake

of simplicity only the ends of the continuum need be'defined.

These two points are simply called manual and automatic.

Manual. performance implies that a man performs the function;

that he generates or accomplishes whatever power, energy, or

energy transduction is required; and, furthermore, that he controls

the application of power or directs the utilization of the given

energy. No assumptions are made about the nature of the activity.

It may utilize human receptors or effectors, or both. The definition does not preclude the use of tools (e.g., a chart, a

lever, or a telescope) which merely extend man's raw capabilities.

Automatic performance implies that a machine performs the

function by 'enerating or accomplishing whatever power, energy,

or, energy transduction is required; and that a man controls, in

real time, t:.e application of the power or directs the utilization of the given energy. In automatic function performance man

participates, but indirectly. He mny determine what is to be,

done, and perhaps how, as in the use of a digital or analog

computer. He usually monitors the output to determine whether

Sit meets certain minimal performance standards. He initiates

and may terminate the operation of the automatic device, as in

the use of an autopilot.

£ A-13

- - ..-. -. . - * §,c
As may be seen from the discussion above, the problem of

allocation of functions to men in many cases becomes a decision

as to the degree of automation. The decision to automate a

function in many cases makes the role of man qualitatively more 14

demanding. RFcenc advances in the state of the art in engineering I

and computers have produced machines with .remendous capacities

and speeds which may require fewer personnel to operate, but which

also may place increasingly more difficult tasks on those personnel who must install, maintain, monitor, override, and.program

these systet's. I1 ttie :ask complexity of personnel performances

in a highly automat.ed s~stem is to be reduced to the point where

highly skilled personnul are not required (or at least reduced),

thei, the burd&n of r•: .ponsibility lies with the human engineering

of the interface L..tween the automatic machine and the human

monitor or operator. In this way, training costs and time for

personnel can. be kept to a minimlim. The general advantages of

the automaLed ý.stem appiy to the extremes of any performance

continuum. For example, monitoring functions which either do

not change over long periods'of time, or change extremely rapidly

in time, are best performed automatically. Those monitoring

functions that are in the middle of the continuum may well fall

within the capabilities of a human monitor and be performed by

man with as 'uch reliability and accuracy as by the machines, and

at much less cost.

Finally, a special consideration that needs to be pursued in

allocating functions is user acceptance. Recent advances in

computer technology have presented a temptation to system

designers to automate functions whenever possible. This may

create more problems than it solves. particularly with respect

to the compatibility impact and user acceptance.

Acceptance prcblems could be defined as any frustration of

any human needs. As an example, excessive automation, particularly

in information processing systems, may end up overloading the

A-14
uiser with more information than he can handle. Excessive automation may also restrict man from performing at his highest skill

level. If automation does not permit man to exercise his

capabilities or sapacity, he 'will become frustrated and lose

motivation for performing his assigned and expected system

functions. Problems created by lack of 'confidence in the effective.

and reliable performance by hardware of automated functions must

be considered independently of whether in fact the hardware is

effective and reliable. If man does not accept a particular

automated function, he simply will not perform in the manner that

the system designer intended.

What is needed is the development of an awareness of the need

for considering user attitudes when system design decisions are

being made. This will permit the incorporation of acceptance

factors as one criterion in tradeoff analyses that already include

a consideration of the performance capabilities and reliabilities

of man and equipment components. It may be found, for example,

that a decision to automate a particular function based upon sound

engineering considerations would produce a degree of negative

acceptance that would clearly offset the anticipated advantages of

the engineering solution. Price, Smith, and Behan (1964), in the

study of pilot acceptance of automated landing systems, offer the

following principles as guidance concerning automation acceptance.

1. The more system experience a man has,' with this

experience including exposure to automated

equipment, the more accepting he is of the

automated equipment and the more he will use

it in the prescribed manner.

2. Those with more status, responsibility, and

authority tend to be more accepting of and make

more use of automated equipment than others.

3. Where failure of the performance of its function

by automated equipment can endanger the life of

the man, he is less likely to accept and use it

despite prescribed procedures.

A-15
4. There is generally high acceptance, within the

limits of the above three principles, of the

automation or servo tasks, particularly those

which must be performed over long periods of

time.

5. There i3 generally rather low acceptance of

automation of decision making functions.

In concluding this discussion, it is salient to note that

the allocation of system functions to man (or machine) is recognized as a h~iman factors product by DOD and all three services.

DOD recognizes the definition and allocation of function

in MIL-H-46855B', Human Engineering Requirements for Military

Systems, Equipment and Facilities.

The Army recognizes function allocation in AR 602-1, Human

Factors Engineering Program, and in HEL Guide 1-69, Manpower

Resources Integration Guide for 'Army Material Devolopment.

The Nay recognizes function allocation in NAVMATINST

3900.9, Human Factors; in the Human Engineering Guide to Ship

System Development iCoburn, 1973); and a report on Human Factors

Engineering for Navy Weapon System Acquisition (Baker et al.,

1979).

Tle hir Force recognizes function allocation (man-machine

analyses) in AUR-800-15, Human Factors Engineering and Managemant;

and in AFSC Design Handbook 1-3, Human Factors Engineering.

A-16
Task Analysis and Determination of

Human Engineering Requirements

As suggested by the title, this human factors product has

two parts. The first part, Task Analysis, is a delineation of

the specific task performance (both operator and maintenance)

required to be performed by man. The second part, Human Factors

Engineering Requirements, is more concerned with how man is

expected to accomplish those tasks (at least some of them will be

aided, by human engineering), and the identification of information and response reqiirements (interfaces) between man and the

system. Human factors personnel should provide (1) both the

methodology for and performanca of task analysis, (2) the

identification of human engineering as a means of achieving

(cr assisting) task performance, and (3) specific requirements

or techniques for man to receive information from the system

and to make re3ponses to the system.

Task Analysis Requirement

Task analysis has been employed by those individuals concerned with the "Personnel Subsystem", ever since people have been

recognized as a integral part of military systems. Task analysis

as it is genera ly practiced today was probably first formalized

by Miller (1953 . It is the basis for human engineering because

it 'is 'necessary to know "what" is expected of people in systems

before we can p escribe "hoW" personnel are to achieve what is

expected of thel.

Task analy is data is clearly required by MIL-H-46855B,

Human Engineeri g Requirements for Military Systemn, Equipment

and Facilities. Also, the Tri-Service Advisory Group for Human

Factors is deve oping a new task analysis requirement.

A-17
The ned •ur tas!'. analysis to support training and performance aids haý als3 bt.-L•n recognized by DOD and the services and

was recentl ]vat Ia.ýz for maintenance) by Foley

(1978) in a reucrt cn die impazt of advanced maintenance data and

task oriented t rain. t ec!;nologies on maintenance, personnel,

and training '

Human Factors ::,.:..:e:uiremefnts r. 

Achievin.- •'Prsc:,- Perdormance. As stated earlier, once

it has been oat. i--:, "what" people will do in systems it is

necessai-y to ditei ',nr.ow" to provide for achievement of the

expecteti ý.ertci u:.

There art , .:',a!, four ways, in which one may develop

personnel pcr ,r:f.nance acnievement:

. P,: , - ction

3. Jcb -1:; 1, manuals

4. Human enoine.2ering.

PersoPnel selection te~chnique. are useful when a small

number of personnel. are required, highly specialized skills are

required,exten;i,,e experience _is required, system personnel are

to' assist in svstem uevelopment, and the system is essentially a

one-shot attempt.

Training is. a valuable (bjut expansive) technique when all of

the performance 'requirements can be specified, a relatively large

rumber of syst-em personnel will be involved, system personnel

will be a permanc-it or 3emi-permanent complement, skill require-,

ments are relativey higjh but not specialized, and extensive

experience is not r••Iired.

A-18
Performance aids and manuals will always be required in

military systems. However, performance aids are particularly

valuable in cases where there is relatively high personnel

turnover, skill requirements are relatively low, task performance

can be specified in detail, and large numbers of system personnel

are involved.

Human factors engineering is also a means of achieving (or

assisting in achieving) personnel performance, as well as reducing

error probability. A system which provides sufficient and

meaningful information to the human operator or maintainer and

provides adequate and compatible methods for responding to system

demands can substitute for selection, training, and performance

aids in some cases. Furthermore, human engineering becomes a

permanent part of the system and the investment is usually made

once, whereas the investment in training must be made over and

over as personnel turnover occurs. Moreover, human engineering

applied throughout a large complex system can make training (and

cross-training) easier, and can make it easier for both a novice

and an experienced individual to operate or maintain the system.

Thus, a significant human factors activity is to determine

what types of human performance can best be achieved or assisted

by human engineering.

No matter what the primary method for achieving human performance is, human engineering will affect the reliability with

which man can perform his intended role, functions, and tasks

whenever he must interface with the system. Data need to be

available to system designers with respect to enhancing human

reliability through enhancing both the behavioral and attitudinal

interface between man and the system--the compatibility factor.

Human Information and Response Reguirements. Finally, as

part of the human factors engineering product a determination

should be made of information and response requirements necessary

A-19

kk
for the user to interface with the system. The terms information

and response are deliberately used in place of displays and

controls since the actual selection or design of displays and

controls should occur after it is known what information and

response will be required of the system user. This will permit

consideration of combining information on a single display or

time-sharing the display or sirAilar considerations which are only

possible after all of the information requirements are known.

The same thing is true for response requirements and the eventual

design of controls.

The human may receive information either from the environment directly or through displays of the machine; he may also

make responses directly to the environment or to the machine

through its controls. In an operating system, all inputs either

to the man or machine may be considered system demands. All outputs from the man or machine may be considered system performance.

A final observation should be made that the man and machine

inevitably operate in some kind of environment. This could be a

physical environment, such as the roadway or the atmosphere; and

it could be an organizational environment such as an aircrew,

communications network, or a management information processing

function. No matter what the context of the enviropm-ent, it

imposes demands on the man-machine system, which in turn responds

to provide systen performance in the operating environment.

Design of Optimal Man-Machine Interface

An optimal man-machine interface design is that which is the

most desirable from the human factors viewpoint while remaining

within the constraints of the overall system design. This human

factors product probably needs the least explanation and the

least justification for being included as part of the system and

A-20

04. .
equipment design decisions during the Full-Scale Development

Phaqe. It is during this phase that design decisions will result

in hardware, software, and procedures with which people in systems

must interact. These design decisions and their compatibility

with the human physically, behaviorally, and attitudinally are

the loci for human errors in system performance. While not all

human errors are disastrous with respect to system performance,

certainly no one would argue that human error must be minimized,

particularly when this can be done at minimum cost before hardware,

software, and procedures are released for production. The answer

simply lies in having available data and expertise which will be

a part of design decisions concerning man-machine interfaces in

the Full-Scale Development Phase.

MIL-STD-1472B is, of course, a fundamental requirement of

complex systems under development and prov.des the basis for the

design of optimal man-machine interfaces. There are many other

widely accepted handbooks or guidebooks which provide information,

primarily about man's physical and behavioral characteristics,

that must be accounted for. However, even at this level of

systemdesign it is still essential to consider the attitudinal

variables.

Through training and experience man has built up many habit

patterns that lead him to expect things to look, sound, or feel

a certain way. Conversely, there is a psychological phenomenon

known as perceptual constancy which allows us to perceive certain

things for what they are, even though they are distorted or

symbolic.. This has relevance in the design of the equipment

interface, as certain types of instrument symbols are more

acceptable than others because they meet man' s perceptual

expectancy or do not exceed his bounds of perceptual constancy.

A practical example of this is the symbolic representation of

A-21

N
the runways as part of a head-up display--some representations

are simply more acceptable than others.

This completes the discussion of the four principal products

of human factors R&D. it is the opinion of the authors that there

is ample rationale and precedent for the establishment of these

products as an integrated part of system development.

A-22
References

Armed Forces Journal International. Don't be hypnotized by the

flashing lights. Vol. 115(8), April 1978.

Baker, C.C., Johnson, J.H., Malone, M.T., & Malone, T.B. Human

factors engineering for Navy weapon system acquisition.

Naval Sea Systems Command, Washington, D.C., July 1979.

Coburn, R. Human engineering guide to ship system development:

Responsibilities, procedures, and methods for carrying out

human engineering programs. NELC/TD 278. San Diego, CA:

Naval Electronics Laboratory Center, October 1973.

Department of the Air Force. Human factors engineering. AFSC

Design Handbook 1-3. Headquarters, Air Force System Conummand,

Andrews AFB, Maryland, January 1977.

Department of the Air Force. Human factors engineering and

management. AF Regulation 800-15. Headquarters,

U.S. Air Force, Washington, D.C., October 1974.

Department of the Army. Human factors engineering program.

AR 602-1. Headquarters, Department of the Army,

Washington, D.C., June 1976.

Department of Defense. Human engineering requirements for

military systems, equipment, and facilities. MIL-N-46855B.

January 1979.

Department of'Defense. Human engineering design criteria for

military systems, equipment and facilities. MIL-STD-1472B.

May 1978.

Department of the Navy. Human factors. NAVMATINST 3900.9.

Headquarters, Naval Material Command, September 1970.

Foley, J.P. Impact of advanced maintenance data and task oriented

trai.ning technologies on maintenance, personnel,' and training

systems. AFHRL-TR-78-25. Advanced Systems Division,

Air Force Human Resources Laboratory, WPAFB, Ohio, September

1978.

Jordan, N. The allocation of functions between man and machines

* in automated systems. Journal of Applied Psychology. June

1963.

Melching, W.H. A concept of the role of man in automated systems.

HumRRO Professional Paper -14-68. Human Resources Research

Office, May 1968.

A -23

..4 04i041 ýrw

'1 U
Miller, R.B. A method for man-machine task analysis. Report

No. WADC-TR-53-136. Wright Air Development Center, WPAFB,

Ohio, June 1953.

Nissen, H.W. The nature of drive as innate determinant of

behavioral organization. In Jones, M.R. (Ed.), Nebraska

Symposiuin on Motivation, 1954. University of Nebraska,

Lincoln, NB, 1954.

Price, H.E., Smith, E.E., & Behan, R.A. Utilization of acceptance

data in a descriptive model for determining man's role in a

system. NASA CR-95. National Aeronautics and Space Administration, Washington, D.C., 1964.

Price, H.E., & Tabachnick, B.J. A descriptive model for determining

optimal human performance in a system. Vol. III: An approach

for determining the optimal role of man and allocation of

function in an aerospace system (including technical appendices).

NASA Ames Research Center, Moffett Field, CA, March 1968.

Swain, A.D., & Wohl, J.G. Factors affecting degree of automation

in testand checkout equipment. Dunlap and Associates, Inc.,

Stamford, CT, 1961.

U.S. Army Engineering Laboratory. Manpower resources integration

guide for Army materiel development. HEL Guide 1-69.

Aberdeen Proving Ground, Maryland, 30 January 1969.

A-24
APPENDIX B

DERIVED VOCABULARY LIST OF HUMAN FACTORS METRICS

System-Related Terms and Associated Dimensions 'Unit of Measurv)

ACCESSIBILiTY suoject;ve: sati-factory/unr,satisfactoty e ase of ad

mission to various areas of an i.em

ACCURACY probability/f;equenci of documented error

CAPABILITY subjective: mission objective achievable given the

condition durinq the mission

COMPATIBILITY subjective: .3bility of items of equipment to

coexist (including effect, of temperature and

moisture

CRITICALITY subjective: relative degree of task importance for

mission success

DURABILITY probability: item will survive

a) its projected life

b) overhaul point

c) rebuild point

without a durability failure (failure that causes

an item to be rebuilt or replaced)

EASE OF USE subjective: tasks associated with simplicity, reada.-

bility, etc.

FAILURE RATE/FREQUENCY 1) number of failed items

2) number of effects (out of tolerances) per month,

week, hour, etc.

FIRING RATE time (measured from inir•, to reloadino of weapon)

HABITABILITY subjective: 3dequacy/eas, of space, transport,

watch standing, rest, relaxation, wfrkspace

and access

MALFUNCTION, SYSTEM' frequency per unit tine (hours) based on availINITIATED' able reliability data & maintenance data

MEAN FLIGHT HOURS mean ixobable flight hours between maintenance

BETWEEN MAINTENANCE actions

ACTION

MEAN-MAINTENANCE TIME 1) mean hours preventive and corrective mainte.

nance

2) total preventive and corrective maintenanCe

time divoed by total number of preventive and.

corrective 'actions during a 'specified interval

(MTBAMA) MEAN'TIME same as MTBF except all mainteanr•e actions are

BETWEEN ANY collected as data

MAINTENANCE ACTION

B.I

• • = I I i I -I I
(MTBF) MEAN TIME 1) mean time .i svqsern function; until occurrence

BETWVEEN FAILURE of a failure requires corrective maintenance

(characteristically over a two-month period)

2) total functioniing life of a population of items

divided by the total number of failures within

the popuilation during a measurements cycle

(time, cycles, miles, events, etc.)

(MTBM) MEAN TIME mean of the distrioution of time intervals between

BETWEEN MAINTENANCE m aintenance actions

(MTBUMA) MEAN TIME same as a~bove except only unscheduled mainteBETWEEN UNSCHEDULED nance is collected as data

MAINTENANCE ACTION4

(MTTR) MrEAN I IME TO total correctiv4e maintenance time divided by total

RiEPAIR number of corre,.,tve maintenance actions

duriig a specified intervial

(MTTRA) MEAN TIME TO total corrective and prevent;ve maintenance

REPAIR (ACT!JAiLY time divided by total number of corrective

ACHIEVED) and preventive rn~ntenance actions during

a -.Vecified interval

(MTTRF; MAEAN TIME TO mean probabltt time spent in flightiine mainteREPAIR IFLIGHTLINE) ;iance before system is returned to a readvfor-operatlion condi'ion

(MTTR,) MEAN TIME TO total corrective maintenance timea dividcd by

REPAIR (INHERENT! total number of correetive maintersance actions

during a specified interval

(MTTR 0 MEAN TIME TO total corrective maintenance time dividet! by

REPAIR (OPERATIONAL) total number of ccrrective, prever-ive; administrative, and suppc.rt maintenanice actions

during a iwecified interal

(OPE RATIONAL) SUITABILITY subjective,:

I? estAblistiment of system operability in

opermitional enviromnent (within stated

constraihts)

2) identificati~on of adequate insyrumentation,

comfort, visibility. handiing, et,!. of systems

by personnel

(PILOT) WORKLOAD subjective: delgrse of of ort required to accomplish

a specific task

PRODUCIBILITY. (T&E application): scibjective ability of differ ences between prototype and production

models to achieve desirable result (as a result

Of E CP & p ~gram change arders

READY RATE, OPE RAT'1 NAL % of assigned itents capablei of perormir-g an

zsigned mission or function

iP-2
SAFETY 1) probability ot injury or damage

2) subjective: satisfactory/unsatisfactory mat.erial,

fire & explosion pr',tec-•on, mechanical

electrical hazards)

SERVICEABILITY time: ability to service :n specified inervr•-'

STANDARDIZATION/ degree of similatity (lack of imbi.guities) of

COMMONALITY OF DESIGN tvo displays designed to same specifications

aad stardards

SUBSYSTEM EFFECTIVENESS tubjective: the technical cazz•ility of a subsystem (RADAR, FLIR, etc...) to accomplish

a specific task

SURVIVABILITY probability that a system will withstand hostile

man-made environment and retain mission

accomplishment capability

TIME, DOWN (DOWN TPME) tiie (hours, frequency, duration) which an item

is not in condition to perform its ipecified

function

TRANPORTAIt ITY subjective: ease of transit, packaging, load/

unloading, security & fastening

WEAROUT rate of increase in failure rate of items over system

life (cycles, time, miles)

B.3 i

- -- •--- 3- -.. -

__ _ _
Pmcnnel-Related Terms and Associated Dimensions (Unit of Measure)

ACCIDENT RATE number per specified number of hours

ACCURACY 1) kill/no kill ratio

2) % correct

3) subjective: associated with cognitive skids

(e.g., observin.;, estiv'ating, detecting, recognizing, positioning, reading, etc...)

4) measure of precision and/or timeliness of

performance

ANXIETY ssbjLective: stress factors assiciated with pilots

(e.g., training, confidemce)

APTITUDE AND SKILL 1) testing wcores (e.g., AFQT)

2) subective: low vs. high

ATTRITION/TURNOVER % attrition-number of attrited personnel divided

by number of attrited personnel plus number

of non vtrited personnel

DISSATISFACTIONS/ subjective: ratings of challenge, ýpersonnel-job

SATISFACTIONS match, perceived degree of utilization

EFFICIENCY rating success on a task

ERROR RATE (ANALYSIS) 1) mean error per performance time

2) percent and/or number of operator error

{e.g., forgetting, iccidents, inability, etc...)

3) analysis: includes

a) amplitude

b) frequency

c) type

d) change over time

ILLUMINATION LEVEL 1) measure: luminance2) subjective: number of lighting deficiencies

INJURY subjective: injury type, severity, frequency

MAINTENANCE, CORRECTIVE number, rate, frequency of acts performed to

restore an item to a specified condition

MAINTENANCE, PREVENTIVE number, rate, frequency of actions performed

to retain an item In a specif led condition

MALFUNCTION, HUMAN frequency of test participant (operator) error

INITIATED resulting in system/item malfunction

(MOBAP MILITARY 1) communications distance (limitations)

OPERATIONS IN. 2) weapons effectiveness.

BUiLT-UP AREAS 3) tactics effectiveness

iB-4
MORALE subjective: ratings Of Midividual personnel identification aid satisfacticn with work group, iob

activities, duties, supervision, etc.

MOTIVATION subjective: rating of desire to perform duties,

obta;ri experience, advance

NIGHT OPERATIONS performance (target identification! in night

missions

NOISE/BLAST sound pressure rneJsurements (e.g.dbs, amplitude,

also velocity, wavelength frequency in herz)

PERFORMANCE TIME mean time/number per some unit/rate

OR RATE

PRODUCTIVITY un';s produced per some interval

PROFICIENCY test scores (written)

RADIATION radiation effects aircrew performance on radiation

environments

REACTION TIME 1) (time reaction): uptime to initiate a mission,

measa'red from the time the command is

receivri

2) operator pirception time (or start time) in

response to some initiating stimulus

STRENGTH amount lifted (kilograms)

STRL3S. GENERAL gas (general adaptation syndroma,

STRESS, TASK OVERLOAD subjective: woikload excessiveness

TASK COMPLEXITY/ subjective:, rating based on knowledge and skill

DIFFICULTY required for performance

TASK DURAT!ON total time required for task completion (also as in tracking targets-% of time on target)

TASK FREQUENCY number of responses made by an -operator(s)

in a specified interval

TEMPERATURE measures of comfort and performance in variable

temperatures

TIME, ADJUSTMENT/ time ,reeiired to make needed response

CALIBRATION

-TIME,CHECKOUT time required to verify performance of an item

(in specified condition)i

TIME, FAULT CORRECTION time iequired to correct a failure

TIME, FAULT (ISOLATION) time (hours) measured from discovery of a fault/

LOCATION failure to correct identifica'ion of failed item

n,--

.. 1

* k.-.
TIME, TASK TIME time required to perform task

TIME, TURNAROUND time required to service or check out an item

for recommitment

USER ACCEPTANCE subjective: underuse. misuse, abuse of equipment

due to dissatisfaction with:

a) machine function

b) status

c) economic fears

d) survival fears

e) enjoyment of manual performance ot tasks

VAPORS/EMISSIONS measured in parts per million (PPM) over specified

time

VIBRATION frequency (in Hz) over a unit exposure time

WINDFORCE (0-FORCE) windspeed indicator (impact on physical operating

environment)

WORKLOAD subjective level of effort required to a•oirnp.i'ti

a task

13-6
APPENDIX C

EXAMPLES OF HUMAN FACTORS EFFORTS RELATED

TO SYSTEM DEVELOPMENT PRODUCTS AND METRICS

Application of the Metrics Approach to the

Evaluation of Human Factors Products

Previous chapters have demonstrated that human factors

products are explicitly tied to the formal sequence of m7ilitary

system development, and that the stages represent a progression

from the general to the specific. It should also be remembered

that each human factors product exists in order to provide a

response to a generic problem in system development.

The mode uf illustration in this section will be to fill in

some of the cells of the matrix in Exhibit C-i with specific

metrics. The intent is to demonstrate that these assignments

can be made in a sensible manner. In several cases, we will

also go a further step and show how empirical measures fit into

the metric level. Again, the objective is demonstration, riot an

exhaustive explication.

Supplementary Evaluation Considerations

An important aspect in the evaluation of the'human factors

contribution to military system development should be the quality

of the human factors products; that is their "intrinsic" merits.

An example is a human factors report that is assessed Qn its,

overall relevance or cogency, the logic of the'derivation of its

conclusions, the validity of the data used in the inferential

reasoning presented, and even its readability.

Our more structured approach is not intended to preclude

such a mode of assessment. Indeed, that kind of evaluation

C-i

\4
0

0Elo

u C IC

CO

00 a

'a

CC

';t ~ C- 2
is virtually spontanFous. We would exýe,•t 1 human factors

products Lo be evaluated in t!,at moxz jiaud_,.entaI mode. Such

judgmental evaluaticis arq useful and neeaitd but they do not go

far enough. Specifically, a rey.-ort zuch a- is mentioned above

could be entirely cogent and valid an,- still not be valuable to

the system designers. Certainly they would not perceive it as

valuable unless the recommeriIdations were seen L.o inahe a positive

difference in the cost, capability or compatibility of the

system. To do so requires the linkage with their concepts of

system impact areas--"caste," in their, (the designer's and system

development manager's) terminology.

Metrics and Principal Human FacLors Products

Human Role x Cost. We have filled-in the cell in the upper

left hand corner of the matrix with the metric of cost associated

with depot maintenance, as shown in Exhibit C-I. The rationale

for this example is that the human role in the maintenance

process is implicitly or explicitly defined when the system

maintenance philosophy is promulgated early in the design process.

A typical parameter at this level is the threshold of the choice

of repairing a subassembly at the operational unit or replacing

it. if the philosophy is to replace most-subassemblies, the

function of repair and the cost associated with that repzirwork

becomes a depot responsibility. The human factor aspect would

be the extent to which quick and accurate fault diagnosis could

be performed at the operations unit level. That outcome, in

turn, would be influenced by the quality of available test

equipment and job aids for operaticnal unit level maintenance

technicians.

Given some actual data on these parameters plus a consideration

of t-he complexity of the system, spare parts availabil.ity, etc.,

and an accurate characterization if the fault diagnosis performance

C-3
of maintenance technicians in similar situations, *the human

factors recommendation might be to work toward the circuit-card

level of replacement as opposed to the subassembly level. If

such a recoxnenuation were valid, it could lower maintenance

costs and could link across to such metrics as system availability

in the sense that time-to-repair could be reduced.

Allocation of Functions x Cost. An example is shown in

which the metric cost associated with a design modification to a

radio device is cast in terms of allocated functions between men

and machine (represented by tihe cell at the intersection of the

Allocation of Function Column and the Cost row). Modern technology

is not prone to simple devices and a "black box" fix can often

add to, rather than alleviate user problems. For example, the

automatic tuner for the AN/GRC-19--a high frequency, AM, medium

power radio--turned out to be. more trouble than it was worth.

Development commenced in the early years of WoIld War II and the

set wis fielded in 1949. Prior to the AN/GRC-19, standard radio

scts required the operator physically to change the length of

the antennae and to go through a series of "dipping" and "peaking"

operations to tune the transmitter to the operating frequency.

A well-skilled operator was required to get the most out of the

radio, and training such operators was not easy. The designers

ef the AN/GRC-19 sought to eliminate this'training problem by

incorporating a "black box", an automatic tuning assembly which.

was expected to substantially decrease requirements for operator

training, as well as increasing the speed and accuracy of tuning.

The AN/GRC7l9ý passed its acceptance tests and was put into use.

But over the years (last procurement was 1965), the Army experienced

a net loss in system cost-effectiveness. While the transmitter

tuned more rapidly, the operator needed as much training to

manipulate the "black box" a• the former manual system demanded,.

arid the training of mainteuance personnel had to be increased to

C-4

• • , , , • .. . . , . •: ' : , .. . •
take care of the tuner. The following data reveal the cost and

other associated effects upon the cost-effectiveness of. having

an automatic tuner added to the AN/GRC-19 radio.

Cost +25%

Size and Weight +15%

Repair Costs +10%

Speed of Tuning Improved

"Operator Training same

Maintainer Training Increased

In sLu, the Army bought rapid tuning at a substantial price in

higher procurement, shipping, maintenance and training costs.

Had the cost-effectiveness of the machine-allocated function

been compared with the original man-allocated version (as was

done retrospectively in the data summarized above), the designers

would have seen no utility in the design modification. (Portions

adapted from TRADOC Pam 71-8)

Task Analysis and Human Engineering Requirements x Cost.

Cost savings are anticipated through increased development and

use of simulators for training and skills maintenance, especially

in the areas of maintenance and flight training. Simulators

have been touted as. capable of reducing or eliminating a need

for operational systems and/or spare parts, since all necessary

functions (e.g., malfunctions) are simulated., In addition, with

increasing costs associated with flight training and air skills

- . maintenance due to energy resource consumption, simulators offer

an economical alternative means for skill development and retention.

* Especially desirable is the maintenance of combat readiness for

pilots in Air Combat Maneuvering (ACM).. The practice required

to reach optimal readiness levels places high cost factors upon

actual aircraft and fuel resources. Use of simulators offers an

IC-

- C-5

I .
opportunity to escape heavy costs and achieve desirable readiness

states. Maintenance trainers such as that developed for the

Heads-up Display tester associated.with the A-6E aircraft offer

an attractive $700,000 savings per copy.

In order to safeguard huge investments in simulator R&D,

much human factors (HF) R&D has been invested to: 1) determine

characteristics relevant to critical performance in the operational environment which must be simulated (through task analysis

and human engineering), and 2) verify the effectiveness of

simulators developed from such research. While the advent of the

age of simulators presents HF R&D with a unique opportunity for

empirical simulator research, the cost-effectiveness of such

research must be demonstrated through actual improvements in

skill proficiency and readiness... While cost savings postulated

through a one--for-one substitution of, simulators for actual

systems results in an obvious numerical cost figure, no accounting

is made of training and readiness losses or gains made through

the substitution. Perhaps more called for is a detailed costeffectiveness modeling approach which takes into account such

benefits as: reduced training time, training experience with

rare or hard-to-duplicate events or contingencies, better

monitoring of student performance by instructors, etc. It is

understood that recognition gained through simulator usage will

of necessity be shared with the simulator design and training

communities, but nonetheless human factors claims a co-equal

share of the responsibilities and benefits. Cost savings

demonstrated through the -use of simulators will to a great extent

vindicate human, factors input and investment in'this technology.

Man-Machine Interface Design x Cost. Cost savings demonstrated

through an increase in the number of personnel made available to

operate a system can be a powerful metric for demonstrating the

C-6

- ., . -.
underlying value of the HF R&D that led to the savings. A

computer modeling procedure? the crewstation assessment of reach

model, was developed to simulate operators in aircraft cockpits

so as to determine, among other things, the percentage of the

pilot population which could safely operate controls as well as

be safely accommodated by cockpit size and arrangement.s. This

model is used to "step-through" various cockpit design configurations, with the intent of evaluating the man-machine match,

before the aircraft and its cockpit are actually built. Use of

this model has increased the percentage of aviators available for

such aircraft as the F-18. This has resulted in the achievement

of subst_...ial cost savings in terms of manpower alone, variously

described as being between $5 and $40 million a year. In

addition, ,other long term cost savings may be achieved, such.

those associated with reduction in aircraft.accidents, possible

redesign/retrofit of aircraft cockpit configurations, and the

availability of a new design evaluation tool for use by industry

on a continuous basis. Whatever the actual cost savings achieved

(and a detailed cost-benefit model may more fully exploit this),

the cost of the 1BF R&D has been fully exceeded by received

benefits to the system as well as reduced costs.

Human Role x Compatibility. Had human motivation been

considered a vital metric in-measures of system performance at

the.-time when large multiple man-machine systems (e.g., SAGE)

first'evolved from manual, to automated operations, user acceptance

would have readily been seen as .a primary component of motivation

leading to mission effectiveness. User acceptance by implication

is heavily compounded with job satisfaction. For jobs to be

satisfying three conditions seem to be necessary: the system

must demand the ooerator to use skills; the job must be meaningful; and the operator iust perceive real responsibil'ity in job

performance. In designing andthinking about our new complex

C-7

-LA
automated man-machine systems we must learn to design for men

jobs that are intrinsically interesting and satisfying.

A researcher who was involved with the SAGE program makes

the following comparisons:

One notices a striking difference when comparing the

behavior of the crews in the old manual Air Defense

Command sites to the crews in the SAGE direction centres.

In the manual site almost every crew member took pride

in his job. I had occasion to visit many of them, and

in every site the crew members to whom I talked would

eagerly go to great trouble to explain'to me the.

intricacies of their job and what it demanded of them

for good performance. A comparable pride and eagerness

was almost completely lacking in the SAGE direction

centres I visited. Men just cannot be proud of something

which bores them.

If we look closely at the job demands in a SAGE

direction centre, we find several striking differences

between it and the manual sites it replaced. First,

for most jobs, skill requirements have been reduced

to a bare minimum. Second, most of the jobs have

become so isolated and fractionated that they have

become meaningless in terms of the overall crew

mission responsibility. One clear-cut example of

this isolation and fractionation will here be given;

there are many others.

Most of the jobs in an air defense kystem involve

relaying information; i.e., information is processed

or acted upon and then relayed to aother position

for further processing and action. Each job by

itself, although clearly defined, enerally has

little meaning when the total pict re of crew's

actions a..e lacking. In the manua sites there was

a central plotting board which sho ed such a picture

for all the crew members to see. o such summarizing

display is available for a member f a SAGE crew.

Third and last, because of the fan astic performance

ability of the computer, because o the inflexibility

of even the most so-called flexibl program, and

because of the mystery, to the cre members, of what

goes on inside the computer, and r inforced by the

effect of the preceding two condit ons, the roles of

a C-8
the human operator in SAGE and the computer have

functionally been reversed. Rather than the machine

being an aid to the man, the man becomes an aid to the

machine. In addition to boredom generated by the

reduction of skill, there is a feeling of futility

generated by the feeling of having lost control over

what is going on. Maybe this is all we desire of the_

men in our emerging complex automated man-machine

systems, that they merely be aids to the machine, but

it is legitimate to raise the question whether this

desire is itself desirable.

In designing complex systems, regardless of our good

intentions, we can often create a situation, that becomes

intolerable for the human being, and as a result he

either leaves the system or, if he cannot, he subordinates

himself to the system and ceases to play the role which is

the ultimate role of men in man-machine systems, to see

to it that the system works.

(from Jordan, 1968).

As was made explicit in the preceding discussion, not only

should the Role of Man be considered early in system development

to include elements of motivation, personnel satisfaction, and

user acceptance; but by direct extension a case can be made for

systematic development of measures such as user acceptance and

metrics such as motivation to document.HF R&D improvements in

systems. Improvements in user acceptance not only benefit man

as a user, but must also ultimately pay off in improvements to

overall system performance as well. This may especially be the

case for emergency and contingency situations in automated

systems (e.g., SAGE).

Allocation of Function x Compatibility. The metric chosen

todemonstrate the linkage in this-case is operability.'The

allocation "problem" is dramatically illustrated by the case of

automated (computerized) landing systems. Such systems may not

be operable because they violate human- factors principles with

respect to user acceptance. Specifically, it has been feasible

C-9

'5 ,,.- - . - ----
for several years to control carrier landings by cihr ters used

in conjunction with some advanced radar telemetry devices. In

spite of their demonstrable accuracy, such systems are not costeffective bec~ause the pilots disengage or under--use them.

All such systems have a manual override feature, for obvious

reasons. Pilots exercise the override feature even when the

system is working perfectly because they cannot bring themselves

to invest their trust in a system in which a "slight" malfunction

could cost their life or career. From the pilot's point of

view, the advantage is not worth the potential cost/risk associated

with a malfunction--particularly if the system might be susceptible

to nonobtrusive malfunctions.

Test data could'show that the probability of a mishap or a

missed approach is significantly reduced by the automatic system.

Thus, from an engineering viewpoint the pilot is wrong. However,

from an outside functional point of view, the system is inoperable

and might as well not be on board.

The empirical measure in this case could have been a survey

of pilot attitudes. Had this been done and a human factors

report been produced when the allocation-of-function decisions

were being made, a system might have been designed. that would

have been compatible with pilot attitudes, operable, and thus

effective. Note that this case also links back to cost considerations. In effect, all the development costs of this system were

lost when it became apparent that the system was inoperable.

Task Analysis and Human Engineering Requirements

x Compatibility. The metrics chosen to demonstrate the linkage

between task analysis and human engineering requirements and

compatibility are: performance and effectiveness. The Army has

shown compatibility factors (e.g., crew turbulence) which have

C-10

4 .ýW\
clear effects upon mission success, armor weapon accuracy, speed

of use, as well as overall crew cohesiveness. Identification of

pertinent design factors at the task analytic and human engineering requirements levels would serve to mitigate some of the

user problems seen in the field. The U.S. Army has long been

concerned with getting the maximum capabilities and effectiveness

out of its armor weapon systems. Much of the capability of any

weapon system is a function of the performance of the crew.en

assigned., Some people in the armor corrmunity have expressed

concern that crew turbulence--the movement of crewmen from crew

to'crew and position to position--may have a negative impact on

tank system effectiveness. Research conducted during recent

years has addressed this notion and attempLed to identify the

relationship between tank 'crew tuibulence and tank crew performance.

Tank crews contain four crewmen, a tank commander--commonly

called a "TC,"--a gunner, a driver, and a loader. For the tank

weapon system to 'achieve full potential, each must perform

effectively in his assigned position. Each duty position within

the tank system requires unique skills and smooth coordination

with the other crew members. The TC must identify and range'on

targets, communicate his findings to the gunner and loader, and

be prepared to guide the driver through difficult terrain based

solely.on voice commands. The gunner's response to the •C's

identification of a target must be coordinated with the loader's

response to the TC's command specifying the type of ammurition

-to be loaded. The accurate synchronization of these duties is

essential.

Three types of turbulence' were identified. They were:

equipment, personnel, and position turbulence. -Equipment turbulence occur3 when a crew is moved from one tank to another.

Personnel turbulence occurs when crewmen are moved from one crew

C-I
to another, but kept in their positions. And position turbulence

occurs when crewmen are moved from one position to another.

Assignment changes which create personnel and position turbulence

are always accompanied by equipment turbulence. From the data,

it appeared that position turbulence had a significant degrading

effect on gunnery performance. However, for equipment and

personnel turbulence, little or no effect was indicated. All

types of turbulence could be minimized if it were possible to

assign each crewman'to a permanent position, tank, and crew upon

his arrival in the unit. However, this ideal procedure is often

not feasible, because a sufficient number of trained TC and

gunner replacements are not always available to fill vacated

positions. Consequently, units must fill TC and gunner positions

from available crewman. To cope with the turbulence required by

the assignment system, a unit may frequently move crew members

up within crews, where possible, or between crews where necessary.

These problems with crew turbulence which have direct effects on

metrics of performance and effectiveness are an example of

compatibility issues which, need to be brought before equipment

design engineers. Human factors personnel need to identify

design recommendatioiis which achieve desirable levels of standardization across crews' positions in order to facilitate (among

other things) cross-training and thus reduce the negative aspects

of crew turbulence.

(Portions irom Eaton and Black, 1980).

Man-Machine Interface x Compatibility. To demonstrate this

linkage we have chosen a particularly challenging example: the

evaluation of the product of'detailed design by the impact area of

compatibility using the producibility metric. This assignment is

again illuntrated in Exhibit C-i. The challenge can be met by the

consideration of the human factors aspect of the production

process itself. What we are saying her3 is that the detailed

C- 12
design must be compatible with the attributes of the workers who

will fabricate the system. Of all the possible linkages between

human factors product, metric, and impact area, this is one of

the more likely to get "swept under the rug" in the develLment

process. However, the logic is not that complicated. For

example, at the design recommendation level in the layout of an

instrument panel, the procedure can involve a design review

against standard industrial fabrication practices that asks the

question: Is there any aspect of the fabrication of this panel

that will require deviation from standard practices? If the

answer is yes, the next question is: Does the nonstandard

requirement generate a possible mismatch between what must be

accomplished and the physiological, behavioral, or attitudinal

attributes of the production workers? Specifically, are parts

involved that are so small that positioning is difficult for

individuals with normal vision and normal dexterity? Does

fabrication involve the assembly of pieces by touch because the

worker cannot observe the back of the panel after a particular

production stage has been reached?

In short, the human factors products at the detailed design

level shou d incorporate not only a recommendation that will

lead to an effective interface between the. human operator of the

system and the control panel, bLt should also include a consideration of the compatibility of the design with the tools,

practices, and attributes of the hiuans who will put the control

pane. together.

As in icated, the main impact can be designated as compatibility but, again, it should be noted that a good deeign in this

instance would have effects in the cost area, and probably other

metrics as well.

C-]3
Human Role x Capability. Automated Battlefield Systems are

being developed at a rate which may surpass the ability of'HF R&D

to offer input to each in sufficient quantity. The Army alone is

procuring approximately 60 new automated .`tical systems in the

coming decade. Role-of-man decisions--mn•e explicitly or implicitly

by design engineers--will have a great influence not only on human

performance capabilities and limitations but •Iso on overall

system and mission capability. To make matte-, more critical,

automated systems have had a history of problems related to man's

role, complexity, as well as system hardware and software

architecture.

As much as comprehensive HF R&D is required to take advantage

of opportunities offered by the use of automated technology, with

the problems just mentioned, a method to assess the impact of human

factors on the course of development for these systems is also

necessary. For example, systems-embedded training within

fielded tactical systems offers a rare opportunity for training

operators to readiness states required for suucessful mission

completion in actual combat environments. The opportunity for

HF R&D to aid in the refinement of embedded training is limited

only by the resources available to fund it. HF R&D may be

invested in: software developments for "canned" scenarios and

training packages, guaranteeing the realistic nature of the

training such as to mimic the expected operational environment

by displaying representative data t'3 system operation; as well

as other areas of concern. Measures of personnel and system

readiness as well as proficiency, productivity (in terms of task

data utilized), and performance time or rate may be used to

evaluate the effectiveness of the embedded training to successfully train operators, and to evaluate skills acquisition and

maintenance by operators undergoing training; they also provide

the capability to verify the value of HF R&D that was invested in

the effort. This component may then be aggregated with others to

C-14
form metrics such as readiness, performance, etc. Finally,

through an analysis of type found in the model described in

Chapter 6, metrics may be merged to assess the impact upon

system capability. Given that the information was available on

the ekample cited, systems-embedded training, as well as other

relevant data (e.g., compatibility of the man-machine integration, feelings of lack of responsibility and detachment from

system performance, effectiveness of manual override as well as

others for each alternative configuration of a system), a strongcase could be demonstrated for the value of HF R&D in battlefield

information systems development.

Allocation of Functions x Capability. That the allocation

of functions between men and machines is dependent upon the role

assigned man is nowhere as clearly demonstrated as when man is

remotely located in the system. Remote location of man in the

system eliminates direct observation and control of system

functions. Also, by implication, remote system monitoring

requires heavy dependence upon automation and communication

links. Needless to say, manual override is totally surrendered

to the system, at least as regards that portion from which the

man is remotely located.

Since military intelligence requirements consist, at least in

part, of locating enemy troops, and identifying concentrations

and types (infantry, armor, vehicle, etc.), the military saw fit

to meet this requirement by engaging in development of Remotely

Piloted Vehicles (RPVs). RPVs function to scout the enemy in' the

battlefield, gathering intelligence data about them through the

use of drone, aircraft (tkte RPV) that carriy video sensors. Of

course, by dppign the human is remotely located in the system.;.

Once the location of man i-n the system is designated to be remote,

man's role would be limited to only those possibilities available
under such circumstances. Decisions :egarding the allocation of

functions to men and machine will te based on this human role

decision, with all its advantages and disadvantages. Therefore,

the mission success of the RPV surveillance depends on the ability

of human operators to (a) detect significant information from

video displays, (b) ignore "noise"' or non-significant video

display information, and (c) distinguish friendly from enemy

forces. It is up to HF R&D at this point to ensure positive

effects upon system capability.

Allocation of functions within human performance capabilities

and limitations becomes a critical concern of system developers.

Care must Le shown in designating man-rated functicns which are

within available human performance capabilities and skill levels,

as well as other more basic concerns such aS perception and

decision, making. Measures of accuracy-in-vigilance type tasks

of the sort to be encountered in actual operation of the system

would be useful in determining human performance capabilities,

as well as tas'- -.. plexity/difficulty. HF R&D may also aid in

selecting aptitL.'.. and skill requirements for RPV operators

through measures such as proficiency.

These :neasures, along with othcrs which represent quantifiable

measures of capability, may be aggregated together in a model

such as that preserted in Chapter 6 to form metrics such as

general skills, performance., and task/workload. These metrics

may then be utilized to determine the impact on capability as a

resulti of this HF R&D investment.

•Task Analysis and Human Engineering Requirements

x Capability,. This cell is filled-in in Exhibit C-i with the

metric "maintainability." The mode of approach is to. show that

the product in this case can be evaluated by me.-l.'-ing that

product against maintainability criteria that link to the

capability of the system.

C-16
Design-for-maintainability is a well established concept in

the development of military systems. In this particular instance,

we can look at something as simple as the design of access

hatches for re-arming combat aircraft. From an aerodynamics

point of view, such hatches must be flush and faired so that the

fasteners and handles do not spoil the air flow in flight.

Consequently, there may be a temptation on the part of the

designer to overlook the fact that the speed with which these

hatches can be removed could be crucial to the combat effectiveness

of the system. Similarly, it can be optimal from a human factors

point of view that removal and replacement be accomplished using

conventional tools, and that the geometry of the fasteners and

the complete hatch be such that it precludes errors such as

mispositioning the hatch upon replacement or not properly

locking the fasteners.

The product, in this case, would be a hatch cover design

that would meet both the requirement for good aerodynamics and

the requirements for quick and easy access and error-free hatch

cover replacement. The evaluation at the empirical level would

be speed and accuracy of performance during re-arm operations in

a prototype test situation. A more comprehensive evaluation

would be one that would reveal the consequences of reduced

turnaround time on the overall mission effectiveness of the

system. This level of outcome can, indeed, be estimated by

impact analysis techniques discussed in Chapter 6.

Man-Machine Interface Design x Capability. Increased

accuracy in airborne weapons use has been a continuous goal of

researchers for obvious reasons. As in so many instances,' weapons

accuracy is highly dependent upon a compatible man-machine

integration. One effort has been to improve the target' acquisition

side of weapon systems (as opposed to the purely engineering side,

C-17.
which involves just machine capability). The primary element of

target acquisition of necessity involves a human performance

component. That component has become more critical in today's

airborne threat environment, which has increased in lethality as

well as requiring aircraft to be upgradad in continuously high

flight regimes'(speed) at the edge of the aircraft's flight

envelope. This has resulted in, a military requirement to aid the

human operator in target acquisition. Addressing the human component in target acquisition has resulted in refinement of

television (TV) and forward-looking infrared (FLIR) systems.

These systems were designed to improve identification and recognition of potential targets, actual target acquisition and,

finally,- the probability of achieving a kill. A critical element

in the viability of such systems is human performance capabilities

associated with perception. HF R&D was required in order to

determine the density of scan lines necessary for optimal target

acquisition performance in one such system. This R&D contributed

to the overall capability potential of the weapon system in-the

following areas:

* Increased'weapon accuracy resulting in higher kill

ratios

* Reduced acquisition time

- correlated with increased probability of achieving

a kill

- reduced vulnerability.,

Taken as measures of performance, these &reas may be aggregated

into a performance metric which may also contribute to effectiveness and system survivability. A model tailored to evaluate the

contribution of this HF R&D to capability added to target

acquisition performance and the overall system could clearly

demonstrate that the HF R&D was instrumental-to this 'system

development effort.

C-18
Summary

As shown above, it is possible to identify a set of metrics

that are demonstrably acceptable to design engineers (because

they are extracted from engineering documentation), and that

also serve to relate the empirical and analytical measures used

by human factors specialists. This circumstance encourages the

view that the human factors products that enter (or should

enter) into the military system development process can be

evaluated in strictly engineering terms.

Several cases were described that tend to verify that view.

What is still lacking is a comprehensive and orderly methodology

for collecting the appropriate data and generating explicit,

quantitative conclusions about the specific value of a particular

human factors product for a particular military system development

effort.

References

Department of the Army. Analyzing training effectiveness. TRADOC

Pam 71-8. Headquarters, U.S. Army Training and Doctrine

Command, Fort Monroe, VA, February 1976.

Eaton, N., & Black, B. Tank crew turbulence. Army Research,

DeveZopment and Aoquisition Magazine, 2'1(2), March-April

1980, p. 10-12.

Jordan, N. Themea in epecuZative paychoZogy. London: Tavistock

Publications, 1968.

C-:19
DI.STRIBUTION

1 US Aw4V 7ESTERN C3MMANU AIINI AIPPL

I 800)A ATIN: DAA6-EI)

I HO, ICATA ATTN: ATCAT-UP-u

? sUR RESEARCH- AND STUL'LLb OfC

1MILITARY OCCUPAT13kAL UlhVLLOPMF.NF UIV DAPC-MS0-O9 RN 652C HOFFMAN BLDG 1

4 OASO (#4h.A ANO L)

I HGOO ATTN: OAMO..RuR

I HO TcATA TECHNICAL LIURARY

I HQUA OUCSPER

I 11SR~ijCU* STC

I HQOO ATTN: UA141-ISI

I USA &VIATION SYSTEM4S CUMU ATTN: ORSAW-LOR

I USA :.ORAUCO,4 ATTN: AMSEL-FA-RM

I USA a&RRAOCOM ATTN: ATl-E-LU-AC

I HE.AUtUARTLRS, US '4ARINL CORPS ATTNI CODIE mPI-aO

7 US Ak.qY EUROPE ANO SEVE.NTuH ARMY

I 1 ST INFANTRY U)IVISION AND FT. RILEY ATTNI AFZbN-OpT-T

I USA ;NTELLIGEAICE tAND StCUNITY COMMAND ATTN$ IAOPSaTN6.T'

2 HO TwADOC TECHNICAL L18WARY

I NAVAi TR4AINING LIUJIPMENT f-tN ATTN: TECHNICAL LIdRARY

I MILI)ARY OCCUPATIONAL UEVLLOPME141 DIRECTORATE ATYNI ATZIt4CR-0SW.N RN 3N33 HOFFooAN BLDG 11

I DATA ANALYSIS OlVISIUN ATTN: ATZI-NCR-Mno HOFFPMAN BlLDG 11

I* USA P41LPENCEN AT TN: L)APCPFOO-T

I USAFACFA CHIEF# )ROANIZATIONAL kFtECTIVENESS BRANCH

1 8Ttt iNFANTRY DIV!SION

I HUUA ARMY FORCE 400LRNIZATION COORDIN4ATION OFPICE

I NAVAl AIR SYSTEM COMMAND

I OCSOw'S (DIST 4) ATTN: OAMO..RUT

I 123D USANCOM RESERVE CENTER

I US AWHY SOLDIER SJPPORT CENTER ATTN; ATSG-HWJ 10R; CAVINESS), I DINC,.TORATE OF. Akvi)R AVIATION ATTN: ATZX-AAD

1 USAA~wMC * FT* KN~OX AVIATION4 DIVISION

1 USA oORCES, COMMAND AFIN - DEPUIV C OFtS FOR WNELLIGENCE

1 JSA pORCES COMMAND AFIJP - DEPUTY CHIEF OF STAPF FOR OPERATIONS

1US Am.AY AIR DEFENSE SCHOOL ATTN& ATSA-OTO

1 DIREcTORATE OF TRAININGi ATINI AbAZQ-T

I OIRE&.TORATE OF CO4tAT.UEVk.LOPMENTS ATTN$ ATLQOG

I HOD)AwCOM MARIN4E CORPS LIAISON LIC

I nERAw.TMENT OF THE ARMY US ARMY INTELLIGENCE * SECURITY COMMAND

I US AwMY SAFETY CENTER ATTN$ LJVRWARIA4 , RLOG 4905

I USA MoISSILE COMe'AW ATTN: 014S.41NTN

I ARTAsiS ATTN$ URCOM-T.OS-TU

1 USA s-OI4CES COMMAND

I1PM TwAU).

I (IS MILITARY DISTRICT OF WASHIN6OTN OFC OF EQUAL OPPORTUNITY

I NAVAI. CIVILIAN PE4SONNEL WHMO SUUTHIEIN 'FLO DIV

2? ARI ilAISON OFF Il.E

1 ?TH .RMY IWAININti COMMAND)

I mw Us~AREUI4 ATINI UOSUPS

I HNJI)A*OCb STUDY 3FFICt

1 IJ.S. NAVY TRAININJG AN^LYSAS tvALUATIAI GROUP

1. USACssEC ATTN$ AfEC-LX-h HUMAN4 VACIORb*

i JSAFs-.bOS/TAC btNIUR ARMY AiOVISOH

I INI~m-UNIV SLO41NAR ON A90MLU FUNC(.S * SOC

I. USA pL.ECTWONIC PRROVING -4jRUUNO ATTN: STEEP-MT-CS

I OASA (ROA) DEI'tily F'UR SGIENCE ANO TECHP40LOGY

I oFC 1,F NAVAL RESEARCH.

I AFt4Rs hL I4

I AF'4R, ,LWLu

1 AlI1 ..ORCE HUMAN 14!bOuRCLS LAts AT1Nj AFHPL/,TS1M

I FEi:Eo.AL AVIATIUN AUMINISIRATION4 CENTRAL REGION LIBRAnve ACl-H
1 AFAM.4L/86

I AFAMpcL/HE

1 NAVAo PLRSONNEI. N AND U CLNTER f.OKM4HAd AND SUPPORT SYSTEMS

I NAVY PERSONNEL R4 AN) L) CEN~TER/

I NAVY I'ERSONNLL R AND UI Ct~NIEN bL.LCrJH (vF PkOGRAMS

I NAVY P'ERSONNEL W4 AND U CtNTEW

I US AmMY AVN ENUIN--tHINj, ýL1GIT ALTIVITY ATTNZ DAVTE-TD

? DFC tuF NAVAL REStAi4CH PER4SONNEL AND THAININb. RESEARCH PROGRAMS

I NAVAs PERSONNEL N 0 U CLNTLR /

I OFC oF NAVAL RL~tARCl PROJECT UFFICER9 ENVIHONMLNTAL PHYSIOLOGY

I DE'PT. OF NATIONAL DJEFENCE OLFFNL.E AND CIVIL INSFITUTE OF ENVIR MED

I NAVAs AEROSPACL MEUICAL HSLH LAO AEROSPIACE PSWCIIOLO6Y DEPARTMENT

i USA iRADUC SYSIkMS ANALYSIS ACTIVITY ATTN: ATAA-TCA

I HEAOwUAWTLPS* LU.ASI bOUARD ChIft# PSYCHOLOGICAL '4SC'I AR

I USA mESEARCtI ANI) TECHNOLO#IY LAN ATTN: UAVOL-AS (OR* Ro S. DUNN)

1 USA P-NGINEER TUPOUI4AFh1C LABS AITN: E1L-GSt.

I USA tNbINE.fR TOUPSRAPHIC LAOS ATTN: 5TINFO CENTtR

I eSA i.NGINLkE TUPUSHAPILC LABS ATTN: E1L-TO-S

I USA NOBILITY LWU12MENT R4 ANU U LOMW &ITNI DRUNE-TO (SCHOOL)

I NIGNI VISION'LA.S ATTN* LINSEL-MV-SDD

1 USA IRAINING HUAICJ ATTN$ ATTtb.ATOTA

I USA mUMAN ENGINEtRINb LAW

I USAHPL LIAISON RLP9 USAAVNL I1

I USA MATERIEL SYSTEM4S ANALYSIS ACTIVITY ATTN: ORXSY-C

I USA wESEARCH OFC/

I NAFEc HUMAN ENGI4IERINO UNANC"

I RAFTh-LLE-COLUMUUS LABORATORIES TACTICAL TECHNICAL OFC

1 USA oRCTIC TESI CEN AlrNI AMSTL-PL-TS

1 USA cOLD REGIONS TLST CEN ATINS STECA-OP

1 USA CONCEPTS ANALySIb AbCY ATTN$ CSCA-RQP

I11,SA .tONCEPTS ANALYvSIS *blw ATTN$ CSCA-JF

1 HQ OWAIR DIV UF '4EUPOPSYCIIIATPY

1 USACmCDA ATTN: ATIL-CAC&IL

I USAC.'CDA ATTN: ATZL-CAC-IM

I uSACA.C ATTN: ATIL-CAC-IA

I USACA&CDA ATTN: ATLL-CAC-A

1 USA PLECTkONIC W *ARFARE LAD -CHILF9 INTELLIGENCE MATER oEVEL *SAJPP Off

I USA ovSCH UEVEL -STANDARDALA OP9 Ue.s

I AFwA$/FIGR (COIC)

JItiSA wESEAI4CH ANI) OVELOPNtNT. LAWS CHIEF, BEHAV SCIENCES OIV4 FOOD SCI LAB

I TRAJ&NA ATTNf SAJS-Ok

I NAVAi AIR SYSTEMS COMMAN 0 JTIN3 A114-5313

.1 ECUM ATTNt AmbtL-CT-O

1 USACUIEC TFCHNICAL INFURM4ATION CENTER

I USAAwL LISRAkv

I HUMAN RESOURCES RSCH ONO (HUMRRO)

I St.VjeLE RESEARCH CORPORATION

1 USA IRAOOC SYSTEMS ANALYSIS ACTIVITY ATTNI ATAASL (

T ECH LISRARY1

I UNIF'lRME0 SERVICtS UNIT Of THE .stAL 1H SCI DEPARTMENT OF PSYCHIATRY

1 USA BompUTER SYSTEMS COMMAND ATTN3 COMM0AND TECHNICAL LIBIRARY M-9

I "UMA-# RESOURCES NSCI. ONO InUMRRJ)

I NUMRwQ LIBRARY

lEUSTiS OIRFCTOkAli, USAAMML.L TECHNICAL LIBRARV

I RAND CORPORATION f

I RAND CORPORAtION ATTN: 1L10RARY V

I FESIEwAL AVIATION AUMINISTRATION ATTN; CAMI LIGRARN ACC.4*O1

1 NAFEC LIBRARY. ANA-64

I GRONoNGER LINRARY ATTNI ATLF-fibL SLOG 1313

I CE,44l. FON NAVAL ANALYSIS

I NAVAI HEALTH RbCH CEN LIWU4ARY

I NAVAf ELECTRONICS LAW ATTN$ WFbtARCM LI;RARY

I 'NAVAv PERSONNEL N ANt) U CtN L114NARY ATTNI CODN P106
I AIR~ o.OCE HIUMAN WESOUkCt5 LAd AiTN: AF'Idn~L/OTS

I NO* P.T. HUACHULA ATTN: TtLH WEF UIV

I USA mCADEHY OF HEALTH SCILNCk.S STIMSON LIBMARW (DOCUMENTS)

I SCHOoL OF SYSTEMS AND L(MtibTICS/

I IJSAMo kOC TECHNICAL LlbHARY

I OEPAwTME.NT OF fIML NAVY ThAININb ANALT'SI5 AND EVALUATION rap

I NA~ltiu-AL CFNTEII F~k HLALIPI STATISTICS I

I US.4A DEPT OF dI.HAVIUHAL SU.. ANU LLAOEHSHIP

1 OLi) ,aOMINION UNIVERSITY PLRPUIMANCE ASSESSMENT LAWORATORY

1 USA COMMANO AND U.ENENAL SIAFF CULLEGiE ATTNS LIBRARY I IJSA 114ANSPORTATIU'4 SCHUOL USA 114ANSP lICCl INFO ANO MSCH CEt4

I NA~iAH /

I NP4k4D PNQGRAN rMA14AbkM FUN HUMAN I'EHFINMANCE

I NAVAI MEDICAL K A4E) U COM14AND **,)

I USA &IJNINCIhN TEC4INICAL WtbEAHCH bkANCH LIHRARV

? HQ')A USA MED HSC'4 AND ULAVtL COMMAND

IUSA i-1ELO ARTY HI) /

I NAT tLEARINDIIOUtiE FOM AENIAL NFAL1tI P4Fv) pAwKLAWN ULFDG

I U OF TEXAS CEN FOH COMMUNICATION I4SCrI

I INSTITUTE FOW DEFZNSL ANALYSES

I USA tHAINING SUPW3NT CLNIEM ATTN: ATIC-D):T-PA

I AFHWB Tt.CHNOLObY 3FC (H)

I PURNkiE UNIV Dtpr OF PbYCIIULObILAL SCIENCES

I UiSA PoUdILITY Ew'Id'-MENT R AND Ui LUMMANO) ATTN: fURUkNL-iG

I NO. aoSA MUW AIIM: ANPE-OL

I DA U, ARMY RETHAIV.IN(. bot RESEANCH # EVALUATION DIVISION4

I CALSwAN HUMAN FACTOKS Ahu TRAININbi CENTER

I USA .EROmEnICAL RESEARCH LAO SCIENTIFIC INFoRMArIOA CENTEN

I USAF SCHOOL OF At-10'3'ACE ML.OICINt AER044EUICAL L16PRAW cTSK%41

I IIS MOLITAR'Y ACAUE4Y ULI"T. OF HISIONY. 13LOG 601

I USA iNTELLIGLNLE. CEN. ANI?) bLM AJIMI SCP4OOL LIOWRAIY

1 USA iNTELLIGENCE CEN A~-w':LM AIINS ATSI-UT-OL

I MARI.VE COkfiS INSTITUTE

1 NAVAl SAFETY Cti-4TER /

I USAAvC AND FT. WJCKLII AINIM ATLO-ES

I US A-My AVM INI, LIBRARY A)TN1 CHIIEF L113RARIAN

I USAAvNC ATTN: Alto-I)

I US MLITARY ACAOE'4Y LJIRELIOR Of INSTITUTIONAL RSC1M

USA &IN DEFENSE bCHOUL AIIN: AISA-CUMS

I iiSAAesS-L IbRAkV-tUCUMEkfs I IJS.a ilk DEFENSe. W)ARV AT'iN3 FTLES REPOSITORY

1'USA INFANTRY OUAW) ATTN: AJll-lb-AE

1ltISA~iNTELLIGENCt~ CEL AND bCH LU1h: ATSL-UOTU-SF

I USA toRDNANCE CtN AND SCh ATTN$. ATSL-ID-TAC:

I OSA 8 RMOH SCHOOL ATTN: AILK-TO)

I oS54 oqm'Jw CENTEM DINILCIUMATE Of COUMBAT D)EVELO'PMENTS

I NAVAI POSTcHWAOUATE SCH 'AlIMI WjULLY AMOX LIBRARY (COOf 14261

.1 USA IMANSiPOWTAIJU-4 SCHOOL DEPUTY ASST. COMANAMIANT EQUCA. TICHNtOLOGY I USA -.IGNAL SC'4UUL AMU FT. BIONUON ATT148 &tZMET

I USA aRMOW CENTtk # FT. KNUA OFFICE OF.-ARMOR FORCE NOV S TANDARDIZATION,

1 CHIE1. OF NAVAL kDJCAT IOk AhD INI /'

I USA %IGNAL SCHOOL 9 FT. IUOMOON EDUCAIIONAL TECHNOLOGY DIVISION I NU AiC/XPrfi* fRAl'INI, SYSTEM.S I)tVELOPOLNT

I IISAI..o ATTN. ArIlE-UT

I US A.wMY AMMON CENTER ATTN: ATIK-TU-PHJ;

1 USA WUARTERMASit" SCHOOL uIRECTORATE OF TRAININU !DEVCOPW4ENTS

I USCAT iNANSIDORTAIIOM 'AOAAMYUA SCHOOL DIPFCIO14ATE OF WkAINING 4 DOCTRINE

I u.SA INFAWfIY SCMOUL LIkIRANY .4

1 USA WiMANiiY SCt4OXI AFTNI ATSN-IýV

I I)% LAw INFANTMI SCHOOL ATYMI AISM-CO I I!S.& I0FANr~y SILMUX ArmNS ATSM4-UOT-LPIU

V-3
I USA iNFANTRY SCriUOL A'TTN- Al!S.4tV

I USA .P #CHEM SCH1/TNb CEN FT. MCCLE~LLAN AITN: ATIN-PTS

I USA A~P # CHEM bCHlNij CEN * PT. MCCLkLLLAN 0114. COMBAT DEVELOPMENT

I U)SA AP # CHEM Srtl/lNb CEN * Fl. MCCLELLAN UliM TRAIN!.IG UEVELOPMENT

I US4 mnP # CHiEM*bCH~/ 'TNbj CLN # Fl. 'MCCLELLAN ATTN: ATZN-mP-ACE

) USA INSFIJUTE UF AU04INISINAYLUN 'ATTN: iRESIULNT TRAINING MANAGEMENT

I USA tIELU A.R1ILLLQY SCHUOL 140WRIS SWETT LIBRARY

1 USA slNSTLILTL Uý AUMINIS.TWATILSN ACADEMIC LIbNARY

I USA wAR COLLEbt A(TN: LIbNAWY I USA PNG1NEFiW SCHiUX LINNAAaY AlNu Lk.AHIING RESOURCES CENTER

1 USA ARMOR SCHOOL (USAWNM,) AIINZ LIURAWY I

I OR(,AamdZATIONAL EFVLCTlvtNtSS CEN # SCHs ATTN: LIBRARIAN

I US Aim"Y INTELLisL4ICE CLNTLN # SCHOOL ATTN: ATSI-TI

I US AmMY !4T-ELL'I~xE,E'CT CtNILM * SCIOU0L ATTN: ATSI-RM-M

I US AwMY IN.TELLILP.'4CE CLNlLW * SCHOOL ATtNl ATSI-TO-LD

JýUS AmMy INTELL16.L4dCE Ck.NTLI * SCHOOL AITTh ATSI-CO-CS-C

I US'AsoMY INIELLLst.'4CL CENTtH * SCHIOOL ATTN: ATSI-DI-SF-10M

I O)EIPA.TMENT OF TaiL At" PUNLL AIN UNIVER4SITY LIBRARY (ATC)

I MQ TaiAOOC. TRAIN144, ULVELUP0Mt17 &NbTIFUTF

2 BRITiSH EMRASSY 4wIT15" uLFLN4Ct STAFF

2 CANA#A#IAN JOINT STAFF

I COLS (w) LIHRAWV,

1 FREN0.4 ARMY ATTACAE

I AUSTwIANEMBASSY UEPENSt., MILT JAIY AV4 AIR ATIACH(

3 CANAsoIAN UEFENCL LIAISON STAFF ATTN$ GOUNSELLOR, OEFENCE R AND D

1 ROYA4 NETHERLANDS EMdbASY MILITARY ATTACHE'

I CANAs'IAN FORCES MASE CORNWALLIS ATTN, PERSONNEL SELECT:ON

7 CANASIIAN FORCES PERSONNEL APPL N45CM U4lT

1 ARMY PERSONNEL RtSEARC" tTAISLISMt.NT

I NETtHPLANUS EMtlAbSY OiFF1LE OF ITHE All ATTACHE

6 LIOWARY Of CO~ibwLSS LACi1ANGL ANti 61Ff OIv

I uEFENiSE TECmNICAL INFORMATION CtM AITNt oTIC-QoA-j 140 LIH~R&WY.OF CONUaMESS UNIT UOCUmt.NTS EXPEDITINII PROJECT

I US GisVERNMENT PRIVeING, OFC, LikkARY9 PUBLIlC IJOCUM4ENFS DEPARTMENT

I US OuVERNMEN1 l'014TING OFC. LIe04AiY AW4 STATUTORY, LIB DIV (SLL)

I THt. &RMY LldI4AMY ATTNS ABIMY SiULDILS SEC

3 ROYAs ARIMY EUUCATIONAL CORPS CENTft. -AMY SCHOOL Of TRAINING SUPPORT

NUMV.Z4 uF* AOURESSt.ES .22

TOTAL Ni#MdER OF COooIES 40Z

D-4