Research Note 85-17

An Analytic Process Model for

Systems Design and Measurement

Richard F. Bloom, John F. Oates, Jr.,

John W. Hamilton, and William A. Leaf

Dunlap and Associates, East, Incorporated

-- a" l ARI Field Unit at Fort Benning, Georgia

! Seward Smith, Chief

i!. Training Research Laboratory

Harold F. O'Neil, Jr., Director "o'

DC

NA

-' U. $. Army

Research Institute for the Behavioral and Social Sciences

February 1985

AppHroved for Nublic r distribution Jleae; unlimited 08 45 10 16 10_o8

C--k

-:dl

> .... 1.. ...;., . ,:...;..4">'. App...,. d for. . pu li . . ., ditib -,ess to u.ie....,: . , .. , .. :., . . ... . ,.-.
... 

THIS DOCUMENT IS BEST 

QUALITY AVAILABLE. THE COPY 

FURNISHED TO DTIC CONTAINED 

A SIGNIFICANT NUMBER OF 

PAGES WHICH . DO NOT 

REPRODUCE LEGIBLYo 
U. S. ARMY RESEARCH INSTITUTE

FOR THE BEHAVIORAL AND SOCIAL SCIENCES

A Field Operating Agency under the Jurisdiction of the

* Deputy Chief of Staff for Personnel

L. NEALE COSBY

EDGAR M. JOH-NSON Colonci, IN

Technical Director Commandcr

Acresion For

-T 1-C 7 2

UI tV t1 -

Tso reort. be osuted as thn onracia, heasbmen ofeahed r polsito policy, orTechisinesmaion einte

by other official documentation.

EL
1y-.

: Unclassified S CRIT'- CLASSIFICATION OF THIS PAGE ("en Date Entered)

REPORT DOCUMENTATION PAGE READ INSTRUCTIONS

REPORT__ DOCUMENTATION__-,,,, __ BEFORE COMPLETING FORM 1. REPORT NUMBER 2. GOVT ACCESSION NO.1 " RE IPIENT'S CATALOG NUMBER

ARI Research Note 85-17 ,F h P

4. TITLE (and Subtitle) 5.TYP 'REPORT & PERIOD COVERED

An Analytic Process Model for Systems Design Final Report

and Measurement

6. PERFORMING ORG. REPORT NUMBER

293-21 7. AUTHOR(e) I. CONTRACT OR GRANT NUMBER(*)

Richard F. Bloom; John F. Oates, Jr.;

John W. Hamilton; and William A. Leaf

9. PERFORMING ORGANIZATION NAME AND ADDRESS 10. PROGRAM ELEMENT. PROJECT, TASK

Dunlap and Associates East, Inc. AREA & WORK UNIT NUMBERS

One Parkland Drive 2Q263743A794

Darien, Connecticut 06820

il. CONTROLLING OFFICE NAME AND ADDRESS 12. REPORT DATE

U.S. Army Research Institute for the February 1985

Behavioral and Social Sciences 13. NUMBER OF PAGES

5001 Eisenhower Avenue, Alexandria, VA 22333 191

14. MONITORING AGENCY NAME & AD.9RESS(t different from Controlling Office) IS. SECURITY CLASS. (of this report)

Unclassified

ISa. DECL ASSI FICATION/DOWNGRADING

SCHEDULE

. 16. DISTRIBUTION STATEMENT (of thle Report)

Approved for public release, distribution unlimited

17. DISTRIBUTION STATEMENT (of the abetract entered in Block 20, it different from Report)

IS. SUPPLEMENTARY NOTES

Seward Smith, Contracting Officer's Representative

IS. KEY WORDS (Continue on reveree aide If neceeary and identify by block number)

Human machine system; analysis; taxonomy; model; training systems; design

requirements; design specification; evaluation performance measurement;

effectiveness measurement; system populations; Bradley Infantry Fighting

Vehicle; BIFV; handgun; systems taxonomy model; computer-aided model;

interactive automated model.

20 ARSTAACT (fCQ0time on rew wao s if Ne iNp ernd dentify by block number)

The objective of this model development effort is to provide a uniform, thorough,

. adaptive and efficient procedure to help derive design specifications and

%F. effectiveness measures for any-planned or existing human-machine system, especially a training system. The-bsenr report describes results of the second

year of work, in which the analytic process model (APMf)was developed in great

-. detail, applied in a sample fashion to an existing system (the Bradley Infantr

Fighting Vehicle Training System) and a newly planned system (the Army 9mm

. Handgun Training System, and placed onto an Apple II computer for demonstratin

DD IM 147 01I TON OF I NOV 65 IS oa$OLETE

A 1~4 Unclassified

SECUOITY CLASSIFICATION OF THIS PA.xE (Whlren Date Entered)

- Li
Unclas sified

SE11tUMITY CLASSIFICATION OF THIS PAGE(Whn D e ERI ,

_---)its potential for computer-aided" applications. The sample applications and

procedures are documented, and recommendations are made to complete developme lt

of the model, its application procedures, and the manner of its dissemination

to Army users. /c .Lj - /j

I

Io.,

Unclassified

SECURITY CLASSIFICATION OF THIS PA~GEW"On DOM. Entere)

.i%

*.*-,
ACKNOWLEDGMENTS

Many individuals helped the authors during the course of this project.

In particular, we wish to acknowledge the invaluable technical assistance

provided by Mr. Hal C. Strasel and Ms. Dorothy L. Finley, of ARI, and

Dr. Edward Youngling of Litton/Mellonics. Constant support and assistance on j technical and administrative issues were generously provided by Mr. Edward

W. Bishop, the Responsible Officer for this project at Dunlap and Associates,

Inc. Finally, we acknowledge the help provided by the Dunlap technical Slibrarian, Bernice Astheimer, and the Dunlap technical assistants, Frances UKowaleski, Mary Ellen Pavlech, Karen Schoelch and Janet Vartuli.

.1

il
TABLE OF CONTENTS

Page

I. INTRODUCTION 1

II. THE ANALYTIC PROCESS MODEL (APM) 4

III. APPLICATIONS OF THE APM TO TRAINING SYSTEMS 12

A. Training Systems in General: Axioms

for APM Application 13

B. Taxonomies of Training System Performance 27

C. The APM for Measurement of Training System

Performance/Effectiveness 51

D. The APM for Development of Training System

Requirements 94

E. The Computer-Aided Model 108

3 IV. APM APPLICATION PROCEDURES 118

V. RECOMMENDATIONS FOR FURTHER DEVELOPMENT

AND APPLICATIONS 133

VI. SELECTED BIBLIOGRAPHY 141

APPENDICES 143

A, Examination of Design Subsystem Performance IPotentialities Taxa to Assess Their Relevance

to Evaluation of Learner Testing Specifications A-I

I B. Application of the APM (Measurement) to the

IFV Training System B-1

C. Documentation for the Computer-Aided Model C-1

I, V

I--

3
·.~ ~li 

.. ::: m 

.til 

·~I 

,, 3·1 

. ~~ 

. ' ·.;WI! ~ lr · )•:;; 

t< ,.~ ' 

I, INTI~ODUCTJON 

Thif; ·port deHcrlhcB rcsult.s of the second year's work on an ann1ytlr. 

pr·oct ~•R nrndel (.A. PM) fol" systems design and measurement. The work was 

<'•'l~>:pl.~h·d f,v Dunlap and Associates, Inc., under contract MDA903-80-C~0)1S• 

hr-twt~~·n t!n· P.S. Anny Rcse<lrch Institute for the Behavioral and Socta1 

r; .. ·itr u.·~· ( f'o1·t Bc.•tming Ficlcl Unit), and the Mcllonics Systems Development 

'!ivi tun uf Litton SyAtcms, Inc. Task 3 (System Development and Evaluation~ TPclt ·nln1• y) of tlr:lt contract wns pedormt"'cl by Dunlap under a subcontr:1r.'f ,.,· 

( ~1o. U')L:'!\) fr·om Litton Mcllonics. 

lhll"ill)' thr· second yoar this task h;ts been directed primarily toward 1 '•rn<Jnstr·:·.t.ing t!Ltt the APM, as orlgin"lly designed !or systems measurement. 

:nuld be :rpplic·d successfully to tl1e analysis and evaluation of training 

;·,rst<J.J<. Thi:c; h;J!l been done in the context of the Bradley Infantry Fightin~ 

•.'t~hic1e Tr,lilliLg System (BIFVTS). The project staff also explored the use of 

~ e :\ ['t.l, ;mr1 its component systems taxonomy model (STM), as a dP.sign tool 

nr t l l: ~;p. ~d fir:• tion of training systems required to support developing 

:·.·:btCI'I.';. 'i !:is dcnign application of the APM was examined in the context of a 

tt·;dnL·f~ ~;;':tern for the new (to be selected) 9mm Army handgun. The third 

rn::Jor" .trtivi: v during year 2 was to examine the potential for computer-aiding 

lhe A::'~J :tppiicalion procedure. For that effort, a sample application of the 

P Phl was programmed on an Apple II computer, in order to demonstrate an 

ability in automate the routine procedures as an interactive analytic process, 

thet·r·l y making the APM application a cost-effective and efficient technique. 

Tf1,; rationall~ l1chind t.bc work on this project rests with the problem that 

~~ .. IC"t·s, a:-Jalyzpr·s and researchers too often use an incomplete or inappropri:tte 

H~l of human performance measures in evaluating or specifying· a 

lnnnan-rn:lchine (e.g., training) system. Those are usually known measures 

oftt·n sc ~~ctcd \l.'ithout adequate consideration of the sys-::em context, which may 

nn: help clearly assess !1ystem performance ar ·' may not provide adequate . 

an \vcn; to the e:,sential questions about systcn, f{ectiveness. Because there 

is no veri Ji,~d analytic process for deriving (, specifying} the optimal 

rnr:1•::u'"f'S of ;1 system's performance or effectiveness, true assessment ueeds 

an difficult to define and the process is relatively haphazard. The- typical 

r,ol11ti<•n i:; 1~.~ test/measure/specify the easy and accessible system points, but 

nrd r'eces:~at·ily thos£! that should be addressed. Without having more 

sy~.:tenw I ic: procedtncs, people measure or specify what can be counted (e. g. • 

pr.); h~nr·s), or observed (e.g., number of troops trained). They desl~n 

w1·•tten lt·nts to assess. facts rather than understanding, and use crll(1rta such 

;u: ('nd-d-cour~;e tests rather than on-the-job performance. Rarely do people 

k:1o\v tl•c relationship between test performance a.nd job performance, m· 

!)1~ 1 wN~n soldic~r job perfr;rmance nnd unit effectiveness. 

'*(;(';!·~T"'J'itlc: "l~ffcctivcncss of lnfantry Systems: TEA, CTEA, and llumnn 

1:.1c!nrt·; in SyA!erns Development and Fiolding. 11 

1Bbo·1, n.F., Oatci', J.F., Jr. and Hamilton, J.W. An Initial Analyti<: 

Proc•·r::. Moch'l for Svstems Measurement: F.xtcnslons of the 

t~c;-;:r T:-·-rr:il-u!n·,c"'Tt Dunlti'p ·ancri\~soc atcs, 

'!rum·~ .:try n~port) • 

·>:::,3T AVAILABLE COPY 
POI: hours),* or observed (e.g. , number of troops trained). They design written tests to assess facts rather than understanding, and use criteria such

aend-of-course tests rather than on-the-job performance. Rarely do people

kno the relationship between test performance and job performance, or between soldier job performance and unit effectiveness.

The results of those inadequacies of the measurement determination

process are the wasting of valuable resources (time, effort, talent, money),

- the failure to provide adequate answers to effectiveness questions, and the

~ relegation to obscurity of the elusive questions regarding human contributions

to system performance. A better method is required to decide what should be

measured and how. Hence, the development of this analytic process model for

system design and measurement.

The APM is intended to enable testers, analysts and researchers to define I system factors or taxonomies, and to translate taxa into measures or design requirements. The model forces one to describe the system of interest and its - human elements in such a way as to suggest a more complete set of

system-human attributes pertinent to system effectiveness. The model aids the

users by providing general "menus" of factors (taxa), and procedures to help

translate those taxa into appropriate measures or design requirements. The

APM for measurement is illustrated in Figure 1 (the APM for design has not

yet been defined this completely). The five blocks (numbers 1 through 5)

across the top of Figure 1 comprise the most highly developed portion of the

model, and are often referred to as the systems taxonomy model (STM). During this second year of work, more development effort than before was

applied to blocks 6 and 7 in an attempt to implement more completely what is

considered to be the basic mission of the model under the present

contract -- routinizing the derivation of appropriate measures and design

requirements. The remaining blocks of Figure I (numbers 8 through 17) have

~ 3 received little attention in this study because our present objecti'~es of

developing procedures for deriving the required information are met if one can

coplete the model steps through block 7. The APM for design specification is

most similar to the measurement version (Figure 1) in the early stages

(taxonomy development), and becomes less similar in the later stages.

Details of the APM and significant accomplishments in its continuing

development and application are described in the following chapter. Specific applications of the model to training systems, and their measurement and initial ~ I design are described in the later chapters. Following that is a description of

* the computer-aided application of the model and future plans.

-2-
*SYSTEMS TAXONO MODEL (STM)

%TAONOINOTAONOIS 

TAXONOMIES TAXONO - IES

P-iRNAC B ERFORMANCE PERFORMANCE ENVIRONMENT GENERA,. POTENTIALITIES PROCESSES PRODUCTS SPECIFICATIONS CONSTRAINTS

PRACTICAL

,,~MSUAFICE

FOA

II

MEA-UR-

(DSPCII

PEFRAC .ASURE- ME. TESTi:K
II. THE ANALYTIC PROCESS MODEL (APM)

A brief summary of significant APM concepts, definitions and development

milestones art! provided here as a departure point for the reader in reviewing

the results of our second year's work.

A. Initial APM Concepts and Definitions

The initial work on manned syftgm measurement and the systems taxonomy

model by Finley and her colleagues ' indicated that certain prerequisites exiF;+

for including "system" factors in manned system performance measurement.

They are: (1) recognition of systems as viable entities in and of themselves;

and (2) development of conceptual tools for the purpose of grouping systems

into populations, defining these populations, and placing them into a context

with other populations.

In all cases, a taxonomy supplies knowledge that is specifically relevant

to ti.e anal-;tic application at hand. Thus, each system taxonomy is unique to

the particular system and to the particular context and purposes in and for I which the measurement or design process is to be applied. What Finley et al.

sought was a systematic way of generating such taxonomies for any given system and analytic purpose. Development of the STM and its encompassing APM by the Dunlap staff in the current project was intended to help meet that

need. A description of work 3 completed during the first year of this project is

found in Bloom, et al. (1981) , and is summarized below. The reader will note I that the emphasis in this early APM development work has been on the STM

portion. Later stages of the APM are addressed in the second year's effort.

First, system populations can be formed according to a contex t factor

consisting of the following: the kind of system each population member is

supposed to be; the kind of mission or job each member is supposed to be able

to do; the circumstances, conditions, and constraints under which each member

is supposed to work; the specific requirements and criteria each member is

I supposed to ; etc. Each such population immediately suggests design

and r:,easurement issues, such as: Is the system what it is supposed to be?

Can it do what it should? Will it work where it is supposed to?

11 Finley, D.L., Muckler, F.A., Gainer, C.A. and Obermayer, R.W. An

*: Analysis and Evaluation Methodology for Command and Control: FinTF

Technical ort . Northridge, CA: Manned Systems Sciences, Inc., INovember 1975.

Finley, D.L. and Muckler, F.A. Human Factors Research and the

Devel ment of a Manned Systems Applications Science: The Sysm Sampling

WI problem and a Solution. Northridge, CA: Manned Systems Sciences, Inc.,

3july 1976.

Bloom, et al., 1981. Op. Cit

J

I -4-

... .... :,. ., ,:-.I:.. .:2..::.,:..: :. . .. - 4 ,. . :: . -: . . : : :. :. .. . . . . . .
A second factor by which any system can be examined is that of I -descriptive level, which refers to the several levels of increasing detail into

which a Eystem can be divided. First. a system can be described as an

indivisible, macroscopic entity, or "black box." This overall level of

''U d scription can be thought of as focusing on the system's basic objectives.

Taxa that would be identified on that level of description would be expec.ed to

generate nominal specifications or taxa of performance/effectiveness, and would

r tend to be particularly directed to generalizable research issues. Second, the

system can be described in more detailed terms of how it can be applied. i.e.,

its functional purposes. This level of description probably would produce

taxonomies directed to both fundamental and system-specific types of research

questions, and would generate both nominal and relative measures. Third, a

system can be described in terms of how it can achieve its purposes, i.e., in

Vol terms of the cha) acteristics of its operations. This level of description would

tend to yield relative measurements, and be applicable mainly to specific

research issues. The distinction between generic and specific research issues

will be illustrated later in Section III (Figure 13).

These first two factors (system context and levels) suggest a descriptive

I matrix for the mode], which in its rudimentary form served as a point of

denarture for the work under this contract. In its subsequent, evolved form, 3 shown in Figure 2, the matrix served as a point of departure for the second

-. year's vork.

Because any system tends to be associated with larger and smaller I systems in carrying out its operations, that system of interest must be viewed

in terms of those operating relationships. Consequently, a third factor of the

I fl APM is the hierarchical structure of system operation, illustrated in Figure 3.

*- Thi- third factor is intended to insure that proper attention to interacting

entities is given in any system design or measurement process. The

hierarc!ical structure includes every system with which the system of interest

dire,' 4y interacts. In general, a system interacts directly with:

Larger systems, of which it is a part, and superior systems, to .1i which it is subservient in a command/control sense; collectively,

these larger and/or superior systems are termed suprasystems in the

hierarchical structure.

I Smaller systems that comprise it, and inferior systems over which it

exercises command/control; collectively, these are termed

subsystems.

Systems that exist and operate on their own level of command and control, with which they share resources and/or exchange input/output; collectively, these are called collateral systems.

.: When the two-factor matrix (Figure 2) is applied to the system of interes*

* and its associated hierarchy, the model takes on a three-factor appearance as

shown iii Figure 4. Each cell of each matrix of Figure 4 yields a list or

.- 5

i-5

1 .. . ..
:~ I c, r EEL ZEo 

IL 0 IIc 3.

J: fr . x '

r. 3t _07o' X I

2. -.L-f-

'aI go t .E

U FI l SLU10arso ~ V6B-l ~ VOZNn ~ ~ O~ ~ = v1:-

_____ 4 ___ ___ ___ GA2H

MOEhd ____ At",6_1U________W3C ____ 

7!r -
'SA

IE

~PRi

IRA

'- U C L~.OL~COLO

SYSEM SYSE COLSTHE SYSTEM SSE YTMS~E SY OLTRLO

E~ ~ Fiue3 ~ ~ eerlRpeettioREaSte SSE 01rrhcl tutr

SYS1o

~r1

ii -7..

*b _n*. . - . . . .
~-. L ~ ~. - L ~ -Taxonomies

IL Z.- - Sfor Each 0 ~,Supra-System

A;= MW Invm 1. II ki67;1±wI=.±lax, lapiaw

ILZ.. auw

Taxonomies

for the - ~m

System of ' swInterest

will- m - .-

Taxonomies *

for Each -. w Tikxonomies

Collateral - -fbr Each

System Subsystem

II

NSA Wii- -~-. *Z1

I'Iarwwar

U Figure 4. The Three-Factor Structure of the APM (STM Segment Only)
taxonomy of descriptors pertaining to the system being analyzed. Each item or

* taxon in each list is a potential basis for design specifications or evaluation I measures. One can readily see the enormous magnitude of effort that could be

required in applying this three-factor model even to a simple system. Hence,

* an increasingly important part of this study is the attempt to aid and yEaccelerate the model application by using an interactive computer. The

computer can lead the analyst through the model steps in a systematic flashiun

j by asking the proper questions, by providing clarification and, especially, by I offering the analyst general taxonomies and measures that can be used

directly, ignored, modified or supplemented.

B. Model Development, Prior Applications and New Objectives

.'The overall goal of this project is to improve the capability for measuring

the effectiveness and performance of training and other human-machine

systems. However, the specific areas of emphasis within that goal have

undergone redefinition and clarification during the project since 1980. In the

first year, it was implicitly assumed that the research would address

measurement of existing (i.e. , at least prototype) systems. In this second

year, however, applications to system cocet also were examined. That is, S3results of the first year suggested tat project's technological

developments may be useful not only for evaluating emerging and operational

systems, but also for helping to define and specify systems at the conceptual level, as an input to system design. Both such applications are now included

within the project's scope of activities.

Also, while the project continues to seek technological developments

suitable for use with the total population of human-machine systems,

* concentration is on training systems. It is somewhat different than the Year I

- model development Mt,~I in which the APM was applied to the combat

performance of an emerging weapon system, the Bradley Infantry Fighting

Vehicle. In particular, the sample BIFV function that was most intensively

analzedwith the model for measurement implications was that of Surveillance.

Applications to systems of lesser complexity were also tried in Year 1, in an

* effort to clarify the elementary stages and procedures required of the analytic

.f~ flprocess being modeled. By the time the project began its second year, there

were three major accomplishments upon which to build:

>~0 A comprehensive analysis of the state of art of human-machine

- system measurement, including In annotated bibliography of

literature pertaining to that field.

An improved, extended Systems Taxonomy Model, developed and

verified through trial applications to systems of increasing

complexity.

Edwards, J.M., Bloom, R.F., Oates, J.F., J~r., Sipitkowski, S., Bann

P.A., Ec'kenro-Je, R.J. and Zeidler, P.C. An Annotated Bibliography of the

Manned Systems Measurement Literature. Darren TTDulap ad Associates,

Inc., 30 November M98.

'18
3 0 An outline of the overall process of human-machine system

measurement, providing the skeletal structure for a total analysis

process model (APM). The STM constitutes the first stage of that

APM.

This second year's work was designed to continue development of the I APM, proceeding from the first stage (STN) to at least the second stage (blocks 6 and 7, Figure 1) of the overall process. Previous accomplishments

demonstrated that a face-valid hierarchy of measures could be derived from the I taxonomies, but explicit procedures and guidelines for deriving those measures

were yet to be developed. One objective of the Year 2 work was to begin developing those procedures and guidelines.

A second objective In Year 2 was to examine the applicability of the model

to the definition and specification of new systems. Third, the project wis to

~ I address the issue of streamlining the process of applying the STM (and AP?!).

In particular, the potential for implementing the first portion of the APM. as a

co!7puter-aided model was to be explored to a reasonable depth.

Six tasks were specified and completed for the Year 2 effort. First, the

STM itself was further developed, through improvement of the taxonomization

~; guidelines and the identification of comprehensive, usable sets of taxonomies

* (when possible) for the model's cells. This further development was extended

* to the procedures for deriving measures hierarchies once taxonomies are

identified. Also addressed briefly were concepts and procedures for:

(1) sorting out taxonomies and measures on a "system vs. subsystem" basis;

(2) differentiating measures of "effectiveness" vs. "performance" in identified

* hierarchies; and (3) specifying the human contribution to any given system

measure.

3 Second, the model was applied to one aspect of an existing system as a

further test of its utility and validity for aiding systems measurement. The ,. - system of interest for that trial application was the BIFV Training System.

The analytic "purpose" selected was to help measure that aspect of gunnery

training design concerned with the specification of gunner-learner testing.

~ 3 The third task was to apply the model to a selected system concept, to

*test its utility in providing input to system definition and specification. The

system selected for this application was the training system for the new 9mm

Army handgun. The analytic "purpose" selected was to help specify that

aspect of the handgun training system concerned with the design of the

curriculum.

As the fourth task, the results of the two major trial applications were examined in detail, and the needs for additional model development were

identified.

The fifth task was to review the potential for implementing a

computer-aided version of the extended STM. This task produced a working

(but reduced-scale) computer-aided model, suitable for demonstration

II -10-
3The final task was to document the results of the second year's work.

The principal documents produced were a working paper describing the results

of the trial application to the 9mm handgun training system concept, and the (I present technical report providing details on all of the project's tasks. As a

third item of "documentation," all products of the computer-aided

implementation were delivered, including hard copy listings and magnetic disk

copies of all programs and data bases (taxonomies), and equipment user manuals.

This second year's effort contributed to the longer term goals of

producing:

0 Automated procedures to help derive training systems effectiveness

measures.

0 Demonstrated applicability and usability of the APM by Army users.

O Demonstrated utility of the APM for specifying new training systems

development needs.

0 Guidelines and procedures for user training and user application of

the automated APM.

*l It is envisioned that the users of the APM for design and measurement

purposes will be at the U.S. Army Infantry School (USAIS) and other schools

of the U.S. Army Training and Doctrine Command (TRADOC), including thDirectorate of Training Development (DTD), the Directorate of Evaluation

(DOE), and the Directorate of Combat Development (DCD). APM users are

also foreseen to include personnel with the U.S. Army Test Boards, as well as

the U.S. Army Operational Test and Evaluation Agency (USAOTEA) .and, of

course, the U.S. Army Research Institute for the Behavioral and 'Social

Sciences (ARI). Among the individuals at those agencies who are seen as the

NE ultimate users of automated APM are such types as training developers and

evaluators, TRADOC system managers (TSMs), system researchers and

designers, analysts and planners.

The computer-aided APM will help realize the ultimate objective of

providing the above-noted users with a uniform, thorough, adaptive and

efficient procedure to aid in the process of deriving the most meaningful

. design specification requirements, design specifications, evaluation measurement

requirements or evaluation measures for any planned or existing

human-machine system, but particularly one in the area of training systems. Progress toward that end, as achieved in Year 2 of this contract work, is

described next.

_ I

-11-

• ~~~~~~~~............... ......... . ... "-.--.....-.. . ..--. . "--V"'.

'': " -"i ' ' i'" '"" "- .... .....-.... .... ':.- .-.-. ...--.. ..-. '....'.'..- ... ....... *-" "". ""' , "'"
:1 III. APPLICATIONS OF THE APM TO TRAINING SYSTEMS

The narrowed focus of the project from the total population of

human-machine systems to the subpopulation of training systems, permitted

considerable progress to be achieved that might not have been possible

otherwise. Clear delineation of a system's hierarchical structure has been

shown to be a sine qua non for the identification of performance taxa and thp

application of tF eA PM. "X long as the focus remained on systems-in-general.

nothing specific could be said about system structure. The "general system's" :3i hierarchy (Figure 3, earlier) is a non-specific conceptualization of

suprasystems, subsystems, and collateral systems, interlinked in some

undefined fashion. However, the shift of interest to one particular

I subpopulation immediately allowed a tractable structure to be defined in

considerable detail. In other words and without extreme oversimplification, it

can be said that all training systems share a common hierarchical structure,

consisting of identical subsystems and analogous collateral systems and

suprasystems; that these structural members can be defined in detail for

purposes of APM application; and that general training system performance

taxa can be identified and subsequently applied, for measurement or design

specification, to any training system of interest. Identification of those

general performance taxa occupied a significant segment of the research effort

expended during the project's second year. Important tools now are becoming

available for evaluation and design application involving this key class of

human-machine systems. Those tools include a set of general performance

attributes for all training systems, with computer-aided procedures for

*- translating them into effectiveness measures or design requirements in specific

training systems.

This second year of research probably would have followed a siinilar path

and led to similar products if the focus had narrowed to any other

subpopulation of systems. However, it seems particularly appropriate to have

chosen training systems as the field of interest, for at least these reasons:

0I Training systems are pervasive. Virtually every other kind of

human-machine system requires a corresponding training system to

insure its operability. Training very likely forms the largest single

subpopulation of systems, and thus represents the largest potential

base of applications for the APM.

Training systems are human-focused, probably to a greater degree

than is any other system subpopulation. Training system

performance has an especially large component of human

performance. Given ARI's fundamental focus on the behavioral and

social sciences, training is an appropriate sphere of interest for this

ARI research effort.

*l0 Training is one area of human endeavor for which rapidly expanding

technology is available, but only slowly or seldom is that technology

I -12
applied. Training system designers often seem to be bound by

tradition and familiar practice to a greater degree than are their Icounterparts in other (e.g., data processing) systems. Newer, potentially more effective and less costly methods of individualized,

self-paced, on-site training often seem to be overlooked in favor of I the time-honored collective, group-paced, instructor-dominated

academies. An analytically rigorous and performance-oriented model

such as the APM should facilitate the modernization of training

system design and help direct training evaluators to Issues of real,

current significance.

Important new Army training systems are presently under

development. Ranging from training to support use of individual

those linked with high level C (command, control and

communications) operations, these new training systems will have a

major impact on the Army's overall combat effectiveness. By

~-: Uinsuring the APNI's applicability to training, this present research

project may provide important assistance toward that larger issue.

3 Accepting, then, that training is a valid and desirable context in which

to continue APM development and application, the project staff undertook the

tssof Constructing a human-machine systems-oriented view of training,

i.e., a view compatible with the APM's fundamental structural

~1 concepts;

Identifying taxa of performance for key elements of a generalized

training system;

0 Applying those generalized taxa to emerging Army training systems,

to assess their validity and their utility for system measurement and

design specification;

0 Investigating the feasibility of streamlining and expanding the APM's

applicability to other training systems, through automation of certain

procedures.

The work and results are described below.

.3A. Training Systems in General: Axioms for APM Application

'I The APM was developed for application to any human-machine system. In

order to apply the model to the training sphere, it must be possible to show

that a training system (as a general concept) possesses the essential

characteri stics of a human-machine system, and that the structure of a

Itraining system conforms to the general architecture of all systems. It is appropriate and instructive to begin by examining training in light of a

general human-machine systems concept, in order to explore how that concept I is reified for this particular subpopulation.

-13-
IOne immediately encounters a stumbling block: "human-machine system"

Is one of those technical/ philosophical notions that, although critically

important for research and scholarship, are seldom defined. The mathematical

concept of a "set" is another of these. When attempts have been made to

define the set concept rigorously, enlightenment and progress sometimes have

resulted, but usually at the price of paradox, limitations in the utility of the

concept, and disagreement among scholars holding to divergent rigorous

so-called naive set theory, in which no attempt is made to define the

fund amental concept precisely. Instead, they adopt the implicit attitude that

they "know a set when they see one," and cheerfully go about the busine'-s of

achieving great breakthroughs in their research.

A human-machine system is something else that people usually feel they 3 know when they see one. The absence of precise definitions fosters a loose

and often confusing use of terms. Consider, for example, the term

h "sub sy stem. " Purists (including the authors of this report) insist that, 3whatever else a subsystem may be, it is itself a system. That is, it must

possess exactly the same essential characteristics that permit any system t-) be

m o termed. However, one often hears references to the "personnel 3 subsystem," which has come to connote simply the people who man a system of

interest, but not its equipment or other components. Further, most

- researchers will agree, at least in the abstract, that a "subsystem" must be .3totally contained within the (larger) system to which it is subservient. Yet in

practice one often encounters entities termed subsystems t at are at least

partly outside the system of interest. For example, it is common to hear

speakers refer to the "maintenance subsystem" of an armored vehicle when

they intend to include the personnel, equipment, and facilities at the direct

support and general support levels of maintenance: components and people

clearly outside the scope of the operating vehicle system itself.

It is not the intent of the authors to impose rigorous definitions upon

systems theory. The purposes of thi; research are quite compatible with a

naive human-machine systems science in which the entities of interest are

VTn -when -seen. However, it is crucial that the readers of this report at .3 least have some idea of what the authors see when they look at a

human-machine system. As a fundamental axiom, the research documented

herein presupposes that a human-machine system can be described (not

defined) as:

A collection of people, equipment, and procedures that :1 work together to accomplish specified functions.

No attempt is made here to impose precision on the notions of people,

a,;' equipment, procedures, collections, functions, or on the concept of "working

together." All of these things are treated naively as things one "knows when

they are seen.*" Nevertheless, some helpful insights can be gleaned from this

naive axiom:

1. There must be at least one human in a human-machine system.

Further, the humans have to take an active part in the work: they

can't simply be "passengers."

-14-
i.

2. There must be machines in a human-machine system. They too must

have an active, useful role in the system's working: they can't

simply be "freight."

3. There must be a purpose to a human-machine system. The humans

and machines can't simply interact blindly, unconcerned about any

chance results that might occur. Their functions must be specified

at the outset. It may be remotely within the realm of possibility

that, given enough time, ribbon and paper, a baby playing with a

typewriter could reproduce the Aeneid. That would be remarkable,

but there wouldn't be a human-maci"ne system in operation unless precisely that result had been intended.

If the reader will accept this loose, axiomatic view of the world of

human-machine systems, it becomes possible to ask: Does training fit this

system view? What does one see when one looks at a training system? What

are its purposes, or specified functions? Who are its actively working people?

What is its functioning equipment? What are its procedures? How are the

people, equipment and procedures organized, and how do they work together?

The authors assert as another axiom that any training system works to

accomplish two fundamental purposes, or master functions: learning and

helpin,-to-learn. These, too, are terms often defined differently by different

users. In this report, the classic, broad definitions are employed:

.o Learning is any activity, involving the senses, that affects behavior.

Thus, in any training system there must be activities requiring use

of a person's sight, hearing, smell, taste, feeling or any combination

of senses, and those activities ultimately must impact in some

intended fashion on what the person does or how the person does

something.

0 Helping-to-learn is simply the process of providing an efficient

learning environment.

In any training system, steps must be taken to make it as conducive

as possible for the learner to learn.

Very simply, then, a training system exists if and only if somebody

learns and somebody or something helps that person to learn. It might be

that the person doing the learning and the person doing the helping-to-learn

are the same individual and no one else is involved. It might be that many

people learn simultaneously and in concert with one another, under the

guidance of another individual, or with automated machinery providing the

helping. It might be that many individuals interact dynamically in an

atmosphere of mutual inquiry, each one learning and helping the others to

learn, with no one exclusively playing the role of learning director. In any

event, the people, equipment and procedures must work together to see to it

I that the intended learning and helping-to-learn take place.

-15-

%' - - I - IV " " 

5.!
N( xt, who are the people involved in a training system? In m re familiar

terms, whuop2 e~ates a training system?

It is self-evident that any tiaining system, no matter how prosaic, r,'st

have at least one type of operator: the person or persons who do the

learning. Training systems always strive to insure that human beings learn

somethini. Helping equipment or machinery to "learn" is the province of

cylc " retics and programming systerns. The time-honored terms for the hmn

who learns in a training system are "student" or "trainee". However, th-,';

terms have connotations that subtly impede the marriage of training syster~s

with modern technology. "Student" connotes a subordinate in the traditional

classroom Petting, for whom the principal sensory activity is listening whilf-

* ketping hands still and mouth shut. "Trainee" also connotes a receptive.

subservient role (where there is a "trainee," one expects to find a "trainer,'

and there is little doubt as to who is in charge). Further, the term "trainee"

conju-es up the image of a complete novice, a virtual tabula rasa. In reality,

learning activities often are essential for people who al 7'i"a kyn'ow a good de-i1

about the subject matter of interest In an effort to remove the exces.s

semantic baggag-, the term "learne or "learners" will be used throurhout

this report to designate the essential training system operator or unified group 3 of operators.

As suggested previously, it is possible for a valid training system to be 11i- Ptrated entirely by a single individual. In that case, the learner also is the

only human being responsible for helping himself or herself to learn. (Legend

has it that Dizzy Dean taught himself to pitch, totally unaided. His practice

consisted of hurling a homemade facsimile of a baseball at a peach basket

hanging from a tree limb.) In many cases, however, other human operators

are involved, even if only behind the scene. A group of learners frequently .3 are involved together in training as a collective group, unit or team (e.g.,

Individual Collective Training Plan).

1Who are these other operators? Since the only two functions of a training

system are learning and helping-to-learn, any operators who aren't learners

must be learning helpers, i.e., those who work to provide an efficient learning U er, vironment.Sevieralsubclasses of learning helpers might participate in any

given training system, including:

0 Training administrators. These are the people responsible for

determining the basic needs for learning and for exercising command

and control over the training system designed to satisfy those 3 needs.

o Curriculum developers. These are the people who plan the sensory

learning activities that will fulfill the identified needs, determine the

resources of material, equipment, facilities and people needed to implement the activities, and assemble the plans and resour-e specifications into a workable package.

o Facilities developers. These are the people responsible for 31 acquiring, fabricating, and/or preparing the hardware, facilities,

equipment, etc., needed to implement the curriculum.

1 -16-

|%
0 Logistics supporters. These are the people who handle the mundane

yet absolutely essential tasks incidental to training system operation, Ue.g. , maintenance of equipment, cleaning of facilities, preparation and provision of meals, transportation, record-keeping, etc.

0 Learning helpers who interact directly with the learners. About

whom, more will be said shortl-y.

From th~e viewpoint of the learner, the first four classes of learning

helpers listed above should be unobtrusive in the extreme. Those people

should be all but invisible to the learner, their interactions with the learner

kept subtle and subliminal. At its best, their work shouldn't really be noticed

by the learner. Like a baseball umpire, they will be noticed only when there

is1 Fum-e fault to find with their performance. To cite a simple example, U sappose training is being conducted over a period of several days in a meeting

room of a hotel isolated from other facilities. The hotel's restaurant, with its

cooks, waiters, waitresses, menu, oven, wine cellar, etc., constitutes an

element of the training system. Those cooks and other restaurant personnel

* aren't likely to think of themselves as trainers in even the broadest sense, but

their work could have a significant impact on the comfort, morale, and even 'Ithe physical well-being of learners and other training system operators. In

short, they have something meaningful to do with the efficiency of the learning

environment.

Similarly, a-, long as the learning needs have been identified accurately

and all of the people and equipment needed to support the learning activities

are present, the learner will not notice the work of the administrators or the

facilities developers. As long as the specific learning objectives are relevant

to 11-p learning needs, and as long as the learning activities are well.planned

and inherently interesting and illuminating, the work of the curriculum

developers will remain unobtrusive. It is the fate, even the aspiration, of

these kinds of training system operators to be taken for granted. Their job is

to do everything possible to prep are an efficient learning environment, and

then to get out of the way and al ow the learner to interact with that

environment.

The last class of learning helpers is a case apart. Their role is one

requiring direct contact with the learner. The common terms for tho;s-e who

play that role are very familiar and steeped in tradition: teacher, instructor,

professor, trainer, etc. Here again, the connotations are not favorable to

modern training technology. Each of those terms conveys the image of a

* classroom dominated by an authoritarian figure, actively imposing knowledge

*upon a group of passive recipients. The major training functions postulated

above clearly demand that the learner be the principal activist in the system:

it is he or she who must actually engage in the sensory activities and it is his

or her behavior that ultimately is to be affected. Furthermore, much of

present-day training is (or should be) tailored in pace, content, and

objectives. Large classes marching in lock-step through identical learning

activities toward identical goals no longer represent the optimum training

modality for many applications. Thus, a better term is needed to label the

-17-
U "interacting learning helpers," a term that connotes the supportive role these

key operators must play and also that connotes their essential responsiveness

to the needs of individual team learners. With some reluctance, the word

facilitator is used in this report. It is a distastefully trendy locution, but it

does atlast emphasize the learning -helping nature of the operator's work and

~ 3avoids the learnin g- dominating taint of the older expressions.

To recapitulate, the general training system model admits up to six "crew

ch. ssificat ions," viz., the learner (the most important member of the crew);

the facilitator (a key learning-helper wl'o interacts directly with the learner);

and the administrator, the curriculum developer, the facilities developer, and

;:~ Ithe logistics supporter (all of whom also are key learning -helpers, but who

work generally behind the learner's scenes). It is another fundamental axiom

of this research that the work done by each of these six types of system

operators is required in ever training system. In very large training

systems, many individuals mgt be "assigned" exclusively to each of these "loperator stations. " In the smallest of all training systems, one individual 3might man all of the stations in a time-shared fashion. Regardless: if

P learning is to occur, all of their jobs must be done.

~ 3What are those jobs? Fundamentally, each operator works with other

* operators and with various system equipment, following prescribed procedures,

to contribute to learning and helping-to-learn. Each does his or her work as

the key operator of a particular training subsystem, and may play important ., Isupportive roles in other subsystems. T1FTiSesiiption and examination of

those subsystems and their performance requirements occupy major portions of

te remainder of this report. For the present, the kinds of tasks required of

teeoperators can be illustrated generally as follows:

The Administrator

O Identifies needs for learning

0 Establishes an overall training schedule

o Allocates resources within the system 0 Recruits arid selects other system operators (i.e. , learners,

facilitators, curriculum developers, etc.) 0 Monitors progress 0 Maintains records

o Evaluates operator! subsystem performance

0 Evaluates system effectiveness

3The Curriculum Developer

o Establishes specific learning objectives

o Determines training content I o Selects training methodologies 0 Prepares/ acquires plans, resources, and resource

specifications 0 eemns aiiao rqieet 0 Determines facilitator requirements

0 Sets learner performance standards

o Establishes specific training schedules
..

i4 I The Facilities Developer

o Develops/acquires training aids and devices required in the

curriculum

o Develops/acquires other supporting equipment

o Develops/acquires any training sites and ranges required in the

currictvklm

Adapte equipment and sites to specific learning activities

T e Logistic Supporter

3 ~ Provides for ammunition, POL supplies, etc.

Provides for transportation, feeding, lodging, recreation,

etc., of all system personnel

- Provides for housekeeping, environmental control and

maintenance of system equipment and facilities

3 'he [ acilitator

o Helps to motivate learners to learn

Provides stimulation to facilitate and enhance learning

0 Provides day-to-day management of learning activities

c Helps to tailor learning activities to individual and

3 team needs

* Provides guidance to learners when needed

c, Counsels learners I C Tests and critiques learners' performance

The learner

0 Carries out appropriate sensory activities to achieve

intended behavioral objectives

The learner's job is most succinctly stated here, and yet encapsulates the

true purpose of the whole training system. The other system operators play m' important roles requiring thorough preparation and much hard, dedicated

- work. 

filins unless 

But their work always is secondary to the learner's: a training system the learner actually learns.

Next, what is the equipment used in a training system? What machinery

do the training operators operate?

i Training system equipment is anything and everything (apart from the

people) required for or contributing to the learning activities and the learning

environment. This includes, most obviously, the equipment, machinery, tools,

etc., that are used in performing the job-being-learned, One component of a

handgun training system, for example, will be the handgun itself. Training

systt.m equipment also includes learning resource material, e.g. books,

. pictures, models, mock-ups, films, tape recording, etc., used by the

learner in the conduct of the learning activities. Equipment needed to support

1. 2 .

A ' q.
or apply,, the resource material would also be included, e.g., chalkboards,

desks, tables, screens, projectors, chart stands, and so forth. Materials

needed to prepare and maintain the facilities also would be part of the training

system's equipment inventory, as would the facilities themselves, whether they

be ranges, ditches, classrooms, bunkers, study halls or whatever. All of the

equipment required to provide logistic support also would be included, e.g,

kitchen supplies, sleeping accommodations, vehicles for transport, etc.

Finally, what can be said in general about the procedures used in a

training system? Procedures are the specific ways in which system operators

wcrk with each other and with the various equipment in order to carry out the

system's functions. Given the focus of a training system on learning and

learning-helping, a training system's procedures must encompass (among many

other things):

0 How basic needs for learning are identified

o Pow learning objectives are derived from identified needs

o Hlow learning activities are planned

0 How facility requirements are determined

o H ow facilities are acquired or developed

Q How the learners acquire information

o How the learners practice and assimilate learning

0 How the environment is controlled

0 How the learners are tested

o etc.

Training systems typically document some of their procedures, for

example in lesson plans, schedules, etc. However, simply documenting

procedures doesn't guarantee that they will be carried out as written.

Leazrners, facilitators, and other operators often revise plans to suit -their own

tastes. In evaluatirng training system performance, it Is important to assess

the procedures actually used, in contrast to those officially specified.

Training, then, is an endeavor that easily lends itself to the concept of

human-machine system. It always involves people, equipment and procedures

working together to accomplish purposeful functions. Those functions revolve

'I around learning, i.e., the creation of desired human behavioral effects

through planned human sensory activities. All of the performance capabilities,

processes, and products built into a training system must enhance or support

learning. Any that do not are at best superfluous, and possibly deleterious.

Given that training systems are indeed human-machine systems, what can

he said about their subs ystem structure? Here again, a "subsystem" is

conceived to be a valid sytem, totally contained within and subservient to

some larger system. The subsystem's humans and machines must all be

members of the larger system, and the subsystem's purposes must be strictly -, supportive of the larger system's purposes.

-20-
As a final fundamental axiom, the authors assert that every training

system contains six subsystems. Each subsystem constitutes the "principal

territory" of one of the six training operators. The operators also play

supportive roles in one another's subsystems as depicted in Figure 5. The

* names given to the six subsystems intentionally are non -traditional. Just as

the terms learner and facilitator were adopted to avoid the undesirable

connotations of "student" and "teacher," new words are needed to describe the

building blocks of training in ways that do not encourage shopworn approaches 3~j to their design. Thus, for example, the term "Curriculum Development" might have been used to name the subsystem we have called "Design"; however, the

older phrase carries with it the stereotypic classroom, academy, professorial

image. Similarly, what we have termed "Enabling" is more commonly known as

"Instructor Training. " However, what unfortunately passes for instructor

training all too often falls far short of the facilitator preparatory work that we

perceive as essential for satisfactory system performance.

The Command subsystem is the main territory of the training

administrator. The administrator, with help from others, operates the

Command subsystem to exercise control over the total training system, e.g.,

by recruiting and selecting personnel to "staff"' the other subsystems, by

~: ~ allocating resources to them, by evaluating their performance, etc. It is

within the Command subsystem that the most fundamental decisions concerning

training are made: decisions regarding what needs to be learned and who

needs to learn it. Each of the other subsystems must submit its plans and

products to the Command subsystem for review and approval. Command thus

stands as a suprasystem for the other subsystems becjuse of their

subservience to its authority.

The principal operator of the Design subsystem is the curr!culL -l

developer. The developer, with help from others, operates the Design

subsystem to trar'slafe the basic learning needs (identified by Command) into

specific training plans. Design determines exactly what human behaviors are

to be affected during learning, and specifies the sensory activities to be

conducted to achieve the effects. In discharging that responsibility, Design

makes basic decisions concerning the kinds, numbers, and qualities of

facilitators, equipment, materials, and facilities needed to implement the

learning activities. Design produces all of the procedures, some of the

materials, and all of the resource specifications required for the Delivery

subsystem (the subsystem in which learning actually takes place). Since it

thus provides essential input to each of the four remaining subsystems, Design

* stands as a collateral system for each of them.

The Delivery subsystem is the heart of the structure. Its principal

operator is the learner, who operates Delivery to implement the prescribed

sensory activities and thereby acquire the intended behavioral effects. In

doing so, the learner is assisted by all of the other operators, each of whom

contributes in some way to the efficiency of the learning activities. Because it

derives essential input from Design, Emplacement, Logistics, and Enabling, the

Delivery subsystem is collateral to each of those four.

Is
Administrator

DEIG

1. Motethan o eeoper

2. Oepeson ad s er

aeelpe pSiupalopeatoei

I.More than one subeystem.

Figure ~ SusytessndThtrOpraor ~ ~ 5.a Tranin ~ or 

-22-

........................................
The principal operator of the Enabling subsystem is the facilitator. This

4 subsystem is unique in that it is itself a total training system: every Enabling subsystem plans and implements activities, involving the senses, that affect

p behavior in a purposeful fashion. The person engaging in those activities is

the facilitator, who does so to acquire learning that will prepare him or her to

work in the larger training system. The kinds of learning activities carried

F' out in the Enabling subsystem are intended to insure that the facilitator has

Y: I the basic subject matter expertise necessary to permit playing a credible and

useful role in helping others to learn; has the ability to implement the kinds of

interactions with learners called for in the training plans produced by the

Design subsystem; and has the ability and the opportunity to tailor those

plans to meet the specific learning needs of given learners. In short, the

Enabling subsystem works to insure that the training plans, procedures, and

resources produced by Design are made workable and viable for each Delivery

application. Among many other things, activities might be conducted in the

* Enabling subsystem to insure that facilitators possess traditional platform t instructional skills, but those activities would be included only if such

instructional skills are relevant to the learning activities of the larger training

system. Even then, instructional skill training never accounts for all of the

work required of the Enabling subsystem.

The Emplacement subsystem is the main province of the facilities

Ideveloper. This subsystem discharges the responsibility for constructing,

acquiring, adapting or otherwise preparing the equipment and materials needed

for the prescribed learning activities, and also for preparing the physical sites

at which the activities are to take place.

- The Logistics subsystem's principal operator is the logistics supporter.

This subsystem carries out the work necessary to maintain the total* system's

humans and machines in good working order.

I Having postulated that all training systems possess these six subsystems, it should be possible to point them out in the operation of any training

system, large or small. This is done in the following two examples.

EXAMPLE 1. A Small Training System

Consider first a bicycle rider training system operated by two people, a I child whn wishes to become a rider and a parent who agrees to help the child

learn. The basic Command decision, that a need to learn to ride exists, might

be initiated by the child and, after some thought, approved by the parent.

;~ I The child's perceived need may be strong, motivated by envy of already-riding * playmates, sharp attraction to a shiny, new bike, the desire for mobility

beyond what the feet or tricycle can provide, or whatever. The parent may

also perceive a nee,- for the child to ride, based on the parent's desire to

* express affection, provide happiness, and aid in the overall growing process,

-' -23-
K J and also perhaps based on the parent's fond memory of having learned to ride

himself or herself. However, the parent probably also assesses that need in

light of other needs, such as safety, economics, etc. In any event, the

decision is jointly made: if either parent or 'child refuses to concede the need

{for the child to ride, this bicycle rider training system will die aborning.

Both parent and child thus are key operators of the Command subsystem, with

i~ 9 the parent (one hopes) serving as the higher ranking operator.

If the training is to be successful, the parent had better be the sole

operator of the Design subsystem. The child simply has no way of knowing, a

priori, what behaviors are required for bike riding or what learning activities

will lead to the mastery of those behaviors. The parent probably has a fair

I idea of what the behavioral objectives and learning activities ought to be,

based on recollection of the parent's own bike training and on a few moments'

reflection on the abilities needed for riding. Although the parent rarely will J express these thoughts in classic behavioral terms, the concepts of balancing, .~ 3 pedaling, braking, steering, turning, signalling, etc., probably form a loose

mental image of a training content outline. The painstaking or extremely .motivated parent might consult a few bicycle safety and

W how-to-teach-your-kid-to-ride manuals. The outcome of all of this will be a

very nebulous (usually) and very flexible plan for exactly what will take place

when the learning actually starts. At roughly this same time the parent

unconsciously should "change hats" and start to function as the operator of

the Enabling system. This basically entails forming some rudimentary ideas

about what will be said to the child (by way of explaining bicycle controls and

operations), what will be demonstrated to the child (to clarify what is

required), how the child will practice (to assimilate learning), and--very fl roughly--how the parent will know when the child has learned well eniough to

cease supervised training. The parent's work in this subsystem also extends

to a "critical examination" of the child to identify any special training <1 requirements. For example: Is the child so short that wooden blocks should

be attached to the pedals? Has this child shown any special problem in

balancing in the past?

~' I Next, the parent and child move into the arena of the Emplacement

subsystem. First of all, they obtain a bicycle. Although the economics of

that step probably fall only on the parent, the child very likely has some

input to the decision as to exactly what bike will be had. If nothing else, the

choice of color might be up to the child. Training wheels might be purchased

at this time. Then, a location for the training has to be chosen. Here again

the parent should take charge, since the parent presumably is much better

aware of the characteristics of a good practice site (traffic-free, level, paved,

no obstructions, etc.) than is the child. It would be a rare parent indeed

who would entertain the possibility of constructing such a site. Instead,

every effort would be made to find a suitable, existing site as close to home as

possible.

-24-
The parent and child operate the Logistics subsystem incidental to each

training session. The parent should be responsible for handling or at least

reviewing the more critical work required: seeing to it, for example, that the

bike's nuts and bolts are secure, its tires properly Inflated, etc. Parent and

child likely will jointly inspect the practice site, finding and removing any bits

of glass or other debris that could degrade safety or otherwise interfere with

learning. The parent might insist that the child take responsibility for giving

the bike a pre-ride cleaning.

Finally, there is the Delivery of learning. The parent explains to the

child the operations required for bike riding, demonstrates these operations to

clarify them in the child's mind, assists the child's firstseveral rides by

holding on to the bike to secure it, coaches the child to guide him or her

toward increasingly better operation, ii7?Ell reesstebkealwnnh

child to practice unaided riding. From time to time, the parent will have to

disentangle thefallen child from the bike, help to alleviate the physical and

emotic al pain created by the fall, boost the child's aspirations and resolve,

and generally counsel the child through the difficult trials of the learning

experience. Notwithstanding all of this parental involvement, the child clearly

plays the key role in Delivery. It is the child who hears the explanations,

sees the demonstrations, controls his or her muscles to move and steer the

* bike, contacts the ground, gets back up, dusts off and starts all over again.

The child is the one living through the sensory activities. The sole

determinant of the success or failure of training is the child's ultimate ability

~:K jEXAMPLE 2. A Large Training System

Next, consider briefly the training system represented by a large, :4'private University. There can be no doubt that this is a training 'System,

s~nce everyr University (officially at least) dedicates itself to lear~ning,

implements numerous sensory learning activities, and confers certificates of

learning accomplishment. Many Universities undeniably pursue other functions

as well (research, football, etc.), but at its heart the essential elements of a

training system are present.

7 1 The University's key Command decision, viz., the identification of

learning needs, is handled by the administrative staff, typically with the

I advice of the faculty. Note that their true purpose in undertaking this effort

IVA is to ascertain the needs that exist in the real world beyond the University's

walls. The administrators should ask themselves: What learning achievements

are in demand out there, by society at large, potential employers, and

potential learners? The University functions In a highly competitive,

service-oriented marketplace. It may cease functioning if it fails to anticipate

and cater to the learning needs perceived by the people willing to pay for that

service.

,*

-25-
The Design subsystem's principal operators are the department heads.

Presumably, they take their cues from the needs Identified by the

administration, filtered perhaps by their appreciation of the nuances of

"marketability" of the skills and knowledge available in their own disciplines.

Presumably, too, they seek the advice of the members of their departme.,its

who may be a bit more in tune with the outside world than are the (typically)

* older and more academically entrenched chairmen. Ideally, the Design work

would include interviews with real-world practitioners of each discipline, to

benefit from their insights concerning the specific learning achievements that

really are needed.

j In universities, the Enabling subsystem usually Is distributed among the

individual faculty members. Their initial acceptance into the university system

comes only after they have shown proof that they possess certain basic skills

.cin their chosen fields of interest and in techniques of instruction. Usually, this "Proof" is in the form of learning certificates (diplomae) awarded to them

by other universities, and of endorsements by other academicians. Once

accepted, they are assigned certain duties as facilitators, acquainted with the

relevant curricula, and then pretty much are left on their own to get ready.

.% Faculty members usually have considerable latitude in adapting the curricula to

their own styles and tastes, in choosing the particular resource material to

which the learners will be exposed, in planning the details of learning

experiences, and in otherwise "fleshing out" the curricula in preparation for

1:Delivery. A university's Design subsystem tends to produce skeletal

curricula, relying on the Enabling process to add the essential substance to

the structure. Thus, the same curriculum, in the hands of different faculty

members, can involve significantly different learning activities and produce significantly different degrees of learning effectiveness. At least in this

* regard, universities tend to differ from military training systems, whose

Design subsystems usually produce fairly detailed curricula and whose

facilitators have considerably less leeway in modifying the curricula.

The university's Emplacement subsystem for the most part is

prefabricated. It is presupposed that, whatever the learning needs and

objectives might be, much the same kinds of learning activities will be

conducted and much the same kind of facilities and equipment will be

applicable. Universities thus expend considerable funds to provide

general-purpose classrooms, libraries, laboratories, auditoria, media centers,

and the like. Undeniably, this impacts on the Design and Delivery subsystems

since it induces a strong pressure to use these facilities, and to shape the

learning activities to conform to the avafl-able Emplacement resources. To the

extent that military training systems have copied the university's predilection

for expensive, fixed facilities, a predisposition against individualized, work-site training of soldiers has been created, even though work-site training

-~ might be more effective in many cases.

son The university's Logistics subsystem also tends to be general-purpose.

Housing, feeding, and recreation of learners, facilitators, logistics supporters,

-26-
Z..

ec., are accomplished on a mass scale, usually in that efficient but colorless

style known as "institutional." Physical maintenance of facilities and equipment

is similarly organized and implemented as a routinized, impersonal process.

As in any training system, it is In the university's Delivery subsystem

that the learner does his or her work. To a great degree, the learner's

sensory activities consist of those that are traditional in an academic setting: ~ - ~ listening to lectures, viewing an occasional film, reading books, writing

papers, participating in laboratory experiments, etc. The more fortunate I learner might occasionally have an opportunity to participate in role-playing

experiences, small interactive study groups, self-paced computer-assisted

activities, and a generally richer environment with more active sensory

j experiences to enhance learning. As a subclass of training systems, however,

I the university may be among the slowest to adopt modern technology to its

learning and learning-helping activities.

The authors hope that these two examples have convinced the reader that

the postulated six subsystems represent a reasonable and useful general model

for any training system. It now remains to apply that model to construct a

I general representation of training system performance, and to extend that

4representation to particular measurement an3 eiT applications.

B . Taxonomies of Training System Performance

The authors have asserted that every training system includes subsystems

tV at have been labeled Command, Design, Enabling, Emplacement, Logistics,

an c Delivery. The level of effort, the complexity, the time, the cost, and

other attributes of the work carried out by those subsystems no doulbt vary

widely from one training system to another, as demonstrated by the examples

above. But even though the amount of work needed varies, the performance

requirements are stable. Each of the training subsystems exists because it

makes some essential contribution to the master functions of learning and Ihelping- to-learn. It doesn't matter whether the training system is large or

small, simple or complex. The subsystem contributions are needed in every

case.

What are the contributions needed from each of the six subsystems? What

performance requirements must they satisfy if the total training system is to

achieve its purposes of learning and helping-to-learn?

The performance requirements of each training subsystem can be

identified by applying the systems taxonomization guidelines (see Figure 2) to

define the taxa or categories of potentiality, process, and product which

describe the subsystem and hence the populations to which it belongs. For

example, one can examine training Command subsystems in general and ask:

- 27-
1. What capabilities are to be provided

by Command? What contribution is

Command supposed to make to the (Potentialities)

abilities the system needs to accomplish learning and helping-to-learn?

2. What activities are supposed to be

carried out by Command? How is

Command supposed to contribute to the (Processes)

procedures the system employs to

accomplish learning and helping-tolearn?

3. What goods or services are supposed

to be produced by Command? What

contribution is Command supposed to (Products)

make to creating the products the

system needs to accomplish learning

and helping-to-learn?

These and similar questions will lead the analyst to discover the

Objectives-level performance requirements of the Command subsystem, i.e., the

fundamental "what" of Command performance in terms of potentialities to be

possessed, processes to be undertaken, and products to be delivered.

Each Objectives-level requirement car. be analyzed in greater depth, again using the taxonomization guidelines, to determine "why" it is needed. That

* step will disclose the Functional Purpose-level taxa to which Command belongs.

These in turn can be analyzed to determine "how" the purposes art to be

fulfilled, producing Command's Characteristics-level performance taxa. This

same analytic approach can be taken to derive the performance taxa for

Design, Emplacement, Logistics, Enabling and Delivery. Taken together, the

six taxonomies will cover all of the performance requirements of the total

generalized training system.

The first stages of the six taxonomies have been derived and are

displayed in Figure 6. The items listed are the Objectives-level taxa, i.e.,

the "what" of performance requirements for each training subsystem. Those

taxa could be applied to specify and/or evaluate the design of a given training

system at the conceptual stage of development. In effect, the Objectives-level

taxa constitute a check list for the training designer/evaluator. The

designer/evaluator can review each taxon and ask, "Is this particular

capability, procedure, or deliverable product relevant to my training

application? If yes, has my training system concept made provision for including the capability, procedure or product? If no, how should my concept

* be modified to correct that deficiency?" Note, however, that the

Objectives-level taxa say nothing about the particular purposes to which the

system in question will put each capability, procedure, and product, and

certainly also say nothing about the design details needed to insure that those

-28-
T A ITIE PRQE S MO

mai PObTTIAITE D RE CEASE PRODUCTSgm LICYIOIND SYTI 0VSSAL AAYSI ofLYI ra ova FLaOIOD551m

ofIf owININ OALU SnCE Of SIS If NI IA6 0wol P&EENImt ssimt

COMMAND :M~1ND.6AL ScPOff if colE NEFIT 1889681011 NAS OF Ae PEFRANSIEMILESTOE 0 ENVI#DWh FANGIAL 0480USC98 PIOYENIIAL TRAINING 0 PSfWOOANCI ASSESSMNTNI

OVALI 160 STAFF AND0 IDOLS SELECTION iOf PtikSONNEL 0 41AM.. UA1111 COTRO 0SCREENING AND WL§CII OF

0 ~ ~ ~ ~ ~ ~ SIS11?o ~ ~ KICLOW ~ ssPO ~ a 0*Also StPPAC PERFORMANCE ollitC,,Y

sil* ANALYIZING THE SamlTives 4, ANALYSIS OF SILICID 1*565 0 TANNN DOCUMENTS agog" 0 SEFNAoiG TRAINING CONTENT A A444ISSWZNI Of MSIhUCT*04.AI 0 ISM1

DCPMN~~~~ RCDRS ~ ~ ~ MLANOT YRM AAMuUO TINAL DATA .1 JALW6O tYlONS FO

matahq:"71460"CtsAWILIl, T 10 PI ooo N S TASE 4165 *V a a SLIVINY APILCATIONS OMISUIIIO AVAILABILIY Of 0 ANALYSIS9 OF FACILIVAIDN'S 0 CURRICULA AGLONO To INSTRUCTIONAL SALIVIRV 1111111710 ABILITY TO MAWAW I IUPLESUNI 4110101 SSLIVSA AFPLICAORSI

IF NISUftING AVAILAILITY OF ISL1IIGAIV19 IULS LIARING OWNANCI IPIRIISS 0 "blIow ACTIVITIES T0 CoomNCt

A PAMLIARI*1ING FACILITATOR$ IW#*1 ADAPTATION 04' INS CURRicuLuw THlE CURRNICUJLUM 0 COMOYCT Of INS FACLIAOR' ATAILORIA10 INS CURNRICULUM TO $ASARNI ACI)VIIIS

-sPACIplC ILIuVLS, APJ.LiCATIOA.. "11IURINO VIASLI or if ANALYSIS OF C&ARicuLum TO #olIIat LIanniNOia FAc '0'1 Va

AfE~uaTEa LIAONING FACILITIES SUTIOMi SIMPLACUE1 FOR IN SEg%9LIVIOV APPLICATIONS

%INSURING AVAII.A64LITY of onowmU ilRsIE # APPROPRIATE LEARNING NATERIAL1

@PAAMlADIOUATI LGARIO AERIALS ACOLIISITION. CONSTRUCTION AMSS' FOR elIv4e 09LIV~IV APPLICATIONS W*Vki.) AVAILAOILI? or CDIFICATION Of LIARRINO FACILITIE1 0 APNIOPRIATI MINSRUCTIONJAL £104

% ADSOLIAT11 "TAVC"TIONA"L 448 ~~~ACOUISIIION,. CONSTRUCTION ADOlPN&i ILIR ".C~tu

AGEOUAU STNCIIOIAL AS ACOUIITION. COSItUCON ANSW If VNSLIIOLILCTOI

ODIPICaTIoN i aU-Njcio&& Am~a

0 OWING AVAILASILITY OF 0 ANALYSIS Of BELIVER' PLANS 0 PINSONNEL SUPPORT SERVIC95

ADSOIJAYS PERSIONNEL SUPPORT 10 DtISSUINE LOIOICS C FACILFTIS M1ANITSRANCS SERVICE

a IGNSING AVAILASILITY if "OUIEUE RIS C *soup""?N iAiTSANOR mo~vies

AOUATE FACILITIES MAIN'.. .C 4 DVElOPmEmioi if SCStIS lu. qAOU"E* AllTSNAIICS1 PUN OF PERSONNEL SUPOR SCSISS

PAIVINTIVIF UITRANCI 8SERVICEA

* S9ILOSNENT if PROCEDURES

too AESTORATIVI UAIRTSNA*Ct *ACOITION OF FACILITIES ROD

MATERIALS rnNEDS FOR SUMMITY

* POOVOWO LSaaawa WIT" low . .NAL 1SIS or SPEIFOIC Lg.%OWORL 0 APPOOPRIATE LCARIIING ACTIVPTIE

tgaakNiu ENYONUDNT wave a LIAUSRS wWO NAVA ACM*vgc

5.0 1WSIJI~ LEANERS WTEACTUN 0 CONDUCT PIMPARATIOO LSAONUNS THE STATED GMCIVES

*4 11111* VO NsVINOaImn I ACtSVITI&S 0 AsuSsMET OF LEASHERS

WLIYEY 0 MANAGING LEANNESS PROGRESS & 0$O*iC P011821109010N IDM. @CGvtumvNs 0SALUATING EV LEAMNECS ACHIIEVC09 slahIIIN ACtI10171 A PoRNAIvE SVAUIATWO BAIA

FRYIN "o* ANS Pool SYSTEM 4, COMDUC1 APPLtICAtIOQ N I ACTICS

EVALUATIEN 0 0ON1"Ct MIARNER timUe IADAUISITION OF PORMAIIYS

Figure 6. Objectives-Level Performance Taxa of Training Subsystems

*-29-

A1
purposes will be accomplished. That is, the "what" of performance says

*| nothing about the "why" or "how" of performance. Taxa must be identified on

j the functional purposes- and characteristics-levels in order to obtain inputs

relevant to "why" and "how" issues concerning design and evaluation.

To date, taxonomies complete on all three levels have been compiled for the Design and Enabling subsystems. These two were singled out to initiate training system taxonomization for two reasons. First, they are the most

extensively studied and best documented of the training subsystems.

Procedural manuals and research reports abound that deal with both civilian

and military applications of instructional system Design and Enabling (the

S..tter, typically under the title "Instructor Training"). Representative

documents from that literature base were reviewed in depth in preparation for

the present work, which then proceeded in a less speculative atmosphere than

might have been the case if other subsystems had been selected. It might be

added, in this same context, that the authors' have substantial prior

experience in training systems Design and Enabling applications.

The second major reason for addressing these two subsystems Is that they

are probably of greatest concern to training systems at early stages of

development. Two important, emerging Army training systems are at such

stages, namely, the Bradley Infantry Fighting Vehicle Training System

(BIFVTS) and the new 9 mm handgun training system. The developers of

those systems conceivably could benefit from a specification of Design and

Enabling performance requirements. Figures 7 and 8, respectively, present

complete taxonomies for the Design and Enabling training subsystems.

The taxa contained in Figures 7 and 8 represent the basic building blocks

for both design and evaluation of the two training subsystems. Each taxon is

a performance-oriented design requirement. The subsystem designer examines

a particular taxon and asks, "How can I best insure that this requirement will . be satisfied by the subsystem I am building?" The evaluator looks

independently at the same taxon and asks, "How can I tell whether this

particular subsyz;tem satisfies that requirement?" Answers to the designer's

question produce system specifications. Answers to the evaluator's question

- -"lead to measures of system performanie/effectiveness.

A decimal indexing code was established to facilitate the categorization of S. "all three levels of system description. Each contextual aspect was assigned a

unique number, as follows:

1.0 Performance Potentialities

2.0 Performance Processes

3.0 Performance Products

At the Objectives Level, the population categories is assigned successive

numbers in the first decimal position. For example,' at-the Gbjeetves -Level,

-30-

... . .. ..........-......................- ,-,. .... " %
PERFORMANCE PERFORMANCE PERFORMANCE

POTENTI1ALTIES PROCESSES PRODUCTS

1.1 Identifying goals and Z.1 Analyze the job for which 3.1 Documented job analvses

priorities training is to be developed 3.2 Stated performance oh1.2 Establishing perfor- 2.2 Assess tasks for training jectives mnce objectives development 3,.3 L.esson plans Cr', 1. .3 Anayzing parlor- 2.3 'Analyze tasks selected 3.4 Training documents

N%',- mane objectives for training 3.S Tests

_1.4 Dfining training content 2.4 Determine instructional 3.6 Training trial data

1.5 Dfining training requirements

procedures 2.S Analyze existing 0 1.6 Evaluating the instruction

cur culum 2.6 Assemble instruction

2.7 Conduct training trials

1.1.1 Defiing the total 2.1.1 Identify the tasks that 3.1.1 Provide a complete

s scope of learning are involved in the job specification of what

1 1 2 Stating the ultimate Z.1.2 Determine the sequence constitutes the job to be.

intended outcomes of and stimuli of the tasks trained

learning 2.1.3 Determine the level of 3.1.2 Provide a complete 1.1.3 Identifying the rela- skill or performance specification of the ortive Importance of in- deemed adequate for the ganJztion of the job

tended outcomes job 3.1.3 Provide a complete

1.1.4 Establishing a basis specification of what

for specifying learning 2.2.1 Determine needs for constitutes job adequacy

objectives training as a function i of task 3.2.1 Define exactly what the

1.2.1 Stating what the learners Z.Z.2 Allocate training learners are expected

will be able to do or how development and to achieve as a result

they will be expected to delivery resources of the training

behave after training 3.2.Z Provide an organiza1.2.2 Providing a basis for ob- 11.3.1 Identify the information tional framework for

jectively Assessing processing entailed in the training

learner performance the task

2.3.2 Classify the tasks in 3.3.1 Provide complete opeci -

1.3.1 ldentifyi,,g pre-e.uisites terms of the domains of fication of the instruc- to learning learning involved tional content for each 1.3.2 Providi.g a basis for 2.3.3 Identify the conditions objective

objectively assessing for learning the task 3.3.2 Provide complote speci-

* csr'i.t.l1 ear .ey's fication of th, instrucuitabi I V 2.4.1 Identify instructional tional proceriures for

events/activities required each objective

Ic 1.3.3 Identifying Interactions/ 2.4.2 Identify strategic Inatruc- 3.3.3 Provide complete speci- dependence among tional requirements fication of the instruc- objectives 2.4.3 Identify tactical instruc- tiona circumstances

1.3.4 Classifying objectives tional requirements for each objective

in terms of domains 2.4.4 Identify learner assess- of learning ment requirements 3.4.1 Provide essential

1.3.5 Identifying component resources to learners

steps or processes 2.5.1 Assess relevance and 3,4.2 Provide essential guidwithin the objectives applicability of existing &nce to facilitators

instruction to present 3.4.3 Provide essential speci- 1.4.1 Selecting stimuli for requirements fications to facilities

each objective 2.5.2 Assess effectivenessI developers 1.4.2 Selecting Instructional efficiency of existing 3.4.4 Provide essential

setting for each instruction in light of support to training sysobjective state-of-art methods! tem administrators

1.4.3 Defining Instructional media se:uence 3.S.1 Acquire qualified

2.6.1 Adapt relevant and appli- candidate learners

I.S.i Specifying preparation cable instruction, where 3.5.2 Tailor Instruction to

procedures possible, to satisfy pre- learners

1..2 Specifying presentation sent instructional 3.S.3 Measure learner's

procedures requirements achievements 1.5.3 Specifying application 2.6.2 Create instruction, where

procedures necessary. to meet unsat- 3.6.1 Provide Justification for

1.3.4 Specifying (learner) isfied Instructional specific features of

evaluation procedures requirements instructional design 3. .6.2 Demionstrate validity

1.6.1 ValJidstingfroviing 2.7.1 Culde the ongoting analysis .2 d utility of the

the bases for the and assembly of instruction training

developing currtulum 2.?,Z Verify the suitability of

1.6.Z Validating revising the assembled instruction

the developing curri- for delivery

culum itself

Figure 7. Taxonomy of Design tubsystem Performance

-31-

~~~~.. ..... . . . . . . . . ... .... ..... ,..,.. ... -. ,. .. .'.-.-
PEfiFORmA14CE PERFORMANCE PERFORMANCE

POTENTIALITIES PROCESSES PRODUCTS

1.1. i Identifying types of 2.1.1.1 Review job specification/ 3.1.1.1 Specification of mental "- &, achievements relevant description processes to the level of to the intended job 2.1.1.2 Interview job/system binary decisions

1.1.1.2 Analyzing achievements designers 3.1.1.2 Specification of physical

to determine suitability 2.1.1.3 Observe job incumbents processes to be iefvl of

for training 2.1.1.4 Interview job incumbents discrete actions 2.1.3.5 Prepare tentative task 3.1.1.3 Specification of all

list process stimuli

1.1.2.1 Determining necessary 2.1.1.6 Authenticate tenstive task 3.1.1.4 Specification of condilevels of achievement hatl using representative tions under which pro- ,% .I;'Z Determing existing Job designers /incumbent a cesses are to take levels of achievement Z.1.1.7 Validate task list using place

1.1.2. Discerning discrepan- other representatives des between existing 3.1.2.1 Operational sequence

%and necessary levels 2.1.2.1 Review job specification/ diagrams

description 3.1.2.2 Decision flowcharts

1.1.3.1 Assessing goal 2.1.2.2 Interview job/system 3.1.2.3 Specification of

importance designers rules and regulations 1.1.3.2 Insuring availability 2.1.2.3 Observe Job Incumbents of a relative (numeric) Z.1.Z.4 Interview job incumbents 3.1.3.1 Performance accurary goal ranking scheme 2.1.2.5 Prepare tentative task specification

flowchart 3.1.3.2 Performance speed 1.1.4.1 Insuring availability 2.1.2.6 Prepare tentative task specification of a hierarchy/taxonony stimuli specifications 3.1.3.3 Performance volume

of objectives 2.1.2.7 Authenticate tentative specification 1.1.4.2 Stating grals in terms flowchart and stimuli 3.1.3.4 Performance duration

that imply behaviors specifications using specification

representative job 1.2.1.1 Defining the perfor- designers /incumbents 1.2.1.1 Specification of the mance action 2.1.2.8 Validate flowchart and basic capabilities the .2.1.? Defining the perfor- stimuli specifications learners are to acquire mance conditions using other representatives 3Z.I.2 Specification of an ob- F 1.2.1.3 Defining the perfor- servable action the trance standard 2.1.3.1 Review job specification/ learners will execute

description demonstrate each 1.2.2.1 Developing pretest 2.1.3.2 Interview johisystem capability 1.2.2.' tteris designers 3.2.1.3 Specification of a mea- -. t2. Developi.g posttest 2.1.3.3 Interview jot) supervisors/ curable object that %i

items cumrmanders result from the actic, 2.1.3.4 Observe/interview *high*- 3.2.1.4 Specification pf the

i.3.1.1 Reveali g presort con- rated Incumbents circumstances under

d~tions of learing 2.1. 3.5 Observe/interview *low'- which the learners will 4 (i.e.. that which rated incumbents execute the action

learners must recall) 2.1.3.6 Prepare tentative skill/

performance specifications

1.3.1.2 Identifying essential 2.1.3.7 Authenticate performance 3.2.1.5 Specification o. ny tools

prerequisites (capatili- specifications using or equipment the learnerr ties Incorporated within representative designers/ will use to execute the

the objective) supervisors /incumbents action 1.3.1.3 Identifying supportive 2.1.3.8 Validate performance 3.2.1.6 Specification of any prerequisites (capabili- specifications using other constraints Imposed upon ties facilitating representatives the learners' execution

achievement of objectives) of the action

3.2.1.7 Specification of the cr- 1.3.2.1 Developing entry test 2.2.1.1 Develop training needs teria the resulting object items criteria In concert with must satisfy In order to - representative system demonstrate acceptable 1.3.3A Discerning independence users/designers achievement of the basic

between objectives 2.2.1.2 Rate tasks in accordance capability 1.3.3.2 Discerning dependency with the specified

between objectives criteria 3.2.2.1 Specification of the

A 1.3.3.3 Discerning supportive essential abilities *nc!qdrelationship between 2.2.2.1 Deterviine degree of ed within each pe-f:, objectives learning difficulty for sance objective

each task requiring 3.2.2.2 Specification of the 1.3.4.1 Identifying information training supportive abilities to

type objectives 2.2,2.2 Determine degree of be included in instuc 1.3.4.2 Identifying mental instruction needed for tIou for each objective

eskill type obje -tives each such task

Figure 7 (Continued

-32-

-' %,9

Z,-" ]. . -: - / ' i ::-T:.-'..'''-,'-...T" -". ---,,, -.., :,,..-.- : ,
- - PERFORMANCE PERFORMANCE JPERFOR

3.2.2.3 Speciflicatior, 01 the se1.3.4.3 Idertrifying physical quence of instruction in

skills type objectives 2.3. 1. 1 Analyze the decision a ssential] and suppoiti~r 1.3.4.4 idelifyng attitude flow to reduce pro- a bilities to be prepared

lype objectives ceasing to a sequence for each objective

of binary decisions 3.2.2.4 Specification of the me- 21.3.5 1 lIentifying overt steps 2.3.1.2 Analyze the decision quence of objectives to

13.5.2 ld'entifying covert levels in terms of be following in the

Iteps nature, complexity, and instruction

1.5. 3 1c4-ritfying unconscious time

at. Ps 3.3.1.1 Specification of the

42.3.2.1 Identify knowledge to be objective .4.1.1 Defining p o positions, acquired .3.1.2 Specification of assume i

naefacts., tc. 2.3.2.2 Identify mental skills to learner prerequisite% relevant to informs- be acquired 3.3.1.3 ItemizatIon of informnatior objectives 2.3.2.3 Identify physical skills tion topics 1.4.1.2 Defining concepts. rules, to be acquired 3.3.1.4 Itemleation of reital

algorilboi~s. etc..* rele- Z. 3.,.4 Identify attitudes to be skills steps

vant to mental skills acquired 3.3.1.5 Itemization of physical obi~ttves skills steps

.4. 13 Vefi iing movements. 2.3. 3.1 Analyze information 3.3.1.6 Itemization of .'..timings, actions. etc.., processing requirements attitudinal components *.relevant to physical to identify prerequisite 3.3.1.7 Itemization of practice *.skills objectives knowledge expected of exercises

*1.4.1,4 Defining v'alues, choices. learners 3.3.1.8 Itemization of test

ttc., relevant to 2.3.3.2 Analyze those require- problems /exercises a ttit udinal objectives ments to identify prerequisite mental skills

1.4.2.1 Assessing applicability 2.3.3.3 Analyze the physical 3.3.2.1 Specification of pro- of Job Performance Aids requirements to identify cedures for preparing (JiAs) prerequisite motor skills learners to achieve the ~ . 22 Asesaisng applicability 2.3.3.4 Analyze the attitudinal objective

Uof Self-Traching Export- requirements to Identify 3.3.2.2 Specification of how the ab sle Packages (STEPs) prerequisite values instructional content 3 .4.2.3 Assessing applicability topics/steps will be

rof Formal on-the-job 2.4.1.1 Select the types of presented ccTraining IFOIT) events/tactivIties required 3.3.2.3 Specification of how the L~1.4.2.4 Assessing aptlicability to prepare learners to learners will practice,

U of 'nstaliatior. 5uppoct learn the various tasks apply the instructional Sch.,ol (ISS) 2.4.1.2 Select the types of content

-C 1.4.2.5 Assessing applicability events/ activ ities required 3.3.2.4 Specification of how the

ofc Reside..t School (RS) to present/ demonstrate learners will be tested -the tasks on the instructional

1.4.1.1 Sequencing In depen- 2.4.1.3 Select the types of content

dency order events/lactivities; required 3.3.2.S Specification of proce- 1.4.3.2 Sequencing in supportive for learners to practice/ dures for dealing with

iationship (efficiency) apply the tasks learners who fail to

order 2.4.1 .4 Select the types of demonstrate achievement

events/lactivities required of the objective 1.S.1.1 Defining facilitator to evaluate learners'

preparation activities performance 3.3.3.1 Specification of the

1.5.1.2 Defining facilities location at which each

preparation requirements 2.4.2.1 Analyze eve nts /activities instructional event will

1.5.1.3 Defining attention- to establish their sequence take place guirlng procedures 2.4.2.2 Analyze events /activities 3.3.3.2 Specification of the .S 1.4 Defining objective- to establish their equipment needed

Infurming procedures delivery media 3.3.3.3 Specification of special

1.. tfinlng procedures for location /equipment

stimulating learners 2.4.3.1 Analyse events factivl ties set-upconfiguation

rrcal of prerequisites to establish thetr requirements

management procedures 3.3.3.4 Specification of Instir--

1.S.2.. Defining presentat~on 2.4,3.2 Analyze eventa/Iactivities tional personnel needs4

!

media/media to establish their 3.3.3.S Specification of time

it e -natives resource requirements and achedule

1 D..2 lefining procedures and requirements

requirements for placing 2.4.4.1 Analyse tasks to identify

emphasis assessment requirements 3.4.1.1 Texts

1.5,2.3 PefIning procedures for for selecting /accepting 3.4.1.2 Workbooksi

.IancnZ; learners' learners 3.4.1.3 Study guides coznprenension and 3.4.1.4 L~sts of references retention 3.4.1.5 Schedule of Instruction

Figure. 7 (Continued)

-33-
i

PERFORMANCE PERFORMANCE PERFORMANCE

POTEN"IALI1ES PROCESSES PRODUCTS

Z.4.4.2 Analyze tasks to identify , assessment requirements

1. e n rer ffor providing remedial

.. 3.1 Defining procedures for instruction 3.4.2.1 Sets of lesson plans

enhancing learners' 2.4.4.3 Analyze tasks to identify 3,4.2.Z Sets of test

ir-volvementl participation assessment for accelerating/tailoring requirements 3.4.2.3 specifications Lists of facil:tior

D;l-ininj. procedures for learners' progress through references

e"l;cing performance the instruction 3.4.2.4 Specifications of faciltfrom learners 2,4.4.4 Analyze tasks to identify tator prerequisites

1.5.3.3 Defining procedures for assessment requirements

providing learning for advancing/delaying 3.4.3.1 Specifications of

guidance to learners learners' progress through physical requirements

1.5.3.4 Defining procedures for the instruction for instruction &l sites

providing learning 2.4.4.5 Analyze tasks to identify 3.4.3.2 Specifications for

feedback to learners assessment requirements audiovisuals

1.5.3.5 Defining procedures for for certifying/decertifying 3.4.3.3 Specilfications for

asessing sufficiency learners as job performers training equipment asoS

of practice supplies

2.5.1.1 Specify criteria of rele1.5.4.1 Defining procedures to vance and applicability 3.4.4.1 Detailed outline of

assess the proper 2.5.1.2 Search for existing instructional objectives.

domains of learning instruction of potential content, procedures. _,.4.2 Defining procedures to relevance and applicability and schedule

-sses the appropriate 1.5.1.3 Access and review existing 3.4.4.2 Specifications of learner

action/behavior instruction deemed to be of prerequisies

%.5.4. 3 Defining procedures to potential relevance and 1.4.4.3 Summaries of personnel

insure assessment applicability requirements

reliability 2.5.1.4 Rate relevance/applic- 3.4.4.4 Summaries of facilities

ability of instruction requirements 1.6.1.1 Entry-level testing of 2.5.1.5 Select existing instruction 3.4.4.3 Documentation of all

representative cardi - warranting application to activities during the

date learners prescnt instructional process W 1.6.1.2 fost-train:ng testing requirements

of represertative 1o

F incumbents 2.5.2.1 Identify instructional 3.5.1.1 Measures of prerequisite

events/activities imple- abilities

W I.. I Obtainlng error data mented by technologically 3.5.1.2 Measures application

t, .6.i! 2 Obtaining design feed- outmoded methods/media procedures - back Z.5.2.2 Assess degree of ineffi- 3.5.1,3 Standards of candidate I .6..3 Clorrelatlrg Irot-test ciencylineffectiveness qualification

- data with subsequent ?,S.2.3 Assess cost of replacing 3.5.1.4 Specifications for dealjob pertormrAre outmoded instruction lg with substandard

2 5.2.4 Formulate decision to use candidates

or replace existing

instruction 3.S.2.1 Pre-traininit measures

2.6.1.1 Identify specific defi- 3.5.2.2 Measures application

ciencies in the eventsl procedures

activities of the 3.S.2.3 Standards of preapplicable instruction training qualification 2.6.1.2 Develop specific changes 3.S.2.4 Specification of pro- to the events/activities cedures for dealing

and their prescriptions with qualified learners

to remove the deficiencies

3.5.3.1 Poet-training measures 2.6.2.1 Identify specific objec- of intended abilities

tives for which new 3.5.3.2 Measures application

Instruction is needed procedures 2.6.2.2 Analyze the objectives 3.5.3.3 Standards of postto identify the Instruc- training qualiftcation

tional events/activities 3.5.3.4 Specification of pro- needed cedurae for dealing with

4 2.6.2.3 Select implementing unqualified learners

methods/media

2.6.2.4 Prepare detailed pre- 3.6.1.1 Entry test data derived

scriptions of the from ropresentative methods and media candidste learners

Figure 7 (Continued)

-34- 3

"" --- "-- -- - - -- - ." - -..-. -"... -- .- . .
1PERFORMANCE 1 PRFORMANCE PERFORMANCE

POTENIIAIfl JES _pE..S s PRODUcTs

3.6.1.Z Pout-test ata derived

2.7.1.1 Apply small blocks of from representative

existing instruction to job incumbents

individual candidate 3.6.1.3 Test item error analyses

learners derived from tryout

2.7.1.2 Apply small blocks of learners new instruction to indi- 3.6.1.4 Instruction design feedvidual candidate learners back data obtained from

2.7.1.3 Apply small blocks of tryout learners and

assembled instruction to facilitatorn

small roups of candi- 3.6.1.5 Specification of reviFions

date learners based on these data

Z.?.Z.1 Conduct field testing of 3.6.2.1 Test item error analythe assembled instruction sea derived from field

test learners

3.6.2.2 Correlations between

learners' post-test

results and subsequent

job performance I 3.6.2.3 Instruction design feedback data obtained from field test learners

and facilitators

3.6.2.4 Specification of final

revisions based on

these data

II

U

F1 .

Figure 7 (Concluded)

-35-

7,-2
'.'.PERFORMANCE PERFORMANCE PERFORMA14CE IPOTENTIAUTIES PROCESSES PRODUCTS

P1. Providing day-to-day 2.1 Analyse facilitator's 3.1 Qualified facilitators training management ability to perform the for given delivery % resourcesa taske applications 1.2 Insuring availability Z.Z Analyse facilitator's 3.2 Curricula tailored to of instructional ability to managelimple- liven delivery appli- delivery expertise sent the learning cations 1.3 Insuring availability activities

of learning guidance 2.3 Design activities to

expertise correct facilitator's . 1.4 Insuring availability deficiencies o f subject matter 2.4 Design activities to

0 expertise allow adaptation of the

I.S Familiarizing facli- curriculum

tators with the 2.5 Conduct the facilita- , curriculum tor's learning activi- 1.6 Tailoring the curricu- ties

mlu to specific

delivery applications

].1.1 Establishing a ached- 2.1.1 Identify tasks for which 3.1.1 Facilitators able to

*W ale of learning a given prospect could help create a rich activities credibly serve as a learning environment - 1.1.2 Establishing a condu- facilitator for the learners

cive learning climate 2.i.Z Identify task performance 3.1.2 Facilitators able to 1.1.3 Monitoring and modify- deficiencies for a given guide learners' inter- ing the learning prospect which could be actions with the experience corrected through cost- environment 1.1.4 Operating the training effective training .Ifacilities and equip- 3.2.1 Curricula relevant to went 2.2.1 Identify learning activi- the learning needs of 1.I.5 Implementing learning ties which a given pros- given learners activities pect could credibly 3.2.2 Curricula geared toward 1.1.6 Acquiring and diasemi- manage/implement the experiences of

nating learning 2.2.2 Identify deficiencies in given learners resources learning activity manage-

. Cment/implementation for a 1].2.1 Conveying learning given prospect which

stimuli could be corrected 1.2.2 Presenting information through cost-effective 1. . 2.3 Demonstrating pro- trainn

cedures

1.2.4 Conducting learner 2.3.1 DeviS plans to insure

I. practice that each facilitator ., 1.2.5 Conducting learner achieves requisite levels

atest of job performance skills

0 1.3.1 2.3.2 Devise plans to insure Motivating learners that each facilitator

to learn achieves requisite levels

" =a 1.3.3 attention Focusing learners' and Interest of instructional skills

1.3.$ Relating the learning 2.4.1 Devise plans to insure

to the learners' that each facilitator experience learns the curriculum and 1.3.4 Coaching learners its instructional

1.3.5 Feeding-back to requirements

learners 2.4.2 Devise plans to insure 1.3.6 Facilitating learner that each facilitator

self-assessment learns the particular 1.3.7 Counseling learners requirements of a given

1 K n delivery application

1.41 onto Kowig of' that te which igrei- is 2.4.3 Devise that each plans facilitator to insure to be learned develops the additional

1.4.2 Performing/demonstrat- details of procedure and

Ing that which is to . materials needed for a be learned given delivery applics- 1.4. 1 Citing illustrative tion

examples to assist

learning

Figure 8. Taxonomy of Enabling Subsystem Performance

- -36-

.----- I- .k I... ..- .
P R PERFORMANCE PERFORMANCE :i EN IALMn PROCESE PRODUCTS

2.S.1 Facilitators demonstrate

satisfactory levels of

1.5.1 Knowing what learners job performance skills

are intended to 2.5.2 Facilitators demonstrate

achieve satisfactory levels of .,. 1.S.2 Knowing the training instructional skills

c eontent 2.5.3 Facilitators demonstrate

1.5.3 Knowing the training satisfactory abilities to

resources Implement the curriculum

1.5.4 Knowing the training for a given delivery procedures and methods application

1.5.5 nowing the training W circumstances

a* 1.6.1 Adapting the curriculum to the facilitaZ tor's style and

0 strengths

2 1..2 Adapting the curricu- z lure to the specific

-learning needs of a

given delivery application

1.6.3 Adapting the curricuum to specific con- straints of a given

delivery application

1.1.1.1 Identifying scheduling 2.1.1.1 Assess prospect's experi- 3.1.1.1 Fadlitators knowledgeoptions ential history with able of the ingredients 1.1.1.2 Devising schedule respect to the tasks of the learning

smonitoring mechanisms 2.1.1.2 Assess prospect's train- objectives 1.1-1.3 Devising schedule Ing with respect to the 3.1.1.2 Role-model performers

control mechanisms tasks of the ingredients

2.1.1.3 Assess degree to which 3.1.1.3 Facilitators knowledge- 1.1.2. 1 Encouraging maximum the prospect manifests able of the activities

learner participation the qualities and charac- that can support 1.1.2.2 Eliciting sense of teristics of a good per- achievement of the - ., responsibility for former of the tasks learning objectives

learning from learners 3.1.1.4 Insurers of Jhe ad*-

1.1.2.3 Treating learners with 2.1. -. 1 Test the prospect's abil- quacy of the physical

respect ity to satisfy the per- learning environment

11.2.4 Encouraging learners' formance objectives asso- 3.1.1.5 Insurers of the availfreedom of expression cated with each task ability of needed mate1.1.2.5 Avoiding disorder and Z.1.Z.Z Estimate the cost of vials and resources

learner distaste training that would be 3.1.1.6 Communicators of internecessary to achieve est and enthusiasm for

1.1.3.1 Measuring learners' satisfaction of the the learning

progress performance objectives 3.1.1 . 7 Facilitators who are 1.1.3.2 Dlagnoeing emerging 2.1.2.3 Formulate decisions to vespectful of learners

deficiencies and dif- provide or not provide 3.1.1.8. Facilitators who are ficulties in learning task performance train- accepting of learners' 1.1.3.3 Simplifying key ing for each prospect needs and differences

operative information on each task 3.1.1.9 Facilitators who are 1.1.3.4 Tailoring learning supportive and encouractivities to speci- 2.2.1. Assess prospect's experi- aging of learners'

fic experiences of ential history with initiatives

1.1.3.5 learners Tailoring respect to managingl 3.1.1.10 Facilitators who are schedule and Implementing those types accessible to learners

sequence of learning nf learning activities

to learners' capabili- 2.2.1.2 Assess prospect's prior 3.1.2.1 Elicitore of learners' 1.' 4 ties and needs training relative to participation in the sthoe types of learning learning planning 1.1.4.1 1.1.4.1 Dev the use lpin of g relevant sdkls In artviis lctlvttes p "

Instructional aids

Figure 8 (Continued)

-37-

-- 3 N .- .. . .. . ,'- .. ".
PERFORMANCE PERFORMANCE PERFORMANCE

POTENTIALITIES PROCESSES PRODUC'rS

2.2.1.3 Assess degree to which 3.1.2.2 Creators of conditions

prospect manifests the to enhance desire for

1.1.4.2 Developing skills In qualities and character- learning

the appropriate con- latics associated with a 3.1.2.3 Dtagnosticians of

W Itol of the physical good managerlimplementer learning needs

environment of those types of learn- 3.1.2.4 Creators of experience

1.1.4.3 Developing skills in ing activities to enhance learning

the preparation of 3.1.2.5 Counselors of learners

relevant special 2.2.2.1 Test the prospect's abil- 3.1.2.6 Roosters of learners'

facilities and ity to managelimplement aspirations and efforts

equipment the learning activity in 3.1.2.7 Managers of media to

question aid the learning

1.1..1 Preparing learners 2.2.1.2 Estimate the cost of process

to learn training that would be 3.1.2.8 Deliverers of informs1.I.S.2 Presenting learning needed to prepare the tion

content to learners prospect to manage/ 3.1.2.9 Coaches of learners

1.1..3 Guiding learners' implement each activity 3.1.2.10 Evaluators of learners'

application of learn- 2.2.2.3 Formulate decisions to progress

ing provide or not provide

1.1..3 Testing learners' facilitator training for 3.2.1.1 Performance objectives

achievement of learn- each prospect on each specifications couched

Sing learning activity In terms relevant to

the given learners'

fl 1.1.6.1 Assembling learning- 2.3.1.1 Identify specific perfor- needs

relevant experiences mance objectives for 3.2.1.2 Plans for learning

1.1.6.2 Acquiring and/or which job performance activities experiences

fabricating resource training is needed that are in accordance

eaterials Z.3.1.2 Analyze the objectives with the given learnto identify the learning ers' needs

"" 1.2.1.1 Informing learners of events/activities needed 3.2.1.3 Presentations. applithe learning oblec- 2.1.3 Select implementing cations, and tests

tives methods/media that are consistent

1.2.1.! Relating objectives 2.3.1.4 Prepare/assemble de- with the given learnto learners' own tailed prescriptiona of ers' intended applicaneeds the methods and media tions of learning

1.2.1.3 Relating objectives 3.2.1.4 Complete sets of

to learners' own 2.3.?.1 Identifv specific perfor- refinements and de-

" experiences mance objectives for tails necessary to

W which facilitator train- Implement the turrtcu1.2.2.1 Gearing presentations nop is n-eded lure in a fashion conto learners' ah h- 2.3.2.2 Analyze the objectives sixtent with ikarners'

ties and achievements to identify the learning needs

S1.2.2.Z Selecting meaningful eventsIactivities needed

images, patterns. 2.3.2.3 Select implementing 3.2.2.1 Performance prerequietc.. to vivify methods/media rite specifications

content 2.3.2.4 Prepare/assemble de- couched in terms role1.2.Z.3 Utilizing multiple tailed prescriptions of vant to the given

senses to reinforce the methods and media learners' experiences

leaming 3.2.2.2 Plans for learning

1.2.2.4 Involving learners 2.4.1.1 Identify specific per- sctivitiesIexperiences

actively in the formance objectives re- that draw upon the

presentation leting to familiarity given learners' experiwith the curriculum and ences

1.2.3.1 Exposing each step in instructional require- 3.2.2.3 Presentations and

the procedure manta applications that make

1.2.3.2 Clarifying the pur- 2.4.1.2 Analyse the objectives use of the given

pose for each step to identify the learning learners' experiences

1.2.3.3 Manifesting perfor- events/activities needed

mance technique 2.4.1.3 Select impltmenting

1.2.3.4 Involving learners methodstme |s

actively In the 2.4.1.4 Preparetsseemble dedemonstration tailed prescriptions of

the methods and media

1.2.4.1 Selecting and creatIng opportunities for 2.4.2.1 Identify spedfic per-

. I practice formence objectives re1.2.4.2 Involving learners lating to familiarity

% actively In practice with the requirements

1.2.4.3 Sequencing practice of a given delivery

experiences to application

enhance learning

development

Figure 8 (Continued)

-38-

-. ... : . . . .-V "
PERFORMANCE PERFORMANCE PERFORMANCE

POTENIAUIES PROCESSES PR11ODUCTS

2.4.2.2 Analyze the objectives

to identify the learn1.2.4.4 Develo~ping and main- Ing eventsa/activities

tamning realism In needd

practice 2.4.2.3 Select implementing 1.2.4.S Observing and tutor- methods/media

ing learners in 2.4.2.4 Prepare Iassemble depractice tailed prescriptions of

1.2.4.6 Helping learners the methoda and media I sense progress toward

learning objectives 2.4.3.1 Identify specific per- and goals formance objectives

Diagnsingrelating to developing 1.25. Dagocngspecific additional detail.

needs for learning 2.4.3.2 Analyze the objectives

progre ~~~~Ingevnstaiiis

1.2..3 iago~i~g dficen-needed

ci' es intelearning 2.4.3.3 Select implementing

environmnent fmethods/media

2.4.3.4 Prepare/Iassemble de1.3.1.1 Clarifying the value tailed prescriptions of

of the learning to the methods and media

the learners

1.3.1.2 Demonstrating enthu- 2.S.1.1 Implement activities to

siasm prepare facilitators to

1.3.1.3 Encouraging learners' achieve the needed job

aspirations and tnt- performance skills

tatives 2.S.1.2 Implement activities to

1.3.1.4 Rewarding learners' present/ demonstrate the

progress and partici- skills to facilitators

pation 2.S.1.3 Implement activities to

S 1.3.1.5 Demonstrating pa- insure that facilitators

tience and compassion practice the skills

2.5.1.4 Implement activities to r1.3.2.1 Cresting situations test the facilitators'

1cand learning activi- achievement of the 8.1ties that are inher- needed job performance .4.. ently interesting skills < 1.3.2.2 Conveying a sense of

exiemn 2.S.2.1 Implement activities to z 1.3.2.3 Dramatizing for em- prepare facilitators to

phasia and rein- achieve the needed

forcement learning facilitation

M1attention ing to learners' 2.S.2. 2 Implement activities to

and present/ demonstrate the

interest skills to the facilitatorl; 1.3.3.1 Exploiting learners' 2.5.2.3 Implement activities to

experiences through Insure that facilitators

discussions, case practice the skills

studies, role- 2.S.2.4 Implement activities to

playing. etc. teat facilitators'

1.3.3.2 Emphasizing Immedl- achievement of the I acy of application needed learning facill- of learning to tation skilla learner-relevant

problems/needs Z.5.3.1 Implement activities to

prepare facilitators to

1.3.4.1 Reaiing and maintain- achieve the needed cur- ing learners' level riculam implementation

of aspiration for Wbiles

*achievement 2.5.3.2 Implemsent activities to

1.3.4.2 Mediating and fadli- present/demonstrate the

tating the triunsfer abilities to the facill- -. ~from training to tators

aperetions

Figure 8 (Continued)

-39-

:7
PERFORMANCE PERFORMANCE PERFORMANCE POTENTLAUTIES PROCESSES PRODUCTS

Z.5.3.3 Implement activities to 1.3.4.3 Serving as a mentor insure that facilitators

to extend and control apply the curriculum learners' talent Implementation abilities

1.3.4.4 Prompting and cuing 2.5.3.4 Implement activities to

, learners, 

ery when neces- test the facilitators' curriculum implement&-

t.on abilities and 1.3.1 2iagrooas Convey.ng application products

.. ":" of mince learners' to leaurners perfor-

*.',1.3.5.2 Positively rein fore-

• "' Ing desirable performnance

1.3.5.3 Constructively criticizing deficient

performance

1.3.6.1 Assisting learners to

develop and apply self-evaluation procedures suited to

learning objectives

1.3.6.2 Assisting learners to

diagnose their own needs for learning

1.3.,.3 Assisting learners to

translate diagnosed

needs into specific

Ing activities

1.3.7.1 Assisting learners to

plan and organize

their learninR

activities ' 1.3.7.Z Assisting learners to " Implement their learn-

) ing activities

i '?: 4C 1.3.7.3 Asssting dealingj with learners In problems

'%" 1affecting their learnU ing progress/achievement

1.3.7.4 Interacting with

learners to bolster

morale and raise aspirations as learners'

motivation and efforts

diminish

1.4.1.1 Knowing the elements -'. Iof the cognitive (factual) aspects of the

training

1.4.1.2 Knowing the elements

of the psychomotor

(skill) aspects of the

training 1.4.1.3 Knowing the elements

of the affective

(attitudinal) aspects of the training

1.4.2.1 Demonstrating and

articulating the cog- .* nitive tasks of the

job 1.4.2.2 Demonstrating and

articulating the psychomotor tasks of the

Figure 8 (Continued)

-40-

........... ,a 16..2
NPOTENTIAL1m 

PERFORMANCE PERFORMANCE PERFORMANCE PROCENES PRODUCTS

1.4.2.3 Demonstrating and

articulating the attitudes required of the

Job

1.4.3.1 ] Relating practical applications of the in- tended learning by role-model perforniera

1.4.3.2 Relating beneficial

"6 learning experiences

Ueed by previous learners

1.4.3.3 Facilitating interchange of relevant

experiences of current

learners

1.5.1.1 Understanding the intended goals of learnIng 1.5.1.2 Understanding the intended performance

objectives 1.5.1.3 Underotanding the prerequisite abilities

expected of learners

1.5.2.1 Understanding the

topics to be covered

1.5.Z.2 Understanding the

association$ among topics and performance

objectives 1.5.2.3 Understanding the organization of the

Content's topical

sequence

1.5.3.1 Understanding the perdonne] resources =€ available and needed

) 1.5.3.2 Understanding the

equipment resources available and needed

1.5.3.3 Understanding the

materiallmedia resources available and

needed

1.S.3.4 Understanding the

facilities resources

available and needed

1.S.4.1 Understanding the

preparation activities

and procedures to be

used

1.5.4.2 Underetanding the pre- 8entation activities

and procedures to be

used

1.5.4.3 Understanding the ap- plication activities

and procedures to be

used

1.5.4.4 Understanding the

evaluation actieiies and procedures to be

used

Figure 8 (Continued)

-41-

.

..

%

- * .~- * ---~. a 

-

r4
PERFORMANCE PERFORMANCE PERFORMANCE

POTENTIALMES PROCESSES PRODUCTS

i.5.S.1 Understanding what conditions are to be

controlled during

learning activities

!.S.5.2 Understanding how to

control those cond1.6.1.1 Capitulizing oan facilitators' own taperiences that are relevant to the intended

learning 1.b.1.Z Selecting learning activity options that are compatible with

facilitators' own instructional skills 1. 6.1.3 Developing procedural

i details to enhance failitators' delivery ? £ of instruction

1.6.1.4 Acquiring andlor re- fining materials to

enhance facilitators'

delivery of instruction

1.6.2.1 Deleting performance

objectives that are - not relevant to the

learning needs of a given delivery appli- cation

1.6.2.2 Defining new perfor- mance objectives relevant to the learning

needs of a given W delivery application

. 1.6.2.3 Modifying content to < reflect changed objectives

1.6.2.4 Modifying resources to

U reflect changed objectives 1.6.2.5 Modifying procedures

and methods to reflect

changed objectives 1.6.2.6 Modifying conditions

to reflect changed

objectives

1.6.3.1 Modifying resources to

reflect specific con- straints

1.6.3.2 Modifying procedures

and methods to reflect

specific constraints

1.6.3.3 Modifying conditions

to reflect specific

constraints

Figure 8 (Coot iowed)

-42-

S., / , ..- .w."%'.', ~ . - . . ... .,... . . - -- - ..
I under Performance Potentialitites, each population category is identified as 1.1, ~ 11.2, 1.3, etc. The Functional Purposes Level descriptions is assigned a two

decimal code, i.e., 1.1.1, 1.1.2, 1.1.3, etc. Finally, the Characteristics

Level of system description is assigned a three decimal code, i.e., 1.1.1.1,

1.1.1.2, 1.1.1.3, etc. The decimal numbering of the population categories

simply aids in organizing the population categories in such a way so that the

-J analyst can easily trace the subpopulation categories of Objectives Level,

Functional Purposes Level and Characteristics Level populations.

toSubsequent segments of this report illustrate the application of these taxa

toproduce measures and design specifications for representative military

training systems. To prepare for those sample applications, this segment

'~: *closes with brief clarifications of the design and measurement implications of

one taxonomiic subset of the Design subsystem. This clarification also will

serve to elucidate certain of the terms and concepts of a general training

systemn.

Sample Taxonomic Subset

*Training Subsystem: Design

Performance Category: 1.0 Performance Potentialities

Objectives-level taxon: "1.2 Establishing Performance

Objectives"

Derivative Functional Purposes-level taxa:

"1.2.1 Stating what the learners will

be able to do or how they will be

e-xprted to behave after completing

training"

"1.2.2 Providing a basis for objectively

assessing learner performance"

Derivative Characteristics-level taxa:

"1211Dfnn6h.efomneato" 11221 eeoigpeeties

"11.2.1.1 Defining the performance conto" "1.2.2.1 Developing prettest items"

"1.2.1.3 Defining the performance standard"

One of the many fundamental capabilities that the Design subsystem is

expected to provide for the benefit of the total training system is the ability Jto establish performance objectives for learning. The Command subsystem

-43-

.J &. &A
-~Z~ j decrees that learning will take place, having identified a "job" for which

training is needed. However, Command defines the "job" only in broad terms,

e.g., "we need to train soldiers to man the Bradley Infantry Fighting BVehicle." It is up to Design to translate that broad statement into a

4 behavioral reference framework that is compatible with learning. Given that

:4 learning is sensory activity intended to affect behavior, one of the Design i~j~ subsystem's first responsibilities is to establish exactly what behaviors are to

be affected, and exactly how they are to be affected. In training system

terms, that is called "establishing performance objectives."

* Performance objectives are simply behaviors, expected of the learner upon

completion of training, that can be defined and observed in quniibeterms. Precise definition of observable behavior ibsuty ssi7i t

determine the trainin~g content and methods and to evaluate training

effectiveness. A training designer would never (or should never) say

something like "one of my objectives is to give the BIFV gunner trainees an

appreciation of the kinds of targets they can engage with the weapons on this

vehicle." There is nothing quantifiable or directly observable about

"lappreciation." What does a learner have to do to acquire an "appreciation"?

* How can one tell whether one learner has achii-ved more "lappreciation" than

another? Of course, what the designer really means is that the

gunner-learners should come to know what targets can and should be engaged

by each type of weapon. That 31-siabetter statement of the training purpose,

but it still isn't expressed in behavioral terms. A true performance objective I expressing what the designer really intends might be stated as follows:

"Given a list of the types of targets to be entgaged, the learner will state the

appropriate weapon /ammunition to be used to engage each target." The

intended behavior is a series of audible statements of weapon /ammunition type,

which is readily quantifiable as to the accuracy of each statement.

In the absence of valid performance objectives, no one really knows what

the intended behavioral outcome of the training is supposed to be, and so no

one can properly design or evaluate the sensory activities that are to form the

:. jtraining experience. Thus, both the designer and the evaluator of any training system must concern themselves with the issue of the Design

subsystem's ability to establish performance objectives. At the top-most level

-. :~j,.jof design and evaluation, crucial questions such as the following must be

asked:

0 Is there a person (or persons) assigned to the Design subsystem

who has the knowledge and skills needed to establish valid

performance objectives for training?

0 Do the plans submitted by Design subsystem for Command's review

and approval call for the establishment of performance objectives?

o Do those plans allocate appropriate manpower and other resources to

the establishment of performance objectives?

.1Y

-44-

%~ S-
0 Do) the plans provide for obtaining the input information necessary

for the establishment of performance objectives?

P Once the designer and evaluator are satisfied (from their different

perspectives) that the basic capability of "establishing performance objectives"

has been provided in the Design subsystem, their attention shifts to

ascertaining how and whether this capability can satisfy its functional

purposes. The Design subsystem is expected to be able to establish

performance objectives for two reasons: first, so that it will be possible to

state what the learners will be able to do or how they will behave subsequent

to training; second, so that it will be possible to develop an objective basis

for assessing learner performance in training. At this level of detail, the

designer and evaluator deal with such issues as:

0 Does the basic capability for establishing performance objectives lend ''I itself to specifying learners' terminal behaviors?

o Do the plans and procedures used by the Design subsystem call for

specifying learners' terminal behaviors?

0 Is the basic capability for establishing performance objectives

actually being applied to prepare for learner performance

assessment?

Finally, the designer and evaluator shift their focus to the technical

details necessary to apply the capability of establishing performance objectives

to achieve the two functional purposes. Numerous issues come to their

attention at this stage, including the following:

o Does the Design subsystem's capability for establishing performance

objectives, extend to the specification of the actions that the learners

are to perform? Does the capability for specifying the actions insure

that the actions are observable and quantifiable? Does it insure that

the actions specified accurately demonstrate the intended behaviors?

0 Does the capability for establishing performance objectives extend to

the specification of the conditions under which the learners are to

perform the actions? Does it insure that the conditions specified are

relevant to actual job requirements?

0 Does the capability for establishing performance objectives extend to

the specification of standards of acceptable performance? Does it

insure that the standards specified are appropriate and relevant to

actual job requirements?

0 Does the liesign subsystem possess the capability of developing 4:' means of testing learners' performance prior to training? Does that

capability insure that the test items reflect the specified performance objectives? Do Design subsystem plans call for establishing and

applying such pretest items?

-45
0 Does the Design subsystem possess the capability of developing

means of testing learners' performance after or at completion of

training? Does that capability insure that the test items reflect the

specified performance objectives? Do Design subsystem plans call for '.5 establishing and applying such posttest items?

Training system designers and training system evaluators work with the 'Ivery same system performance requirements. The designer's genius lies within

his or her ability to select the approaches, methods, and means for insuring

that the rectirements are satisfied by the system being built. The evaluator's

genius is found in his or her selection of the approaches, methods, and means

of determining how well the designer's job was done. After they have both

completed their work, the evaluator is required to "report back" his or her

d fincings so that the designer may correct any deficiencies that may have been

uncovered. With respect to the Design subsystem performance requirements

surrounding the potentiality of "establishing performance objectives," the

fvaluator could submit any of the following "reports" to the designer:

Possible Report No. 1:

"Look, you never saw to it that your Design subsystem included a

capability for establishing performance objectives. None of the

people that you assigned to the subsystem had any knowledge of

what constitutes a valid, useful performance objective, and none of

them had any experience in conducting the kinds of front-end

analyses nueded to identify appropriate objectives. The result was

that the so-called job and task analyses that they performed were

* badly botched and are just about useless, and their so-called

training objectives don't deserve the name. The Design subsystem

has failed to specify what the learners are supposed to be able to

do. That means there is no way of telling whether the curriculum's

content, schedule, sequence, or procedures are any good at all."

Possible Report No. 2:

"Okay, you provided the basic capability for establishing

ri performance objectives when you assigned John Jones and Mary

Smith to that task. They have solid credentials in that area;

they're good analysts, and you gave them sufficient resources to get

the job done. They produced what appears to be a very good set of

statements concerning the learners' intended achievements: the

actions, conditb6ns, and performance standards are well described

and based on very thorough job analyses. The problem is, yru

didn't see to it that their excellent work would be applied to

assessing learner's performance. The people you assigned to the

* task of developing test items seem never to have even considered the

performance objectives, and certainly never consulted John or Mary.

At least half of the stated performance actions aren't elicited by any

* '.~-46-
of the test itcems,- either or post-. Almost all of the actions

that are elicited are called for under conditions that aren't even

close to those specified in the performance objectives. The net

result is that you won't be able to tell whether your learners have

actually achieved their intended learning."

I Possible Report No. 3:

'John's and Mary's assignment to the Design subsystem provided a

solid capability for establishing performance objectives, and you were

I wise to task them also with the assignment of preparing pre- and

post-test items to address each objective. They're -good, hard,

skillful workers, and they produced excellent statements of the I performance actions and conditions required of the learners, and

some very well constructed test items. The only problem is that you

didn't give them quite the support they needed. Mary tells me that I she and John just didn't have enough time and funds to do a proper

analysis of the job performance standards. She admits that some of

the criteria they state in their performance objectives are largely

wild guesses. I'd suggest that you scrape up some more funds to

allow them to study the jobs and tasks to greater depth, so that we

1 can be sure that the performance standards reflect what really is

required in the field. Otherwise, we might find that we are either

undertraining or overtraining our learners, and we might be giving * j them tests that are either too easy or too hard."

Possible Report No. 4:

"You did a fine job in seeing to it that the basic capability for

establishing performance objectives was built into the Design

subsystem. In choosing Mary and John to operate that subsystem,

you obtained the services of two of the best in the business. The y

tell me that you saw to it that they received all of the resources and

support they needed to do the job the right way. It's clear that

your system had everything needed to produce comprehensive, well

stated, and complete performance objectives, and to reflect the

objectives in the test items that were developed. There is absolutely

no fault to find with this element of your system's performance."

: I These are not all of the possible reports that an evaluator might submit

cciicerning a Design subsystem's potential for establishing performance

*objectives, but they do span the full range of possible findings. In the first

report, the evaluator finds that a basic required potentiality simply is absent

J ~ from the system being studied: this particular system has no capability for

3establishing performance objectives. This is a failure detected at the

' objectives level of design /measurement. A fundamental element of the "what"

* ,,of performance is unsatisfied. Note, too, that the failure is diagnosed as a

missing 2 t+f ialt. The evaluator does comment on the fact that there are

Iprocess tailures Kadly botched job and task analyses") and product failnires

-47-
*'. i ("stated performance objectives that don't deserve the name"), but it is clear

that the evaluator perceives these to be the natural results of the more

fundamental potentiality failure ("none of the people had any knowledge of or

* experience with establishment of performance objectives"'. At this stage the

evaluator is focusing on a taxon of potentiality. In h - _r her complete

report, detailed comments presumably also would be made on the related

process and product taxa, i.e., the evaluator would document exactly how the

job and task analysis processes were "botched" and exactly how the stated

performance objectives products were deficient.

in Report No. 2, the evaluator finds a different situation. Here, there is

evidence th:it the essential requirements for establishing performance objectives

were met. Qualified people were brought to the assignment, and they were

given the tools needed to do the job of producing the objectives. Indeed, the

et'aluator finds that they did produce the objectives. However, their

capabilities and their produc-ts were not applied to the task of providing a

basis for assessing learner performance. This is a failure on the function:i

purposes level of design/measurement, a deficiency in the "why" of this

system's performance. A key potentiality was made available in the system,

but the system failed to apply that potentiality properly.

The evaluator finds still a different situation in the third report. The

system has the basic (objectives level) potential for establishing performance

objectives. The system applies that potentiality to achieve its two intended

(functional) purposes. However, detailed analysis discloses that the

potentiality breaks down at the characteristics level of design/measurement,

i.e., in the "how" of performance. The potentiality for establishing

performance objecti,'es is deficient in one characteristic, namely, the ability to

specify stindards of performance. The evaluator traces this to an

insufficiency in the resources provided for applying the capability, which

prolably is a typical reason for a malfunction on the characteristics level.

At this point it is approrpiate to reflect on the hierarchical relationship

* that binds the objectives, functional purposes, and characteristics levels of

system design/measurement. It is clear that a system performance failure on

the objectives level automatically implies corresponding failures on the

functional purposes and characteristics levels. Very simply, if a system does

not possess some essential potentiality, or does not carry out some essential

proce!:s, or does not deliver some essential product, there is no way that the

system can then apply that potentiality, process, or product to serve its

intended purposes: you can't use what you don't have. Also, the absence of

the potentiality, process, or product itself certainly implies the absence of all

of its characteristics that are required for any intended purpose. However,

the rmere fact that the system satisfies its objectives-level performance

requirements does n t guarantee that the associated functional purposes and

. characteristics requirements will be met. The system might have all of its

basic potentialitie::, carry out all of its basic processes, and produce all of its

basic products, but fail to apply them for the required purposes. Further,

those basic potentialities, processes, and products may not have exactly the

* right characteristics needed to satisfy those purposes.

-48-

. . . .. . . * .. . .. . . . . . . . . . . . . . . .
i il In the evaluator's fourth report, a happy finding is documented. The

system under study has everything necessary to insure that the potentiality of.

- establishing performance objectives is present in all of its characteristic

details, and will be applied to serve its intended purposes. The evaluator

- commends the designer on this particular element of his or her work.

But what exactly is the evaluator saying in that excerpt? The statement

on says that a requirement for potential performance has been met. The

evuator has worded the statemen~t in a way that carefully avoids saying

9 J anything about the processes carried out or the products delivered.

NPotentiality does not guarantee achievement. Any person, no matter how

capable, can make mistakes. Any machine can break down. In the context of

the simple example we have been using, it is possible that John Jones and

Mary Smith could "botch up" a job analysis in spite of all their experience.

The system designer and the evaluator cannot be satisfied with seeing to it

~ j that the system has the potential to do its job. They also have to see to it

that the proper processes are planned and implemented and that the proper

products are specified and actually delivered.

This brings us to the point where it is appropriate to reflect on the

hierarchical relationship involving the three aspects of system performance,

viz., potentialities, processes, and products. Their relationship Is similar to

that involving the three levels of system description. The absence of some

required potentiality will give rise to corresponding breakdowns in the

system's processes and products. If a basic capability is missing, the process

used to apply that capability cannot be carried out successfully. If the

process cannot be carried out, the fruits of the process (i.e. , some product)

will not be delivered as needed. On the r'1.her hand, the potentiality might be

there but the processes using that potentiality might nevertheless go awry. A

process might be carried out without fault, but the resulting product might

nevertheless be deficient. The expression "garbage in, garbage out" familiar

to data processors refers to this situation: the data processing might be fine

but the output products might be worthless, because -tF-inputsE also were of

no value. Thus, just as each level of system description must be designed

and evaluated in turn, so must each aspect of performance. The system

designer and the system evaluator should always proceed from general to

specific, from the basic to the detailed, from the potential to the actual, to

insure that all appropriate steps are taken to bring about a system that does

exactly what it is supposed to do. Figure 9 illustrates this essential

~ I developmental flow within the general systems taxonomy structure. The basic

flow for either design or measurement applications is from the most fundamental

requirements of potentiality to the most detailed requirements of product,

~ I always insuring that all intermediate processes and purposes receive careful

attention. Sample applications of this concept are given in the segments that

* follow.

-49-
CD C

LL 0

C00

uiU

00

050

cc*

a. C)- ~ '**,
C The APM for Measurement of Training System Performance/Effectiveness

The purpose of this segment is to illustrate how the performance taxa

identified for any particular human-machine system can be applied to derive

useful measures of that system's performance/effectiveness. The approach

taken proceeds through the following steps:

(1) Define precisely the system of interest.

(In this example, the system of interest will be one subsystem

of one portion of an emerging Army training system.)

(2) Define precisely the measurement application.

(Here, the application will focus on one aspect of the training

:, system's planned learning activities.)

(3) Identify the particular performance taxa that are relevant to that

measurement application.

(Not all of a system's performance requirements are of interest

to every measurement application. This example will illustrate

.- .the procedures to be used to determine which requirements are

relevant and which are not.)

"* (4) Establish the issues and implications for measurement that derive

from each relevant taxon.

(The derivative issues and implications clarify exactly what

.' relevance the particular performance requirement has for the

-measurement application at hand.)

(5) Fosit measures that will address each issue and implications.

(Here, the measures will be workable means of determining

whether the training system satisfies its relevant performance

requirements in a way that insures the adequacy of the

particular learning activities being examined.)

1. The System of Interest: The Design Subsystem of

BIFV Gunnery Trainini

The U.S. Army Infantry School presently is developing a training

system that will be employed to supply qualified operators for the new Bradley

Infantry Fighting Vehicle (BIFV). This new Bradley Infantry Fighting Vehicle

Training System (BIFVTS) will prepare soldiers and officers for assignments as

BIFV squad members (skill level 1 soldiers); BIFV drivers; BIFV gunners;

BIFV Commanders; and various support functions on various levels of command

(platoon, company, etc.). Figure 10 depicts the network of other systems

that interact with BIFVTS. These include:

0 The system known as the U.S. Army Infantry School. At its

A present stage of evolution, BIFVTS is wholly contained within

the Infantry School, which therefore is a BIFVTS suprasystem.

As will be shown below, the school presently operates the

. BIFVTS Command and Design subsystems.

-51- :-- 
jU.S. Army Fr U.S. Army iInfantry eng Armor I School Lgsi a col I (Supra-syste m)Supr

Braidley BalyTan

CavalryInatyI- t n Fighting FightngIfant__g_

VehicleVeil nrn

TrainingTriigTang3

Sy1e S stem Sse

Typ I

Bradldlyy ; !~ICavalry

FFighting

Vehiicle

IM I

:1 Figure 10. Major Systems Interacting with BIFVTS

-52-
a Various existing and possibly emerging infantry training

systems other than BIFVTS. Initially, BIFVTS will design and

deliver "add-on" training in BIFV operations, which training

must be compatible with other on-going training that the BIFV

learner-operators have or will have received. At a later stage

of evolution, BIFVTS will be integrated with other infantry

training systems, a step that will require careful coordination '- with those systems. At present, those systems are collateral to

BIFVTS.

.- 0 Various U.S. Army infantry units. BIFVTS learners, facilitators

and other personnel will be drawn from infantry units. Those

units thus are human-machine systems collateral to BIFVTS.

The Fort Benning Logistic Support System. This entity works

closely and often with the Infantry School, although it is f functionally distinct from the school. The Fort Benning

Logistic Support System will operate a significant portion of the

BIFVTS Logistics subsystem, especially during the early stages

of BIFVTS operation. In the future, as greater shares of

BIFVTS operation move "into the field," the infantry units will

take over increasingly more of the Logistics functions in

support of training.

0- The BIFV itself, together with its full complement of personnel

and equipment. The BIFV clearly is an essential collateral

system to BIFVTS.

0 The Bradley Cavalry Fighting Vehicle Training System

-" (BCFVTS). This is a fraternal twin of BIFVTS, and many of

its operations and performance requirements will be identical to

those of BIFVTS. Pooling of resources and sharing of

information and products between these two systems is highly

desirable if not absolutely essential. (Parenthetically, BCFVTS

>is itself imbedded in an interactive network of systems that

exactly parallels the BIFVTS network. A portion of the

BCFVTS network is also shown in Figure 10.)

Since BIFVTS is a training system, it must (according to one of the

fundamental axioms) incorporate the six general training systems within itself.

How are those subsystems structured and operated in BIFVTS?

The BIFVTS Command subsystem is operated and equipped by the

" BIFV Task Force, an a hoc subsystem of the U.S. Army Infantry School.

The BIFV Task Force has been created and staffed to exercise overall

responsibility for the design, development, testing, and final production and

dissemination of BIFV training. The specific immediate responsibilities of the

Task Force are to:

- ,5 . . .- ". 5..5 . S . .
%

: 0 Identify the specific skills and knowledge required of BIFV

operators.

0 Develop BIFV training strategies.

0, Establish new equipment training (NET) for the BIFV.

1 0 Administer, monitor, and evaluate all BIFV training

development, testing, and production.

The reader will recognize that several of the above-listed

responsibilities are actually training design functions. Indeed, the BIFVTS

Design subsystem also is operated (at present) solely by members of the BIFV

Task Force. Their current design work is oriented toward the production of

five distinct curricula:

0 An "add-on" course for Skill Level 1 BIFV Soldiers (MOS

IIMI0).

0 A Basic Gunnery Course (for Skill Level 2 Soldiers (MOS

1IM201 and some Skill Level 1 Soldiers).

0 A BIFV Commander's Course (for Skill Levels 3-5 111M30,

11M40, 11M501 and for officer grades 01-05).

0 A transitional new equipment training (NET) course covering

the above-listed skill levels.

•~ i 0 A Master Gunner's Course.

" The Master Gunner's curriculum principally is intended as an input

to the BIFVTS Enabling subsystem. One E7 or E8 from the S3 section of each

BIFV mechanized infantry battalion will receive Master Gunnery training, as

will one E6 from each BIFV company. The Master Gunner's duties include the

administration, supervision and delivery of Basic Gunnery training within his

or her unit. The Master Gunner, in effect, will be a field

facilitator/administrator within BIFVTS. Other current or planned operations

within the BIFVTS Enabling subsystem include the organization and deployment

of a team of facilitators to support new equipment training on a transitional

basis as the BIFV is introduced into operational units; familiarizing selected

mechanized infantry instructor/facilitators with the emerging BIFV curricula;

and delivering Mastery Gunnery training.

It is not presently envisioned that the BIFVTS Enabling subsystem

will provide formal instructional skills training to all selected facilitators.

Instead, persons who are selected to serve as BIFVTS facilitators will be

recruited from among those who already possess instructional delivery and

guidance skills. Note, however, that this approach in no way changes the

performance requirements faced by the Enabling subsitiem.- That subsystem

-54-

.,-,%
~K still must have the potentiality for insuring that adequate instructional skills

resources are provided for BIFVTS Delivery applications. It still must carry

out the processes of assessing the candidate facilitators' instructional skills

and of diagnosing their needs for improvement and remedial training. It still

and always has the responsibility of producing facilitators who are qualified in

every respect for their assignments. The decision to recruit experienced

instructors rather than to provide basic instructional training to novices is a

system design choice that may or may not be optimum in a particular case,

depending on the exact circumstances. The design choices taken are simply the variable means to the immutable ends represented by the performance

requirements, or taxa. The system evaluator is far less interested in the

approaches the designer took and in why those approaches were taken than he

or she is in determining whether the system--as designed-does the job it is

V~ Zsupposed to do. It may well be (and probably is) an excellent decision of the

BIFV Task Force to recruit experienced instructors to serve as BIFV training jfacilitators. But that decision, for good or ill, does not lessen the BIFVTS

Enabling subsystem's responsibility for providing fully qualified facilitators.

The r3IFVTS Emplacement subsystem is responsible for the design

and selection/ fabrication of al audi'ovisuals and other learning aids necessary

to support implementation of the various curricula, and for the acquisition or

construction and preparation of all sites at which learning activities will take

place, whether in a classroom or field setting. One important element of this

responsibility will be the design and construction of gunnery practice ranges.

The ran~ge requirements presently envisioned include:

0 Firing port -weapons range

0 BIFV Basic Gunners range 0 BIFV Squad subcaliber range

o BIVV Squad qualification range

0 Platoon qualification attack range

0 Platoon qualification defense range

Note that all of the BIFVTS Emplacement subsystem's performance

requirements derive from the need to insure that all facilities and materials

necessary to support delivery of BIFV training are available and operable at

the times and places needed. The Emplacement subsystem designers might

determine that the best way to insure the availability of ranges where and

when needed is to construct six separate ranges, one for each gunnery 7 requirement. Conversely, they might analyze the proposed Delivery schedules

and conclude that two physical ranges will suffice, time-sharing the six

requirements between them. The former approach might satisfy the

subsystem's product performance requirements, but if it turns out that the ~ I ranges are significantly underutilized, the evaluator may conclude that certain

analytic process performance requirements were not met, resulting in an

expensive, wasteful Emplacement subsystem. Similarly, the two-ranges approach might turn out to have been a brilliant design choice, but not if

S severe scheduling conflicts arise because the two ranges cannot handle the

training load.

-55-

***- *%* *-* *-*
The BIFVTS Logistics subsystem presently is operated solely as a

component of the Fort Banning logistic support system. In time, increasingly

greater shares of the operation of that subsystem will be exercised by various

mechanized infantry units in the field.

The BIFVTS Delivery subsystem will allow and insure that learning

takes place in the contexts o the various curricula produced by the Design

subsystem. Soldiers and officers will carry out prescribed sensory activities,

guided and assisted by facilitators, and will achieve the specified performance

objectives applicable to their BIFV operational assignments.

BIFVTS th-.s is a well structured training system, possessing

functionally oriented subsystems compatible with the general training model and

pursuing a broad range of administrative, developmental, and delivery goals to

support the introduction of the new BIFV into the mechanized infantry. It

does not detract at all from that image to note that BIFVTS can also be viewed

as a loose consortium of smaller scale, relatively independent training systems

that pursue more narrowly defined goals. These smaller systems are organized

around the five BIFVTS curricula now in development. These systems may be

denoted as:

0 The BIFV NET System 3 0 The Basic BIFV Soldier Training System

0 The Basic BIFV Gunnery Training System

- 0 The Master BIFV Gunnery Training System

0 The BIFV Commander Training System

Each of these has its own Command, Design, Enabling, Emplacement, Logistics,

and Delivery subsystems. Their development is proceeding more or less in

tandem, and there is significant overlap in their objectives, content, and ,

:. 1 learning activities, as well as in their administration. But they are

independent in the sense that any one of them could be developed and

operated in the absence of the others. (Admittedly, there wouldn't be much

point in designing, developing, and implementing Master Gunnery training if

there were to be no Basic Gunnery training, but it still could be done.)

In order to reduce the scope of this trial application to a practical

level, the project staff elected to narrow their focus to the Basic BIFV

Gunnery Training System. Like all systems, this one operates in the context

of an interactive system network of suprasystems, collateral systems, and

subsystems. The network is shown in Figure 11. The Basic Gunnery system

is subservient to such entities as the U.S. Army Infantry School, the Fort

Benning Logistic Support System and, of course, the total BIFVTS. It

interacts collaterally with the other four independent BIFV training systems

and with portions of U.S. Army Infantry units, namely, those portions from

which the E4's and E5's will be recruited as candidate gunners. It also

interacts collaterally with the BIFV Carrier Team subsystem. That is the

portion of the total BIFV system for which the gunner-learners are to become

qualified as operators. And finally, the Basic BIFV Gunnery Training System

NE -56-

.S
Overall U.S. Army Fr

DIFVTS Infantry Denin - e'l- (ur ytm School Loinig U.S. Army BrSadley vilf (ur-ytn) (Supra-ystem) Supgotc

I Armor i I Fighting SkillLevel_

Sii~yystem

GuCar ner y unrse

T ya m SytmGunr

Unaitin Ne 

i~uEquipment Subtter

IB/- F- F 

e

i Care re

EUnaiten Lnigisticsr

Figurest_ Susy.e Major SytmAneacigwtrh

Basic~atr GunrITanngSse B4s, Uni

(Slcedqs E-swit-n

-57-11

..... .. .... . . ...- in... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .ion)

~~~~~~~o m n .'.E.* . .,b*~* *. f* *
has a fraternal (collateral) twinship with the Basic BGFV Gunnery Trainin g

*~: ISystem. The subsystems of Basic BIFV Gunnery Training are, of course, the

six familiar subsystems of Command, Enabling, Design, Emplacement, Logistics, and Delivery.

Even this narrowed focus remains too broad for a trial measurement

application. Thus far in the present research, performance taxa have been

identified only for two of the six training subsystems. The tools are not yet

* available to permit assessment of a total training system's performance, not

even one concerned with such a specific issue as BIFV Gunnery training. It 4 is necessary to narrow the focus further, to select measurement issues lodged

strictly within one of the two subsystems for which taxa now are known.

~ I We choose to address the Design Subsystem of BIFV Basic Gunnery Training, rather than the Enabling Subsystem, because of the following 4 considerations:

- Design operations are at a further stage of development than

are Enabling operations. Any inputs that this research can

provide for assessing Design performance would be more timely

for the BIFV Task Force than would inputs to assessment of

Enabling performance.

- Many of the performance requirements of the Enabling

subsystem relate to the facilitators' interaction with the

'~ Iproducts of the Design subsystem. Thus, it is logical to

establish first that Design has achieved its performance

requirements before attending to Enabling subsystem

performance.

U - As a collateral to all of the other training subsystems, Design

performance has a marked effect on the work that the other

subsystems can accomplish. Any strides that can be made now

toward measurement of the Gunnery Design subsystem will

provide a firmer base for future applications of the APM to

BIFVTS and other training systems.

Figure 12 depicts the network of systems that Interact most closely

with the Design subsystem of Basic BIFV Gunnery Training. As in any

training system, Gunnery Training Design is subservient to the Command I subsystem, and collateral to the Enabling, Emplacement, Logistics, and

Delivery subsystems. Key sub-subsystems of both Emplacement and Delivery

have been included in Figure 12 to emphasize the special gunnery instruction

and practice requirements that must be reflected in Design's products.

Gunnery Training Design is also collateral to the Design subsystems of the

other BIFV Training "packages. " This is because all of those packages share

some performance objectives, learning activities, etc. , with gunnery training.

.58-
COMMAND

Administration

and

Coordination

of the Basic

Training

Development Development Development Development of Curriculum of Curriculum of Curriculum of Curriculum

for for for for SLMaster BIFV BIFY New Skill Level 1 BIFV Gunr qupetBJFV Soldier Commanders TannTriigTraining Training

ENABLINGEMPLACEMENT

Pe ion Development of Instructors ve opeto aclte

Gunryfr VGunnery '

Syte IIriigSse

I .of InstructionSupr

Squadin andCrwiVhilenea

uni Trinn Training Exercise

~~Sytem I ofantee Thea Deind Subyem of haic Team Sunner ytmba Coiig 

(Continueded)

Figure 12. Sample Hierarchical Structure Centered on I a Subsystem of a Training System

-59-
-..,.*1 iU

P I e- ire 

Squad and" Gunnery Training I

Crew Instruction

Drill Instruction

Squad and

Crew

Subealiber " an -- - -- DELI--RY

Instruction Vehicle Team - ..... - Delivery adCrew - ombat - -- -- - - - ------- - of Instruction ._._______a____-_ for the

Sq uad In struc tion . . . . fo.. , theasi BI V

Combat -- ' - ---- " - . .... " " -1 Bai (un- V

Qualification . -. . . . . - - " Gunnery

Instruction -- -- Trainin

latoon/ - - -

Section

Battle Drill Platoon/

Instruction Section

Qualification I

Firing ort

EMPLACEMENT Wane Gunners

Development . . . .Raenger I S

of Facilities -----. ---- . . aier

f o r . . . . . . .

B a sic B IF V Sq u ad

Gunnery Rane-.....Qaii at

Training - - aoPlatoon Qu a ge i

System FIii~a-ion Qualification

Qualification Atak Mae lDefense Range]I

Carrier

Team

Subsystem

- ---- I - -a

[7.62m 25 TOW

Machine Gun Cannon ATOM Sub-Subsystem Sub-Subsyste m SulPSulsystem

Figure 12. Sample Hierarchical Structure Centered on

a Subsystem of a Training System (Cont'd)

-60-

L..... . . . .
The sub-subsystems of Gunnery Training Design correspond to the

thus far identified segments of the gunnery curriculum. These represent the

sequential stages of learning through which BIFV gunner-learners must pass to

become fully qualified. The initial stages stress individual training; the

latter, team training.

2. The Measurement Application: Specification of

-Gunner- Learner Testing

Having selected the Design subsystem of Basic BIFV Gunnery Training as the system of interest for this trial, it remains to select an issue

of focus for the application. It would be possible, if sufficient resources were

available, to examine the global performance of the Design subsystem, i.e.,

simply tu ask, "How well does the Gunnery Training Design meet all of its

performance requirements?" Within the scope of this present study, and given

the current state of APM application procedures, development of a

comprehensive and complete measures set would be unreasonably consumptive

of project time and funds. A more restricted measurement application seems in

". order, i.e., an assessment of how well Design meets some limited aspect of its

performance requirements. Of course, it is highly desirable to insure that the

aspect chosen for study is of interest and value to the BIFV Task Force.

What, then, are some typical aspects of Gunnery Training Design

" performance? What are the issues that might be of interest to system

designers and evaluators?

Recall that the fundamental function of any training system is

learnin. Everything that goes on anywhere in any training system, or in any

ofits subsystems, is supposed to contribute in some way to learning.

Anything within the system that detracts from or degrades learnin'g is a

. system defect. At rock bottom, all assessments of training system or

subsystem performance must supply some portion of the answer to this

";il fundamental learn? question: Did the learners learn what they were supposed to

What does Design contribute to learning? What light can an

examination of Design performance shed on the quality and quantity of learning

that the system delivers?

Design's basic job is to provide training specifications. It specifies

what is to be learned, i.e., the goals and (especially) the performance

objectives that learners are supposed to achieve. It specifies the context of

learning, i.e., the content of knowledge, abilities, and attitudes required o

the learning and the informational material that manifests the content. It

* specifies the learning activities, i.e., the sensory experiences that will affeLt

the learners' havior in the intended vmanner. The quality and quantity of

learning will suffer to the extent to which Design badly specifies what is to be

learned (e.g., produces poorly or erroneously stated performance objectives);

or to the extent to which Design badly specifies the learning context (e.g.,

-61-

, ..... ...................... . ... .. ........
by including topical material that has no bearing on the performance

objectives, thereby confusing the learners); or to the extent to which Design

badly specifies the learning activities (e.g., devises activities that produce - . ~ unexpected, deleterious effects on learners' behavior). Any issue that

pertains to some component of Design's specification of the learning objectives,

context, or activities is one very worth exploring.

The issue selected for this trial application is Gunnery Training

Design performance in specifying gunner-learner testing. That is a sub-issue

~ of its performance in specifying learning activities. The other sub-issues of

that heading are: specifying gunner-learner preparation activities; specifying -~ activities for making presentations (or demonstrations) to gunner-learners; and

specifying gunner-learner application (or practice) activities. Virtually all ~ I learning theorists* agree that preparation, presentation, application, and

*See Bibliography for representative documents.

.. ~ Jevaluation (or testing) constitute the four classes of learning activities.

The issue of how well a Design subsystem specifies learner testing

procedures is of special importance in any training system. Certainly, that

issue will affect the appropriateness of decisions made by facilitators and

learners themselves concerning whether particular learners need more training,

or any training at all, to achieve particular performance objectives. Even

* rno-e significantly, it will affect the appropriateness of the system's

certification of learners as L_ Apetent job performers, thus affecting the entire 7 jtraining system's reputation. Most basically, if deficient learner testing

specifications are produced by the Design subsystem, the entire training

system will be at a loss as to assessing its most fundamental performance

requirement: it will not really know whether the learners have learned.

U Thus, no training system can afford to allow its Design subsystem tQ produce

poor measures of learners' achievements. The Basic BIFV Gunnery Training J System is no exception.

Before moving on to the next step in this trial application, in which * J particular performance taxa associated with learner testing specification will be

* identified, it is worthwhile to set tne stage by informally reviewing some of the

requirements and implications of learner testing in any training application.

The following, at least, seem almost self-evident:

- Testing must be objectives-referenced

The purpose of testing is to evaluate learners' progress toward

or achievement of the performance objectives, i.e., that which

::: jis to be learned. No performance objective should go untested.

No test that does not relate to some performance objective

U should be included.

%L;

-62-
Testing must becriteria-referenced

It is not enough that a test simply requires the learner to

exercise the behavior called for in a performance objective.

The test must disclose whether the behavior is performed "well

enough" to satisfy learning needs.

'I - Some kind of _pre-training testing is necessary

All training assumes some prerequisite behavior. If nothing

~ else, it is always assumed that the learner is able to carry out

the sensory activities called for in the training. Many training

applications admit the possibility that some learners may already k' I have achieved certain of the specified performance objectives. There is no point in commencing training unless the learner

first is tested to insure that he or she possesses the assumed

prerequisites. A very informal test might suffice for that

purpose. perhaps only a records check or even a cursory

visual examination of the learner, but some testing for

prerequisites surely is needed. When there is reason to believe

that some learners may already have achieved the learning

objectives, pre-testing of those objectives also will be

necessary.

Testing is after all a learnn activity. Learners need to know

- *- how well they are doing in order to progress and persevere.

Any training system should provide at least some testing for

the purpose of feeding back progress and diagnosis information

to the learners.

Testing must diagnose learning deficiencies

It is not sufficient that tests disclose that a learner cannot

perform the intended behavior. In order to provide meaningful

feedback to the learner and to facilitate subsequent improvement

of the curriculum and the delivery, tests that diagnose the

specific ingredients of substandard learner performance must be

included in the specifications. Only then can the learner's

performance be corrected and brought up to standard.

- Testing must be reflective of on-the-job performance

* The tests specified must require the learners to manifest the

job-relevant behaviors under job-relevant conditions. The

testing conditions should resemble job conditions as closely as

possible, since the test outcome otherwise may not be a valid

predictor of the learner's job performance.

-63-
- Some kind of qualification testing is necessary

No learner should be "certified" as having successfully

completed training unless he or she demonstrates, through *testing, that the performance objectives have been achieved.

Depending on the circumstances of the particular training

experience, the qualification testing might be formal or

informal, but some testing surely is needed.

The preceding list is an "off-the-top-of-the-head" compilation of

factors that probably would occur to any evaluator interested in measuring the

adequacy of a Design subsystem's learner testing specifications. What follows

is an attempt to impose analytic rigor upon the measurement process.

3. Design Subsystem Performance Taxa Relevant to

Kfeasurement of the Adequacy of Gunner-learner

Testing Specifications

Two hundred and eighty-one taxa of Design Subsystem performance requirements are listed in Figure 7, above. These are distributed among the

three aspects of performance (potentialities, processes, and products) and

across the three levels of system description (objectives, functional purposes, and characteristics). Some of the 281 taxa are relevant to evaluation of a

Design Subsystem's performance in specifying learner testing. Others are not.

At the present stage of development of the APM, well defined procedures for

determining which taxa do or do not bear on a given measurement application

are yet to be established. Indeed, one purpose of this trial application is to

add to the base of experience from which such procedures may be derived.

Currently, assessment of taxa relevance is a "judgment call" by an analyst or

team of analysts working together. The thought process can be described as

follows: "Here we have some aspect of this system's performance, and over

here we have a performance taxon for the system. That taxon represents a particular requirement that the system is supposed to satisfy. Suppose the

system does a bad job in trying to satisfy that requirement. Might that have

any bearing on the particular aspect that we are studying?" If the analyst

concludes that the answer is "yes," the taxon in question would be included , among the subset from which measurement issues and implications, and

ultimately measures, will be derived. If the answer is "no," the particular taxon will supply no input to the measurement application at hand.

Identification of relevant taxa, then, is a step-by-step march through an

entire system taxonomy, during which each taxon is picked up in turn,

examined from various angles, and finally accepted or rejected in accordance with whether or not it appears to the analyst to have something of interest to

say. This process can be streamlined somewhat, simply by examining only the

characteristics-level taxa. If a taxon on the characteristics level is found to

be relevant to the particular measurement application, then the functional

purposes-level taxon from which it derives also must be relevant. A

characteristics-level taxon, after all, is simply a detailed performance

requirement that the system must satisfy if it is to achieve the related

-64-

! t *: ..:~ ;7~&1*2.
functional purpose. Thus, the functional purpose must be subsumed under

r.-.the aspect of performance being studied. Similarly, if a functional

purposes-level taxon is relevant to the measurement application, so must be

the objectives -level taxon from which it derives. The functional purpose,

after all, is only a required application of the basic potentiality, process, or

product defined on the objectives level. For example, if the analyst in using

Figure 7 were to conclude that taxon number 2.1.3.1 (a characteristics-level

process requirement) is of significance to the measurement application, he or i she would automatically know that taxa 2.1.3 and 2.1 (its functional

purposes-level and objectives-level antecedents) also are relevant. These

inter- relationships are described in more detail in the Year 1 Final Report

(Bloom, et. al. , 1981).

Appendix A of this report documents the examination of every

characteristics-level taxon of potentiality of the general Training Design

subsystem for possible relevance to assessment of the subsystem's performance jin specifying learner testing. Rationale for concluding whether an individual

taN-on is or is not relevant to the measurement issue is presented there.

Similar (but undocumented) examinations were made of the process and product

taxa. The final conclusion of these examinations was that 119 of the 281 taxa

are relevant to evaluation of the Design Subsystem's specifications of learner

testing. Those relevant taxa form 14 taxonomic subsets, of which 5 are

potentialities, 3 are processes, and 6 are products. Figure 13 arrays these

subsets, displaying their member taxa as issues relevant to evaluation of

learner testing specifications.

04. Addressing the Issues: Attributes and Measures of

the Adequacy of Gunner-Learner Testing Specificatons

Alongside each of the evaluative issues arrayed in Figure.13, the

reader will find a set of attributes deemed to be relevant to the particular

issue. Referring to the overall APM (Figure 1), the issues listed in Figure 13

derive from Blocks 1-3, whereas their correlated attributes for potential

measurement are the result of carrying out the Block 6 operations. Each

attribute is a concrete element of the Design subsystem's work. It is

something that can be seen, examined, submitted to analysis, taken apart, etc.

In short, each attribute is something that is measurable. Further, each

attribute is something that, when measured, can supply a portion of the

answer to the question posed in the evaluative issue.

;. I Some examples may help to clarify the relationship between issues

and attributes. Consider first the final issue in Subset No. 1 (Figure 13),

namely, "Can the BIFV Gunnery Training Design subsystem insure that the

training goals will be stated in terms that imply the gunner behaviors

required, so that the testing specifications ultimately may address those

behaviors?" This issue arises from a characteristics-level potentiality, i.e.,

the potential for stating goals in behavioral-implication terms. That isa

particularly important capability for test specification. Each gunnery test will

require the gunner-learners to manifest some behavior, for instance a behavior

K~ .J-65-
r q

PART 1: ISSUES AND ATTRIBUTES OF POTENTIALITIES

Subset No. 1: *Identifying Goals and Priorities"

Issues Attributes

1.1 Does the system have the ability Scope of Information available concern- U to nsure that all gunnery train- ing BIFV gunnery operations

Ing goals will be Identified so 0 Staff experience and qualifications in

that they may be addressed in the identification of training goals

gunner testing specifications? 0 The planned approach to Identifying

BIFV gunnery training goals 1 o The resources allocated to identifying

BIFV gunnery training goals

1 1.1.1 Can the system insure that the 0 The elements or aspects of each gun- /' total scope of learning required nery training goal intended to be

for each gunnery training goal defined

will be defined so that the full 0 The plans for defining the total scope

scope of each goal may be covered of BIFV gunnery learning

in the testing specifications?

1.1.1.1 Can the system insure that all * The types or classes of achievement

achievements constituting the intended to be identified

scope of BIFV gunnery learning 0 The plans for identifying all types will be identified so that the of BIFV gunnery achievements

testing specifications may

address each achievement?

. 1 .2 Can the system insure that the 0 Elements of ultimate outcomes that

ultimate intended outcomes of are intended to be stated

BIFV gunnery learning can be 0 Intended formats for stating the ultistated so that appropriate mate intended outcomes of BIFV

tests of those outcomes may be gunnery learning

specified? 0 The plans for stating the ultimate

intended outcomes

1.1.2.1 Can the system Insure that the 0 Elements or factors of the necessary

- F necessary levels of achievement levels Intended to be determined .',.j for acceptable BIFV gunnery can (i.e.. factors of acceptability)

be determined, so that the spe- 0 The plans for determining the neces- cified tests may assess whether sary levels of BIFV gunnery achievewhether those levels are reached? went

1.1.2.2 Can the system insure that candi- o Elements or factors of the current

date gunners' current levels of levels of achievement intended to be

achievement can be tested to help tested

determine learning needs? 0 The plans for testing candidates' current levels of BIFV gunnery achievement

,m 1.1.4 Can the system insure that the 0 Elements of the basis Intended to be

goals will be analysed to produce established

a basis for specifying BIFV gun- 0 Plans for establishing the basis for

nery performance objectives? specifying BIFV gunnery performance

objectives

1.1.4.1 Does the system propose to use an Goal tax& intended to be employed

appropriate taxonomy for clasi- Plans for employing the taxa to

fying goals in a fashion condu- classify goals for BIFV gunnery cive to establishing correct per- training

formance objectives and correspondingly correct testing

*. specifications?

1.1.4.2 Can the system Insure that goals " Behavioral terminology Intended to be

will be stated in term that used In stating BIFV gunnery training

Imply the gunner behaviors re- goals

quired. so that the testing ape- * Intended formats for stating BIFV

difications ultimately may gunnery training goals

address those behaviors?

4- Figure 13. Measurement Issues and Measurable Attributes Relevant to Evaluation

of BIFV Gunnery Testing Specifications

-66-

+ . +''T wv v,: ''''+"''+ 7+''''',?L' ""'''%"'++
Subset No. 2: 'Establishing Performance Objectives*

Issues Attributes

1.2 Does the system have the ability o Scope of information available conto insure that all BIFV gunnery cerning BIFV gunners' tasks

performance objectives will be e Staff experience and qualifications

identified so that they may be in establishing performance objectives

addressed in gunner testing spe- 0 The planned approach to establishing

cifications? BIFV gunnery performance objectives

0 The resources allocated to establishing BIFV gunnery performance objectives

1.2.1 Can the system insure that all 0 The elements or factors of gunner

things BIFV gunners will be able abilities intended to be stated

to do after training will be 0 Intended formats for stating BIFV

stated, so that tests of all gunner abilities

required abilities may be 0 The plans for stating intended

specified? abilities

1.2.1.1 Can the system insure that the 0 The types of performance actions

stated abilities will include intended to be defined

definitions of the required * The plans for defining actions

gunner actions, so that the test required of BIFV gunners

specifications may address those

actions?

1.2.1.2 Can the system insure that the o The elements or factors of perforstated abilities will include mance condition intended to be

definitions of the appropriate defined

conditions under which BIFV gun- 0 The plans for defining BIFV gunnery

ners are to perform the actions, performance conditions

so that the test specifications.

may reflect those conditions?

1.2.1.3 Can the system insure that the Elements or factors of performance

stated abilities will include standards intended to be defined

definitions of the appropriate 0 The plans for defining standards for

standards of acceptable perfor- BIFV gunnery performance actions

mance of the BIFV gunner

actions, so that the test specifications may incorporate those

standards?

1.2.2 Can the system insure that the 0 lemento of the basis for assessment

identified BIFV gunnery perfor- intended to be provided

mance objectives will be applied 0 Plans for providing a basis for objecto provide a basis for objec- tively assessing BIFV gunners'

tively assessing gunners' per- performances

formances?

1.2.2.1 Can the system insure that 0 Types of BIFV gunnery pretest Items

appropriate pretest items intended to be developed

addressing the performance ob- e Plans for developing BIFV gunnery

jectives will be developed, so pretest items

that they may be included in the

testing specifications?

1.2.2.2 Can the system insure that e Types of BIFV gunnery pottest Items

appropriate poettest items intended to be developed

addressing the performance ob- 0 Plans for developing BIFV gunnery

jectives will be developed, so poettest items

that they may be included in

the testing specifications?

Figure 13. (Continued)

-67-

m1V
Subset No. 3: 'Analyzing Performance Objectives"

Issues Attributes

1.3 Does the system have the ability 0 Intended scope of analysis of BIFV

to Insure that all BIFV gunnery gunnery performance objectives . performance objectives will be 0 Staff experience and qualifications

analyzed to determine the ape- in analyzing performance objectives

cific elements of gunner perfor- 0 The planned approach to analyzing

mance that need to be tested? BIFV gunnery performance objectives

0 The resources allocated to analyzing

'J BIFV gunnery performance objectives

1.3.1 Can the system insure that the 0 Types or classes of BIFV gunnery

analysis of BIFV gunnery perfor- prerequisites intended to be

mance objectives will disclose identified

the gunnery training prerequl- 0 Plans for identifying BIFV gunnery

sites, so that appropriate tests training prerequisites

of candidate gunners may be

specified?

1.3.1.1 Can the system insure that the * Types of information recall require-

-:2 analysis will uncover what can- ments intended to be revealed

didate gunners must recall in 0Plans for revealing the BIFV gunorder to be qualified for train- ners' recall requirements

ing. so that appropriate tests

of that recall may be specified?

1.3.1.2 Can the system insure that the 0 Types of essential prerequisite

analysis will uncover abilities abilities intended to be revealed

that the candidate gunners abso- 0 Plans for revealing the BIFV gunlutely must possess in order to ners' essential prerequisite J achieve the performance objec- abilities

tives, so that appropriate tests

of those abilities may be

specified?

1.3.1.3 Can the system insure that the 0 Types of supportive prerecjuisite

analysis will uncover abilities abilities intended to be revealed

which, if possessed by candidate 0 Plans for revealing the BIFV gungunners, will facilitate ners' supportive prerequisite

achievement of the performance abilities

objectives, so that appropriate

tests of those abilities may be

;;, specified?

1.3.2 Can the system insure that the Elements of the basis for candidate

analysis of performance objec- assessment intended to be provided

ives will be applied to provide 0 Plans for providing a basis for a basis for objectively assess- objectively assessing a candidate

ing a candidate gunner's suit- BIFV gunner's suitability for

ability for training? training

1.3.2.1 Can the system insure that 0 Scope of BIFV gunner abilities

. appropriate entry test items intended for entry testing

addressing candidate BIFV gun- 0 Plans for specifying entry test Items

ners' suitability will be for candidate BIFV gunners

included in the testing specifications?

1.3.4 Can the system insure that per- * Domains of Wearning intended for use

formance objectives will be in classifying BIFV gunnery perforclassified into appropriate mance objectives

domains of learning, so that o Plans for classifying SIFV gunnery

tests specified for each objec- performance objectives into learning

tive may exercise the appro- domans 'I priate domain?

i dFigure 13. (Continued)

-68-
Subset No. 3 (Cont'd)

Issues Attributes

1.3.4.1 Can the system insure that all * Criteria intended to be employed for

information-type objectives classifying BIFV gunnery performance

will be correctly classified, objectives as information-type

so that appropriate informa- objectives

tion-type tests may be specified for them?

1.3.4.2 Can the system insure that all Criteria intended to be employed for

mental skills-type objectives classifying BIFV gunnery performance

will be correctly classified, objectives as mental skills-type so that appropriate mental objectives

skills-type tests may be specified for them?

1.3.4.3 Can the system insure that all 0 Criteria intended to be employed for

physical skills-type objectives classifying BIFV gunnery performance

will be correctly classified, so objectives as physical skills-type

that appropriate physical objectives

skills-type tests may be specified for them?

. 1 1.3.4.4 Can the system insure that all Criteria intended to be employed for

attitude-type objectives will be classifying BIFV gunnery performance

correctly classified, so that objectives as attitude-type objecappropriate attitude-type tests tives

may be specified for them?

1.3.5 Can the system insure that the 0 Types or classes of gunners' compo- analysis of BIFV gunnery perfor- nent steps or processes intended to

.- mance objectives will identify be identified

the component steps or processes 0 Plans for identifying gunners' comrequired of BIFV gunners, so ponent steps or processes within the

that tests of the gunners' abil- BIFV gunnery performance objectives

Ities to perform those component

steps or processes may be

specified?

1.3.5.1 Can the system insure that all 0 Types of overt steps intended to be

overt steps required of BIFV identified

gunners will be identified, so 0 Plans for identifying the overt

". that they may be addressed in steps involved in BIFV gunnery

the testing specifications? performance objectives

1.3.5.2 Can the system insure that all 0 Types of covert steps intended to be

covert steps required of BIFV identified

gunners will be identified, so 0 Plans for identifying the covert

that they may be addressed in steps involved in BFV gunnery

the testing specifications? performance objectives

1.3.5.3 Can the system insure that all 0 Types of unconscious steps intended

unconscious steps required of to be identified

BIFV gunners will be identified, 0 Plans for identifying the unconscious

so that they may be addressed in steps involved In BIFV gunnqry

the testing specifications? performance objectives

Figure 13. (Continued)

-69-
Subset No. 4; ODefining Training Procedures'

Issues Attributes

1.5 Does the system have the ability * Categories or types of BIFV gunnery

to insure that all BIFV gunnery training procedures intended to be

training procedures will be cor- defined

rectly defined, so that gunnery 0 Staff experience and qualifications

testing procedures will play in the definition of training

their proper role in BIFV procedures

gunnery training? 0 The planned approach to defining

BIFV gunnery training procedures

0 The resources allocated to defining

BIFV gunnery training procedures

1.5.3 Can the system insure that BIFV 0 Classes/methods of BIFV gunnery

gunner application (or practice) application intended to be specified El procedures will be correctly 0 Intended involvement of testing in

specified, so that the role of BIFV gunnery application

testing-in-practice will be 0 Plans for specifying BIFV gunnery

properly defined and reflected application procedures

in the testing specifications?

1.5.3.4 Can the system insure that the 0 Intended role of learning feedback

application (practice) pro- in the BIFV gunnery application

cedures will provide learning procedures

feedback to the BIFV gunner- 0 Intended contribution of testing to

qq learners? BIFV gunnery learning feedback

0 Plans for providing for BIFV

gunnery learning feedback

1.5.3.5 Can the system insure that the 0 Criteria intended to be employed in I application (practice) pro- assessing sufficiency of practice

cedures will enable the BIFV 0 Intended contribution of testing gunner-learners to assess to the employment of those criteria

sufficiency of practice? 0 Plans for enabling BIFV gunnerlearners to assess sufficiency of

practice

1.5.4 Can the system insure that BIFV 0 Categories/methods of BIFV gunnery

gunner evaluation (or testing) evaluation intended to be specified

procedures will be correctly 0 Plans for specifying BIFV gunnery

defined, so that those defini- evaluation procedures

tions may be reflected in the

testing specifications?

1.5.4.1 Can the system insure that the 0 Plans for tailoring test specificaBIFV gunner tests that are spe- tions to the domain of the objective

cified will address the appro- to be tested

priate domains of BIFV gunnery

learning?

1.5.4.2 Can the system insure that the 0 Plans for tailoring test specifica- BIFV gunner tests that are spe- tions to the action called for in

cified will elicit the appro- the objective to be tested

priate actions from the BIFV

gunner-learners?

1.5.4.3 Can the system insure that the Plans for tailoring test specificaBIFV gunner tests that are ape- tions to the conditions and stand-

% j cified will provide a reliable ards appropriate to the objective to

assessment of the BIFV gunner- be tested

learners' performances? Plans for devising test specifica- tions that wil support statistical

reliability roquirements

Figure 13. (Continued)

-" -70-

I% ,-]!
Subset No. 5: "Evaluating the Curriculum"

Issues Attributes

1.6 Does the system have the ability 0 Intended scope of BIFV gunnery

to insure that the BIFV gunnery training curriculum evaluation

training curriculum will be 0 Staff experience and qualifications

properly evaluated, and in par- in evaluation of trainir g curricula

ticular that all BIFV gunner 0 The planned approach to evaluating

testing appropriate to curricu- the BIFV gunnery training curriculum

lum evaluation will be specified 0 The resources allocated to evaluatand conducted? ing the BIFV gunnery training

curriculum

1.6.1 Can the system insure that all 0 Intended elements/methods of curappropriate steps will be taken riculum basis validation/revision

to validate/revise the bases for to be employed

the developing BIFV gunnery 0 Intended contribution of BIFV gunner

training curriculum, including testing to those elements/methods

in particular the specification 0 Plans for validating/revising the

and conduct of BIFV gunner test- bases for the developing BIFV

validation/revision?

1.6.1.1 Can the system insure that all 0 Assumptions concerning prerequisites

appropriate entry-level testing and candidates' qualifications

of representative candidate BIFV intended to be verified

04 gunners will be specified and 0 Intended contribution of BIFV entryconducted to verify assumptions level gunner testing to that

concerning prerequisites and verification

candidates' qualifications? 0 Plans for specifying and conducting

BIFV entry-level gunner testing to

verify the assumptions

1 .6.1.2 Can the system insure that all 0 Conclusions concerning performance

appropriate post-training test- objectives and test specifications

ing of representative current intended to be verified

BIFV gunners will be specified 0 Intended contribution of BIFV postand conducted to verify conclu- training testing to that verification

sions concerning the performance 0 Plans for specifying and conducting

objectives and the tests speci- 1IFV post-training testing to verify

fied for those objectives? the conclusions

1.6.2 Can the system insure that all 0 Intended elements/methods of curricuappropriate steps will be taken lum validation/revision to be emto validate/revise the develop- ployed

ing BIFV gunnery training cur- 0 Intended contribution of BIFV gunner

riculum itself, including in testing to those elements/methods

particular the specification and 0 Plans for validating/revising the

conduct of BIFV gunner testing developing BIFV gunnery training

. appropriate to that validation/ curriculum

, !revision?

1.6.2.1 Can the system insure that all 0 Aspects/segments of the curriculum

appropriate testing of 'pilot intended for 'pilot training'

training" BIFV gunner-learners 0 Intended role of BIFV gunner testing

will be specified and conducted, in 'pilot training' so that necessary error data on 0 Plans for specifying and conducting

the specified tests may be BIFV gunner testing in conjunction

oLtained? with 'pilot training'

1.6.2.2 Can the system insure that 0 Aspects/segments of the curriculum

'pilot training' gunner-learners' for which job performance measures performance on the specified are intended to be developed

tests will be correlated with 0 Plans for correlating job perforsubsequent job performance mance measures with 'pilot training'

measures. so that the predictive gunner-learners' test scores

validity of the specified tests

way be assessed?

Piue 13. lContinued)

-71 -

.. . . . . . .

- ..' '.": . : .: . : ... ',,. . . : . . . . . ... . .- ,, ..,. '.., 
j- PART 2: ISSUES AND ATTRIBUTES OF PROCESSES

Subset No. 6. "Analyze Tasks Selected for Training'

Issues Attributes

2.3 Does the system conduct a proper 0 The task analytic methods applied to

analysis of BIFV gunner tasks, the BIFV gunner tasks so that appropriate tests of 0 The scope of the task analyses conthose tasks may be specified? ducted (i.e., factors determined in

the analyses) o The completeness of the BIFV gunner task analyses

o The accuracy of the BIFV gunner task

analyses

2.3.2 Does the system accurately iden- 0 Methods employed to identify the

tify the domains of learning domains of learning involved in the

involved in the BIFV gunner BIFV gunner tasks

tasks, so that tests may be spe- 0 The domains of learning identified

.. i cifled and developed in the for each BIFV gunner task • proper domins?

2.3.2.1 Does the system accurately iden- o Methods employed to identify the

tify the knowledge that BIFV the knowledge requirements associSgunners must acquire to perform ated with BIFV gunner tasks

their tasks, so that appropriate 0 The BIFV gunner task knowledge

tests of knowledge may be speci- requirements identified

fled and developed?

2.3.2.2 Does the system accurately iden- 0 Methods employed to identify the

tify the mental skills that BIFV mental skills requirements associgunners must acquire to perform ated with BIFV gunner tasks

their tasks, so that appropriate 0 The BIFV gunner task mental skills

tests of mental skills may be requirements identified

- specified and developed?

2.3.2.3 Dues the system accurately iden- o Methods employed to identify the

tify the physical skills that physical skills requirements associBIFV gunners must acquire to ated with BIFV gunner tasks

perform their tasks, so that 0 The BIFV gunner task physical skills

appropriate tests of physical requirements identified

skills may be specified and

developed?

• 2.3.2.4 Does the system accurately iden- 0 Methods employed to identify the

" tify the attitudes that BIFV attitudinal requirements associated

gunners must acquire to perform with BIFV gunner tasks

their tasks, so that appropriate 0 The BIFV gunner task attitudinal

tests of attitude may be speci- requirements identified

fied and developed?

2.3.3 Does the system accurately Iden- 0 Methods employed to identify the

tify the conditions for learning conditions for learning the BIFV

associated with the BIFV gunner gunner tasks

tasks. so that appropriate tests 0 Types of conditions for learniqg

'. may be specified and developed identified for each BIFV gunner task

for assessing candidate gunners'

qualifications?

2.3.3.1 Does the system accurately iden- 0 Methods employed to identify the pretify BIFV gunners' prerequisite requisite knowledge required for

knowledge, so that appropriate learning the BIFV gunner tasks

tests of the knowledge prerequi- 0 Knowledge items identified as presites may be specified and requisite for learning the 3rFV

developed? gunner tasks

* gFigure 13 (Continued)

%

-72-

i. . . 1. 4 - . .... , .. .. . .. . .-.,- - .- . . . . ,. . .:, ' ..., " . ' ,' " -: ' . '" : " " ' -:':
d

Subset No. 6 (Cont'd)

Issues Attributes

2.3.3.2 Does the system accurately iden- 0 Methods employed to identify the pretify BIFV gunners' prerequisite requisite mental skills required for

mental skill. so that appropri- learning the BIFV gunner tasks

ate tests of the mental skills Specific mental skills identified as prerequisites may be specified prerequisite for learning the BIFV

i and developed? gunner tasks

2.3.3.3 Does the system accurately Iden- 0 Methods employed to Identify the pretify B1FV gunners' prerequisite requisite physical skills required

physical skills. so that appro- for learning the BIFV gunner tasks

priate tests of the physical o Specific physical skills identified

skills prerequisites may be spe- as prerequisite for learning the

cified and developed? BIFV gunner tasks

2.3.3.4 Does the system accurately iden- 0 Methods employed to identify the pretify BIFV gunners' prerequisite requisite attitudes required for

attitudes, so that appropriate learning the BIFV gunner tasks

tests of the attitudinal pre- 0 Specific attitudes identified as prerequisites may be specified and requisite for learning the BIFV

developed? gunner tasks 1~

Figure 13 (Continued) L

I

-73-

!4

-%/ o:.y
Subset No. 7: 'Determtine Instructional Requirements'

,Issues Attributes

2.4 Given the results of the analy- 0 The methods employed to determine

sis of the tasks selected for instructional requirements from the

training, does the system propT results of the analysis of BIFV

erly determine the BIFV gunnery gunner tasks

instructional requirements, 0 The types of instructional requireincluding in particular the meants determined

requirements for testing gunner- 0 The completeness of the methods' learners? applications 0 The accuracy of the methods' applications

2.4.1 Does the system accurately iden- 0 The methods employed to identify

tify all required BIFV gunnery required BIFV gunnery learning

learning activities/events, so activities/events

that the proper role of testing 0 The BIFV gunnery learning activiin those activities/events may ties/events identified as required

be specified?

2.4.1.4 Does the system accurately iden- 0 Methods employed to identify compotify the gunner testing compo- nents of gunner testing included

nents of the required BIFV within required BIFV gunnery learngunnery learning activities/ ing activities/events

events? o The gunner testing identified as

Scomponents of the required BIFV gunnery learning activities/events

2.4.4 Does the system accurately iden- 0 The methods employed to Identify

tify all needs for testing BIFV requirements for BIFV gunnery testgunner-learners in conjunction Ing in conjunction with BIFV

with BIFV gunnery training? gunnery training

: (This is in addition to needs 0 The types of gunnery testing needs

for testing as component parts identified

of BIFV gunnery learning activities/events.)

2.4.4.1 Does the system accurately iden- o Methods employed to identify

tify all needs for testing acceptance-testing requirements

candidates for accer-once as 0 Identified needs for acceptanceBIFV gunner-learners? testing of BIFV gunner-learner ,- candidates

-1 2.4.4.2 Does the system accurately iden- 0 Methods employed to identify testing

" tify all needs for testing needed to assess remedial training

accepted BIFV gunner-learners to requirements

assess their requirements for * Identified needs for testing

remedial training? accepted BIFV gunner-learners to

assess their remedial training

requirements

2.4.4.3 Does the system accurately iden- 0 Methods employed to identify testing

tify all needs for testing needed to assess possible training

accepted BIFV gunner-learners to tailoring requirements

assess possible requirements for 0 Identified needs for testing I tailoring the training to their accepted BIFV gunner-learners to

abilities and experience? "sess possible training tailoring

required for them

%lt Figure 13. (Continued)

-74-

. .. .. .. . -. -- - ;:-. . -- .. , . . , . ,

, ,.- , , -. .:- . ._. .-. -.-. -.- , ... . . .,. ..... ? ..- .. :., .. , .; . :. ,.''.' .,,:,\ / :L. ,, ,',C.; '.,,\V. -.
I.. Subset No. 7: (Cort'd)

-Issues Attributes

2.4.4.4 Does the system accurately iden- 0 Methods employed to identify testing

tify all needs for testing BIFV needed to assess learner advancement

gunner-learners during training requirements

to assess requirements for 0 Identified needs for testing BIFV

advancing or delaying their gunner-learners to assess their

progress through the training? training advancement status

2.4.4.5 Does the system accurately iden- • Methods employed to Identify gunner

tify all needs for testing BIFV certification testing requirements

gunner-learners at completion of 0 Identified needs for testing BIFV

scheduled training to assess gunner-learners to determine their

their qualifications for cer- certification status

tificatton as BIFV gunners?

FIgure 13. (Continued)

: I

~-75

%
1 Subset No. 8: 'Conduct Training Trials"

Issues Attributes

2.7 Does the system appropriately o The aspects/segments of BIFV gunnery

conduct trials of BIFV gunnery training submitted to trial

training, including in particu- 0 The kinds of trials conducted

l ar trials of gunnery testing? 0 The role of gunner-learner testing

in the trials

The appropriateness of the trials

O The completeness of the trials

O The reliability of the trials

2.7.2 Does the system adequately and 0 Methods employed to verify the suit-

" accurately verify the sultabil- ability of the BIFV gunnery instrucity of the assembled instruction tion for delivery

for delivery. including in par- 0 The role of gunner-learner testing

ticular the testing segments of in those methods

instruction? 0 Conclusions reached concerning the

suitability for delivery

2.7.2.1 Does the system conduct appro- 0 Kinds of field tests conducted of

priate field tests of the assem- the assembled BIFV gunnery instrucbled BIFV gunnery instruction, tion

including field tests of the 0 Information gleaned from field tests

testing specifications? 0 The role of gunner-learner testing

in the field tests

Figure 13. (Continued)

I-

~.1

1,

' % " "-76-
Subset No. 9: 'Documented Job Analyses"

Issues Attributes

3.1 Does the system produce docu- 0 The BIFV gunner job analyses documented analyses of the BIFV gun- mented

ner's jobs that completely 0 The elements of BIFV gunner perfcrdefine the performance elements mance identified in the documenessential for adequate gunnery, tation

so that tests of those elements 0 The completeness of the documents

may be specified? 0 The accuracy of the documents

3.1.1 Does the system produce a com- 0 BIFV gunner performance elements

plete specification of the per- for which detailed specifications

- formance elements of the BIFV are produced

gunner's jobs in sufficient 0 Types of details specified for

detail to permit those elements each performance element

to be tested?

3.1.1. 1 Does the system produce accurate 0 BIFV gunner mental processes specispecifications of all BIFV gun- fled to the level of binary deciner mental processes to the sions

level of binary decisions, so 0 The binary decisions specified for

that tests of those decisions each BIFV gunner mental process

* may be constructed?

3.1.1.Z Does the system produce accurate 0 BIFV gunner physical processes spespecifications of all BIFV gun- cifled to the level of discrete

ner physical processes to the actions

level of discrete actions, so o The discrete actions specified for

that tests of those actions may each BIFV gunner physical process

be specified?

3.1.1.3 Does the system produce accurate 0 BIFV gunner processes for which

specifications of the stimuli stimuli are specified

for all BIFV gunner processes, Q The stimulus specified for each

so that those stimuli may be BIFV gunner process

- - incorporated into the tests of

those processes?

.2- 3.1.1.4 Does the system produce accurate 0 BIFV gunner processes far which

specifications of the conditions conditions are specified

under which BIFV gunners perform 0 The conditions specified for each

their processes, so that those BIFV gunner process

conditions ma-y be incorporated

into the tests of those 4 processes?

- 3.1.3 Does the system produce a corn- 0 BIFV gunner performance elements for

plete specification of the fac- which adequacy factors are specified

. tors that the BIFV gunner- 0 Types of adequacy factors specified

learners' performance elements for each performance element

must satisfy for adequate

gunnery?

3.1.3.1 Does the system produce correct 0 BIFV gunner processes for which

specifications of the accuracy accuracy requirements are specified

requirements associated with 0 The accuracy requirements specified

BIFV gunners' performance, so for each BIFV gunner process

that tests can be designed to

assess whether sufficient accuracy is achieved?

3.1.3.2 Does the system produce correct 0 BIFV gunner processes for which

specifications of the speed speed requirements are specified

requirements associated with * The speed requirements specified

BIFV gunners' performance, so for each BIFV gunner process

that tests can be designed to

assess whether sufficient speed

is achieved?

Figure 13. (Continued)

-77-

:_I , .. .... .... . ...

:_---:,: : ., .. , ., .. . ., ., .. . :: .. , -.. , ... , ., ., . , ,, .. .. .
Subset No. 9: (Cont'd)

Issues Attributes

3.1.3.3 Does the system produce correct * BIFV gunner processes for which

specifications of the volume volume requirements are specified

requirements associated with 0 The volume requirements specified

BIFV gunners' performance, so for each BIFV gunner process

that tests can be designed to

."• assess whether sufficient

volume is achieved?

3.].3.4 Does the system produce correct 0 BIFV gunner processes for which

specifications of the duration duration requirements are specified

requirements associated with 0 The duration requirements specified

BIFV gunners' performance, so for each BIFV gunner process

that tests can be designed to

assess whether sufficient duration is achieved?

;I Figure 13. (Continued)

AW

1

w.. ,

,._ -78-

• i

:::::::::: . . .. .. .. .. .. ::::::::::: .. .. .. .. ..::':-::--::.: .. .. .. .. .. ... . . . . . . .. *.==================== . . . .;:i::. . ,-:. . . :-*.. . . . . . . . . . . . . ::::::::::::::::::.::
L-6L

in~ozuU;.j; Surna (peuwao 

suotp ;0.q4ax spawx~x ARq *qi 

0 psjueq 0) psodpnra od aql wqAInbG Aq ,uzsuUsJwatsUunl suopw 

suollvoypoade jo a41 -uoo1 UOnbepe j Oq AqsBUO[P aunuj 

LtV-Z- saoQ a4; m;As a snpo&d jo ~ * euils;.z psposd ipouu;e sjo 

Omun CV4 an t uzx Iq aq 

;uaOulnba A.1180 Iq21uvruunS Air 9-9*UJWOI-40uunSsq 

atIs3uwoo sq paow0 toot pq; aAIB *.zss qjjoujnsj-.zsul 

a ped asuo;;. tjow~ sq;oz paoo solnzx pto .q uuuw pagqd x; 

*V soaw waol pond aqj sgpad. uuu~uuuoo*lpezza £05 041 

s*oju3gpaads jo ; -w3q 10u~l al H9 suOPO taunB A -.

pTv saoa a41 wa;vlo npoizd .,d 0piw zo v3vxnj:) -n~~~~~~~axa~~~.LO 

uo;1;naad jo qs suoliow A Uq AAlt uusqjzsun el~a 

vuoi~vujjaps jo aq; oloqww u;sm AAE n solAqe Ad.zuI -&uuntS 

£ j1 saoCJ 341 waloLs aaLOC*npoid sjsq0uduiiw pa~pa w LOsisj aap 

suo;;joU:apd jo iq eq zu -. sJIt euivgepun 

tZcIZ saocj sq; W50ls a XSenpoizd ~ *43jo pago LtzeLsulU uolln0a5143U Aq 

al 95A1111-auns4vl oil511 

3U011t vu. eq -qpve!JpadU ua Aq~ AJuIII jo eq; souzojvsd 

eqaI--- psqt £0 1141 q3noi a * .qJ *esalqtdwo3 payod ao; e-ulaoJpe 

V sao(3 541 wualsis a3npoad ;z1Ll3 , a.j .zuAvauunAl euwwzo;qe -s;(o 

soli qAu s Zptla wa. 1oAainV fl *g *auuuo 

* POIWS. *3U15.Z0J.Lod AJI sa,%jpa~~*3.;~3qq 0 saun IaxoulaalqfIn 1 

slart.a.:3.':. slua>$KKKi' pp q ~ D -OlfOTjq 

~ augAI.t ope ajti CI.fq aessjsoI ~pi i33 
Subset No. 11: "Lesson Plans'

Issues Attributes

3.3 Does the system produce adequate 0 Lesson plans prepared for BlFV

lesson plans outlining the con- gunnery learning activities

tent and procedures for all 0 Segments of the plans outlining

learning activities, including gunner-learner testing

gunner-learner testing in con- 0 Completeness of the testing segments

junction with learning activi- 0 Appropriateness of the testing seg-

. ties? ments to the learning activities

3.3.1 Does the system produce complete 0 BIFV gunnery learning activities

specifications of the instruc- for which content is specified

tional content for the learning in the lesson plans

activities, including the con- 0 Content elements specified for

tent of gunner-learner testing each learning activity

in conjunction with learning 0 Content elements pertaining to

activities? testing

3.3.1.8 Does the system produce correct 0 Test problems/exercises itemized

specifications of test problems in the lesson plan content

and exercises in the instructioznal content for the BIFV

gunnery learning activities?

3.3.2 Does the system produce complete 0 BIFV gunnery learning activities

specifications of the instruc- fur which procedures are specified

tional procedures for the learn- in the lesson plans

ing activities, including the 0 Procddural elements specified

procedures for gunner-learner for each learning activity

testing in conjunction with 0 Procedural elements pertaining

learning activities? to testing

3.3.2.4 Does the system produce correct 0 BIFV gunner-learner testing

specifications of testing pro- procedures itemized in the

cedures in the instructional lesson plan procedures

procedures for the BrFV gunnery

learning activities?

Subset No. 12: 'Training Documents'

aIssues Attributes

3.4 Does the system produce all 0 BIFV gunnery training documents

necessary training documents, prepared

. including in particular docu- 0 Documents and segments of documents

ments needed for BIFV gunner- that are relevant to gunner-

" learner testing? learner testing - Completeness of the testing

documents I segments

0 Accuracy of the testing documents/

segments

3.4.2 Does the system produce docu- 0 Documented guidance for BIFV

ments providing all necessary gunnery training facilitators

guidance to BIFV gunnery train- 0 Guidance elements provided in the

ing facilitators, Including documentation

guidance for Implementing 0 Guidance elements pertaining to

gunner-learner testing? gunner-learner testing

3.4.2.2 Does the system produce complete e Testing applications/requirements

sets of test specificationso for which BIFV gunner-learner test

addressing all applications and specfications are documented

requirements for BIFV gunner- 0 Elements or aspects of testing

learner testing? included in the specifications

Figure 13. (Continued)

-80-

+J, % ".-- + , " . .'.% + .". .-+,', " % ",.-' + , ". " ,+.. . " % " .+ ,.%=".% ."." % - . ,- ." " " * . ° V" "." " 

++ .++ 4 ,,, ...+++ +.+,+";,,. . ."..'-."-'-.+ . -':' -+ N"- -. , :.'-,-++." + L,+ + ++'-+-N -.+. gn.- -'-' +' -.-- ..- .'7. '-L 
- J Subset No. 13: "Tests"

Issues Attributes

3.5 Does the system produce appro- 0 BIFV gunner-learner tests prepared

priate tests for administration * BIFV gunnery testing applications/

N to BIFV gunner-learners? requirements for which tests are

prepared o BIFV gunner behaviors for which the

tests are prepared

o Relevance of the tests

3.5.1 Does the system produce appro- 0 Candidate gunner-learner qualificapriate tests for assessing the tions for which tests are prepared

qualifications of candidate BIFV ' Tests prepared for assessment

gunner-learners? of those qualifications

3.5.1.1 Do the candidate qualification 0 Domains of learning addressed

tests provide adequate measures by the tests

of the candidates' prerequisite 0 Actions elicited by the tests

abilities? 0 Objects intended to result from the

actions elicited by the tests

3.5.1.2 Are appropriate procedures 0 Circumstanc-s specified for the

specified for applying the tests

qualification tests to BIFV 0 Tools and equipment specified for

gunner candidates? the tests

o Constraints specified for the tests

7 o Other aspects of applications procedures Specifled for the tests

-.. 3.5.1.3 Are appropriate standards for 0 Specific standards specified for

qualification as BIFV gunner the qualification tests

specified for the candidate

qualification tests?

3.5.2 Does the system produce appro- 0 BIFV gunnery abilities for which

prlate tests for assessing tailoring tests are prepared

requirements for tailoring the 0 Tests prepared for assessinent of

instruction to accepted BIFV needs for tailoring instruction to

gunner -learners? fit gunner-learners' abilities

3.5.2.1 Do the tailoring tests provide * Domains of learning addressed by

adequate pre-training measures the tests

of accepted BIFV gunner- 0 Actions elicited by the tests

learners' abilities? 0 Objects intended to result from the

actions elicited by the tests

3.5.2.2 Are appropriate procedures pe- Circumstances specified for the tests

cified for applying the tailor- 0 Tools and equipment specified for

ing tests to accepted BIFV the tests

- gunner-learners? 0 Constraints specified for the tests o Other aspects of applications procedures specified for the tests

3.5.2.3 Are appropriate standards for 0 Specific standards specified for

tailoring instruction specified the tailoring tests

for the BIFV gunnery tailoring

tests?

3.5.3 Does the system produce appro- 0 BIFV gunnery abilities for which

priate tests for measuring what achievement tests are prepared

gunner-learners have achieved 0 Tests prepared for assessment of

2 as a result of BIFV gunnery BIFV gunner-learners' achievements

training?

Figure 13 (Continued)

,--..
Subset No. 13: (Cont'd)

Issues Attributes

3.5.3.1 Do the achievement tests provide 0 Domains of learning addressed by

adequate post-training measures the tests

of BIFV gunner-learners' o Actions elicited by the tests

abilities? * Objects Intended to result from the

actions elicited by the tests

3.5.3.2 Are appropriate procedures spe- 0 Circumstances specified for the tests

cied for applying the achieve- 0 Tools and equipment specified for

ment tests to BIFV gunner- the tests

learners? 0 Constraints specified for the tests

0 Other aspects of applications procedures specified for the tests

3.5.3.3 Are appropriate standards for 0 Specific standards specified for

BIFV gunner-learner achievement the achievement tests

specified for the achievement

tests?

Figure 13. (Continued)

I

1

:.. : ,'.--- . --. . . - . .-. , .. .-.. .... ,-o, 8-2--,., - . , , ,', . , ,- . - - - . .- .:
Subset No. 14: @Traininp Trial Data'

Issues Attributes

, 3.6 Does the system produce all 0 Training trials from which data > necessary and appropriate train- are obtained ing trial data. including in 0 Types or categories of data obtained particular data on BIFV gunner- 0 The data obtained within each learner testing? category

Completeness of the data o Accuracy of the data

M 6.1 Does the system produce suffi- 0 The data produced to provide justifi- dent data to provide justifi- cation for specific features of BIFV cation for specific features of gunnery instructional design the 

design? BIFV gunnery instructional 0Gmner-.learner testing conducted to help supply those data

3.6.1.1 Does the system produce appro- 0 Representative candidates to whom priate entry test data derived BIFV gunnery entry testing is i from representative candidates administered for BIFV gunnery training? o Entry level tests administered to

the representatives i"' , i0 The entry test data obtained

I 3.6.1.2 Does the system produce appro- 0 Representative current BIFV gunners priate post-test data derived to whom post-testing is administered from representative current Post-tests administered to the BIFV gunners? representatives -,* The post-test data obtained

3.6.1.3 Does the system produce appro- 0 The tryout gunner learners to whom priate test item error analyses test items are administered derived from tryout gunner- o The test items administered in the learners? tryouts

* The test item data obtained o The those analyses of error conducted on data

3.6.2 Does the system produce suffi- 0 The data produced to demonstrate the cient 

validity data 

and 

to 

utility 

demonstrate the validity and utility of the BIFV of BIFV gunnery training

gunnery training? o Gunner-learner testing conducted to

help supply those data

* 3.6.2.1 Does the system produce appro- 0 The field test BIFV gunner-learners priate test item error analyses to whom test items are administered derived from field test BIFV 0 The test items administered in the gunner-learners? field tests

The test item data obtained The analyses of error conducted on those data

Figure 13. (Concluded)

.°; .J

-83-

.o4,
Srequired for detecting targets, or tracking, or shooting, etc. Unless the

system can accurately define exactly what behaviors gunners must be able to

manifest to accomplish effective gunnery operations, the system will not know

what behaviors should be tested. If the training goals are loosely or

m ambiguously stated so that the required gunner behaviors are not clear, it is

possible that the entire structure built upon the foundation of those goals will

be inappropriate. That is, the gunnery performance objectives, the learning activities, the training documents, the testing specifications, and all other T~ j facets of the assembled instruction might be focused on a set of behaviors that

* are not quite the ones that BIFV gunners really need to perform. The people paying for the training design effort certainly would want some assurance, at

the outset, that the system they are funding will be able to define the correct

gunner behaviors.

But, how does one determine whether an ability is present before

any processes are carried out and before any prout are delivered?

Obviously, one examines the people, plans and tools assembled for carrying out the processes and producing the products, and tries to determine whether

theyhave what it takes to do the job right. In the case of the particular ability to state training goals in behaviorally referenced terms, the evaluator

wudexamine the training design plans and perhaps speak with the training designers to ascertain:

- The behavioral terminology they intend to employ in their

statements of BIVgunnery triiggoals.

v~ ~- The formats they intend to follow in preparing their statements

of BIFV gunnery training goals.

These two attributes of Design subsystem potentiality will allow the evaluators

to form reasonable judgments about the training designers' knowledge of

behaviorally referenced goals, awareness of the importance of such goals for

all facets of training development (including the specification of gunner-learner testing), provisions they are making for producing such goals, etc. In short, these two attributes provide the evaluator with something concrete with which

>q*. to deal in his or her efforts to assess the system's ability to specify the

proper kinds of training goals. After examining those attributes, an evaluator

might report back something like the following to the keepers of the purse strings:

"Look, I examined the proposal your training designers submitted,

and I found nothing in it to indicate any recognition of the need to

produce behaviorally referenced training goals. I spoke to the

designers, and asked them to give me some examples of the kinds of

goal statements they intended to develop. The examples they came

up with were absolutely devoid of any behavioral terms. They simply weren't the kinds of statements that can lead to a proper analysis of learner bahavioral requirements. I asked them if they

couldn't suggest a format for their goal statements that would help

~..' ~-84-

'AA
make explicit the learners' intended behavioral achievements, and it

was clear that they didn't really understand what I was seeking. I *':-.~ Iconclude that, as things now stand, your system can't insure that

the goals will be stated in terms that imply the required gunner *behaviors. I suggest that you delay starting the development of

this tr-aining until you've either brought these designers up to

speed, or until you've replaced them with some people who

understand behaviorally referenced training. "*

As a second example, consider the fifth issue in Subset No. 6,

"Does the system accurately identify the physical skills that BIFV gunners

must acquire to perform their tasks, so that appropriate tests of physical

skills may be specified and developed?" This issue arises from a

characteristic s-level process, i.e., the analytic process used to identify the

physical skills learners must acquire in order to serve as BIFV gunners.

Clearly, tests of all required physical skills must be specified and administered

as part of the gunner certification process. If the system's analysis is

deficient, so that certain essential physical skills are not identified, or so that

some irrelevant physical skills are mistakenly identified as essential, the

testing specifications also will be deficient. As the design work progresses,

the sponsors certainly would wish to know whether a thorough and accurate

analysis of required physical skills is underway.

In order to assess that issue, the evaluator would look first at the

analytic methods the training designers are using to identify required physical

skills. Are those methods appropriate to the analytic needs at hand? Are all

of the data required for application of those methods being obtained and used

appropriately? Is there any fault to find with the way in which the training

designers are applying those methods? As the analyses begin to produce

results, the evaluator would examine the physical skills requirements

identified. Are all of those skills really pertinent to BIFV gunnery dperations?

Are there any essential skills that haen't been identified in this analysis? Are

there any skills that the designers have labelled "physical" that really are

"1mental?" Based on these and similar considerations, the evaluator can form a

well reasoned judgment about the accuracy of the designers' approach to

identifying physical skills requirements. That reasoned judgment will be based

on measures applied to the two concrete attributes, viz., the analytic methods

of skills identification and the identified skills themselves.

The evaluative issues thus lead to measurable attributes, and the ...- 'Iattributes in turn lead to measures of performance and effectiveness. The

question is: How does all this "leading" proceed? How does one discern the

*This arnpl Teport" is purely hypothetical, and is intended only to illustrate

2K. how examination of concrete attributes could provide answers to the question

* posed in a particular evaluative issue. In no wa'y should the reader infer

that the authors of this report expect that the ITFV Task Force will produce

poorly stated goals for gunnery training. The authors are much impressed

with the experience, professionalism, and dedication of the Task Force

-04 members they have met, and have no doubt of the high quality of their work.

-85-

- - * .- *
%~--r rr r r '

attributes in the issues, and how does one derive the measures from the ~ J attributes?

No precise or even preliminary guidelines, algorithms or heuristics

OW for extracting attributes from issues have thus far been developed. A major purpose of this trial application of system performance taxonomization to

evaluating BIFV Gunnery testing specifications is to generate an experiential base from which such guidelines can begin to be derived. But in very general

terms, the analyst's thought process in searching for the attributes of a

particular issue relevant to testing specification can be sketched this way:

"Here is an issue that bears on how well the BIFV Gunnery Design

Subsystem has (or can) define gunner testing specifications. I'm

supposed to suggest a way of answering the question posed in this

issue. First, I have to ask myself: which of the subsystem's

people, resources /equipment, and procedures are involved in this

issue? Next, what qualities or characteristics do those involved ~ Ipeople, resources/ equipment, and procedures have to possess, or

produce in their work, if this issue is to be resolved affirmatively? Finally, what concrete, observable factors associated with those

people, resources/ equipment, procedures, or their work can I

examine to determine whether those qualities or characteristics are

present?"

The concrete, observable factors uncovered by the analyst are the

attributes associated with the issue in question. The authors applied the

mental process outlined above to each of the 119 performance-related issues

deemed relevant to evaluation of BIFV gunner testing specifications. The

outcome is the set of attributes presented along with the issues in Figure 13.

It is the authors' considered opinion that the attributeT set

documented in Figure 13 is a face-valid and comprehensive basis for evaluating

the adequacy of the gunner-learner testing specifications. But that is only opinion, not proof. It is hoped that the reviewers of this report, including

representatives of the BIFV Task Force and other specialists in training development and evaluation, will assess and critique the attribute set so that a

much broader base of expertise and insight will be applied to the continuing devcl1opment of APM procedures.

If the reader will accept the attribute set given in Figure 13 as

being at least illustrative of the bases from which the evaluative issues can be

* addressed, attention can now turn to the derivation of measures from

*attributes. A measure, in most general terms, is a judgment or appraisal

about the thing-being-measured. Measures applied to system attributes

* derived from various taxa of performance thus are judgments or appraisals of

performance. Each such measure contributes some small bit of wisdom or

*insight about the total performance of the system-bein g-measured. Every

measure "looks" at its attribute in its own unique way, and 'weighs" the

attribute on its own unique scales. Collectively, the measures are Intended to

-86-
% determine whether the attribute is "good enough" to meet the demands of the J performance requirement from which it derives. Individually, each measure

* focuses on some particular aspect or dimension of "goodness."~

Consider this example: the BIFV Gunnery Training Design

subsystem must have the ability to insure that all BIFV gunnery performance

objectives will be identified so that they may be addressed in gunner testing

specifications. One attribute of the system's ability to do that is the scope of

information available to it concerning the BIFV gunners' tasks. Gunnery

performance objectives must be derived from the gunners' tasks. If the

system's knowledge of the gunne~rs' tasks is not "good enough," it will not

have the ability to identify gunnery performance objectives accurately.

How does one measure whether a set of knowledge is "good enough."

First, one specifies or states exactly what is known, i.e.,* the contents of the

set of knowledge. Next, one identifies missing elements in the set of * I knowledge, i.e., facts or other information that should be known, but aren't.

Finally, one identifies inaccuracies in the set of knowledge, i.e. , supposed

"facts" and other information that are not true. This is a roundabout way of

saying simply that a person's knowledge of a given subject might be "bad" in

any of three ways:

- the person might know nothing about the subject; or

might know some things but not others; or - might "know" some things wrong.

A combination of the last two is also possible.

In the specific context of the Design Subsystem's knowledge of the

BIFV Gunner Tasks, the evaluator would need to determine, first, wtxether the

subsystem knows anything about any of the tasks; second, whether some tasks

are unknown to the subsystem; and, third, whether some of the "known" tasks

are in fact not tasks required of BIFV gunners. Each of those determinations

provides a separate judgment or appraisal of the goodness of the scope of

information the Design Subsystem proposes to use as the basis for establishing

BIFV gunnery performance objectives. That is, each is a separate measure of

that one attribute of the system's potential for establishing objectives. It is to

the credit of the BIFV Task Force that they have assembled a very

comprehensive task data base which, in fact, helps to meet the Potentiality

aspects analysis of this exercise.

Consider another example in the context of this same performance

potentiality. Along with having good information about gunnery tasks, the

Design Subsystem must have people who are qualified to do the job of

establishing performance objectives. The staff's experience and qualifications

thus canstitute another attribute of the system's ability to produce well

founded objectives, If their qualifications aren't "good enough," the system's

* ability to establish objectives also won't be good enough.

* -87-
How does one measure the "goodness" of an individual's or team's

qualifications for doing a particular job? When the job is relatively simple and

of short duration, a candidate's qualifications sometimes can be measured by

requesting him or her to actually do the work in question, on a sample basis.

A typing test, for example, often is administered to candidates for a secretarial position. Then, actual job performance measures (e.g., words typed per -minute, percentage of errors, etc.) can be applied to the sample work and used as job qualification measures. However, when the work is

complex, cerebral and non-routine, this approach may not be practical. In

such cases, clinical judgments usually are employed to appraise the candidate's

qualifications. Subject-matter experts review resumes of the available

candidates' experience and training relevant to the job to be filled, interview

personal references named by the candidates, perhaps examine the products

delivered by the candidates during previous assignments similar to the job to ii be filled, and maybe interview the candidates themselves to obtain greater insight concerning the skills and knowledge they could bring to the job. Each

expert reviewer then produces an independent rating of each candidate, based

on the reviewer's perception of the candidate's strengths and weaknesses.

Such ratings often are placed on an interval or ordinal scale. Then, the

expert reviewers meet as a group, report and explain their individual ratings,

deliberate on and debate the issues raised concerning each candidate, and form

a consensus rating of the qualifications of each.

I his clinical approach can be applied to measure the Training Design staff's qualifications for establishing gunnery performance objectives. The

appropriate measures would include the consensus rating of each individual I proposed for assignment to the task of establishing objectives, and an overall

consensus rating of the total proposed staff as a team for handling that task.

Both types of consensus ratings should include or be augmented with detailed

explanations of all identified personnel deficiencies bearing on the ability to

establish objectives.

Referring next to Block 7 of the overall APM (Figure 1), it is

relatively easy to suggest measures for any given attribute. Unfortunately, it

is quite something else to devise a general procedure for doing so in all cases.

Within the scope of this current study, measures have been suggested for all

of the f ,sues and attributes of four of the taxonomic subsets of Figure 13.

These measures are arrayed in Figures 14 through 17. The authors believe * that each suggested measure has something of value to say about the "goodness" or "badness" of the attribute to which it relates. The authors also

believe that the measures suggested for each given attribute collectively say

everything that is pertinent to the "goodness" or "badness" of that attribute.

Obviously, both of these beliefs need to be subjected to the test of a careful,

critical review by the readers of this report. It is quite possible, and

probably very likely, that the measures hierarchies shown in Figures 14

through 17 will undergo significant revision after readers' comments are

received. Subsequent to such revision, the measures hierarchies will provide

a base from which \ke can begin to extract measures specification procedures.

-88-
Hierarchy Number 1: BIFV gunnery design subsystem potential for

establishing performance objectives

(Note: Issue numbers refer to the taxa in Figure 7 from which they are derived.)

Issues

1.2 Ability to insure that all BIFV gunnery

performance objectives will be identified so that they may be addressed in

gunner testing specifications

" Attributes Measures

Scope of information A. BIFV gunners' required tasks that are known (listing and description

available concerning of each such task)

BIFV gunners' tasks B. BIFV gunners' required tasks that are unknown to the Design Subsystem (listing of each)

C. Tasks that actually are not required of BIFV gunners, but which nistakenly are classified as required by the Design Subsystem (listing

and description)

Staff experience and D. Reviewers' cumulative ratings of each Design staff member's trainiTIg,

qualifications in experience and previous performance in establishing performance

establishing perfor- objectives for training (using a specified point allocation system

mance objectives for training, experience. etc.)

E. Reviewers' overall assessment of total Design staff's qualifications

(based on specified method of accumulating individual members'

ratings)

Planned approach to F. Essential steps missing from the plans (listing and description of

establishing BIFV each such step)

gurnery performance G. Inessential steps included in the plans (listing and description of

obiectives each)

H. Inconsistencies/deficiencies in the sequencing of steps in the plans

I. Reviewers' cumulative ratings of the soundness/workability of the

plans (using a specified point allocation system)

Resources allocated J. Percentage of required person-hours proposed for allocation to

to establishing BIFV establishing objectives

gunnery performa(_c- K. Percentage of required materials/goods proposed for allocation to

objectives establishing objectives

L. Percentage of required support services proposed for allocation to

establishing objectives

M. Ratio of dollar value of total resources proposed for allocation to

establishing objectives to dollar value of total resources

actually required

1.2.1 Ability to insure that all things BIFV

gunners will be able to do after train-

*. ing will be stated so that tests of all

required abilities may be specified7

Elements or factors A. Essential elements of abilities intended to be included in the stateof gunner abilities ments of BIFV gunner abilities (listing and description of each such

* intended to be type of element)

stated B. Inessential elements intended to be included in the statements of

BIFV gunner abilities (listing and description of each type)

C. Essential elements of abilities not intended to be included in the

statements of BIFV gunner abilities (listing and description of

each type

Figure 14. Sample Measures Hierarchy Number 1

1

' 89a

I ...... * . .. .

, :-~~~~~~~~~~~~~~~~~~~~~~~.-:. -. .'.-. /'.-'.......,......."-. ...- .'..--....... -.... .-.-..-......-- ... '. . .'..- . .
Intended formats for D. Reviewers' ratings of the degree to which the intended formats will

st3ting BIFV gunner enable clear expression of BIFV gunners' abilities

abilities E. Specific deficiencies in the intended formats identified by

reviewers (listing and description of each deficiency)

Plans for producing F. Essential steps missing from the plans for producing statements of

statements of In- abilities

tended abilities of G. Inessential/inappropriate steps included in the plans

BIFV gunners H. Inconsistencies/deficiencies in the sequencing of steps in the plans

I. Deficiencies in the staff's abilities to carry out these particulaplans

'2: J. Deficiencies in the resources allocated to implementing these particular plans

1.2.1.1 Ability to insure that definitions of

the required gunner actions will be

included in the statements of intended I

gunner abilities, so that the specified

tests may address those actions

Types of PIFV gunner A. Relevant types of actions not intended to be defined (description ef

performance actions each type)

. it ended to he B. Irrelevant types of actions intended to be defined (description of

defined each type)

C. Reviewers' ratings of the observability of the types of actions

intended to be defined

Plans for defining D. Essential steps missing from the plans for defining performance

the actions re- actions

quired of BIFV E. Inessential steps included In the plans

gunners F. Inconsistenciesldeficiencies in the sequencing of steps in the pW-A .

G. Deficiencies in the staff's abilities to carry out these particular

plans

H. Deficiencies in the resources allocated to implementing these particular plans

.. 1.2 Ability to insure that definitions of

the appropriate conditions under which

the gunner actions are to be performed

will be included in the statements of

intended gunner abilities, so that the

specified tests may reflect those

conditions

Elements or types of A. Relevant elements or types of conditions not intended to be defined

BIFV gunner perfor- (descriptions of each) marce conditions in- B. Irrelevant elements or types of conditions intended to be defined

tended to be defined

.',.7 Plans for defining C. Essential steps missing from the plans for defining performance

the BIFV gunnery conditions

performanc, con- D. Inessential/inapprc Hate steps included in the plans

ditiurns E. Inconsistencies/deficiencies in the sequencing of steps in the plans

F. Deficiencies in the staff's abilities to carry out these particular

plans

G. Deficiencies in the resources allocated to implementing these particular plans

Figure 14. (Continued)

89b
1.2.1.3 Ability to insure that dfinsTof

the appropriate standards of accept-

~ I able performance of the gunner actions

will be included in the statements of

intended gunner abilities, so that the

specified tests may incorporate those

standards

Elements or factors A. Relevant elements or factors of performance standards not intencded

of BlFV gunner per- to be defined

formanc(. standards B. Irrelevant elements or factors of performance standards intended

intended to be to be defined

defined

Plans for defining C. Essential steps missing from the plans for defining performance

the standards of standards

BIFV gunnery D. Inessential /inappropriate steps included in the plans performance E. Inconsistencies/ deficiencies in the sequencing of steps In the plans

F. Deficiencies in the staff's abilities to carry out these particular

plans

G. Deficiencies in the resources allocated to implementing these par- I ticular plans

1 .2.2 Ability to insure that the identified

BIFV gunnery performance ojbectlves will be applied to provide a basis for

objectively assessing gunners'

performance

.niements. of the A. Essential elements or segments of the basis for objective assessbasis for assess- ment of BIFV gunner performance intended to be provided (listing

ment intended to he and description of each)

provided B. Inessential elements or segments of the basis intended to be provided (listing and description) -C. Essential elements or segments of the basis not intended to be

provided (listing and description)

Plans for providing D. Essential steps missing from the plans for providing a basis for

a basis for objec- objective assessment

tively assessing E. Inessential /inappropriate steps included in the plans

BIFV gunners' F. Inconsistencies/ deficiencies in the sequencing of step in the plans performance ~ G. Deficiencies in the staff's abilities to carry out thesparticular

plans

H. Deficiencies in the resources allocated to implementing these particular plans

1.2.2.1 Ability to insure that appropriate pretest items addressing the performance

objectives will be developed and included in the testing specifications

Types of BIFV gun- A. Relevant types of pretest items not intended to be developed (listing

nery pretest items and description)

intended to be B. Irrelevant /inappropriate types of pretest items intended to be

developed developed (listing and description)

Plans for develop- C. Essential steps missing from the plans for developing pretest items

Ing BIFV gunnery D. Inessentiall/inappropriate steps included in the plans

pretest items E. InconsistencieslIdeficiencies in the sequencing of steps in the plans

F. Deficiencies in the staff's abilities to carry out these particular

plans

G. Deficiencies in the resources allocated to implementing these particular plans

Figure 14. (Continued)

.Id 89C

- ~ % L.6- e _.-. .. ..
1.2.*2.2 Ability to Insure that appropriate posttest items addressing the performance Iobjectives will be developed ad~ in

Icluded in the testing specifications

Types of BIFV gun- A. Relevant types of posttest items not intended to be developed (listnery poattest items Ing and description)

intended to be B. Irrelevant /inappropriate types of posttest items intended to be

developed developed (listing and description)

Plans for develop- C. Essential steps missing from the plans for developing posttest items

ing BIFV gunnery D. Inessential /inappropriate steps Included in the plans

posttest items E. Inconsistencies f deficiencies in the sequencing of steps in the plans

F. Deficiencies in the staff's abilities to carry out these particular

plans j~JG. Deficiencies in the resources allocated to implementing these particular plans

Figure 14. (Concluded)

89

I%

IA
Hierarchy Number 2: BIFV gunnery design subsystem process of analyzing taks selected for training

(Note: Issue numbers refer to the taxs in Figure 7 from which they are derived.)

Issues 2.3 Conduct of anailysis of BIFV gunner tasks, so that appropriate tests of

those tasks may be specified

Attributes Measures

Task analytic A. Deviations between the methods actually applied and the methods spe- methods applied to cified in the plans for analyses (listing and description of each

the BIFV gunner such deviation)

: tasks B. Aspects or elements of the methods applied that are irrelevant to or inconsistent with the analytic purposes (listing and description

of each)

C. Aspects or elements missing from the methods applied that are essential for the analytic purposes (listing and description of each) D. Types of data needed for the methods that were not obtained (listing and description of each data deficiency)

The scope of the E. Factors supposed to be determined from the task analyses that

BIFV gunner task actually were not determined

analyses conducted F. Factors irrelevant to the analytic purposes that were determined

in the task analyses

Completeness of the G. Relevant BIFV gunner tasks to which the analytic methods were not

B7FV gunner task applied

analyses conducted H. Factors missing from the analyses applied to other relevant BIFV

gunner tasks

. Accuracy of the I. Inaccuracies in the input data obtained for application of the

BIFV gunner task analaytic methods

analyses conducted J. Specific misinterpretations of the analyzed data

K. Specific aspects or elements of the analytic methods that were

misapplied

2.3.2 Classification of BIFV gunner tasks

in terms of the domains of learning

ers fthey involve

Methods employed to A. Aspects or elements of the methods employed that are irrelevant to

identify the do- or inconsistent with identification of the domains of learning

main: of learning involved in a task (listing and description)

of the BIFV gunner B. Aspects or elements missing from the methods employed that are

tasks essential for identifying the domains of learning (listing and

description)

C. Input data required for the methods that are missing or insufficient

(listing and description)

. The domains of D. Domains identified as involved in a task that actually are not relelearning identified vant to that task (listing and description of each such incident) for each BIFV gun- E. Domains not identified as involved in a task that actually are inner task volved in that task (listing and description of each such incident)

.-- a

b

Figure 15. Sample Measures Hierarchy Number 2

90a
2.3.2.1 Identification of the knowledge that

to perform their tasks, so that

appropriate tests of knowledge may

be specified and developed

Methods employed A. Aspects or elements of the methods employed that are irrelevant to

to identify the or inconsistent with identification of the knowledge required for

knowledge require- performing a task (listing and description)

ments of BIFV B. Aspects or elements missing from the methods employed that are * ~.gunner tasks essential for identification of the knowledge required for performing a task (listing and description)

:~C I . Input data required for the methods that are missing or insufficient (listing and description)

The knowledge D. Knowledge elements identified as required for a given task that

Arequirements iden- actually are not needed for performance of that task (listing and ~ itified for BIFV description)

gunner tasks E. Knowledge elements not identified as required for a given task that

actually are needed for performance of that task (listing and

description)

2.3.2.2 Identification of the mental skills

that BIFV gunners must possess in

order to perform their tasks,* so that

appropriate tests of mental skills

-May be specified and developed

Methl.ds employed A. Aspects or elements of the methods employed that are irrelevant to

to identify the or inconsistent with identification of the mental skills required

mental skill for performing a task (listing and description)

requirements of B. Aspects or elements missing from the methods employed that are BIFV gunner essential for identification of the mental skills required for

tasks performing a task (listing and description)

C. Input data required for the methods that are missing or Insuffi-

~:: ~cient (listing and description)

The mental skill D. Mental skills identified as required for a given task that actually

requirements ideii- are not needed for performance of that task (listing and description)

tified for BIFV E. Mental skills not identified as required for a given task that actugunner tasks ally are needed for performance of that task (listing and description

2.3.2.3 Identification of the physical skills

that BIFV gunners must possess in

order to perform their tasks, so that

appropriate tests of physical skills

may be specified and developed

Methods employed to A. Aspects or elements of the methods employed that are irrelevant to

identify the physi- or inconsistent with identification of the physical skills

Wcal skill require- required for performing a task (listing and description)

ments of BIFV B. Aspects or elements missing from the methods employed that are

gunner tasks essential for identification of the physical skills required for

performing a task (listing and description)

C. Input data required for the methods that are missing or insuffi- .~ ~ cient (listing and description)

The physical skill D. Physical skills Identified as required for a given task that actually

requirements iden- are not needed for performance of that task (listing and description)

tified for BIFV E. Physical skills not Identified as required for a given task that

gunner tasks actually are needed for performance of that task (listing and

description)

Figure 15. (Continued)

90b
! %

~ 12.3.2.4 Identification of the attitudes that

BIFV gunners must possess in order

to perform their tasks,* so that appropriate tests of attitudes may be

specified and developed

Methods employed to A. Aspects or elements of the methods employed that are irrelevant to

dinal requirements performing a task (listing and description)

ofBIFV gunner B. Aspects or elements missing from the methods employed that are tasks essential for identification of the attitudes required for performing a task (listing and description)

C. Input data required for the methods that are missing or Insufficient (listing and description

. IThe attitudinal D. Attitudinal elements identified as required for a given task that

requirements iden- actually are not needed for performance of that task (listing and

*tified for BIFV description)

gunner tasks E. Attitudinal elements not Identified as required for a given task that

actually are needed for performance of that task (listing and

description)

2.3.3 Ientification of the conditions for

I learning the BIFV gunner tasks

Methods employed to A. Aspects or elements of the methods employed that are irrelevant to

identify the condi- or inconsistent with identification of the conditions for learning tions for learning a task (listing and description of each such aspect) the BiFv gunner B. Aspects or elements missing from the methods employed that are tasks essential for identifying the conditions for learning (listing and

description C. Input data required for the methods that are missing or insuffi- - cient (listing and description)

lypes of conditions D. Types of conditions for learning identified as associated with a for learning that task that actually are not relevant to that task (listing and are identified for description)

each UUIFV gunner E. Types of conditions for learning not identified as associated with

task a task that actually are relevant to that task (listing and

description)

2.3.3.1 1 Identification of the prerequisite

-> I knowledge that BIFV gunners should I have in order to learn theirtsk

Methods employed A. Aspects or elements of the methods employed that are irrelevant to

to identify the or inconsistent with identification of the knowledge prerequisites prerequisite required for learning a task 4.knowledge for B. Aspects or elements missing from the methods employed that are BIFV gunner essential for identification of the knowledge prerequisites tasks required for learning a task .1C. Input data required for the methods that are missing or insufficient

Knowledge items D. Knowledge items identified as prerequisites for learning a given identified as task that actually are not needed or helpful for learning the

prerequisites for- task (listing and description) learning the E. Knowledge items not identified as prerequisites for learning a hlFV gunner given task that actually are essential for learning the task

tasks F. Knowledge items not identified as prerequisites for learning a

given task that actually would facilitate learning the task,

although not essential for learning

Figure 15. (Continued)

90c

r. . . .
I

2.3.3.2 Identification of the prerequisite mental skills that BIFV gunners

should have in order to learn

I their tasks

Methods employed A. Aspects or elements of the methods employed that are irrelevant to

to identify the or Inconsistent with identification of the mental skills preprerequisite requisites for learning a task

mental skills for B. Aspects or elements missing from the methods employed that are

"" BIFV gunner essential for identification of the mental skills prerequisites

tasks for learning a task

C. Input data required for the methods that are missing or insufficient

Specific mental D. Specific mental skills identified as prerequisites for learning a 4 skills identified given task that actually are not needed or helpful for learning

as prerequisites the task

for learning the E. Specific mental skills not identified as prer' quisites for learning

BIFV gunner a given task that actually are essential for learning the task

tasks F. Specific mental skills not identified as prerequisites for learning

a given task that actually would facilitate learning the task,

although not essential for learning

2.3.3.3 Iden tification of the prerequisite

physical skills that BIFV gunners

should have in order to learn

I their tasks

Methods employed A. Aspects or elements of the methods employed that are irrelevant to

to identify the or inconsistent with identification of the physical skills preprerequisite requisites for learning a task

physical skills B. Aspects or elements missing from the methods employed that are

for BIF V essential for identification of the physical skills prerequisites

gunner tasks for learning a task

C. Input data required for the methods that are missing or insufficient

Specific physical D. Specific physical skills identified as prerequisites for learning a

skills identified given task that actually are not needed or helpful for learning

as prerequisites the task

'5' for learning the E. Specific physical skills not identified as prerequisites for learning

BIFV gunner a given task that actually are essential for learning the task

tasks F. Specific physical skills not identified as prerequisites for learning a given task that actually facilitate learning the task, although

not essential for learning

2.3.3.4 Identification of the prerequisite

attitudes that BIFV gunners should

% have In order to learn

"I their tasks

Methods employed A. Aspects or elements of the methods employed that are irrelevant to

to identify the or inconsistent with identification of the attitudinal prerequiprerequisite atti- sites for learning a task

tudes for BIFV B. Aspects or elements missing from the methods employed that are gunner tasks essential for identification of the attitudinal prerequisites for

learning a task

- C. Input data required for the methods that are missing or insufficient

Specific ati~tudes D. Specific attitudes identified as prerequisites for learning a given

identified as pre- task that actually are not needed or helpful for learning the task

requisites for E. Specific attitudes not Identified as prerequisites for learning a

learning the given task that actually are essential for learning the task

I3IFV gunner F. Specific attitudes not identified as prerequisites for learning a

tasks given task that actually facilitate learning the task, although

not essential for learning

Figure 15. (Concluded)

90d

,,.,..,. .. .-' - ; . ..'... . : - .: _: , •-€' Lfs f.. a i /tli i ..d, = . . . . .
Hierarchy Number 3: BIFV gunnery design subsystem product

consisting of stated performance objectives

(Note: Issue numbers refer to the taxa in Figure 7 from which they are derived.)

', Issues

3.2 I Production of accurate and complete Istatements of BIFV gunner performance objectives, so that tests of gunners'

qualifications may be specified

* Attributes Measures

The BIFV gunner A. Statements that define performance objectives that actually are not

performance needed for BIFV gunnery (listing and description of each such

objectives that statement)

are stated B. Performance objectives that actually are needed for BIFV gunnery

that are not addressed in any of the statements produced (listing

and description of each such objective)

The format of the C. Essential elements of performance objective statements not Included

statements of in the format employed for stating BIFV performance objectives

BIFV gunner (listing and description of each missing element)

performance D. Inessential/inappropriate elements in the format employed for statobjectives ing BIFV performance objectives (listing and description of each

such element)

The completeness E. Elements missing from the statements of particular BIFV perforof the statements mance objectives (tabulation of missing elements as a function of

of BIFV gunner the stated objectives)

performance

objectives

The accuracy of F. BIFV performance objective statements that contain erroneous or

the statements of inaccurate elements (tabulation of inaccurate elements as a

BIFV gunner function of the stated objectives)

performance G. Description of the nature of the inaccuracies, for each stated

objectives objective

3.2.1 Production of exact definitions of

the BIFV gunner-learners' expected

achievements as a result of

the training

The BIFV gunner- A. Achievements defined that actually are not necessarily expected of

learner achieve- BIFV gunner-learners during training (listing and description of

ments that are each such achievement)

defined B. Achievements not defined that actually are expected of BIFV

gunner-learners during training (listing and description of each

such achievement)

C. Inexact or ambiguous elements in the definitions of BIFV gunnerlearner achievements (listing and description)

D. Inaccurate elements in the definitions of BIFV gunner-learner

achievements (listing and description)

The factors in- E. Factors included in the definitions that actually are irrelevant

eluded in the to the description of intended achievements of BIFV gunnerdefinitions of learners (listing and description)

BIFV gunner- F. Factors missing from the definitions that actually are essential

learner achieve- for describing intended achievements of BIFV gunner-learners

ments (listing and description)

j( Figure 16. Sample Measures Hierarchy Number 3

91a

,-' -j,......... ILI ....... ...... 

, _ : r0 * . % . " . -- a ' I , a• m • - --
J3.2.1.1 Production of correct specifications

of the capabilities that BIFV

I gunner-learners are expected

*to achieve

S' Capabilities spe- A. Specified capabilities that actually are not relevant to the intended

I cified for achievements of BIFV gunner-learners (listing and description)

achievement B. Capabilities not specified that actually are essential to the

by BIFV intended achievements of BIFV gunner-learners (listing and

gunner-learners description)

I3.2.1.2 Production of correct specifications

of the actions that BIFV gunnerlearners are expected to execute in Iorder to demonstrate each capability

The actions speci- A. Specified actions that actually do not necessarily demonstrate the

fied for execution intended capabihties (listing and description)

by BIFV gunner- B. Actions not specified that actually would demonstrate the intended

learners capabilities (listing and description)

The observability C. Reviewers' ratings of the inherent observability of the specified

of the specified actions (Likert or numeric scale ratings)

- actions D. Requirements for imposing artificial constraints on the gunnerlearners' execution of the actions to enhance observability

(listing and description)

- E. Reviewers' ratings of the degree to which observation will impede

or interfere with gunner-learners' execution of the actions

(Likert or numeric scale ratings)

S.2.1.3 Production of correct specifications

of the objects that are to result

from the actions executed by the

BIFV gunner-learners

Objects specified A. Specified objects that actually do not necessarily result from exeas results of the cution of the actions of BIFV gunner-learners (listing and

actions to be exe- description)

cuted by BIFV B. Objects not specified that actually would result from execution of

gunner-learners the actions of BIFV gunner-learners (listing and description)

C. Specified attributes/features of the objects that do not actually

indicate achievement of the intended gunner-learner capabilities

(listing and description)

D. Non-specified attributes/features of the objects that actually are important indicators of achievement of the intended gunner-learner

. capabilities (listing and description)

Measurability of E. Potential methods of direct measurement of the relevant attributes/

the specified features of the specified objects (listing and description of

"1 objects each method)

F. Potential methods of indirect/inferential measurement of the

relevant attributes/features of the specified objects (listing I jand description of each method)

G. Probabilistic distribution of measurement error for each of the

potential measurement methods

'1

Figure 16. (Continued)

91b

. . . . . .. .. . . . . .ALA. .
3.2.1.4 Production of correct specifications of the circumstances under which

the BIFV gunner-learners are to exe- cute the actions to produce the objects

Circumstances A. Specified circumstantial factors/categories that actually do not

specified for exe- apply to execution of the actions under realistic job conditions

cution of the (listing and description of each such factor/category) actions by BIFV B. Non-specified circumstantial factors/categories that actually do

gunner-learners apply to execution of the actions under realistic job conditions,

and that might affect the execution of the actions (listing and

description of each)

C. Deviations between the values, ranges, limits, etc., specified for

circumstantial factors/categories and the values, ranges, limits,

etc., that actually exist in those factors/categories under realistic

job conditions (magnitude and direction of each such deviation)

3.2.1.5 Production of correct specifications

of the tools and equipment to be

used by BIFV gunner-learners in

L ,executing the actions

Tools and equip- A. Specified tools and equipment that actually would not be available

ment specified for or employed under realistic job conditions (listing and description) use in executing B. Tools and equipment not specified that actually would be applicable the gunner- and available under realistic job conditions (listing and

learners' actions description)

3.2. 1.6 Production of correct specifications - of the constraints to be imposed

upon the execution of actions

by BIFV gunner-learners

Constraints speci- A. Specified constrained behaviors/activities that actually would not

fied for the exe- be constrained under realistic job conditions (listing and

cution of the description)

gunner-learners' B. Behaviors/activities not specified for constraint that actually

actions would be constrained under realistic job conditions (listing and I description) C. Deviations between the degree of constraints specified and the

degree of constraints that actually would exist under realistic job

conditions (magnitude and direction of each such deviation)

3.2.1.7 Production of correct specifications

V. of the criteria to be used to judge the adequacy of the objects resulting

from the actions executed by the

BIFV gunner-learners

Criteria specified A. Deviations between the criteria specified and the standards actually

• as standards of required for adequate performance on the job (magnitude and

adequacy for the direction of each such deviation)

objects

Figure 16. (Concluded)

91C

.- , - - - .- - - . %
Hierarchy Number 4: BIFV gunnery design subsystem product

consisting of tests

(Note: Issue numbers refer to the taxa in Figure 7 from which they are derived.)

40 Issues

3.5 F Production of appropriate tests for

administration to BIFV gunner-learners

. I Attributes Measures

The BIFV gunner- A. Prepared tests that address each BIFV gunnery performance objeclearner tests that tive (listing and description as a function of each performance

are prepared objective)

B. Prepared tests that relate to no BIFV gunnery performance objective (listing and description of each such test)

BIFV glinnery C. Reviewers' ratings of the utility of each prepared test for each

testing applica- testing application/requirement (yes/no rating of each test for

tion/requirements each application)

for which tests

are prepared

BIFV gunner D. Listing of tests prepared for each behavior associated with each

behaviors for performance objective

which tests are

prepared

Relevance of the E. Deviations between the behaviors elicited by the tests and the

tests behaviors that actually are to be addressed (listing and description of each such deviation)

3.5.1 Prod-ction of appropriate tests or

- assessing the qualifications of

candidate BIFV gunner-learners

Candidate gunner- A. Relevant qualifications for which tests are prepared (listilig and

learner qualifi- description of each such qualification)

cations for whir-h B. Relevant qualifications for which no tests are prepared (listing

tests are prepared and description of each such qualification)

C. Irrelevant "qualifications" for which tests are prepared (listing

and description of each)

The tests pre- D. Elements of each test that deviate from the qualifications actupared for assess- ally required of acceptable gunner-learners (listing and description

ment of the of each such deviating element)

qualifications E. Probability that each test will be passed by candidates who

actually do not possess the qualification being tested

F. Probability that each test will be failed by candidates who

.I actually do possess the qualification being tested

G. Cumulative probability that an actually unqualified candidate wll

pass sufficient qualification tests to be accepted as a gunnerlearner

H. Cumulative probability that an actually qualified candidate will

fall sufficient qualification tests to be rejected as a gunnerlearner

Figure 17. Sample Measures Hierarchy Number 4

92a

1J
3.5.1.1 Proction of qualification tests that

provide adequate measures of candidate

BIFV gunner-learners'

prerequisite abilities 9

Domains of A. Qualification tests that can only be passed via application of a learning addressed domain of learning other than the domain relevant to the qualifi- tby the tests cation supposedly to be tested (listing and description of each

such test)

B. Qualification tests that can also be passed via application of a

domain of learning other than the domain relevant to the qualification supposedly to be tested (listing and description of each

such test)

Actions elicited C. Deviations between the actions elicited by the tests and the actions by the tests required to demonstrate each qualification (listing and descrip- .. ~.I D.tion of each such deviation)

D.Probability that the execution of the elicited action implies the

ability to execute the action required to demonstrate the

qualification

E. Probability that a failure to execute the elicited action Implies

the inability to execute the action actually required to demonstrate the qualification

Objects intended F. Deviations between the objects intended to result from the test

to result fromt actions and the objects intended to result from the actually required

the actions actions (listing and description of each such deviation)

elicited by the G. Probability that the production of the test object implies the

tests ability to produce the actually required object

4 H. Probability that a failure to produce the test object implies the

inability to produce the actually required object

- 3.51.2 rodutionof appropriate procedures

forappyig the qualification tests to I BIFV gnner-learner canriidates

Circumstances A. Deviations between the circumstances specified for the tests and 7specified for the circumstances under which the actual qualification abilities

the tests need to be applied during training (listing, description. magnitude and direction of each such deviation)

B. Probability that passing the test under the specified circum- I stances implies the candidate can apply the qualification ability during training

C. Probability that failing the test under the specified circumstances implies the candidate cannot apply the qualification

ability during training

Tools and equip- D. Deviations between the tools and equipment specified for the

ment specified tests and the tools and equipment to be used by gunner-learners

for the tests to apply the qualification abilities during training (listing and

description of each such deviation)

E. Probability that passing the test using the specified tools and

equipment implies the candidate can apply the qualification ability

using the tools and equipment to be provided during training

F. Probability that falling the test using the specified tools and

equipment implies the candidate cannot apply the qualification

ability using the tools and equipment to be provided during training

Figure 17. (Continued)

92b
Constraint, speci- G. Deviations between the constraints specified for the tests and the

fied for the tests actual constraints governing the gunner-learners' application of

the qualification abilities during training (listing and description

of each such deviation)

H. Probability that passing the test under the specified constraints

implies that the candidate can apply the qualification ability

under the constraints that will exist during training

I. Probability that failing the test under the specified constraint,

implies that the candidate cannot apply the qualification ability

under the constraints that will exist during training

3.5.1.3 Production of appropriate qualification

test standards for accepting candidate

BIFV gunner-learners into training

tanda7-!s sDeci- A. Deviations between the specified test standards and the actual

fied for th,' standards of adequate performance of the qualification abilities

qualificatiorn during training (listing, description, magnitude and direction of

tests each such deviation

B. Probability that satisfaction of the test standards implies that

the candidate can satisfy the actual standards of adequate performance during training

C. Probability that failure to satisfy the test standards implies

that the candidate cannot satisfy the actual standards of

adequate performance during training

,3... Production of appropriate tests for

assessing requirements for tailoring

the instruction to accepted

BIFV gunner-learners

BiFV g-innery A. Relevant gunnery abilities for which tests are prepared (listing and

ablities; for description of each such ability)

-,.hih tail, -nrg B. Relevant gunnery abilities for which no tests are prepared (listing

terls ale pr-- and description of each such abihty)

pared C. Irrel-van' "abilities" for which tests are prepared (listing, and

description of each)

"he tests prepared D. Elements of each test that deviate from the abilities actually

for asessment of required of BIFV gunners (listing and description of each such

the needs for deviating element)

tailoring instruc- E. Probability that each test will be passed by learners who

tion to fit actually do nut possess the ability being tested

learners' abili- F. Probability that each test will be failed by learners who

ties actually do possess the ability being tested

G. Cumulative probability that a learner's test performance will

result in inappropriate tailoring of instruction for that learner

. . .1 Production of tailoring tests that

provide adequate pre-training

measures of accepted BIFV

[' guriner-learners' abilities

Domains o" learn- A. Tailoring tests that can only be passed via application of a

ir addressed by domain of learning other than the domain relevant to the ability

- the tests supposedly to be tested (listing and description of each such test)

B. Tailoring tests that can also be passed via application of a

domain of learning other than the domain relevant to the ability

supposedly to be tested (listing and description of each such test)

Figure 17. (Continued)

92c

% - % . . . . . . . . . -
Actions elicited C. Deviations between the actions elicited by the tests and the

by the tests actions required to demonstrate each tested ability (hAting and

description of each such deviation)

D. Probability that the execution of the elicited action implies the

learner can execute the action actually required to demonstrate

the tested ability

E. Probability that a failure to execute the elicited action implies

the learner cannot execute the action actually required to

demonstrate the tested ability

Objects intended F. Deviations between the objects intended to result from the test

to result from the actions and the objects intended to result from the actually

actions elicited required actions (listing and description of each such deviation) by the tests G. Probability that the production of the test object implies the

ability to produce the actually required object

H. Probability that a failure to produce the test object implies

the inability to produce the actually required object

3.5..2 1Production of appropriate procedures

3.5.2.2 for applying the tailoring tests

I to BIFV gunner-learner

Circumstances A. Deviations between the circumstances specified for the tests and

it.-Aspecified for the circumstances under which the actual gunnery abilities need

the tests to be applied on the job (listing, description, magnitude and

direction of each such deviation)

B. Probability that passing the test under the specified circum- I stances implies the learner can apply the gunnery ability under

realistic job circumstances

C. Probability that failing the test under the specified circumstances implies the learner cannot apply the gunnery ability under

realistic job circumstances

Tools and equip- D. Deviations between the tools and equipment specified for the tests

ment specified and the tools and equipment to be used by BIFV gunners to apply for the tests the tested abilities on the job (listing and description of each

such deviation)

E. Probability that passing the test using the specified tools and

equipment implies the learner can apply the gunnery ability using

the tools and equipment to be provided on the job

F. Probability that failing the test using the specified tools and

equipment implies the learner cannot apply the gunnery ability

using the tools and equipment to be provided on the Job

Constraints spedt- G. Deviations between the constraints specified for the tests and the

fied for the tests actual constraints governing the BIFV gunners' application of the

tested abilities on the job (listing and description of each

such deviation)

H. Probability that passing the test under the specified constraints

implies that the learner can apply the gunnery ability under the

constraints that will exist on the job I. Probability that falling the test under the specified constraints

Implies that the learner cannot apply the gunnery ability under

the constraints that will exist on the job

* Figure 17. (Continued)

92d

%. . . . . .
3.5.2.3 Production of appropriate tailoring Itest standards for tailoring the

instruction to the BIFV

gunner-learners' abilities

Standards speci- A. Deviations between the specified test standards and the actual

fied for the standards of adequate performance of the tested abilities on the

deviation)

B. Probability thtstsato ftetest sadrsIpisthat the A learner can satisfy the actual standards of adequate performance on the job C. Probability that failure to satisfy the test standards implies that

the learner cannot satisfy the actual standards of adequate performance on the job

3.5.3 Production of appropriate tests or assessing BIFV gunner-learners achievements as a result oai ig

Gunner-learner A. Relevant achievements for which tests are prepared (listing and ~: Iachievements for description of each such achievement)

which tests are B. Relevant achievements for which no tests are prepared (listing and

N: a prepared description of each such achievement)

C. Irrelevant "achievements" for which tests are prepared (listing and

description of each) IThe tests prepared D. Elements of each test that deviate from the achievements actually far assessment of required of certified BIFV gunners (listing and description of

the BlFV gunner- each such deviating element) learners' achieve- E. Probability that each test will be passed by learners who actually

ments do not possess the achievement being tested

F. Probability that each test will be failed by learners who actually do possess the achievement being tested

G. Cumulative probability that an actually unqualified learner will

pass sufficient achievement tests to be certified as a gunner H. Cumulative probability that an actually qualified learner will

fail sufficient achievement tests to be rejected as a gunner

3.5.3.1 I Production o? achievement tests that

provide adequate post-training measures]

A ing addressed by oai BIFV gunner-learners' abilte

Domains of learn- A. Achievement lerigohrta tests that can only h be oanrlvn passed via application ote of alt

-. -the tests supposedly to be tested (listing and description of each such test)

B. Achievement tests that can also be passed via application of a

domain of learning bther than the domain relevant to the ability

supposedly to be tested (listing and description of each such test)

Actions elicited C. Deviations between the actions elicited by the tests and the actions

by the tests required to demonstrate each achievement (listing and description

of each such deviation)

D. Probability that the execution of the elicited action implies the

ability to execute the action actually required to demonstrate

the achievement

E. Probability that a failure to execute the elicited action implies the inability to execute the action actually required to demonstrate the achievement

Figure 17. (Continued)

92e
Objects intended F. Deviations between the objects intended to result from the test Ito result from the actions and the objects intended to result from the actually

actions elicited required actions (listing and description of each such deviation)

by the tests G. Probability that the production of the test object implies the

, I ability to produce the actually required object

H. Probability that a failure to produce the test object implies

the inability to produce the actually required object

2.::3.5.3.2 IProduction of appropriate procedures

I for applying the achievement tests

I to BIFV gunner-learnrs

Circumstances A. Deviations between the circumstances specified for the tests and

specified for the the circumstances under which the actual tested abilities need to

tests be applied on the job (listing, description, magnitude and

direction of each such deviation)

B. Probability that passing the test under the specified circumstances implies the learner can apply the gunnery ability under

realistic job circumstances

C. Probability that failing the test under the specified circumstances implies the learner cannot apply the gunnery ability under

realistic job circumstances

Tools and equip- D. Deviations between the tools and equipment specified for the tests

4. rent specified and the tools and equipment to be used by BIFV gunners to apply

for the tests the tested abilities on the job (listing and description of each

such deviation)

&. IE. Probability that passing the test using the specified tools and equipment implies the learner can apply the gunnery ability

using the tools and equipment actually available on the job

F. Probability that failing the test using the specified tools and

.1.1 equipment implies the learner cannot apply the gunnery ability using the tools and equipment actually available on the job

Constraints spe- G. Deviations between the constraints specified for the tests and the

cified for the actual constraints governing the BIFV gunners' application of

tests the tested abilities on the job (listing and description of each

such deviation) S1H. Probability that passing the test under the specified constraint.

jimplies that the learner can apply the gunnery ability under the

constraints that will exist on the job

1. Probability that failing the test under the specified constraints

implies that the learner cannot apply the gunnery ablity under

~ Ithe constraints that will exist on the job

3.5.3.3 Production of appropriate achievement

test standards for certifying I learners as BIFV gunners

< IStandards speci- A. Deviations between the specified test standards and the actual

fied for the standards of adequate performance of the tested abilities on the

achievement tests job (listing, description. magnitude and direction of each such

deviation

B. Probability that satisfaction of the test stardards implies that

the learner can satisfy the actual standards of adequate performance on the job C. Probability that failure to satisfy the test standards implies

that the learner cannot satisfy the actual standards of adequate

performance on the job

§1~ ~Figure 17. (Concluded)

92f

%. % - . . . . . .. . . . . . .

'.1 pgh:. M---_ A .- -: . - - - - -
-N

*

S. S.-V

N. .1"

-C'.

4

-N

*0~*

'U.

-p..

-V

-Pt

V.,.

'4:'

Page intentionally left blank

92 0

t} ~Q~j53-ZKKK<K-QQ :YK: )4V2tK: 2- y-t~ ~ -&itt§s -

-

-

-

k 

Cr 

'.-t-.c-4

V '"'?'~-~ *- N
5. Concluding Remarks Concerning the Trial Application to

Measurement of Training System Performance/1 Effectivenes~s

Clearly, much remains unproven as we conclude this trial

application. Its final product is merely a partial set of measures for assessing

one very narrow aspect of the work required of the BIFV Gunnery Training

Design Subsystem. Most glaringly, those measures are untried. We have not

attempted actually to apply them, nor even to verify that the information

needed to generate them can be obtained. Those measures, moreover, derive

from attributes that also are unproven. No attempt has even been made to

I. demonstrate convincingly that the attributes stated really constitute all of the

factors pertinent to each evaluative issue, and only those factors.

0What, then, has been demonstrated? If nothing else, a set of SImeasures has been prod~uced that definitely is issues-oriented. We have been

able to proceed from a framework of basic performance requirements to a

statement of exactly what requirements apply to a given measurement

application. We have shown that such requirements can be expressed in terms

that permit concrete, tractable attributes to be defined. None of the

attributes stems from any particular approach to training system design. All

of the attributes strictly derive from the performance requirements that are

common to all such designs. This is a critical point:- it demonstrates that the

APM at least is rooted firmly in valid measurement issues. Too often,

evaluations of new systems are clouded by irrelevant and unfair comparisons

with older design concepts. The basic question of whether the new system

does what it is supposed to do sometimes is overlooked because the evaluators

actually measure how the new system "looks" in comparison to older models.

Conclusions based on such evaluations typically boil down to this: "The

system is no good because we've never done it this way before." That

design-oriented philosophy of measurement suppresses technological

*breakthroughs. in the training sphere, it prevents modern and effective

I methods of work site, learner-tailored instruction from playing their

appropriate role in the total learning environment and it masks the relative

ineffectiveness of the traditional, institutonal methods of training. The

authors consider it to be a major step forward to have shown that a training

measurement scheme can be kept free of artificial judgments of design. Use of

the APM has forced a rather complete taxonomy of performance requirements to

be considered in a measurement application; more complete than a typical,

less-structured analysis would provide.

Finally, it has been shown that measures can be identified for these

performance-oriented attributes. Much remains to e done to develop

*guidelines for insuring that the best such measures always are identified. But

the first step, at least, has been taken in that direction.

The next section of this report explores the utility of the APM for

specifying how training system performance requirements can be met.

L -93-
D. The APM for Development of Training System Requirements

The purpose of this section is to illustrate how training system

performance taxa identified for any particular human-machine system can be

applied to produce training system requirements for that particular system.

The approach presented proceeds through the following steps:

(1) Define precisely the system of interest.

(For this example, the system of interest will be one training

subsystem, i.e., the training design subsystem of an Army

training system under consideration for further development /

procurement.)

(2) Determine and identify training system performance requirements. (This example will illustrate the procedures to be used in

translating general training design performance taxa for any

system into training system requirements for a system of

interest.)

(3) Present an approach on how these training subsystem design

requirements may be satisfied by the training design subsystem.

(Here examples are presented to illustrate how the training

design subsystem plans its approach to satisfy the training

design requirements.)

1. The System of Interest: The Design Subsystem of

U. S. Army Handgun Training

Currently, several U.S. joint military services study groups have

been formed to dete!rmine the need and feasibility of adopting a standardized

9mm handgun as a replacement for more than 25 different makes, models and

types of handguns currently in the U.S. services inventory (Air Force, Army,

Navy, Marines and Coast Guard). A Joint Service Small Arms Training Task

Force (JSSATTF) has recommended that all U.S. armed services adopt a single

family of 9 mm handguns; however, to date no final conclusions have been

made. The U.S. Army currently requires the largest number of handguns

(approximately 331,000) as a personal defensive weapon for such Army

personnel as certain front-line officers, non -commissioned officers, operators of

crew-served weapons, aircrews, drivers, equipment operators, rear echelon

personnel, etc.

In the context of this application an assumption is made that the

JSSATTF would assign the overall responsibility for training development to

the Army. Since the new handgun is considered to be in the family of

Infantry weapons, the U.S. Army Infantry School would develop the curricula

* for the Army 9mm Handgun Training System (AHTS). The AHTS is a training

system that incorporates the six general training subsystems within itself;

these are the Command, Design, Enabling, Emplacement, Logistics and Delivery

subsystems. Figure 18 depicts the network of other systems that interact with

the AHTS. These include:

-94-
0 The system known as the U.S. Army Infantry School. At this

stage of training development, the AHTS would be wholly J" contained within the Infantry School, which would be considered

a suprasystem to the system of interest.

0 JSSATTF. The Task Force is comprised of highly qualified

representatives of each of the U.S. military services. The

JSSATTF will determine likely handgun users and uses. The

JSSATTF will develop the training system requirements for the

AHTS. The JSSATTF is a suprasystem to the system of

interest.

U.S. military services. These macrosystems impact on training

system requirements through the JSSATTF. The AHTS has

to satisfy their training system requirements. Eventually, each

of these services will require its own service-specific handgun

jtraining system. They will, in fact, have a network of training

systems which will be somewhat analogous to the AHTS, i.e.,

various MOS training systems, unit training systems, school

training systems, etc.

0 Various U.S. Army Branch School Training Systems (Armor,

Artillery, Signal, etc.). The AHTS will be integrated

eventually into these training systems. AHTS training must be

compatible with these training systems. These systems -therefore must be considered as collateral systems to the system of interest and the AHTS.

0 Various U.S. Army Training Systems. The personnel who will

be assigned the new handgun function within a specific military

occupational specialty (MOS). The AHTS must be cdmpatible

with each of these training systems. These personnel will be

the AHTS learners. The systems are collateral to the AHTS.

o Various types of U.S. Army unit training systems (mechanized

infantry, self-propelled artillery, heavy-construction engineer,

etc.). The AHTS learners/users are within these types of

training systems as depicted by their Tables of Organizational

Equipment (TOE). The AHTS must be compatible with the

other on-going training the AHTS learners have or will have

received. These systems are collateral to the AHTS.

o- The 9mm handgun itself with its characteristics is an essential

collateral system to the AHTS.

7

-95- I

-4*4* .,- ., ,. -'.t-,. : .. .,. . % .- . *.. ". ~ -*- .".". . " "
cro S. U.S. U.S. U.S. U.S. NATO

~ S% ystem5) Air Force Navy Army Guard MaI Countrie

Supr Sericeer len)

System Small Training Arms U.S. AyoI i_ nch

ASrstems

Task Force U-.Ar. hool 0

Branch 4 e ° 0o

m U..U.S. pe AAy fantry Branch 3

School U.S. Ary

ydBran gun Training Sehoo

Army

igSyste o Handgun

"- . (Interest Design

TSubsystDeminSbse

Trining

rm tem

1 ining [cN

-- . .**. . . . . . . . " .

%',.S -'emL m Arm 0

Jng n 4

U..Army tem ,

"Training 2

CoserlSyste m

SystemI

U.SaAryguter

" nt -0 Sytems

Arimy p

% ' U.S. Army pe =-~~ I it pe

U.S Armyp3

2U.S Army |p e2

• .':unit ::::'Type I

,"- ]Figure 19. Major Systemns Interacting with the Army Handgun

=,..• lTraining" Design Subsystem

1 " -96-

, . . . ..
The Army Handgun Training Design Subsystem (AHTDS), a

subsystem of the AtITS, is the system of interest. The AHTDS is given the

,, I training requirements identified by the JSSATTF and the AHTS Command

subsystem and translates these requirements into specific training

plans/sp,-cifications. The AlTDS is the curriculum developer. As such, the

AHTDS determines the learner behaviors that are to be affected during

learning, arid specifies the sensory activities to be conducted to achieve the

effects. The AHTDS produces all of the learning procedures and resource

specifications and some of the material required by the AHT Delivery

subsystem. The AlTDS makes basic decisions with regard to learning

facilitators, facilities, material and equipment needed to implement the learning

activities.

Determining Training System Performance Requirements

At the conceptual stages of a particular training system's

development the suprasystem (which has direct administrative control of the

training system), has the overall responsibility to develop training system

performance requirements for each of the training system's six subsystems

(Command, Design, Enabling, Emplacement, Logistics and Delivery). For this

application, the JSSATTF is the suprasystem and therefore the training

administrator and, as such, exercises administrative control over the total

training system, i.e., the AHTS. The JSSATTF must specify the performance requirements in such detail as to insure that the total training system achieves

its purposes of learning and helping-to-learn.

This application focuses on the Training Design Subsystem of the

AHTS. In developing training performance requirements, the JSSATTF can

apply the general design subsystem performance taxa as shown in Figure 7.

Each taxon is a performance-oriented Design subsystem requirement.

,* YThereforo, in developing the Design subsystem training performance

-I requirement;, the complete taxa shown in Figure 7 applies. The taxa

constitute a performance checklist of capabilities, procedures and products that

the I)esign subsystem must plan for in meeting the training systems purposes.

The JSSATTF must translate the general Design subsystem training

performance requirements specifically for the AHTS. At this stage of system

development, the JSSATTF knows who the learners are, (i.e., the Army

handgun users) and, generally, for what purposes the Army handgun is to be

used, (i.e., the jobs for which the handgun is to be employed).

As an aid in writing the AHT Design Subsystem (AHTDS) training

requirements, it is helpful to organize the performance taxa identified in

Figure 7 into logically associated groups within a particular contextual aspect

of system performance, e.g., Performance Potentialities. A specific objectiveslevel taxon should be followed by the first functional purposes-level taxon; the

first functional purposes-level taxon should then be followed by its supporting

characteristics-level taxa. The process is continued for that objective level

taxon until all of the functional purposes-level taxa and their supporting

-97-

.'. . . ." ,

• , .' . b, , ,%.',. " '. ' ,.. .,- -. -', ',, '€ '-' -

" • ' w ' .',,°,, . . r ,.. ,-... -.. . • . - . .* .- . . ,.
'*0

characteristics-level taxa have been tied together explicitly. For example, to -p.:: Jprepare the AHTDS training performance requirements, the taxa for

* Performance Potentiality 1.4 shown in Figure 7 could be organized as follows:

Objectives Level 1.4 Defining training content

Functional Purposes Level 1.4.1 Selecting stimuli for each

objective

Characteristics Level 1.4.1.1 Defining propositions, names,

facts, etc., relevant to

information objectives

1.4.1.2 Defining concepts, rules,

algorithms, etc., relevant to

mental skills objectives

1.4.1.3 Defining movements, timings,

actions, etc., relevant to physical

skills objectives

1.4.1.4 Defining values, choices, etc.,

relevant to attitude objectives

Functional Purposes Level 1.4.2 Selecting instructional setting for

t each objective

Characteristics Level 1.4.2.1 Assessing applicability of Job

Performance Aids (JPA's)

S1.4.2.2 Assessing applicability of

Self-Teaching Exportable Packages

(STEP's)

1.4.2.3 Assessing applicability of Formal

On-the-Job Training (FOJT)

L 1.4.2.4 Assessing applicability of

Installation Support School (ISS)

1.4.2.5 Assessing applicability of Resident

School

Etc.

Use of this or some other suitable scheme greatly facilitates

organization of the performance taxa needed to generate a particular training

subsystem's performance requirements.

16. - 9 8 -

S -... .
The next step is to translate the organized general performance J taxa into subsystem training performance requirements for a specific training

system. Again for this application, training performance requirements are for

the AHTDS. Each requirement must address the specific subsystem which

must perform the work. Each of the contextual stage performance taxa

systematically will identify the performance requirements which must be

addressed by the training subsystem. Generally, the contextual stage

requirements for the AHTDS will state the following:

(1) AHTDS Performance Potentialities Requirements

These requirements state the capabilities/potentialities the

AHTDS must address.

(2) AIITDS Performance Processes Requirements

These requirements state the processes/procedures the AHTDS

must address.

(3) AHTDS Performance Products Requirements

- The requirements state the products the AHTDS must address.

To continue the above example, the performance taxa for

Performance Potentiality 1.4 and Functional Purpose 1.4.1 could be restated

into AHTDS training performance requirements as follows:

Objectives Level 1.4 The Handgun Training

Design Subsystem must

possess the ability to define .- ~ the handgun training content

required to achieve the

user's performance- objectives

Function,] Purposes Level 1.4.1 Given a list of handgun

users and uses, and given

the defined training go7als

.: I and analyzed performance

objectives, the Design

Subsystem must be able to

select stimuli for each

performance objective

relevant to the domains of

learning

Characteri.;tjc, ILevel 1.4. 1.1 The Handgun Training

Design Subsystem must

demonstrate that it possesses

the ability to define

propositions, facts, names,

etc., that are relevant to

information-type objectives

9

I" -99-
1. 4.1 .2 The Handgun Training

Design Subsystem must

demonstrate that it possesses

the ability to define

concepts, rules, algorithms,

etc., relevant to mental

skills objectives

1.4.1.3 The Handgun Training

Design Subsystem must

demonstrate that it possesses

the ability to define handgun

user/uses movements,

actions, timings, etc. , for

relevant physical skill

objectives

1.4.1.4 The Handgun Training

Design Subsystem must

demonstrate that it possesses

the ability to define handgun

user/uses choices, values,

etc., relevant to attitude

objectives

It should be noted by the reader that each requirement in the above

example is specific to the AIITDS. Further, since these are Performance

Potentialities Requirements, they are stated in terms of "abilities" or

"capabilitie.," which the AHDTS must demonstrate that it possesses to

effectively respond to the requirement. Similarly, the Performance. Process

Requirements are stated in terms which require the AHTDS to demonstrate its

ability to effectively plan an approach to satisfy these requirements. With

regard to Performance Product Requirements, these are stated in terms which

require the AtITDS to demonstrate its ability to plan how it will deliver or

produce the specified product requirements. Figure 19 is a sample of detailed

training performance requirements specific to the AHTDS. Figures BI through

B6 present all of the training performance requirements for the basic

potentialities specific to the AHTDS.

3. An Approach to Developing Training System

D esign Specifications

The general training system performance taxa shown in Figures 7

and 8 cinsctitute the performance requirements faced by every Design Subsystem and Enabling Subsystem, respectively. Figures 19 and B1-B6 in

Appendix B present Design Subsystem taxa in terminology that is specific to

the Army Handgun Training System. Those figures, too, deal in performance

requirements. In earlier segments of this report, the point was emphasized

tThat system evaluation should address itself solely to issues stemming from

-100- /( "

• ., , €, .. . .~...........-..... .......... ..... .. -.......... ,.. . . .
Ferformatnce Potentiality Hierarchy Number 1: The capability of identifying training goals and priorities

1.1

l~c Handgun Training D'esign Subsystem

must possess the ability *2 identify trainIng goals and priorities for all intended .r J military users and uses of the handgun.

1.1.1 1.1.2 .. 31.1.4

Iie a list of handgun users Liven a list of handgun users M-na liot of handgun users M-na list of handgun users and uses, the Design Subsystem and uses, the Design Subsystem and uses, and given the train- and uses, and given the trainmust be able to define the must be able to define what Ing goals defined for each ing goas defined for each

scope of learning needed for each user ultimately will be user/use, the De~ign Subsystem user/use, the Design Subsysteri schhngnuserluse in terms able to do with the handgun, must be able to identify the must be able to expressi the of th nweg.skillr.a, i training goal terms, relative importance of each goals in behaviorally oriented attdsrequired. dfndgoal. terms to establish a basis for

identifying handgun user per- * Lformnance objectives.

Yl;-Vndgun Training The Handgun Training YT1Lidgun Training T~eWandgun Training Design subsystem must Design Subsystem must Design Subsystem must Design Subsystem must demonstrate that it poe- demonstrate that it pos- demonstrate that It has demonstrate that it can sesses the ability to sessets the ability to the ability to formulate devise or select a suit- identify, for each hand- define the necessary ssessment& of training able taxonomy of goas/ gun user/use, exactly levels of performance goal importance, objectives to be used to what the user must know& achievements that each classify its intended exactly what skills the handgun user must mani'- training outcomes.

age- must possess; and fest Im order to apply

-exactly what attitudes the handgun to its

the user must manifest. intended uses.

77F-Tndgun Training T7i'T~indgun Training The Hndgun Training The Handgun TrsininR

Design Subsystem must Design Subsystem must Design Subsystem must Design Subsystem mi1st demonstrate that it pos- demonstrate that it is demonstrate that it can demonstrate that it is sesses the si~ity to able to determine cor- devise or select a suit- capable of stating its analyte the identified ractly the pre-training able scheme for ranking gtoals im terms that perknowtledgte, skills, and levels of the relevant the relative importance mit the necessary user

attitudes to determine performance achievements of its training goals. behaviors to be identified. their suitability for possessed by members of 4training, the intended handgun user populations.

YIWllandgun Training

Design Subsystem must

deo nstrate that It poe- assess the ability to

discern correctly the

discrepancies existing

between current and required levels of relevant

performance achievements

within the intended hand-

* gun Use populations.

Figure 19. Army Handgun Training Design Subsystem Performance Requirements

Basic Potentialities Needed: Hierarchy Number 1

_101-
·. 

.· 

-- .. ~-. 

... ·< 

I ' 

p0rft:rmancc rccp:i:rcments. The evaluator's proper frame of mind can be 

illustrated ns follnws: 11 1 don't care what it looks like, and I don't care why it 

]•wks that w;• y: · only w:.nt to know if it performs the way it is supposed to 

perform. 11 The .... J~tluator needs to have that point of view in order to insure 

that . fhc evaluation pror.css and its conclusions are 'lOt contaminated by 

b·re1(~vant consi rations 0f design. The evaluator cannot allow himself or 

herse1f to jump to lhc concluswn that a system is no good simply because it 11 looks" different from its predecessors. Neither can the evaluator 

autc-·rn·ltica1ly ass:1me th:;t the system is good simply because it embodies 

radically new d<':dgn concepts. The evaluator needs to preserve a strict 

di 11 ter~::~t in issues of design, 

: om~; body, however, does have to worry about what the system 

shot:'· l 11 look like. 11 Somebody has to build the system to satisfy its 

performance requirements. If it isn't built properly, it surely won't meet 

those requi:r(~r:1~nt:.:. And the proper building of anything has to start with 

the proper design. 

TF.e specification of a training system design is nothing more nor 

less tha~·· the plan for building the training system to satisfy its performance 

requirements. The design specification thus must evolve from those 

requircn~nts, i.e., from the same taxa of potentialities, processes, and 

product:. fror:1 which the evaluator extracts the measures of system performance 

and effc ~tiveness. The system designer loc,ks at each taxon and asks, "How 

can I Sf·e to it that the system will have all of these potentialities? How 

should I arrange for the system to carry out all of these required processes? 

How can T best in~;ure that it will deliver all of these required products? 11 Of 

course, the designer must come up with answers to these questions that are 

compatible with tbz~ taxa of Environment and Constraints under which the 

syste!11 must oper:.~tc. But the questions themselves are rooted in the taxa of 

performance. 

Our question is: How does the designer answer these questions? 

How does a design specification evolve from the system performance 

req t:irer:-:en ts? 

The first step in this process is identical to the first step in the , 

evaluation process: one examines each taxon to identify its associated systeq1 

attributes. Here again, an attribute is some concrete element of the system, · 

something th.:1t car· b~ seen, examined, submitted to analysis, etc, It is, 

the::eforc, sr:·nethin7, for which concrete plans may be formulated. 

Tl,e De;3L,;n attributes associated with a particular taxon are any and 

all features nf the ;ystem1s people, equipment, and procedures that may affect 

satisfaction ,,f the :)erformance requirement rapresented by that taxon, The 

features of ::"'!]evan;e, of course, may vary from one taxon to another. The 

relevant fealures cf people associated with one taxon, for example, might 

include thcil· vir;ud acuity, their physical strength, their height, their 

rcsist:mce tc rnot.im1 sickness, etc. That is, the nature of the particular 

performance ·equir<:ment wou1d be such that the system would need to have 

some operate ·s pos::;essing those physical attributes if the system is to be able 

to meet that requh:::mcnL Analysis of some other taxon might disclose that 

BEST AVAILAEJLE COPY 

-102-

... 

-

" 

I 
, ,.. . . .. . . . . . . .

'- - people are needed who possess a certain degree of knowledge about some

particular subject, who can apply certain mental skills, and who manifest

certain attitudes concerning particular topics. The nature of that particular performance requirement would demand that it be addressed by people who had

those specific intellectual and affective attributes.

The system designer's job thus begins with the identification of all

of the features of people, equipment, and procedures that relate to the

satisfaction of all of the performance requirements that the system-to-be-built will face. Then, the designer must prepare plans or statements showing

exactly how he or she will see to it that all of those features are included in

the system.

To illustrate, recall that one performance requirement that a training system designer faces is job analysis. That is an objectives-level taxon of

process. The designer knows that the plans for building any training system * must include a provision to "analyze the job for which training is to be

developed." The problem is that the form or method of job analysis that may work well for one training system may not be most appropriate or even

workable fir another training system. A designer who doesn't pause to

consider the attributes of the specific system to be built might simply select a

job analysis plan "off the shelf," dust it off and incorporate it into the system specification. That plan might place heavy emphasis on the conduct of

observations of actual performances of the job to be analyzed. That is

certainly a traditional and often effective approach to job analysis. But in the

case of a brand new job, never before performed, it may be totally iiappropriate to rely on observational methods of job analysis: there simply wouldn't be any job performers to observe. A study of the attributes of the

data available for the job analysis would disclose that other methods-would be

required, e.g., methods relying on review of the specifications for the newly developed job, interview with the designers of the new job, etc. Similarly, the training system designer would determine that the people who will be

assigned to perform the job analysis would need to be skilled in interviewing

techniques and "paper-and-pencil" analytic methods, rather than simply being

well trained observers.

Figure 20 presents a sample of the design specification that might be - written in response to the performance requirements of the Army Handgun Training System. The selected sample addresses the first hierarchy of

performance potentialities of the Design Subsystem, i.e., the objectives,

functional purposes, and characteristics taxa associated with the potential for

identifying training goals and priorities. Appendix B provides additional

samples of design specifications for the process and products aspects of the

AHTDS.

Preparation of a design specification such as that shown in Figure 20

"merely" is a matter of deciding how best to insure that all appropriate

41. features cf people, equipment, and procedures are built into the developing system. Of course, a good deal of design talent is needed to do "merely" that. Good system designers are paid well, and deserve to be. Deciding how

a set of diverse performance requirements can be met most effectively is never

1 -- 103-

:::~~~~~~~~.-..-.: ........-......-...... _....................+.....................:........... ............ :, .. ,:..:-: ,. : ," ,.... .: .,.,, .
Potentiality Performance Requirement Number 1:

The capability of identifying training goals and priorities.

The Army Handgun Training Design Subsystem Task Force

recognized that a solid capability for identifying its training goals and

establishing their relative importance must be brought to bear on the

training design effort. The Task Force has assembled a highly qualified

team, including representatives of all user services whose members

* collectively possess skills and experience in such relevant disciplines as

task analysis, operations research, human factors engineering, and

training program design. The members of the team will have access to all

relevant documentation on the handgun, and will have ample opportunity

to interview current and prospective users of the handgun, including

some Allied military personnel. The team will develop separate lists of

training goals and priorities for each service's intended user populations,

as w~ell as a master, all-services list. Names and relevant experience of

- the te-,m members are given in Section ** of this specification.

The Task Force has determined that the principal use of the

h-andgun will be as a sh~ort-term personal defensive weapon. Users who

will so employ the haadgun include front line of ficers and

* non- commissioned officers; operators of crew-served weapons; pilots;

driver-,; various equipment operators and technicians; and, certain rear

echelon personnel. A secondary use of the handgun will be as a law

enforcement offensive and defensive weapon, to be used by military

police. The charter of the task Force explicitly excludes the 'latter use

from the scope of the training design requirements. Accordingly,

member-s of the team will focus exclusively on the handgun's use as -i

personal defensive weapon. They will analyze a representative set of

close -contact- combat reports from World War II and the Korean and Viet

2 Nam conflicts to develop scenarios in which the handgun user tasks can

be identified. Appropriate methods of task analysis will be applied to

determine handgun user usage requirements in those scenarios, and to

identify the knowledge, skills, and affective characteristics needed to

satisfy those usage requirements. Using the Delphi and/or other suitable

rating techniques, the team members will determine the criticality of the

knowledge, skill, and attitude ingredients. This will be coupled with a

review of the intended user populations to assess the current status of

the knowledge, skills, and attitudes. The critical deficiencies thus

identified will serve as the basis for identifying the training goals. The

Figure 20. Sample Design Specifications for the AHTDS

(Potentialities Aspect)

[Appendix B provides sample design specifications, for the Processes and

Products Aspects of the AHTDS1

-104-

L A.- n!,_.. n.-
WA

~ I goals will be evaluated in terms of their relative importance; this step will

solicit input from representatives of the intended user populations as well

as from the team members individual and collective judgments. The team

members will apply their human factors and training development expertise

to insure that the goals express the quantifiable actions and conditions

required for effective use of the handgun.

Analysis of the knowledge, skills, and attitudes required for

effective use of the handgun for personal defense will proceed in

accordance with the information - decision -action attribute model, described

in detail in Section ** of this specification. Briefly, this model calls for

analysis of the sequential operations required of a task under study.

The operations are examined at steadily increasing levels of detail until all J human actions, movements, queries, decisions, etc., have been identified,

together with their stimuli and outputs. Increasingly detailed

formulations of the human knowledge and skill requirements emerge from

this analysis, and operator attributes and attitudinal characteristics can

be inferred from those requirements. The team will evaluate each

identified requirement to determine, first, the need for training to insure I that it will be satisfied by intended handgun users; second, the expected I amount and cost of training that would be needed to insure satisfaction of

the requirement; and, third, the practicality of providing that training. J Any knowledge, skill, attribute, or attitudinal requirement not deemed

I suitable for training will be established as prerequisites for candidate

handgun learners.

(1.1.2.1, 1.1.2.2, 1.1.2.3)

The team sill examine the required knowledge, skills, and attitudes

in the cnetof terepresentative scnro.The proewill beto

assess the achievement standards required for effective use of the

handgun as a personal defense weapon. Factors such as accuracy, A repeatability, duration, etc., of knowledge/skill performance will be

Vvaried to determine their impact on the scenarios' outcomes. Attitudinal

and attribute characteristics also will be varied and assessed. Minimum

standards of achievement then will be postulated. Representatives of the

intended user populations will be examined to determine their current

levels of achievement. These representatives will form a sample of users

stratified by characteristics relevant to handgun usage, e.g., such

*characteristics as the user's job assignment, anthropometric

characteristics, GS scores, and other relevant factors that may have emerged from the analysis. Findings will be compared with the identified

minimum achievement standards, to discern achievement discrepancies that _are to be addressed in training. This will produce o refinement of the

training needs analysis discussed above.

IFigure 20. (Continued)

-105-
" " I(1.1.3.1, 1.1.3.2)

The training requirements emerging from the team's analysis assessed

as to relative importance. Team members will produce individual,

independent ratings for the training needs identified for their own

services' intended users. These individual ratings will be submitted

II- anonymously, and will be reviewed, discussed, and critiqued in a

round-table format. That will be done first on an individual service basis

and finally collectively for all users/uses. Consensus sets of needs

importance ratings will emerge from this activity. Finally, the needs list

4 !will be reviewed and rated by the representative intended users and by

selected Allied current handgun users, for validation and refinement.

(1.1.4.1, 1.1.4.2)

Each selected training goal will be classified as to its relevant domain

of learning. The tripartite taxonomy of B.S. Bloom (1956) will be used

- for this purpose, i.e., cognitive-affective-psychomotor. Within each

do"-main, the goals will be expressed as observable actions which the

handgun learners will demonstrate under defined conditions.

I

Figure 20. (Concluded)

-106-

I-
easy, and may demand considerable levels of such difficult-to-instill qualities

as intuition, creativity, and insight. System design may well be more science

than art, but good system design is always the province of artful scientists.

The authors make no claim to artistry, nor to possession of the

*creative or insightful instincts of the training designer. The sample design

K- j specification given in Figure 20 undoubtedly is deficient in numerous respects.

* In one major respect, however, it is a shining example: it addresses system

performance requirements and nothing but performance requirements. Experts

in handgun training possibly could look at Figure 20 and uncover instanI-ces

where the offered proof of the ability to identify training goals is weak,

*inconclusive, or even incompetent. We submit, however, that those design

specifications do not overlook even one aspect of the constituents of that

ability, i.e., the specifications at least offer some proof for everything that

needs to be proven. Neither do the specifications deal with anything that is

not germain to the ability to identify training goals. Our design specifications

* are firmly rooted in the performance requirements that the Army Handgun

Training System really faces. The APM provided the capability of identifying

* those requirements. As such, it is a valuable tool for any training designer.

The sketch at the right outlines the General

context in which the specification of design Training Design

*plans for the Army Handgun Training System Performance

takes place. The total process begins with Requirements

the generic performance requirements of the

Design Subsystem of any training system, i.e., -the taxonomy given in Figure 7 above. The Design Requirenext step is the translation of those general ments specific to

performance taxa into terms keyed to the the Training

peculiarities of the tr'aining application Application of

of interest; in this case, Army Handgun interest

Training. Next comes the development of

4 

t-oncrete 

Handgun plans for insuring that the Army I ~ z

Training Design Subsystem will Specification

possess its specifically required of a Plan for

potentialities, carry out its particular satisfying the

processes, and deliver the unique products Design Requirerequired of it. Sample segments of such ments for the

concrete plans appear in this section of Training Applicathe report and in Appendix B. Once the tion of

plans are approved, they must be imple- interest

mented. In the context of our particular

example, that means that steps must be

taken to assemble the people, equipment, Implementation

-and procedures needed to develop effective of the Design

*hand-!un training for Army handgun users; Plan

t,) t-i,duct activitie-, ippropriate to that

deve-lop~ment; and finally, to deliver usnbie

*handgun training products. At the final Program of

stage of development, those products are Instruction for

collected into an effective Program of Ithe Training

Army Handgun Instruction that will enable Application of

soldiers to use their handguns to do the interest

jobs for which the weapons are intended.

-107-
-Y E. The Computer-Aided Model

I. General

The interactive computer system is being developed to aid users of

the APM understand and apply the structure of the training system design

model and to relate training goals and activities to appropriate learner

progress measures. The system is being illustrated through a set of

PASCAL-based programs operating on complex data bases using an Apple-I

desktop computer.

It was recognized early that the STM and its emcompassing APM are

somewhat toilsome to use. It was foreseen that a streamlining of the process,

a reduction in the magnitude of effort required, and greater ease in applying

" the model can be achieved if an interactive computer system is employed. This

research task was intended to determine 'he feasibility and resource

implications of incorporating parts of the APM (specifically, the STM) onto an

interactive computer system for machine-aided applications. The task included

the examination and step-by-step documentation of procedures followed in one

or more manual applications of the model. It then assessed how each step can

be programmed on an interactive computer, especially those steps in the model

application which necessitate the analyst's review of taxonomies. A sample

application of the model on a small interactive system was also begun in this

task.

2. Points of Departure for This Task

Since the APM is intended for use with any human-machine system,

and all system development or evaluation efforts have finite and limited

- resources of time, money, and personnel, it is necessary to find ways of

simplifying and hastening the analytic processes without undermining

confidence in the results. If found to be too time-consuming or costly, any

model risks being abandoned by potential users, regardless of how effective it

can be.

Consider the magnitude of the process (or algorithm) for creating a

gen-ral set of taxonomies and guidelines for applying the STM to any

particular human-machine system. There is no doubt that the total number of

populations that exist among the universe of human-machine systems is

- enormous. However, if one's principal interest is centered on a particular

sub-universe of systems (e.g., training systems for military weapons), it may

be possible to identify a finite, manageable set of populations whose members

span the range of variables pertinent to performance/effectiveness specification

of measurement of (at least) most of the systems belonging to that

sub-universe. In particular, it may be possible to organize that finite,

manageable set into subsets corresponding to the cells formed by interaction of

the first two dimensions of the STM (see Figure 2).* This would produce

*At present, tie model's third dimension (the System Hierarchical Structure)

will be implicitly represented, based on the extent to which other systems

which interact with the system of interest are covered by the automated

model's data base.

-108-

% N %

12f;.! -*N. *,c 
fifteen sets of system taxonomies, with each set corresponding to the

interaction of one of the three Descriptive Levels with one of the five

Contextual Aspects. The existence of these sets presumably would greatly

I facilitate the application of the STM to any particular human-machine system

belonging to the sub-universe of interest. To identify the total set of

* taxonomies for that system, one would search through each of the fifteen sets

in turn, "pulling out" the populations to which the system belongs.

Occasionally, one possibly would discover that the system also belongs to a

population not contained in the available sets. That population could be

added, so that the fifteen basic sets would "grow adaptively", incorporating

more of the taxonomies associated with more of the systems of their

sub-universe.

The compilation and use of taxonomy sets should be greatly

*facilitated by automated data processing. In one of the trial applications to

just one subsystem of the BIFV, the STM produced several hundred

taxa. The "finite, manageable set" of taxonomies for the class of all weapons

- systems easily could have several thousand or even tens of thousands of

entries. Manual searching and extracting from such voluminous sets would be

very laborious and error-prone. But if the sets were maintained in an

* interactive computer system, it might be possible to reduce the labor involved

significantly. Ultimately, supporting software could be written so that, given certain descriptive features of the system of interest as input, the automated

* routine might take a "first cut" at selecting at least the Objectives Level Taxa

(corresponding to the "what" of system performance). Of course, the human

analyst could always amend those initial choices by deleting or adding taxa,

* but at least his or her search time should be shortened substantially.

- Furthermore, the computer-based implementation eventually could include a

permanent memory of all of its past taxonomization applications. This could

allow keeping track, for example, of the frequency with which each population

in each lexicon has been selected and of the groupings of populations that

have occurred in past applications. The sequence in which populations are

presented to the human analyst for his or her consideration could be

continually refined based upon the observed frequencies and groupings of

previous selections. Given enough previous applications, the software could

j become progressively "smarter," and become capable of making steadily more

accurate initial selections of taxonomies based on what it has "learned" in the

past. Although it may never be possible to dispense entirely with the human

analyst in the final selection of taxonomies, it should be possible to eliminate

much of the work that presently is required. It should also be relatively easy

to design the memory to permit new entries to be made, and even to delete

* old, never-chosen entries as time goes on.

The PASCAL programming language was selected for this sample

application because a form of PASCAL is expected to be available in about 1984

-for use at remote terminals throughout the Department of Defense. Various

versions of PASCAL are now available for use on such computers as the Apple

11 and the DEC-20, both of which were potentially available for this task.

Except for the newly announced IBM Personal Computer, no PASCAL compilers Ihave been developed for other 113M computers to date, as far as can be

-109-
determined. Recently published manuals on Standard PASCAL, UCSD*

PASCAL, Apple PASCAL and PASCAL-20 have been obtained for reference in

this task. Although a desktop computer like the Apple II would not have the

capacity to handle the model's ultimate information content, it was chosen as

.." useful and convenient in this early demonstration stage to execute a sample

program which can later be expanded and transferred to a larger capacity

machine.

3. Selection of the Sample Application to be Programmed

t The BIFV system has been analyzed (manually) more than any other

with this STM/APM model, and it also is one in which major practical interest

lies. Specifically, the most advanced application of the model to date is with

the BIFV Training System, an important entity for which a general set and a

particular subset of taxonomies have been identified under the first several

model components. Because of this experience in applying the model and the I project team's existing knowledge of the BIFVTS, the basic algorithm, guidelines and data sets are fairly well understood and were adaptable to a

sample computer application with less effort than if one were to select a system

not previously examined.

In order to keep the scope of this task within project resource

limits, and since this was to be a sample application only, just one subsystem

(Design) of the BIFVTS was considered for the main computer implementation

(see Figure 5 earlier in this report). Furthermore, this implementation was

"" Icarried down through only one set of task requirements associated with

Curriculum Development, namely, Specification of Learner Testing. Other sets

m of tasks within Curriculum Development are:

o Specification of Training Objectives

0 Specification of Training Content

o Specification of Training Methods

O Specification of Texts, Lesson Plans and Reference Sources

o Specification of Instructor Requirements

o Specification of Facilities Requirements

o Specification of Instructional Schedules

0 Secification of Training Standards

o Etc.

By means of the narrowly defined example, this task attempted to illustrate the

feasibility and value of guided general conceptualizations for helping to

evaluate any system, the use of pre-established taxonomies at general and

specific levels, thc improvement of data bases in adaptive computer-aided

processes, and the ability of the process to yield specific implications for

system definition or evaluation measurement (if not some recommended

specifications or effectiveness measures themselves).

*UCSD: University of California at San Diego.

-110-

.............................

................................
4. Implementation Concepts

The key notion of branching down through sets, to subsets, and to

sub-subsets is evident from the first steps in preparing for STM application.

Figure 21 illustrates how the focus of evaluation and measurement narrows from

all human-machine systems, to all training systems, to the BIFV Training

System, to the BIFVTS curriculum development subsystem, and finally to the

specification of learner testing within that subsystem.

The STM/APM components have a similar nesting structure, such as

illustrated in Figure 7. In that Figure, taxonomies for a general design

for Curriculum Development subsystem are listed for the first three contextual

aspects (Performance Potentialities, Performance Proceses, and Performance

Products) at all three levels of system description (Objectives, Functional

*Purposes, and Characteristics). It is seen, by referring to the numbering

scheme, that the sets of taxonomies at the higher levels branch down to nested

subsets at the lower levels. The taxonomies for the "Specification of Learner

Testing" are a subset of those in Figure 7. In operation with a computer, the

analyst could be shown a display of the general Design taxonomies, and could

identify for the computer those which apply to the area of specific interest

(i.e., Specification of Learner Testing in the particular training system being

considered). The computer could then guide the analyst through subsequent

"menus" and selections until all the STMI components have been applied at all

system levels, and implications for evaluation measurement or system

specification can be derived. Similar programming of the remaining components

of the overall APM (beyond the STM portion) could lead to identification of

*actual measures or performance requirements, and eventually, perhaps, to

recommendations for applying those identified measures or requirements.

5. Computer Programming Implications

* Ultimately, a computer-aided realization of the entire analytic process

is a major design, programming, and clerical task. The sample illustration of

the computer-aided analytic system, developed using PASCAL on a small

computer, contains only a small fraction of the information intended for the full

*system. More critically, although the demonstration appears to have the

capabilities of the full system, many of the capabilities are simulated rather

than developed as in the full system. (For example, searches may be done by

simple linear procedures that would be too inefficient for use in the full

system.)

In the development of the illustration, however, design concepts for

the full system were considered. Only after the major design decisions for the

full system h,-ve been made can a realistic subset be extracted for use as an

illustration. In this development, the user-computer interactions and the

user's abilities and experience are key concerns. As interactive computers

have become more common in recent years, techniques have been developed to

enable more efficient use of computer facilities by less expert users.

Particular concern is being paid to relatively naive users--ones who seldom use

47
computers or who are unfamiliar with the current system. Techniques such as

"1menu selection," extensive "help" facilities, and interactive error correction

procedures all make it easier for naive users to conduct work successfully on

the computers without the need to reference written support documents or to

*call upon system experts for aid. The best of thase procedures are sought

and used as appropriate in the design of the user-machine interface for this

task. For example, split-screen formats are considered to display "menu" data

and user guidelines or instructions at the same time.

The data structure--the taxonomic information is being organized to

reflect the complex interrelationships to be exploited by the users. The basic

elemients, or- nodes in the structure, will be made up of key pieces of

information for the user. But relating the nodes to each other is the key to

having a powerful and useful data base. Multiple kinds of links were explored

and are implemented as appropriate. For example, the overall data base has a

tree structure like that partially shown in Figure 21, with successive nesting

of subsets to more and more specific detail. But, to be useful in an

interactive environment, the data base must have other ways of being scanned.

For example, one must be able to access all "related" taxa of, say, performance processes. And "related," to be meaningful, may require each

node or cluster of nodes to be judged on several con tent- descript iv e

dimensions.

To access the data structure, the user must be able to ask powerful

questions, to use general queries to expand understanding, and to use

intermediate answers to guide subsequent questions. This query system,

dependent upon the data structure for its detail, must be a helpful and

productive tool for the user.

Obviously, the more features present in the full computer,' system,

the more flexible, powerful, arnd ultimately valuable to the user it can be. To

program such features and to input a data base of the required complexity can

be extremely costly, however, and a middle ground must be chosen such that

tlfc end product is both useful and cost-effective. At this point in the

project, it is intended that the system initially be designed with as many

features as possible. Designing is relatively inexpensive. With an

overdesigned starting place, one can more knowledgeably develop the actual

implementation. That is, one can compromise more effectively knowing the

costs and benefits of specific capabilities than by starting with a reduced

design in which the "extra" features have never been specified. Moreover, it

may be that some desired features initially judged too cumbersome can

eventually be put into the system, as costs of some operations are reduced or

as more efficient procedures are devised. Such additions will be more easily

implemented if the original design included them than if they were never

contemplated in the initial work.

The outline in Figure 22 is an ordered list of items relevant to the

design of a complete computer system. Included are features for the system,

possible capabilities of the system, key questions about the user population
All Sys

I

Others: Atomic, Planetary, Animal

i1 Human-Machine Geological, Weather, etc.

Systems

Others: Communication, TransportaAll Training tion, Weapon, Maintenance,

Systems Health and Welfare, Supply,

Production, Navigation, etc.

I I ________ _ Others: University, Parent-Child,

BIFVTraining Auto-Driving School, Radio

System Operator, 9mm Handgun,

I. etc.

< ~ ~ -- ~-i I

S Design Others: Command, Enabling, Emj..Subsystem placement Logistics, Delivery * __I__

Others: Training Objectives! Training

I Specification of i Content, Training Methods,

Learner Testing Information Sources, InstrucI< tor Requirements, Facilities

(Begin Detailed Application of Requirements, Instructional

* the STM/APM) Schedules, Training Standards,

etc.

_ Performancel Performance Performance Environment

Potentialities Processes Products

Objectivei Level Obje ctives5 Level Objectives..

-Functional Pur - Functiona .

poses Level

:-Characteristics Level

Figure 21. Branching Down Through Nested Sets as a

Basic Concept in Applying the STM

-113-

........... •... .
-'ItL Users: "tng of kinds of users, subgroups described in terms of..

A. Rank or position

IL bItlligene

C. Content as experieneftraining

D. Computer oeperiiic* /taining

L. Purposes in sini, the system

F. Skperienee with tis system-Omount. reeeney

A. Dots low

-AA. Magnitude of the data bese

2. now much text

3. Now many nodes (eag.. taxs, pefomnce mpwut. measuemnent beums,

2. now many node descriptors MOE&/~tOPst

4. Now much total storage required

B. ructure. *4g.,

1. Treelike hierarchy

a. tUp and down links

b, Jump to parallel branches

2. Overall structural dimensionls. eg.. matrix dimensions:

a. .Descriptive levels

b. Contextual aspects

e. System Families (AWrchicall 3tuctwel)

2. Node or subtturte Qualifiers

a. Attributes. relevne Will VOlWS or value Tangies

b. Ordered liks (eag.. A souses 3. 3 precedes C)

e. *miiarity links (e*g.. A ilke B. (Ai~1 ist keo 

t

d. Co-ccurrence links 44e7gtomes lip whene-ver 31 is

merntioned, or A ever exists alongside B)

a. intensity of links eg. weak links pawsed over i general

searehes but followed i hi-depth searches

A. Node eantents

a. Same or differpent I differet parts of structure

b. Range of types of node eanitents. if mere than sone

IL Necessary type/attribute ifentifiers

- BS. Pomibilty c, multiple elaedlfieatiofl ltrtaetwe) Wisaes

M. Quer7 honguqe/user interface

A. General design features e..

2. Menus for users to search to make branching ehoce.

S. Interrupt with stack memory--whether users can iterruapt to

performamsaw task. then resume the prior task

3. 'IVp-4 general Panic button to have the system provide

extra Information to "el the laew find out where hes is and

warit Ie ean do

4. Rtestoretback up Viae *Wily, In the middl of a @omplex

search, to back vp wie or more decision points and resume In archdif~ti~nwithul WtW hvin t art anew

S. Trees printout-the ability to order. at any point. a pemann

spy of somne or All of tae previous march results

I. ITatoria subprogram, to train naive sers of the system on whot the capabilities are and how to interact

I. Veer eomments-e facility by which wers eon comment on

.1 usything related to he system I massages to the system

mitainers-with eventual feedback to specific mrs

3. Nature of interactions

I. Fized-format ser inputs vs. relatively free-format

3. Command defaWu to allow straightforward actions fW orlave

Wm plus powerful abiities for sophisticated votrs

3. Command prompting to help usrs sompoa gommands

4. lalishtike eommands

6. Forliving of minor armors it eenstroctling gommanida

L Quick system response time

C. 

- Iae Powes

I. Rayred to WweWo of sets bas

3L *DquntWa or parallel wreenlng

8. Allow bgiealy amples mielnm roke

Figure 22. Items Relevant to the Design of the

Coinputer-aided System

-114-
-~~~ ..- r2' .- - 7- 77 

and the system uses, and other related issues. The list is an initial gathering

~ J of items considered in the design, rather than an outline of the system or a

set of steps to follow in the design process.

Other aspects of the system, particularly record-keeping and

dynamic modifications, represent a secondary level of design which need only

be briefly discussed here. The system must keep track of who has used it

and for how long, and there may be value to recording some session details as

well. The system will permit modification, of course, but an important

consideration is the source of such changes. The system designers can

enlarge or modify the data base and change the interactive capabilities of the

query system at will. The direction for such changes may come from outside,

i.e., from content or system experts who have determined that a system

expansion is in order. Changes may be originated by user comments--ones

which report errors, awkwardness, or weaknesses in system performance.

Finally, changes may occur because of the nature of activities on the system.

If users always make one choice at a decision point, for example, the overt

decision may be eliminated. If some choices or associations are always

rejected, the data may be restructured to eliminate the possibility of those

links. In these last examples, because the information needed for system

change is resident in the system activity, it is theoretically possible to

program the system to monitor such activity and modify itself.

6. Specific- Demonstration of Interactive Computer Capabilities

Design goals for the ultimate computer system include power and

user-friendliness. While the material to be make available to users through

- the computer system could be aggregated into printed volumes, the interactive

approach has several potential advantages. Paramount is the opportunity for

* the user to only investigate the sections of the full STM relevant to -a specific

task and to be able to study--under the guidance of the system--all the trelevant components. For a computer system which is well designe7 and

implemented, users will come back to it frequently and benefit from the

information available. The simplified program to demonstrate the kinds of

capabilities attainable with a comp uter- aided version of the STM actually

possesses far less than the full range of capabilities ultimately envisioned.

However, the program and data bases are designed to clarify the full range of

capabilities, through a combination of simulation and inference. The STM

demonstration program is a partially working model: virtually all of the

options, functions, data bases, etc., that ultimately will be available to a user

of a full-fledged automated STM appear to be present in the demonstration

version, but many of these would o actually "work" if the user were to

attempt to select, implement, or access them.

Using this approach, the computer system presently includes four

components briefly noted below:

14

It
0 General Design Subsystem database -a tree-like hierarchy of

curriculum development concepts--goals, processes, and

p rod ucts--cap able of being divided into subsets based on

content as well as hierarchy factors. Each "node" in the tree

includes limb information to related nodes in the tree,

descriptive information, and links to measures useful in

determining whether, in an actual training situation, the node's

goals have been met.

0 Measures database - a structure of kinds and example-, of

measures for training situations to scale how well certain

training activities have met their goals. The measures are

ordered in a simple structure of more general and more specific

examples and, in this instance, are linked to specific points in

the content database.

* t 0 Programs to allow system maintainers to conveniently add to or

modify the data bases.

0 Programs for system users to selectively study the portions of

the databases relevant to their own activities.

The system is intended for sophisticated and for naive users.

Typically, user activity choices are displayed on the screen whenever user

inputs are required. Abbreviations are permissible, so that users can type as

much or as little as they need for their own confidence. A wide raiage of help

is available at nearly every point in a session, and repeated errors lead to

attempts by the system to help the user out of his dilemma. Much of the

user's activity involves tracing paths through the databases to identify helpful

content nodes and associated measures. For any session, the system retains

the search route !-o that users can retrace their steps as desired. The system

also allows information to be printed at nearly any point so that permanent

records are available after the sessions.

Figure 23 compares the ultimate capabilities desired of a full-scale

* automated STM with the representations of those capabilities in the

demonstration program. Appendix C presents details on the program, in the

form of a flow chart with explanatory narrative. The demonstration based on

the concepts of Figure 23 and Appendix C are considered to convey adequately

the' ultimate power of an automated STM, without having required a severe

drain on pr,,ject res~ources.

Lurrently, the initial formulation of the programs and initial

exemplary databases are being completed on the Apple 11. More detailed

design goals and descriptions of the computer system are presented in

* Appendix C.
-- -- W I

THE FULL-4CALE. ULTIMATE

VERSION OF THE AUTOMATED THE DEHONSTRATION VERSION OF

11M SHOULD: THE AUTOPIATED TI DOES:

Contain complete taxonomies for a Contain taxonomies only for two sub- wde variety of system types. Each systems of a generalized training sys- taxonomy, for each type of system. tem. Those two subsystems will be should encompass all of the system's curriculum development and nstrucor

subsystems, and should cover poten- preparation. Further, no tax& of tialities. processes, products, environment or general constraints

environments, and eonstraints on will be included. The demonstration the objectives, funetional purposes, program will convey the impression

an cardeter terisies levels of system that other types of systems are description. available in the data base, but in reality none will be seieet&ile.

Provide two different, but related. hnply that both applications are elasses of application. viz., 1) the available, but in reality software will selection of measures of perfor- be prepared only to support measures

mance and effectiveness, and selection.

.) the development of system .dsign ppecifications.

. Provide measuresielevant taxa for Provide measures-relevant tax& for no

a wide variety of aspects of per- more than two aspects of eurriculum' formonee for any selected system. development performance (probably

In particular. If the wer has idi- Nesting specification" and "earning sated interest in 0training systems." objectives specification") and for onlv

thuser lould be able to specify one aspect of instructor preparation

any of a large number of aspects performance (probably "conduct of of training system performance for testing"). Other aspects of perfor- which particular subsets of tWxe manee wW appear to have been ki"-

have been Identified In the data tified in the data bass. bu their iixa base. subets will not really have' been

defined. Note, however, software will

be prepared that will permit a user t

define a tax& subset, on-line, for a

Newly specified apeet of performance

Provide a somprehensive listing of Provide selected, exemplary measurethe isues and implications for ment isues--strictly relevant tO "testmeasurement, relevant to a spee- Ig specification" performance-and fed Mset of perfomane, tIt only fo? the objectives level and fwn,- derive from each member of the tional purposes level taxa associated

taxe subset associated with that with a euirriculum development subaspect of performance. system's performance of "testing ape- eifieation." Note, however, software

will be pepared that will permit a

user to define measurement issue aInformation, on-ine, for any specified - taxon.

Provide sidtable measur or meas- Provide measures for the Issues raised mes to address every iSue and by only a slet sample of the tax&

implication raised by every taxon associated with 'testing speiofieation." amociated with a speWef aspect Note. however, the ability of the ser

of performance. to define measures, n-lne, will be

0demonstrated.

Figure 23. Comparison between Demonstrated and Ultimate

Capabillties of a Computer-aided STM

%-117-
IV. APM APPLICATION PROCEDURES

The two general applications of the Analysis Process Model are to the

-. design of human-machine systems and to the measurement of the performance

and effectiveuess of those systems. The foregoing discussions and examples of

those applications in the training system context should have made one

fundamental point abundantly clear: system design and system measurement

deal with exactly the same issues. Those issues are the constituent

requirements of system performance. The designer and the evaluator are (or

shca ld be) interested strictly in the capabilities (potential performance) their

system ir supposed to have; the activities (process performance) it is

supposed to carry out; and, the goods and services (product performance) it - is supposed to deliver. The designer's job is to see to it that the system does

in fact meet all of its requirements for potential, process, and product

performance. The evaluator's job is to check on how well the designer's job is

done.

Sometimes a system designer may achieve somewhat more than was

required: the system might not only deliver all required products, for

example, but also some additional useful goods or services that weren't among

those essential to meet the needs of the system's sponsor. Designers expect

those kind of achievements to be applauded and rewarded. Evaluators,

however, have to take a long, skeptical look at such design "extras." The

evaluator has to ask, how much is this additional item going to cost the

sponsor? Could any economies in time, equipment, funds, or other resources

be realized if we dispensed with this "extra"? Most importantly, does the

- delivery of this "extra" in any w impair the system's ability to have

*." cvailable its essential capabilities, carry out its essential activities, or deliver

its essential goods and services? Depending on how these kinds of -questions

are answered, the designer's expected applause might materialize as boos.

The evaluator might report that, far from deserving additional credit, the

* "extra" good or service might really be a system defect, that absolutely must

be removed. The evaluator is the designer's advocatus diaboli: the one who

has to insist on rigorous proof that the designer's work has met all applicable

requirements. "Extras" are nice, but only after all of the essentials are met.*

Sometimes, too, the performance-oriented shoe is on the other foot. It is

far too common that evaluators find fault with a system designer's work that

really satisfies the performance requirements very well. Such invalid

objections arise fromr, the evaluators' preconceived ideas about how the system

should achieve its requirements, instead of from the basic question of whether

the system performs as it should. A case from the early days of World War It

is illustrative:

TDesign "et rasnr that truly do not impair achievement of real performance

requirements in any way probably are as rare as the free lunch.

,':: -118-

........ ......................................................
In order to maintain her essential supply of materials

from abroad, Britain equipped much of her merchant

fleet with anti-aircraft guns. Those guns were in

short supply, and were much in demand by the military

for installation in shore batteries and on men-of-war.

The Royal Navy conducted an "evaluation" of the

effectiveness of the merchantmen's guns which showed

a very small number of German aircraft had been

brought down by the merchant fleet. The ratio

of a-a gun shots to aircraft "kills" was vanish-

* ingly small. On the strength of that study, the

Navy argued for the removal of guns from the

merchant fleet, so that the weapons could be

employed maire "effectively" by the "professionals."

That argument nearly carried the day, until someone

pointed out that the guns weren't there to kill

German aircraft, but rather to keep British ships

afloat. In fact, the a-a guns were highly effective, in that a much smaller percentage of armed

merchantmen were sunk. The sailors used their

firepower to keep the aircraft at bay, and that

was all that they had to do to achieve their

performance requirement.*

The designer, then, must stand ready to defend the system and its

specifications strictly on the issue of its performance. The designer has the right, and the obligation, to insist that any criticism of his or her work be

shown to stem from a real performance deficiency. Otherwise, good work may

be needlessly corrupted.

System design and system measurement thus really are two different

aspects or views of precisely the same thing. They are more closely related

than are two sides of the same coin, for they interact constantly. At the

outset of a system development project, a designer takes a set of performance

requirements, identifies the attributes of people, procedures, and equipment

" needed to meet those requirements, and develops a preliminary plan for

securing those attributes in the system to be built. The designer's work at

this conceptual stage is oriented heavily toward the performance poetaiis

The plan of course addresses how the processes will be carried ouand what

the products will look like. But the plan's basic purpose is to demonstrate

that the system will have what it needs (in terms of abilities) to carry out the

processes and deliver the products.

*Given that one of our sample applications of the APM is to Army handgun

training, it is relevant to note that a soldier's handg,- .3 very closely

analogous to the merchant ship's a-a gun. In each case, the basic purpose is :<i. to defend the user, not necessarily to kill the enemy. Handgun training thus

*logically should empi'ize defensive operations. Unfortunately, the training

often boils down to target practice.

-119-

%................ . . . . . .
Meanwhile, the evaluator examines those same performance requirements

and their associated system attributes, and forms a set of measures to be used

to determine whether the system has those attributes and meets those

requirements. The evaluator applies those measures to the designer's plan.

The evaluator's interest, too, is at this time principally on the requirements

arid attributes of potentiality. Will the system have what it takes to carry out

" the processes and deliver the products actually required? Is the plan sound

and workable? This is the proposal stage of design and evaluation. The

designer says "here is how I intend to go about building the system to meet

its performance requirements." The evaluator responds "here are the specific

performance dtfects that I have found in your plan." The designer should

have a chance to rebut, to challenge whether the evaluator's conclusions truly

are based on the performance requirements. Once the evaluative conclusions

are validated, the designer modifies the plan to remove the performance

defects, and begins to build the "real" system. This means that attention

shifts from potential ("on paper") performance to actual process performance.

As the designer implements the processes the evaluator again applies the

measures to determine whether the requirements are being met. At this stage,

A| the evaluator might well uncover some defects in potentiality that escaped

notice during the proposal evaluation. To the extent that some early products

begin to emerge, the evaluator might also find some product defects. But the

focus here is on process performance. Is the system conducting the activities

that should be conducted? Is the plan being followed? Are the processes

being adapted to meet the needs emerging in the real world of system

development? The evaluator continually monitors the designer's work at this

stage, from time to time reporting "what you are doing there isn't quite what

is needed." Again, the designer may challenge the evaluator's observations,

to insure that they address real performance defects. Upon conceding that

the defects are real, the designer modifies the processes (and takes steps to

acquire any missing potentialities needed to implement the processes properly)

to correct those defects. Work continues, progress is made, always under the

watchful eyes of the evaluator. At some point, the system begins to deliver

its products. Once again, the evaluator applies the measures. Here, too, it

is possible that previously unnoticed defects in potential or process

* performance may surface. The accent, however, is now on how well the

- delivered goods and services meet their requirements for acceptance. The

' designer says "here are the goods you wanted." The evaluator might reply

, "these goods aren't good enough," and state exactly how they are deficient.

Provided that the evaluator's dissatisfaction is based on valid product

performance requirements, it will be incumbent upon the designer to "make

good" the goods. The designer might be able to do that simply by repeating

the production process somewhat more carefully. However, it might also be

necessary to re-design the processes, or even to re-design the system's

potentialities, in order to produce the proper products.

,. a What we have sketched on the previous few pages of course is a general

outline of an interactive design/measurement process for human-machine

systems: in other words, a general procedure for applying the APM. It

should be evident that design and measurement are indispensible activities

within this Analy.,4s Process, Further, they are living activities, born with

a

-..- -120- ;

... . .. ... , ..'. ,: : -. ., .. -:,.. . .
I .. ... . .. !! - .rf .- . .. . -, . ... . • - - " .. .

the system concept, growing with it throughout its construction, and remaining

with it throughout the system's active life. Every design step demands a

corresponding measurement activity. Every measurement finding demands a

design response.*

Figure 24 depicts the general procedures for APM application in the

Training Systems context. As shown, the responsibility for designing and

measuring a training system is shared by the suprasystem sponsoring the

development and operation of the training system; by the Command Subsystem;

and by each of the other training subsystems. Training system design

activities take place at two levels: at each of the operating subsystems, and

at the total training system level; Command performs the design function

commonly known as "system integration" in support of total system design.

Training system evaluation activities take place at all three levels: each

subsystem informally measures its own performance in designing its approach

to its performance requirements, and in meeting those requirements; Command L

formally evaluates each of the individual subsystem's performance, and

informally evaluates total training system effectiveness; the sponsoring

suprasystem formally evaluates the effectiveness of the total training system.

In carrying out these activities, all three levels participate in applying the

APM.

Details on the APM application procedures are given below, in reference

to each functional block in Figure 24.

Block 1: Sponsor identifies the basic needs for which training is to be

developed and delivered.

Following the definition used in Interservice Procedures for

Instructionzil Systems Development (Branson et al., 1975), 'a training

need is "a measurable discrepancy between te-actual world as it

exists now and the world as it ought to be." Such discrepancy of

course is perceived as a dearth of people qualified to perform some

job that the sponsor wants performed. This could arise because the

job is something brand new: for example, BIFVs are going to be

deployed and there is no one who has ever manned a BIFV.

Alternatively, it could arise because the sponsor is unsatisfied with

the way in which an existing job is being performed: for example,

combat studies may show that soldiers are woefully unsuccessful in

defending themselves with their handguns. A discrepancy could also

stem from a combination of these circumstances: for example, the

Army may be about to introduce a new handgun to replace its

current .45 and.38 calibre models, and may be dissatisfied with the

way in which soldiers were trained to use the older weapons. This

could lead to a decision like the following: "As long as we're going

*In many cases, the appropriate design response to a given measurement

finding is "don't cf. inge anything!"

-121-

'$ " - , " " . - " € "r: , : ''€ - ' "- - .* " ' - - - - " ' ' - " - " - " , "" . . - - - -" '.%%
V1

OTHER TRAINING

.. SUBSYSTEM

TRAINING SPONSOR TRAIDIG COMMAND Wt. Design

|Supra System) SUDYSTEM Enabling. etc.)

(j) Identifies the Basic (. Triigi ob the General Training

Developed and System Performance

Delivered Requirements to the

- Identified Need.

-* Selcts age of Translates General PerI T=aSstem formAnce Requirement.

Effectiveness vent to the Identified Role- |Requirement into Needs- -Specific Identifies Attributes Sabsystem Relevt 0 Needs to the Needsi_ SltsMaU'lOiSpecfc Perform -

g- Selects Measures of ( neRqieet

Subsystem Perform-

[ance Relevant to the - - Needs-*ecilfc Require- Develops PrelimiayPlan for (i

Securing the Needed

_T Evlute Subsystem Attributes SEvaluatse Subsyste m

IPlans and Identifes

[Performance

Deficiencies Modifie Plan to

Evaluates Initial Correct Deficiencies

1-' Design Spo1cAfiction Assembles Subsystem 1

and Identifies |Plans into Initial TrainDefciencies in ling System

Training System

Effectiveness_____________

Translates Identmed @

System EWfectiveness Reises Plan to

Deficinc ies into Sub- Correct subsystem

System Performance Performance Def- Deficiencies Iclenaies

"' I Implements Sub- )

Evaluates Suzbsystem Ime* Proceses In

" 'tI performance lDelclenciels ]plan

Cen tnued

eNow Pae*

Figure 24. General Procedures for Applying the APM

to Design and Measure Training Systems

-122-

*. . \. *
From

Previous

Page

_________________Revisea Subsy stem )

Controls Integration of 17OCesses to Correc

Subsystem Processes rformanc Delinto Total Training

System Performance

i1 Evaluates System Process Implementation

Processes and ~Identifies Defi- " €ciencioes in

Training System

jEffectiv es

Translates Identified

System Effectiveness

Deficiencies into

Subsystem Performbance Deficiencies

b e .zn "________ ______yte

Products and Identifies Products i. Accord

Conrol D InternfRse Subsystem 

ste P n Deficiencies

rotelier Traisngsyste

EvlaeiSbytm Pout Performance 

: ___ __aA_ and Id _d e Pter ldentfles DsfteeelesI CotrlsInegatonofRevise systnem Evlats TraTnn System, Sysetm ,f Ttin Pefrmne rdut

inTraining Syem RequiremePltn

E~f e c t

f

v ene ' ' Prcse n

Figurn ral de ifod ApProduct$ As

-', System Effectiveness , Needed to Correct

M a Defieenn SysteI S r s into ub- siigystem Perfr- m Perftormace g~gtDai~ce

+- .l ' lDeficiencies Fcgce

Fs -' -ction of Sub- (

sytm PerformleRequirments :' J .tifacflo @ o! S a tisfactiol n of T r

a

J i

ing

Saifato of System Effectiveness

p.'" lTrainn Nel [Requirements

p.+,5t" Figure 24. General Procedures for Applying the APM

I:I to Design and Measure Training Systems (Cont'd)

-123-

~4.

• :" "+ :" ? :-.' -' " :( S. : " ' : -. -. -... v ----. " .. 'Y ."... .. .+. :.. -. -(') - :(
to have to develop new handgun training anyway, let's do it right

this time arid try to teach solders to use the weapon as it actually I needs to be used in combat."

* It is up to the sponsor to define the training needs, and thus to

*establish the scope of APM application. These needs usually aren't,

and need not be, expressed in behavioral terms: extraction of

required behaviors from the identified needs is a subsequent

responsibility of the Training Design subsystem. But the expressed

needs must totally encompass the job that the sponsor wants

performed, so that the training system designers can do their work

in the appropriate ballpark. Along with the needs, it is also up to

the sponsor to provide or point the way to all available information

concerning the job) to be trained.

Block 2: Command assesses the relevance of the general training system

performance requirements to the identified training needs.

The general performance requirements for the Training Design * - subsystem were given in Figure 7, while Figure 8 presented the

general requirements for the Training Enabling Subsystem. Similar

general performance requirements exist for the Emplacement,

Logistics, and Delivery Subsystems as well as for the Command 1Subsystem itself. Depending 'upon the particular training needs to

be satisfied, it might be that some of those general performance taxa

do not apply to the Training System in question. The sponsor

might, for example, have already performed a job analysis with

which the sponsor is satisfied (in effect usurping a portion of the

performance requirements and responsibilities of the Design

Subsystem). As another example, if the job to be trained is truly

* bran(-1 new and radically different from all previous jobs, the Design

* Subsystem wouldn't need to analyze existing instruction for that job

(none would exist). Thus, one of the initial responsibilities of the

Command Subsystem would be to determine exactly which of the

general training system performance requirements do apply to the

:~ ~ application at hand.

Block 3: Sponsor selects measures of training system effectiveness

relevant to the identified needs.

The sponsoring suprasystem, is interested in "bottom line" results:

it needs to determine, ultimately, whether its identified training

needs are met. From the sponsor's point of view, the training

system will be effective if and only if it produces the qualified

people needled to do the job that the sponsor wants done. The

measures of training system effectiveness thus are job performance

measures. The sponsor needs to know: will the learners who

emerge from the training system be able to perform the job the way

we want it to be performed? For example, will the soldiers who

* receive handgun training actually be able to defend themselves

-124-
successfully in close-quarters combat with enemy soldiers? Will the

soldiers who are trained to be BIFV gunners actually be able to

operate the weapons to engage and destroy enemy targets? If these

kinds of questions are not answered affirmatively, the sponsor will

conclude that the training system is ineffective, because it fails to

satisfy the sponsor's training needs. It will not impress the sponsor

if the training administrator were to argue that "Okay, so our

learners can't really perform your job; but you have to admit that

we built some really superb classrooms and prepared some very

attractively packaged lesson plans, and our cafeteria turned out

some very tasty meals. Doesn't that count for something?" In fact,

it counts for just about nothing: about as much as it counts to a

bereaved widow to be informcd that her late husband's operation was

"technically, a success."1

N The sponsor will derive direct measures of training system

effectiveness by applying the APM to the opprating system for which

training is to be developed. For example, !eAP iio be applied

to the BIFV Carrier Team Subsystem to determine its performance

requirements, from which in turn gunner performance requirements

and measures of gunner performance would be derived. However,

the spon sor will also need indirect measures of training system

effectiveness, i.e., measures that will disclose whether the planning

and development of the training system is on the right track. Those

indirect measures will be derived from the training system - performance requirements submitted for review and approval by the

Command Subsystem. That derivation is depicted in Figure 24 as

the dashe~d line connecting blocks 2 and 3.

Block 4: Command translates the relevant generic training system

performance requirements into expressions specific to

the identified training needs.

In the trial application to the U.S. Army Handgun Training System,

the authors learned that the general Design Subsystem performance

taxa (as presented in Figure 7) did not lend themselves as readily to

handgun training specifications as one would wish. The problem was

not that the taxa weren't relevant to handgun training. Rather, the

general expressions simply weren't tailored to the nuances of that

*particular training enterprise: they didn't explicitly impart a

"handgun flavor" to the performance requirements. It was

determinedl, therefore, that an intermediate step was needed to

bridge the gap between generic performance requirements and a

,vstem-specific plan for meeting those requirements. Th

intermediate step was to translate the general requirements into

expressio:1s tailored to the particular system of interest (in our

case, Army 9mm Handgun Training). Samples of the "tailored" taxa

were given in Figure 19, above, and in Appendix B. One

* I comparison is reproduced here:
Generic D~esign Corresponding Handgun

Subsystem taxon Training taxon

"The Subsystem must be "Given a list of Army handgun

able to define the total users and uses, the Design Subscope of learning." system must be able to define the

(Taxon 1.1.1) scope of learning needed for each

user/use in terms of the

* knowledge, skills, and attitudes

required."

This translation produced much additional verbiage, but it was

helpful verbiage: specific details to which design specifications

could easily be written. Hence, this "tailoring" of generic taxa to

specific training needs is felt to be an essential step in the APM

Application Procedures.

Block 5: Subsystem identifies its attributes relevant to its needsspecific performance requirements.

Upon receiving (from Command) its own, tailored set of performance

requirements, each training subsystem takes the first step toward

planning for satisfying those requirements. Of course, the various

subsystems do not do this simultaneously, but rather in a logical

sequence: much of the Design Subsystem's inital work must be

completed, for example, before the Enabling Subsystem's performance

requirements can be properly tailored. However, at the proper time

each subsystem applies the APM to take that first step, which

consists of identifying the attributes or features of the people,

eqluipment, and procedures it must bring to bear to do the job.

This is a subsystem-level analysis of the performance req~.irements

to identify how those requiements impact on the subsystem's basic

components.

Block 6: Subsystem develops preliminary plans for securing the J required attributes.

These preliminary plans are the first output of the training design

process. In the plans, each subsystem states the attributes it needs

to achieve its assigned performance requirements; documents proof of

its possession of certain of those attributes; identifies those

attributes that it does not now possess and documents what it feels

will be workable means of acquiring those missing attributes; and

states the resources and other support it must have to implement

those means. The Subsystem is aware that Command will carefully

review and evaluate the submitted plans. Ideally, therefore, the

Subsystemn attempts to "anticipate" the evaluative measures that

Command will apply to the plans, and writes the plans in a way that

will produce "high scores" on those measures. That is, the

Subsystern informally evaluates its own plans (and measures its own

-126-
potential performance) before submitting the plans to Command.

This "first tier" of evaluation gives a valuable boost toward an

effective design specification provided that both the Subsystem and

Command have applied the APM properly to develop valid measures of

Subsystem performance.

Block 7: Command selects relevant measures of Subsystem performance. ...

This is the formal application (by Command) of the APM to develop

the valid measures of Subsystem performance discussed above. It is

this set of measures that the operating Subsystem attempts to

"anticipate." In practice, of course, it is completely reasonable that

Command may inform the operating Subsystem of the measures that

will be applied to the Subsystem's plans. Command, after all,

desires that measurably good plans emerge from the Subsystem.

Block 8: Command evaluates Subsystem plans and identifies performance

deficiencies.

This starts the first wave of formal evaluation in the APM application

sequence. Command applies its measures to the plans, producing

concrete, quantitative ratings of their "goodness." The emphasis

naturally is on the degree to which the plans demonstrate

achievement of the potentialities of performance, although such

measures of process and product as can be applied at this planning

stage will also be taken. Command reports its findings back to the

operating subsystem, pointing out specific deficiencies that must be

corrected. These evaluative findings demand a design specification

response.

Block 9: Subsystem modifies its plan to correct the deficiencies.

This is the design specification response to the first wave of

evaluation. Command has informed the Subsystem that specific

deficiencies exist in the plan. Those deficiencies include particular

Subsystem attributes that are essential for adequate performance,

and which are missing or inadequately demonstrated in the plan.

The deficiencies may also include some inessential or even detracting

attributes of the Subsystem that have been built into the plans.

The operating Subsystem modifies its plan to remove the attribute

deficiencies, i.e., modifies the proof of its fitness for the particular

training application.

Block 10: Command assembles the Subsystem plans into an initial

design specification for the total training system.

At this stage, Command exercises its overall system management

responsibility and integrates the various plans into one

comprehensive proposal for system design. This is an important

responsibility. The integrated plan must demonstrate not only that

-127-

: ....,-,.. :•,.... ., . . . . ,... . .. . . . . ... . . . . . .
each subsystem can do its proper work, but also that all their work

will "fit together" well. A particularly significant aspect of the

integrated plan is its ability to prove that Command can adequately

discharge its management responsibility on a continuing basis.

Block 11: Sponsor evaluates the initial total system design

specification and identifies deficiencies in training

system effectiveness.

This is the higher level equivalent of Block 8. It completes the first 4wave of formal evaluation. The emphasis remains on performance

potentialities. Here, however, the focus is on the integrated

* Utraining system's potentialities. This reflects the Sponsor's abiding

interest in "bottom line" training results. In this present case, the

interest is on the potential for satisfying the Sponsor's training

needs. The Sponsor Joesn't much care whether a defective

potentiality traces back to the Design Subsystem, or the Enabling Subsystem, or whatever. Sponsor has "hired" Command to perform

that level of diagnosis. Sponsor simply documents total system

deficiencies, and passes these back to Command with an order to

correct them.

.1 Block 12: Command translates identified system effectiveness

deficiencies into subsystem performance deficiencies.

Command performs the detailed diagnosis for which it was "hired,"

and reports its findings back to the operating subsystems. This

forces another (final) design specification response.

Block 13: Subsystem (again) revises its plan to correct its

performance deficiencies.

This is the "final" design specification response to the first wave of

evaluation. Of course, this response will be evaluated by Command

and subsequently by the Sponsor (after plan integration) . It is

possible that those evaluations could lead to further revisions,

further evaluations, more re-visions, etc. Fairly soon, however,

Sponsor would be expected to lose patience and start to look for

another training team to satisfy its needs.

Block 14: Subsystem implements its performance processes in

accordance with its plan.

Assuming that Command finally approves the Subsystem's plans, and

Sponsor finally approves the integrated plan, the plans finally go

into effect. The focus of Subsystem performance shifts from

potentialities to processes.
.. ' Block 15: Command evaluates Subsystem processes and identifies

performance deficiencies.

This starts the second wave of formal evaluation. Just as in

Block 8, Command applies its previously selected measures (Block 7)

to the Subsystem's performance. But now, the principal focus is on

proces performance. The Subsystem has moved from the "on

paper' design specification stage to the active design implementation

stage. Evaluation similarly moves away from the activities-asplanned to the activities -as-imp lemen te d. Any deficiencies noted in

that implementation are reported back to the operating Subsystem.

Block 16: Subsystem revises its processes to correct the deficiencies.

This is the design implementation response to Command's second

wave evaluative findings. Command has pointed out that the

Subsystem's activities are deficient: it may be doing some things

that it shouldn't, it may be failing to do some things that it should

% do, or both. The Subsystem responds by changing what it does.

Note that some of these process deficiencies may arise because the

Subsystem has deviated from its plans: it may be doing some things

differently from the way it proposed to do them. But note, too,

that some process deficiencies may arise precisely because the

Subsystem adheres to the plan, refusing to change with changing

conditions. The plan, any plan, rests on certain assumptions about

how the work will unfoF No plan is foolproof. The wise system

designer will not abandon the plan at the earliest opportunity, but

neither will he or she insist blindly on sticking to it when it becomes

clear that things aren't working out as they were expected to.

Process evaluation is not simply a matter of checking up on

deviations from planned performance, although that is one facet of

concern. Rather, true process evaluation is the determination of

* whether the system is doing what it should, given the conditions of

the real world. If the evaluator's process measures are based on

the true process performance requirements, he or she will uncover

the real process defects, and not simply disparities that may or may

not be bad.

Block 17: Command controls the integration of the various Subsystem

processes into a total training system process implementation.

This is another stage at which Command executes system

* management. Now, Command's responsibility is to see to it that the

work -as-c arried-out by the individual Subsystems merges smoothly as

it should. Previously, the concern was with the fitting together of

the work- as -planned.

-129-
'-I

Block 18. Sponsor evaluates total system process implementation and

identifies deficiencies in training system effectiveness.

This completes the second wave of formal evaluation. Now, the

.,pomor looks at what the total training system actually is doing,

and asks whether that promises progress toward satisfaction -f 'he

training needs. Any system activities that are either a hind ;,. to

. realizing those needs or merely superfluous are pointed out to

Command as process defects.

Block 1l: Comnitnd translates identified system effectiveness deficiencies

into Subsystem performance deficiencies.

This is idfentical, in word and spirit, to Block 12, although the

-eficiencies with which Command now deals are likely to be processrather than potentiality-based.

Block 20: Subsystem revises its processes and its plans, as needed, to

correct its performance deficiencies.

This is aiiother design implementation response to a set of evaluative

findings. Of course, changes in processes that are underway may

necessitate changes in processes planned for subsequent

implementation. Thus, design specification response might also be

needed.

Block 21: Subsystem delivers its performance products in accordance

with its plan.

After all p:,rties have approved the revisions to the processes, the

work picks up again. Eventually, products emerge. The focus has

nov' shifted from design specification through design implementation

to design fruition.

Block 22: Command evaluates Subsystem products and identifies

performance deficiencies.

Now the third wave of formal evaluation begins. The Subsystem

performance measures once more are applied, this time with special

* 1 emphasis on product measures. Command now has something to say

directly about the goodness of the Subsystem's goods.

.',-ck 23: Subsystem r,,ises its products to correct the deficiencies.

This is the design fruition response to the third wave of evaluation.

. To the extent that the Subsystem's goods are deficient, the

Subsystem must "make good" the goods. Sometimes this merely

requires placing a few "final touches" on the delivered goods: a bit

of polishing here and there, a few more dabs of paint, correction of

some typographical errors, deliverance of the service with a smile

-130-

+.._ . ++. . , .- . % . . . . . . . . t 
rather than a frown, etc. Other times the "making good" requires

scrapping the submitted product and completely replicating the

process that produced it, this time with much greater care. When

that sort of situation develops, the blame often can be partially

placed in a defective process evaluation: if the original process was

performed so poorly that the product had to be scrapped, the

evaluator probably should have discovered that fact before the

product emerged. At still other times, the only way to"miVe good"

on the product is to scrap the processes of production and devise

new, more effective products, that is, a fundamental redesign of the

system may be needed. All evaluators know that this kind of

sitliation does sometimes arise. But they also know that, when it

does, there often are defective evaluations of potentiality and

process buried somewhere in the system's history.

Regardless of whatever it takes, any defective goods need to be

"made good."

Block 24: Command controls the integration of the various Subsystem

products into a set of total training system products.

* 1 Now it is the work-as-delivered that must be made to "fit together."

Conflicts and inconsistencies between the interacting products of two

or more subsystems must be removed, so that an acceptable total

package may be delivered to the Sponsor.

Block 25: Sponsor evaluates total system products and identifies

deficiwncies in training system effectiven'-ss.

T his completes the "final" wave of formal evaluation. Here, the

Sponsor can come to grips with the bottom line of all bottom lines:

Does this now-fully-assembled training system actually satisfy the

Sponsor's training needs? To the extent that any needs remain

unsatisfied, training product defects will have been found.

Block 26: Command translates identified system effectiveness deficiencies

into Subsystem performance deficiencies.

Just as in Blocks 12 and 19, Command performs the detailed

diagnosis to determine how the defective work can best be corrected.

Now, of course, it is diagnosis of defective products that claims most

of Comriand's attention.

Block 27: Subsystem revises its products, processes, and plans, as

needed, to correct its performance deficiencies.

This is another design fruition response to a set of evaluative

findings. Depending upon exactly what is needed to "make good" its

goods, the Subsystem may also have to make a design implementation

-131-

-1:1EV.. . . . .................................................
response (revise its processes), or even a design specification

response (revise its fundamental plans and capabilities).

Blocks 28, 29 and 30: Suprasystem, system of interest, and other I! training subsystems affirm satisfaction of training needs and requirements.

K ~Provided that all concerned ultimately "sign off" on the revised

products, there will be across-the-board satisfaction of everyone's

"bottom line." Each Subsystem is pleased, because it has met all of Iits performance requirements. Command is pleased not only because

it has met its personal performance requirements, but even more

because the total training system has been effective. Sponsor is 4 happy because the training needs have been met. Each has its own

point of view, its own definitions and standards of "happiness."

That is to say, each faces its own taxa of performance requirements.

There is a direct parallel between this three-tiered training

structure and a professional baseball club. After a game, the center

fielder (for example) might say, "I'm happy: when they threw it, I

hit it, and when they hit it, I caught it. " The manager might say,

"I'm happy: we won." The owner might say, "I'm happy: by

winning, and by continuing to win, we've brought paying customers KI into the ballpark."

This discussion, and Figure 24 on which it is based, of course is very Jmuch oversimplified. Evaluation and design don't (or shouldn't) actually

proceed in three distinct waves. Rather, they are continual or even - continuous activities that always are blends of potentiality, process, and

product issues. Also, there truly is nothing "final" about a system design or

a system evaluation, until the system ultimately goes out of business. All

functioning systems always admit the possibility of improvement, and so always 1admit the need for additional evaluation and additional design. System

technology, after all, steadily improves. As it does, formerly state-of-art

system designs approach obsolescence, a trend that can only be halted by j modifying the system's design to incorporate the newer technology. Evaluation

and design are forever interlinked and forever functioning within any

operating system. Evaluation and design are the two threads that weave the

fabric of a total Systems Analytic Process.

-132-

. . . . .. ....
. 8o

V. RECOMMENDATIONS FOR FURTHER DEVELOPMENT

AND APPLICATIONS

As noted in Section II, the ultimate objective of this APM development

work is to provide users with a uniform, thorough, adaptive and efficient

S-. procedure to aid in the process of deriving the most meaningful design

specification requirements, design specifications, evaluation measurement

requiremeats or evaluation measures for any planned or existing

human-machine system (particularly a training system). In keeping with that

objective, the APM development project so far has resulted in the following

main products:

0 An annotated bibliography of the manned systems measurement

literature (244 documents cited and abstracted).

0 An initial analytic process model for systems design and

measurement.

o A generalizable taxonomy of measurable attributes for the

surveillance function found in many manned systems.

0 A generalizable model of training systems and their component

subsystems.

0 Generalizable taxonomies for the curriculum development (or design)

and trainer training (enabling) subsystems of training systems.

o A sample application of the model for deriving effectiveness*

rmnasures for Learner Testing Specifications within the emerging

BIFVIS including a generalizable taxonomy and application

procedures.

0 A sample application of the model for establishing Training System

-. design requirements for the Army's new 9mm Handgun, including a

generalizable taxonomy and application procedures. =.

o An untested procedural outline of a computer-based application of

the APM for training systems measurement (using an Apple II

computer).

This list represents considerable progress in an area that has been so

difficult to standardize for so many years. The model development is just now

. beginning to yield the kinds of guidelines and sample applications that soon

will allow it to be learned and applied by those who are its potential users.

"* To help bring the APM development process to the point of application by the

ultimate users, several tasks need to be carried out. These are listed in

Figure 25. It is noted that the items are not necessarily independent of each

other, but some tasks can be seen as being a part of others. They are listed

-133-

C *.. ~ 7-~.k* :..A- .- *.**i**
b

I

DEVELOPMENT

1. Further Develop the APM for Training Systems Measurement

2. Further Develop the APM for Training Systems Specification

I 3. Further Develop Training Systems Taxonomies for the APM

4. Further Develop Procedures for Measures Derivation

P 5. Further Develop Procedures for Identifying the Human Operators'

Contribution to Specifications or Measures

6. Develop Procedures for Differentiating Specifications and SW Measures in Terms of Weighting Factors, Delay Tolerance and

Criticality

APPLICATION

7. Apply the APM (Measurement) to an Existing Army Training System

8. Apply the APM (Design Specifications) to a New Army Training

System

9. Improve Procedures for User Application of the APNI (Measurement a

Design)

10. Test Procedures for User Application of the APM (Measurement &

Design)

11. Improve and Test Routinized Procedures Using Interactive Computer

12. Develop and Test a Training Program for APM Users

13. Train Users to Apply the APM (Measurement & Design)

14. Provide Mechanism for Future Modification/Improvement of the APM

L.

Figure 25. Recommended Further Research on APM for Training

Systems Specification and Measurement

,I

-134-
separately, however, because they are identifiably distinct or limited in their

purpose. For example, Item 1 can be seen as a general task, within which are 4. several others including part of Item 3 and all of Item 4. Item 4, in turn, can

be seen as having within it several others including parts of Items 5 and 6.

Likewise, Item 2 can be seen as a general task, within which one could also

place part of Items 3, 5 and 6.

Item 1. Further Develop the APM for Training Systems Measurement

The purpose of this research item is to continue developing the APM

for measurement toward the ultimate goal of providing a practical and useful

end product. The research task ultimately should address all stages of the

model in order to extend its conceptualization and utility in each of its

aspects. The research method should include model application with several

existing or developing training systems to verify or modify its component

P arts, its adequacy, and its practicality. The effort should also produce improved and amplified procedures and guidelines for using the model.

*Item 2. Further Develop the APM for Training Systems Specification

The purpose of this research item is analogous to that of Item 1, but

as applied to the APM for specification. Sample applications should involve the

specification of several new training systems. All component parts of the

specification model should be addressed; concepts and definitions should be

clarified and application guidelines should be developed. In addition, the

utility of the model for assessing previously generated design requirements

should be evaluated.

Item 3. Further Develop Training Systems Taxonomies for the APM

4 devlopedCurrently, generalized taxa for any Design Subsystem ha'.e been

deveopedand are being applied to BIFVTS issues for verification. A second

generalized taxonomy for any Enabling Subsystem is also completed. Taxa for

the additional generalized training subsystems should be developed and their

applicability to BIFVTS issues should be verified through some real world

comparison or evaluation, by subject-matter experts (SMEs) or otherwise. The

* same procedures should be applied as used in prior taxa, development and

verification of the taxa should be carried out. Taxa should be completed for

each of the training subsystems of the BIFVTS.

IItem 4. Further Develop Procedures for Measures Derivation

One of the most di fficul t-to- develop segments of the APM for

measurement is that in which measurable attributes are determined and actual

measures are selected (Ref. : Figure 1, blocks 6 and 7). Some progress has * * been made during the past year in beginning to identify specific procedures

for measures derivation. More needs to be done, however, especially if

computer-aiding is to be used for this segment. In particular, the analytic

steps leading to measures derivation involve a determination of issues that

imply a need to measure particular attributes.

-135-
Additional work is also needed to incorporate the MOE-MOP

development steps into the APM, identify the relationships among MOE-MOP for

systems-subsystems levels, and identify the relationships among the machine

and human components in terms of contribution to MOE-MOP development or

specification. This subtask can build on the current BIFVTS work, by

continuing more sample or partial applications to increasingly complex issu,.s of

military training, both to verify and to improve the procedural guidelines.

This is analogous to the earlier process of generating taxonomization guidelines

in this research effort. The procedural guidelines allow for the derivation of

MOE-MOP specific to given measurement purposes and provide for direct

MOE-MOP (type) selection given the defined measurement purpose. Those

guidelines, when better understood, will define the algorithm for measures

derivation that then can be included in the computer-aided process.

Item 5. Further Develop Procedures for Identifying the Human

, Operators' Contribution to Specifications or Measures

One aspect of the measures derivation process (Ref.: Item 4)

ruquiring particular attention is the determination of operator-centered

measures. This task should identify the many (sometimes subtle) roles of

human operators in the total system measurement and evaluation process.

Researchers should apply the appropriate techniques of Human Factors

Engineering to define the roles and influence of operators in the various

system attributes selected for specification or measurement. Some researchers

using the model in the future may require assistance in maintaining the

necessary awareness of the human's role, so that their derived specifications

- and measures will give due consideration to operator-determined aspects of

system performance.

Item 6. Develop Procedures for Differentiating Specifications

and Measures in Terms of Weighting Factors, Delay

Tolerance and Criticality

, The pr;,ctical aspects of model utilization suggest that there will be

times when the full range of specifications or measures cannot be applied in its

entirety, and that choices will have to be made to select only the most

important ones for application. To aid the user in selecting those

specifications or measures which are most important, a procedure should be

developed which helps to rate or rank measures in accordance with such

criteria as: criticality to mission performance, amenability to alternatives or

corrective actions, tolerance to postponement or delay in application, or other

criteria to be determined.

Item 7. Apply the APM for Measurement to an Existing Army System

This item involves extending the current developments of the

-"Curriculum Development Subsystem of the BIFVTS to a practical application 1 that also can serve as a validation or verification exercise. For example, the

el can be applied in the context of the Training Effectiveness Analysis

S(TEA) of the BIFVTS planned by USAIS/USAIC. We understand that the TEA

-136-

W.. .. .. . .. . .. . . .. . . .. . . : . .. .. ...- .....".... .. ,. : .z£;, .. .. ., ...e " """" " """' ". ..""" ""' " "" " " " = " """' ." "" " " "' "" -"". " " " " " " " -....-..- .-.., . ... - . .-- , ..,. ...-.. A,,w.- . -.". ...- .. : ... '. ,. .- .. .. '. . .. - .' ... .-. -,_-A
Is to begin in 4th Qtr FY 1982. Current developments can be extended to the

point where derivation of specific measure sets for the effectiveness of the

BIFVTS subsystem can be generated. These measure sets can then be

examined for feasibility, validity and utility in the context of the TEA. It

would be necessary to review the measurement issues to be addressed in the

TEA that are relevant to the BIFVTS curriculum, and one or more of those

issues should be selected as the basis for this task. The performance taxa

pertaining to each selected issue and the measurement derivation guidelines I should be used to generate a measures hierarchy for each selected issue.

Differences between APM and TEA measures should be examined so that

* reasons can be determined for those differences. Commonalities should also be

noted and their reasons understood. For practical reasons, this in-depth

application and comparison should focus on a narrow aspect of the BIFVTS,

such as Basic Gunnery Learner Testing. This task would also provide an

opportunity to focus on human operator contributions to performance, such as

motivation, attitudes, capabilities, and procedures under given environmental

administrative and other constraints.

Item 8. Apply the APM for Design Specification to a

New Army Training System

Initial extension of the APM to the specification of training system

requirements was begun during the current contract year. This item is

designed to extend and further explore this use of the APM as a tool through

application to another new (or developing) training system. The purpose is to

develop further and to verify, to the extent possible, the capability of the

APM process to serve this function. This can be done through the "blind"

development of training system specifications for a new or developing system to

be selected by ARI. A comparative assessment can be carried out between

design specifications or specification requirements as developed by using the

APKI, and similar specifications or requirements as developed by other

methods, such as the Early Training Estimation System under development by

the ARI Fort Bliss Field Unit. This task could also make use of the. APMI's

capabilities both to generate new design specifications and to aid in the

diagnostic assessment of previously generated design specifications. For the

same practical reasons as mentioned under Item 7, this comparativ.e application

should take place in depth over a narrow aspect of the new training system.

Item 9. Improve the Procedures for User Application of the

APM (Measurement and Design Versions)

This task would enlarge and improve the documented procedures and

guidelines to be followed by users in applying the APM. It would also help to

define the steps and algorithms essential to a computer-aided application of the

model. The effort, in general, would make use of all the prior and concurrent

applications of the APM, documenting the steps, the decision rules, the

criteria and other factors that are essential to application procedures. One set LI of procedures would be expected for the development of performance

taxonomies (Figure 1, blocks 1-3), but later blocks in the model would require

-137-

--. L*~ A*.~~:. J - S
separate procedures for the Measurement version as compared to the Design version.

UItem 10. Test the Procedure for User Application of the

APM (Measurement and Design)

This task is preliminary to training actual users in the application of

the APM. It would consist of providing a skilled, but APM-naive analyst

(such as a "non-project" member of the Dunlap staff) with the procedures and

* guidelines, to be used in developing effectiveness measures in one application

and design specifications in a second application. The purpose of this trial

application effort is to refine and -larify the procedures prior to making them

available to users through computer-aided facilities and training programs

(Ref.: Items 11-13). If this task can be conducted with the interactive

computer capability, then Item 11 becomes unnecessary as it would only

duplicate this one. If the computer-aided capability is unavailable, then this

task would be conducted manually with printed guidelines and instructions.

.- The final product of this task should be an updated set of guidelines,

procedures, sample applications, taxonomies, checklists, algorithms or any

other available, documented aid to APM application.

Item 11. Improve and Test Routinized Procedures

Using the interactive Computer

This task is intended to build upon the existing computer-aided

capability, by adding flexibility, instruction options, thoroughness, and data

bases to the rudimentary computer-based model. It would also Include a test

of the procedures by a skilled, but APM-naive analyst (as in Item 10), to

". determine where modifications are required to clarify, simplify or otherwise

improve their value and ease of use.

Item 12. Develop and Test a Training Program for

APM Users

This task should include whatever remaining development effort is

necessary to yield routinized (computer-aided) procedures for application by

users. It should also include a test of those procedures by user personnel.

Among others, likely candidate "users" for this test would be administrators

and training developers of the USAIS. This task should consider the

development of training/usage handbooks or guidelines for users; contractor

development and conduct of user-directed workshops; and other techniques for

user training in the APM process and its application. The task should also

consider the training/usage of the APM process by various personnel types

and for both possible purposes (evaluation of training systems and

- specification of training systems). The eventual procedures for training of

users and for their usage of the APM should be as programmatic,

proceduralized, routinized and simplified as possible. Advantage should be

taken of the model's already demonstrated capabilities for flow charting,

programming, indexing and other operations which make it readily reducible to

routinized procedures. The user field test of these procedures should aim to

-138-

- .l-
confirm their completeness and accrac, their utility (based on the acceptance

by users) and their validity (based on the quality and comprehensiveness of

products).

Item 13. Train Users to Apply the APM (Measurement

and Desigi

This task is one in which the previously developed and validated

training program (Ref.: Item 12) is placed into operation to promulgate the

APM as an analytic tool among potential users. It requires the sponsorship of

a promulgating agency, which will be identified for this purpose by ARI. One

possible agency to be considered for this dissemination function is the

U.S. Army Training and Doctrine Command (TRADOC). Administrative

preparations for this task item, in the form of preparatory explorations and

information exchange with the possible promulgating agencies, should begin in

the near future, even though the task itself is not likely to begin until a later

date.

The task should identify specific users in specific agencies, and

plans for administering the APM training program should be developed. A key

office should be designated to administer and coordinate the subsequent

training activities, and should be accessible for future inquiries from trainees

and users. A cadre of trained instructors and an instructor training program

*should also be established as a continuing entity. One of the tasks of the

APM program administrator should also be to serve as an information exchange

and documentation center for users. The mechanisms and procedures for those

* administrative and technical capabilities should be designed as part of this task

item.

Item 14. Provide Mechanisms for Future Modification or

Improvement of the APM

To aid the promulgating office in carrying out its functions as an

information exchange and documentation center for users, a technical capability

should be established to include analysts, programmers and the interactive

computer facility for APM applications. These resources would be used to

maintain the APM, its associated data bases, and its various application

routines and records. Users can then be helped with the latest fund of

knowledge, techniques and experience. A current mailing list of users would

allow for rapid distribution of application documents, program modifications,

-' ' information requests, and suggestions to improve the value of this resource to

users.

One of the most important sources of ideas to keep the APM as a

useful and valuable resource is the user population itself. A method should be

established for permitting and, in fact, encouraging users to supply feedback

of application ideas to the promulgating office, with the assurance that those

ideas will be given careful consideration and dissemination in the most

appropriate manner. Dissemination can employ a newsletter, a "new

application"t description, a revised data base or even a message for CRT

-139-
display when users log on to use the model. In fact, one way in which users

can conveniently communicate briefly with the APM central office could be via .,. Ithe interactive terminal, such as by typing in messages or inquiries after using the program or by pausing during use when a problem is encountered.

* The specific mechanisms for keeping the APM current and useful should be

determined and implemented in this task item.

-140
| n

 ... | ,.  

-v,

VI. SELECTED BIBLIOGRAPHY

U BlooLr, R.F., Oates, J.F., Jr., and Hamilton, J.W. An Initial Analytic

Process Model for Systems Measurement: Extensions of the Systems Taxonomy

Model. (Final Summary Report).Darien, CT: Dunlap and Associates, Inc., 30

Apri 1%1.

Branson, R.K., Rayner, G.T., Cox, J.L., Furman, J.P., King, F.J., and

Hannum, W.11. Interservice Procedures for Instructional Systems Development.

Tallahassee, FL: Florida State University, 1975.

Briggs, I.J. (Ed.) Instructional Design: Principles and Applications.

Englewood Cliffs, NJ: Educational Technology Publications, Inc., 1977.

Dunlap ad Associates, Inc. Instructor Training Institute for the Police

Traiiic Services Teacher: Participant's Notebook. Darien, CT: Dunlap and

Associates, Inc., 1980.

Edwards, J.M., Bloom, R.F., Oates, J.F., Jr., Siptkowski, S., Brainin,

P.A., Eckenrode, R.J. and Zeidler, P.C. An Annotated Bibliography of the

Manned Systems Measurement Literature. Darien, CT: Dunlap and Associates,

Inc., No-ember 1981.

" . Finch, G. (Ed.) Educational and Training Media: A Symposium. Washington,

DC: National Academy of Sciences--National Research Council, Publication 789,

1960.

Finley, D.L., and Muckler, F.A. Human Factors Research and the

Development of a Manned Systems Applications Science: The System Sampling Prolem and a Solution. Northridge, CA: Manned Systems Sciences, Inc., July 1976.

Finley, D.L., Muckler, F.A., Gainer, C.A., and Obermayer, R.W. An

Analysis and Evaluation Methodology for Command and Control: FinaTechnical Report. Northridge, CA: Manned Systems Sciences, Inc.,

November 1975.

Greer, G.D., Jr. Basic Training Effectiveness: A Discussion of Instruction

Centralization, the Training Curriculum and Achievement Evaluation. Fort

Ord, CA: U.S. Army Leadership, Human Research Unit, 1957.

Klott-r J.C. T'echricques for Police Instructors. Springfield, IL: Charles C

Thomas, 1963.

* Longo, A.A., Gaddis, J.T., and Whitehouse, B. Preliminary Evaluation Plan

for U.S. Army Computerized Training System. Fort Monmouth, NJ: U.S.

Army Sig'.al center and School, 1974.

-141-

-~~ .,' . .. . .
McDonnell Douglas Astronautics Co.-East. A Survey and Analysis of Military

Comuter-Based Training Systems: (A Two Part Study). St. Louis, MO:

McDonnell Douglas Astronautics Co.-East, 1977.

Smode, A.F., and Lam, K.D. (Eds.) Military Instructor Training in

-" oTransition. Orlando, FL: Training Analysis and Evaluation Group, 1975.

Thurmond, P. Development of Analysis, Design, and Development Techniques \: for Team ISD. San Diego, CA: Path Research, 1980.

U.S. Department of the Army. Interservice Procedures for Instructional

Systems Development: Executive Summary and Model, Vol. I-V. Fort

Benning, GA: U.S. Army Combat Arms Training Board, TRADOC Pamphlet

No. 350-30, August 1975.

U.S. Department of the Army. Toward a New Hand Gun. Army Research,

Development and Acquisition Magazine, Alexandria, VA: Hq., U.S. Army

Materiel Development and Readiness Command, January-February 1981.

771

U..

- . - ..

t"l. t S~- -. - * .... -- - .,. .- ~ . . . . .. * . . ,
I

APPENDICES

I

I

I

-K I

~; 1

~ ~

II

J -143-
9.'

i APPENDIX A.

Examination of Design Subsystem Performance

A Potentialities Taxa to Assess Their Relevance to

Evaluation of Learner Testing Specifications

a. "Identifying Goals and Priorities for Training" (Taxon 1.1)

1.1.1.1 "Identifying types of achievements relevant to the intended

job. "H

The job-relevant achievements ultimately represent the

subject matter of learner testing. The single most important

reason for testing learners is to determine whether they are

ready to do the job for which they have been trained. If the

Design subsystem does not have the capability of identifying

accurately the job-relevant achievements, the testing

-- specifications almost certainly will be deficient. If important

achievements are overlooked in the training design, they will

also not be covered by any specified test. If non-relevant

achievements are included in the training design, then

corresponding non-relevant tests will be included in the

specification. In either case the tests specified would produce

distorted images of learners' qualifications for the job.

Conclusion: This taxon is relevant to the measurement

application.

Note that this conclusion automatically insures the

relevance of taxon 1.1.1 ("Defining the total scope of learning")

and taxon 1.1 ("Identifying goals and priorities").

1.1.1.2 "Analyzing achievements to determine suitability for training."

The testing specifications must include provisions for

assessing the candidate job performer's abilities relative to

every achievement deemed relevant to the job. This is true of

- al achievements that are selected as suitable for training. It is

f".:a.' true of those achievements deemed not suitable for

training: those become cast as prerequisites for the job. It is

just as necessary to know whether a prospective learner

e'," possesses the prerequisites as it is to know whether the learner

has achieved the abilities addressed in training. The Design

subsystem's capabilities for assessing whether or not a

job-relevant achievement is suitable for training will have a

major impact on the subsystem's performance in specifying

learning content. It shuuld not affect the system's performance

in specifying testing. Of course, if the content is poorly

analyzed, the tests probably will disclose that the training fails

to qualify learners as job performers.

Conclusion: Thie taxon is not relevant to the application.

A-1
1.1.2.1 "Determining necessary levels of achievement."olwnne

i:JLearner testing is supposed to disclose not ol hte

the candidate job performer can carry out the actions relevant

* to the job, but also whether he or she can do so well enough to

-~meet the job's requirements. For example, some job ight

require a performer to lift a 50-lb. weight and hold it at

* shoulder height for a period of time. It would be critical for

Straining design, personnel selection, and learner testing to

determine whether that "period of time"~ is instantaneous, a few

seconds, several minutes, or whatever. If the job's

requirements called for holding the weight for, say,

15 seconds, it would be inappropriate to test the learner by

j requiring that he or she hold it for only 5 seconds; it would be

I equally wrong to conduct a test that required holding the

weight for a full minute. The former test would "qualify" some

I learners who actually do not meet job requirements; the latter I test would "disqualify" some who can really do the job. Thus,

a Design subsystem that is not capable of accurately

determining necessary levels of achievement very likely will

produce test specifications that are deficient.

Conclusion: This taxon is relevant to the application.

Note that this insures as well the relevance of taxon 1.1.2

4 ("Stating the ultimate intended outcomes of learning").

1.1.2.2 "Determining existing levels of achievement."

This capability explicitly calls for testing. It requires

that the current status of the candidate learner populations'

abilities relevant to the job be determined. This implies that at

least some testing, even if only informal, be applied to at least

some representatives of those populations.

Conclusion: This taxon is relevant to the application.

1.1.2.3 "Discerning discrepancies bet-ween existing and necessary levels

of achievement."

This requirement pertains to the Design subsystem's ability

to accurately assess differences between what the populations of

candidate job performers can do now, and what they need to be

able to do to perform the job satisfactorily. That ability

demands access to data that can be acquired through testing,

and demands the subsystem's ability to apply those data

correctly. But, it has no direct bearing on the test

specifications themselves.

Conclusion: This taxon is not relevant to the application.

A-2
*1.1.3.1 "Assessing golimportance."

' J The absolute or relative importance of the various training

goals have no impact on the need to test all achievements

.1 relevant to the job.

Conclusion: This taxon is not relevant to the application.

2- 11.1.3.2 "Insuring availability of a relative (numeric) goal ranking

scheme.

The comment given under taxon 1.1.3.1 also applies here.

j Conclusion: This taxon is not relevant to the application.

1.1.4.1 "Insuring availability of a hierarchy /taxonomy of goals."

A Design subsystem must be able to classify its training

goals and objectives in a way that permits the kinds of actions

required of learners and job performers to be identified

accurately. This is of particular significance to test

specification, for without a suitable classification scheme, the

wrong type of test might be specified. For example, if an

A ' objective requires a learner to demonstrate some physical act, it

would not be sufficient to test the learner' s achievement of that

objective only by requiring him or her to state the physical

activity. That test would disclose whether IMF learner knows

what is needed, but it wouldn't disclose whether the learner

can really do what is needed.

Conclusion: This taxon is relevant to the application.

Note that taxon 1.1.4 ("Establishing a basis for specifying

performance objectives") is thus also relevant.

1.1.4.2 "Stating goals in terms that imply behaviors.

If a Design subsystem is unable to state its training goals

-4 in behaviorally relevant terms, it is unlikely that the actions

required of learners will be identified correctly or expressed in

a fashion insuring they will be observable and quantifiable. If

the required actions are unclear, then the actions to be tested

also will be unclear. Proper test specification demands

behaviorally defined goals and objectives.

Conclusion: This taxon is relevant to the application.

b. "Establishing Performance Objectives" (Taxon 1.2)

1.2.1.1 "Defining the performance action."

If action cannot be precisely defined, then the learner's

ability to produce the action cannot be exercised in a test.

A-3

% ~. ~ .
Conclusion: This taxon is relevant to the application.

0 do Note that taxa 1.2.1 ("Stating what learners will be able to

dor how they will be expected to behave after completing

training") and 1.2 ("Establishing performance objectives")

automatically are relevant as well.

1.2.1.2 "Defining the performance conditions."

If the performance conditions required for performing the ,.1 action on the job cannot be precisely defined, then no

assurance can be had that the specified tests will exercise the

~ $ learner's ability under the appropriate set of circumstances.

Conclusion: This taxon is relevant to the application.

1.2.1.3 "Defining the performance standard."

If the standard for "1good enough" performance cannot be

-. precisely defined,' the test specifications likely will not provide

for accurate interpretation of the adequacy of the learner's

action.

Conclusion: This taxon is relevant to the application.

* 11.2.2.1 "Developing pre-test items."

Pre-test items form an integral portion of the test

specifications.

Conclusion: This taxon is relevant to the application.

Thus, taxon 1.2.2 ("Providing a basis for objectively

assessing learner performance") also is relevant.

1.2.2.2 "Developing post-test items."

Post-test items form an integral portion of the test

specifications.

Conclusion: This taxon is relevant to the application.

C. "Analyzing Performance Objectives" (Taxon 1.3)

1...1.3.1.1 "Revealing present conditions of learning."

1.3. 1. 2 "Identifying essential prerequisites."

Z1.3.1.3 "Identifying supportive prerequisites."

Test specifications must provide for diagnosing the cause

of any deficiency in a learner's performance. Such deficiencies

might arise because a learner falls to accomplish some j prcrequisite to a given objective. iLor example, the learner may

fil torecall something that is relevant to learning (a deficiency

in the present conditions of learning); or the learner may not

A-4
have mastered some achievement that is an essential milestone

toward the objective (a deficiency in an essential prerequisite);

or the learner may not have mastered some achievement that,

while not essential for the objective, would have facilitated

dattainment of the objective (a deficiency in a supportive

prerequisite). Unless these types of prerequisites can be

identified, they won't be reflected in the test specifications.

Conclusion: These three taxa are all relevant to the

application.

:-. iNote, thercfore, that taxon 1.3.1 ("Identifying

prerequisites to learning") and taxon 1.3 ("Analyzing

performance objectives") also are relevant.

1.3.2.1 "Developing entry test items."

Entry test items form an integral portion of the test

specifications.

Conclusion: This taxon is relevant to the application.

Taxon 1.3.2 ("Providing a basis for objectively assessing a

candidate learner's suitability) thus also is relevant to the

1.33. application.

1.3.3.1 "Discerning independence between objectives."

1.3.3.2 "Discerning dependency between objectives."

1.3.3.3 "Discerning a supportive relationship between objectives.

These potentialities express the capability that the Design

a 'subsystem must have for determining the sequence in which

performance objectives should be achieved during training.

Certain objectives logically must be achieved before others are

pursued. In other cases, the achievement of a given objective

will aid a learner's pursuit of another objective even though it

is not essential for that pursuit. The Design subsystem must

have the capability of identifying these dependency

-L. relationships if it is to structure the training efficiently and

effectively. However, the dependency among objectives really

has no direct bearing on the testing of those objectives.

Conclusion: These three taxa are not relevant to the

application.

1.3.4.1 "Identifying information type objectives."

1.3.4.2 "Identifying mental skills type objectives."

1.3.4.3 "Identifying physical skills type objectives."

1.3.4.4 "Identifying attitude type objectives."

The comment given above in reference to taxon 1.1.4.1 is

applicable here as well. Unless the Design subsystem can

accurately classify its performance objectives in terms of the

type of learning called for, the test specifications may not

A-5

...-......... ..................... ........-.... .. . .
insure that the proper type of testing takes place.

Measurement of a learner's achievement of a mental skills

objective, for instance, requires a mental skills test;

achievement of a physical skills objective must be measured by

a physical skills text; and so forth.

Conclusion: These four taxa are relevant to the

application.

Note that taxon 1.3.4 ("Classifying objectives in terms of

the domains of learning involved") automatically is relevant as

well.

1... Ietfin vr tp .

1.3.5.1 "Identifying covert steps."

1.3.5.3 "Identifying ucoerto steps."

The Design subsystem must have the capability of

analyzing the actions required of the performance objectives to

identify the constituent steps that make up the action. This is

essential for structuring the training content in a way that

fully explains each task to the learners, and that enables the

learner to practice performing the task. The capability is also

essential for insuring proper testing of learners, since the

testing should diagnose performance deficiencies. In other

words, if a learner fails in performing a task on which he or

she isbeing tested, it should be possible to determine which of

the overt, covert, or unconscious steps led to the failure.

This is a significant challenge to the test designer, since he or

she must devise ways to translate the covert and unqonscious

steps into observable, quantifiable behaviors. This often

requires that the learner -being -tested state aloud the results of

mental processes, decisions, recognitions, etc., that proceed in

silence under normal on-the-job conditions. But certainly, the

test designer cannot develop means of diagnosing the learner's

performance of the constituent steps unless those steps can be

identified.

Conclusion: These three taxa are relevant to the

application.

Note that taxon 1.3.5 ("Identifying component steps or

processes within the objectives") automatically is relevant as

well.

d. "Defining Training Content" (Taxon 1.4)

1.4.1.1 "Defining propositions, names, facts, etc. , that are relevant to

information objectives."

This capability is essential to insure that the Design

subsystem can accurately select the content to be provided to

learners to assist them in achieving information -type objectives.

This is an extremely critical capability bearing on how well the

A-6
V ~rwi~r

subsystem specifies the learner presentation/ demonstration

activities. However, how the learner learns is of no real

significance to the sp~ecilcation of means for determining

whether the learner has learned. Indeed, test designers often

err byproducing test specifications that measure how well the

learner recalls his or her learning experiences and the material

to which he or she was exposed rather than measuring how well

the learner has achieved the stated objectives. If the

instructional content provided to support a given

information -type objective was poorly chosen, the tests specified

for that objective probably will disclose widespread performance

deficiencies among learners. But the central fact is that the

objectives-referenced tests should be independent of the

instructional content.

Conclusion: This taxon is not relevant to the application.

-1.4.1.2 "Defining concepts, rules, algorithms, etc., relevant to mental

skills objectives."

1.4.1.3 "Defining movements, timings, actions, etc. , relevant to

physical skills objectives."1

1.4.1.4 "Defining values, choices, etc., relevant to attitude objectives."

The comments given above in reference to taxon 1.4.1.1

apply here as well.

Conclusion: These three taxa are not relevant to the

application.

1.4.2.1 "Assessing applicability of Job Performance Aids."

1.4.2.2 "Assessing applicability of Self-Teaching Exportable Packages." *.1.4.2.3 "Assessing applicability of Formal On-the-job Training."

1.4.2.4 "Assessing applicability of Installation Support School."

1.4.2.5 "Assessing applicability of Resident School."

The Design subsystem must possess these five capabilities - if it is to do a proper job in choosing the settings for the

various learning activities. However, the issue of where

training takes place has no bearing on the specification- Ttests

7 to determine whether the training was successful.

Conclusion: These five taxa are not relevant to the

application.

1.4.3.] "Sequencing in dependency order."

*1.4.3.2 "Sequencing in supportive relationship (efficiency) order."

The Design subsystem must possess these capabilities to

insure that the performance objectives will be pursued in a

logical, efficient sequence. That is absolutely essential if the

learning experiences are to be as effective as possible. But

the order in which objectives are pursued should have no

bearing on the tests to be used to determine whether the

objectives are met.

% A-7

................. . . . .. . . . . . .
Conclusion: These two taxa are not relevant to the

application.

e. "Defining Training Procedures" (Taxon 1.5) II~ 1.5.1 .1 "Defining facilitator preparation activities." 1.5.1.2 "Defining facilities preparation requirements." ~N-.1.5.1.3 "Defining attention gaining procedures." 1.5.1.4 "Defining objective-informing procedures."

1.5.1.5 "Defining procedures for stimulating learners' recall of

prerequisites."

All of these capabilities must be present in the Design Asubsystem if it is to be able to specify appropriate and efficient

means of assisting learners to prepare for learning. However,

none of these capabilities bear on the subsystem's work in

specifying appropriate means of testing learners.

~~ IConclusion: These five taxa are not relevant to the

application.

51.5.2.1 "Defining presentation media/media alternatives."

1.5.2.2 "Defining procedures and requirements for placing emphasis."

1.5.2.3 "Defining procedures for enhancing learners' comprehension and

retention."

* All of these capabilities must be present in the Design

subsystem if it is to be able to specify appropriate and efficient

means of presenting/ demonstrating the training content and

material to the learners. However, none of these capabilities

bears on the subsystem's work in specifying appropriate means

of testing learners.

Conclusion: These three taxa are not relevant to the

application.

1.5.3.1 "Defining procedures for enhancing learners' participation /

involvement."1

1.5.3.2 "Defining procedures for eliciting the performance from

learners."

1.5.3.3 "Defining procedures for providing learning guidance to

learners."

These three capabilities must be present in the Design

subsystem if it is to be able to specify appropriate and efficient

c~ means of assisting the learners to practice/apply what they are

learning. However, none of these capabilities bears on the

subsystem's work in specifying appropriate means of testing

learners.

Conclusion:- These three taxa are not relevant to the

application.

A -8
1.5.3.4 "Defining procedures for providing learning feedback to

:1 ' , learners."

1.5.3.5 "Defining procedures for assessing sufficiency of practice."

The two capabilities also are essential to insure that the

Design subsystem will be able to specify appropriate and

efficient practice for the learners. The key ingredient that

these capabilities supply is learner feedback, i.e., the ability

to inform the learner of how he or she is doing in practice, and

to help the learner and facilitator judge whether the learner has

practiced enough, at least for the time being. These abilities

demand some form of testing. Usually, this is learner

self-testing, perhaps under observation and "scoring" by a

facilitator.

Conclusion: These two taxa are relevant to the

application.

Note that this automatically implies that taxon 1.5.3

("Specifying application procedures") and taxon 1.5 ("Defining

training procedures") also are relevant.

1.5.4.1 "Defining procedures to assess the proper domain of learning."

This capability aims at one key ingredient of learner

testing, namely, the ability to insure that the proper kinds of

tests are used for a given class of learning objectives.

Conclusion: This taxon is relevant to the application.

Note that taxon 1.5.4 ("Specifying learner evaluation

procedures") therefore is relevant as well.

1.5.4.2 "Defining procedures to assess the appropriate action/ -S behavior."

This capability goes to the heart of another key ingredient

of learner testing, namely, the ability to insure that the tests

require the learner to demonstrate the actions that are

appropriate for the performance objectives.

Conclusion: This taxon is relevant to the application.

1.5.4.3 "Defining procedures to insure assessment reliability."

This capability focuses on yet another key ingredient of

learner testing, namely, the ability to insure that the tests are

conducted subject to conditions and performance standards that

reflect real job requirements.

Conclusion: This taxon is relevant to the application.

A-9

. . ' . . .. .. .. , ... -.. .. -....... .. " .... , .. -... .-. ...''.,' ,, ,-'.-,. -.- :..'.."' ."..
* f. "Evaluating the Curriculum" (Taxon 1.6)

*1.6.1.1 "Entry-level testing of representative candidate learners."

In order to validate the assumptions underlying the :4. curriculum, the Design subsystem must be able to ascertain

whether the candidate learner populations actually satisfy the

prerequisites specified for the training. Usually, if not always, *2- Ithis entails some type of testing of at least some representative

members of those populations. The ability to construct and

apply such tests is an ingredient of the subsystem's overall

performance in specifying learner testing.

Conclusion: This taxon is relevant to the application.

Note that this implies that taxa 1.6.1 ("Validating /revising

the bases for the developing curriculum") and 1.6 ("Evaluating

the curriculum") also are relevant.

1.6.1.2 "Post-training testing of representative job incumbents."

The training is supposed to prepare the learners to do the

job for which the training was designed. Post-training tests I specified by the Design subsystem are supposed to provide a

basis for determining whether the learners are in fact prepared

to do the job. The post-tests, in other words, are supposed to I predict job performance. Whenever possible, it is desirable to

_ validate the post-tests by applying them to current

(satisfactory) job incumbents. The ability to do so clearly is

an ingredient of the subsystem's overall performance in

specifying learner testing.

1 Conclusion: This taxon is relevant to the application.

- 11.6.2.1 "Obtaining error data."

One key question to be faced at the final stages of a

curriculum development effort is: "Given a group of learners

who have received training in accordance with the curriculum,

how well do those learners perform?" Part of the answer is

1 found by administering specified post-tests to a sample of (pilot

test) learners. Such testing produces error data, disclosing

which objectives or portions of objectives were not achieved by

significant numbers of learners. The ability to obtain this

error data is an ingredient of the subsystem's overall -~ performance in specifying learner testing.

Conclusion: This taxon is relevant to the application.

Note that taxon 1.6.2 ("Validating /revising the developing curriculum itself") thus also is relevant.

A-10
1.6.2.2 "Obtaining design feedback."

Design feedback consists of general and specific criticisms

of the curriculum by representative "users" of the curriculur:i,

i.e., learners and facilitators. Like error data, design

feedback typically is obtained in pilot-test settings, often at

virtually the same time that learner post-testing occurs.

However, the solicitation of design feedback is not a test, nor

is it in any way part of the curriculum's test specifications.

Conclusion: This taxon is not relevant to the application.

1.6.2.3 "Correlating post-test data with subsequent job performance."

IThe ultimate evaluation of a new curriculum is found in the

learners' job performance subsequent to training. The Design 4 subsystem must have the capability of obtaining measures on job

performance, at least for a sample of "graduated" learners.

Further, it must be able to compare those measures with the

end-of-training tests administered to the same learners, to

identify any deficiencies in the test specifications. The test

specifications cannot be considered to be "complete" if the

ability to correlate test results with real job performance is not

built into the Design subsystem.

Conclusion: This taxon is relevant to the application.

I

4

%

.1

:--i

- ,..-A . -
APPENDIX B.

Application of the APM (Measurement)

to the IFV Training System

Appendix B supports the textual material presented in Section III of this

report. It contains Figures BI through B10. Briefly, Figures B1 through B6

,. present all of the Army Handgun Training Design Subsystem Potentialities

. Requirements. Figure B7 illustrates a sample AHTDS Performance Process

Requirements, and Figure B8 illustrates an example of the process design

j specifications for the Figure B7 requirements. Similarly, Figure B9 illustrates

a sample AHTDS Performance Product Requirements, and Figure B10 illustrates

an example of the product design specifications for the Figure B9

requirements.

" L

,

Q1B-

.1'I

(5

;B-

"J

!...................................................,..-
Performaince Potentiality HierArchy Number I: The capability of Identifying training goals and priorities

L1.2

We hlandgun Training Design Subsystem

must possess the ability to Identify training goals and priorities for all intended

ovilitary users and uses of the handgun.

Given a bat of handgun users Given a Ust of handgun users Cena list of handgun users Er-na list of handgun users

and uses. the Design Subsystem and uses. the Design Subsystem and uses. and given the train- and uses, and given the trairmust be able to define the must be able to define what ing goals defined for each ing goals defined for each

scope of learning seded for each user ultimately will be userluse, the Deign Subsystem uaser/use, the Design Subsystem

each handgun userluse in terms able to dn with the handgun, must be able to .dentify the must be able to expresa the

of the knowledge. akills, and in training goal terms. relative importance of each goals in behaviorally oriented

attitudes requtred. defined goal. terms to establish a basis for

VMS identifying handgun user perLosance objectives.

YT Rndgun Training 7e'Vindgun Training Tghe Handgun Training TI'Wandgun Training

Design Subsystem must Design Subsystem must Design Subsystem imust Design Subsystem must I demonstrate that it poo- demonstrate that it poa- demonstrate that it has demonstrate that it can

insses the ability to sesses the ability to the ability to formulate devise or select a suitidentify, for each hand- define the necessary assesments of training able taxonomy of goals/

Sunt userluse, exactly levels of performance goal importance. ebjectfves to be used to

what the uaser must know. achievements that each classify its intended

~ Meactly what slkills the handgun user must mai- training outcomes.

oeer must possess; and feet In order to apply

eaactly what attitudes the handgun to its AM the Inner must mnanifest, intended uses.

'TWTindgui' Training 7%eTalndjgun Training YTeWandgtun Trainingt ThEV andgtun Training

.. ,Design Subsystem must Design Subsystem must Desigtn Subsystem vaust Design Subsystem must

demonstrate that it porn- demonstrate that it is demonstrate that it can demonstrate that it is

*ease& the aility to able to determine cor- devire or select a suit- capable of statingit Is

analyse the identified rectly the pro-training able scheme for ranking poals in terms that perknowledge, skills, and levels of the relevant 11%, "elative importance mit the necessary user

attitudes to determine performance achievements of its training goals. behavior@ to be Identified.

their suitability for possessed by members of

training, the intended handgun user populations.

F; aindgun Training

Design Subsystem must

deonsstrate that it pon

0 msas the ability to

vdiscern eorrectly the

iscrepancies existing

between current and required levels of relevant

withn theintended handgun nowr popualations.

Figure B-L Army Handgun Training Design Subsystem Performance Requirements

Basic Potentialities Needed: Hierarchy Number 1

B-2
Performance Potenti6ity Hirrchy Number 2:

The capability of est&bihn er7Forance objective$

YFe Handgun Training Design Subsystem

must pose*& the ability to establish

performance objectives for all intended

military users and uses of the handgun.

1.2.11.2.2

tvnalist of handgun users and t van a list of handgun users and

uses, the Design Subsystem must Ases and given the stated parlorbe able to state in performance- eatice objectives. the Design Suboriented terms what the handgun system must be able to define a

users will be able to perform as scheme for objectively assessing

a result of training. handgun user performance prior to

and after receiving the designed

training.

Th iHndgun Training Design The Handgun Training Design

Subsystem must demonstrate Subsystem must demonstrate

that it possesses the abil- that It possesses the abilIty to define all of the ity to develop pre-tst Items

IStasks that the handgun users to determine giFZun users'

ZM le expected to perform. existing knowledge&. skills

and attitudes.

1.2.1.2 1.2.2.2

The Handgun Training Design Thie Handgun Training Design

Subsystem must demonstrate Subsystem must demonstrate

that It possesses the abil- that it possesses the ability to define all of the ity to develop pst-teatt

eondltionalcircumetances Items to dete mne-%an gun

under which each of te users' poet-training knowitasks is to be performsed, edges, skills and attitudes.

Nameangu Training Design

Subsystem maust demonstrate

that it possesses the abiltty to define all of the

!!,ainIng/performace Otandad or sach an .7s or Tof the counditiouns under

which the task ia to be

performed in terms of mini- mrum acceptable proficiency.

Figure B-2. Army Handgun Training Design Subsystem Performance Requirements

Basic Potentialities Needed: Hierarchy Number 2

B-3
'-:~ I

-.- ,l j, ii i

':I a .

I" ,:I "Li 1 LilE iit h•EL

I V-A

z JI z i2 E

~~~"> w3IIIU

gil

~I htU. {li" -'i " " "' !i: l i" °

E cc

il l I

Il *1I:=:

I. {1) "1.1ii Z i '

elii

B-45

S.. : _.

"",j I C z

.: . . . . . . -- I . .. o. . *
P,.formn'o-c Potentiality Hierarchy Number 4: The capability of defining the training content

L 1.4

'" e Handgun Training Design Subsystem

most possess the ability to define the handgun training content required to achieve the users' performance

objectives.

1.4.1 1.4.2 1.4.3 a" S list of handgun users and revni a list of handgun users and 1e a lit of handgun users ano

goals 

s. , and given the defined training uses, and given the defined training uses, and given the defined trai, a and analyzed performance goals and analysed performance goals and analyzed performance ebjectives. 

must be able 

the 

to 

Design 

select 

Subsystem objectives, the Design Subsystem objectives, the Design Subsystem stimuli for must be able to select an appro- must be able to define an instru- ech performance objective relevant pilate instructional setting for tional sequence which provides for to the domains of learning, each performance objective, the optimum transition and supp-,,

from one performance objective i,,

another.

i ! 4.4.].1 1.4.2.1 T H-aIndgun 1.4.3.1 Training Design YlirW-ndgun Training Design ThiHandgun Training Desipt

m Subsystem must demonstrate Subsystem must demonstrate Subsystem must demonstrale that it possesses the abil. that it possesses the abil- that it possesses the abil- ity to define procedures, try to assess the applicability Ity to sequence the training fats, names. etc.. that are of Job Performance Aids (iPAs) telative to the objectives relevant to informsation-type to enhance the learning dependency order nf requireri objectives, process. skills. knowledges and

attitudes.

1.4.l. 1.4.2.2 T.e Tindmun 1.4.3.2 Training Design T 1 Nindgun Training Design Mel-mndgun Training Design Subsystem must demonstrate Subsystem must demonstrate Subsystem must demonstrate that it possesses the abil- that it possesses the abil- that it possesses the ability to define concepts, Ity to assess the applicability rules, allorithms ity to sequence the training

tMtevant to mental , etc. of Self-Teaching Exportable based upon the supportive sk .h Patkutes (ST.Ps) an a vehicle order of knowledges. skiitl objectives, to enhance the learning and attitudes from one per- process. formsnt. objective to other

performance objectives.

S!.4.1.3 1.4.2.3 T _rWaindgun Training Design FTWondgun Training Design Subsystem must demonstrate Subsystem must demonIstrate - that it possesses the abil- thz; it possesses the abilIty to define handgun user/ Ity to assess the applicability uset movements, actions, of formal on-the-job training tmings, etc., for relevant (OJT).

physical skWl objectives.

'. . 1.4.1.4 1.4.2.4

lW-Mndgun Training Design Te amndgun Tralning Design Subsystem must demonstrate Subsystem must demonstrate

that it p"sesees the abil- that it possesses the abil- I'a i defilne handg i- vserl Ity to asseas the applicability Is - choices, values. etc., of Installation Support Schools relevant to attituee objec" (ISS) as a possible instructves. tional setting.

1.4.2.5

T "O'rindgun Training Design Subsystem must demonstrate

that It possesses the abil- rii ity to ases the applicability

ef Resident Schools (RS) an a pesible instructenal setting.

Figure B-4. Army Handgun Training Design Subsystem Performance Requirements

Basic Potentialities Needed: Hierarchy Number 4

B-5 --.
.. > .1Performance Potentiality Hierarchy Number S: The capability of defining training procedures

1'We Handgun Training Design Subsystem

must possess the ability to define V the training procedures to be employed

for all intended military users and

uses of the handgun.

M-na list of handgun users ive n a list of handgun users M-na list of handgun users M-na list of handgun -P-. and uses. and given the train- and uses, and given the train- and uses, and given the train- and uses. an't giver, ins.. ig goals and priorities. an&- in& goals and priorities. ana- Ing goals and priorities, ans- performance ohiectives or., 17ted performance objectives lyzed performance objectives. lyzed performan:e objectives. defined training content, and defined training content, and defined training content, and defined training content. specifv evaluation pr~cre.-. specify the training prepara- specify the training presents- specify the application pro- to assure Achievement of p.. tior procedures required to tion procedureit required to cedures required to assure formance ob~ectives.

"asure optimum learning, assure optimumi learning, optimum trainee achievement of

performance objectives.

The Handgun Training We'Mandgun Training TRT'Windgun Training TF77Tidoun Traini,,a Design Subsystem must Design Subsystem must Design Subsystem must Design Subsvster, rrs *demonstrate that it porn- demtonstrate that It pos- demonstrate that it pos- demonstrste that it pc,. * muses the ability to misses the ability to sesses the ability to @ease@ the abd'jtv to emdefine required teacher define presentation define trainingt applica- define procefiures to A-, preparation activities to medislmedia alternatives tion procedures for en- the learner's achievr~r, assure optimum learning, to assure optimum hancing the handgun of Information. mental

learning. learner's involvement or skills, physical skill, a,, J

participation In the attitude oibjectives. i.e

handgun training, the domains of learning

1..12 .5.2.2 1.5.3.2 1.5.4.2 ?Te Hadgun Training W5Mrandgun Training 'TFeTandglun Training YTWindgun Training Design Subsystem must Design Subsystem must Design Subsystem must Design Subsystem MIu-t *demonstrate that It porn- demoinstrate that it pos- demonstrate that It pos- demonstrate that it rui nse the ability to sesses the ability to &eases the ability to sesses the ability to define facilities prepa- define presentation pro- define procedures which define procedures for ration requiremnrts to codures and requirements will enhancel/stimulate assessing appropriai. -accomplish hiandgun for placing emphasis to learner performance. handgun learner brla-..

training objectives. enhance the need-to-learn. action.

1..13 .S.2.3 1.S.3.3 1.5.4.3 ne-Mfidgun Training TK;Wi-ndgun Training 7%T*Wndgun Training YT~eandoturi Tram't, * .Design Subsystem must Design Subsystem must Design Subsystem must Design Subsysiemn demon strata t,,at It pos- demronstrate that it pos- demon strate that It pos- demonstrate that it noose@ the ability to sessms the ability to sesses the ability to sesses the abilit - define handgun learner define learning pro- define procedures that define procedures tha- attention-gaining cedures to enhance hand- will provide learning will assure reii.b'e procedures. Sun learner's comprehen- guidance to handgun assessment of han.; iom end retention, lEarners, learner achievement it -. performance oblective'-

Y~WdgnTraining 75 WmTandgun Training Design Subsystem must Design Subsystem must demonstrate that it pos- deo nstrate that it p..- esas the ability to misses the ability to define handgun perfor- define procedures which *mince objective- will assure that learning Inforesing procedures. feedback is provided to

handgun learners.

1.5.1.51.5.335

UDesign ?W~adgun Training YWeTMandgun Training Subsystem must Design Subsystem must sotstrata that It PON- demon strata that it poe- moes the ability to masses the ability to define procedures for define procedures which stimulating handgun asure that adequate/ laarner's recal of ruffldient practice has handgun tieining pre- been accomiplished to sequisitse. coniltently achieve

performance objectives.

Figure B-5. Army Handgun Training Design Subsystem Performance Requirements Basic Potentialities Needed: Hierarchy Number 5

jB -6
, a

°

]Performance Potentiaity Hierarchy Number 6:

The capability of evatuatins the currieun

1.6

TFe Handgun Training Design Subsystem

must possess the ability to evaluate

the curriculum developed for intended

Army users and uses of the handgun.

Um 1.6.1 1.6.2

V', a lis of handgun users and M a list of handgun users and uses. as well as defined handgun uses. as well as defined handgun b'aining system goals and priori- training system goals and priori- ties, defined performance objec- ties. defined performance objec- tive.. defined training content tives. defined training content

and training procedures, the and training procedures, the

Design Subsystem must be able to Design Subsystem must be able to

validate and/or revise the bases validate/revise the developing for developing the handgun train- handgun curriculum itself.

Ing system curriculum. :1.6.1.1 1.6.2.1

YT"Frandgun Training Design Te "a'ndgun Training Design Subsystem must demonstrate Subsystem must demonstrate

that it possesses the abil- that it possesses the ability to conduct entry-level ity to obtain error data

testing of representative related to trial learner

handgun learners. training.

1.6.1.2 1.6.2.2 "eT Jndlun Training Design The'Tandgun Training Design Subsystem must demonstrate Subsystem must demonstrate

that it possesses the abil- that it possesses the ability to conduct post-training ity to cbtain curriculum

tsting of representative design feedback from poten- handgun msers. tial learnersijob incumbents.

1.6.2.3

1Wandgun Training Design . Subsystem must demonstrate

that it possesses the ability to correlate post-test

training date with subsequent learners' job perfor- Stance.

Figure B-6. Army Handgun Training Design Subsystem Performance Requirements

Basic Potentialities Needed: Hierarchy Number 6

B-7

..................................
Performance Processes Hierarchy Number 3: An approach for analyzing tasks selected for trsainins

2.3

IWe Handgun Training Design Subsystem

must be able to plan an approach for

analyzing handgun user tasks which

are selected for training.

.. 12.3.2 2.3.3

Maing the Job analysis of hand- UU~i-ing the job analysis of hand- Mafling the job analysis of handgun users and uses and an "ase- gun users and uses and an asessn- gun users and uses and the assess- mont of those handgun users' tasks want of those handgun users' tasks ment of those handgun users' tasks

selected for training development, selected for training development, selected for training development. the Design Subsystem must be able the Design Subsystem must be able the Design Subsystem must be able

to plan an approach to Identify to plan ani approach to classify to plan an approach to identify the Information processing the tasks selected for training in the conditions for learning each en mtaled in each of the tasks terms of the domains of learning of the tasks selected for training.

selected for training. involved.

2.31.12.3.2.1 2.3.3.1

TS; andgun Training Design TK7Wmrndgun Training Design The H ndgun Training Design Subsystem must be able to Subsystem must be able to Subsystem must be able to

plan an approach to analyze Ian an approach to identify pIan an approach to analyze

processing to a sequence of acquired. requirements to identify prebinary decisions, requisite knowledge expected A- of learners.

2.3.1.2 2.3.2.2 2.3.3.2

YW-Rindgun Training Design Th1W-Mndgun Training Design TEl rndgun Training Design Subsystem must be able to Subsystem must be able to Subsystem must be able to

plan an approach to analyse plan an approach to identify plan an approach to analyze the task decision levels in the mental skills which task requirements to identerms of nature, complexity, must be acquired. tify prerequisite mental

and time. skills.

2.3.2.3 2.3.3.3

yThindgun Training Design YE-Tandilun Training Design

Subsystem must be able to plan Subsystem must be able to plan an approach to identify the an approach to analyse the

physical skills which must be physical requirements to Idenacquired. tiy prmeqtsisite motor skills.

2.3.2.4 2.3.3.4

TWe-Mndgun Training Design Yierndifun Training Design

Subsystem must be able to plan Bubsystemo must be able to plan

an approach to identify atti- an approach to analyse the

tudes to be acquired. attitudinal requirements to

Identify expected prerequisite - larner attitudes.

5 Figure B3-7. Army Handgun Training Design Subsystem Performance Requirements

Basic Processes Needed: Hierarchy Number 3

B -8
Process Performance Requirement Number 3:

An approach for analyzing tasks selected for training

(2.3)

The AHT Command Subsystem will have reviewed as part of the

curriculum design effort, the handgun user's job analysis and the

analysis to determine which tasks should be selected for training

development. The Design Subsystem will closely interact with the

Command Subsystem in determining which of the tasks considered for

training will actually be selected for training development. The final

determination of tasks selected for training will be based on such factors

as task training costs, available time and resources as well as tasks which

may be already learned through previous training and consequently may

be considered as a prerequisite to the AHT. Once the Design Subsystem

has received the approved list of tasks to be trained they will be

analyzed utilizing operational sequence diagram (OSD) techniques and

decision trees which will describe in detail the learning hieiarchy

m associated with each task.

(2.3.1, 2.3.2, 2.3.3)

The team will examine the information processing entailed in each of

the tasks selected for training in the context of representative scenarios

designed specifically in the use of the handgun for personal defense

actions. It is anticipated that the Command Subsystem will supply

realistic scenarios in such detail to provide the analysts with information

which describes the conditions and performance environment in which the

user must utilize the handgun for self defense purposes. The sequential

operations in performing each of the tasks will be analyzed via an OSD

technique until all of the task elements and their required sub-elements

have been defined. The analysis will identify all the handgun user task

inputs, the processing required, the resulting outputs, as well as task

element inter-relationships. At each level of task element analysis, the

analyst will determine what information the learner must know, recall or

state to perform the task. The analysis will identify categories of * -information which the potential handgun learner must process. These

analyses also will be the basis for identifying prerequisite abilities that

the handgun learner will bring to the training situation, as well as for

identifying the learning the AHT must provide as described below.

Figure B-8. Sample Design Specifcation of a

Process Aspect of the AHTDS

(In response to the Performance Requirements illustrated in Figure B-7.)

B-9
(2.3.1.1, 2.3.1.2)

The sequential analyses of tasks will identify handgun user decision

flows. These decision flows will be further analyzed utilizing a binary

A decision tree (yes/no) technique. The decision flow analysis will include

an appropriate time line as well as columns to identify the nature of

decisions and to classify the complexity of the decision levels from simple

to complex using a scale of 1 to 5. The complexity of these decisions will

be based on such factors as criticality of the decision with regard to task

completion delay tolerance and number of factors influencing the yes/no

branches of the decision tree.

(2.3.2.1, 2.3.2.2, 2.3.2.3, 2.3.2.4)

The information-decision-action attribute model, described in detail in

Section ** of this specification, will be the basis for performing the

analyses to classify the tasks selected for training in terms of the

domains of learning involved. An analysis of each of the tasks will be

performed to develop categories of information the potential handgun

learner must possess to complete tasks/task elements. The analysis will

detail the bodies of knowledge the learners must be able to recall or

state. To analyze mental skills required, the task elements will be

analyzed in detail to determine rule learning and using, pattern

recognition, symbol identification, detection cues and decision making.

The team of analysts will prepare a detailed hierarchial documentation of

the mental skills a potential learner must possess in the performance of

tasks. To analyze physical skills required, again each task element will

be sequentially analyzed to determine the sequence of physical (motor)

skills required, a hierarchy of the sequence of physical skills required to

perform required tasks will be fully documented. An analyses will also be

performed to determine the attitudes a handgun learner should possess in

performing the handgun systems functions i.e., serve as a short-termr

personal defense weapons system. This analysis will involve some

speculative thinking since It is not always possible to observe attitudes

directly. The team will analyze each task/task element and ask the

question, "What would a potential handgun learner do if he had the

desired attitude?" The team will use a consensus approach to develop a

list of the attitudes potential learners must possess in order to exhibit

the desired attitudes.

(2.3.3.1, 2.3.3.2, 2.3.3.3, 2.3.3.4)

The analyses to determine potential learner prerequisite knowledge,

mental skills, physical skills, and attitudes will be accomplished utilizing

the outputti of the analyses described in the previous paragraph and the

analyses performed in Section ** which identifies entry-level behaviors

that the potential learner brings to the learning situation. A comparison

of entry level behaviors/attributes that a learner brings to the training

situation and those behaviors/attributes that a trained handgun user must

r exhibit will define those knowledges, mental and physical skills as well as

attitivdes the learner must acquire via the AHT. The results of these

analyses will be documented for each of the domains of learning.

Figure B-8 (Concluded)

B-10

.. . . , . -
Performance Products Htierarchy Number 3: The caaiiyof producing lesson plant

17 ~~~~must be able tprdclesnplans to support intended handgun user

training objectives.

3.3.1 3.3.2 3.3.3 V~ing the developd handgun user Uifl-ing the developed handgun user Uttlisting the developed handgun user job aiTajyis and performance abjec- job analysis and performance objec- job analysis and performance objec- tiveis. the Design Subsystem must tives. the Design Subsystem must tives. the Design Subsystem must provide complete specification of provide coot~lete specification of provide complete Specification of the Instructionial content for each the Ittatruc ional procedures for the instructional circumstances for training objective, each training objective, each training objective.

3.31.13.3.2.1 3.3.3.1 YrdunTraining Design TgeTfmndgun Training Design Yhe-Tndgun Training Design Subsystem must produce specift- Subsystem must produce specifi- Subsystem must produce specifi- cationts of each handgun user cations of procedures for pre- cations of the location at which training objective, paring handgun learners to each instructional event will

achieve each training objective, take place.

>:WWa'du Trinn Design 4jj±L2;dsu Training Design Th adu Training Design Susse utproduce specifi- Subsystem must produce epecill- Subsystem must produce specifi- cain fasmdhandgun cations of exactly how the cations of the equipment needed Isrve ps iie.isrconlContent topics/ to support the training.

steps wlbepresented.

3.3.1.3 unTaiin esg 3.3.2.3 3.3.3.3

Sy stemnust Triding aein YW-eWndgun Training Design The Mandgun Training Design Subystm mstprouceanSubsystem must produce specifi- Subsystem must produce specifi- Itemization of information cations of exactly how the cations of special location/I topics. handgun learners will practice/ equipment set-up /configuration apply the instructional content, requirements.

3.3.1.4 3.3.Z.4 3.3.3.4 TWri'Wndilur, Training Design Tlel1 agun Training Design TeHngnTraining Design Subsystem must produce an Subsystem must produce specifi- Subsystem must produce specifi- Itemization of mental skill cations of exactly how the cations of instructional per- steps. handgun learners will be tested sonnel needed. on the instructional content.

3.31.53.11.23 3.3.335 Y3WWandgun Training Design a'lndgun Training Design The Handigun Training Design Subsystem must prodouP. an Subsystem must produce specifi- Subsystemn must produce specifi- Itemization of physical all cations of procedures for deal- cations of time and schedule steps. ing with handgun learners who requirements.

fai to demonstrate achievemen t

of any of the training objec-

'a" tives.

.... ... YFWbrdgui Training Design

Subsystem must produce an Ite~atlo oflearer ttitudinal smons~t.

YWIRandgun Training Design Subsystem must produce an Itelsation of practice eserIIW~mdgun Training Design 4 Sumbsystem met produce an ke~satiot of tes problm/

Figure B-&. Army Handgun Training Design Subsystem Performance Requirements

Basic Products Needed: Hierarchy Number 3
Product Performance Requirement Number 3:

The capability of producing lesson plans.

(3.3)

The team recognizes that the development of lesson plans is one of

the most important requirements in the training development process. It

* is at this point that all of the preceding analyses will be utilized to

produce the actual instructional products, i.e., lesson plans which

support the AHT objectives. The team that has been assembled are well

versed in producing lesson plans to meet these objectives. The analyses

to develop the handgun user's job analysis arnd performance objectives will

be the basis for lesson plan development. These analyses include the

following information:

0 Training objectives 0 Training objective sequence

o Tasks to be performed number and how

0 Conditions of performance objectives were structured

o Training standard of 0 Learning category and

acceptable performance sub-category for each training

o Test items for all objectives objective

o Reference to the task/task 0 Learning activities for each

element from which the objective

training objective was 0 Planning information from other

derived AHT subsystems

0Existing handgun instructional

programs

The team will use Field Manual 21-6 "Military Training" as a guide to

produce lesson plans which provide practical, economical aid in preparing

and delivering the training. The lesson plans will not be designed to

record every word of the instructor's presentation. The following

minimum information will be included in all of the produced lesson plans:

0 The Training Objective(s) - Principal and assistant * 0All Intermediate Training trainers

-. Objectives (if any) listed - Training aids, devices and

in the sequence to be taught equipment to be used

o Administrative instructions - References

(as required) 0 Training sequence and time -When the training will be estimate

conducted 0 Safety restrictions -Training location 0Additional information required - Who will be trained by local SOPs

Figure B-10. Sample Design Specification of a

Product Aspect of the AHTDS.

(In responsc to the Performance Requirements illustrated in Figure B-9.)

B~-12
(3.3.1, 3.3.2, 3.3.3)

t In providing complete specifications of the instructional content, the

team will prepare a content outline to satisfy each training objective.

Each training objective will contain the tasks which must be learned, the

conditions of task performance and the task standard of acceptable

performance. The specifications will include learner prerequisites as well

as the inti rim objectives which must be performed to achieve the

.'. ~objective. The outline will itemize the information topics, mental and

physical skills, and attitudes which must be acquired as a result of

learning. The lesson plans will include practical exercises and test items

to stimulate learning as well as providing the instructors with criteria for -evaluating trainee learning relative to the training objective.

Instructional strategies will be included for each lesson plan. They will

be in the form of guidelines and will offer strategies which will aid the

instructor in presenting the material. The strategies will suggest ways to

keep the learners involved. They will cue the instructor as to when I instructional aids are appropriate as well as when to emphasize/lrepeat instructional material. The specifications for instructional circumstances - will be detailed in the administrative instructions of the lesson plan as

well as in the POI course guide.

1 (3.3.1.1 - 3.3.1.8)

The Task Inventory developed in Section ** of these specifications,

will be used as the basis for deriving the specific terminal and

- -intermediate Training Objectives. Each Training Objective will be clearly

stated and will consist of the following three elements:

0Tasks - What knowledges /skills the handgun learners must

acquire ,...0 Conditions - Under what conditions the learners must acquire the

knowledges/ skills

0 Training standard - A description of minimum acceptable

performance

The various kinds of learned capabilities have various kinds of

prerequi sites -- previously learned entities which are available from the

learners memory at the time new learning begins. A notation of the type

of prerequisite required for, or helpful to, the learning appropriate to

each listed Training Objective will be identified. For each category of

learned capability, the following will be considered:

Figure B-10 (Continued)

B-13
S0 Mental skill 0Cognitive Strategies U- Simpler component - specific intellectual skills

Intellectual skills 0Attitudes

0 Physical skills - Intellectual skills, verbal

-Part skills, procedural information

rules

0 Verbal information - Meaningfully organized sets ofinformation

To demonstrate how objectives in the various domains of learning interact

with each other in reaching the final outcomes of the training objectives,

the team will use the format titled "Instructional Maps." This map will

show the teaching sequence and interactions for specific training

objectives in the domains of information, intellectual skills, attitudes,

cognitive strategies and motor skills as well as the relations of

<intermediate objectives within a single training objective. This itemization will help insure that all relevant content and skills are included in the

- training program.

To insure learning, sufficient practice time and exercises must be

available. At a minimum, one practice exercise will be designed for each

intermediate and training objective. To insure the transfer and

integration of learning, the practice exercises will relate directly to the

required job skills.

Means to evaluate learner achievement within each of the domains of

learning will be itemized. The training objectives for the POI will be

used as a basis for deriving test problems and exercises. In addition to

itemizing the questions deemed essential, a few additional questions will

be designed and included to check the reliability of responses and to

measure the influence of changes in wording and teaching techniques. Two or more roughly equivalent or closely related questions (wellseparated in the test) will be included in order to measure consistency

(i.e. , validity) of answers.-

(3.3.2.1, 3.3.2.2, 3.3.2.3, 3.3.2.4)

Each lesson plan will specify the instructional strategies to be

employed to meet both interim and training objectives. The strategies will

be keyed to the lesson plan outline and serve to cue the instructor.

Training content related to knowledge will present strategies to guide

instructor-learner discussions. Skill related training (physical -mental) strategies will present cues to guide performance /practice oriented

training. Attitude training cues will aid the instructor in presenting role

Figure B-10 (Continued)

B-14
mi

plays, demonstrations or discussions. Each lesson plan will be designed ':- J so that the interim objectives are sequenced to provide a learning

* hierarchy to meet the training objective. Through performance oriented

training the learners will apply the instructional content. The lesson

plans will provide cues to the instructor to guide each phase of

" instructions. All tasks, conditions and standards will be presented as

well as associated instructional strategies. Each training objective and

associated interim/intermediate objectives will be tested against the stated

task conditions and performance standards. A training record, keyed to

the POI, will be designed as an aid to record learner achievement. I Strategies will be provided on each lesson plan to guide remedial training.

(3.3.3.1, 3.3.3.2, 3.3.3.3, 3.3.3.4)

1 The team will insure that the administrative instructions on each

lesson plan include the training facilities required to support the

training, equipment needed to support the training, instructional d personnel required and specifications of time. A training course guide

will be developed to support required POI. The course guide will contain

the following four sections:

0 An Introduction, which will describe the Guide's organization,

list the program's instructional units and provide an overview I of the training administrator's responsibilities.

0 iA detailed discussion of the instructional program, including its

overall objective(s), specific learner performance objectives and

the contents and duration of each instructional unit.

S, A detailed discussion of program planning considerations,

including scheduling, desirable class size, instructor

qualifications, materials and equipment required, facilities

needed and procedures for estimating program costs.

0 A detailed discussion of program management and evaluation

considerations, including suggestions for record keeping and

identification of potential improvements to the program.

-I ," Figure B-10 (Concluded)

- B-15
I:C-

.-.- .A PPENDIX C.

~Documentation for the Computer-Aided Model

., 0 General Flow Chart for Sample Demonstration Program

j Explanatory Notes for Flow Chart

-p

c-o
General Flow Chart for Sample Demonstration Program

Application of the APM for Developing Measures of the following Training

Systems components:

0 Curriculum Development Subsystem

- -. 0 Instructor Preparation Subsystem

1~. C-2
a. .-- ~ -CC- ~ ~ ~ 6 -0. 4 .-.

C2

sq~~r &~v~

Is

El n5

-'5 G

to AT ?

I%

'i A ,RT

Ic

cc4

-4p C-3.
A

0

IIt

Ii

.9C-4

- - - - - - -
4 IV A 1

4'4

%Uh

4O1XA

SF.

C5)
lim

4,4

P4 4A

C2.3)2~

AA U"'

( 2 0) _ _ _

1C-6 !x41
F

(32)

OD4 f

iT#

(33

I3

tN

Ic

IyVe

OJ

C-
EXPLANATORY NOTES

Page Block Block Functional

Number Number Name Description

1 SIGN-ON Typical sign-on procedures for accessP ing an interactive computer system

and specific software packages.

Restricts acbess to STM to those

who are authorized, i.e., those who

know the sign-on code.

1 2 DECISION/QUERY: For the novice user, a capability will

. does operator wish to exist to access and review basic

view some basic descriptive information about STM, -* Idescriptive info about e.g., its applications, its structure and

STM? data base, its interactive procedures,

a list of more detailed references,

etc. If the user so chooses, this

information can be called up on the

screen, in a series of display pages.

1 3 DECISION/QUERY: At any stage of its evolution, STM's

does operator wish to data base will contain complete

c I view the list of sys- taxonomies for only a finite (but

tern types to which steadily growing) set of system types.

STM presently is Unless the user is interested in one

applicable? of the types presently "stored" in the

data base, STM will not be applicable

to the user's needs. If the user so

chooses, the list of presently "stored"

types can be called up on the screen.

For the proposed demonstration, that

list will include 5 entries, only one of

which (Training Systems) actuatly will

have taxonomies in storage.

1 4 SYSTEM TYPE The user will type in the full name of

SELECTION one of the system types for which

taxonomies presently exist in the data

base. For the proposed demonstration,

even though four other choices will

"appear" to be available, the user will

have to select "Training".

1 5 APPLICATION The computer will acknowledge the

SELECTION user's interest in the selected system

type (Training) and inquire whether

the user wishes to apply STM to 1)

select MOEs/MOPs or 2) specify a system design. Even though both options

*wHi be displayed, only 1) _. measures

selection ... actually wil be available

for the proposed demonstration. Thus,

the user will have to type In "1".

C-8
EXPLANATORY NOTES

Page Block Block Functional

Number Number Description

2 6 SUBSYSTEM The computer will acknowledge the

SELECTION user's interest in the specified STM

application (measures selection), and

will present a list of the general or

typical subsystems of the selected system type (Training). The computer

then will ask the user to indicate

which subsystem is to be addressed

first. Even though all six general subsystems of Trrainiing will be displayed

to the user, only two of them

(Curriculum Development and Instructor

Preparation) actually will have taxono- 4 mies in storage for the proposed demo.

'02 7 DECISION/QUERY: In any measurement application, the

does operator wish to user typically is interested in assessing

view the list of cur- how well the system performs some

rent Performance particular aspect of its work. For Iaspect KEYWORDS for example, one might be preparing to

the selected subsys- select MOEs/MOPs to investigate how

tem? well a curriculum development subsystem has performed with respect to

producing specifications for testing

m learners. Each aspect of performance

associates with some particular subset

of the system's complete taxonomy.

Each taxon in the subset gives rise to

certain issues and implications for

assessing the associated aspect of performance.

At any stage of its evolution, STM's - data base will contain specifications of

the taxonomy subsets associated with a

finite list of performance aspects for

any given system /subsystem. Having

signified interest in "Training" systems

and (for example) their "curriculum

* development" or "instructor preparation"

subsystem, the user can elect to call

* up display of a list of the performance - aspects for the selected subsystem for

L. which subsets of taxa are defined in L. the data base.

2 8 PERFORMANCE The user can type in any KEYWORD

ASPECT ... whether or not it exists in the curSPECIFICATION rent list ... to signify the performance

aspect of interest. For the proposed demonstration, the KEYWORD current _____ ______________lists for the Curriculum DevelopmentJ

C-9
EXPLANATORY NOTES

Page Block Block Functional

NubrNubr .... E Description

4entry. nec) uthr culsb

DECISION:set is tha.w~ e defined in tersoh dsgaesb

performaneatspec se taxa) fo th copr w nti offe

desinatebyheueath usr ane potnty int reietwihe

ontained/bedefined inr subystm' copeentaxny. i

29 CMUEIfthe crented orde KEYWORD tot idniytoetx ote

DElIStON or the eetdewihte (nely spcfiederforaesb

subsystmnc aspecst of intrest cots i oeram

desinatd b th usrple ofe ho h pplicaiity o Siwte

conaind /efiednaubthoiedm's canpee a newmsubthe urret KYWORoseto thetxnoy, ands this assae

list ~ h ~ eece ~ ~ a ihte bo acewss int ifed e frmbnoter

subsytemuseso inetedesin tht sme aspcto

2 0 ~ DEIINQUR: ~ ~ ~~l Uowdn thaiat the enerdEY iit 

does teopertorwthorid se cnandedfine thew cuen

to efie asubet f lst ofo the subsnoye ofd titer et

the previously Pearaton)ste bnde compeute notisther

fin ~ ~ ~ uersrthtanudeie KEYOR ntrsed that aspect of r

has beenentered? formance.sbe eintd n

2 1 DCISONQUE: inquies Upo whthr the user wses toY

does~~~I the userto electOD s not t dfine aurn

to efie suse of litaxoomye subsset, noth intmrescat b

fined EYWORDthadoe cot nnn undefined aspectpr

hasbee enere? ofrmance. STMbee defats tonth

inur, whtcan the chose toisecify

anoifthe aspc ofrerforanc or

addessaohetyeofsse

entirelysor elsimplycomefne SIN

2 11 DEIGNATETAXA aUonom recei oting mdctoe tha the

(seqentilprcedue)oue inen t udefined axooysub-t

se, assocae th thoe newl specifie I~aote aspect of performance, h optr begins ato diplypte e spfiesubsys

(seqentil prced e 'sinndto taxonomy tothuse.-h efea 

- a1
a

EXPLANATORY NOTES

Page Block Block Functional

Number Number Name Description

.2 11 concept here is that the user will make

(cont'd) an "include or exclude" decision concerning each taxon's membership in the

new subsystem. Experience sugests

that it is sufficient to examine only

the characteristics level taxa. If a

L* taxon on that level associates with

some aspect of performance, it automatically follows that its hierarchically

superior taxa on the functional purposes level and the objectives level

also must associate with the performance aspect. Thus, for example, if

the user decides that taxon 1.2.4.1 • " (characteristics level) is relevant to the

aspect of interest and signifies that the

taxon is to belong to the subsystem

being formed, the computer automatieally will include taxa 1.2.4 (functional

purposes level) and 1.2 (objectives

level) in the subsystem as well.

• Thus, the computer will display every

characteristics-level taxon to the operator, in sequence, and will "Insist" that

the user signify either INCLUDE or

EXCLUDE before moving on to the

next taxon. Once all characteristicslevel taxa have been reviewed and

designated for inclusion/exclusion, a

subset will have been defined for the

new performance aspect.

2 12 DECISION/QUERY: The computer will not permit "casual"

does operator wish to manipulation/modification of its data

record for permanent base of taxonomies, systems, subsysstorage the new tems, performance aspect subsets, etc.

taxonomy subset just Any user can, at any time, define and

defined? operate on a new performance aspect

and taxonomy subset. But the computer will not permanently store those

data unless proper authorization is

received.

3 13 ENTER If the user desires to order permanent

AUTHORIZATION storage of a newly defined aspect of

CODE performance and its associated taxonomy subset, the user must type in a

special authorization code. This is not

the same as the SIGN-ON code. OnyC-11

Z,...."".t,.
EXPLANATORY NOTES

Page Block Block Functional -~4Number Number NaMeC Descri~t ion

3 13 special, designated users will have the

(cont'd) authority to add or delete data in the

permanent storage bases.

3 14 COMPUTER If the entered code is valid, the comnDECISION: is the puter will enter the newly defined

code entered by the performance aspect and its associated

user valid authoriza- taxonomy subset into permanent stortion for permanent age. Future users then will be able

storage? to access and use the subset. If the

entered code is not valid, no permanent storage will take place, but the

newly defined aspect and subset

remains available for use by this user

on this application.

3 15 DISPLAY: the struc- The computer presents the user with

ture and size of the a summary of the structure and size

taxonomy subset of the taxonomy subset associated

with the specified aspect of performance. This will entail a listing of

the numbers of hierarchies and individual taxa in each STM "column"

associated with the specified performance aspect. A hierarchy, in this

context, is a set of associated taxa, headed by one objectives-level taxon

and followed by its descendants on

the two lower levels, but including

only those descendants that are relevant to the performance aspect of

interest. For example, one relevant

hierarchy might consist of:

1.2

1.2.1

1.2.1.3

1.2.1.4

% 1.2.4

1.2.4.2

1.2.4.3

1.2.4.6

That particular example depicts a

hierarchy of performance potentiality

taxa (leading digit =1) containing 8

entries, namely, the second objectiveslevel "patriarch" (1.2), its first functional purposes-level derivative taxon

(1.2.1), that taxon's third and fourth

V characteristics-level descendants

C-12
EXPLANATORY NOTES

Page Block Block Functional

-Number Number Name Description

3 15 (1.2.1.3, 1.2.1.4), the "patriarch's"

(cont'd) fourth functional purposes taxon

(1.2.4), and that taxon's second, third,

and sixth characteristics-level derivatives (1.2.4.2, 1.2.4.3, 1.2.4.6).

For the proposed demonstration, a

typical size/structure message displayed to the operator might appear as

follows:

SYSTEM: TRAINING

SUBSYSTEM: CURRICULUM

DEVELOPMENT

ASPECT: TESTING

POTENTIALTIES: 5 hierarchies 42 taxa

PROCESSES: 2 hierarchies 10 taxa

PRODUCTS: 6 hierarchies 44 taxa

This information conveys a summary of the

scope of the measurement application

at hand, and provides a "feel" for the

distribution of measurement issues

among potentialities, processes, and

products.

3 16 DECISION/QUERY: The user may wish to examine the hierdoes the operator wish archies and their constituent taxa. The

to review, and possibly STM data base "claims" that those hieredit, the taxa hier- archies are relevant, in general, to the

archies relevant to the kind of measurement application at

specified aspect of hand. However, the user might decide

performance? that certain taxa, or even entire hierarchies, are not really pertinent to the

particular application and the particular

system with which the user is concerned.

Also, the user may have reason to

believe that other taxa deemed not

generally relevant are in fact of importance in this particular case. Thus,

each user needs to have the ability to

review and edit (through deletion or

addition) the taxonomy subset generally

associated with the performance aspect

to insure that the subset is "tailored"

to the user's particular application.

C-13

LI ,1 .. .. U,-,"" 4', *. '' - . , . - -/ : o , * r * ..* . .,.,-' . , ,. ....- , , ..
-- w r. trir i w-- - V

EXPLANATORY NOTES

Page Block Block Functional

Number Number Name Description

3 17 CONTEXTUAL If the user signifies a desire to review

CATEGORY and possibly edit the taxonomy subset,

SELECTION the computer inquires as to which contextual category of taxa the user

wishes to review first. The choices

correspond to the five columns of STM.

For the proposed demonstration, even

though all five options will "appear" to

be available, no taxa actually will be

in storage for the "Environment

Specification" or "General Constraints"

categories. Thus, the demonstration

user will be restricted to choosing one

of the three "Performance" categories

(Potentialities, Processes, or Products).

3 18 REVIEW (AND EDIT) Once the user indicates which contexHIERARCHIES tual category of taxa is desired, the

(sequential procedure) computer begins to display the hierarchies of the subset that belong to

that category. At each taxon, the

user will be able to command a deletion. Such action will drop the indicated taxon (and its associated

measures and measurement implica- - tions) from the set of taxa relevant

to the particular measurement application. The user also will have, the

option of commanding an addition, i.e.,

the option of defining a new taxon and

including it in the set of taxa for the

particular application. The user will

review the entire collection of hierarchies in this fashion, indicating (at

each taxon) whether to retain, delete,

augment, or replace the various taxa.

Thus, the hierarchies will be "tailored"

to the user's particular application.

3 19 DECISION/QUERY: Having completed the review/editing of

does the user wish to hierarchies in one column of the STM

,. review/edit the hier- (e.g., Potentialities), the user will have

archies associated with the option of performing the same

M. another contextual function for the hierarchies of another

category? column (e.g., Processes or Products).

3 20 PERMANENT Certain specially authorized users will

RECORDING OPTION have the capability of making permaROUTINE nent editorial changes to the data

bases of hierarchies. This will be done

4- C-14
EXPLANATORY NOTES

Page Block Block Functional

Number Number Nam_. Description

3 20 via entry of a special code number, in

(cont'd) a fashion similar to that described in

blocks 12, 13, and 14.

4 21 DECISION/QUERY: Up to this point, the user will have

does operator wish to 1) specified the system and subsystem

review, and possibly of interest, 2) defined the measureedit, the measurement ment application in terms of the speissues associated with cific aspect of performance of interest

the relevant taxa? and 3) selected the hierarchies of

taxa that are relevant to that measurement application. The next step is

to examine precisely what those taxa

portend for that measurement application, i.e., the issues and implications

for measurement associated with each

taxon. Of course, if a particular

taxon has been newly defined by the

user, the computer will as yet have

no associated issues for that taxon in

storage. Thus, the user may wish to

define such issues, and place them in

storage.

4 22 DESCRIPTION If the user signifies a desire to review

LEVEL SELECTION and possibly edit the measurement

issues associated with the selected

taxa, the computer inquires as to

which level of system description

(objectives, functional purposes, or

characteristics) the user wishes to

review first. Even though all three

choices will "appear" to be available,

the proposed demonstration will be

limited to only the Objectives and

Functional Purposes levels.

4 23 REVIEW (AND EDIT) Once the user indicates which level

ISSUES SETS of system description is desired, the

(sequential procedure) computer begins to display the taxa

of that level, along with the measurement issues and implications associated

with each taxon. The user has the

opportunity to edit those issues and

implications, by deletion, modification,

or addition. Thus, the user can select

precisely which issues will be addressed

in the measurement application at

hand.

C-15

. . .. . . . . . . . . . . . . . . . . . . .
SEXPLANATORY NOTES

Page Block Block Functional

Number Number Nam .. escription

i 4 24 DECISION/QUERY: Having completed the review/editing of

does the user wish to measurement issues in one row of the

Sreview/edit the meas- STM (e.g., Objectives Level), the user

urement issues associ- will have the opportunity to perform

ated with another the same process for the issues of

level of system des- another row (e.g., Functional Purposes

cription? level or, except for the proposed demonstration, Characteristics level).

4 25 PERMANENT Certain specially authorized users will

RECORDING OPTION have the capability of making perma- ROUTINE nent editorial changes to the data

bases of measurement issues. This

will be done via entry of a special

code number, in a fashion similar !,-t

that described in blocks 12, 13, ana 14.

4 26 DECISION/QUERY: This is the final, and most important,

does the user wish to step in the STM application. It is at

review, and possibly this point that a user extracts and/or

edit, measures defines specific measures suited to

applicable to selected the performance aspect of interest.

issues? By signifying a desire to review the

applicable measures, the user informs

the computer that access to the data - bases of measures is desired.

4 27 INDIVIDUAL TAXON The computer responds by requesting

SPECIFICATION the user to designate a relevant taxon

from the (previously identified andpossibly--edited) subset. The user

types in the taxon's number (two-tofour digits, depending on the level of

system description). For the proposed

demonstration, no measures will actually be in storage for taxa on the

characteristics level of system deseription. Thus, the demonstration user

will be restricted to entering two- or

three-digit taxa numbers.

4 28 MEASURES SET Once the user designates a taxon of

DISPLAY interest, the computer responds by displaying all of the relevant measurement issues associated with that taxon,

along with specific measures suitable

for addressing those issues.

c-16 , .
" oEXPLANATORY NOTES

Page Block Block Functional

Number Number Name Descriotion

4 29 REVIEW (AND EDIT) The user has the opportunity to edit

MEASURES the measures set asociated with the

sp zified taxon, through deletion, modification, or addition. Thus, the speS cific measures selected can be

"tailored" to the particular measurement application.

4 30 DECISION/QUERY: Having completed the review/editing of

does the user wish to measures associated with one member

j review/edit the mess- of the taxa subset, the user will have

ures set associated the opportunity to perform the same

with another taxon? process for another taxon's measures.

Again, for the proposed demonstration,

• Ionly Objectives level and Functional - Purposes level taxa actually will have

measures sets available for review.

4 31 PERMANENT Certain specially authorized users will

RECORDING OPTION have the capability of making permaROUTINE nent editorial changes to the data

bases of measures. This will be done

via entry of a special code number, in

a fashion similar to that described in

blocks 12, 13, and 14.

5 32 DECISION/QUERY: Once the user has reviewed/edited all

does the user wish to measures of interest to the perforaddress another aspect mance aspect previously specified, the

of performance for computer inquires whether the user

this same subsystem? wishes to consider some other performance aspect as well. An affirmative

response by the user causes STM to

branch back to block 7 on page 2.

5 33 DECISION/QUERY: If the user indicates that no more

does the user now work remains to be done in the conwish to address some text of the originally specified subother subsystem of system, the computer inquires whether

this same system? the user wishes to consider another

subsystem as well. An affirmative

response by the user causes STM to

branch back to block 6 on page 2.

5 34 DECISION/QUERY: If the user indicates that no more

does the user now work remains to be done with any of

wish to address some the subsystems of the originally speother system entirely? cified system, the computer inquires

whether the user wishes to consider

:: , ,-

C-17

. %"
- ." EXPLANATORY NOTES

Page Block Block Functional

Number Number Name Description

5 34 another system as well. An affirma-

(cont'd) tive response by the user causes STM

to branch back to block 3 on page 1.

5 35 SIGN-OFF Typical sign-off procedures for terminating work with an interactive

computer system. Permits storage of

data processing in user-controlled

memory files, initiates cost-accounting

processing, etc.

I xxx

jC-18

*,

'S--. ' C218.5 *