» 11 I M fc^^—^l^^l——~

^

Research Note 80-21

0

The Analytic Process Model for

System Design and Measurement:

A Computer-Aided Tool for Analyzing Training Systems and

Other Human-Machine Systems

CM

<

I

Q

<

Richard F. Bloom, John F. Gates, Jr.v

Ronald Q. Shapiro, and William A. Leaf

Dunlap and Associates. East, incorporated

Submitted by

ARI Field Unit at Fort Banning. Georgia

Seward Smith. Chief

Training Raaaarch Laboratory

Harold F. CNall, Jr., Director

DT1C

ELECTE

SEP 0 3 «86

U. S- Army

Research Institute for the Behavioral and Social Sciences

February 1985

Approvtd for public ralaatt; diitribution unlimittd.
·•· 

THIS DOCUMENT IS BEST 

QUALITY AVAILABLE. THE COPY 

FURNISHED TO DTIC CONTAINED 

A SIGNIFICANT NUMBER OF 

PAGES WHICH DO NOT 

REPRODUCE LEGIBLYo 
i&TiV&FttVV??^^*FTTrr

&T*^&mr^,T',*'r7*TrTzr*rTKTiri*T^*^

' ^

U. S. ARMY RESEARCH INSTITUTE

FOR THE BEHAVIORAL AND SOCIAL SCIENCES

A Field Operating Agency under the Jurisdiction of the

Deputy Chief of Staff for Personnel

EDGAR M. JOHNSON

Technical Director

L. NEALE COSBY

Colonel, IN

Commander

[ Accesion For f |

1 NTIS CRA&I

DTIC TAB

Unannoui ced

JiiStificaiiou _

D |

D \

3y

1 Dist ib LltiJl, /

vailabiiit r

A

y Codes 1

[Düt

Avail c

Sp.

i d/or "1

Cldl <

TWi rtport ht» bMn daartd for rtlMW to tht DtfanM Ttchnical Information Canttr (DTIC). It ha« baan |iv«n no

othar primary dittribution and will ba avallabla to raquattora only through DTIC or othar raftränet tarvieas mch

at tha National Tachnical information tarviot (NTIS). Tht «iom, opinions, and/or findings oontaintd in thn

raport art thost of tha author(s) and should not ba conttrutd as an official Dtpartmant of tha Army position,

policy, or dacision, unlatt so dnignsttd by othar official doeumantation.
ir^VCJi".* 'yrjf^y.'^ "* "»"'J *"• "^ '"- "- r^.-^.-«i.-'«T^BBiMB^j«rr

jr-T",.-

i

'i .^i^i.-'v^i-TATJr^irMr" A^.V •k"". k'1 BBBHBAuBaZJCiW»

Unclassified

UCUKITV CLAStiriCATION OP THIS PAOK («kan Bata t-frHO

REPORT DOCUMENTATION PAGE READ INSTRUCTIONS

BEFORE COMPLETING FORM

i. nran SDSHI

ARI Research Note 85-21

i. eevr AcctuiON NO. t.I NCC RKClPlKNT'l CATALOG NUMtKR

«. TlTLK fMtf MMita!)

The Analytic Process Model for System Design and

Measurement: A Computer-Aided Tool for Analyzing

Training Systems and Other Human-Machine Systems

• TVPC Of MKVORT • »KftlOO COVCNCO

Final Report

March 1980 - February 1983

• PKRFORMINO One. HKROMT NUMVCM

ED-83-3 (293-24)

•. CONTRACT OH ONANT NUMICMfa;

MDA903-80-C-0345

?. AuTNonr^) ~~ '

Richard F. Bloom; John F. Gates, Jr.;

Ronald G. Shapiro; and John W. Hamilton*

«0 PROGRAM CLCMCNT.^ROJtCT. TAtK

ARK A • WORK UNIT NUMBCRt

2Q263743A794

1. rCRrORMINOOROANlZATION NAMC AND AOORKM

Dunlap and Associates East. Inc.

17 Washington Street

Norwalk. Connecticut 06854

1«. CONTNOLLINO OPriCC NAME AND AOORH«

U.S. Army Research Institute for the Behavioral

and Social Sciences

5001 Eisenhower Avenue. Alexandria. Virginia 22333

U. MONITORINO AeiNCY MAMK • ADDNUIfll «IfafMit BS 855383 OWlMj

U. REPORT DAT!

February 1985

It. NUMtKR OP PACK»

17A

U. aKCURITV CLASS. (•! «Ma

Unclassified

Id! DKCLASSiriCATIOM/OONNORAOINC

»CMtOIULKI«. MSTNiauTlON STATEMENT r«f«»laRRpaf«> '^~

Approved for public release; distribution unlimited

H. MSTWauTlON STATEMENT (»I »a akalrMl anfaratf la Bttk 10. If «Ifaranf «TMH ft«M*J

I«. MPPkEMBMTARV NOTES

*One chapter contributed by Dr. Frederick A. Muckler, Canyon Research Group, Inc.

(now Essex Corporation). Seward Smith, Contracting Officer's Representative

is KEY «OROt CCMKIMM an ravwaa al* l/Maaaaatr mtt lO-ttllr Sp *<•«* ****•')

Human-machine system; analysis; taxonomy; model; training systems; design

requirements; design specification; evaluation; performance measurement; effectiveness measurement; system populations; Bradley Infantry .Fighting Vehicle; BIFV;

Analytic Process Model; APM; systems taxonomy model; computer-aided model;

interactive automated model.

\

VThe objective of this model development effort was to provide a uniform, thorough,

adaptive and efficient procedure to help derive design specifications and effectiveness measures for any planned or existing human-machine system, and especially for

a trainii* system. The present report describes results of almost three years of

work, in which the analytic process model (APM) was developed from earlier models,

applied in sample fashions to an existing system (the Bradley Infantry Fighting

Vehicle) and placed onto an Apple n computer to demonstrate its automated.

(continued)
*" M in ITimir' --- -TI ^- ^ *.-!*.-. KJ- «* »r ».-»->

Unclawified

MCUWITV ctAmriCATtow or TMI» g^gggfcjg ggj «gjg^ iö tt'h

/

20. Abstract (continued)

'interactive potential. The APM guidelines and procedures are also documented here.

Recommendations are made for enlarging and completing the model and for its

dissemination to Army users. A separate Operations Handbook was issued for the

computer-aided APM demonstration system.

\
I I

TABLE OF CONTENTS

I. EXECUTIVE SUMMARY 1

II. THE ANALYTIC PROCESS MODEL 7

A. The APM Concept and Definitions 7

B. Applications of the APM for Measurement 35

III. APM APPLICATION PROCEDURES 71

A. General 71

B. The Computer-Aided APM 73

IV. RELATIONSHIP OF THE APM TO OTHER MODELS 80

A. Bases of Comparison 80

B. System Development Procedures 81

C. Human-Machine System Development 82

D. Training System Development 83

E. Behavioral Simulation Models 85

F. Logistic Support Analyses 86

G. Test and Evaluation 86

H. Summary 87

V. RECOMMENDATIONS FOR FURTHER DEVELOPMENT

AND APPLICATIONS 88

A. Data Base Development 89

B. Procedures Development 91

C. Applications and Application Aids 92

D. Administrative Items Development 94

VI. SELECTED BIBLIOGRAPHY 95

VII. ANNOTATED GLOSSARY OF ANALYTIC PROCESS

APPENDICES;

90

A. Training Design Subsystem Data Base A-l

B. Training Enabling Subsystem Data Base B-l

C. Training Delivery Subsystem Data Base C-l

Hi
VS^" ." '." • '

_

'.^ • '.^';^T.'n*i T•."T•.T,r'1».'r•?'^.''. ^A".''."".'."^'."^^.".''.".^."-'."'"." '.' '-'•' " '•'»' '.'<»MHl^P"^P"«i*"****"^',^P^lB^"^"«^BM«

LIST OF FIGURES

Page

1. Overview ot the APM for System Design and Evaluation 2

2. Sample Illustration of the Sequential Steps in Applying

the APM 4

3. Simple Printout of Analysis Results 5

4. Four Stages of the APM 8

5. Sample 2-Dimensional STM for the liradley Infantry

Fighting Vehicle (Carrier Team Subsystem) 11

6. Example of a System Hierarchical Structure 14

7. Guidelines for Identifying Dimensions/Taxa of System

Performance 16

8. Sample Set of Attributes Associated with System

Work Requirements 17

9. Training Subsystems and Their Operators 21

10. Objectives-Eevel Performance Taxa of Training Subsystems 23

11. Sample Attributes Associated with a Taxon of

Training Design Subsystem Performance 26

12. Schematic Representation of a General Hierarchy of

Measurement 28

13. Mission Scenario (First page only for sample purposes) 38

14. Verbal Squad Order 39

15. Carrier Team System Taxa Associated with "Surveillance11

Performance Element 42

16. BIFV Surveillance: Potentiality-Oriented Measures 44

17. BIFV Surveillance: Process-Oriented Measures 45

18. BIFV Surveillance: Product-Oriented Measures 46

19. Major Systems Interacting with BIFVTS 49

20. Major Systems Interacting with the Basic BIFV Gunnery

Training System 52

^r
PI|ppjif.fU*fVHl

ffVl«Mi,WWmn««'Tr«reTOVan«iiun*Ty*«HBa^Bi^BiHHaaHB«

• •

LIST OF FIGURKS (Continued)

Pace

21. Sample Hierarchical Structure Centered on a Subsystem

of a Training System 53

22. Measurement Issues and Measurable Attributes Relevant

to Evaluation of BIFV Gunnery Testing Specifications 59

23. Sample Measures Hierarchy Number 1 66

24. Sample Measures Hierarchy Number 2 67

25. Sample Measures Hierarchy Number 3 68

26. Sample Measures Hierarchy Number 4 69

27. General APN! Application Procedures 72

28. Simplified Flow Chart for Computer-Aided APM

Demonstration System 76

29. Comparison between the Ultimate Computer Package

and the Demonstration Package 78

30. Recommended Further Research on the APM 90
TlWT*JiK'%

-*f'*rTrrrm'?wrwwmmimmm^mm^immm^mmmm^^m^mtmmmam^imn

1. EXECUTIVE SUMMARY

This report describes an analytic process model (APM) for systems design and

measurement. This model provides a uniform, thorough, adaptive and efficient

procedure to derive design specifications and effectiveness measures for any

human-machine system, including any training system.

The APM is designed to enable testers, analysts and researchers to define sets

(or taxonomies) of system performance requirements, and to translate then into

measures and/or design requirements. It forces one to describe the system of

interest and its human elements in terms of a more complete set of

system-human attributes pertinent to system effectiveness. The general APM

is illustrated in Figure 1. An interactive demonstration was developed to

illustrate the potential for a computer-aided APM to generate effectiveness

measures (Figure 1, Blocks 1, 2 and 4) for training systems in particular. It

provides general "menus" of performance items and procedures to help translate those items into appropriate evaluation measures. Work with this

demonstration computer-aided model suggests that similar routinized programs

can also be developed for other particular kinds of systems (e.g., communication, transportation, weapon, etc.).

This project addresses the problem that testers, analysts and researchers too

often use an incorrect, incomplete or inappropriate set of human performance

factors in evaluating or specifying a human-machine (e.g., training) system.

Those are usually known measures and traditional design choicer often

selected without adequate consideration of the system qua system. They may

not ask the essential questions about system performance and effectiveness or

they may provide inadequate answers to these essential questions.. Because

no verified analytic process for deriving (or specifying) the optimal measures

of a system's performance or effectiveness exists, true assessment needs are

difficult to define and the process is relatively haphazard. The typical

solution is to test/measure/specify the easy and accessible system points, but

not necessarily those that should be addressed. Without having more systematic procedures, people measure or specify what can be counted (e.g., POI

hours), or observed (e.g., number of troops trained). They design written

tests to assess the learners' rote memories rather than their understanding,

and use criteria such as end-of-course tests rather than on-the-job performance. Rarely do people know the relationship between test performance and

job performance, or between soldier job performance and unit effectiveness.

This results in wasting valuable resources (time, effort, talent, money),

failing to provide adequate answers to effectiveness questions, and losing

important facts regarding human contributions to system performance. The

APM is designed to be a better method for deciding what should be measured,

or designed, and how.

The recent development and tryout of the APM were performed within the

context of the Bradley Infantry Fighting Vehicle (BIFV) development program,

a complex man-machine and training system. The following products have

resulted: an outline APM (Figure 1); a de.ailed STM (Figure 1, Block 1) and

associated procedures for deriving dimensions or taxa of system performance;
Q)

i Determine The

System's Work

1 Requirements

| SYSTEM

PERFORMANCE

| TAXONOMIES

1

1

g

© j

Design

j Determine What

The System Needs

To Do Its Work Evaluation

® ,

SYSTEM

' ATTRIBUTES

«o

i ® .

i

Determine How

The Needs Can

Best Be Provided

Determine How

Best To Judge If

The Needs Are Met

SYSTEM

DESIGN

SPECIFICATIONS

SYSTEM

MEASUREMENT

SPECIFICATIONS

® ^ ® ]

Implement Steps

To Meet The

System's Needs

Implement Steps

To Judge Satisfaction

Of The Needs

s

SYSTEM

DESIGN

SYSTEM

TEST

1 ©

£

Implement The

Operational

System 1»a

a.

s

SYSTEM

ASSEMBLY/

MODIFICATION

® 1

Ui

1

Perform The

Required

Work |

SYSTEM

OPERATION 1

Figure 1. Overview of the APM for System Design and Evaluation
^»^»■■liMMKrrwT'jwwvFVTj^^iOT'VVi-r.».^^ \.> ww-irv»^

initial procedures for deriving attributes (Figure 1, Block 2), design

specifications (Figure 1, Block 3) and measures (Figure 1, Block 4) from the

taxa; identification of taxa for the surveillance function of the BIFV; definition

of what functional suhsystoms constitute a training system; partial or, in some

cases, complete identification of taxa for the subsystems of the BIFV training

system (BIFVTS); and a demonstration computer-aided APM program.

Although the sample taxa have been developed for the BIFV, they appear to

he generic and applicable to any manned system.

The APM application is illustrated in Figure 2. This shows how an analyst

could produce effectiveness measures using the computer-aided model. The

procedure consists of branching through a series of nested sets. The analyst

chooses from the computer's data base of recommended options or creates new

options at each branch. Special instructions and help are available on-line and

in an operating manual to guide the analyst through the model. Hard-copy

printouts of the selected options are available immediately upon request at key

stages of the analytic process. Figure 3 is a sample of such a printout,

including a portion of the Figure 2 illustration. The sequential digits in the

index numbers of Figure 3 correspond to the analytic items and levels illustrated in Figure I (Aspects, Objectives, Functional Purposes, Characteristics,

Attributes and Measures, respectively). As seen in Figure 3, attributes and

measures can be derived from any of the three analytic levels (Obj-'ctives,

Functional Purposes or Characteristics). Consequently, zeros can appear in

the third (Functional Purposes) or fourth (Characteristics) positions of the

index number. The header information and index numbers provide a means

for tracing the connection between any selected performance measure and its

determining prerequisites. One can thereby insure that all system objectives

are addressed by derivative evaluation measures and that all measures are

justified.

As described in Chapter TV, the APM compares in several important wavs with

other models for evaluating or developing training systems and ether

human-machine systems. For example, it is totally consistent with the general

objectives of MIL-H-46855B ("Hunan Engineering Requirements for Military

Systems, Equipment and Facilities"), by providing a general, sequential method

by which human-machine systems can be developed in a total systems context

and with careful emphasis on the human element. It complements a variety of

methods grouped under the generic term Instructional System Development

(ISE) by facilitating the front-end analysis of training systems. ISD

frequently comes too late to identify some potential training problems, whereas

the APM can be applied earlier when such issues as performance goals,

mission, objectives, functions, tasks, job analysis, validation and logistic«?

must be first addressed. The broader and earlier analysis provided by the

APM can set the stage for the detailed processes associated with ISD. The

APM is also complementary to the Early Training Estimation System (ETES).

The APM is of special value in facilitating the generation of system and

subsystem objectives and the development of performance measures, while ETFS

is of particular value in evaluations of media and instructional methods. The

APM is compatible with various methods for Logistic Support Analysis (LSA),

in that earlier estimates of logistics requirements can be prepared with the aid

of the APM. Because the APM can be applied to developing systems in which

operating parameters are still evolving, it also can be useful for generating
JÄTV^V*"»' \ » .frTTTV^BBBBBM« ' AnVlKn'V^aiWUSW^^iM^Hl^^MlB^WBRaBBl^l^^^ - .• ..- v~ ^' B^B^OH

APM

Step

No.

1.

Analytic

Determination

To Be Made

Purpose in

Applyir« the APM:

Application:

Options Selected for this Illustration

To help evaluate the BIFVTS design subsystem's effectiveness in specifying

gunner learner testing.

I

Design Specification

|

Performance Evaluation

(Measurement)

3. System Class:

4. System:

5. Subsystem:

6. Aspects:

7. Objectives:

10.

11.

Functional

Purposes:

Characteristics:

Measurable

Attributes:

Measures:

Communication Training Weapon Transportation

1 1 1

Handgun BIFVTS 155-mm FA

Training System Training System

j 1 1 1 1 1

Command Logistics Emplacement Design Enabling Delivery

I I l' 1 1

1. Potentialities 2. Processes 3. Products 4. Environment 5. Constraints

i —i 1 1 r 1—

1.2. The subsystem must be capable of; establishing performance

objectives.

[ ' 1 1 1 1 1

T

1.2.1. This subsystem capability allows: stating expected performance

and behavior after training.

—I ' 1 1 1

1.2 1.1. For this purpose, the system must have the potential for:

defining the performance action.

—j 1 ! ! , ! !

1.2.1.1.1. To evaluate effectiveness in meeting this characteristic, the

following system attribute(s) can be measured: types of B1FV

gunner performance actions intended to be defined.

—I * 1 1 1 1 1

1.2.1.1.1.1. This system attribute can be analyzed by comparing the

following parameter(s) with established criteria: relevant

types of actions not intended to be defined.

Figure 2. Sample Illustration of the Sequential Steps

in Applying the APM

-4
TW————^—^p—^ ii < ! PIWIIH ■■ii i in» mm ■■ wmi^m^i^m^immmu

Finol Sumniorx Rvportr

Fvbruorv 28» 1983

11 1 cist. rot. 1 on of Anolv*is Prlrftout.

MVOKU n«ni»n i. Purposes To tvoluäte trommg dtsign «ubtyttM «fftctivfnvit in

tptcifying Ivorntr tttting

Sv»-t,»iTi Closst TroiningtO

Syst-emX frodlty Infantry Fighting Vthiclc Training Syttn (BirvTS)[l]

Subsyst,»iti t Dt»»gn[33

1.0.0,0.0.0. PotentiolitittS Tht tyttm »utt bt copablt of!

1.1.0.0.0.0« Identifying goals ond prioritiot

1.1.0.0.1.0. Scope of inforaetion ovoilable concerning (IFVG operations

1.1.0.0.2.0. Staff experience and qualification« in identifying training goal»

1.1.0.0.3.0. Planned approach to identify PIFVG training goals

1.1-0.0.4.0. Resources allocated to identifying HFW training goals

1.1.1.0.0.0. Defining the total scope of learning

1.1.1.0.1.0. Aspects of each gunnery training goal to be defined

1.1.1.0.2.0. Plans for defining the total scope of IIFV6 learring

1.1.1.1.0.0. Identifying types of ochieveaents relevant to the intended job

1.1.1.1.1.0. Types or classes of achieventnt intended to be identified

1.1.1.1.2.0. Plans for identifying all types of BIFVG achievetients

1.1.2.0.0.0. Stating the ultieote intended outcomes of learning

1.1.2.0.1.0. Cleeents of ultieote outcoaes that ore intended to be stated

1.1.2.0.2.0. Intended fomats to state ultieot» intended outcomes of IIFVG Irng

1.1.2.0.3.0. Plans for stating ultioate intended OJUOMS

1.1.2.1.0.0. Petereining necessary levels of ochieveeent

1.1.2.1.1.0. Eletents or factors of necessary levels intended to be deternined

1.1.2.1.2.0. Plans for deternining necessary levels of FIFVG achieveeent

1.1.2.2.0.0. Detereining existing levels of ochieveeent

1.1.2.2.1.0. Elenent» of the current levels of achicveeent intended to be tested

1.1.2.2.2.0. Plans for testing candidates' current levels of MfVG achievement

1.1.4.0.0.0. Establishing a basis for specifying learning objectives

1.1.4.0.1.0. Clesents of the basis intended to be established

1.1.4.0.2.0. Plans for establishing the basis for specifying MFVG perfor» obj

1.1.4.1.0.0. Insuring availability of a hierarchy/taxonoey of objectives

1.1.4.1.1.0. Goal taxa intended to be employed

1.1.4.1.2.0. Plans for eeploying the toxa to classify goals for BIFVG training

1.1.4.2.0.0. Stating goals in teres that ieply behaviors

1.1.4.2.1.0. lehaviorol tereinology intended for use in stating IIFVG goals

1.1.4.2.2.0. Intended foroats for stating BIFVG training goals

1.2.0.0.0.0. Establishing perforoance objectives

1.2.0.0.1.0. Scope of inforoation ovoilobe concerning ilFVG task

1.2.0.0.1.1. BIFVG required tasks that ore known (list t descrb each)

1.2.0.0.1.2. BIFVG required tasks that ore unknown to Design Subsystesdnt ea)

1.2.0.0.1.3. Tasks that ore not reqd of BIFVG but Design Subsyt classif at reqd

1.2.0.0.2.0. Staff experience end qualifications in establishing perfor» ob.jctvs

1.2.0.0.2.1. Revwrs cue ratings of ea Design staff qual to estab trng perf ob.j

1.2.0.0.2.2. Revwrs overall ossessMnt of total Design «toff qual

1.2.0.0.3.0. Planned approach to establishing BIFVG performance objectives

1.2.0.0.3.1. Essential steps »issing froo plans (list I describe ea)

1.2.0.0.3.2. Inessential steps included in plans (list t describe ea)

1.2.0.0.3.3. Inconsistencies/deficiencies in sequencing steps in plans

1.2.0.0.3.4. Reviewers cue rating» of soundness/workability of plans

1.2.0.0.4.0. Resources allocated to establishing BIFVG performance objectives

1.2.0.0.4.1. Z of reqd person-hrs proposed for alloc to establish objectives

1.2.0.0.4.2. X of reqd motenoU/good» proposed for alloc to establish objctvs

1.2.0.0.4.3. I of rtqd support services proposed for alloc to establish objctvs

1.2.0.0.4.4. Ratio of • proposed to needed for establishing objectives

1.2.1.0.0.0. Stating expected performance ond behavior after training

1.2.1.0.1.0. Elements or factors of gunner abilities intended to be stated

1.2.1.0.1.1. Essential elements of abilities to be Ind in stmnt of BIFVG ablty

Figure 3. Sample Printout of Analysis Results

-5-
lum'jiMuii'ww.M *.'i ^i »j'"

early inputs (objectives, parameters, measures) to the Army's Human

Resources Test and Evaluation System (HRTES). While the APM provides a

measurement framework for the HRTES technique, HRTES extends APM further

into the operational testing domain.

Now thnt a computer-aided demonstration system of the APM is available,

Chapter V considers how it can be improved. The demonstration system can

now be used to develop evaluation measures for three of the six training

subsystems: Design, Enabling and Delivery, but not Command, Logistics or

Emplacement (see Appendices A, B and C). It is essential to develop the taxa

of subsystem performance items, subsystem attributes and attribute measures

for the three remaining training subsystems to use the APM fully for

evaluation. Procedural guidelines must be developed for generating design

specifications from the performance taxa to use it in developing design

specifications. More complete computer "help" instructions must be made

available on request. Eventually, the model must include data bases (taxa) for

more than training systems; computer assistance should be tailored to

communications systems, weapon systems, transportation systems, and many

others of interest to particular users. The package could also incorporate a

routine for costing guidelines to aid in early evaluation decisions. Clearly, if

these capabilities are added to the computer-aided APM, it will require a

computer of significantly greater capacity than the present Apple II Plus which

barely contains the sample demonstration program.

-6-
.^W^^^AV^^J/V^'^'J"'^/^

II. T1IL ANALYTIC PROCESS MODEL

The APM is a conceptual framework for systematizing the process of designing

and evaluating human-machiic systems. Impetus for APM development is the

experienre that system design and evaluation often proceed haphazardlv.

Design and measurement decisions are frequently formulated out of familiar

practice, without due consid nration of new approaches that may be preferable.

Conversely, designers and evaluators sometimes adopt emerging technology or

methods because they arc no-el, without adequate regard for what is best

suited to the application at hand. In cither casp, the system may be built

and/or tested incorrectly, Incomj'letely or inappropriately. This result'- in

misleading evaluative findings, a .d systems that do not work as well as they

should.

The fundamental concept behind the APM is that the haphazard design and

evaluation typically begin with a failure to consider fully the system's intended

work. That is, designers often start to build systems, and evaluators often

start to test them, without a clear understanding of what the systems are

supposed to do, and where, when, why and how they are supposed to do it.

The APM Ir designed to remind (or force) designers and evaluators to consider

all relevant factors.

A. The APM Concept and Definitions

Based on the original work of Finley et al. (1975, 1976) and extended by

Bloom et al. (1981, 1982), a four-stage functional sequence was defined as the

top level (or macroscopic) outline of the APM. This is illustrated in Figure 4,

with a cross-reference to the blocks of Figure 1.

Stage 1, the contextual stage, is the Systems Taxonomy Model (STM) as

initially defined by Finley et al. (1975). It is an organized conceptual

framework and a set of data bases and guidelines that assist an analyst in

determining what the system of interest is supposed to do and why, how,

where and under what circumstances it is supposed to do it. The STM is a

model for taxonomization since it provides a means of generating dimensions

which would be taxa in that they could be used for syr.tems classification and

as generic terms for analysis to develop measures of effectiveness and

performance. The STM has three dimensions:

0

/. >pects of system performance (of which there are five).

Levels of system description (of which there are three).

0

System hierarchical structure (which can contain any number of

suprasystems, subsystems and collateral systems).

Stage 2, the focal stage, includes analytic procedures for translating the

required work into corresponding system attributes. This stage also includes

procedures for determining how to insure the presence of those attributes, and

for determining whether they are present to the appropriate degree. The

outputs of Stage 2 are the basic tools of a system evaluation application

(measures of performance/effectiveness) and/or of a system design application

(design specifications).

-7-
HMHBMHHMBn*«»1*' F » ' i

111 nn >» Y w nirrrrrsvT'vrrf^nvrvifwr^jv.'.» r* -*.'.» v "> -.

Stage 1

Establish the

Work Context

Stage 2

Focus on the

System Features

Stage 3

Implement the

Application Details

Stage 4

Interpret, Conclude

and Recommend

Corresponding Blocks

in Figure 1

©

2,

®

®

®

®

(!)

Figure 4. Four Stages of the APM

-8-
L'JrfrJ rs -f "-•' ^-^.T "f ri&.nr r*nr ^.nr.nrj-y immmmmm^mmmmmmmmmmmmmmmmmwmmmmm 11 WJT»wma rvirmwmjrirwH " • ^ «"»n »r » T »- -^ Wi/-\ »

Stage 3, tho implementation stage, consists of methodology for employing

the application tools. For a design application, Stage 3 is where the system is

actually built, in accordance with specifications. For an evaluation application.

Stage 3 is where the system is actually tested, using the measures.

Stage 4, the interpretation stage, includes analytic procedures for

determining whether the system satisfies its work requirements and, when

deficiencies exist, for determining how the system must be modified to correct

the deficiencies.

To date, the key concepts have been developed for Stages 1 and 3.

They are described in the following paragraphs.

1. Aspects of System Performance

A system's work is broadly defined by what it is supposed to do and

where it is supposed to do it. To support design or evaluation, the "what" of

performance can be divided into three mutually exclusive classes, or aspects:

c

Performance potentialities--the capabilities for doing work that

the system is supposed to possess;

0

Performance processes—the activities that the system is

supposed to carry out;

Performance products—the outputs that the system is supposed

to deliver.

A system could fail at any of these three aspects of performance. Tt

could simply not have the prerequisite abilities needed to get its job done; it

might have the ability, but fail to implement the necessary procedures; it

might have the ability and carry out the procedures, but fail to deliver the

required goods and services in the quantity and quality needed. Too often,

system evaluators concern themselves only with the output, or product, aspect

of system performance. But if the product is deficient, no corrective action

can be taken unless the evaluation can diagnose the process and potential

performance to uncover the root cause of the failure.

The "where" of performance similarly subdivides into two mutually

exclusive aspects:

Performance environment--the naturally occurring set of

conditions and circumstances under which the system must

provide its capabilities, carry out its processes, and deliver its

products.

Performance constraints--the man-made or artificial limiting

factors that apply to the system's potential, process and

product performance.

System designers and evaluators always must consider the real world, where

the system actually will be used. It isn't good enough that a system can

provide its required abilities, carry out its processes and deliver its products

-9-
under the most favorable circumstances, or without regard for cost, safety,

time, or other considerations. Performance under actual operational conditions

and in full compliance with all necessary restrictions is demanded of every

system.

2. Levels of System Description

Three levels of system description are defined:

0 Objectives

0

Functional Purposes

Characteristics

The "what" and "where" of system performance provide a sterile or

abstract view of a system's work. It is a view of work as something that

merely is, without regard for its purpose. This is the Objectives-level of

work definition, i.e., the performance elements are defined as objects or ends

in themselves. Quite different types of systems may have similar work

requirements at the objectives-level of description. The process known as

"Task Analysis," for example, is required in: tool desigt: systems, training

development systems, personnel evaluation and management systems, and many

others.

Rut systems do not perform work merely for its own sake. There

always is a purpose behind a system's work, a function to which each work

element is to be applied. Otherwise, the work really is not required.

Furthermore, if the system works to accomplish purposes, there must be a

method to its work. That is, it must approach its work requirements in

characteristic ways which insure that the purposes will be met. These

observations lead to definition of two other levels of system description. The

Functional Purposes-level corresponds to the "why" of performance: the

reasons For tnc existence of objectives-level work requirements. The

Characteristics-level corresponds to the "how" of performance: the

characteristic ways in which the system provides its potentialities, carries out

its processes, and delivers its products to insure that their purposes are met.

3. Interaction of the First Two Dimensions: Taxa of System

Performance

The five aspects of system performance interact with the three levels

of system description to form a fifteen-cell matrix. Each cell represents a

particular taxonomy of work requirements, each member of which is a

particular taxon or dimension of work. As an example, Figure 5 arrays the

work requirements taxonomies for the Carrier Team Subsystem of the Bradley

Infantry Fighting Vehicle, and exhibits sample member taxa for each cell.

Within each column in Figure 5, the member taxa are hierarchically

linked from cell to cell. In the topmost cells, general statements of the "what"

and "where" of performance are found. For example, the system is supposed

to have the potential for "providing surveillance" (taxon 1.1), and is supposed

to do so in a "rolling terrain" environment (taxon 4.1). In the cells of the

middle row, the purposes behind the required work are stated. The reasons

for the potential of "providing surveillance" include having the capabilities of

-10-
FFTrxrtFTTJVTrTT'rrrr W*} ''- ^ ""-"Ivr"'-7."r\' V^-'^^-V7.T

r^^".A.Tr^-Arjt\}n%icn*T\k?.,-»-v- , „'- .-»-r-. -VJ

0

5.1 Communication

constraints

etc.

5.1.1. Communication

security

5.1.2. Contact

information

exchange

etc.

5.1.1.1 Avoid radio

transmission

prior to

contact

5.1.1.2 Maintain

light and

noise discipline

etc.

1 -' c

i

0

E

P

>

u

4.1 Rolling terrain

etc.4.1.1 Hills

4.1.2 Valleysetc.

4.1.1.1 Maximum

upslope of

eight

degrees

etc.

(A—'

1

D.

3.1 Squad bounds3.2 Threat detections

etc.

3.2.1 Enemies

located

3.2.2 Intelligence

information

provided

etc.

3.2.2.1 Information

regarding

what was

observed

3.2.2.2 Information

regarding

how many

were

observed

etc.

*-

CL

2.1 Carry out overwatch bounds

2.2 Engaged identified threats

etc.

2.1.1 Bound to

secure forward position

2.1.2 Enable platoon

movement

etc.

2.1.2.1 Assess

security of

bound position

2.1.2.2 Communicate

security to

overwatch

element

etc.

i M

•-

c

—'

du

1.1 Providing surveillance

1.2 Conducting

weapons fire

etc.

1.1.1 Conducting

sector surveillance

1.1.2 Detecting

targets

1.1.3 Acquiring

targets

etc.

1

1.1.2.1 Unaided

visual target

detection

capability

1.1.2.2 Aided visual

target detection capability

1.1.2.3 Infrared sensor target

detection

capability

etc.

S3A!)03fqo sasodjiitj iBuoi;ounj sonsusioojBqo

>>«-

-•

c

33 «~,

C

— *

>t

<1) c

•o 0) <o **

c in

CO >>

1

&

a

a

E

35

<U

s H

H t_

Vi 0)

<—t t.

CO

c

o

CO Ü

(/>

c a>

V

5

«J

S

a ii

i >

e^

be

0) c

n, *->

E

a] tf CJin

2

3

.a>

NOIldlHOSHQ IMHISAS iO Sn3A3T

-11-
Wjvr,'

r

^'>r^fr "^^y ".K^>^>^t ^.w^v^v^>,^'^\f^^^v"^'^r^'.^y'.v^r^'.^ij^\-^\^v^v^,r

g

v

-

»v^t^^^v^'r^T^'.'^v^

"conducting sector surveillance" (taxon 1.1,1), "detecting targets" (1.1.2),

"acquiring targets" (1.1.3), and possibly other purposes. Further, the

system is required to be able to achieve those purposes while operating on

hills (4.1.1) and in valleys (4.1.2). The decimal numeric notation links the

various funrtional-purposes-level taxa with the individual objectives-level taxon

they support. Taxa 1.1.1, 1.1.2, and 1.1.3 all support or "descend from"

taxon 1.1.

The colls of the bottom row state how each purpose is to be

satisfied. For example, the capability of "detecting targets" will be met bv

providing the potentials for "unaided visual target detection" (1.1.2.1), "aided

visual target detection" (1.1.2.2), "infrared sensor target detection"

(1.1.2.3), and perhaps through other capabilities as well. The decimal

notation traces the vertical path from cell to cell. Thus, characteristics-level

taxon 1.1.2.3 descends from functional-purposes taxon 1.1.2, which in turn

descends from objectives taxon 1.1.

As Figure 5 illustrates, the matrix resulting from the interaction of

the first two dimensions of the STM provides a framework for organizing:

The basic work requirements for which the system is intended,

including its potential, process and product aspects and the

natural and man-made factors impeding those aspects;

0

The purposes behind each work requirement, and the

environmental and constraining components that bear on those

purposes;

The ways in which the system is supposed to achieve its

purposes, and the factors that may impede those ways.

4. System Hierarchical Structure: The Third Dimension of the STH

Systems seldom work in isolation. Usually they interact with other

systems, so that the work they do influences, and is affected by, what other

systems do. Designers and evaluators need to account for these other

interacting systems so that all of the requirements of the svstem of interest

may be identified and so that appropriate interfaces may be designed and

appropriate evaluation controls may be established. Three kinds of interacting

systems can exist:

Larger systems, of which the system of interest is a part, and

superior systems exercising command and control over the

system of interest. Collectively, these larger, superior systems

are termed suprasystems. It is often at the suprasystem level

that the work requirements of a system of interest are

established.

Smaller systems wholly contained within the system of interest,

and systems over which the system of interest exercises

command and control. Such smaller, subservient entities are

called subsystems. The ability of a system to satisfy its work

requirements oTten depends on the subsystems' abilities to

perform as they are supposed to do.

-12-
r#CW^WTr^^rV^77^FriK*CWWWWrW7W^7V7^J'^,

^VJV^^^ .". - -r n* -k-j-.- nrrvsr.rvjv..^VTI

0

Systems that exist and operate on the same level of command

and control as does the system of interest, and with which the

system of interest shares resources or exchanges inputs and

outputs. These equal ranking, interacting systems are termed

collateral systems. A system of interest must be designed to

interlace compatibly with its collateral systems.

Figure 6 illustrates the concept of system hierarchical structure in the context

of the Bradley Infantry Fighting Vehicle. The BIFV is shown as subservient

to one guprnsystem, namely, the mechanized infantry platoon of which the

individual vehicle systi m is a member squad. It has two subsystems, namely,

the Carrier Team Unit (Track Commander, Gunner, Driver, the vehicle control

apparatus, the vehicle-mounted weapons, etc.), and the Crew Compartment

Team (Firing port weapons and firing port weapon operators). Four collateral

systems are shown. Three of these are the other individual fighting-vehicle

systems that make up the platoon (including the particular vehicle that

happens to house the platoon headquarters). Another collateral system is the

enemy armored-vehicle system, i.e., the threat. In the operational

environment, the BIFV derives its work requirements from the platoon's

orders, translates those requirements into the corresponding responsibilities of

its two subsystems, and carries out its duties in concert with (and without

interfering with) its sister squads. And undeniably, it carries out its duties

in concert with the threat system, since those duties include detecting,

acquiring, identifying, tracking, engaging, and neutralizing or destroying the

threat. The relationship between the BIFV and the threat is certainly

antagonistic, but equally certainly interactive and thus collaterd. If the BIFV

is not designed with the threat's capabilities and work requirements in mind,

the BIFV likely will fail to achieve its purposes. In an engagement situation,

the outcome may depend not only on how well the BIFV performs its intended

work, but also on how well the threat does its.

5. Summary of the 3-Dimensional STM

To recapitulate briefly, the STM is the first stage of the APM. It is

the stage at which a svstem's work requirements are identified and organized

so that they will be fuily considered in the design and evaluation of that

system. Identificaticn jf the work requirements demands that the analyst

examine not only the system itself, but also its operational environment,

practical constraints, vrd the performance factors of other systems with which

it interacts. In organizing "what" work the system is supposed to accomplish,

it is necessary to consider not only the products, goods or services to be

delivered, but also the processes intended to lead to those products and the

potentialities needed to carry out those processes effectively. Further, for

each element of required work, it is necessary to identify all of the purposes,

or reasons, why the work is needed, and exactly how those purposes are to

be achieved. Finally, each system of interest must be examined in the context

of the hierarchical system structure in which it is necessarily embedded—the

supra-, sub- and collateral systems.

6. Guidelines for Identifying P'-nensions/Taxa of System Performance

Through experience gained in a series of trial applications of the

APM concept, guidelines have evolved for identifying a system's work

requirements within each cell representing interaction of a particular aspect of

-13-
I———————— !■■ MmIIII I i I M I I i| MM I III I' III I 'I——will in i in ! i inn IT« \~K I^VM

SUPRASYSTEM 1

BIFV |

Platoon

1 COLLATERAL

SYSTEM

[ SYSTEM OF

INTEREST

BIFV

Squad

Armored

Threat

(Target)

COLLATERAL

SYSTEM

BIFV Squad

and

Platoon Hq.

COLLATERAL"!

SY COLLATERAL

SYSTEM

BIFV

Squad

SUBSYSTEM

Carrier

Team

SUBSYSTEM

Crew

Compartment

Team

Figure 6. Example of a System Hierarchical Structure

-14-
r^ryr.« r?-ji —»'.JI-H ".«-,.-k " vT-ir-v"^ " WFV "« • --^w« ir^'y .TIWTIWI" UT ^» t> .^k^ .•.»'.• i. •. -u^- .^ . • . • .

performance with a particular level of description. The guidelines are given in

Figure 7. Each cell's guidelines begin with the opening phrases of a statement

that sets the context for the particular cell, and that establishes the format

for the member taxa or dimension cf that cell. A list of questions follows,

each of which is intended to establish an analytic perspective that will aid the

identification of those member taxa. The questions are not intended to be

mutually exclusive. Neither do they necessarily constitute all of the analytic

perspectives that might help in the identification of the system's work

requirements. Rather, they reflect conceptual and organizational points of

view that evolved from early trial applications of the emerging STM/APM, and

that proved helpful for identifying taxa in subsequent applications.

7. Guidelines for Deriving Measurable System Attributes

Application of the guidelines discussed above (Figure 7) allows an

analyst to identify and organize a system's work requirements (Figure 1 ,

Block 1). Those work requirements establish the context for system design

and system evaluation. The designer examines the work requirements and

asks, "How can I build the system so that it satisfies the work requirements?"

The evaluator examines those same requirements and asks, "How can I test the

system to determine whether it satisfies the work requirements?" The designer

must translate the work requirements into concrete design decisions, i.e.,

consciously control the physical features of the system so that adequate

provision is made for the system to work as it should. The evaluator must

translate the work requirements into a concrete set of measures that can be

applied to observable features of the system to determine whether the system

can work as it should, is working as it should, or has worked as it should.

The translation of work requirements allows the designer and the

evaluator to focus on the system's ATTRIBUTES (Figure 1, Block 2). An

attribute is a concrete, observable feature of the system that bears directly on

the work that the system can and does perform. To the designer, an

attribute is something that can be controlled. By controlling the attributes,

the designer controls what the system can do and how well it can do it. To

the evaluator, an attribute is something that is measurable. By measuring the

attributes, the evaluator arrives at a judgment or appraisal of the system's

work.

Figure 8 illustrates the translation of work requirements into

attributes. The requirements chosen for the example are a sequence of the

performance processes required of the Design (or Curriculum Development)

subsystem of the Bradley Infantry Fighting Vehicle Gunnery Training System.

The selected sequence consists of one objectives-level performance process

(Analysis of Tasks), together with one of its functional purposes, and one

characteristic of that functional purpose.

In order to build the system to carry out the "Analysis of Tasks"

requirement properly, the training system designer must choose some

appropriate analytic methods; insure that those methods address the purposes

of the analysis (e.g., classification of domains of learning) and the technical

details required to achieve those purposes (e.g., identification of physical skill

-15-
•> ->->>^.'.->X">""5^J.^> •> •:^. 'y i^'.

ASfii-Ts o» svrriM I-IH(OMM*NI

PlHIUNMANll.

rOIIKri*LfTIIA

WIMI miMl this »yvtrm br

rt^Mtr of ODing er provtfi^g1

What prtnqwmu tbtiMi«!,

ltd iiilo Ih* ir*l«ni*

mimi k.ndv of rviourm

»Wal IM tytiem pantM4

Mm n»«l it il'ii lyfltm

to luirill»

HMI ^rcifir lyptl of

tipCflitt " J»I M butll mlo

IM ijntm'

»Ml kindi of toctiniquot

mmx ii» iinKm w «u» 10

riNHlHMANI l

mi" i.ssis

•• iwai «nr wi

rioilao -.•

Whol |ot>t or« rv^wirod of

(In» fjiUm»

WIMI prvrmurti %n involvot]

in Ikt miuion lh«t Ihit

tytioio it to porform'

Whtl s^nnt«nli doc» Ihn

syttom rtMivt to «wort

(HfMf «Mnmond*

»hoi Mlhoda «ill ihn ipl»» toptoy'

»hol or» Ih. •yoltni'I Urt«'

Ha« i« Ih» tytun n^vaiod

lo eouM lU mundod

•rtocu*

PIHMlMMAhl t

moiiti-n

»hol produrls muit ihr tyol»m d*li»«r m th» tourm of

rorrying owl IU miuM«9

»hoi food» o« lor»nt mun

II oulpul*

Mhol ooncrol* r»iulii or«

toini ««MT

»Ml Olli 11« •ytuin H^fy

to olhrr. mlororunf lyilomt*

»hot rhoi^oi i» th» tyoiom

»hoi if» IH« lytttm'i

InKlOM tllocU*

PiP»OKMANCC

CNVIIIONMtNT

»hot noturolly oocymng

foetan moy tffocl iho lyllom't »«ft'

»hol •l»monl> of Ih« »nvtronmoitt mifhi offo»! th«

•yolom't potootioliliot,

PtllMjPM/tl.1 I

CONmtAIHIi

»hot •rtulrory limiltlium

MM MO» lOtpaood «pan IM

y»l»mi Mug« M opor»

NHl

«Ml torlon trtoclini IM

oyoltm'i worti M»o toon

ixpcMO Wl doruu» a>

OlMf MMOrvOOIMII*

•Ml nmiramMi ttimonii

tomtit »ff»«! IM tytiti«'»

pountioiiim

fl oorh »opobility lulad

in IM Mil oMvt

H»

U

Ml on th» purpo« lo

»icft tftu pwlthlMlity

For ooeft ftrtivity Uolftd

M IM MU OftOM

For »am ptsduet lulM

ui tM nil oho»»

»hy Mo« IM tytUm M«tf

to Mvt IM parlieulftr pr«-

loqaMt» atainy. Mftftlo^r

r tftiU'

»hy M IM porlirulir «iporlilt IQMlMlid ftOMtMTJlt

No» H IM lyoloni

to Hi Uft» OMftbiillyt

•Ml D^lirftliOM laut for

IM locMMpr Hi III» lytl»«*

If IM qritom did MI MM

UM Hrl'eiiiar obilily. »Ml

Mffof?

•MMly

or» M .-"

•Ml or» IM pufpowi (or

porfanmni tftu pftrltftiUr

»*'

•Ml ftMitwMl Mllrllit«

III IM Hb »ilftMr IM ly»-

looi U porforoi*

Ml fti* IM noMM lor

imummiH UM prooodnroT

•Ml »III IM lyilrm M

» tpiifirt I« ftooomplith ui

IM mmm »t tint u*»

•hy Ml IM tylKm ftfton

fl»«* I*« MnfMMillT

•Mt »ill M Itftrnrd m •

MMM of tftu Ulk*

•Ml ill M ftWMftplltMd

M • Pnwlt o( tlu« toik'

•Ml MMflU ro« M <l»ri»»d

rreo UM portHMiof ftutpul«

Ml Ml ill IM

•ytwift MftM of UM

' It

Ml « »»»—iitul foclon

•iifhl oflftel IM lyoun'i

ftMllly to orply ill pot»nllftllti»! M UM WIOhrtM

MOd •II

MM foM M otnio» MlMt

»Ml IwiMr oolwMmonu

»III IK« prodMI ftllo» IM

tytlrio to pww*

u uw tmtm u MI MII»-

OfM»'

MlgM OflMI IM >

III orftlOM M ftM»|l^ ft»l

IU

Ml «mrftnawflUI tftrton

oi«»t oflort IM ftMlity u

HM IM rrtum'« pindoiu I»

tf-•

Ml impilMll fMUWlKMIO

«hl «riool IM liol»!«'!

•ftilily U ftpplll lk Mirnllftlllm I» IM wlOftMd

Ml lOipnwd MMnoliaM

•«III «ffo«! IM MMIII-

(MftftMU IMI IM lyilon

III wftMM In »orryMi

Mt It» priwi'

«Ml iit»Md IMUittiftM

•11011 ftrfwl IM IMily to

ua qnUiftf prodMU

ft Mft MUlMM mmrnt'

lor »..t ni^HtCMtion h.inl

ift IM n It uuivr

"To ipfly IM roMfobly lo

UM P»IH«UI PMVOM. IM

»Ml »jrcific oftllili»! or«

Miftid for IMI pftrlmlar

opplictliix.'

•Ml difdrrnl c»i»D.lili«i

Mft M troifhl U Mor on

this piirpoi«'

»Ml or» IM lll|r«di«nU of

kno«l»<Hr «ill ftnd oftility

IMt Ihi» 9**yam MmonM*

•Ml iptc.fw aroci of np»'

ri»or» or» rvkvftftl U thn

•pplinlM«»

•Ml or« IM MHipftMntt at

IM tovhni^M« N^uifftd U

tftu

lor »orli pyrpoM lutrd in

IM orll otoM

»Mt itop» muftt IM iyil»m

in IM

For »orli opiHirition liklod

ui IM Mil oteM

111» «joM-ft pn

•Ml rwBlllMnl .^rrauono

orr ro<|L,.rnl lo ftftply IM

ppftoftM lo Ihn pftriicutar

•Ml i«ilo>tonri «Ml M

ocftirM^ if IM oolpwl it U

M taftlftl Far Uw pftruaalar

pplmtiont

»Ml wtrrftiaAfttt fftftM ft»*

nim tr» WHtm U

tM»»iU» 0 Mftfol fHlllMd

I?

FtoftUd If »

•Ml tnvironattnlol fortort

•U|»l offMI IM lytltn't

aoMtilwAl aapftUlilwa Mi

piotoftiwui tor parlor-

«llotud k( «•

Ml Mpeud raamrlmm

i>i|f.l tlf»«i Mft tyiiom't

eontiiiMni «MftMilm ftnd

far MiFftr

In »Ml difftrtnt anyt »on

IM »ytltm orh*»-» lint porimitar atrowpluMMftt*

ill M tlufnrd la HlAtldlMl

I of UM parlKaUr

•Ml or» IM indoidual

MlitftroMo lUm IMI ftM

«uMtial u orroioplitn uw

pftrticiMr HlliftdM purpOM

Ml iMividuoi >rae«

Ml M proHMd 10 IM

tytloift Ift MTM

Mil Iftrlon

ignt tfltrl parltrmonr«

of IM tytloin'i üMülMftl

•Ml oftmr—nftUI fortort

«»I ftfrMI IM rMrtclrrMIM of IM IliidMU Mli»-

•M k| IM lyt Ola?

mlnrliunt

il|(il tflort porlornonr»

o/ Ua »ytuat toMlitwnt

Ml impoood MitrialMM

l(kt tt'oet IM tOftTMUrittltt of IM prodMCtt

daliMMd »y IM lytu«.*

Ml MauiaM MM U M

(ornuloiod W punwl at Uw

WVftM*

•Ml mformotion tmwil«

Ml M iMdftriftkan m puttut a( uw

•Ml Mat IM produrl MM

U tarMla la M MftM (or

UW ftMlKftllM*

Figure 7. Guidelines for Identifying Dimensions/Taxa

of System Performance

-16-
WORK REQUIREMENTS ATTRIBUTES

OBJECTIVES-LEVEL PROCESS:

2.3 The System must carry out

an analysis of the tasks

selected for training.

Functional Purposes of that

Process:

«•Oil •• •

2.3.2 One purpose for the task

analysis is to classify

the tasks in terms of

their domains of learning.

Characteristics of that Functional

Purpose:

£• w*Z« X ••• •

£fv* £*• i» ••• •

2.3.2.3 To Classify tasks in

terms of domain, it is

necessary to identify

the physical skills to be

acquired.

Task analytic methods applied to

the BIFV Gunnery Tasks

Analytic factors assessed for the

tasks

Completeness of the task analysis

Accurac;, of the task analyses

Classification methods employed

to identify domains of learning

for BIFV Gunnery Tasks

The domains identified for each

BIFV Gunnery Task

Methods used to identify the

physical skills required to perform BIFV Gunnery Tasks

The physical skill requirements

identified for each task.

2.3.2.4 • »• •

Figure 8. Sample Set of Attributes Associated with

System Work Requirements

-17-
m^w^.-^n^7TmjTr.rmrmTwirmrmj~w*rmr^rwi*^ -m.r

requirements); insure that that the analysis considers all factors pertinent to

the tasks; insure that the analysis extends completely to all of the tasks

selected for training; maintain quality control over the analysis to insure that

it is performed accurately; and, ultimately, see to it that the analytic results

are produced when needed, and fully document all necessary items (e.g.,

domain classifications, skill requirements, etc.). These attributes are the

leatures of the system that the designer must control (Figure 1, Block 3) to

see to it that the task analysis work requirement is satisfied.

As seen in the APM Overview, these are also the very same

attributes that the training system evaluator must measure to judge whether

that task analysis work requirement is satisfied (Figure 1, Block 4). To

appraise the system's task analytic work, tho evaluator must consider whether

appropriate analytic methods were selected; whether they were applied to serve

the intended purposes, and whether they were applied in manners suited to

those purposes; whether all pertinent factors were considered in the analysis,

and whether anv extraneous factors were included; whether all tasks selected

for training were analyzed, and whether any extraneous or superfluous tasks

were analyzed; whether the analyses were conducted accurately; and whether

the intended analytic findings were produced.

8. Design and Evaluation as an Interactive Process

The designer's first step is to develop a preliminary plan for

securing the identified system attributes. The plan addresses the products to

be delivered and the processes to be implemented, but its principal focus is on

the system's required potentialities. The designer's main task at this stage is

to demonstrate that the system will have the prerequisite abilities needed to

carry out the processes and deliver the products.

In parallel, the evaluator examines those same system attributes and

develops a plan for testing whether the system has the attributes and satisfies

the work requirements. The test plan includes measures and measure

application procedures for all attributes. However, at this stage the evaluator

principally is interested in selecting and applying measures of potential

performance. This is the proposal stage of design and evaluation. The

designer says, "Here is how I intend to see to it that the system will have the

attributes it ne^ds to work properly." The evaluator responds, "Here are the

specific performance defects that I have found in your plan." The evaluator

and designer work together to determine how the plan should be modified to

correct the defects.

Now the designer implements the (modified) plan. The focus shifts

from potential ("on paper") performance to in-process performance, as the

specified activities are carried out. The evaluator again applies the measures

to appraise the work, but now interest centers on the measures of process

attributes. The kinds of questions that arise may include: Arc the right

kinds of activities being conducted? Is the plan being followed? Are we on

schedule? Are procedures being adapted properly to meet the needs emerging

in the real world of system development? The evaluator monitors the

designer's work, and from time to time ma, report, "What you are doing there

-18-
tnJy 'XT; ^^"tr-T'Ti ** *%. rmmmmmmmmmm^ammmmmmm

isn't quite what is needed." In a cooperative effort, the designer and

evaluator modify the on-going system processes to correct the defects

uncovered.

At some point, the system is assembled and begins to deliver its

products (Figure 1, Block 7). Again the evaluator applies the measures. The

accent now Is on whether the output goods and services satisfy the system's

product requirements. If defects are noted, the measures of product,

process, arid potentiality help to diagnose their causes. It may be that all

that is needed is to repeat the processes more carefully, i.e., to re-produce

the products with greater attention to quality control. However, the

evaluation might disclose that it is the processes themselves, or even their

prerequisite potentialities, that must be reconstructed. An evaluation that

interacts cooperatively with the design effort and that addresses all aspects of

performance will be much more likely to help develop a well working operational

system than will the old-fashioned "bottom line only" approach that

traditionally is antagonistic to design.

Design and evaluation do not end with the "christening" of the

operational system (Figure I, Block 8). They remain living functions as long

as the system continues to operate, always looking for possible improvements

that will allow the system to respond to changing needs and/or to adopt

newer, better technology and procedures.

9. Data Bases of System Performance Ta?:a: Focus on Training

Applications

Extensive practical application of the APM requires the availability of

taxonomies of system performance. Each such taxonomy is an organized data

base. Its contents are the work requirements of the particular system, or

family of systems, for which the taxonomy was prepared. A design or

measurement analyst may examine the total data base and identify the specific

performance elements relevant to the application at hand. As more taxonomies

are developed, the scope of applications available to the model will increase.

To provide one such data base, generic taxonomies were developed

for major functional subsystems of training systems. The decision to focus

APM development on training applications reflects the observations that

training systems are pervasive, critical to modern military operations involving

sophisticated equipment, and consistent with ARI's primary interest in human

behavioral issues.

Development of training system data bases for APM applications

necessitated identification of the training system hierarchical structure, and

subsequent generation of performance taxonomies for the members of that

structure. It is functional, not organizational, structure that is of interest

here. The organizations of two particular training systems may differ widely,

but each will accomplish the same basic functions required of any training

enterprise.

The principal generic function of training systems is postulated to be

learning, defined as any activity involving the senses that affects behavior in

some purposeful fashion. Learning is a human function: it is people who

•19-
[W.WWLWLffWIH.'WKWtiJnWWWt"^' WWIß.V V.Kfl^rJtSWmiXWyitftlfHil'^WWMJWlMMUWiiM» tm\n^-v^s wrr~i^i wi-wjv-vw'-• -

carry out the sensory activities to experience the intended behavioral effects.

The principal generic operator/staff merriber of a training system, thus, is the

learner.

The other major generic function of training is postulated to be

helping to learn, defined as providing an efficient learning environment to the

learner. Numerous types of people, using a variety of equipment, may work

in any given training system to make it conducive for the learner to learn.

All such people are learning helpers. They acquire more specific titles in

accordance with the particular types of help they provide.

The two major functions can be broken down into more detailed

operations, or subfunctions. These can be grouped conveniently into six

generic training subsystems, as illustrated in Figure 9.

a. The administrative control, or Command Subsystem—which deals

with identifying needs for training, allocating resources,

recruiting training personnel, monitoring and evaluating learner

performance, etc. Command Subsystem is staffed principally by

training administrators. They help the learner to learn by

constructing and managing the system in which the learning

activities can take place.

b. The curriculum development, or Design Subsystem--which is

responsible for planning the instructional activities, selecting

training technology, assembling content material, defining

instructor and student requirements, etc. The Design

Subsystem is the portion of the training system in which the

Instructional Systems Development (ISD) model is applied.

Design Subsystem is staffed primarily by curriculum {tevelopers.

They help the learner to learn by determining the specific

behavioral effects that are needed and by planning the sensory

activities that will lead to those effects.

c. The facilities development, or Emplacement Subsystem—which

constructs or acquires and assembles all facilities and equipment

needed to support the sensory learning activities. The

Emplacement Subsystem takes the plans developed by the Design

Subsystem and insureb that all materials, equipment,

installations, supplies, etc., needed to carry out tho^e plans

are made available. Emplacement Subsystem is operated mainly

by facilities developers. They help the learner to learn by

providing the tools needed to carry out the learning activities.

d. The logistics support, or Logistics Subsystem—which deals with

maintenance of facilities and equipment, housing, feeding and

recreation of training system personnel, replenishment of

consumables, transportation of people and supplies, etc. The

logistics supporters help the learner to learn by attending to

the myriad of details necessary to keep the training system

running smoothly and free of discomfort and distraction.

-20-
Command

Training

Administrator

and others

Design

Curriculum

Developer

and others

Emplacement

Facilities

Developer

and others

Logistics

Logistics

Supporter

and others

Enabling

i

Notes:

Delivery

Learner(s)

and others

More than one operator

may be involved in any

subsystem.

One person may serve

as principal operator in

more than one subsystem (i.e.. an individual

may "wear two or more

hats" and simultaneously

serve as two or more

different kinds of

operator).

Figure 9. Training Subsystems and Their Operators

-21-
BHWWWrg^i

1".'. W1L» K\ J.* J.'U.« l^l^^-L' -L1

.'-' ^'

J

*'.'f'.-'.'A'A'J.T'i! 1'».'^• ',^'.''-.V."»/ ^^'^:^^ ^AT'.".'.'.'-^.'-^^.^^.'''.'.^^.'-".-^.'-"^

e. The instructor preparation, or Enabling Subsystem—which deals

with familiarizing instructors with the plans, content, equipment

and facilities involved in the intended learning and with

preparing the instructors to teach the course effectively. The

Enabling Subsystem also is responsible for tailoring the general

plans and material to the specific learning needs of a particular

class or team of learners. The principal staff of the Enabling

Subsystem are the instructors, teachers, professors, etc., who

directly assist the learner in carrying out the prescribed

sensory activities. Collectively, these people are called

facilitators. They help the learner to learn by presenting

information, demonstrating techniques, coaching the learner's

efforts, evaluating and correcting performance, etc.

f. The instructional implementation, or Delivery Subsystem—which

is where the learning/training actually takes place. In the

Delivery Subsystem, the prescribed sensory activities are

carried out, and the learner experiences the intended

behavioral effects. The principal staff member of the Deliverv

Subsystem is the learner, aided by all of the other staff

members identified above.

Figure 10 presents "what" performance is required of each

subsystem, '.e., their generic objectives-level taxa of potentialities, processes

and products. Those taxonomies were compiled by applying the objectiveslevel guidelines presented previously in Figure 7. No attempt was made to

identify generic taxa of environment or constraints, since the scope of work

did not allow for such development.

The functional purposes-level guidelines and characteristics-level

guidelines of Figure 7 were applied to develop complete taxonomies for the

Design, Enabling and Delivery Subsystems. Those are presented,

respectively, in Appendices A, B and C. Those taxonomies provide a general

data base for addressing virtually any design or evaluation issue concerning

those three subsystems in virtually any training system.

10. Sample Derivation of Attributes for a Training System

An attribute previously was defined as a concrete, observable

feature of a system that bears directly on the work that the system can

perform. For each taxon of performance required of a system, there exist

certain system attributes that determine whether and how well the system can

satisfy that taxon. The designer controls those attributes to make the system

work well. The evaluator examines those attributes to assess how well the

system works.

To aid in identifying the system attributes associated with a given

performance taxon, the analyst can "decompose" the system into its constituent

parts. The "parts" of a human-machine system are the people who operate the

system, the equipment/materials employed in the operation, and the

procedures/methods by which the people employ the equipment and interact

with each other. The derivation of attributes then proceeds as follows:

-22-
r

jrmytf^f ^i»ii ;. j/ji/y.-gi^jTyrjy/rjw:Jyy>^.'FV^l*Vff<.*.V^angH."^7TT1TFrV."r.»^.T^-.-r7T-7rr«yrT».V*V

Subfy.ctpm Potentialities Proeesaes Productü

Command

• Providing overall system

manugemrnt

. Determining learning needs

• Determining overall scope

of effort

. Providing financial

resources

• Insuring availability of

qualified staff and tools

• Maintaining quality control

• 'Market analysis' of learning demand

. Analysis of resource requirements to

meet demand

• Cost-benefit assessment of potential

training

• Recruitment, screening, and selection

of personnel

> Screening and selection of equipment

« Review and evaluation of performance

• Svstem personnel

• System materiel

. Funds

• Work assignments

• Performance milestones

• Performance assessments

Design

• Identifying goals and priorities

• Establishing performance

objectives

• Analyzing the objectives

• Defining training content

. Defining training procedures

• Evaluating the curriculum

. Job analysis

. Assessment of tasks for training

development selection

. Analysis of selected tasks

• Assessment of instructional requirements

• Analysis of existing instruction

. Instruction assembly

. Conduct of training trials

• Documenled job anaK

sis

• Stated performance

objectives

• Lesson plans

• Training documents

. Tests

• Training trial data

Enabling

. Providing day-to-day training management resources

• Insuring availability of

Instructional delivery

expertise

« Insuring availability of

learning guidance expertise

• Insuring availability of subject matter expertise

• Familiarizing facilitators

with the curriculum

• Tailoring the curriculum to

specific delivery applications

• Analysis of facilitator's ability to perform the tasks

• Analysis of facilitator's ability to manage/implement the learning activities

• Design activities to correct facilitator's

deficiencies

. Design activities to allow adaptation

of the curriculum

• Conduct of the facilitator's learning

activities

• Qualified facilitators

for given delivery

applications

• Curricula tailored to

given delivery applications

Emplacement

• Insuring availability of

adequate learning facilities

• Insuring availability of

adequate learning materials

• Insuring availability of

adequate instructional aids

• Analysis of curriculum to determine

emplacement requirements

• Acquisition, construction and/or modification of learning facilities

• Acquisition, construction and/or modification of learning material

• Acquisition, construction and/or modification of instructional aids

« Appropriate learning

facilities for given

delivery applications

• Appropriate learning

materials for given

delivery applications

• Appropriate instructional aids for given

delivery applications

Logistics

• Insuring availability of

adequate personnel support

• Insuring availability of

adequate facilities maintenance

• Insuring availability of

adequate equipment maintenance

• Analysis of delivery plans to determine

logistics requirements

• Development of schedule of personnel

support services

• Development of schedule of preventive

maintenance services

• Development of procedures for restorative maintenance

• Acquisition of facilities and materials

needed for delivery of the services

. Deliverance of the services

• Personnel support

services

• Facilities maintenance

services

• Equipment maintenance

services

Delivery

• Providing learners with the

learning environment

• Insuring learners' interaction with the environment

• Managing learners' progress

• Evaluating learners'

achievement

• Providing bases for system evaluation

• Analysis of ipecific learner's needs

• Conduct preparation learning activities

• Conduct presentation/demonstration

learning activities

• Conduct application/practice

• Conduct learner testing

• Acquisition of formative «valuation data

• Appropriate learning

activities

• Learners who have

achieved the stated

objectives

• Assessment of learners'

achievements

• Formative evaluation

data

Figure 10. Objectives-Level Performance Taxe of Training Subsystems

-23-
WWWPWWWWWM MW-TWw I ^ '* * '.' *-*y «V-WJ^HWY"! 'v^rrg^^.7r£fyyv^£rc*V>rr-"'iy'^PWfyt"

,W7y nrry r,^,7v ^'«»■■■■■■ ■■VTvr/' TP -»:.'W

a. Determine which of the system's "parts" are involved in the

performance taxon of interest.

Identify the specific operators or classes of operators who are

responsible for the particular potentiality, process or product

and/or who are included in that potentiality, process or

product. Similarly, identify the specific equipment or materials

used in or contained in that potentiality, process or product.

Finally, determine which of the procedures/methods by which

the operators employ the equipment pertain to that potentiality,

process or product.

b. Identify the qualitative and quantitative traits or properties

that the involved people, equipment and procedures must have

in order to achieve or satisfy the performance taxon of

interest.

The requisite traits might include specific aptitudes, expertise,

strengths, reliability, fidelity, etc. These traits need not be

directly observable. However, they must be directly relevant

to the particular system potentiality, process or product being

studied. And, if one of these traits is not observable, it must

at least be possible to infer whether or not the trait is present.

c. Identify concrete, observable features of the involved people,

equipment and procedures that provide bases for inferring

reliably whether the essential qualitative and quantitative traits

are present.

These concrete, observable features are the attributes of

interest. That is, they are the controllable and measurable

entities that determine the system's achievement of the

performance taxon of interest.

To illustrate this process, consider Training Design Subsystem

performance taxon 1.2, "Establishing Performance Objectives," (see Appendix

A). Performance objectives can be defined as written, explicit descriptions of

specific terminal behaviors that learners are required to demonstrate to signify

successful training. Each performance objective states something specific that

the learner will be able to do upon completion of training. (See the Glossary

for a more complete definition of "Performance Objective.") Clearly, the

potential for establishing performance objectives accurately and completely is

an essential ingredient of the subsystem responsible for developing the

training curriculum.

What "parts" of the Training Design Subsystem are involved in the

potentiality of "Establishing Performance Objectives"? First and foremost,

specific people are involved: the subsystem must designate key instructional

system designer (s) having the ability to establish accurate, complete

performance objectives. Secondly, important materials are involved: these

materials consist of information or data that disclose what it is that the

learners need to be able to do, on the job, following the end of their training.

Lastly, instructional design methods are involved: the methods of interest are

-24-
'.-Vl——iM——^1^—1—1M^W—i—i—' ill !! mimwmBmK^mimt^imwiMwm-m-mmnm-^^mT

the procedures that the instructional system designer(s) will use to collect and

analyze the relevant data, and to translate the analytic findings into

statements of performance objectives. In order to demonstrate that their

training system has the potential for "Establishing Performance Objectives,"

the training designers must:

0 Name and/or describe the individuals who will establish the

objectives!

0 Define the information to be collected and analyzed to establish

the objectives;

o Define precisely how the information will be collected and

analyzed.

What are the essential qualitative and quantitative traits that these

people, materials and methods must have in order to insure the potentialit/ of

"Establishing Performance Objectives"?

Certainly, the people designated for this task must possess expertise

in the establishment of performance objectives, and they must be prepared to

devote sufficient level of effort to that task for this particular training design

project. The information to be used as the basis for establishing the

objectives must be appropriate for that task, and it must be accessible to the

training deüigners. The methods to be used to collect and analyze the

information likewise must be appropriate, and adequate resource support for

implementing those methods must be provided. In naming their staff and in

defining the information and methods that will be used, the training designers

must provide concrete evidence that these essential traits will be present.

What is the concrete evidence that the traits needed for

"Establishing Performance Objectives" are present? That is, what are the

system attributes that can be directly observed and examined to determine

whether the training system satisfies that particular taxon?

Figure 11 lists concrete, observable features of training system

people, materials and methods from which reliable inferences concerning the

system's potential for "Establishing Performance Objectives" could be drawn.

11. Guidelines for Developing Measures Specifications

A measure is a judgment or appraisal about the thing-beingevaluated. Thus, a measure of a human-machine system attribute is an

appraisal of the "goodness" of that attribute. Each measure supplies a

particular quantum of evidence or insight about the degree to which the

system possesses some feature that it needs to perform some element of its

work. The guidelines for systematizing the specification of measures derive

from a number of conceptual bases, or postulates.

0 A system attribute is a variable across the population of

human-machine systems tHat are supposed to possess the

attribute.

-25-
Taxon; Establishing Performance Objectives

Essential Traits j Concrete Attributes

[

• Expertise in the estabI lishment of performance

| objectives

• Sufficient level of effort

for the assigned task

- specific efforts to be undertaken by each |

person !

j - specific prior experience in the establish- i

menl of performance objectives for other i

training applications |

- education and training relevant to the \

establishment of performance objectives

- samples of prior work in this area i

- endorsements of people for whom this !

type of work previously was done

- time scheduled to be devoted to the !

establishment of performance objectives i

- remuneration to be received for that task

.11.

Ü

00

s

c

• Information to be obtained

must be appropriate

• Information to be obtained

must be accessible

- the information specified for collection \

- source(s) from which the information is to |

be acquired

- volume of data to be obtained j

- descriptions of applications of these types j

of data in previous, similar work

- agreement/permission received from the \

source(s) to access the data |

- specific proof of the accuracy, complete- !

ness and representativeness of the data in i

the source(s) • j

i

• Data collection and

analysis procedures must

be appropriate

• Sufficient resources

must be provided to

implement the resources

- prior success in applying the collection/ j

analysis procedures j

- compatibility of the procedures with the 1

characteristics of the data to be collected j

- sequence and timing of procedural steps |

- compatibility of the procedures with the j

specific efforts to be undertaken by each |

staff member

- cost of implementing the procedures

- funds provided for implementing the

procedures j

- supportive equipment/materials needed to j

implement the procedures j

- availability status of the supportive

equipment/materials j

Figure 11. Sample Attributes Associated with a Taxon of

Training Design Subsystem Performance

-26-
< minUU^UahBUCSX&JKBMV^VBaWMHHVBWBHMWV«~V TW »r, \ ~. > nkibV«*^^-UN

The value that the attribute takes on, in any given system, is

a function of the degree to which the system possesses the

attribute.

The attribute's value may be a function of the values of

simpler, constituent features termed the dimensions of the

attribute. For example, if "area" is a relevant attribute of a

rectangle, the value of that attribute can be expressed as a

function (product) of the rectangle's base and height values.

In some cases, it may be possible to estimate the value of an

attribute directly. In other cases, it may be possible to

estimate the values of the attribute's dimensions, and then to

compute the attribute value.

Different kinds of estimates of an attribute's value may be

made. One kind is an estimate of absolute, or "true" value

(e.g., obtained by using a yardstick to determine a man's

height). A second kind of estimation deals with relative values

of an attribute (e.g., identifying the man who is tallest, 2nd

tallest, 3rd tallest, etc.). Still a third kind of estimate

produces threshold-referenced attribute values (e.g., a rod of

unknown absolute length could be used to partition the group

of men into two classes, one class taller and the other shorter

than the rod).

A measure of an attribute is the estimate of the attribute's

value. Thus, a measure may be direct or computed/derived,

may be absolute, relative, or threshold-referenced, and also

may come from nominal, ordinal or ratio scales.

Numerous possible measures exist for any given attribute. The

evaluator's principal challenge Is to select measures that provide

the kinds of estimates of attribute values that are appropriate

to the measurement purpose at hand, and that permit those

estimates to be made accurately, reliably, and efficiently.

The attribute measures, when properly combined, will supply

appraisal of the more general traits with which the attributes

are associated (see Section 10.b., above). Similarly, the trait

measures. In combination, provide the judgment of the system's

performance with respect to the particular taxon of Interest.

The taxon measurement requirements thus dictate the kinds of

estimates that are needed for the traits. In turn, the trait

estimation requirements dictate the kinds of attribute estimates

that are needed.

Figure 12 summarizes these concepts. The figure represents a

general hierarchy of measurement. At the top, to which all branches lead. Is

the desired measurement of system performance: that Is a judgment or

appraisal of the system's satisfaction of a particular performance taxon of

Interest.

-27-

o
Measurement

of

Performance

of a

Taxon

Measurement Measurement Measurement "1

i

of of of i

Trait Trait Trait \

No. 2 No. i No. n |

j i A

Measurement

of

Attribute

No. il

Measurement

of

Attribute

No. jj

!

Measurement

of

Attribute

No. ij

Measurement

of

Attribute

No. im

—f

Value Value Value

of of • •• of

Dimension Dimension Dimension )

1 No. ijl No. ij2 No. ijk

Figure 12. Schematic Representation of a General

Hierarchy of Measurement

-28-
At the second level appear the traits or properties that the system

must possess in order to satisfy the taxon. For the sake of generality, n of

these appear in Figure 12. Each trait belongs to one of the three fundamental

elements of a human-machine system: the people, the equipment, or the

procedures. The measurement of system performance is a function, f, of the

individual trait measurements. That is equivalent to saying that system

performance, is the sum total of the contributions of the system's people,

equipment and their interactions. By measuring at the trait level, the

evaluation process permits the respective contributions of these fundamental

system elements to be appraised, and functionally combined to supply the

judgment of system performance.

But any given trait may not be directly observable, and hence not

directly measurable. Human traits in particular, but also equipment and

procedural traits, may be abstract. However, if a trait is real, it must be

possible at least to infer it from concrete, observable factors. Those factors

are the attributes associated with the trait. The measurement of any given

trait, e.g., trait No. i, is a function, £i, of the measurements of its

individual attributes. Each trait has its own set of attributes, and its own

functional relationship with its attributes.

Every attribute is concrete and observable. Thus, every attribute

is at least theoretically directly measurable. But many attributes are complex,

and difficult or expensive to measure directly. Often, those kinds of

attributes are multi-dimensional, and thus their measurements may be computed

from the values of simpler, unidimensional factors that are more amenable to

direct measurement. That is, the measurement of a particular attribute, e.g.,

attribute No. ij, might best be obtained as a function, hii, of the values of its

constituent dimensions. Each dimension is a factor amenable to practical direct

measurement.

The links between given performance taxa and their related attribute

dimensions may vary widely. Consequently, the intervening traits and

attributes may be simpler (even identical from one level to the next) or more

complex as well. Furthermore, the number of levels one chooses to analyse in

the measurement hierarchy of Figure 12 may also be a function of what is

feasible, how diagnostic one wishes to be and what is cost-effective.

The concepts discussed, exemplified and depicted above provide the

basis for developing guidelines for selecting attribute measures, and for

specifying how those measures are to be combined to supply a judgment of the

system's performance relative to some taxon of interest. These guidelines

proceed as follows:

a. Determine the kind of performance estimation that is required

and appropriate for the particular taxon and measurement

purpose at hand.

b. Determine the trait value estimates that are needed to support

the required taxon measurement, and identify the functional

relationship between the trait values and the taxon value.

-29-
c. For each trait, determine the attribute value estimates needod

to support the trait measurement, and identify the functional

relationship between those attribute values and the trait value.

d. Identify the dimensions of each attribute that may be used to

generate the required value estimates.

e. Identify the functional relationship between the attribute value

and the constituent dimension values.

f. Define the methods by which the dimension values will be

obtained.

To illustrate the application of the measures specification guidelines,

consider the following hypothetical situation:

Thn U.S. / rmy Infantry School has issued a

Request for Proposals (RFP) to undertake

development of a training system to support

introduction of a new squad automatic weapon.

Responses have been received from several

different organizations. A committee of training

development experts has been assembled to review

the proposals, and to select the organization best

qualified to conduct the project. The committee

has 6 members.

For simplicity, this illustration will address only the following

Measurement Purpose:

To review and evaluate a set of proposals received from

training development organizations to identify the organization

best suited to establish performance object; es for squad

automatic weapon training.

The guidelines (a-f) are applied as follows:

a. Determine the kind of performance estimation required for this

measurement purpose.

A relative appraisal of the competing organizations' capabilities

for establishing training objectives will suffice. The estimates

should permit the organizations to be rank-ordered (e.g., "best

qualified," "Znd best," ... etc.).

b. Determine the trait value estimates needed to support this

performance measurement. Identify the functional relationship

between the trait values and the performance measurement.

The traits essential for satisfying the potentiality of

establishing training objectives are (from Figure 10, earlier):

-30-
1) Expertise in establishing performance objectives

2) Adequate level of effort proposed

3) Appropriateness of information proposed for collection

4) Accessibility of the proposed information

5) Appropriateness of data collection/analysis procedures

6) Sufficiency of proposed resources

Every member of the review committee will review each

proposal, and will make an independent judgment of each

proposal with respect to the above-listed traits. Each trait will

be assigned a value from 0 (insufficient) to 3 (excellent).

Thus, each reviewer will assign each proposal a value from 0

(totally insufficient on all traits) to 18 (excellent relative to all

six traits). In turn, each proposal will receive a summary

score from 0 (totally deficient in all respects to all reviewers)

to 108 (excellent in all respects to all reviewers). Propo;als

will be rank-ordered in accordance with their respective

summary scores.

c. For each trait, determine the attribute value estimates needed

to support the trait measurement. Identify the functional

relationship between the attribute values and the trait

measurement. For example:

Trait No. 4: Accessibility of proposed information

Each offerer's proposal will be assessed relative to the

following two attributes to arrive at a measurement of this

trait:

1) Evidence that the offerer has secured or can secxire

permission to extract the proposed data from the

proposed source(s).

?) Evidence that appropriate data of sufficient quantity

and quality exist at the offerer's proposed source(s).

Each reviewer will make an independent assessment of these

attributes for each proposal. Relative to attribute 1), the

reviewer will assign a value from 0 (cannot be done) to 2 (can

be done).

Relative to attribute 2), the reviewer will assign a value from C

(not in existence) to~5 (exists in high quality and quantity).

After the reviewer has assessed a proposal relative to attributes

1) and 22, he/she will compute the measurement of Trait No. 4

using the table set forth below.

-31-
Value of attribute 2)

r 0 1 2

\ 1 Value of Ö 0 0 Ö 0

attribute 1) 1 0 0 1 2

2 0 1 2 3

Table entries are values for the "Accessibility of Proposed

Information" (Trait No. 4). Other traits are determined using

similar procedures defined by the analyst.

d. For each attribute, identify the dimensions that may be used to

generate the required value estimates. For example:

Attributes associated with Trait No. 4: Accessibility of

proposed information

1) Dimensions of the attribute: "Evidence that offerer

has secured or can secure permission to extract the

proposed data from the proposed source(s)."

Each reviewer will consider the following dimensions

in assessing the evidence of the offerer's ability to

access the proposed source(s):

a) The explicit documentation, in the offerer's

proposal, that access to the proposed source(s)

has been granted to this offerer for this project

by authoritative representative (s) of the

source(s).

t

b) Documentation, in the offerer's proposal, of

his/her prior success in accessing similar

information, for other projects, from the same

proposed 3ource(s).

c) Documentation, in the offerer's proposal, of

his/her prior success in accessing similar

information from source(s) similar to those

proposed.

d) Potential impediments to access of the proposed

source(s) that the offerer has identified In

his/her proposal, but has not adequately

addressed.

e) Potential impediments to access of the proposed

source(s) known to the reviewer but not

identified or addressed in the offerer's proposal.

N.B. The reviewer shall immediately inform the

chairman of the review committee of any such

impediments.

-32-
f) Reviewer's knowledge of the prior history of the

proposed source(s) in granting/refusing access

to this or similar offerer.

N. B. The reviewer shall immediately convey any

such knowledge to the chairman.

The reviewer will consider no dimension or factor

other than the six listed above when appraising this

attribute, unless directed to do so by the review

committee chairman.

2) Dimensions of the attribute: "Evidence that

appropriate data of sufficient quantity and quality

exist at the offerer's proposed source(s)."

(not included in this sample)

For each attribute, identify the functional relationship between

the attribute value and the constituent dimension values. For

example;

1) Attributes associated with Trait No. 4; Accessibility of

proposed information

a) Functional computation of the value of the attribute

"evidence that offerer has secured or can secure

permission to extract the proposed data from the

proposed source(s)."

The reviewer must assign this attribute a value of 2

(highest possible value) if:

0

Explicit documentation is included in the

offerer's proposal to establish that

grant-of-access explicitly has been received from

authoritative representative(s) of all proposed

sources, or

0 All of the proposed sources are known to grant

access to all users similar to this offerer, for

purposes similar to those intended in the

proposed project.

The reviewer must assign this attribute a value of 0

if:

0 Any key source (i.e., the only source proposed

by this offerer for accessing certain essential

data) is known to refuse access to all users

similar to this offerer, for purposes "iTmilar to

those intended in the proposed project, or

-33-
0 Any key source is known to have refused access

to this offerer in the past, for purposes similar

to those proposed, and the offerer has not

demonstrated a method for securing future

access in his/her proposal.

In the absence of all of the four conditions listed

above, the reviewer will exercise his or her own best

judgment in assigning a value of 0, 1, or 2 to this

attribute. The reviewer is to be prepared to explain

the rationale for his or her judgment.

b) Functional computation of the value of the attribute

"evidence that appropriate data of sufficient quantity

and quality exist at the offerer's proposed

source(s)."

(not included in this sample)

f. Define the methods by which the dimension values will be

obtained.

The principal method by which reviewers will obtain values for

the dimensions described under item d. , above, will be through

a careful and complete study of the oTferer's proposal.

In addition, for certain dimensions, a reviewer may possess

relevant prior information that might affect the values of those

dimensions. When a reviewer believes that he or she possesses

such information, that information should be conveyed to the

review committee chairman as soon as possible. The.chairman

will decide whether the information is to be considered in

assessing the value of a particular dimension. Upon so

deciding, the information will be conveyed to other reviewers.

Occasionally, due to the availability of relevant prior

information, the chairman may determine that a particular

dimension must be assigned a particular value by all reviewers.

12. Initial Concepts for Developing Design Specilications

Although the project's scope did not allow for as much development

in the APM's design specification application as for the performance

measurement applications, a preliminary set of concepts has been prepared.

They are summarized briefly in this section.

A Vsign specification of a human-machine system is a description of

exactly what ihe system design should include in order for the system to

achieve all of its performance requirements. As with the APM's measures

specification procedures, the notion of a systematic design specification process

rests on a number of conceptual bases, or postulates:

-34-
c Design and measurement deal with precisely the same issues:

namely, the concrete, observable attributes that make the

system what it is and that cause it to work in the way that it

does.

0

The design specifier, like the evaluator, thus must know how

the concrete, controllable attributes are linked to the taxa they

^erve. Specifically, the design specifier must know the

functional relationship between a taxon and its essential traits,

and the functional relationship between each of those traits and

its constituent attributes. Only then can he/she identify the

attribute design choices to make to effect the appropriate status

of the taxon.

0

The design specifier's choices determine the value that each

attribute will have in the system being built. TKose choices

determine the system's trait values through the functional

relationships that link the traits and the attributes. Similarly,

the trait values determine the values of the performance tn\a.

0

The design specifier's intention is to make the set of choices

that maximizes the values of the taxa. Often, but not always,

that involves attempting to maximize the attribute values. In

some cases, the best choice may be to arrange for certain

attributes to take on sub-maximum values, because the resulting

combination produces the best attainable trait value.

0

In sum, in making choices the design specifier needs to know

and to apply the functional relationships that link the system

attributes to the traits, and also the functional relationships

that link the traits to the taxa.

0

The design specifier also needs to know how to assess the

values of the attributes, traits and taxa, to insure that

appropriate design choices are made. That is, the design

specifier needs to know the measures.

The concepts described above provide the basis for defining an

initial set of guidelines fcr developing design specifications. That effort is

left for future projects.

B. Applications of the APM for Measurement

The APM concepts and guidelines for measurement applications are next

illustrated with more realistic examples. These illustrations summarize more

complete descriptions to be found in the earlier project reports (Bloom, et al.,

1981, 1982). Two narrowly focused measurement illustrations, related to the

Bradley Infantry Fighting Vehicle (BIFV), are given: (J) Derivation of a

BIFV Measures Hierarchy for the Surveillance Function, and (2) Derivation of

Measurable Performance Objectives for the BIFV Training System.

-35-
1. Derivation of a B1FV Measures Hierarchy for the Surveillance

Function

a. General Conditions

This example begins with a general look at many BIFV

performance aspects, of which surveillance is only one. The application to

surveillance is pursued after ertablishing that more general context. For the

general case, the overall measurement purpose was taken to be:

c

To evaluate the combat readiness of a BIFV squad serving

as the gounding element of a platoon in bounding

overwatch during an offensive maneuver within an armored

threat environment.

This measurement purpose was chosen for several reasons:

0

First, the BIFV weapons system is a complex system and

any attempt to apply the APM to a unit size larger than a

squad would require analyses beyond the scope of the

present effort.

0

Second, a mounted operation with the squad activity

limited to conducting bounding overwatch maneuvers and

an engagement with a stationary armored threat were

selected primarily to keep the number of variables

manageable.

0

Third, it was intended that this application would serve as

a useful experience in utilizing available system

documentation and could adapt in formation/doctrine from

similar type systems as would a military analyst in

applying the methodology to a new system.

Certain assumptions made were with regard to the qualifications

of the BIFV squad/platoon personnel and the BIFV Table of Organization and

Equipment (TOE). These assumptions were as follows:

0

The BIFV/Squad/Platoon personnel are MOS-qualified and

have completed the necessary unit team training in order

to participate in combat readiness exercises.

0

The BIFV squad is manned by TOE-authorized levels. For

the purpose of this application, the assumption was made

that the BI^V squad consists of nine personnel, i.e..

Track Commander, Gunner, Driver and six (6) Firing Port

Operators, one of whom is a fire team leader.

0

The BIFV is prepared for this mission with maximum

authorized TOE supplies and equipment.

-36-
0

The B1FV squad will remain mounted during the entire

tactical operation and would only dismount personnel if

absolutely necessary, e.g., prepare vehicle for fording,

etc.

b. Determine the Hierarchical Structure in Which the System of

Interest is Imbedded

Figure 6 (presented earlier) depicts the external view of the

hierarchical structure providing the immediate context of the BIFV Squad. For

this measurement purpose, the system of interest is imbedded in a

subsystem/system/suprasystem structure that encompasses a BIFV Platoon.

Within the system of interest there are two subsystems: (1) the Carrier Team

Subsystem, consisting of the Track Commander, Gunner and Driver, with their

associated work stations and equipment, and (2) the Crew Compartment Team

Subsystem, consisting of the six Firing Port Weapons Operators and their

associated work stations and equipment. The two subsystems could be further

divided into sub-subsystems which would bring the level to individual system

operators and their associated equipment. However, considering the

measurement purpose, this level is beyond the application interest of the APM.

The collateral systems include two (2) other BIFV squads, one (1) BIFV

squad/platoon headquarters and the threat systeni(s). "ihe suprasystem is the

BIFV platoon.

In applying the APM, one should develop the total number of

taxonomic sets to which the system of interest belongs, including sets for each

subsystem and corresponding sets for the collateral systems and for the

suprasystems. The development of taxonomies for the various system levels

leads to the identification of system-level interactions which influence relevant

performance requirements and. In turn, affect performance measurement issues

for the system of interest.

c. Define and Review Scenarios anil or Documents Required for

This ApplicationIn order to become thoroughly familiar with the BIFV

Squad/Platoon system, available system documents were reviewed. Those

documents included the BIFV Materiel Needs document. BIFV Systems

Specifications, Mission Scenarios, BIFV System Task Descriptions, and Field

Manuals (for similar systems) that provided guidance with regard to tactical

operations. As a point of departure and initial guidance with regard to the

measurement purpose, use was made of the Mission Scenario #1 (Bounding

Overwatch Operation) from the BIFV/CFV Personnel Selection Study (Bloom et

al., 1979). The first page only of that scenario is reproduced here in

Figure 13. A squad verbal order was also developed to serve as guidance,

and to aid in the identification of system population categories relevant to the

contextual stages of the measurement process. The verbal order is presented

in Figure 14. Its associated map is not shown here.

Specifying the measurement purpose also requires description of

the measurement conditions. These include such factors as the geographic

area(s) in which the system is to perform its mission, the time period during

-37-
it

85« Iiuzuu

11

a2I

riaI

is

i!

2>

isut*

il;..sf

«a•

2 JK •* «

1]

15ii

ITn

11II

iittmü1m * i

»5J

til I« ' «•

!i If! M

5'8

I1

?*

8

I]

I*i

J

2 Iaa

iIs3a1I

II IHlllI I 5

iI!

iIiI

isH

IJ6

ijiiIIi

As

111

IJI &z •

tiff « I* -* 1 c -

Iii

1111

b9£

-38-
ITUATION

*W« «re |oinj lo conduct an «ttack ! 0600 hour» today. This tcrrun

ketch IVc made rtpratentt the ground we've jot to cover. As you can see

the attack •ill cover a distance of about 3 kilometers. These little hills.

149 and S4( are platoon intermediate objectives. Hill J53, west of Lahm if

the platoon objective We are presently on Hill S7I. From ohet I was id r

to tee from the limited reeen with the platoon leader, it looks like the encny

hat withdrawn lo positions north of Lahm. Although I didn't tee any armored

vehiclat, Lt.Daviat said the enemy hat been identif>ed at part of a motonted

nflr unit. That meant we have to be on the lookout for BMPs tnd maybe tomr

tanks. We may alto encounter tuch obstacles as minefields and wide trcrrhes.

The minefields are more than likely going to be forward of the trenchlmr.."

Throughout the attack we will stay mounted. The 1st Platoon will be or

our left and the 3rd Platoon on our right. The platoon objective is going to be

hit with artillery and mortar fire just before we go In. The company has ttnks

attached to support us through the attack.

MISSION

"Our mission it to attack each of thete intermediite objectives and final

objective. We have to clear the trenches and bunkers in our tone. I'pon securing Hill 553 we are to prepare defensive positions. Once we're there and sec the

terrain we'll talk mere about it. We will move out at 0600."

EXECUTION

"When we move out. the platoon It going lo be using bounding overwttch

and we are the bounding tquad. With the platoon ovcrwatchmg us from this hill,

we will move up this draw until we get to the patch of woods. I couldn't tee

nn'Ch of these woods: to once we gel close I'll guide the track ovn the remainder

m the route. At toon as the rest of the platoon moves up to us. we'll bound

around to the left and up thit ridge line. I could see it pretty clearly, and I

don't think anything it up there. When the rett of the platoon has moved up

and all the tracks art in position we'll wait for Lt. Davits' order to bound to

the firtt Intermediate objective. Hill 54». Once we've taken Hill S49 we'll wait

for Lt. Daviet' order to continue bounding overwateh. It't important that we

don't gel loo far ahead or behind the rett of the company. If we detect t

mine field we'll try to go around it so keep your eyes open."

"If wc ha"« lo dismount, the platoon targetnt will be controlling the firet

of the carrier team but that it very unlikely. Our priority it to engage any

armorad vchielet In our sector and then bunkers."

"Whan we get to the trenches and enemy contact it made. Igt. Jonas

«ill control the fires of the FPWO's Don't lake any chances, lay down a

good bate of fire. Report all poaaible targelt for the carrier team."

KKVlCt »UPFORT

tfi Jcnaa, you make sure that the aquad compartment and crew have

the required batie load and equipment. Let me know when you're all compleie.

I'll check the Mirier team tuppliet and equipment."

McCarthy, the POL track will be hare around 0430; make sure our fuel

tank ia topped off."

Take care of any casualties within the track) give Mm tome quick first

aid and get back to your position. The company aid track will ecmc forward

)uat at aeon at each Intermediate and final objective is aeeured."

COMMAND AMD SIGNAL

"We will croat the SP at 0(00 on Lt Daviet' hand tignal. We will maintain radio tilenct until we reach Hill S49. During the attack no unnecessary

intercom chatter.'

"In caae I get hit, Igt.Kowaltki, you take ovci our call sign it TANGO 11."

OK, the time it now 0230; I will inapect at 0515 hours It wiU be daylight

at 0S45. Wc have lest than 4 hours before we move out. We will maintain radio

alienee until contact with the enemy it made. The p>atoon and the aquad will

we arm and hand tignalt during the maneuvers. Remember to maintain light

and nei-o diteiphnt. Do you have any questions? if net, get ready."

V

v

Purpose:

Figure 14. Verbal Squad Order

To Attack Hills 549, 546 and 553 (Lichtenfels, Deutschland)

-39-

!

8
which the system is to perform its mission, the weather conditions and any

other limiting variahles. As noted in the verbal squad order for this

application, the geographic area selected is a temperate zone of operations in

Southern Germany; the timf period of performance is 0600 to 1200 hours; the

weather is defined as fair, no precipitation, and good visibility.

d. Begin Applying the APM to the System of Interest to Derive

Taxonomy Sets

The analyses of the two subsystems were initiated first, since it

was felt that combining the subsystem taxonomies would simplify the derivation

of the taxonomy for the system of interest itself. The initial subsystem

selected for analysis was the Carrier Team. The APM revision was applied to

Carrier Team and Crew Compartment Team subsystems, separately, to derive

the fifteen different taxonomy subsets. These correspond to the

"intersections" of each of the three levels of system description with each of

the five contextual stages of the measurement process. Figure 5 (presented

earlier) is a sample of that effort, and incorporates the decimal number

indexing scheme described earlier.

At the Objectives Level, the population categories were assigned

successive numbers in the first decimal position. For example, at the

Objectives Level under Potentialities, each population category was identified

as 1.1, 1.2, 1.3, etc. To be specific, the taxa identified on the Objectives

Level system description for the Potentialities of the BIFV Carrier Team were

designated as follows:

1.1 Providing surveillance

1.2 Conducting weapons fire

1.3 Providing squad command and control

1.4 Providing transportation

1.5 Providing communication

1.6 Providing squad protection

The Functional Purposes Level descriptions were assigned a

two-decimal code, i.e., 1.1.1, 1.1.2, 1.1.3, etc. To continue the above

example, the Functional Purposes Level taxa of "1.1 Providing surveillance"

were designated as:

1.1.1 Conducting sector surveillance

1.1.2 Detecting targets

1.1.3 Acquiring targets

1.1.4 Identifying/classifying targets

1.1.5 Tracking targets

The Characteristics Level of Potentialities was assigned a

three-decimal code, i.e., 1.1.1.1, 1.1.1.2, 1.1.1.3, etc. In subdividing the

functional purpose "1.1.1 Conducting sector surveillance," the characteristics

which describe what capabilities the system must have in order to satisfy that

application were numbered as follows:

1.1.1.1 Unaided visual sector surveillance capability

1.1.1.2 Aided visual sector surveillance capbility

1.1.1.3 Infrared sensor sector surveillance capability

-40-
e. Derivp'ion of a BIFV Measures Hierarchy for the Surveillance

Function

As a test of the utility of the resulting BIFV taxonomy, a set

of measured was derived from those taxn associated with the Carrier Team

Unit's performance of surveillance. The advantages in choosing the

performance element of surveillance were that, first, it is a requirement

common to many military human-machine systems and, second, it entails many

potentialities, processes, and products of performance and is subject to the

influence of many environmental and constraining factors. Thus, as a test of

the measures-derivation process, it permitted a wide variety of system taxa to

be included and offered the possibility of producing measures relevant to many

system applications.

Figure 15 displays the Carrier Team System taxa associated with

the "sirveillance" performance element. On the Objectives Level of system

descripion, these taxa include:

The basic capability (potentiality) of surveillance.

0

The process of sector surveillance, for both air and

ground sectors.

0

The product of threat detections (for which surveillance is

a prerequisite).

0

The physical and clinatological environment.

0

Surveillance constraints.

o Threat constraints (the nature of the threats, and the

tactical response required, may affect how the system

performs surveillance).

All taxa that "descend" from these on the Functional Purposes Level and the

Characteristics Level also are included.

The numbers shown in parentheses in Figure 15 refer to the

taxonomy codification scheme illustrated earlier in Figure 5. They allow the

links among the three levels of description to be indicated explicitly. For

example, taxon 1.1.2.1 "descends" from taxon 1.1.2, which in turn "descends"

from 1,1.

The taxa arrayed in Figure 15 could next be used as search

terms for accessing the body of published literature on human-machine systems

me^si'rement, with a view toward identifyirg measures, analytic methods, and

other test procedures that previously were applied successfully to evaluate

similar systems. For purposes of this example, however, it was not deemed

necessary to conduct a formal literature review. Instead, the staff members

relied on their own svt.tem measurement experience with surveillance systems

(especially those in the infa'.try and ah defense artillery contexts) as the

source from which measures relevant to the listed taxa would be drawn.

-41-
zaw

<

x

8

Q. I

I e <r c

5 <

I

I

u

I

&lu

11 n

33 «

r

sh a

132 3

I«; 1

-2 2 o .£

S

z

o

K

5

z

Ui

PS"*"

0

S« Ü *

"iiii Ji r

< 1 fe =

£ — *.!$ er

u. u. X S t- S i

3

.. -

8 -

0

=

'S S .o

lull

3 s §• k ;

3

£[< WS D u. bu

"a^c.. ä t • * 11 ti 1 f

.2 c

a

o

53

lisll » I If!

.iiiifllll ifi

OS

I

I 3

I I

a a

ni ** ^ •• Hi

a

<

P

z 11

15

P

II

5 a

II

1 I

* 1 2 I

in! 13In

i

J<

9 3

S pi I 3

11

i 2

1= t

inin

SHAUoarao SaSOdMrtd IVNOIXONOi SOIXSIH3XOVHVI'D

-42-
Figures 16, 17, and 18, respectfully, depict the measures that

are potentiality-oriented, process-oriented, and product-oriented. The first

set of measures (Figure 16) are intended to answer questions about the basic

capabilities for surveillance that the system possessed. The second set

(Figure 17) answers questions concerning how the system performs its various

surveillance activities. The third set (Figure 18) addressed questions

concerning the results of those surveillance activities.

Each set of measures is organized in conformance to the APM's

three levels of system description. The uppermost row in each figure contains

only those rroasures associated with Objectives Level taxa. The second row is

reserved for measures associated with Functional Purposes Level taxa. All

lower rows deal with measures associated with Characteristics Level taxa.

Measures arc labelled with reference to the particular attribute (taxon) of

performance on which they bear. Each such taxon is itself designated by a

numeric-decimal label. Each measure associated with a given taxon is labelled

by that taxon's numeric-decimal, with parenthetic alphabetic postscript. For

example, three measures are identified in Figure 16 for the "Surveying Sector"

taxon designated by 1.1.1 (a Functional Purposes Level taxon under the

Performance Potentialities column). Those three measures are designated by

1.1.1(A), 1.1, KB), and 1.1.1(C), respectively.

One important point that bears re-emphasis is that all measures

listed in Pgures 16, 17 and 18 focus on the system's performance of

surveillance. Each measure derives from some particular system taxon, but

the measure is concerned only with »he interaction of that taxon with

surveillance. If some other aspects of system performance (e.g., "Squad

Command and Control," "Communications") had been under study, the relevant

taxa might have included some items that are similar to those listed for

"surveillance." However, even if the taxa were identical for the .several

performanco aspects, different measures would most likely have been derived

because of differing measurement purposes.

Some examples can clarify the preceding point. One of the

product-oriented taxa associated with surveillance is "Tactical Decisions" (Item

5.2.3, on the Functional Purposes Level in Figure 13). The association stems

from the fact that surveillance is a principal source of input for BIFV Squad

tactical decisions. If an analyst wishes to undertake a comprehensive

assessment of the system's performance of surveillance, one issue that must be

addressed, therefore, is "How well does the system's surveillance provide

input to its tactical decisions?" It is precisely that issue that is addressed by

the measures listed under "Tactical Pecision?" in Figure 18. The reader will

note that those measures have nothing directly to do with the kinds of tactical

decisions made, with the correctness of those decisions, with the

decision-making process itself. The measures are concerned only with the

kinds and quantities of tactical decision input data supplied or overlooked by

the surveillance process. Those measures are applied In the context of tactical

decisions, but they remain focused on surveillance.

Suppose for the moment, however, that measures were being

sought not for the "Surveillance" performance element, but rather for "Squad

Command and Control." Tactical decisions would also be included in the set of

43-
*!&£

lip

a«nIISIII

taAUsirio

HI

1 !

iiiyiihiiilii'

IM

a3

I, I1' I }

llllll

I !i J - -

3

»1|

till liM iJlHU

?M I

«I «I• I

ilgillll

l

2SS

I8S

iH- JI31ii~3-

i

8

Iill ill

idIII

III i Ia

«ill i'll pii 'ill 4

80C

444

44

| ill

mil

sasMmw

IVMOUOMfli

Hi1

Uli | it

lei

m8

44

4 I i i

ill! Ilii,! iliil!

lull piili |!li!l I i ii Ml 3 4 4 4

SOUSIMiXOVVVHO I Ig5

-44-
ill

bl

i

!

o m

JI

III If]

liii

1

11ii

5s

< OQ

e4 e4

U

n

vi

<-4

I hill! > § ^ • P. "

< S ü §

R S S »

I

t> ? •* E ^ t

lllllf I

llflell llllili

ft

B Ü

S

llll

!i Hiii

jjllSsSs

< S Ü

5 S "

•4 •I*• H»i HM

I« IIS1

I

I

i

mm

m

< S

»i r«

C4

B

ri

m

ri

ei I

1

1

I

Cfl

b

saALLoaraol SHSOdHfld

IVNOUDNni sDiismaiovHVHO

-45-
iiiiii

mi

3

11

33AI103rSO

5

•4

a

a

a

s

•43

sac

X S X

•< 4 ft

..- «i *

a

a

X

•4a

i § 8 S

s s s

11 li Ji Uhu

^II i! i! lillll

I ! ! § S § •4 •4 i j 2

SHSOdHfld

IVNOUONflJ

111,

mil1

lilili

5m 8m 5m*

«4 «4 «4 •t •< a

« « «

J

11! 1

IM

huh 5 * i ' » *

8 s i I 8 «

1|f«c«

lln-1

« a c

ill

•4 «4 «4

•4 «4 «4

* 4 m

lilili!

i ui! II

3 § § § 9 | 31 222222 32

iiiiii ii

Itli

!

r.i!i,!

IIHlll

liilili

%%%

i i i

SOUSIHaiOVMVHO

f.

c

tr.

X

?

c

o

"S

c

a.

|

>

3

CQ

CO

1—

s

-46-
taxa associated with that performance element because the formulation of

tactical decisions is an integral part of the exercise of command. Now,

however, the key question would be, "How well does the system formulate its

tactical decisions, given the information available?" The measures needed to

address that question would focus on the kinds of decisions made, the decision

algorithms used, the volumes of decisions made, their correctness, their

timeliness, etc. In short, different sets of measures might derive from a

common system taxon, depending upon the performance element of interest.

The basic point is that the taxa are abstractions of measures. The concrete

specification of the measures they represent depend on exactly which facet or

element of the system's performance is being studied.

The measures hierarchy depicted in Figures 16, 17 and 18, of

course, is an idealized set. That is, no attempt has been made to determine

the feasibility of using any of the measures, or to eliminate those that may

require data that cannot be obtained under realistic circumstances. That

"weeding out" activity properly belongs at a later stage in the overall

measurement process. The purpose of the taxa is to guide the analyst in

identifying and organizing a set of measures that, if they can be used, would

answer all the research questions at hand. The taxa themselves shed little or

no light on the practicality of those measures.

There is no conclusive test or set of standards that can be

used to judge the merits of the "Surveillance" measures hierarchy. The true

merits of any measure can be known only when the measure is actually

employed under the intended circumstances. Nevertheless, the measures

themselves appear intuitively to be well suited to assess the BIFV Squad's

performance of surveillance. The measures would allow the analyst to proceed

in a careful and logical step-by-step fashion. Applied in the proper sequence,

they would address the following kinds of questions:

What capabilities does the BIFV Squad possess in the area

of surveillance?

What surveillance capabilities did the system actually use

in this particular measurement application?

0 How were these capabilities affected by the environmental

and other circumstances that pertained to this particular

measurement application?

0 What procedures did the system actually use to apply its

surveillance capabilities?

0 How did these procedures differ from the prescribed

surveillance procedures?

0 How did the environmental and other circumstances affect

the surveillance procedures?

For what specific applications were the surveillance

procedures conducted?

-47-
0 Was sufficient and appropriate information or other input

supplied to those applications through surveillance?

0 What specific deficiencies were found In the information/

input provided by surveillance?

Achieving the ability to answer questions such as these in

every analytic application would constitute substantial progress in the

state-of-art of human-machine systems measurement. In this specific instance

at least, the APM has produced a measures set that apparently is structured

around the right kinds of research questions, and that possesses the logical

sequential organization required to address those questions.

2. Derivation of Measurable Performance Objectives for the BIFV

Training System (B1FVTS)

a. General Conditions

In this second sample illustration of applying the APM for

Measurement, the focus is shifted to the training system associated with the

BIFV. The information in these paragraphs summarizes a more extensive and

detailed discussion reported in an earlier project document (Bloom, et al.,

1982, Section IIC and Appendix A). Using the training system hierarchical

structure defined earlier in this report, and illustrated in Figure 9, this APM

application was conducted for the purpose of deriving measures to assess how

well the Design subsystem of the BIFVTS specifies the testing of gunner

trainees.

b. Determine the Hierarchical Structure in Which the System of

Interest is ImbeddedFigure 19 shows the system hierarchy for the BIFVTS—that is,

its relationship to other training systems. Every training system can be

conceived of as depicted previously in Figure 9, with six subsystems

(Command, Design, Enabling, Emplacement, Logi.ctics and Delivery).

The BIFVTS Command subsystem is operated and equipped by

the BIFV Task Force, an ad hoc subsystem of the U.S. Army Infantry School.

The BIFV Task Force has been created and staffed to exercise overall

responsibility for the design, development, testing, and final production and

dissemination of BIFV training. The specific immediate responsibilities of the

Task Force are to:

Identify the specific skills and knowledge required of BIFV

operators.

0 Develop BIFV training strategies.

0

Establish new equipment training (NET) for the BIFV.

Administer, monitor, and evaluate all BIFV training

development, testing, and production.

-48-
U.S. Army

Armor

School

Bradley

Cavalry

Fighting

Vehicle

Training

System

Bradley

Cavalry

Fighting

Vehicle

Figure 19. Major Systems Interacting with BIFVTS

-49-
The reader will recognize that several of the ahove-listed

responsibilities are actually training design functions. Indeed, the BIFVTS

Design subsystem also is operated (at present) solely by members of the BIFV

Task Force. Their current design work is oriented toward the production of

five distinct curricula:

0 An "add-on" course for Skill Level 1 lilFV Soldiers (MOS

11M10).

0 A Basic Gunnery Course (for Skill Level 2 Soldiers [MOS

11M20| and some Skill Level 1 Soldiers).

0 A BIFV Commander's Course (for Skill Levels 3-5 I11M30,

11M40, 11M501 and for officer grades 01-05).

0 A transitional new equipment training (NET) course

covering the above-listed skill levels.

0 A Master Gunner's Course.

The Master Gunner's curriculum principally is intended as an

input to the BIFVTS Enabling subsystem. One E7 or E8 from the S3 section of

each BIFV mechanized infantry battalion will receive Master Gunnery training,

as will one E6 from each BIFV company. The Master Gunner's duties include

the administration, supervision and delivery of Basic Gunnery training within

his or her unit. The Master Gunner, in effect, will' be a field

facilitator/administrator within BIFVTS. Other current or planned opeiations

within the BIFVTS Enabling subsystem include the organization and deployment

of a team of facilitators to support new equipment training on a transitional

basis as the BIFV is introduced into operational units; familiarizing .selected

mechanized infantry instructor/facilitators with the emerging BIFV curricula;

and delivering Mastery Gunnery training. The Enabling subsystem must have

the potentiality for insuring that adequate instructional skills resources are

provided for BIFVTS Delivery applications.

The BIFVTS Emplacement subsystem is responsible for the

design and selection/fabrication of all audiovisuals and other learning aids

necessary to support implementation of the various curricula, and for the

acquisition or construction and preparation of all sites at which learning

activities will take place, whether in a classroom or field setting. One

important element of this responsibility will be the design and construction of

gunnery practice ranges. The range requirements presently envisioned

include:

0

Firing port weapons range

0 BIFV Basic Gunners range

0 BIFV Squad subcaliber range

0 BIFV Squad qualification range

0

Platoon qualification attack range

0

Platoon qualification defense range

Note that all of the BIFVTS Emplacement subsystem's

performance requirements derive from the need to insure that all facilities and

-50-
materialb nccossary to support delivery of BIFV training are available and

operable at the times and places needed.

The RIFVTS Logistics subsystem presently is operated solely as

a component of the Fort Benning logistic support system. In time,

increasingly greater shares of the operation of that subsystem will be

exercised by various mechanized infantry units in the field.

The BIFVTS Delivery subsystem will allow and insure that

learning takes place in the contexts of the various curricula produced by the

Design subsystem. Soldiers and officers will carry out prescribed sensory

activities, guided and assisted by facilitators, and will achieve the specified

performance objectives applicable to t'aeir BIFV operational assignments.

BIFVTS thus is r well structured training system, possessing

functionally oriented subsystems compatible with the general training model and

pursuing a broad range of administrative, developmental, and delivery goals to

support the introduction of the new BIFV into the mechanized infantry. It

does not detract at all from that image to note that BIFVTS can also be viewed

as a loose consortium of smaller scale, relatively independent training systems

that pursue more narrowly defined goals. These smaller systems are organized

around the five BIFVTS curricula now in development. These systems may be

denoted as:

0

The BIFV NET System 0

The Basic BIFV Soldier Training System

0

The Basic BIFV Gunnery Training System

0

The Master BIFV Gunnery Training System 0

The BIFV Commander Training System

In order to reduce the scope of this trial application to a

practical level, the focus is narrowed to the Basic BIFV Gunnery Training

System. Like all systems, this one operates in the context of an interactive

system network of suprasystems, collateral systems, and subsystems. The

network is shown in Figure 20. The Basic Gunnery system is subservient to

such entities as the U.S. Army Infantry School, the Fort Benning Logistic

Support System and, of course, the total BIFVTS. It interacts collaterally

with the other four independent BIFV training systems and with portions of

U.S. Army Infantry units, namely, those portions from which the E4,s and

EB's will be recruited as candidate gunners. It also interacts collaterally with

the BIFV Carrier Team subsystem. That is the portion of the total BIFV

system for which the gunner-learners are to become qualified as operators.

And finally, the Basic BIFV Gunnery Training System has a fraternal

(collateral) twinship with the Basic BCFV Gunnery Training System. The

subsystems of Basic BIFV Gunnery Training are, of course, the six familiar

subsystems of Command, Enabling, Design, Emplacement. Logistics, and

Delivery.

The focus of analysis was narrowed even further by addressing

only the Design Subsystem of BIFV Basic Gunnery Training. Figure 21

depicts the network of systems that interact most closely with the Design

subsystem of Basic BIFV Gunnery Training. As in any training system.

-51-
U.t. Army

Armor

School

r övirilf

i Bridlcy

i Cavalry

i Fifhting

i Vehicle

i Trüning

i System

Overall

BIFVTS

(8upra-»yitem)

U.S. Army

Infantry

School

(Supra-tyttem)

asic

BCFV

Gunnery

Training

Syitcm

fort

Benmng

Logistic

Support

System Skill Uvel 1

BIFV Soldier

Training

System

Batic

BIFV

Gunnery

Training

Syitem

- -UfFvi 1 " 1 i Carrier i Carrier

i Tiam i Team

• Unit i Unit

Subayatemj Subsystem

Command

BIFV

Commander»

Training

System

1

—nrv— NCM

Equipment

Training

Syatem

Matter

BIFV

Gunnery

Training

Syttem

LB. Army

Wantry

Unit

.S. Army

(Selected E4t,

ESs with

MOS 1IM

qualifteatlon)

4s,

M».

h)

S. Army •

InfanWsT

M E4s,

with

I1M

ation)

n)

Daalgn

EnabUng

Delivery

EmpUcemenl Ufistiea

Figure 20. Major Systems Interacting with the

Basic BIFV Gunnery Training System

-52-

a
Envelopment

of Curriculum

for

Master BIFV

Gunnery

Training

System

ENABLING

Preparation

of Instructors

for the

Basic BIFV

Gunnery

Training

System

I -^

**!-/.:? ^ ^

DELIVERY

Delivery

of Instruction

for the

Basic BIFV

Gunnery

Training

System

COMMAND

Administration

Coordination

of the Basic

BIFV Gunnery

Training

System

Development

of Curriculum

for

BIFV New

Equipment

Training

tern

Development

of Curriculum

for

Skill Level 1

BIFV Soldier

Training

System

DESIGN

TTT

III

'II

Hi

< i i

' i i

i i i

' i •

1

i i

"TIFV—

Carrier

Team

Unit

Subsystem

Development

of Curricula

for

Basic BIFV

Gunnery

Training

System

/

, L Development of

Squad and

Crew Drill

Training

Curriculum

Development

of Pre-fire

Gunnery

Training

Curriculum

Development o

Squad and Crew

Subcaliber

Training

Curriculum

Development of

Platoon/Section

Battle DriU

Training

Curriculum

velopment of

Vehicle Team

and Crew Combat

Exercises

Curriculum

(Continued)

Development'

of Curriculum

for

BIFV

Commanders

Training

System

EMPLACEMENT

Development

of Facilities

for

Basic BIFV

Gunnery

Training

System

LOGISTICS

Logistic

Support

for the

Basic BIFV

Gunnery

Training

System

* o

Development of

Platoon/Section

Qualification

Exercise

Curriculum

Development of

Squad Combat

Qualification

Exercise

Curriculum

System of Interest: The Design Subsystem of the Basic BIFV Gunnery Training System

(Continued

Figure 21. Sample Hierarchical Structure Centered on

a Subsystem of a Training System

-53-
Squad and

Crew

Drill

Instruction

Pre-fire

Gunnery

Training

Imtruction

Squad an3~

Crew

Subcalitxr

Instruction

Bqusd

Combat

Qualification

Instruction

• •

I""^- DELIVERY \

Vehicle Teem

and Crew

Combat

Instruction

Delivery J

of Instruction

for the

Basic Bin

Gunnery

Training 1

---"•"* ^^"-z.^' " '

PlätöönT

Section

Battle Drill

Instruction

EMPLACEMENT

Development

of Facilities

for

Basic BIFV

Gunnery

Training

System

\ Platoon/

Section

Qualification

| Instruction

Firir« Port

Weapon

Range Basic BIFV

Gunners'

Range Squad

Subcaliber

Range

Platoon

«4 Qualification

Defense Range

Platoon " Qualification

Attack Range

Squad

.Qualification

Range

TITC—

Carrier

Team

Unit

Subsystern

7.S2mm

Machine Gun

Sub-Subsystem

XSmm

Cannon

lub-Subayitem

T&R

ATGM

Jub-Bubayatam

Figure 21. Sample Hierarchical Structure Centered on

a Subsystem of a Training System (Concluded)

-54-
Gunnery Training Design is subservient to the Command subsystem, and

collateral to the Enabling, hmplacement, Logistics, and Delivery subsystems.

Key sub-subsystems of both Emplacement and Delivery have been included in

Figure ^1 to emphasize the special gunnery instruction and practice

requirements that must be reflected in Design's products. Gunnery Training

Design is alfo collateral to the Design subsystems of the other BIFV Training

"packages." This is because all of those packages share some performance

objective?, learning activities, etc., with gunnery training.

The sub-subsystems of Gunnery Training Design correspond to

the thus far identified segments of the gunnery curriculum. These represent

the sequential stages of learning through which BIFV gunner-learners must

pass to become fully qualified. The initial stages stress individual training;

the latter, team training.

c. More About the Measurement Application; Specification of

Gunner-Learner Testing

Having selected the Design subsystem of Basic BIFV Gunnery

Training as the system of interest for this trial, it remains to select an issue

of focus for the application (i.e., an assessment of how well Design meets some

Mmited aspect of its performance requirements). Ultimately, all assessments of

training system or subsystem performance must supply some portion of the

answer to this fundamental question: Did the learners learn what they were

supposed to learn?

Design's basic job is to provide training specifications. It

specifies what is to be learned, the context of learning, and the learning

activities. One sub-issue of specifying learning activities was selected for this

trial applic ..ion: specifying gunner-learner testing. The other sub-issues of

that heading are: specifying gunner-learner preparation activities; specifying

activities for making presentations (or demonstrations) to gunner-learners; and

specifying gunner-learner application (or practice) activities. Virtually all

learning theorist- agree that preparation, presentation, application, and

evaluation (or testing) constitute the four classes of learning activities. The

issue of how well a Design subsystem specifies learner testing materials and

procedures is of special importance in any training system. No training

system can afford to allow its Design subsystem to produce poor measures of

learners' achievements. The Basic BIFV Gunnery Training System is no

exception.

d. Begin Applying the APM to the System of Interest to Derive

Performance Attribuus and Measures

Before moving on to the next step in this trial application, In

which particular performance taxa associated with learner testing specification

will be identified, it is worthwhile to set the stage by informally reviewing

some of the requirements and implications ox learner testing in any training

application. The following, at least, seem almost self-evident:

0

Testing must be objectives-referenced

The purpose of testing is to evaluate learners' progress

toward or achievement of the performance objectives, i.e.,

-55-
that which is to be learned. No performance objective

should go untested. No test that does not relate to some

performance objective should be included.

Testing must be criteria-referenced

It is not enough that a test simply requires the learner to

exercise the behavior called for in a performance objective.

The test must disclose whether the behavior is performed

" »ell enough" to satisfy learning needs.

Some kind of pre-training testing is necessary

All training assumes some prerequisite behavior. If

nothing else, it is always assumed that the learner is able

to carry out the sensory activities called for in the

training. Many training applications admit the possibility

that some learners may already have achieved certain of

the specified performance objectives. There is no point in

commencing training unless the learner first is tested to

insure that he or she possesses the assumed prerequisites.

A very informal test might suffice for that purpose,

perhaps only a records check or even a cursory visual

examination of the learner, but some testing for

prerequisites surely is needed. When there is reason to

believe that some learners may already have achieved the

learning objectives, pre-testing of those objectives also will

be necessary.

Some testing for feedback is necessary

Testing is, after all, a learning activity. Learners need

to know how well they are doing in order to progress and

persevere. Any training system should provide at least

some testing for the purpose of feeding back progress and

diagnostic information to the learners.

Testing must diagnose learning deficiencies

It is not sufficient that tests disclose that a learner cannot

perform the intended behavior. In order to provide

meaningful feedback to the learner and to facilitate

subsequent improvement of the curricul; .n and the

delivery, tests that diagnose the specific ingredients of

substandard learner performance must be included in the

specifications. Only then can the learner's performance be

corrected and brought up to standard.

Testing must be reflective of on-the-job performance

The tests specified must require the learners to manifest

the job-relevant behaviors under job-relevant conditions.

The testing conditions should resemble job conditions as

-56-
closely as possible, since the test outcome otherwise may

not be a valid predictor of the learner's job performance.

0

Some kind of qualification testing is necessary

No learner should be "certified" as having successfully

completed training unless he or she demonstrates, through

testing, that the performance objectives have been

achieved. Depending on the circumstances of the

particular training experience, the qualification testing

might be formal or informal, but some testing surely is

needed.

The preceding list probably would occur to any evaluator

interested in measuring the adequacy cf a Design subsystem's specification of

its learner testing program. With that list in mind, the next step is to review

the Desipn subsystem performance taxa of Appendix A for relevance to

measuring the adequacy of specifying the gunner-learner testing program.

Two hundred and eighty-one taxa of Design Subsystem

performance requirements are listed in Appendix A. These are distributed

among the three aspects of performance (potentialities, processes, and

products) and across the three levels of system description (objectives,

functional purposes, and characteristics). Some of the 281 taxa are relevant

to evaluation of a Design Subsystem's performance in specifying learner

testing. Others are not. At the present stage of development of the APM,

well defined procedures for determining which taxa do or do not bear on a

given measurement application are yet to be established. Indeed, one purpose

of this trial application was to add to the base of experience from which such

procedures may be derived. Currently, assessment of taxa relevance is a

"judgment call" by an analyst or team of analysts working together. The

thought process can be described as follows: "Here we have some aspect of

this sytern's performance, and over here we have a performance taxon for the

system. That taxon represents a particular requirement that the system is

supposed to satisfy. Suppose the system does a bad job in trying to satisfy

that requirement. Might that have any bearing on the particular aspect that

we are studying?" If the analyst concludes that the answer is "yes," the

taxon in question would be included among the subset from which measurement

issues and implications, and ultimately measures, will be derived. If the

answer is "no," the particular taxon will supply no input to the measurement

application at hand.

Identification of relevant taxa, then, is a step-by-step march

through an entire system taxonomy, during which each taxon is picked up in

turn, examined, and finally accepted or rejected in accordance with whether or

not it appears to the analyst to have something of interest to say. This

process can be streamlined somewhat, simply by examining only the

characteristics-level taxa. If a taxon on the characteristics level is found to

be relevant to the particular measurement application, then the functional

purposes-level taxon ^rom which it derives also must be relevant. A

characteristics-level taxon, after all, is simply a detailed performance

requirement that the system must satisfy if it is to achieve the related

-57-
functional purpose. Thus, the functional purpose must be subsumed under

the aspect of performance being studied. Similarly, if a functional

purposes-level taxon is relevant to the measurement application, so must be

the objectives-level taxon from which it derives. The functional purpose,

after all, is only a required application of the basic potentiality, process, or

product defined on the objectives level. For example, if the analyst in using

Appendix A were to conclude that taxon number 2.1.3.1 (a characteristicslevel process requirement) is of significance to the measurement application, he

or she would automatically know that taxa 2.1.3 and 2.1 (its functional

purposes-level and objectives-level antecedents) also are relevant.

An examination was made (Bloom, et al. , 1982) of every

characteristics-level taxon of potentiality of the general Training Design

subsystem for possible relevc -n to assessment of the subsystem's performance

in specifying learner testing. Rationale for concluding whether an individual

taxon is or is not relevant to the measurement issue is presented there.

Similar (but undocumented) examinations were made of the process and product

taxa. The final conclusion of these examinations was that 119 of the 281 taxa

are relevant to evaluation of the Design Subsystem's specifications of learner

testing. Those relevant taxa form 14 taxonomic subsets, of whic!» 5 are

potentialities, 3 are processes, and 6 are products. Figure 22 is a sample of

4 of the 14 subsets.

Alongside each of the evaluative issues arrayed in Figure 22,

the reader will find a set of attributes deemed to be relevant to the particular

issue. Each attribute is a concrete element of the Design subsystem's work.

It is something that can be seen, examined, submitted to analysis, taken

apart, etc. In short, each attribute is something that is measurable.

Further, each attribute is something that, when measured, can supply a

portion of the answer to the question posed in the evaluative issue.

An example may help to clarify the relationship between issues

and attributes. Consider the fifth issue in Subset No. 6 (Figure 22), "Does

the system accurately identify the physical skills that BIFV gunners must

acquire to perform their tasks, so that appropriate tests of physical skills ma^

be specified and developed?" This issue arises from a characteristics-level

process, i.e., the analytic process used to identify the physical skills learners

must acquire in order to serve as BIFV gunners. Clearly, tests of all

required physical skills must be specified and administered as part of the

gunner certification process. If the system's analysis is deficient, so that

certain essential physical skills are not identified, or so that some irrelevant

physical skills are mistakenly identified as essential, the testing specifications

also will be deficient. As the design work progresses, the sponsors certainly

would wish to know whether a thorough and accurate analysis ol required

physical skills is underway. In order to assess that issue, the evaluator

would look first at the analytic methods the training designers are using to

identify required physical skills. Are those methods appropriate to the

analytic needs at hand? Are all of the data required for applicaton of those

methods being obtained and used appropriately? Is there any fault to find

with the way in which the training designers are applying those methods? As

the analyses begin to produce results, the evaluator would examine the

physical skills requirements identified. Are all of those skills really pertinent

-58-
PART 1: Issues and Atlributcs of "Potentinlities" (Subsets 1-5)

Subset No. 2: 'EstablUhing Performance Objectives"

Issues Attributes

1.2 Does the system have the ability

to Insure that all BIFV gunnery

performance objectives will be

identified so that they may be

addressed in gunner testing specifications?

1.2.1 Can the system insure that all

things BIFV gunners will be able

to do after training will be

stated, so that tests of all

required abilities may be

specified?

1.2.1.1 Can the system insure that the

stated abilities will include

definitions of the required

gunner actions, so that the test

specifications may address those

actions?

1.2.1.2 Can the system insure that the

stated abilities will include

definitions of the appropriate

conditions under which BIFV gunners are to perform the actions,

so that the test specifications

may reflect those conditions?

1.2.1.3 Can the system insure that the

stated abilities will include

definitions of the appropriate

standards of acceptable performance of the BIFV gunner

actions, so that the test specifications may incorporate those

standards?

1.2.2 Can the system insure that the

identified BIFV gunnery performance objectives will be applied

to provide a basis for objectively assessing gunners' performances?

1.2.2.1 Can the system insure that

appropriate pretest items

addressing the performance objectives will be developed, so

that they may be included in the

testing specifications?

1.2.2.2 Can the system insure that

appropriate posttest items

addressing the performance objectives will be developed, so

that they may be included in

the testing specifications?

Scope of information available concerning BIFV gunners' tasks

Staff experience and qualifications

in establishing performance objectives

The planned approach to establishing

BIFV gunnery performance objectives

The resources allocated to establishing BIFV gunnery performance objectives

The elements or factors of gunner

abilities intended to be stated

Intended formats for stating BIFV

gunner abilities

The plans for stating intended

abilities

The types of performance actions

intended to be defined

The plans for defining actions

required of BIFV gunners

The elements or factors of performance condition intended to be

defined

The plans for defining BIFV gunnery

performance conditions

0 Elements or factors of performance

standards Intended to be defined

0

The plans for defining standards for

BIFV gunnery performance actions

Elements of the basis for assessment

Intended to be provided

Plans for providing a basis for objectively assessing BIFV gunners'

performances

Types of BIFV gunnery pretest items

Intended to be developed

Plans for developing BIFV gunnery

pretest items

Types of BIPV gunnery posttest items

Intended to be developed

Plans for developing BIFV gunnery

posttest items

Figure 22. Measurement Issues and Measurable Attributes Relevant to

Evaluation of BIFV Gunnery Testing Specifications

-59-
PART 2: Issues and Attributes of "Processes" (Subsets 6-8)

Subset No. 6: "Analyie Tasks Selected for Training"

Issues ~ Attributes

2.3

2.3.2

Does the system conduct a proper

analysis of B1FV gunner tasks,

•o that appropriate tests of

those tasks may be specified?

Does the system accurately identify the domains of learning

Involved in the B1FV gunner

tasks, so that tests may be specified and developed in the

proper domains?

2.3.2.1 Does the system accurately identify the knowledge that BIFV

gunners must acqvire to perform

their tasks, so that appropriate

tests of knowledge may be specified and developed?

2.3.2.2 Does the system accurately identify the mental skills that BIFV

gunners must acquire to perform

their tasks, so that appropria'

tests of mental skills may be

specified and developed?

The task analytic methods applied to

the BIFV gunner tasks

The scope of the task analyses conducted (i.e., factors determined in

the analyses)

The completeness of the BIFV gunner

task analyses

The accuracy of the BIFV gunner task

analyses

Methods employed to identify the

domains of learning involved in the

BIFV gunnrr tasks

The domains of learning identified

for each BIFV gunner task

Methods employed to Identify the

the knowledge requirements associated with BIFV gunner tasks

The BIFV gunner task knowledge

requirements identified

Methods employed to identify the

mental skills requirements associated with BIFV gunner tasks

The BIFV gunner task mental skills

requirements identified

2.3.2.3 Does the system accurately identify the physical skills that

BIFV gunners must acquire to

perform their tasks, so that

appropriate tests of physical

kills may be specified and

developed?

2.3.2.4 Does the system accurately identify the attitudes that BIFV

gunners must acquire to perform

their tasks, to that appropriate

tests of attitude may be specified and developed?

2.3.3 Does the system accurately identify the conditions for learning

associated with the BIFV gunner

tasks, so that appropriate tests

may be specified and developed

for assessing candidate gunners'

qualifications?

2.3.3.1 Does the system accurately identify BIFV gunners' prerequisite

knowledge, so that appropriate

tests of the knowledge prerequisites may be specified and

developed?

Methoda employed to identify the

physical ski.U- require tients associated with B'FV gunner tasks

The BIFV gur. er task physical skills

requirements identified

0 Methods employed to identify the

attitudinal requirements associated

with BIFV gunner tasks

0

The BIFV gunner task attitudinal

requirements identified

Methods employed to identify the

conditions for learning the BIFV

gunner tasks

Types of conditions for learning

identified for each BIFV gunner task

Methods employed to identify the prerequisite knowledge required for

learning the BIFV gunner tasks

Knowledge items identified as prerequisite for learning the BIFV

gunner tasks

Figure 22 (Continued)

-60-
PART 3: Issues and Attributes of "Products" (Subsets 9-14)

Subset No. 10: 'Stated Performance Objectives*

Issues Attribut es

3.2

3.2.1

3.2.1.1

3.2.1.2

3.2.1.3

3.2.1.4

3.2.1.5

3.2.1.6

3.2.1.7

Does the system produce accurate

and complete statements of the

performance objectives that BIFV

gunners must achieve in order to

be qualified for that job, ao

thai tests of gunners' quslificationa may be specified?

Does the system produce exact

definitions of what the gunnerlearner are expected to achieve

through training, so that tests

of thoe achievements may be

developed?

Does the system produce correct

specifications of the capabilities that gunner-learners are

expected to achieve?

Does the system produce correct

specifications of the actions

that the gunner-learners are to

execute to demonstrate the

capabilities?

Does the system produce correct

specifications of the objects

that are to result from the

execution of the actions by the

gunner-learners?

Does the system produce correct

specifications of the circumstances under which the gunnerlearners are to execute the

actions to produce the objects?

Does the system produce correct

specifications of the tools and

equipment BIFV gunner-learners

are to uae in executing the

actions?

Does the system produce correct

specifications of the constraints to be Imposed upon the

execution of the actions by the

gunner-learners?

Does the system produce correct

specifications of the criteria

to be used to judge the adequacy

of the objects resulting from

the actions executed by the

gunner-learners?

The BIFV gunner performance objectives that are stated

The format of the statements of

BIFV gunner performance objectives

The completeness of the performance

objective statements

The accuracy of the performance

objective statements

The gunner-learner achievements

that are defined

The factora included in the definitions of gunner-learner achievements

0 Capabilities specif'-d for achievement by BIFV gunner-learners

Actions specified for execution by

BIFV gunner-learners

Observability of the actions

0 Objects specified as results of the

action executions by BIFV gunnerlearners

0 Measurability of the objects

Circumstances specified for execution of the actions by BIFV gunnerlearners

Tools and equipment specified for

use in the execution of the actions

by BIFV gunner-learners

Constraints specified for the

execution of the actions by the

BIFV gunner-learners

Criteria specified as standards of

adequacy for the objects resulting

from the gunner-learners' actions

Figure 22. (Continued)

-61-
PARI' 3 (Continued)

Subset No. 13: Tests"

Issues Attributes

3.5 Does the system produce appropriate tests for administration

to BIFV gunner-learners?

3.5.1 Does the system produce appropriate tests for assessing the

qualifications of candidate BIFV

gunner-learners?

3.5.1.1 Do the candidate qualification

tests provide adequate measures

of the candidates' prerequisite

abilities?

3.5.1.2 Are appropriate procedures

specified for applying the

qualification tests to BIFV

gunner candidates?

3.5.1.3 Are appropriate standards fo»-

qualification as BIFV gunner

specified for the candidate

qualification tests?

3.5.2 Does the system produce appropriate tests for assessing

requirements for tailoring the

Instruction to accepted BIFV

gunner-learners?

3.5.2.1 Do the tailoring tests provide

adequate pre-training measures

of accepted BIFV gunnerlearners' abilities?

3.5.2.2 Are appropriate procedures specified for applying the tailoring tests to accepted BIFV

gunner-learners?

3.5.2.3 Are appropriate standards for

tailoring instruction specified

for the BIFV gunnery tailoring

tests?

BIFV gunner-learner tests prepared

BIFV gunnery testing applications/

requirements for which tests are

prepared

BIFV gunner behaviors for which the

tests are prepared

Relevance of the tests

" Candidate gunner-learner qualifications for which tests are prepared

0 Tests prepared for assessment

of those qualifications

0 Domains of learning addressed

by the tests

0 Actions elicited by the tests

0 Objects intended to result from the

actions elicited by the tests

Circumstances specified for the

tests

Tools and equipment specified for

the tests

Constraints specified for the tests

Other aspects of application!! procedures specified for the tests

Specific standards specified for

the qualification tests

BIFV gunnery abilities for which

tailoring tests are prepared

Tests prepared for assessment of

needs for tailoring instruction to

fit gunner-learners' abilities

Domains of learning addressed by

the tests

Actions elicited by the tests

Objects intended to result from the

actions elicited by th.- tests

Circumstances specified for the tests

Tools and equipment specified for

the tests

Constraints specified for the tests

Other aspects of applications procedures specified for the tests

Specific standards specified for

the tailoring tests

3.5.3 Does the system produce appropriate tests for measuring what

gunner-learners have achieved

as a result of BIFV gunnery

training?

BIFV gunnery abilities for which

achievement tests are prepared

Tests prepared for assessment of

BIFV gunner-learners' achievements

Figure 22. (Concluded)

-62-
to BIFV gunnery operations? Are there any essential skills that haven't been

identified in this analysis? Are there any skills that the designers have

laaelled "physical" that really are "mental?" Based on these and similar

considerations, the evaluator can form a well reasoned judgment about the

accuracy of the designers' approach to identifying physical skills requirements.

That reasoned judgment will be based on measures applied to the two concrete

attributes, viz., the analytic methods of skills identification and the identified

skills themselves.

The evaluative issues thus lead to measurable attributes, and

the attributes in turn lead to measures of performance and effectiveness. The

question is: How does all this "leading" proceed? How does one discern the

attributes in the issues, and how does one derive the measures from the

attributes?

No precise or even preliminary guidelines, algorithms or

heuristics for extracting attribui.es from issues have thus far been developed.

A major purpose of this trial application of system performance taxonomization

to evaluating BIFV Guunery testing specifications Is to generate an experiential

base from which such guidelines can begin to be derived. But in very general

terms, the analyst's thought process in searching for the attributes of a

particular issue relevant to testing specification can be sketched this way:

"Here is an issue that bears on how well the BIFV Gunnery

Design Subsystem can define (or has defined) gunner testing

specifications. I'm supposed to suggest a way of answering the

question posed in this issue. First, I have to ask myself:

Which of the subsystem's people, resources/equipment, and

procedures are involved in this issue? Next, what qualities or

characteristics do those involved people, resources/equipment,

and procedures have to possess, or produce in their' work, if

this issue is to be resolved affirmatively? Finally, what

concrete, observable factors associated with those people,

resources/equipment, procedures, or their work can I examaine

to determine whether those qualities or characteristics are

present?"

The concrete, observable factors uncovered by the analyst are

the attributes associated with the issue in question. The mental process

outlined above was applied to each of the 119 performance-related issues

deemed relevant to evaluation of BIFV gunner testing specifications. A sample

of the outcome is the set of attributes presented along with the issues in

Figure 22.

If the reader will accept the attribute set given in Figure 22 as

being at least illustrative of the bases from which the evaluative issues can be

addressed, attention can now turn to the derivation of measures from

attributes. A measure, in most general terms, is a judgment or appraisal

about the tl ing-being-measured. Measures applied to system atti ibutes

derived from various taxa of performance thus are judgments or appraisals of

performance. Each such measure contributes some small bit of wisdom or

insight about the total performanre of the system-being-measured. Every

measure "looks" at its attribute in its own unique way, and "weighs" the

attribute on its own unique scales. Collectively, the measures are intended to

-63-
determine whether the attribute is "good enough" to meet the demands of the

performance requirement from which it derives. Individually, each measure

focuses on some particular aspect or dimension of "goodness."

How does one measure whether a set of knowledge is "good

enough." First, one specifies or states exactly what is known, i.e., the

contents of the set of knowledge. Next, one identifies missing elements in the

set of knowledge;, i.e., facts or other information that should be known, but

aren't. Finally, one identifies inaccuracies in the set of knowledge, i.e.,

supposed "facts" and other Information that are not true. This is a

roundabout way of saying simply that a person's knowledge of a given subject

might be "bad" in any of three ways:

the person might know nothing about the subject; or

might know some things but not others; or

0 might "know" some things wrong.

A combination of the last two is also possible.

In the specific context of the Design Subsystem's knowledge of

the BIFV Gunner Tasks, the evaluator would need to determine, first, whether

the subsystem knows anything about any of the tasks; second, whether some

tasks are unknown to the subsystem; and, third, whether some of the "known"

tasks are in fact not tasks required of BIFV gunners. Each of those

determinations provides a separate judgment or appraisal of the goodness of

the scope of information the Design Subsystem proposes to use as the basis for

establishing BIFV gunnery performance objectives. That is, each is a separate

measure of that one attribute of the system's potential for establishing

objectives. The BIFV Task Force, in fact, has assembled a very

comprehensive task data base which helps to meet the Potentiality aspects of

this exercise.

Consider another example in the context of this same

performance potentiality. Along with having good information about gunnery

tacks, the Design Subsystem must have people who are qualified to do the job

of establishing performance objectives. The staff's experience and

qualifications thus constitute another attribute of the system's ability to

produce well founded objectives. If their qualifications aren't "good enough,"

th»~ '"stem's ability to establish objectives also won't be good enough.

How does one measure the "goodness" of an individual's or

team's qualifications for doing a particular job? When the job is relatively

simple and of short duration, a candidate's qualifications sometimes can be

measured by requesting him or her to actually do the work in question, on a

sample basis. A typing test, for example, often is administered to candidates

for a secretarial position. Then, actual job performance measures (e.g.,

words typed per minute, percentage of errors, etc.) can be applied to the

sample work and used as job qualification measures. However, when the work

is complex, cerebral and non-routine, this approach may not be practical. In

such cases, clinical judgments usually are employed to appraise the candidate's

qualifications. Subject-matter experts review resumes of the available

candidates' experience and training relevant to the job to be filled, interview

-64-
personal references named by the candidates, perhaps examine the products

delivered by the candidates during previous assignments similar to the job to

be filled, and maybe interview the candidates themselves to obtain greater

insight concerning the skills and knowledge they could bring to the job. Each

expert reviewer then produces an independent rating of each candidate, based

on the reviewer's perception of the candidate's strengths and weaknesses.

Such ratings often are placed on an interval or ordinal scale. Then, the

expert reviewers meet as a group, report and explain their individual ratings,

deliberate on and debate the issues raised concerning each candidate, and form

a consensus rating of the qualifications of each.

This clinical approach can be applied to measure the Training

Design staff's qualifications for establishing gunnery performance objectives.

The appropriate measures would include the consensus rating of each

individual proposed for assignment to the task of establishing objectives, and

an overall consensus rating of the total proposed staff as a team for handling

that task. Both types of consensus ratings should include or be augmented

with detailed explanations of all identified personnel deficiencies bearing on the

ability to establish objectives.

It is relatively easy to suggest measures for any given

attribute. Unfortunately, it is quite something else to devise a general

procedure for doing so in all cases. Within the scope of this current study,

measures have been suggested for all of the issues and attributes of the four

taxonomic subsets of Figure 22. These measures are arrayed in Figures 23

through 2f). Each suggested measure has something of value to say about the

"goodness" or "badness" of the attribute to which it relates. The measures

suggested for each given attribute collectively try to say everything that is

pertinent to the "goodness" or "badness" of that attribute. Nevertheless, it is

quite possible (and probably very likely) that the measures sets shown in

Figures 23 through 26 will undergo significant revision after a critical review

by subject-matter experts.

Concluding Remarks Concerning the Two Trial Applications of

APM to Measurement of vSystem Performance/Effectiveness

the

Clearly, much remains unproven at the conclusion of these trial

applications. The final products are merely partial sets of measures for

assessing very narrow aspects of the work required of the BIFV and its

Gunnery Training Design Subsystem. Most glaringly, those measures are

untried. No one has actually attempted to apply them, or even to verify that

the information needed to generate them can be obtained. Those measures,

moreover, derive from attributes that also are unproven. No attempt has even

been made to demonstrate convincingly that the attributes stated really

constitute all of the factors pertinent to each evaluative issue, and only those

factors.

What, then, has been demonstrated? If nothing else, sets of

measures have been produced that definitely are issues-oriented. It has been

possible to proceed from a framework of basic performance requirements to a

statement of exactly what requirements apply to a given measurement

application. It has been shown that such requirements can be expressed in

-65-
Hierarchy Number 1; BIFV gunnery desien »ubgygtem potential for

establishing performance objectives

1.2

Attributes

Scope of information

available concerning

BIFV gunners' tasks

Staff experience and

qualifications in

establishing performance objectives

Planned approach to

establishing BIFV

gunnery performance

objectives

Resources allocated

to establishing BIFV

gunnery performance

objectives

Issues

Ability to insure that all BIFV gunnery

performance objectives will be identified so that they may be addressed in

gunner testing specifications

Measures

A. BIFV gunners' required tasks that are known (listing and description

of each such task)

B. BIFV gunners' required tasks that are unknown to the Design Subsystem (listing of each)

C. Tasks that actually are not required of BIFV gunners, but which mistakenly are classified as required by the Design Subsystem (listing

and description)

D. Reviewers' cumulative ratings of each Design staff member's training,

experience and previous performance in establishing performance

objectives for training (using a specified point allocation system

for training, experience, etc.)

E. Reviewers' overall assessment of total Design staff's qualifications

(based on specified method of accumulating individual members'

ratings)

F. Essential steps missing from the plans (listing and description of

each such step)

C. Inessential steps included in the plans (listing and description of

•ach)

H. Inconsistencies/deficiencies in the sequencing of steps in the plans

I. Reviewers' cumulative ratings of the soundness/workability of the

plans (using a specified point allocation system)

J. Percentage of required person-hours proposed for allocation to

establishing objectives

K. Percentage of required materials/goods proposed for allocation to

establishing objectives

L. Percentage of required support services proposed for allocation to

establishing objectives

M. Ratio of dollar value of total resources proposed for allocation to

establishing objectives to dollar value of total resources

actually required

1.2.1

Elements or factors

of gunner abilities

intended to be

stated B.

C.

Ability to insure that all things BIFV

gunners will be able to do after training will be stated so that tests of all

required abilities may be specified

Essential elements of abilities intended to be Included in the statements of BIFV gunner abilities (listing and description of each such

type of element)

Inessential elements intended to be included In the statements of

BIFV gunner abilities (listing and description of each type)

Essential elements of abilities not intended to be included in the

statements of BIFV gunner abilities (listing and description of

each type

Figure 23. Sample Measures Hierarchy Number 1

-66-
Hierarchy Number 2; B1FV gunnery design «ubsygtem process of

«nalyzing talks »elected for training

2.3

Attributes

Task analytic

methods applied to

the BIFV gunner

tasks

The scope of the

BIFV gunner task

analyses conducted

Completeness of the

BIFV gunner task

analyse« conducted

Accuracy of the

BIFV gunner task

analyses conducted

Issues

Conduct of analysis of BIFV gunner

tasks, so that appropriate tests of

those tasks may be specified

Measures

A. Deviations between the methods actually applied and the methods specified in the plans for analyses (listing and description of each

such deviation)

B. Aspects or elements of the methods applied that are irrelevant to or

inconsistent with the analytic purposes (listing and description

of each)

C. Aspects or elements missing from the methods applied that are essential for the analytic purposes (listing and description of each)

D. Types of data needed for the methods that were not obtained (listing

and description of each data deficiency)

E Factors supposed to be determined from the task analyses that

actually were not determined

F. Factors irrelevant to the analytic purposes that were determined

In the task analyses

G. Relevant BIFV gunner tasks to which the analytic methods were not

applied

H. Factors missing from the analyses applied to other relevant BIFV

gunner tasks

I. Inaccuracies in the input data obtained for application of the

analaytic methods

J. Specific misinterpretations of the analysed data

K. Specific aspects or elements of the analytic methods that were

misapplied

2.3.2

Methods employed to

Identify the domains of learning

of the BIFV gunner

tasks

The domains of

learning identified

for each BIFV gunner task

B.

C.

D.

Classification of BIFV gunner tasks

in terms of the domains of learning

they involve

Aspecis or elements of the methods employed that are irrelevant to

or inconsistent with identification of the domains of learning

involved in a task (listing and description)

Aspects or elements missing from the methods employed that are

essential for identifying the domains of learning (listing and

description)

Input data required for the methods that are missing or insufficient

(listing and description)

Domains Identified as involved in a task that actually are not relevant to that task (listing and description of each such incident)

E. Domains not identified as involved in a task that actually are involved in that task (listing and description of each such incident)

Figure 24. Sample Measures Hierarchy Number 2

-67-
Hierarchy Number 3: BIFV

contisting of «Ute

Sunrtery de«ign »ub«y

performance objectiv

»tern product

ves

3.2

Attributes

The BIFV gunner

performance

objective» that

are stated

The format of the

•tatcments of

BIFV gunner

performance

objectives

The completeness

of the statements

of BIFV gunner

performance

objectives

The accuracy of

the statements of

BIFV gunner

performance

objectives

Issue»

Production of accurate and complete

statements of BIFV gunner performance

objectives, so that tests of gunners'

qualifier dons may be specified

Measures

A. Statements that define performance objectives that actually are not

needed for BIFV gunnery (listing and description of each such

statement)

B. Performance objectives that actually are needed for BIFV gunnery

that are not addressed in any of the statements produced (listing

«nd description of each such objective)

C. Essential elements of performance objective statements not included

in the format employed for stating BIFV performance objectives

(listirg and description of each missing element)

D. Inessential/inappropriate elements in the format employed for stating BIFV performance objectives (listing and description of each

such element)

E. Elements missing from the statements of particular BIFV performance objectives (tabulation of missing elements as a function of

the stated objectives)

F. BIFV performance objective statements that contain erroneous or

Inaccurate elements (tabulation of inaccurate elements as a

function of the stated objectives)

G. Description of the nature of the inaccuracies, for each stated

objective

3.2.1

The BIFV gunnerlearner achievements that are

defined

The factors included in the

definitions of

BIFV gunnerlearner achievements

Production of exact definitions of

the BIFV gunner-learners' expected

achievements as a result of

the training

A. Achievements defined that actually are not necessarily expected of

BIFV gunner-learners during training (listing and description of

•ach such achievement)

B. Achievements not defined that actually are expected of BIFV

gunner-learners during training (listing and description of each

such achievement)

C. Inexact or ambiguous elements in the definitions of BIFV gunnerlearner achievements (listing and description)

D. Inaccurate elements in the definitions of BIFV gunner-learner

achievements (listing and description)

E. Factors included in the definitions that actually arc irrelevant

to the description of intended achievements of BIFV gunnerlearners (listing and description)

F. Factors missing from the definitions that actually arc essential

for describing intended achievements of BIFV gunner-learners

(listing and description)

Figure 25. Sample Measures Hierarchy Number 3

-68-
Hierarchy Number 4: BIFV gunnery design subiyttew product

_ . con«ignng of test»

3.5

^_ ___ Issues

Production of «ppropriate tests for

Attributes

The BIFV gunnerlearner tests that

are prepared

BIFV gunnery

testing application/ requirements

for which tests

are prepared

BIFV gunner

behaviors for

which tests are

prepared

Relevance of the

tests

administration to BIFV gunner-learners

Measures

A. Prepared tests that address each BIFV gunnery perfornance objec

tive (listing and description as a function of each performance

objective)

B. Prepared tests that relate to no BIFV gunnery performance objective (listing and description of each such test)

C. Reviewers' ratings of the utility of each prepared test for each

testing application/requirement (yes/no rating of each test for

each application)

Listing of tests prepared for each behavior associated with «ach

performance objective

Deviations between the behaviors elicited by the tests and the

behaviors that actually are to be addressed (listing and description of each such deviation)

3.5.?

Candidate gunnerlearner qualifications for which

tests are prepared

The tests prepared for assessment of the

qualifications

Production of appropriate tests for

assessing the qualifications of

candidate BIFV gunner-learners

A. Relevant qualifications for which tests arc prepared (listing and

description of each such qualification)

B. Relevant qualifications for which no tests are prepared (listing

and description of each such qualification)

C. Irrelevant "qualifications" for which tests are prepared (listing

and description of each)

D. Elements of each test that deviate from the qualifications actually required of acceptable gunner-learners (listing and description

of each such deviating element)

£. Probability that each test will be passed by candidates who

actually do not possess the qualification being tested

F. Probabf ity that each test will be failed by candidates who

actually do possess the qualification being tested

G. Cumulative probability that an actually unqualified candidate will

pass sufficient qualification teats to be accepted as a gunnerlearner

H. Cumulative probability that an actually qualified candidate will

fail sufficient qualification tests to be rejected as a gunnerlearner

Figure 26. Sample Measures Hierarchy Number 4

-69-
terms that permit concrete, tractable attributes to be defined. None of the

attributes stems from any particular approach to system design. All of the

attributes derive from the performance requirements that are common to all

such designs. This is a critical point: it demonstrates that the APM at least

is rooted firmly in valid measurement issues. Too often, evaluations of new

systems are clouded by irrelevant and unfair comparisons with older design

concepts. The basic question of whether the new system does what Tt is

supposed to do sometimes is overlooked because the evaluators actually measure

how the new system "looks" in comparison to older models. Conclusions based

on such evaluations typically may suggest: "The system is no good because

we've never done it this way before." That design-oriented philosophy of

measurement suppresses technological breakthroughs. In the training sphere,

for example, it prevents modern and effective methods of work site,

learner-tailored instruction from playing their appropriate roles in the total

learning environment, and it masks the relative ineffectiveness of the

traditional, institutional methods of training.

It is a step forward to have shown that a measurement scheme can

be kept free of artificial judgments of design. Use of the APM has forced

rather complete taxonomies of performance requirements to be considered in

measurement applications, more complete than a typical, less structured

analysis would provide. It has also demonstrated the existence of generalized

and also useful tax, an essential ingredient in constructing a "systems

application science." Though much remains to be done to insure that the best

measures always are identified, it has been shown that measures can be

identified for these performance-oriented attributes.

-70-
HI. APM APPLICATION PROCEDURES

A. General

The two intended applications of the APM are to specify the design of

human-machine systems and to define measures of performance and effectiveness for those systems. As described previously, system design and

system measurement deal with exactly the same issues. Those issues are the

constituent requirements of system performance. The designer and the

evaluator are (or should be) interested strictly in: the capabilities (potential

performance) their system is supposed to have; the activities (process performance) it is supposed to carry out; and the goods and services (product

performance) it is supposed to deliver. The designer's job is to see to it that

the system does in fact meet all of its requirements for potential, process, and

product performance within the environmental and design constraints imposed.

The evaluator's job is to check on how well the designer's job is done. Their

jobs must be performed as an interactive process.

To apply the APM, the designer and evaluator should follow the steps

depicted in Figure 27. In Step 1, the analyst is required to define the

purpose of this analysis, in terms that are as specific and confining as

possible. This information will be used later in helping to assess which of the

many system performance items (which may be presented to the analyst from a

computer data base) are relevant to the present analysis. Specificity and

narrowness of limits in defining the analytic purpose can help keep the

ultimate number of design specifications or evaluation measures to a minimum.

Step 2 of the procedure requires the analyst to identify the overall class

of systems to which the system of interest belongs. The purpose of this

designation is to permit the computer to aid the user by presenting for review

any known performance items, attributes, design specifications and measures

that are stored in its data base from previous analytic efforts.

Step 3 allows the user to identify the specific system of interest so that,

for computer-aided analyses, the proper identification automatically appears on

all displays and printouts. This serves as both a record and a reminder of

the application at hand.

In Step 4, the analyst is required to consider all the other "systems"

which interact with the one of central interest. Interactions with

suprasystems, collateral systems and subsystems call the analyst's attention to

the fact that the system of interest must provide outputs to, and use inputs

from, external entities. The requirements of those external entities therebv

have an influence on performance requirements being developed for the system

of interest. This step is not presently implemented on the computer-aided

APM demonstration package.

Step 5 requires the user to construct system performance taxonomies in

accordance with the 15-cell (5 Aspects x 3 Levels) matrix referred to

-71-
©

Step

Define the Analytic Purpose

©

Identify System

Class of Interest

©

Identify System

of Interest

©

Identify Hierarchical

Interacting Systems

(Supra-, Collateral, Sub-)

©

Define Performance

Taxonomies at all Levels

(Potentialities, Processes,

Products, Environment and

Constraints at Objectives,

Functional Purposes, and

Characteristics Levels)

©

• To evaluate BIFV training design

subsystem effectiveness in specifying

learner testing.

• Training

Bradley Infantry Fighting Vehicle

(BIFV) Training System

• Suprasystems: U.S. Army Infantry

School

• Collateral Systems: Other Infantry

Training Systems, the BIFV itself,

U.S. Army Infantry Units

. Subsystems: Command, Design, Enabling, Logistics, Emplacement, Delivery

• The BIFVTS must be capable of

determining the existing levels of

learner achievement.

Define Measurable

System Attributes

Design

©

To insure compliance with that specified performance characteristic, one

attribute to be examined is the

BIFVTS achievement testing plan.

Evaluation

Define System

Design

Specifications

®

Define System

Measurement

Specificationa

Apply Results to Systems of Interest,

taking Corrective Action as Required

Figure 27. General APM Application Procedures

-72-
previously as the Systems Taxonomy Model (STM) portion of the APM.

Because of the voluminous, equivocal and vital nature of this step, it is of

special importance that the analyst be given systematic directions and a large

data base of existing, representative taxa to aid in its satisfactory completion.

In recognition of that need, the computer-aided APM demonstration package

and this report (appendices) contain comprehensive performance taxa for

several of the six defined training subsystems (for design, enabling and

delivery, but not for command, logistics and emplacement). The number of

performance items that arc retained as relevant to a given analysis will depend

upon the purpose of that analysis (see Step 1).

The task in Step 6 is to identify those system attributes that are both

reflective of each performance requirement and susceptible to specification or

measurement of some kind. Here, too, the user can be aided by having the

opportunity to review existing, representative attributes that a have been

derived from the system performance measures. The computer-aided APM

demonstration package and this report (appendices) contain a sample of such

representative attribute lists for training systems.

Step 7 is the point in the APM where the analyst decides what will be

done with the relevant performance attributes that have been identified. It is

here that the model divides into one procedure for developing der .gn specifications and another procedure for developing measurement specifications.

In Step 8, the user acts on the choice to identify design specifications

for the relevant performance attributes. Ideally, the analyst will state

objective, operational criteria to be achieved by the designers in providing for

the required system attributes.

In Step 9, the user acts on the choice to identify evaluation measures for

the relevant performance attributes. The measurement specifications can be

developed with the aid of the model and the existing sample of representative

measures to be found in both the computer-aided package and this report

(appendices).

Step 10 calls for the user to apply the results of this analysis to the

system of interest, either in aiding its evolving design or measuring its

operational performance. Comparisons of design specifications with

performance measures can also result in decisions to modify the system, its

performance requirements, its design specifications, its effectiveness measures,

or all of these. This must result from the interactive design/evaluation

processes referred to earlier.

Clearly, the cost-effective application of this model to typical military

systems requires some kind of routinized procedure, such as the one illustrated by the APM demonstration package developed as a part of the present

effort. The application of that computer-aided APM is described next.

B. The Computer-Aided APM

The ultimate computer-aided APM will help the analyst to design or

evaluate the effectiveness of any (conceptual or real) system. The analyst will

-73-
take one of several approaches to working with the system of interest. If the

system is totally different from any existing system, he will start out by

defining new performance items (objectives, functional purposes, and

characteristics). Then, the analyst will specify the attributes and measures

(or the design specifications) for each of the performance items. Once this is

accomplished, he might specify reasons for performing an analysis and

associate each of these purposes with a set of characteristics. Printing the

data set in a variety of formats as well as editing the data set for corrections

and/or revisions will, of course, he possible.

If the system of interest is very similar to (but different from) an

existing system, the analyst will be able to review, on a line-by-line basis,

each of the performance items, design specifications, attributes, and measures

for the existing similar system and decide whether to include these taxa in the

new system unedited, edit the items for inclusion in the new system, or

exclude them from the new system. Then, the analyst will be able to associate

each of the performance items with appropriate measurement purposes, and

print the dpta set in a variety of different formats.

Probably the most frequent use of the APM will be to analyze and/or

review the design specifications for an existing system (any system which has

prcviouslv been entered into the computer is an existing system). In this

case, one might start by examining a specific measurement purpose for the

existing system, including, excluding, editing, or otherwise reviewing

individual taxa associated with the measurement purpose. Alternatively, the

analyst might decide to review all of the performance items design

specifications, attributes, or measures; or the analyst might decide to review a

specific performance item, a specific attribute, and/or a specific measure.

After completing this editing procedure, the analyst will have the opportunity

to print out the results of his analysis.

Another application for the APM would be for an analyst to decide to

analyze part of a system, but not the entire system. The analyst might

decide, for example, to perform only 30 or 40 percent of the recommended

measurements. In this case he would have the option to tell the computer

which measures he would or would not perform. The computer would then tell

hin which performance items were going to escape untested, so that he could

review, once again, his testing procedures.

This ultimate system would be implemented on a main-frame computer with

adequate on-line disk storage for several systems. Analysts would have the

opportunity to draw upon master files of taxa so that they could extract what

they would need to construct their systems. Furthermore, they would have

the ability to compare their system with a host of other systems to discover

various per .irmance items which they may have inadvertantly left out of their

data sets. The ultimate system would be user-friendly, offering help as

needed. It would come with an on-line tutorial progrnm, and taxa for many

systems. Furthermore, the system would be compatible with both touchsensitive screens (to insure maximal user-friendliness) and keyboard entry

devices (to avoid excluding users without touch-sensitive screens).

A demonstratioi model which illustrates many of the important features of

the ultimate system for measurement (but not for design) was programmed in

-74-
PASCAL on an APPLE II PLUS Computer System with an 80-column VIDRX

board. While companion volumes to the present report (Shapiro et al., 1983;

and Shapiro, 1983) provide tutorial and operating instructions for the

demonstration system as well as listings of the actual PASCAL code, Figure 28

presents an overall schematic flow chart of the demonstration system.

When the computer is first turned on, a title page appears on the display

screen. As soon as an analyst presses a button on the computer's keyboard,

he/she can choose to read instructions which explain the operation of the

demonstration system in detail. The instructions are user-friendly in that

they are written with minimal computer jargon and the computer refers to itself

as "I" and to the analyst as "you." The analyst may branch through the

instructions reading only those which are of interest.

Upon completing the instructions, the analyst selects which class of

systems (e.g., Training, Communications) to work with (only Training is

implemented). Then he/she selects a particular system (e.g., Bradley

Infantry Fighting Vehicle Training System). Finally, he/she selects a

subsystem (e.g., emplacement, delivery, or design). After the system class,

system, and subsystem are selected, a menu of all available analytic procedures is displayed for analyst selection. The analyst might, for example,

choose to Add, Modify, or Delete Measurement Purposes. Thus, he/she would

define a particular meas'.rement purpose (or use an existing measurement

purpose). Then, the analyst would have the opportunity to associate relevant

characteristics with the measurement purpose, or dissociate irrelevant

characteristics.

Alternatively, the analyst might select to add, modify, or delete

performance items (objectives, functional purposes, and characteristics) for the

5x3 matrix of the system. The analyst could also add, modify, or delete

attributes and measures for any performance item. Whenever the analyst is

performing a procedure, he/she has the option of making a paper copy of the

taxa which he is analyzing at a given time. Furthermore, the analyst has the

opportunity to print out all of the taxa for the subsystem of interest, or all of

the taxa for a given measurement purpose. Samples of this printout for the

design, enabling and delivery subsystems appear in Appendices A, B and C,

respectively. The printouts present the subsystem (or the taxa which are

relevant to a given measurement purpose for the subsystem) in n hierarchy;

the aspects of the system (Potentialities, Processes, Products, Environmental

Constraints, Constraints); the relevant objectives; and the functional purposes, characteristics, attributes, and measures for each objective. Each

taxon bears a unique number which follows the general hierarchical plan.

Other analytic procedures available include packing disk files more efficiently

(a basic housekeeping function), reviewing instructions, stopping, or specifying the system class, system, or subsystem to be used for the remainder of

the session.

The demonstration model Is user-friendly. In addition to easy-to-read

Instructions, help messages are available In the Initial program where one

specifies which system he/she wishes to use, and which analytic procedure

he/she wishes to perform. However, they are not available in other programs.

The demonstration system Is also very forgiving of typing errors. Whenever It

-75-
-76-
catches an illegal response, it tells the analyst what the legal responses arc.

The system Hoes, however, have certain very strict limitations. Most of these

limitations are caused by the relatively slow processing speed of the Apple II

Plus computer, and the relatively limited core and disk storage available on the

Apple II Plus, as well as the attempt to avoid excessive use of machinedependent code. A comparison of the specific limit tions of the de.nonstration

system and a list of features to be available in tie ultimate system are

presented in Figure 29.

-77-
Ultimate Package

(preliminary

DeicrlDtion estimate) Demonstration Package

vroRAOB cAPAcmr

Number of SYSTEM CLASSES, SYSTEMS, A SUBSYSTEMS that can be ONLINE at the aame time

Number of OBJECTIVES, FUNCTIONAL PURPOSES,

•nd CHARACTERISTICS that can be specified for a

given subsystrm

Number of OBJECTIVES that can be specified for a

given ASPECT

Number of FUNCTIONAL PURPOSES »hat can be

apecifked for a given OBJECTIVE

Number of CHARACTERISTICS that can be specified

for • given FUNCTIONAL PURPOSE

Number of ATTRIBUTES that ean be specified for a

given subsystem

Number of ATTRIBUTES that ean be specified for a

given PERFORMANCE ITEM (OBJECTIVE, FUNCTIONAL

PURPOSE, or CHARACTERISTIC)

Number of MEASURES that can be specified for a

given subsystem

Number of MEASURES that can be specified for a

given ATTRIBUTE

Number of MEASUREMENT PURPOSES that can be

specified for a given subsystem

Number of PERFORMANCE ITEMS that can be associates with a given MEASUREMENT PURPOSE

Number of characters allowed for description of a

PERFORMANCE ITEM. MEASUREMENT PURPOSE,

ATTRIBUTE, and MEASURE

Number of characters available for explanation of a

given taxon

PROCESSING SPEED

Execution time for disk searche" and sorts

Printer speed for printing an entire data set

HELP MESSAGES

Number of messages available

Ability to ^>ecify new help messages while using

program

Length of each message

AvciUbUity

50 each

•00

Fast

Fast

Online

1,000 characters

Anywhere In

programs

1 each

5,000 total .«0Ü total

99 20

99 20

99 20

2,000 200

99 20

10,000 400

99 20

99 5

5,000 HP

400 each •0/Performance Item

61/Attribute A Measure

136/Measurement Purpose

Slow

Slow

60

Offline

100 characters

In 2 of the 6 programs

(continued)

Figure 29. Comparison between the Ultimate Computer Package

and the Demonstration Package

-78-
Ultimate Package

(preliminary

Description estimate) Demonstration Package

ACCOUNTING FUNCTIONS

Keep track of usage for each system class, system,

subsystem, and help message Online

DBR BACKUPS

Daily backup of each system modified during the day

Backups available upon request

FILE MODIFICATION

Ability to modify a disk file for personal use

without destroying the master file

Restricting modification of master files to

privileged accounts

MEASUREMENT PURPOSE

Ability to edit performance items, attributes

and measures for a giver MEASUREMENT PURPOSE Yes

OVERHEAD

Number of simultaneous users (either independent or

interacting following a teacher's demonstration) 15

Power failure protection Yes

DATA SETS SUPPLIED WITH PACKAGE Many

ONLINE MESSAGE FACILITY

Allows analyst to send Computer Package Staff

messages about system at any time Yes

CODE EFFICIENCY

Use search routine which is optimal for task

at hand

DISPLAY UPDATES AFTER SMALL MODIFICATION

TO TEXT

TUTORIAL PROGRAM

SPECIAL MESSAGES

Yes

Single-line updates

Online

After the print routine, displays the measures which

ought to be used, and the computer asks, "Which

measures do you really plan to use?" and shows

which performance items were not analyzed as a

result of the omissions. Yes

in the attributes and measures specification program,

computer reminds analyst (in generic terms) what

ought to be measured for each performance item.

Thus, potential measures are not neglected. Yes

Logbook

Automatic Manual pr. cess

Yes Yes

Automatic Manual recopy necessary

before making changes

Automatic Keep master files in

locked file cabinet

No

1

No

3 subsystems

No

No

Must reconfigure entire

display

In booklet form

No

No

Figure 29. (Concluded)

-79-
IV. RELATIONSHIP OF THE APM 10 OTHER MODELS*

A. Bases of Comparison

There -".re, of course, many "models" available for the development and

evaluation of human-machine and training systems. Indeed, there are so miny

such models that it is difficult to list them let alone select an appropriate

model for an application. These models tend (1) to be based on a variety of

mathematical and descriptive bases, (2) to be specific system- and/or

aggregate systems-oriented, (3) to concentrate on macro- or micro-levels or

somewhere in between, and (4) to focus on parts of the total human resources

in system problems. It is, therefore, not easy to place the APM within the

general framework of human-machine and training systems models since that

framework is, at present, not clearly defined and structured.

One way of looking at the APM is within some of the major acquisition and

development steps for all military systems. In this chapter, six such steps

are noted as follows:

0 H . ian-machine system development procedures

0 Human-machine system development

0

Training system development

0

Behavioral simul^^'on models

0

Logistic support analyses

0

Test and evaluation methods

Thus, th3 emphasis here is on the possible use of the APM, as compared with

other models, in some of the majoi" events in system development." The

question, then, is not so much the internal structure of the APM insofar as

completeness, coherence, and validity but, rather, how relatively well it can

be used in a comparison with other models and methods in the technical

inventory.

With this approach, certain utility features of any model become

paramount. Questions arise, such as: (1) How well does the model fit the

system (or systems) under development? (2) What kinds of data are needed

for the model to work well? (3) How much time and project resources are

needed to U&L the model? (4) Does the model produce requirements and data

useful in system development? and (5) What kinds of analytic skills are

needed to apply the model (i.e. , can it be used by normal project people or

does it require very highly talented people who may or may not be available to

the program)? These are all very practical problems which seem to determine

to a great extent whether or not models are used in the system development

project.

'This chapter was prepared by Dr. Frederick A. Murkier of Canyon Research

Group, Inc. (now Essex Corporation), Westlake Village, California.

-80-
B. System Development Procedures

The major single document that defines the procedures for the design and

development of human-machine systems is MIL-H-46855B (Department of

Defense, 1979) "Human Engineering Requirements for Military Systems,

Equipment and Facilities." This general specification, and its subsequent

revisions, hat- been in effect since 1968, and is presumably mandatory for all

military system development projects. Basically, it provides a top-level

procedure for analysis, design and development, and test and evaluation of all

human-machine military systems and subsystems. In practice, many variations

have been followed to implement the specification. Indeed, it was designed to

allow for such variations consistent with differing weapon systems across the

military services. The end goal of the specification is to attempt to assure

that the humnn element of military systems is used properly and effectively.

The Ai'M is totally consistent with the general objectives of

MIL-H-46855B. The APM may, in fact, be considered an implementing and

detailed model of that specification. The APM places primary emphasis, in its

present form, on system, subsystem, and mission analysis (to use the

terminology of MII.-H-46855B) and stresses "... the functions that must be

performed by the system in achieving its mission objectives (which) shall be

identified and described." (MIL-H-46855B: 3.1.1.a) Two major strengths

may be noted in the APM process:

An initial emphasis on system goals and objectives which forms the

framework for all subsequent analysis, design, and test.

A major requirement to evaluate all system and subsystem levels of

potentialities, processes, and products.

Within each system level, specific answers must Ve given to the questions:

What? (Objectives) Why? (Functional Purposes and How? (Characteristics)

This is precisely the intent of the analytic steps in MIL-H-46855B; the APM is

the most highly structured tool for accomplishing this objective currently

available. Further, in its computerized form, it assists the analyst in rapid

use of the model. Finally, the computerized APM is "user friendly," and takes

into account a widely varying skill level among potential users.

With respect to model structure, the APM is consistent with standard

system engineering methods which have been in use for the past ?.0 years

(cf., Machol, 1965; Byrne, Mullally, and Tohery, 1969). Within that

framework, the APM may be classified as:

0 A descriptive model with, however, specific quantitative outputs in

the measurement subsystem; and

0 A static, multi-level model (cf. Niehaus, 1979) which can give rapid

snapshots of the development status of the system insofar as human

resources are concerned, as well as system performance implications.

-81-
On the latter point, a choice is available to system analysts between static

and dynamic models. Dynamic models assume a time-varying, scenario-based

simulation. At the present time, the APM is not constructed for this use

although future development could make it so. Further, no explicit use has

been made of a great number of optimization techniques (cf., Rouse, 1980)

that might be used for human-machine systems problems.

While all systems models are concerned with measurement and measure?,

the APM appears to be unique in its stress on the methods for deriving

specific and practical measures. Often, measurement is left to the skill of the

analyst with the method undefined; APM requires a direct and explicit

development of system and subsystem measurement. As will be noted, the data

estimates so derived will be useful for many purposes.

C. Human-Machine System Development

The major purpose of MIL-H-46855B was to give a general, sequential

method by which human-machine systems could be developed with careful

emphasis--in the total systems context—on the use of the human element. As

noted, the APM is a specific method for doing exactly that. While there arc a

number of particular techniques for human engineering development (cf. ,

Meister, 1976), the APM appears to be the most comprehensive, thorough, and

detailed methoJ to date. It fulfills the data requirements for MIL-H-46855B,

but goes beyond those requirements to specify training system requirements

and provides basic data for logistic system support analyses.

Much practical experience tends to confirm the fact that systematic

attention to human resources within the system development process varies

enormously. The range is from very detailed and competent contributions to

ignoring the human element altogether. The reasons for this are complex (cf.,

Carr, et al. , 1980), but one problem has been the lack of methods that

significantly answer all of the manpower, personnel and training questions that

arise from a new system.

One method (Mackie, 1979) has organized the system development process

into nine basic questions concerning the manpower, training, and personnel

elements of the system:

1. How do various design concepts affect manpower and training

requirements?

2. How do manpower and other evaluative criteria interact?

3. What numbers and types of personnel are associated with this

system?

4. What is their projected availability?

5. Which design alternative best satisfies all criteria?

6. What operator and maintenance tasks are implied by this design?

7. What are the operational and maintenance training requirements?

-82-
8. What skill levels and numbers of operators are required?

9. What are the estimated personnel life-cycle costs to man the svsten?

The procedure also suggests the kinds of data that would be needed to answer

these questions.

One way of evaluating the APM—or any other model framework for system

development application—is to estimate the degree to which APM can answer

these questions. Following the order of the questions as just given;

1. Alternative design concepts can be evaluated within successive runs

of the APM with its major strength in its present form in personnel

and training estimates.

2. Criteria interaction assessment should be particularly strong since

the APM model requires careful delineation of goals and objectives

with specific measurement dimensions.

3. There is no clear path, at present, in the APM to estimate numbers

and types of personnel.

4. At present, APM does not take into account projected manpower

availability, although it certainly could be expanded to do so.

5. Trade-off studies based on explicit criteria from APM should be

particularly effective (cf., Askren, 1975).

6. Task analysis data are an explicit output of the APM.

7. Training requirements analysis is an explicit output and strength of

the APM.

8. Identification of skill levels is a specific output of the APM.

9. Life-cycle costing is not, at present, an identified process or output

for APM.

Valuable outputs from the APM which are not included in these questions (and

which should be) include logistic support analysis and specific test and

evaluation criteria.

No current model, method, or procedure satisfies all of the development

tasks and data that are needed for system design and development. At

present, the APM is strongest in system, task, training and measurement

definition but appears to be weak in responding to manpower and cost

questions associated with new systems.

D. Training System Development

The past 30 years have seen the most remarkable advances in training

technology in the entire history of human learning. Part of this is due to

-83-
applications of computer technology, part to empirical facts about optimal

human learning, and much to systematic methods of analyzing and developing

training systems. In a broad sense, the last can be termed the "Systems

Approach to Training" or "Instructional System Development" (O'Neil, l^?^;

Vineberg and Joyner, 1980). Instructional system development (ISD)

methodologies are in use throughout all military services, and are widely used

in non-military education and training. Unfortunately, several current

problems are apparent with the ISD technique:

0

There is a variety of methods for generating training systems called

by the loose generic label of ISD;

n

There is no guarantee that any of the ISD methods obtain maxinun

system performance efficiency or training efficiency; and

0

ISD applications frequently come too late in the system development

cycle where potential major training problems could be identified

before design freeze and corrected through job and task redesign.

To date, the major applications of the APM have been on training systems

development. This is, in effect, a test of how well APM can work to generate

training requirements and training system data bases (see Appendices A, B,

and C). APM is particularly strong in:

o

Performance goals and objectives

Need/discrepancy analysis

Identification of job requirements

Selection of tasks for training

Analysis of tasks

Construction of job performance measures

Development of training objectives and objective hierarchies

0 Validation of instruction

0

Logistic requirements for training systems

It appears to be less useful in such steps as:

0 Methods for evaluating optimum training efficiency

0

Formal cost-effectiveness evaluations of training systems

Perhaps another way of stating it is that APM is currently emphasizing the

"What?" (Objectives) and "Why?" (Functional Purposes) of new training

systems rather than the "How?" (Characteristics). For the latter, additional

development is needed. There is, however, nothing inherent in the APM

concept that would preclude its addressing the "how" of new training systems.

Indeed, the APM is so structured that the "how" would be a natural output.

It would appear that the major strength of the APM at present is with

respect to front-end analysis of training systems. As noted above, training

system analysis has usually come very late in the system development cycle,

and there is much concern that these analyses be conducted as a part of the

system development process (cf., Seidel and Wagner, 1970). APM seems

particularly potentially useful in the initial steps of mission, function, task,

and job analysis.

-84-
Another similar system under development at this time is the Early

Training Estimation System (ETES) which will attempt "... to deal with two

major deficiencies in existing technologies: (1) the lack of a systematic tool

for describing, storing, and updating system concepts and for transmitting

this information to all of the various participants in the development/acquisition

process, and (2) the lack of a comprehensive set of training analysis tools

which are appropriate for the early phases of design." (O'Brien, 1982).

ETES has three major functional components: System Description Technology,

Training Estimation Aids and Procedures, and Evaluative Technology. It

appears to be particularly strong in comparative evaluation of instructional

methods and media and cost-effectiveness evaluation. These are areas in

which APM har, not moved into heavily. Note should be made of the fact that

ETES is under development for computer application with a microprocessor.

The APM and the ETES appear to be complementary. ETES can be of

particular value to APM in comparative evaluations of media and instructional

methods. APM appears to have a sounder approach to the generation of

system and subsystem objectives and development of performance measurement.

Both systems appear to generate the kinds of support data for early training

projections as described by Jorgensen (1981).

E. Behavioral Simulation Models

For almost 30 years there has been a continuing attempt to develop

quantitative simulation models for behavioral, social, and systems processes

(cf. , Siegel and Wolf, 1969; Pew, Baron, Feehrer, and Miller, 1977; Rouse,

1980). Initial theoretical attempts concentrated on human tracking

performance, and have now expanded to encompass decision making and

information processing at both direct control and monitoring levels. The

question might be asked: What relevance do these models have to APM?

The primary emphasis of these models has been a better understanding

and description of human behavior often (but not always) in a system context.

For the systems context and tasks for which they are appropriate, these

models have sometimes provided very specific quantitative predictions of hum \n

behavior. This has normally concerned specific operator and maintenance

tasks with little concern for manpower and/or training implications. They are

normally time-varying, stochastic models with much concern about task

performance and often human performance optimization.

As Siegel, Wolf, and Ozkaptan (1981) have pointed out, early in the

system development cycle it may be wise to use qualitative and deterministic

methods before switching to complex, computer-based, behavioral simulations.

If that is true, APM would in general be the method of choice early in design.

APM could then subsequently provide performance parameters, tasks, and

measures for time-varying, scenario-driven, stochastic simulation models. In

short, APM would describe the objectives, structure, and measures of the

system for computer simulation of the system.

-85-
F. Logistic Support Analyses

Perhaps even to a greater extent than training, logistic support analyses

have traditionally been executed well into the system development cycle and

usually after design freeze. It is now apparent that this can no longer be the

case; early estimates of logistic requirements are essential since they may

determine whether or not a system is operationally feasible. The goal, as

Weddle (198]) puts it is "... to build supportability into a system rather than

inefficiently bolting it on after the system is deployed."

There are a very large number of logistic support analyses models, two of

which may exemplify their structure and purpose:

1. Kyle and Markisohn (1974) have described an Integrated Facilities

Requirements System for pilot training. It contained five principal

modules:

Logistic support requirements for calculating total manpower,

aircraft and fuel requirements for training;

Base loading for assignment of students and resources for each

phase of training;

Facilities requirements for estimating specific permanent

facilities for the total pilot training program at each base;

Efficiency/deficiency estimates to evaluate the training input

load with facilities; and

Total systems cost which attempts to predict the total cost of all

resources required, including investment costs and operations

and maintenance costs.

With the exception of cost data, the ARM is designed to supply these types of

logistics data (see specifically Appendices B and C).

2. While all services have developed extensive logistic support analyses

models, the Air Force has particularly pioneered the inclusion of the

human resourcrs element in logistics (cf., King and Askren, 1980).

They have developed and tested a coordinated human resources

technology (CURT) method for logistics analyses combining:

maintenance manpower modeling, job guide development, instructional

system development, system ownership costing, and human resources

in design trade-offs. Most important is an associated requirement

for a unified data base to support logistic analyses in these areas.

/^PM stresses, to a greater degree than CHRT and its derivative

Navy system, Hardman, system and task performance objectives and

performance measurement.

G. Test and Evaluation

Very early in the introduction of human resources work in system

development stress was placed on adequate test and evaluation, particularly of

-86-
the human elrment in systems (of., Meister and Rabideau, 1965). APM

continues this tradition with a strong and explicit module for test and

evaJuation. Although there has been a considerable literature and practice

developed, little standardization has been achieved for human resources test

and evaluation either in developmental or operational testing. Major techniques

have been developed, including the HEDGE system for the Army's Human

Engineering Laboratory and the HFTEMAN (Human Factors Test and Evaluation

Manual) for the Navy.

A major Army test and evaluation system has been the Human Resources

Test and Evaluation System (HRTES) principally developed for operational

testing (cf., Kaplan and Crooks, 1980). HRTES assumes a prototype

operational sy?tem ready for operational test and evaluation. A test plan is

generated by HRTES, performance measures obtained, and the performance

evaluated by an application of multi-attribute utility theory. A principal

feature is the diagnosis and evaluation of unsuccessful performance. In short,

when did an error occur; why did it happen; and what should be done about

it?

In contrast, APM is directed toward developing systems where specific

system parameters are evolving in system design, places particular emphasis on

system goals and objectives, but at the same tine assigns particular measures

to system parameters and objectives. These data, if generated, would be

particularly pertinent as critical inputs to the HRTES. APM would provide a

system analysis framework for the HRTES evaluation. HRTES, then, extends

APM into the domain of operational testing but, at the same time, APN!

provides a measurement framework for the HRTES technique.

H. Summary

APM is a static, deterministic model principally useful for early system

development and for front-end analysis of human resources elements of military

systems. It is specifically valuable in deriving system, subsystem, and human

performance objectives in a manner consistent with standard system

engineering methods as well as the goals of MIL-H-46855B. No other available

models have this capability. Outputs of the APM may be used within other

model frameworks for:

0

Stochastic, time-varying, scenario-driven simulations

0

Training system development

0

Logistic support analyses

0 Human resources test and evaluation

It is, therefore, complementary to other models in these areas. Perhaps its

majc- weakness as a model is a lack of costing data which might be essential

for early evaluation decisions. In its computerized version it allows for quick

development and assessment of initial system goals, processes, and products

before actual system hardware and software have been selected.

-87-
V. RECOMMENDATIONS FOR FURTHER DEVELOPMENT AND APPLICATIONS

As noted earlier, the ultimate objective of this APM development work is to

provide users with a uniform, thorough, adaptive and efficient procedure to

aid in the process of deriving the most meaningful design specification

requirements, design specifications, evaluation measurement requirements or

evaluation measures for any planned or existing human-machine system

(particularly a training system). The users of the APM for design and

measurement purposes are expected to be at: the U.S. Army Infantry School

(USAIS); other schools of the U.S. Army Training and Doctrine Command

(TPADOC); the Directorate of Training Development (DTD); the Directorate of

Evaluation (DOE); and the Directorate of Combat Development (DCD). APM

users are also foreseen to include personnel with the U.S. Army Test Boards,

as well as the U.S. Army Operational Test and Evaluation Agency (USAOTEA)

and, of course, the U.S. Army Research Institute for the Behavioral and

Social Sciences (ARI). Among the individuals at those agencies who are seen

as the ultimate users of automated APM are training developers and evaluators,

TRADOC system managers (TSMs), system researchers and designers, analysts

and planners. The APM project has supplied the following products to help

these users perform their work:

0 An initial analytic process model for systems design and

measurement.

0 A generalizable taxonomy of measurable attributes for the

surveillance function found in many manned systems.

0 A generalizable model of training systems and their component

subsystems.

0 Generalizable taxonomies for the Design (curriculum development),

Enabling (trainer preparation) and Delivery (instruction) subsystems

of training systems.

0 A sample application of the model for deriving effectiveness measures

foi Learner Testing Specifications within the emerging BIFVTS,

including a generalizable taxonomy and application procedures.

0 A preliminary review of concepts for applying the APM in

establishing design requirements.

0 An untested procedural outline and demonstration package of a

computer-aided APM for deriving training systems measures (using

an Apple II Plus computer).

An annotated bibliography of the manned systems measurement

literature (244 documents cited and abstracted).

This list represents considerable progress in an area that has been so difficult

to standardize fur so many years. The model development is just now

-88-
beginning tu yield the kinds of guidelines and sample applications that soon

will allow it to be learned and applied by those who are its potential users.

To help bring the APM development process to the point of application by the

ultimate users, several tasks need to be carried out. These are listed in

Figure 30. ]t is noted that the items are not necessarily independent of each

other, but some tasks can be seen as being a part of others. They are listed

separately, however, because they are identifiably distinct or limited in their

purpose. For convenience, the tasks are divided into four groups, according

to their primary concerns: data base development, procedures development,

applications and application aids, and administrative items development.

A. Data Base Development

1. Develop Performance Items, Attributes, Measures and Specification

Requirements for the Remaining Training Subsystems (CommanfTLogistics and Emplacement!

Currently, generalized taxa for any design, enabling and delivery

subsystem have been developed, though their measures and specification

requirements remain in need of completion. Taxa for the additional generalized

training subsystems also should be developed. Their applicability to BIFVTS

or other issues could be verified through some real world comparison or

evaluation, by subject-matter experts (SMEs) or otherwise. The same

procedures should be applied as used in prior taxa development.

2. Develop Data Bases for Other Than Training Systems

This task is intended to make the APM useful for analyzing

human-machine systems in addition to those in the training system class.

Depending upon the active work areas of the using agency, generalized

taxonomies of performance items, attributes, measures and specification

requirements could be developed for such system classes as weapons,

communication, navigation, search and surveillance, target acquisition and fire

control, command and control, intelligence, electronic warfare or any other

type.

This task should identify the many (sometim»;s subtle) roles of human

operators in the total system measurement and evaluation process. It should

clarify the contribution ?.nd influence of operators in the various system

attributes selected for specification or measurement.

3. Identify the Sub-Taxonomies for Generally Stated Analytic Purposes.

This task is intended to provide the user with computer-aided

recommendations of appropriate performance items for certain stated purposes

of measurement or specification. It consists of a mapping process in which

subsets of performance items are identified for each generally stated analytic

purpose. The APM would then present the user with those recommendations,

if his purpose were one of those in storage, thus easin; the user's task of

generating a completely new subset of purpose-relevant performance items.

The user would then proceed by editing the recommended subset co suit his

needs. For reasons o

f

efficiency, this task should be carried out together

with Task 8.

-89-
A. Data Base Development

1 . Develop performance items, attributes, measures and

specification requirements for the remaining training

subsystems (commnnd, logistics and emplacement)

2. Develop data bases for other than training systems

3. Identify the sub-taxonomies for generally stated analytic

purposes

B. Procedures Development

4. Further develop procedures for APM application to

measures derivation

5. Further develop procedures for APM application to system

design specification

6. Develop procedures for differentiating specifications and

measures in terms of weighting factors, delay tolerance

and criticality

7. Modify procedures and programs for use with a computer

large enough to provide the user assistance, data base and

processing capacity, and speed consistent with the

expanded capabilities

8. Improve procedures for specifying new analytic purposes

C. Applications and Application Aids

9. Train users to apply the APM

10. Apply the APM to the measurement of an existing Army

system

11. Apply the APM to the specification of a new Army system

12. Provide the user with a computer-generated assessment of

each analysis

13. Develop computer-assisted instructions (CAI) for users of

the APM

D. Administrative Items Development

14. Incorporate on-line security and accounting procedures

15. Provide mechanisms for future modification of the APM

Figure 30. Recommended Further Research on the APM
•. w~'

B. Procedures Deve.opment

4. Further Develop Procedures for APM Application to Measures

Derivation

One of the most difficult-to-routinize segments of the APM is that in

which measurable attributes are determined and actual measures are selected.

Some progress has been made during this project in beginning to identify

specific procodurrs for measures derivation. More needs to be done, however,

especially if computer-aiding is to be used for this segment. In particular,

the analytic steps leading to measures derivation involve a determination of

issues that imply a need to measure particular attributes. The research

n ethod should include model application with several existing or developing

training systems to verify or modify its component parts, its adequacy, and its

practicality. The effort should also produce improved and amplified procedures and guidelines for using the model.

Additional work is needed to incorporate the measures of effectiveness and performance (MOD-MOP) development steps into the APM, identify

the relationships among MOE-MOP for systems-subsystems levels, and identify

the relationships among the machine and human components in terms of contribution to MOE-MOP development or specification. This subtask can build on

the current B1FVTS work, by continuing more sample or partial applications to

increasingly complex issues of military training, both to verify and to improve

the procedural guidelines. This process is analogous to the earlier one of

generating taxonomization guidelines. The procedural guidelines would allow

for the derivation of MOE-MOP specific to given measurement purposes and

provide for direct MOE-MOP (type) selection given the defined measurement

purpose. Those guidelines, when better understood, will define the algorithm

for measures derivation that then can be used to improve the computer-aided

process.

5. Fui ther Develop Procedures for APM Application to System Design

^pecification~

The purpose of this research item is analogous to that of Item 4, but

as applied to the APM for specification. Sample applications should involve the

specification of several new Army systems. All component parts of the

specification model should be addressed; concepts and definitions should be

clarified and application guidelines should be developed. In addition, the

utility of the model for assessing independently generated design requirementr.

should be evaluated.

6. Develop Develop Procedures Procedures for Differentiatina Differentiating Specifications Specifications and ^Measures in

Terms of Weighting Factors, Delay Tolerance and Criticalit

The practical aspects of model utilization suggest that there will be

times when the full range of performance taxa, attributes, specifications or

measures cannot be applied in its entirety, and that choices will have to be

made to select only the most important ones for application. To aid the user
in selecting those taxa, attributes, specifications or measures which are most

important, a procedure should be developed which helps to rate or rank those

items in accordance with such criteria as: criticality to mission performance,

amenability to alternatives or corrective actions, tolerance to postponement or

delay in application, or other criteria to be determined.

7. Modify Procedures and Programs for Use with a Computer Large

Enough to Provide the User Assistance, Data Base and Processing

Capacity, and Speed Consistent with the Expanded Capabilities

This task is intended to produce a practical and useful tool by

building upon the existing computer-aided demonstration capability. It can be

accomplished by adding flexibility, instruction options, thoroughness, and data

bases to the rudimentary computer-based model. It should also include a test

of the procedures by a skilled, but APM-naive, analyst to determine where

modifications are required to clarify, simplify or otherwise improve their value

and ease ol use.

8. Improve Procedures for Specifying New Analytic Purposes

This task can build upon the experience gained in conducting

Task 3. It ronsists of analyzing and documenting the process by which

analytic purposes are specified and how they subsequently influence the

model's application (as in Task 3). It is intended that the procedures

developed be incorporated in the computer-aided model, to assist users at the

appropriate stage of application.

C. Applications and Application Aids

9. Train Users to Apply the APM

This task is one in which the previously developed APM procedures

are Placed into operation with some potential users. One possible agency to

aid in this dissemination function is the U.S. Army Training and Doctrine

Command (TRADOC). Administrative contacts, explorations and information

exchange with possible interested agencies should begin well in advance of the

potential application efforts. The results of these trials should be used to

support Task 13 in the development of CAI resources.

10. Apply the APM to ihr Measurement of an Existing Army System

This task involves extending the preliminary DIFVTS applications of

the APM to a different measurement purpose that also can serve as a validation

or verification exercise. For example, the model can be applied in the context

of the Training Effectiveness Analysis (TEA) of the BIFVTS which was planned

by USAIS/USAIC to begin in the 4th Qtr FY 1982. Current developments ran

be extended to the point where derivation nf specific measure sets for the

effectiveness of the BIFVTS subsystem can be generated. These measure sets

can then be examined for feasibility, validity and utility in the context of the

TEA. It would be necessary to review the measurement issues to be addressed

in the TEA that are relevant to the BIFVTS curriculum, and one or more of

those Issues should be selected as the basis for this task. The performance

taxa pertaining to each selected Issue and the measurement derivation
guidelines should be used to generate a measures hierarchy for each selected

issue. Differences between APM and TEA measures should be examined so that

reasons can be determined for those differences. Commonalities should also be

noted and their reasons understood. For practical reasons, this in-depth

application and comparison should focus on a narrow aspect of the BIFVTS,

such as Basic Gunnery Learner Testing. This task would also provide an

opportunity to focus on human operator contributions to performance, such as

motivation, attitudes, capabilities, and procedures under given environmental,

administrative and other constraints,

11. Apply the APM to the Specification of a New Army System

Initial extension of the APM to the specification of system design

requirements was begun during the latter portion of this contract. This

follow-up task should extend and further explore the use of the APM as a

tool, through application to a new (or developing) system. A training system

is recommended as one possibility. The purpose is to improve and to verify,

to the extent possible, the capability of the APM to serve this function. This

can be done through ihe "blind" development of training system design

specifications for a nev or developing system to be selected by an interested

Army office. A comparative assessment can be carried out between design

specifications as developed by using the APM, and similar specifications as

developed by other methods, such as the Early Training Estimation System

under development by the ARI Fort Bliss Field Unit. This task could also

make use of the APM's capnbilitieG both to generate new design specifications

and to aid in the diagnostic assessment of previously generated design

specifications. For the same practical reasons as mentioned under Task 9, this

comparative application should take place in depth over a narrow aspect of the

new training by stem.

12. Provide the User with a Computer-Generated Assessment of Each

Analysis

This is a relatively small task which will provide the user with

feedback on how each analysis was carried out. It will provide assessments in

terms of thoroughness or breadth of the analysis, conformance with or

departures from computer recommendations, completeness or lack of same in

associating performance items with final measures or specifications, and other

criteria by which the computer can report on the user's activities. It is

intended that these assessments will prompt the user to make changes or

improvements where necessary or to obtain some measure of assurance that the

analysis was probably conducted in a satisfactory manner.

13. Develop Computer-Assisted Instructions (CAI) for Users of the APM

This task would provide the computer-based package with a built-in

training capability, using the self-pacing techniques of computer-assisted

instructions (CAI). Completion of this task should follow whatever remaining

development effort is necessary to yield routinized (computer-aided)

procedures for application by users. It should also follow application tests of

those procedures by user personnel (see Task 9). This task can make use of

any handbooks, guidelines, workshops or other items that were previously
developed to train or assist users. The task should also consider the

training/usage of the APM process by various personnel types and for both

possible purposes (evaluation and specification of human-machine systems).

CAI can insure that procedures for training users of the APM are as

programmatic, proceduralized, routinized and simplified as possible. Field

tests of these CAI procedures should aim to confirm their completeness and

accuracy, thuir utility (based on the acceptance by users) and their validity

(based on the quality and comprehensiveness of products).

D. Administrative Items Development

14. Incorporate On-Line Security and Accounting Procedures

This task is designed to protect the integrity of the model and its

data bases, once it becomes accessible to a wide variety of users. It requires

the incorporation of appropriate access codes in order to modify data basep or

programs. It also provides for a record of usage by amount, type, user

description and other accounting-type information that can serve to describe

how the APM is being used, and to suggest how additional developmental

resources can be best applied.

15. Provide Mechanisms for Future Modification of the APM

To help the computer-aided instruction system in carrying out its

functions of information exchange and documentation for users, a technical

capability should be established for maintaining the APM, its associated data

bases, and its various ^plication routines and records. Users can then be

helped with the latest fund of knowledge, techniques and experience. A

current directory of users would allow for rapid distribution of application

documents, program modifications, information requests, and suggestions to

improve the value of this resource to users.

One of the most important sources of ideas to enhance the APM's

utility and value is the user population itself. A method should be established

for permitting and, in fact, encouraging users to supply feedback of

application ideas to the central APM technical office, with the assurance that

those ideas will be given careful consideration and dissemination in the most

appropriate manner. Dissemination can employ a newsletter, a "new

application" description, a revised data base or even a message for CRT

display when users log on to use the model. In fact, one way in which users

can conveniently communicate briefly with the APM central office could be via

the interactive terminal, such as by typing in messages or inquiries after

using the program or by pausing during use when a problem is encountered.

The specific mechanisms for keeping the APM current and useful should be

determined and implemented in this task item.
VI. SELIXTEÜ BIBLIOGRAPHY

Askren, W.n., and Linz, L.M. Human resources cata in system design trade

studies. Human Factors, 1975, 17(1), 4-12.

Bloom, B.S. (Ed.) Taxonomy of educational objectives. New York: Longman,

Inc., 1956.

Bloom, R.F., Gates, J.F., Jr., and Hamilton, J.W. An initial analytic process

model for systems measurement: Extensions of the systems taxonomy model.

Darien, CT: Dunlap and Associates, Inc., 30 April 1981. (Pinal Summary

Report)

Bloom, R.F., Gates, J.F., Jr., Hamilton, J.W. and Leaf, W.A. An analytic

process mode] for systems design and measurement; Applications to trnininp

systems! Darien, CT: Dunlap and Associates, Inc., 2 May 1

0

82. (Final

Summary Report)

Bloom, R.F., Pepler, R.D., Schimenz, M.V., and Lenzycki, H.P., IFV/CFV

?ersonnel selection analysis. Darien, CT: Dunlap and Associates, Inc., Julv

979. (Army Research Institute Research Note 80-41)

Branson, R.K., Rayner, G.T., Cox, J.L., Furman, J.P., King, F.J., and

Hannum, W.II. Interservice procedures for instructional systems development.

Tallahassee, FL: Florida State University, 1975.

Briggs, L.J. (Ed.) Instructional design; Principles and applications.

Englewood Cliffs, NJ; Educational Technology Publications, Inc., 1977.

Byrne, B., Mullally, A., and Rothery, B. The art of systems analysis.

Englewood Cliffs, NJ; Prentice-Hall, 1971. ~

Carr, R.M., et al. An overview of human factors in system development.

Washington: National Security Industrial Association, May 1980.

Department of Defense. Military specification: Human engineering

requirements for military systems, equipment and facilities. M1L-H-46855PJ , 31

January 1979.

Dunlap and Associates, Inc. Instructor training institute for the police traffic

services teacher; Participant's notebook. Darien, CT; Dunlap and

Associates, Inc., 1980.

Edwards, J.M., Bloom, R.F., Gates, J.F., Jr., Sipitkowski, S., Brainin,

P.A., Eckenrode, R.J. and Zeidler, P.C. An annotated bibliography of the

manned systems measurement literature. Darien, CT: Dunlap and Associates.

Inc., 30 November 1981.

Finch, G. (Ed.) Educational and training media; A symposium. Washington,

DC; National Academy of Sciences—National Research Council, Publication 789,

1960.
Finley, D.I.., and Muckler, F.A. Human factors research and the development

of a manned systons applications science: The system sampling problem and a

solution] Northridge, CA: Manned Systems Sciences, Inc., July 1976.

Finley, D.L., Muckler, F.A., Gainer, C.A., and Obermayer, R.W. An

analysis and evaluation rncthodology for command and control; Final tecHrncal

report. Northridge, CA: Manned Systems Sciences, Inc. , November 19?5.

NT IS No. AD A023871.

Greer, G.D., Jr. Basic training effectiveness: A discussion of instructior;

ccntraliyation, the training curriculum and achievenent evaluation. Fort Ord,

CA: U.S. Amy Leadership, 11 uman Research Unit, 1957.

Jorgensen, C.C. Farly training projeetion—the support data problem—an

interim solution. In B. Monroe and F. Farrell (Eds.) Proceedings of the first

annual conference on personnel fc training factors in "system effectiveness.

San Diego: NSIA, July 1981. Plges 135-138.

Kaplan, J.D., and Crooks. W.H. The concept and evaluation of the Human

Resources Test and Evaluation System (HRTES). Proceedings of 24th annual

meeting of the Human Factors Society. Eos Angeles: HFS, 1980. Pages

557-559.

King, G.F., and Askren, W.B. Human resources, logistics, and cost factors

in weapon system development: Project summary. USAF: AFHRL-TR-80-8,

September 1980.

Klotter J.C. Techniques for police instructors. Springfield, IE: Charles C

Thomas, 1963." ,

Kyle, T.N., and Markisohn, G. The Integrated Facilities Requirements System

(1FRS): A planning tool for the U.S. Navy's pilot training program. In

D.C. Clough, C.C. Eewis, and A.E. Oliver (Eds.) Manpower planning models.

London: English Universities Press, 1974. Pages 255-261.

Longo, A.A., Gaddis, J.T. , and Whitehouse, B. Preliminary evaluation plan

for U.S. Army computerized training system. Fort Monmouth, NJ: U.S. Army

Signal Center and School, 1974.

Machol, R.E. Methodology of system engineering. In R.E. Machol (Ed.)

System engineering handbook. New York: McGraw-Hill, 1965. Chapter 1.

Mackie, R.R. An engineer's guide to the use of human resources in electronic

system design. USN: NPRDC TN 79-8, June 1979.

McDonnell Douglas Astronautics Co.-E^st. A survey and analysis of military

computer-based training systems: (A two part study). St. Louis, MO:

McDonnell Douglas Astronautics Co.-East, 19/7.

Meister, D. Behavioral foundations of system development. New York:

McGraw-Hill, iTRT.

Meister, D., and Rabideau, G.F. Human factors evaluation in system

development. New York: John Wiley, 1965.
Niehaus, R.J. Computer-assisted human resources planning. New York:

John Wiley, 1979. Pages 269-279.

O'Brien, L.Il. Early training estimation system: Draft yearly report

number 2. Wilmington, MA: Dynamics Research Corporation, ARI Contract

MDA 903-80-C-0525, August 1982.

O'Neil, H.F. (Ed.) Procedures for instructional system development. New

York: Academic Press, 1979.

Pew, R.W., Baron, S.f Feehrer, C.E., and Miller, D.C. Critical review and

analysis of performance models applicable to man-machine systems evaluation.

Washington, D.C: AFOSR/AFHRL, BBN Report No. 3446, March 1977.

Rouse, W.B. System engineering models of human-machine interaction. New

York: North Holland, I960.

Seidel, R.J., and Wagner, H. Front-end analysis to aid emerging training

systems: Workshop summary. Alexandria, VA: HumRRO, DARPA Contract

MDA 903-78-C-0023, December 1979.

Shapiro, R.G., Bloom, R.F., and Gates, J.F., Jr. The computer-aided

analytic process model: Operations handbook for the APM demonstration

package. Norwalk, CT: Dunlap and Associates East, Inc., 28 February 1983.

Shapiro, R.G. The computer-aided analytic process model; Appendix to the

operations handbook for the APM demonstration package" Norwalk, CT:

Dunlap and Associates East, Inc., 28 February 1983.

Siegel, A.I., and Wolf, J.J. Man-machine simulation models. New York:

John Wiley, 1969.

Siegel, A.I., Wolf, J.J., and Ozkaptan, H. Behavioral simulation modeling:

A front-end analytic tool. In B. Monroe and F. Farrell (Eds.) Proceedings of

the first annual conference on personnel & training factors in system

effectiveness. San Diego: NSIA, July 1981. Pages 59-64. ~ ~~~

Smode, A.F., and Lam, K.D. (Eds.) Military instructor training In

transition. Orlando, FL: Training Analysis and Evaluation Group, 1975.

Thurmond, P. Development of analysis, design, and development techniques

for team ISD. San Diego, CA: Path Research, 1980.

U.S. Department of the Army. Interservice procedures for instructional

systems development; Executive summary and model, Vol. I-V. Fort Benning,

GA: U.S. Army Combat Arms Training Üoard, TRADOC Pamphlet

No. 350-30, August 1975.

U.S. Department of the Army. Toward a new hand gun. Army Research,

Development and Acquisition Magazine, Alexandria, VA: Hq., U.S. Army

Materiel Development and Readiness Command, January-February 1981.
Vineberg, R., and Joyner, J.N. Instructional system development USD) in

the Armed Services: Methodology and application. Alexandria, VA: HumRRO

TR 80-1, January 1980.

Weddle, P.D. Front-end analysis as a tool for system supportability

assessment. In R. Monroe and F. Farrell (Eds.) Proceedings of the first

annual conference on personnel & training factors in system effectiveness.

San Diego: NS1A, July 1981 . Pages 80-83.
VII. ANNOTATED GLOSSARY OF ANALYTIC PROCESS MODEL

TERMS AND CONCEPTS

Numerous concepts, procedures, techniques and data bases are presented in

this report. Of necessity, these are expressed in technical terminology, much

of which is specific to training applications. Understanding and application of

thpse concepts, data bases, etc., requires understanding of the terminology.

This glossary defines and clarifies the terms used, in an attempt to foster that

understanding.

The defini'ions and clarifications presented herein are drawn principally from

the following sources:

Bell, D.Ai An introduction to the performance-based training system.

Ottawa, Ontario, Canada; ACFP (undated).

Bloom, R.F., Oates, J.F., Jr. and Hamilton, J.W. An initial analytic

process model for systems measurement! Extensions of the systems

taxonomy modeFi Darien. CT: Dunlap and Associates, Inc., 30 April

1981. fFinal Summary Report)

Bloom, R.F., Oates, J.F., Jr., Hamilton, J.W. and Leaf, W.A. An

analytic process model for systems design and measurement; Applications

to training systems. Darien, CTl Dunlap and Associates, Inc., 2 May

1982 . (Final Summary Report)

Branson, R.K., Rayner, G.T., Cox, J.L., Furman, J.P., King, F.J.

and Hannum, VV.H. Interservicc procedures for instructional , systems

development. Tallahassee, Florida: Florida State University, 1975.

Briggs, L.J. (Ed.) Instructional design; Principles and applications.

Englcwood Cliffs, New Jersey: Educational Technology Publications, Inc.,

1977.

Finley, D.L. and Muckler, F.A. Human factors research and the

development of a manned systems applications science; The system

sampling problem and a solution. Northridge, CA: Manned Systems

Sciences, Inc., July 1976.

Finley, D.L., Muckler, F.A., Gainer, C.A. and Obermayer, R.W. An

analysis and evaluation methodology for command and control: FinäT"

technical report"! Northridge, CA; Manned Systems Sciences, Inc.,

November 1975.

McLagan, P.A. Helping others learn; Designing programs for adults.

Reading, Massachusetts: Addison-Wesley Publishing Company (undated).

Smode, A.F. and Lam, K.D. (Eds.) Military instructor training in

transition. Orlando, Florida: Training Analysis and Evaluation Group,

T97^
GLOSSARY

ACHIEVEMENT.

See: LEARNING ACHIEVEMENT

ACTION VERB. The action verb is the behavior indicator of a performance

objective.

In a statement of a performance objective, the action verb discloses the

observable behavior that the learner is required to manifest.

See also: PERFORMANCE OBJECTIVE

CAPABILITY VERB

ADULT LEARNING CLIMATE. The adult learning climate is that set of

environmental, procedural, and interpersonal features that, typically,

facilitates learning by ndults.

Features of the adult learning climate:

1. Learners are treated with respect.

2. Learners participate in the learning planning process, helping to

translate diagnosed needs Into specific performance objectives and

learning activities.

3. Learners participate in the diagnosis of their own needs.

4. Learners' comments, questions, discussions are encouraged and

welcome.

5. Learners share in conducting learning experiences.

6. Learners have opportunities for self-evaluation.

7. Emphasis is placed on the practical applications of the learning.

8. Experiential techniques are used to the fullest possible extent (e.g.,

group discussion, simulations, case studies, role-playing, etc.)

Synonym: ADULT LEARNING ENVIRONMENT

See also: EXTERNAL CONDITIONS OF LEARNING

INDIVIDUALIZED INSTRUCTION

ANALYSIS. The first phase of instructional system development.

Comprises the early stages of curriculum design, such as the analysis of

needs, goals and objectives, and the organization of course units.

Entails procedures for defining the job to be trained; dividing the job

down into statements of tasks; and selecting the tasks to be trained.

Further extends to constructing job performance measures for tasks

selected for training, and to deciding whether the tasks should be trained

in schools, on the job, or elsewhere.
Synonym: FRONT END ANALYSIS

See also: TRAINING NEEDS ASSESSMENT

JOB ANALYSIS

ANALYSIS OF EXISTING COURSES. In instructional system development,

analysis of existing courses is undertaken to avoid needless duplication of

effort, by permitting existing courseware to be adapted to new training

needs.

Factors to consider in the analysis of existing courses:

1. The acceptability of the job analysis on which the existing course

was based.

2. The appropriateness of the criteria that were used to select tasks to

be trained through the existing course.

3. The validity of the job performance measures used in the existing

course.

4. Validation of the existing course.

ANALYSIS OF PERFORMANCE OBJECTIVES.

See: TASK ANALYSIS

ANALYTIC PROCESS MODEL. The Analytic Process Model is a conceptual

framework for systematizing the design and evaluation of human-machine

systems.

Stages of the Analytic Process Model:

1. Establish the Design/Evaluation context by identifying the system's

work requirements. (This is the Systems Taxonomy Model stage.)

2. Establish the focus of the Design/Evaluation in terms of necessary

system attributes.

3. Implement the application details to Design/Evaluate the attributes.

4. Interpret the outcome of the Design/Evaluation, and develop

conclusions and recommendations.

Synonym: APM

See also: SYSTEMS TAXONOMY MODEL

DESIGN

EVALUATION

APM.

See: ANALYTIC PROCESS MODEL
ASPECTS OF PERFORMANCE. The aspects of human-machine system

performance form one dimension for organizing the system's work

requirements. The aspects of performance address the work itself and

the circumstances under which the work is required.

The aspects of performance include five mutually exclusive categories:

1. Performance Potentialities

(inherent capabilities)

2. Performance Processes

(activities, methods, techniques)

3. Performance Products

(goods, services, other deliverables)

4. Performance Environment

(natural impediments to the work)

5. Performance Constraints

(man-made, artificial impediments)

The aspects of performance form one dimension of the Systems Taxonomy

Model. They interact with another dimension, viz., levels of system

description, to form fifteen cells or sub-taxonomies within which any

system's work requirements can be exhaustively identified and organized.

See also: SYSTEMS TAXONOMY MODEL

LEVELS OF SYSTEM DESCRIPTION

PERFORMANCE POTENTIALITIES

PERFORMANCE PROCESSES

PERFORMANCE PRODUCTS

PERFORMANCE ENVIRONMENT

PERFORMANCE CONSTRAINTS

ATTITUDINAL DOMAIN OBJECTIVES. Performance objectives belonging to the

attitudinal domain require that the learner demonstrate that he or she

chooses some prescribed course of personal action, or behaves in

accordance with a prescribed set of values.

See also: PERFORMANCE OBJECTIVES

LEARNING DOMAIN TAXONOMY MODELS

INFORMATION DOMAIN OBJECTIVES

MENTAL SKILLS DOMAIN OBJECTIVES

PHYSICAL SKILLS DOMAIN OBJECTIVES

ATTRIBUTE. An attribute of a human-machine system is a concrete,

observable feature of the system that bears directly on the work that the

system can and does perform.

Attributes are linked to specific taxa of performance. For each

performance taxon, there exists a set of system attributes that

collectively determine whether the system satisfies that taxon.

102-
A systenf! attribute is something that a system designer can and must

control, through design decisions, to insure that the system will satisfy

the performance taxon to which the attribute is linked. An attribute also

is something that an evaluator can and must measure to determine whether

the performance taxon is met.

See also: TAXON

MEASURE

DESIGN SPECIFICATION

BEHAVIORAL OBJECTIVE.

See: PERFORMANCE OBJECTIVE

CAPABiLITY VERB. The capability verb of a performance objective statement

indicates the particular domain of learning to which the objective belongs.

See also: PERFORMANCE OBJECTIVE

DOMAINS OF LEARNING

ACTION VERB

CHARACTERISTICS-LEVEL TAX A. Characteristics-level taxa are statements of

how a specific purpose behind a system's work is required to be

achieved. Characteristics-level taxa define the particular Ingredients,

methods and milestones that the system is to include in its work in order

to satisfy its reasons for working.

A Characteristics-level taxon is linked to or descends from a particular

Functional Purooses-level taxon. The Characteristics-level taxon defines

a detailed capability ingredient, or procedural step, or intermediate

output necessary to achieve a particular purpose behind some basic

potentiality, process or product. Characteristics-level taxa also include

definitions of the relevant environmental and constraining factors that may

interfere with the system's ability to provide these capability ingredients,

carry out these procedural steps, or deliver these intermediate outputs.

Characteristics-level taxa consist of the system's performance

potentialities, processes, products, environment and constraints viewed at

the bottom-most level of system description. It is a view of work

requirements as things that are to be achieved in certain ways.

See also: TAXON

OBJECTIVES-LEVEL TAXA

FUNCTIONAL PURPOSES-LEVEL TAXA

LEVELS CF SYSTEM DESCRIPTION

ASPECTS OF PERFORMANCE

COGNITIVE STRATEGIES. A learner's cognitive strategies are the internally

organized mental skills by means of which the learner exercises control

over his or her own processes of attending, learning, remembering, and

thinking.

See also: LEARNING DOMAIN TAXONOMY MODELS

INTERNAL CONDITIONS OF LEARNING

-103-
COLLATERAL SYSTEM. A collateral system of a particular human-machine

system of interest is, itself, a human-machine system that interacts

directly with the system of interest so that it affects the work required

of the system of interest.

A collateral system may work either cooperatively or in competition with

the system of interest.

Collateral systems are members of the system hierarchy of the system of

interest.

See also: HUMAN-MACHINE SYSTEM

SYSTEM HIERARCHY

COMMAND SUBSYSTEM. The Command Subsystem is one of the six generic,

functionally oriented subsystems of any training system. Command

provides administrative control over the training system by assessing

training needs; allocating resources; recruiting training system personnel;

and monitoring, evaluating and recording learner and system performance.

Command contributes to the learning-helping function by constructing and

managing the system in which the learning activities can take place.

The principal operator or staff member of the Command Subsystem is the

training administrator.

See also: TRAINING SYSTEM

LEARNING

LEARNING-HELPING

CONDITIONS OF LEARNING. The conditions of learning are those features or

qualities of the training system, including the personnel, equipment and

procedures, that affect how well a particular learner learns.

See also: INTERNAL CONDITIONS OF LEARNING

EXTERNAL CONDITIONS OF LEARNING

CRITERION-REFERENCED TEST.

See: OBJECTIVE-REFERENCED TEST

CUE. A cue is a subtle or indirect form of guidance, given to help a learner

recall and apply some particular fact, procedure, word, phrase, etc.,

without actually supplying the fact, procedure, etc.

See also: GENERAL LEARNING GUIDELINES

PROMPTING

DECISION FLOW CHARTING. Decision flow charting is a procedure for

identifying and assessing the decisions required in a task, by means of

reduction to a series of binary choices.

See also: INFORMATION PROCESSING ANALYSIS

DECISION LEVEL ANALYSIS

-104-
DECISION LEVEL ANALYSIS. Decision level analysis is applied to the

decisions required in a task to establish a rating of the time, complexity

and nature of each decision.

See also: INFORMATION PROCESSING ANALYSIS

DECISION FLOW CHARTING

DELIVERY SUBSYSTEM. The Delivery Subsystem is one of the six generic,

functionally oriented subsystems of any training system. Delivery is the

most important training subsystem, in that the primary training function

(learning) takes place there. In the Delivery Subsystem, the prescribed

sensory learning activities are carried out and the learner achieves the

intended behavioral effects.

The principal operator of the Delivery Subsystem is the learner. The

iearner is assisted in Delivery by all other training system personnel,

who play various learning-helping roles.

See also: TRAINING SYSTEM

LEARNER

LEARNING

LEARNING-HELPING

DEPENDENT OBJECTIVE. One performance or enabling objective is dependent

upon another if it cannot be achieved unless the other first is achieved.

If one objective is dependent upon another, it must he pursued after the

other, during training.

See also: INDEPENDENT OBJECTIVES

SUPPORTIVE OBJECTIVES

PERFORMANCE OBJECTIVE

SEQUENCING OBJECTIVES

DESIGN. Usage No. 1 (Principal Definition): One of two applications of the

Analytic Process Model. Design of a human-machine system is the process

of selecting and securing all of the attributes that the system must

possess in order to satisfy its work requirements.

Usage No. 2 (Secondary Definition): One of six generic subsystems of

any training system.

Synonym; DESIGN SUBSYSTEM

See also: ANALYTIC PROCESS MODEL

ATTRIBUTE

DESIGN SPECIFICATION

EVALUATION

DESIGN SPECIFICATION. A design specification is a plan for insuring that a

system will have some particular required attribute.

-105-
A design sppcification describes how the system will be built, maintained

and operated so that a specific attribute needed to satisfy a particular

performance requirement or taxon will be present.

See also: ATTRIBUTE

TAXON

MEASURE

DESIGN SUBSYSTEM. The Design Subsystem is one of the six generic,

functionally oriented subsystems of any training system. Design carries

out the development of the curriculum/program of instruction. In so

doing. Design plans the instructional events, selects training technology,

assembles content material, and defines facilitator and learner

requirements.

The Design Subsystem is the portion of the training system in which the

Inst/uctional System Development (ISD) model is applied.

Design Subsystem contributes to the learning-helping function by

identifying the specific behavioral effects needed by the learners and by

planning the sensory activities that will lead to those effects.

The principal operator of the Design Subsystem is the curriculum

developer.

See also: TRAINING SYSTEM

INSTRUCTIONAL SYSTEM DESIGN

LEARNING

LEARNING-HELPING

DEVELOPMENT OF OBJECTIVES. Development of objectives is the process by

which performance objectives, enabling objectives and prerequisites for

training are identified.

Steps in the development of objectives:

1. Identify performance objectives by translating job performance

measures into training-oriented terms.

2. Analyze the performance objectives (tasks) to identify constituent

prerequisite or supportive achievements.

3. Continue the analysis until a level of prerequisites is reached which

all learners have achieved.

Product of the development of objectives;

A list of performance objectives and prerequisites analyzed to the

level where it en be assumed that the individuals to be trained are

prepared to begin.

-106-
See also: TASK ANALYSIS

PERFORMANCE OBJECTIVE

ENABLING OBJECTIVE

ESSENTIAL PREREQUISITE

SUPPORTIVE PREREQUISITE

DOMAINS OF LEARNING« The domains of learning are categories Into which

performance objectives, enabling objectives, and prerequisites may be

grouped, in accordance with whether the prescribed action/behavior is

mental, physical, or attitudinal.

See also: LEARNING DOMAIN TAXONOMY MODELS

EMPLACEMENT SUBSYSTEM. Thr Emplacement Subsystem is one of the six

generic, functionally oriented subsystems of any training system.

Emplacement takes the plans developed by Design Subsystem and

constructs, acquires or assembles all materials, equipment, installations,

supplies, etc., needed to implement those plans.

Emplacement contributes to the learning-helping function by providing the

tools needed to carry out the sensory learning activities.

The principal operator of the Emplacement Subsystem is the facilities

developer.

See alco: TRAINING SYSTEM

DESIGN SUBSYSTEM

LEARNING

LEARNING-HELPING

ENABLING OBJECTIVE. Ar enabling cbjective is a learned capability that

facilitates the learning of a larger performance objective. An enabling

objective can be conceived as an intermediate achievement, or milestone,

that must be or can be accomplished to progress toward the final

achievement represented by the performance objective.

See also: PERFORMANCE OBJECTIVE

ESSENTIAL PREREQUISITE

SUPPORTIVE PREREQUISITE

LEVELS OF OBJECTIVES

LESSON PLAN

ENABLING SUBSYSTEM. The Enabling Subsystem is one of the six generic,

functionally oriented subsystems of any training system. Enabling is,

itself, a self-contained training system: it carries out learning and

learning-helping activities to enable facilitators to (1) become familiar with

the plans, training content, equipment and facilities provided by the

Design and Emplacement Subsystems; (2) become qualified and prepared

to teach the curriculum effectively; and (3) tailor the curriculum to the

specific needs of a particular class or team of learners.

•107-
Within the total training system, Enabling contributes to the

learning-helping function by providing personnel qualified to assist the

learners directly by presenting information, demonstrating techniques,

coaching learners' efforts, evaluating learners' performance, and

correcting learners' deficiencies. Enabling also contributes to

learning-helping by fine-tuning a general program of Instruction to

specific learning needs of specific learners.

The principal operator of the Enabling Subsystem is the facilitator.

Common synonyms for Enabling Subsystem are "Instructor Prepnration"

and "Training-the-Trainer."

See also: TRAINING SYSTEM

DESIGN SUBSYSTEM

EMPLACEMENT SUBSYSTEM

FACILITATOR

LEARNING

LEARNING-HELPING

ENTRY TEST. An entry test is a test of a prerequisite for training.

Entry tests provide a basis for accepting or rejecting candidate learners,

and for determining whether particular learrers require remedial training.

See also: OBJECTIVE-REFERENCED TEST

PRETEST

POSTTEST

ERROR DATA. Error data consist of item analyses of pilot test learners' test

responses. The analyses disclose achievement failures, which in turn

point to needed improvements in the instruction.

See also: FEEDBACK

ESSENTIAL PREREQUISITE. An essential prerequisite for a particular

performance objective or enabling objective is a performance capability

that actually is incorporated within the objective in question.

The objective cannot be achieved unless the essential prerequisite is

satisfied.

See also: PERFORMANCE OBJECTIVE

ENABLING OBJECTIVE

DEPENDENT OBJECTIVE

DEVELOPMENT OF OBJECTIVES

SUPPORTIVE PREREQUISITE

EVALUATION. Evaluation is one of two applications of the Analytic Process

Model. Evaluation of a human-machine system is the process of

determining whether the system satisfies all of its work requirements.

Synonym; MEASUREMENT

•108-
See also: ANALYTIC PROCESS MODEL

MEASURE

DESIGN

EXTERNAL CONDITIONS OF LEARNING. A learner's external conditions of

learning for a particular performance or enabling objective are those

features or qualities supplied by the facilitator, the learning materials and

media, the facilities, etc., that affect the particular learner's ability to

achieve the particular objective.

See also: TACTICS OF INSTRUCTION

INSTRUCTIONAL FEATURES

CONDITIONS OF LEARNING

INTERNAL CONDITIONS OF LEARNING

ADULT LEARNING CLIMATE

FACILITATOR. A facilitator is a key learning-helper in a training system,

and is the principal operator of the training Enabling Subsystem.

The facilitator interacts directly with the learner to aid the learner in

carrying out planned sensory activities to achieve desired behavioral

effects.

Common synonyms for "facilitator" include "teacher," "instructor," and

"trainer."

See also: LEARNING-HELPING

TRAINING SYSTEM

ENABLING SUBSYSTEM

DELIVERY SUBSYSTEM

LEARNER

FEEDBACK. In course design, feedback is the process of gathering

information from learners as the basis for improving the training.

See also: FORMATIVR EVALUATION

PILOT TEfiTING

ERROR lUTA

FIELD TRIAL.

See: PILOT TESTING

FORMATIVE EVALUATION. Formative evaluation is the tryout and revision of

a training system prior to its full-scale operational delivery. Formative

evaluation entails delivery of the instruction to representatives of the

intended learner population, collection of useful data from those

representatives, analysis and interpretation of those data, and revision of

the instruction based upon the findings.

The word "formative" implies that the evaluation occurs while the

instructional program is still in development. Formative evaluation

provides an opportunity to correct mistakes made in the instructional

design process before the delivery process fully matures.

-109-
See also: PILOT TESTING

FEEDBACK

SUMMATIVE EVALUATION

FRONT END ANALYSIS.

See: ANALYSIS

FUNCTIONAL PURPOSES-LEVEL TAX A. Functional Purposes-level taxa are

statements of why specific performance is required of a system, without

reference to how the requirement is to be achieved.

A Functional Purposes-level taxon is linked to or descends from a

particular Objectives-level taxon. The Functional Purposes-level taxon

defines a specific reason why the system is expected to possess a

particular basic capability, or carry out a basic activity, or deliver a

basic output.

Functional Purposes-level taxa also include definitions of the relevant

environmental and constraining factors that may impede satisfaction of the

reasons behind the system's work requirements.

Functional Purposes-level taxa consist of the system's Performance

Potentialities, Processes, Products, Environment and Constraints viewed

at the middle level of system description. It is a view of work

requirements as things that exisit for specific purposes, the achievement

of which merits study.

See also: TAXON

OBJECTIVES-LEVEL TAXA

CHARACTERISTICS-LEVEL TAXA

LEVELS OF SYSTEM DESCRIPTION

ASPECTS OF PERFORMANCE

GENERAL LEARNING GUIDELINES. The general learning guidelines consist of

four requirements deemed essential for helping learners to learn:

1. Inform the learner of the objective.

2. Provide for active practice by the learner.

3. Provide guidance and prompting to the learner.

4. Provide feedback to the learner concerning his/her performance.

See also: INSTRUCTIONAL EVENTS

PROMPTING

CUE

GOAL. A goal is a statement of what "ought to be" with respect to learners'

or job performers' achievements.

In the training context, goals usually are broad statements of the ultimate

intended outcomes of educational enterprises.

See also: NEED

NEEDS ASSESSMENT

•110-
HUMAN-MACHINE SYSTEM. "Humnn-machine system" is the general entity of

interest to the Analytic Process Model. It is a collection of people,

equipment and procedures that work together to accomplish specified

functions.

INDEPENDENT OBJECTIVES. Two performance objectives, or two enabling

objectives, are independent if the learning/achievement of one of them has

no effect on the learning/achievement of the other.

If two objectives are independent, the sequence in which they are

pursued during training is irrelevant.

See also: DEPENDENT OBJECTIVE

SUPPORTIVE OBJECTIVES

PERFORMANCE OBJECTIVE

SEQUENCING OBJECTIVES

INDIVIDUAMZED INSTRUCTION. Individualized instruction is a training

delivery subsystem that permits:

1. Learners to choose their own individual performance objectives, or

2. Alternative media to be selected by individual learners to pursue

objectives, or

3. Learners to progress at their own rates.

Individualized instruction is training in which differences in individuals'

needs, learning rates, attention spans, learning styles, levels of

competence, etc., are reflected in the learning activities experienced by

individual learners. Individualized instruction is a basic requirement for

all adult training.

An individualized instructional system should make it possible for

individual learners to progress at their own rates of learning and to

continue the instructional process until mastery. The instructional system

must obtain reliable and timely information on an individual learner's

learning progress and make adaptations appropriate to the individual

learner.

See also: INTERNAL CONDITIONS OF LEARNING

ADULT LEARNING CLIMATE

LEARNING STYLE

INFORMATION DOMAIN OBJECTIVES. Performance objectives belonging to the

information domain require that the learner demonstrate the ability to

recall bodies of knowledge.

-Ill-
N- tinI) ff >^

i

,

ISee also: PERFORMANCE OBJECTIVES

LEARNING DOMAIN TAXONOMY MODELS (cont'd)

MENTAL SKILLS DOMAIN OBJECTIVES

| PHYSICAL SKILLS DOMAIN OBJECTIVES

ATTITUDINAL DOMAIN OBJECTIVES

'

INFORMATION PROCESSING ANALYSIS. A component of Task Analysis,

information processing analysis is applied to a performance objective to

reveal the sequence of mental operations it requires.

Information processing analysis is premised on the notion that complex

human performance can be conceived as composed of simpler parts. The

fundamental rule for information processing analysis is: carry the

analysis to the point where you are convinced that the revealed

operations are simple.

Information processing analysis often is depicted via flowcharting; the

flowchart nodes consist of information inputs, mental actions/computations,

and decisions.

See also: TASK ANALYSIS

TASK CLASSIFICATION

LEARNING TASK ANALYSIS

INSTRUCTIONAL EVENTS. The instructional events are the distinct steps in

a lesson through which the learner, facilitator and material interact to

progress toward the lesson objective.

The instructional events include:

1. Gaining learner's attention.

2. Informing learner of the objective.

3. Stimulating recall of prerequisites.

4. Presenting stimulus material.

5. Providing learning guidance.

6. Eliciting the performance.

7. Providing feedback about performnMce correctness.

8. Assessing the performance.

9. Enhancing retention and transfer.

See also: LESSON PLAN

STRATEGY OF INSTRUCTION

TACTICS OF INSTRUCTION

INSTRUCTIONAL FEATURES. Instructional features are those components of a

lesson that are distinct from the instructional content. The instructional

features include the st?tements of performance objectives and enabling

objectives, self-tests, practice exercises, discussion topics, etc.

INSTRUCTIONAL SETTING. The instructional setting for a particular task or

performance objective is the context, situation or type of environment in

which training will take place for that task/objective.

_1 17.
Military training systems typically use one or more of these settings:

1. Job Performance Aid (JPA)

2. Self-Teaching Exportable Package (STEP)

3. Formal On-the-Job Training (FOJT)

4. Installation Support School (ISS)

5. Resident School (RS)

INSTRUCTIONAL SYSTEM DESIGN. Instructional system design is the broad

application of the "systems approach" to training. It entails the actual

preparation of instruction. It involves determination of how the learners

will be managed, the kinds of learning experiences they will have, the

activities in which they will engage, and the form and content of the

instructional delivery system.

Steps in instructional system design:

1. Determine training needs.

2. Organize the course and the units of instruction.

3. Write performance objectives.

4. Analyze the objectives.

5. Prepare assessments of learner performance.

6. Sequence instruction within each objective.

7. List instructional events for enabling objectives.

8. Select stimulus types.

9. Select instructional media.

10. Select conditions of learning.

11. Write prescriptions/plans.

12. Develop materials.

13. Conduct formative evaluations.

14. Train pilot test facilitators.

15. Conduct pilot test.

16. Make final revisions.

17. Conduct summative evaluations.

Key products of instructional system design:

1. A performance objective corresponding to ea«.h task to be trained.

2. Test items to measure each performance objective.

3. Test of entry-level behaviors to validate original assumptions.

4. Sequencing of instruction for all dependent tasks/enabling

objectives.

Synonym: INSTRUCTIONAL SYSTEMS DEVELOPMENT

INTERNAL CONDITIONS OF LEARNING. A learner's internal conditions of

learning for a particular performance or enabling objective are those

features or qualities within the learner, himself or herself, that affect the

learner's ability to achieve that objective.

A learner's internal conditions of learning include his or her motivation

with respect to the objective of interfest, recall of prior learning,

-113-
satisfaction of prerequisites, cognitive strategies, and any other

tendencies, practices, memories, etc., of that particular learner that

affects his or her achievement of that particular objective.

Individual learners' motivation levels may be affected by:

1. The learning content.

2. The information format.

3. The sequencing of instruction.

4. Personal attitudes and values.

Individual learners vary in:

1. The learning sources they prefer and from which they learn best.

2. The learning conditions under which they are most comfortable and

effective.

3. Some aspects of information processing style.

4. Other personal abilities and characteristics.

See also: CONDITIONS OF LEARNING

PRESENT CONDITION OF LEARNING

COGNITIVE STRATEGIES

EXTERNAL CONDITIONS OF LEARNING

JOB ANALYSIS. The process of describing all of the performance elements, or

tasks, that make up a job.

Entails finding out exactly what job performers do when they do the job,

the order in which they do it, and the level of skill or performance

deemed adequate for the job.

Job analytic procedures:

1. Initial development of a tentative task list from existing written

doctriae, expert panel, analysis of similar jobs, and/or joint effort

with developers of a new human-machine system for which the job is

required.

2. Authentication of the tentative task list using actual job incumbents.

3. Validation of the task list using other job incumbents.

4. Collection of task conditions; cues that initiate and guide the task

performance; and standards representing adequate task performance.

Job analytic products:

1. A validated list of the tasks comprising the job under study.

2. Itemization of the conditions under which each task is performed.

3. Specification of the clues for initiating the tasks.

114-
4. Specification of the standards of performance of the tasks.

5. Listing of the elements of each task.

JOB PERFORMANCE MEASURE. A job performance measure is a test of the

adequacy of a job incumbent's work under actual job application

conditions. It is an on-the-job assessment of how well the job is done.

A job performance measure must be established for each task that is

selected for training. The job performance measure should be the best

available indicator of how well the task is performed.

The job performance measure must be validated under field conditions.

To be valid, when it is administered to job incumbents, it must

distinguish those who can perform the task satisfactorily from those who

can't.

Job performance measures form the basis for developing performance

objectives.

See also: JOB ANALYSIS

TASK ANALYSIS

PERFORMANCE OBJECTIVE

PREDICTIVE VALIDITY OF TRAINING

LEARNER. A learner is the key operator of a training system, and the

principal operator of the training Delivery Subsystem. The learner is the

person who carries out and experiences planned sensory activities to

achieve intended behavioral effects.

Common synonyms for "learner" include "student" and "trainee.1*

See also: LEARNING

TRAINING SYSTEM

DELIVERY SUBSYSTEM

FACILITATOR

LEARNING. Learning is any activity involving the senses that affects

behavior.

Learning is the principal function of a training system.

In a training system, desired human behavioral effects are achieved

through the conduct of planned human sensory activities.

See also: LEARNER

LEARNING-HELPING

TRAINING SYSTEM

LEARNING ACHIEVEMENT. A learner's learning achievement is how well he or

she performs specific performance objectives of interest.

•115-
Learning achievement is performance evaluated with regard to its

adequacy.

Synonym: ACHIEVEMENT

LEARNING DOMAIN TAXONOMY MODELS. Each learning domain taxonomy

model consists of a set of categories into which performance objectives,

enabling objectives, and prerequisites may be grouped. In each taxonomy

model, the categories are mutually exclusive and jointly exhaustive, i.e.,

every objective and prerequisite belongs to one and only one category of

each taxonomy model. In every model, the categories relate in some

fashion to mental, physical and attitudinal actions/behaviors.

The three-category taxonomy model (after Bloom, et al.)*.

1. Cognitive domain.

2. Affective domain.

3. Psychomotor domain.

The four-category taxonomy model (Branson, et al.):

1. Information domain .

2. Mental skills domain.

3. Physical skills domain.

4. Attitudinal domain.

The five-category taxonomy model (Gagne, et al.):

1. Information learning domain.

2. Intellectual skills domain.

3. Cognitive strategies domain.

4. Motor skills domain.

5. Attitudinal domain.

See also: DOMAINS OF LEARNING

LEARNING-HELPING. Learning-helping is the secondary, supportive function

of a training system.

Learning-helping is any activity undertaken to provide an efficient

learning environment for the learners.

All training system personnel contribute to the learning-helping function.

See also: LEARNING

TRAINING SYSTEM

LEARNING HIERARCHY. The learning hierarchy of a given objective is an

arrangement of its essential prerequisites in a format identifying the

sequencing of instruction for that objective.

See also: SEQUENCING OBJECTIVES

116-
LEARNING OBJECTIVE.

See: PERFORMANCE OBJECTIVE

LEARNING STYLE. A particular learner's learning style is that type of

sensory activity that, typically, is the best way for him or her to learn

(examples: reading, listening, audio-visuals, etc.).

See also: INTERNAL CONDITIONS OF LEARNING

LEARNING TASK ANALYSIS. A component of task analysis, learning task

analysis is applied tn a performance objective and to its mental operations

to reveal the prerequisites of learning and the desirable sequence of

learning events.

The focus of learning task analysis is on the prerequisites for learning

the task/performance objective.

learning task analysis pertains both to the present conditions for

learning and to the prior effects of learning.

See also: TASK ANALYSIS

INFORMATION PROCESSING ANALYSIS

TASK CLASSIFICATION

LESSON PLAN. A lesson plan is a written outline of the instruction to be

conducted to help the learner achieve a particular performance objective

or enabling objective.

Steps in lesson planning:

1. Identify the perfor: lance objective or enabling objective for which

the lesson plan is to be written.

2. List the desired instructional events.

3. Select the ideal media.

4. Select the materials and activities.

5. Analyze the materials to identify the events they supply.

6. Plan other means of implementing the remaining events.

See also: STRATEGY OF INSTRUCTION

TACTICS OF INSTRUCTION

INSTRUCTIONAL EVENTS

LEVELS OF OBJECTIVES. The levels of objectives form a hierarchical

arrangement beginning with the most general definition of the ultimate

intended outcomes of learning, and proceeding to the intended

achievements associated with the smallest units of instruction.

Seven levels of objectives are defined:

1. Needs.

2. Goals.

3. Life-long objectives.

-117-
4. End-of-course objectives.

5. Units of objectives.

6. Performance objectives.

7. Enabling objectives.

See nlso: NEED

GOAL

PERFORMANCE OBJECTIVE

ENABLING OBJECTIVE

LEVELS OF SYSTEM DESCRIPTION. The levels of human-machine system

description form one dimension for organizing the system's work

requirements. The levels of description address the work, itself, why

the work is required, and how its requirements are to be met.

The levels of system description include three hierarchically related

categories:

1. Objectives level

(What work is required, and what impediments the work faces.)

2. Functional Purposes level

(Why each Objectives level requirement exists, and the

impediments that apply to satisfying each purpose.)

3. Characteristics level

(How each Functional Purposes level requirement will be met,

and the impediments that apply to carrying out each

Characteristics step.)

The levels of system description form one dimension of the Systems

Taxonomy Model. They interact with another dimension, viz., aspects of

performance, to form fifteen cells or sub-taxonomies within which any

system's work requirements can be exhaustively Identified and organized.

See also: SYSTEMS TAXONOMY MODEL

ASPECTS OF PERFORMANCE

OBJECTIVES-LEVEL TAXA

FUNCTIONAL PURPOSES-LEVEL TAXA

CHARACTERISTICS-LEVEL TAXA

LOGISTICS SUBSYSTEM. The Logistics Subsystem is one of the six generic,

functionally oriented subsystems of any training system. Logistics

Subsystem deals with maintenance of facilities and equipment; housing,

feeding and recreation of training system personnel; replenishment of

consumables; transportation of people and materials; general

housekeeping; etc.

Logistics contributes to the learning-helping function by attending to the

myriad of details necessary to keep the training system running smoothly

and free of discomfort/distraction.

-118-
The principal operator or staff member of the Logistics Subsystem is the

logistics supporter.

See also: TRAINING SYSTEM

LEARNING

LEARNING-HELPING

MEASURE. A measure is a judgment or appraisal of some attribute of a

system.

A measure is a test or assessment of whether the attribute is "good

enough" to meet the demands of the performance requirement or taxon to

which the attribute is linked.

Every measure contributes an important bit of information about the

system's satisfaction of *he performance taxon of interest.

See also: ATTRIBUTE

TAXON

DESIGN SPECIFICATION

MEASUREMENT.

See: EVALUATION

MEASUREMENT APPLICATION.

See: MEASUREMENT PURPOSE

MEASUREMENT PURPOSE. The measurement purpose for a particularapplication of the Analytic Process Model to some system is the specific

work of that system that is to be evaluated in the application.

A measurement purpose is equivalent to the particular sub-taxonomy of

system performance requirements for which measures are to be selected

and applied.

Synonym; MEASUREMENT APPLICATION

See also: TAXONOMY

MEASURE

MENTAL SKILLS DOMAIN OBJECTIVES. Performance objectives belonging to

the mental skills domain require that the learner demonstrate the ability

to identify, classify, use rules, or solve problems involving thinking,

creating and analyzing.

See also: PERFORMANCE OBJECTIVES

LEARNING DOMAIN TAXONOMY MODELS

PHYSICAL SKILLS DOMAIN OBJECTIVES

INFORMATION DOMAIN OBJECTIVES

ATTITUDINAL DOMAIN OBJECTIVES

-119-
NEED. A need is a discrepancy or gap between the way things are and the

way things ought to be.

A need exists, for example, when a person who cannot presently perform

a particular job wishes or is expected to perform that job.

See also: NEEDS ASSESSMENT

TRAINING NEEDS ASSESSMENT

GOAL

NEEDS ASSESSMENT. A systematic process for determining performance goals,

identifying discrepancies between goals and the current status of

performance, and establishing priorities for action.

Tt should not be assumed that all needs (performance discrepancies) can

be solved through training. Some arise due to external conditions.

Needs assessment enables the curriculum developers to . focus on

high-priority training needs.

See also: NEED

GOAL

TRAINING NEEDS ASSESSMENT

PROBLEM ANALYSIS

OBJECTIVES-LEVEL TAX A. Objectives-level taxa are statements of what

performance is required of a system, without reference to why the

performance is required or how it is required to be performed.

Objectives-level taxa include definitions of the basic capabilities,

activities, and outputs expected of the system and of the relevant

environmental and constraining factors that apply to those basic

capabilities, activities and outputs.

Objectives-level taxa consist of the system's performance potentialities,

processes, products, environment and constraints viewed at the topmost

level of system description. It is a view of work requirements as objects

for study in and of themselves.

See also: TAXON

FUNCTIONAL PURPOSES-LEVEL TAXA

CHARACTERISTICS-LEVEL TAXA

LEVELS OF SYSTEM DESCRIPTION

ASPECTS OF PERFORMANCE

OBJECTIVE-REFERENCED TEST. A test designed to determine a learner's

achievement of a specified performance objective, enabling objective, or

prerequisite.

An objective-referenced test must be keyed to the statement of the

objective, and especially to the capability verb and the action verb in the

statement.

-120-
The capability verb indicates the domain of learning to be addressed by

the test. The action verb indicates the exact behavior that the test must

elicit.

See also: ENTRY TEST

PRETEST

POSTTEST

PEER TUTORING. Peer tutoring is a training procedure in which one learner

first observes, then practices and learns a task, and then demonstrates

the task to another learner.

PERFORMANCE CONSTRAINTS. A system's performance constraints are the

artificially imposed factors governing the circumstances and situations of

the system's operation that may affect whether and how the system

satisfies its work requirements.

Constraints include all man-made impediments to the system's potential,

process or product performance.

See also: ASPECTS OF PERFORMANCE

PERFORMANCE ENVIRONMENT

PERFORMANCE POTENTIALITIES

PERFORMANCE PROCESSES

PERFORMANCE PRODUCTS

LEVELS OF SYSTEM DESCRIPTION

PERFORMANCE ENVIRONMENT. A system's performance environment is the set

of naturally occurring factors associated with or incidental to the

circumstances and situations of the system's operation that may affect the

system's satisfaction of its work requirements.

The environment includes all natural impediments to the system's

potential, process, or product performance.

See ilso: ASPECTS OF PERFORMANCE

PERFORMANCE CONSTRAINTS

PERFORMANCE POTENTIALITIES

PERFORMANCE PROCESSES

PERFORMANCE PRODUCTS

LEVELS OF SYSTEM DESCRIPTION

PERFORMANCE OBJECTIVE. Performance objectives are written explicit

descriptions of specific terminal behaviors or instructional outcomes

required of learners to signify successful completion of their study.

Performance objectives are statements of what the learner will be able to

do, or how the learner will be expected to behave, after completing

instruction.

Performance objectives identify the end products of instruction in terms

of observable, measurable behavior.

-121-
The objective is a specific description of the action the learner is to

exhibit after training, the conditions under which the action will take

place, and the standard or criterion which must be reached for

satisfactory performance.

The objective may state a directly observable action, or an observable

product resulting from the action.

Performance objectives must clarify, in unambiguous terms, the behavioral

competencies to be developed by the learner.

Potential errors/defects that may exist in statements of performance

objectives:

1. They may refer to learners' mental processes rather than to

observable actions or behaviors.

2. The performance standard may be missing or deficient.

3. The conditions may not be stated or may be poorly specified.

4. The objective may state what the facilitator does rather than what

the learner will do.

5. The stated behavior may be too broad to describe specific intended

outcomes adequately.

Synonyms; LEARNING OBJECTIVE

BEHAVIORAL OBJECTIVE

See also: PERFORMANCE OBJECTIVE MODELS

ENABLING OBJECTIVE

LEVELS OF OBJECTIVES

LESSON PLAN

TASK

PERFORMANCE OBJECTIVE MODELS. The three-component model (after

Branson, Mager, et al.):

1. Statement of the action involved in the performance.

2. Statement of the conditions under which the action will take place.

3. Statement of how well the action must be performed.

The five-component model (after Gagne, Briggs, et al.):

1. Action—something observable that the learner will be doing.

2. Object—something concrete that the learner will produce via the

action.

3. Situation—the "givens."

122-
4. Tools and constraints—applicable to the user's implementation of the

action.

5. The learning capability—that the action/object indicates has been

acquired.

See also: PERFORMANCE OBJECTIVE

ENABLING OBJECTIVE

PERFORMANCE POTENTIALITIES. A system's performance potentialities are

those taxa of required performance that define the basic, inherent

capabilities that the system is to possess.

See also: TAXON

PERFORMANCE PROCESSES

PERFORMANCE PRODUCTS

ASPECTS OF PERFORMANCE

LEVELS OF SYSTEM DESCRIPTION

PERFORMANCE PROCESSES. A system's performance processes are those taxa

of required performance that define the activities the system is to carry

out, or the methods/techniques the system is to employ.

See also: TAXON

PERFORMANCE POTENTIALITIES

PERFORMANCE PRODUCTS

ASPECTS OF PERFORMANCE

LEVELS OF SYSTEM DESCRIPTION

PERFORMANCE PRODUCTS. A system's performance products are those taxa

of required performance that define the goods, services or other output

that the system is to deliver.

See also: TAXON

PERFORMANCE POTENTIALITIES

PERFORMANCE PROCESSES

ASPECTS OF PERFORMANCE

LEVELS OF SYSTEM DESCRIPTION

PHYSICAL SKILLS DOMAIN OBJECTIVES. Performance objectives belonging to

the physical skills domain require that the learner demonstrate the ability

to carry out some physical, manipulative action.

See also: PERFORMANCE OBJECTIVES

LEARNING DOMAIN TAXONOMY MODELS

INFORMATION DOMAIN OBJECTIVES

MENTAL SKILLS DOMAIN OBJECTIVES

ATTITUDINAL DOMAIN OBJECTIVES

PILOT TESTING. Pilot testing is the delivery of a newly developed program

of instruction, in the actual intended operational environment, for the

purpose of evaluating and improving the program.

123-
Synonym: FIELD TRIAL

See also: FORMATIVE EVALUATION

FEEDBACK

POSTTEST. A posttest is a test of a performance objective or enabling

objective, administered subsequent to delivery of prescribed training for

that objective.

Posttests provide the basis for determining whether particular learners

require additional training, or whether they can be certified as having

achieved a given set of performance objectives.

See also: OBJECTIVE-REFERENCED TEST

ENTRY TEST

PRETEST

PREDICTIVE VALIDITY OF TRAINING. The predictive validity of the training

undertaken to pursue a particular performance objective is the correlation

between a learner's posttest score on that objective and the learner's

subsequent job performance measure on the corresponding task.

See also: POSTTEST

JOB PERFORMANCE MEASURE

PERFORMANCE OBJECTIVE

PREREQUISITE. A prerequisite is a performance objective which candidate

learners are expected to have achieved prior to the prescribed training.

See also: PERFORMANCE OBJECTIVE

ESSENTIAL PREREQUISITE

SUPPORTIVE PREREQUISITE

ENTRY TEST

PRESENT CONDITION OF LEARNING. The present condition of learning is

that which the learner must recall to make achievement of the particular

performance objective possible.

The present condition of learning is revealed by Learning Task Analysis.

See also: LEARNING TASK ANALYSIS

INTERNAL CONDITIONS OF LEARNING

PRETEST. A pretest is a test of a performance objective or enabling

objective, administered prior to delivery of the prescribed training for

that objective.

Pretests provide the basis for determining whether particular learners

should "skip" blocks of instruction corresponding to objectives already

achieved.

See also: OBJECTIVE-REFERENCED TEST

ENTRY TEST

POSTTEST

-124-
PROBLEM ANALYSIS. Given evidence of a failure or discrepancy in

performance, problem analysis is conducted to determine whether the

problem/need should be solved by means of training, equipment,

administrative action, or some other method.

See also: NEEDS ASSESSMENT

PROMPTING. Prompting is providing direct guidance to a learner by

supplying a specific fact, procedure, word, phrase, etc., that the

learner is attempting to recall.

See also: GENERAL LEARNING GUIDELINES

CUE

PURPOSES OF AN INSTRUCTIONAL UNIT. Identification of the purposes of

an instructional unit provides a first step toward defining the

performance objectives of that unit.

The stated instructional purposes must:

1. Identify what learners will do after or as a result of the instruction,

not what they will do during the instruction.

2. Identify current or immediate outcomes expected of the instruction,

not long-range, future outcomes.

REMEDIAL TRAINING. The remedial training conducted in support of a

particular program of instruction is the training given to certain

candidate learners to help them achieve performance objectives that are

considered prerequisites for the program of instruction.

See also: ENTRY TEST

ESSENTIAL PREREQUISITE

SUPPORTIVE PREREQUISITE

SEQUENCING INSTRUCTION.

See: SEQUENCING OBJECTIVES

SEQUENCING OBJECTIVES. Sequencing objectives is the process of

determining the order in which performance objectives and enabling

objectives will be pursued during training. Proper sequencing demands

that objectives be pursued in a logical "stepping-stone" fashion.

Specifically, an objective should not be pursued until those that are

essential or helpful for its pursuit have first been achieved.

In sequencing objectives for pursuit during training:

1. Place dependent objectives after those on which they depend.

2. Of two supportive objectives, the one that is easier/cheaper to

achieve should be placed first.

•125-
Possible errors that may occur in sequencing objectives:

1. A supportive relationship could be misinterpreted as a dependent

relationship, and result in increased training time and expense.

2. A dependent relationship could be misinterpreted as independent,

and result in a very difficult-to-achieve sequence.

3. A supportive relationship might not be recognized, and result in

inefficient instruction.

Synonym; SEQUENCING INSTRUCTION

See also: DEPENDENT OBJECTIVE

SUPPORTIVE OBJECTIVES

STIMULUS OF A TASK. The stimulus of a task is that which "triggers" the

performance of the task by the job performer.

STIMULUS OF INFORMATION PRESENTATION. The stimulus for a particular

block of information is the form in which that block is presented to the

learner.

STM.

See: SYSTEMS TAXONOMY MODEL

STRATEGY OF INSTRUCTION. The strategy of instruction for a given

performance objective consists of 1) determining the sequence of pursuit

of enabling objectives for that performance objective, and 2) choosing how

to implement instructional events for each enabling objective.

See also: LESSON PLAN

TACTICS OF INSTRUCTION

SUBSYSTEM. A subsystem of a particular human-machine system of interest

is, itself, a human-machine system that is subservient to the system of

interest in the work it performs.

Subsystems may be totally contained within the system of interest.

Subsystems are members of the system hierarchy of the system of

interest.

See also: HUMAN-MACHINE SYSTEM

SYSTEM HIERARCHY

SUMMATIVE EVALUATION. Summative evaluation is the process of gathering,

combining and interpreting data to measure the effectiveness of a training

system operating at full-scale delivery.

-126-
Summative evaluation occurs after all formative evaluation has been

completed.

See also: FORMATIVE EVALUATION

SUPPORTIVE OBJECTIVES. Two performance objectives, or two enabling

objectives, are supportive of one another when achievement of one

facilitates achievement of the other, regardless of the order of

achievement.

If two objectives are supportive of one another, the one that is

easier/cheaper to achieve should be pursued first, during training.

See also: DEPENDENT OBJECTIVE

INDEPENDENT OBJECTIVES

PERFORMANCE OBJECTIVE

SEQUENCING OBJECTIVES

SUPPORTIVE PREREQUISITE. A supportive prerequisite for a particular

performance objective or enabling objective is a performance capability

that is not actually incorporated within the objective, but which makes it

easier to learn/achieve the objective.

Learners who possess the supportive prerequisite will usually achieve the

objective more readily than will learners who do not possess it.

See also: PERFORMANCE OBJECTIVE

ENABLING OBJECTIVE

SUPPORTIVE OBJECTIVES

DEVELOPMENT OF OBJECTIVES

ESSENTIAL PREREQUISITE

SUPRASYSTEM. A suprasystem of a particular human-machine system of

interest is, itself, a human-machine system to which the system of

interest is subservient in the work it performs.

The system of interest may be totally contained within the suprasystem.

Suprasystems are members of the system hierarchy of the system of

interest.

See also: HUMAN-MACHINE SYSTEM

SYSTEM HIERARCHY

SYSTEM.

See: HUMAN-MACHINE SYSTEM

SYSTEM HIERARCHY. The system hierarchy surrounding a particular

human-machine system of interest is the network of other systems with

which the system of interest interacts in the performance of its work.

•127-
System hierarchy is one dimension of the Systems Taxonomy Model.

Sec also: HUMAN-MACHINE SYSTEM

COLLATERAL SYSTEM

SUBSYSTEM

SUPRASYSTEM

SYSTEMS TAXONOMY MODEL

SYSTEMS TAXONOMY MODEL. The Systems Taxonomy Model Is a conceptual

framework and a set of analytic procedures for identifying the

performance requirements (or taxa) associated with a system of interest.

The Systems Taxonomy Model is the first stage of the Analytic Process

Model.

The Systems Taxonomy Model identifies and organizes the performance

taxa of a system of interest along three dimensions:

1. System Hierarchy—This dimension insures that work requirements

will be identified by studying the system of interest, itself, as well

as by studying the work requirements imposed upon the system of

interest by other, interacting systems that are linked to it

hierarchically.

2. Aspects of Performance—This dimension insures that the identified

work requirements will include not only what the system is to

produce, but also the processes it is to carry out, the capabilities it

is to possess, and the circumstances under which it is to operate.

3. Levels of System Description—This dimension insures that the work

requirements will be identified not only in terms of what is expected,

but also why it is needed and how it is supposed to be

accomplished.

Synonym: STM

See also: ANALYTIC PROCESS MODEL

TAXON

TAXONOMY

ASPECTS OF PERFORMANCE

LEVELS OF SYSTEM DESCRIPTION

SYSTEM HIERARCHY

TACTICS OF INSTRUCTION. Tactics of instruction are the specific details or

prescription? for the instructional events in a lesson plan, with special

reference to how the needed conditions of learning are incoprorated into

the events.

See also: LESSON PLAN

STRATEGY OF INSTRUCTION

CONDITIONS OF LEARNING

128-
TASK. A task is an element of a job. A task can be described In terms of a

discrete, observable action carried out by the job performer. Every task

must be addressed in the training provided for the job. Any given task

may be represented by a performance objective, an enabling objective, or

a prerequisite.

See also: JOB ANALYSIS

TASK ANALYSIS

PERFORMANCE OBJECTIVE

TASK ANALYSIS. The analysis of an individual task, or performance

objective, to provide a basis for determining the appropriate

teaching-learning strategy for that task/objective.

Task analysis has three components:

1. Information processing analysis.

2. Task classification.

3. Learning task analysis.

Task analytic products:

1. information-processing diagram showing the mental operations

Involved in each performance objective.

2. Classification of each objective into the appropriate class of learning

outcome, or domain of learning.

3. Specification of essential and supportive prerequisites for each

performance objective.

Synonym; ANALYSIS OF PERFORMANCE OBJECTIVES

TASK CLASSIFICATION. A component of Task Analysis, task classification

categorizes a performance objective as to its domain of learning.

Task classification provides a means of identifying the necessary

conditions of learning that are to be included in instructional events.

See also: TASK ANALYSIS

INFORMATION PROCESSING ANALYSIS

LEARNING TASK ANALYSIS

LEARNING DOMAIN TAXONOMY MODELS

TAXON. A particular category or class pertaining to some thing or concept of

interest.

A taxon is an element of a taxonomy.

A system performance taxon is a particular type or category of

performance that is required of a system of interest.

-129-
See also: TAXONOMY

SYSTEMS TAXONOMY MODEL

ASPECTS OF PERFORMANCE

LEVELS OF SYSTEM DESCRIPTION

TAXONOMY. (When written with an initial capital "T"): The science or

technique of classification, as in the technique ot classifying and

organizing the performance requirements of a human-machine system.

(When written with a lower case "t"): A particular classification of

things or concepts, as in a classification and organization of a particular

system's performance requirements.

See also: TAXON

SYSTEMS TAXONOMY MODEL

TEST.

See: OBJECTIVE-REFERENCED TEST

TEST DISTORTION. Test distortion is a defect in a test resulting from the

fact that the test can be answered/performed successfully by applying a

domain of learning and/or action other than that intended to be tested.

See also: OBJECTIVE-REFERENCED TEST

TEST RELIABILITY

TEST VALIDITY

TEST RELIABILITY. The reliability of a test is assessed by examining

whether the test requires the learner to perform the action verb long

enough/often enough to permit a dependable inference that the objective

has been demonstrated.

See also: OBJECTIVE-REFERENCED TEST

ACTION VERB

TEST DISTORTION

TEST VALIDITY

TEST VALIDITY. The validity of a test is assessed by comparing the test

with the objective it is intended to measure. The action, situation,

standards, etc., of the test must conform exactly to those of the

objective.

See also: OBJECTIVE-REFERENCED TEST

PERFORMANCE OBJECTIVE

TEST DISTORTION

TEST RELIABILITY

TRAINING NEEDS ASSESSMENT. Training needs assessment is a process for

determining what shuald be taught/learned. The process is based on the

active involvement of those directly affected by the training system

(e.g., the learners), in addition to professional educators.

•130-
Training needs assessment consists of:

1. Analysis of the job to be trained.

2. Selection of fhe tasks to be trained.

3. Construction of job performance measures for the selected tasks.

4. Analysis of existing courses.

5. Selection of the instructional setting for each task to be trained.

See also: AI ALYSIS

NEED

NEEDS ASSESSMENT

TRAINING SYSTEM. A training system is a particular type of human-machine

system, intended to accomplish two major functions: learning, and

learning-helping.

Training systems have six types of operators, or staff members:

learners, facilitators, administrators, curriculum developers, facilities

developers, and logistic supporters.

Training systems have six functionally oriented subsystems, each of which

is operated principally by one of the types of operators:

Command (training administrator)

Design (curriculum developer)

Emplacement (facilities developer)

Logistics (logistics supporter)

Enabling (facilitator)

Delivery (learner)

See also: LEARNING

LEARNING-HELPING

COMNiAND SUBSYSTEM

DESIGN SllESYSTEM

EMPLACEMENT SUBSYSTEM

LOGISTICS SUBSYSTEM

ENABLING SUBSYSTEM

DELIVERY SUBSYSTEM

HUMAN-MACHINE SUBSYSTEM

TRAIT. A trait is a qualitative or quantitative property (aptitude, expertise,

strength, reliability, etc.) possessed by a system component (person,

equipment or procedure) which is necessary in order to achieve or satisfy

a performance taxon of interest. The presence of a trait is determined

by one or more concrete, observable attributes.

-131-
\PPENDIX A

Training Design Subsystem Data Base

A-l
in

o

tn

5

E 5

II t

4) tn

« c

><

••<

e

I e

£

E

3 ^ •

(0 a

u

o •w

C 1

9 O»

4» -H M e

u i

0 1

a« o>

e -^

ana ••4 li.e

ICO

fftt K- C i-J

HO e c

x ♦♦£•5

(- »c $»&

000 *z

IN-H 0|^

1 4> Hi £

3>.0 u •

Wt t

"4>

04» EE«

HSU II X

0fc3 4)4)«

CfiH M£

•H |H VXd

LlLM 0)0)0)

a ^« c 11 ti at

o -••-»»••

e> o • «1 »

X) ?> «- •>« £

CT> O •» U- •»•*•> .O •( -^ U

c -^ -o C O « «lltO£>

OC -o 0» M • » "O • > u • o>

««•** IIHCW*^ tl Or-« C

> O -< T3 «». H O • > -O 11

2i- e c ^ w o ft «fe »ox ^

0.0« SMCSI* oi > u * f« * i kit

•HfHC •*«( -HtlO u -o e ft U •-4-H-p

5<^oc ft o> £ 01 0» o« 0» a -o :> o S 9 oy«

•-1 » o «V-HC « • •-< em "?H

0» c » x>«*«*o^E4*o*i c 1 01 e •* u o A

0< "O H U. -rt O C T3 o > ••< v w>ev n o

C ••* C CD Oi-n-t'-OOAft^CUC O » > 0» C fc 0<V

-« H^ »CQCOI^O«^ * ••»« c 01 n o> ti 0» c m

CCOIJL o-ou.4* ••»»5i'-'wt

rt

c » i-^oi

— 0» * « eoftiAsoo-H >>o»Buc*»- c «- o»

e ceoo> o> a« 01 e **- **»o»Oi^^o»o» «••»««» ft

ft u-HU.-ne>e>o c f -c o e w >» xoaft > '

^4 *»-i>.cCL»UI*»0-»4O*»ftftOfc'>.u U ft •WO>

JO ft0MV->4HW«*CftpU •** *» O *- O » CO » C Tl

0 ~4U H C C cftQ.»-*»l«*>C^.*0 ^H •> ft C 0*ft

a. Ä»HX*>«--^«-HOi«voi3ft — -»«o^wvftofto i-i >. -o

0 0i*.<«-co00aS**«*ei9 o» m o > ft i «* • ^ f

U«^4-'*Hftft(.«*ft> ft O ft « U ft ft > » »- •- •« ft

•( •1rt«>-0"-»>0>ft-4-O^BUft'**'-<ftUIII>10ftO U *»

• H0OC-H «) «, .^ ^ ft-^<**ofteft^ *o ^ a. o ft c

JO »» > 3 ft ««- x M£ao-o9oi> c ft -P -^ «ft o. «

•x o o-v o o t- a* £ u *• c o •»* -^ 01 v ycom-^u^if

«>k -^ * ft £ U O 9 • « »^OO'ftftCft^COK«

0 o c -o ft c *> o c»<**ft ^ ^ ¥•« P K 9! ff 1 fcS 9HOCO-OO.C Q. ^ C » O 3 X 0 •»« •.UU>*»XO«/»

ib-^o^ftos o»^ ove--0*» (-ke»9 e-'jt^vo

t0ft£00 -^ 0-Mjft-«0c0*»ft-^ c a, o a 'H w

o • u u u £e0ft«*i-**«*v^0ui>«*ft'Hft^fl-r<->4ft

«*et-eoO'HUHft0c>9'^o«*ft0ft0>e<«*Cfti«w-c

0OOSO^OO<*>O.0%fti9i0U<^«*-^«*0U(- o o v

00c(>o.0O •0*»^iiiÄ*»«tO0et^ft«t'i»-»<fto-^_«jj

«-t-rlfta •»»«*. U U<-lO«b O O T» Ä o o o o

ft O ftO0 0(-0> b09 b o> »- O» * > O" >

Ä 5 ««- X ft ft OCbO 0-pOC0OC0OO' o ? *

Po^oft-oufftv-novotft^v^-^^^-HV^evo1 c >

ft*.«*«« K C £ c ^o c c c c -MCC o>-^ c

•• 0> ft <^ C 9 UIÄ^IÄ^-PftCKt^fttfl^ftÄCHHC^ft

ftHOoo«ncQ.o-*»o.oxo«ft.*»ot.fto».ftOft«»-0«-'-.ft

t4X U > ^« ft <-• Wl •-• C X <-• -< C «-• Cn 0» -• ^ ft rH -H o •• • ft 9 —< >-•

«* v to tn rL ac e«a. SH-O. 0Hiij«-ia.*»tiia.«»uia. 0 •» ui 0 x> UJ

•*•** -«4 -o C * ft ft -r« c « e o

M «> • •• . c*. • . »f • »co • • • «a • «a • • o ft <:•-• > •

ocooooftoo 00 > o o o eo 00 "o 00

•»«»••••©••••••«•••••••••••^••UJ«

» -o «-• r* ro <r ^NOICMO nMno»«oio*-iMO oe <-•

ft 00000000«i«iOOOOOO«i«40rsilNOOOOOO

ooooooo«4«4«4*4»4r*o(MNr4«4r*r4r*n(Nmo»4r40*>

o«^«iV4«^o«4v4«H«i«i«40vi«i«4r><v4«4fM«i«4r>toronov4

• • • • •• • • •••••

• • • • • •

• •

o

A-2
-5

mE

o

«*•

• «

a >

a »*

o

e v

••< o

II

e o

u o

> o>

o (9 y

:>

u. | 1 •^ »1

A a,

-«

t- tr « u

O c w » 9 <•> •H "4 4* tl

-ss

*» u

• • -^

SSf

4* w o

« 9 V

>.<-< «.

w u •»

« a.

9 «

to v o>

« c

e A -^

o> - -

•8

m

a.

?l

s

o> c c

w -rt 11

x <- e

<*• o ti ^

•»«••«« o

W > 9 <- (- «* we v

O »v O * t .

<• >. IF« 01 O C9

_« "g U Ä • >

« U ft U-

.4 (. >. o x -o •-•

•i o o » 'H • a»

e <- I-I o."o

JO • O. O CO'

•rt • X n • C

• f «l O >P •»<

c «> «« e «*

V O tl 0 -M O

O»««. £ > X K»

e o o «> o>

vl v * o *-

jr v or> « ^ o

• > -o c u o «»-

•*-*& •-> ti e

-* I-« -o >. ' •« *>

A -rt C C « »

o ä o» <-< e *- o

o

— *> HO

A «I O M «»

e e e

» >-* c •

a VH tH <»• i- > o «- o o

o o x o o fp

o>^ -^ V

« c w o> > e

C -x •-< C CO«

O (- O O H £ v>

•~> S O —* •*> 9> c

i- S

• o

u e

g^

:5o

> o >

IH i^ O

•*' -M x:

few

0 0

• -H **

U 4*

c o -o

k o •*<

«»-coO.V

O 19

c • u. •t* a,i-4

x: o «

Si .

9 «^

*: «

J-Ä O

•*> o •*

0 •«

X-0 -«

V 0 0

• *&

a» ** "o

0 0 C

«* e 0

0 0

94* •*«

O 0 0

b. W V

2 0«^

« h- o

?

C 4» ^

«- u 0

> 0 0

, JO O .fi 0

0 9 o ia

4» o- •»<

W 0 »-

0 <*- U U

<*> C 0

o q o 0

v •*> 0 -o

0 «-

-< o ••

0 c«». ..

O'H 0 0

0 O. H

«^ 0 •*

«*• O O ^

> «H U. 0

0 O I-I c

e o •-•

o-*» e> a.

•H C

0 o £ 5

o « *-

4* •>< «O

O C «

0 0«»

• o c

«f •» 4» -H

O IA Ut Ul

0 0 0

0 0 **

O « O 0

0 0 4*

•H 0

4»^ x: a

O 1-4 u 0

«- o o »

3 > a. «-•

u o a, o

o ** 0 0 v

0hi 0 C 0

0 ** U

0 0 U -»

0 > -?.0

> ft O O

•*« 4« O 0

4* U £ 0

W U 0 JC 0 >

e 0 o «t H ii

0 0 -» J3 •« 1 4>

'-t C X> O r-l A u

Q. O O JO O 0

0 r-< 0 4»-»

o

4*

0

0 -»^ 0

0 0 4* 0 «

^g

a u „ ,

«4- c i 0 e o

m o 0 -c o o> H v

Q. o o 4* c e

«t x (- «* «> -M -x T)

4*4*O0 UJCO0 0-M<*>0UO0(>-o

r4 k O ^4 •*< 4* e

Ot-*4 0O>M>H^ 0

e .a a, 4* >H o ^ t. «>

4 o 0 0 0 e

U AC CD U k4>4*->4

(. > 5 «- O 0

o u. -< _o <

0

0S

BrvA0 p«-*-.»*

0 0 V 0 O O 4*

0 09»- 0 0 «*• H •»«

0 6 0 0 0 > -4

C C -H «•- O CfO O »4

•* -9 JS O. O 0 C O

£0? P«-p00

0 5 rl 0 O. 0 i.

- O 0 e « 0

u o Q. 0 0 e e

e <*. 4* a -o u 4» e e

* o 0 fc 0 -r" a

-» 0 Q. O > O 0 O

U 0 P» «- » u

rt o> O IA V 0 *-

(^ e4* (> 0 0 -e

0 H x: -4 0

•o 4» -o i a 4> 8

"s o 0 e -i t* e

«. 4» o

o 0

«- O Q.«» O

•• a o «-

en

• •-•••en**«**

o oo o o o o —I

• • • • • • • ^> • •

CM O »4 CN O -4 CNJ 0v4O

CMW

4-1 V4 O

• » •

OO CM

*4 CM «4

S 0U«--»*Q.«-0U

u9O0O9aa0

SU "4 o. • 0 v.

4* 0 O "O "O O 0 (• S P 0- p«£ 4» O

Ij>g0004*'

M ae 0 M N N de . -

. .1

O 4* tA

•»4 C •I

4* 0 0

sss

0 0 U

0 e e

ill HI »H SS,?

CM W 4r -4 CM M 4» »4 CM M ^ CM

WtnWO^'*«'0,4»O»4»44HONCMO • ••••••in********

OOOTOOOO 4^ooo(MOon

«4 tMn ^ in 0

?*> M M W fJ

• • • * • «

o oo o o e

CM CM O W * • * •

o o n o

o o -4 -H o rJ CM oooooooooo

»^ CM^-^OOfMNtMOCMfMOfMfM

OOOOOOOOO

CMCMOCMtMCMfMO-4

.4 «4 «4 O

(M CM CM «^

•4*4 O

CM CM -4 CM CM CM CM CM .4

O

O »4

0

O

CM

CM

CM

A-3

- i

k

• .
"9 * c

* *

e -o

mm" -^

w fee JH-H -• o c «n

• C X> O V V W > ««- « C

ST3 10 W «*. Al C * « A«ue^ Wllill-H o «^ ^t.

e e ••ixitfio c c "e c e a. c u -o A > ?o 0.0

g

* « o OT>« cu o e * o o o •»» a c A o u. «4

c <-i «-< «to --< *•• c •»>••*• w c ^« o > o «- «-i m es

o a o. % o A -^ • a. a. '* *- m Q.«o*>o.-o«rt^rtu

JO-O-M*«« XI«*V<*- v -o <o o v «- «- • «o c -< S > *» e o> on*. e o> •< •» ti v «- w ao o.*» •• -o o< o o «n •*

oc-ou u ••< * c o -o * -o • •'««e 'o TJ ^ «- • o. « e «^ cjo-^c-or-to-t.

vc t. wo«* T) -o o ewe«* ttSSa o*»o«»*x»tooo*o» > o.

«»•«•«»•»a a.^ c •tctiuo»oQ.»-4c « **«> o>-« in ri c a n «> t> M c at tn

801 -o K a 0. * » • •^eBue •« > 0» 4* » c e o •»* 01 ot «Sii«-H 00 Ji 0t D> 3 r-. «> -^ «> C -^ C 3 r-« ^ 10 C x H 3 »^ C "O "O XI O > C

> »a v o> o o. -«H e0«*-f49Oo. 10 o -o c <«- a> o a. t- it •-> c o -^ o> x

c o A >> > 01 r • -o ** * m •> e • oc«*90t«u •o-o.eSovt- c «. •*4^ v u. T3 * •-< >. •»« co E-O-O-M».«« «* ioo>'Q»''«V'4»eO-*'<^>^ct.xi-4i_

O I-I c u «- ECMO 0» u «- 10« T9Uk «H • e 01 0« u o

»•o*>*i-aa«-oct.o u c ««. »- -o c *- o T>C>«>3C«-O «»<.»4V>b-ecu 8« o o -< * o •»> «ot-osSo«* «»•4*<^b.o>Hfio<*>o>ce 0^03» •O TJ V«*>a3U «C-»<0»«^-H3U (•Wt.i-lVuSU C-»««^l0C*><«-^3O

-»« C ft X O CT V «^ O ** CX UP" -O ft CkOk C O- "O rt f« u U tT-P

» ft -o > «) 01 ft o 01 t, ** ** 10 c ft o 01 a. • ott4ftQv««*«io«fti«*cft

u*»c^'oc<cio*»*> ft#>-o>vCH«o<**4' <». «^ (,. e * *•»» to c •»< o •»<-> c -< io »o

**«*-x(-«-i c 10 y w« voft'-iiociou 01 ft -«o.ciou*io e o ^ 10 c M

ft Cia-^a.cHftOci-cucao.^fto co.ioioa.ftHftO«9ia<*-A o. a.r4 «>

ociot-ftO'eft^ovc oc>O(0ft-Hooo>o*»o0 ft-4 0>>3oioo<«-oioft.«

• c ft L> £ «t H ^^ ^ g «». «. -HtH ü o "o fc ft « <-» «HM «* u * &

Z m O * «^-ou-^r-owo •»=>«»- fto»« 4oc>«ooev«>u-««ftoi0e««<«- ft u o

o c -^ A 10 3CAife ft u. «»CAftoii.*-«^« OCA«» ^> 1 «* .^ » c

<*- o «> o c e>^ 01 o u 5 > « a,»» o>oftou<**»-<o •«* o»-^ ft oun«.eo>cioo>0ftift

»- •»< o o c u -x t.uu.ftvoöc-»H^ t.ioa^(>ute<>'*4 *• » o « «o 1 o c •»« -^ *

ft«*ou-n#4euu-3 »-i a. * H «- u <^ 3 uo •»4o.u«t-3u«^o>v.o>Ä»<«-u«*.

au o*>*-.-H««.oft«>vXftio ou-rt <•• o ft «* o ft 10 0 •« «^ o ft * «o «o * a.* u.

a*. u to <*• o m u >iox:«oo«».oiouo«^i*£*».v«t>^io"s*»>so*ovo

«- o*o»4ioft*»ftcs.*s.*»*>-Ht.ft*»tic ^ » »•^aft'^»*"*c* ^feft«* 0» «^ o> •ok-Dtot-ooioc a a-e m i. 0 10 « e ft a« io(>Oft-**ft*>o>a SL-OIO

c o 10 c o> 0« ^ i > ft 0« ^ av i <- »> ft co ON oeiee OLV

c ftHc*^»cc*.ioe«ciooioc eCociefteft c e (- .a ft 0» S-M« O IO C

3 10 o.*> *40.ftft'<-i-HOftwft'<eLcft^->4 0vfiS->4$kev-^-^0 i'ni-oa.cft-^

a> ft x 0 c » •»* va.i'-icft-H-^ vuft'HCftwM «» ft ft 0« ft -H ft ^ .4

a.«3(.^«*r-iU^l0(.>.«iftn*>VU«l(0t>Oftft^4>VU«0« « ^ »H > • * v o 10

> X (^l0OCftftft«'-H «*.t0^Cft0lftVr-l tt.ft^Cftftft^ft'^ftOlO^Cft

U. «> »>> ft H ft -«I -M Q. ft »» ft Oft--«>*«0. ft »» ft Oft>M-<4<*4 O «- O ft H

*i c«-o—i.»>«'uu »- c -o <-••»< > u u »- c'o«-"H«*uo«<».rtM^a-«»<*»o M«*0(- 0efteeftO«*e e^fteefto^o o^Meeooo^tf o > «o c

e>ftb-*ift-*iift£ c><-^c-^ftft.c c>«-Kc«4ftftA -^ c •«•-•"« c ^ 01 •»•OftiO^»W»4-H«*«IOftO«*ftW-»«-*4-**«9ft04*ftW-«4-H « «« «i •> o ** ftlOX o>>Hft<*.eioeuu 4*><-t«*-e«euu -»«^^(«-cMcuuo^eiocs-ciocu ftftx ftftO-»<.-o>cftft ft«o»4.<o>cftft ftno-iH-H e ft « ft ft to o •-< 8-<«->lOtOcU«^«4.Cft^«-"«<»>ftU,

^«^Cft'-4«-lO*ftU«*.VO<ft«ftlO«lOftU«*- ftkft c«oocftftHift tcwccftftHftftfcc io c c • • c i«e «e« e e *

aac >-• a: o ui z *-• o o eftaci-ioujiHM6oeftflC*-«ouiiHMOa-M ftuj»-iuiouj»-i>-«o

^««»a »»»»«»-- 1.1 »»ft »«»» »c^ixl*»^»»» •••MUJ***a* •••

««otro «^ rM ro « v) «i ^M «« CM n ^ m ft «^ CM ^cMW^-ioi »< CM w •« CM r»

• *» ••• ♦♦»&»» •» » • »Ck» *»»•»* »»O ****** » • •

Ov^^MOCMCMCMCMCM O^xOMCMNCMCM 0^*^OCMtMMCMlM»-0*^««OCMCMtNfM

«iM^^N^^^^^oMCMCMCMrMCMCMNCMO^roncMnnmmm «^OOOCMOOOO

*««4«4«4«4v4v4«4»4«40CM«4«4CM'l««H«4«4 ^Or)*^«4m«4v4«i v4«400CMCMCMOCM CM CM CM

««CMCMCM««CMCMMCMCMCM«4C<4CM«4CMCMCMCM CMW'-tCMCM^fMCMCM CMCMOCMCMCMCMCMCM CMCMCM

f>|«4»4«4f>|«4v4«*v4v4v4(<4v4*4(<l|v4v4v4v4v4V*r>4v*<-4CM«,4v4«,*«,4«40>CM<»4»'l««CMv**4*-4''4 • • ••• «•• •• •

• • •

«4 ^4 CM

A-4

..
x si

«- •* * a

_.-•» M U »4

» e> o fi JD ft •

U. C •1 .4 0 u >

•^ •<- fc. U k -«4

~ * Q.f4 «^ 9 «•

« O *£ «*• Q U

'2 *- * 0 S *

ay -^ • "^

t v I • P- c ft c c

4* V u ^ o

* ^ c * *» c

« ft » ft o

"• •*« «* u u •

o «* « e f >-

•H e -«4 ii ft O

<*> ft « •*< -rt «^

c MI e u u ft.

ft « o ««•*< ft i ft u •*. «2 a.

u £ M a o 9

o -^ e

# f* H

r* -4 y-

« O «H

>» 9 0

^ o- C

o o c

^» -. e e o

^^ft« •^ o

•x o A at tf I I

ft 0» tl u

^ ft H ft A

A •** 5

_ C ^4 »- o

«* > « O "O

•M -o o * a»

o * » • -o

ft ** T) • *l C

a e « e ** ft ft

c i • • •»« »* »

ft i -H V ft ^ C

•i» ft «* e t* •**•**

C fc ^ •»« 9 tH

3 — • ft XI ft

f«i ft C o x

o ^ ft a.«*

o -* & u 1

F4 U ^^ «r »«H

iH w -F« H A

• JQ O W O

,ij:o55ft

ft V ft C »4 »

•»4 > ft » »4

o w -^ 2 ^4 m

5 .4 » ft 9 -^

« It 9 H ft or 3

o • ft PS? * ft Ü ft £ £ I C t

ft -^ ft

«* e -fi

•H *» >

r-l i-t O

•4 9«- Ä w o. 0

ft ft

ft > XI

•P (-

•r< ft O

ft c •*>

•*4 t.

3 O "O

O- ft ft

ft •-) -O

«- c

ft ft ft

«-•»» »

Q. 0 C

r-s

•4 C C

k u i

o 3

a o» ui

a e ft

9 -^ M

• * tft

«0 0

tfl ft

fe II* ft ft tf> •»

COO

e t»

9 > -H

o>« «

***

-« fM M * Irt N

• • • • • K

N CMM NN^

• • • • • a

•< MM CM M (MM N fM

o e o o • • • • • • •

«^ M O «< M O «^

O «- o

• a. •

CM «4

CMO»HfMr<(-4CN4fMt<JMCMO CM CM (N* CN CN CM CM CM fM

fM»^<^fM^«^^»4«^0

e o o

o o o

MM«

O O O O O -4

O O «4 «4 «4 «4

•4 O M N or» n o o

• N CM *< •* O* **

• • •

CM -4 «4

^4 «-I *H v4 rs

CM • • • • •

n «4 «« «« n

ro

A-5
i

11

» > -» 5 2

9 U ^ 4 O ti •*» 9t m

> O -5 J3 O X» "? > U-9 ft.

U • JD O • •» ö H tl ^ UJ

-J O O •-• Or> U O*» '^O -

,o -o «!-< »MAC u « tn

o »»-••-< o o p w won w

0> Q. rH JC * • «- -5 t*>-4 «eST-xm y x> v «- o A >«~4 «•

«. t. ^ «: tit. o ** O -rt f-i M a*

> 0>H|M«i-i-»>0ll «*> (• -»J ^ «e o>

b. •*• o o JO « ** o. «. ti • u-Hintn o

t- »i ». o ««-•-• u o »» tt a. «- ti^ti ^

•» «0 • •»> C O -•-• UWCp QL O "9«l»4> U

eo«tt « •»< cxa."»oü. o > • * o >* u ** * a.

set» 0) o>o w * £ >^ x» «- •-. r> u. •»< •• o •« u <

o>->«0 > c > w o a o. *> o a.» u. M wo. o +> in at o. «

»•Ok" -M-Mlk«! I HH tt-H V- C X "? -3 -H

» •» -rt * *>c*->>-» « i« ti «i «- c A -e »* o cfifxa «^ ä

b. «»-o > u «-« •« « e o -o JC o x -o tie eS •xia.oii a

»H » c " ti o •** o 3**£0IC-»<H*U. > m *»

A <? » -3 Oi X u -»"^V I»*» -H-H V ^»H (. » «( ^ MH -Q t.

>»UU «r1<*-tl A A •»« C O- »< <* -x-D »< « O «- U O > — O

•i i. * o-H'^Coo^-xfis-M-p**!» «*- c u c a.

v * «- -s <*- m A o m * £ > v at e> • c * «. ^ ai x

e c o -o m c o m o v vai<*- « »mm c > t> ^ « ^ «^ • • -o -9 «u UJ

T3 Svoat ti o «. m »- > t- ••< D P^H "o o • u u s> -o u

•«I > in a» «»>-• u * o» ft ^ * m 3 * m -o o •* > o -o u ***» *> o c o>

•o (-incT>r4ai3cucina.>a-*>o. o cm •>«> e * ft -** ft ft n oc

e o • c » > > -i oft •* u * ft ft e ft-* > ft *» xz m >-

e 4^ft0u-iHfteop>ip<^(9ft(9 *-9ftcu ft-* A •Of-' ••. » u «- .c

u vPft^Aa«*t>-H>u>-9>«).oinoo JB W ft O •• » in O O O U

0-HO-9U O 0«>ü.ftU.Aü.ftOino.(. «I OQ. -O > MAC ft ««. O

m ft .«fta.-Oft<^uM-9i-io»-i> ftfa. o o. **• e c ft-^^Oft •- ft

m -o *» ft o -5 * wbpAjaA n * «t- u o <**ft »» ft-* >uv •*•-*-•»-

ft emu .o ^ »^ a ft r» o ft «»( out- <•* "o m ** ^o-*«* o a. i m ft ft c c o m o o.4>» x o. x u ft «- o -o m ft cm .»* < . «. 0 * « . «^

m > *> ft oi c *- o«4-ft«^v«^fta.Q.» ft no • > * o. y GO« «^

e c -o « c o m o o n a.H > -^ -5 mm -o » et-« p ft » o> u o

1 o ft

-*xc3oiM«<*.^i)inxin m A o 1- «• o. et- ft ft o. •« «* ••9in^«.^c-)co

o w (.ft^jp^it. u.ain-»<ininino=>Oftft ftp *> > ft a « ^ ft o in üJ •*

•»>cin<»>o.ai3oftT3i-ixe c -^ o u. cv •**> c o «* ft op. o> v«^v

• ftcftÄ**rH*>ai«-^«ini-i'-«>-«ft»-<inc» eo * u m •»» m o»e>**oo

W4*-*ftT> «in T» u^iu->4uano.9 •* m «9 x c m H m ft

•n-M** v«i^(.ceo>c —1 tt x ft o> ^ w o> a. m 9 MI o u ft • « m x x

m -* o> m o •Hpe6o-»40ino4*x*> «a.tAepa.e9in-H 9 •. ^ ^ o ^ >

04*<-«eecftp » * * » it > *> <*-«**-« «a. x^P-wosu^PiMg«*-* 1-4-H*

£in*'r4 0aiLi>incx-p m r* ft-n Oft«>ftXtfi«>XfiO«ne c •- oor-ii-i

ftj>x-H-oeMft-<*<*»e'o -o o -o T» m > c m »* <«- «i<^uHcft«-o »«e-*-*

tt4>o««--<Jcft*>> -»«ftr-iftuftsmcinc m * » -<imuo-^on..inuOAo

T> rtuai-o«--»<o>in«.-oo"o-*-o-»*oftft3*> ***. *»** e*ue**-**-** * o o

•*xc9uOo.co-»»c«noc-Pcinc-'4^cuio>(.vcft(-cocco •« » c - »* u w

>k>p(-pftO.U(-O«^ftCftXft«>u0(n l(.p>ftPu09U-H«p.WlAU-H-*

0-PU.O.ft-OQ.CLft^^CVft«>£V-»> O.0-D>ftT>O>-DCU ^OftCftS^^

«-ct-i(n**cfto-5 u-xe «cacao fi-i-* o > •<t o o •« 9 c -o o> s ct u ft o «- a. a.

o. ft a e-x-o«««*- ■■» •* -* «*Ou o u 3-«c«oc>-H*»aQ.

t.-* o o *- o> o« o> 9 u t- o« ••a* »- o> -* •*'- 000 moo

OQK^O o>o>o> ococococo «'Oc«^oevoe<«-oC'**a.u«>c

** c o v w c c c 0

1 m «».•»<•»<•* K n -»<•* * o> o ««. * o <»- -«-i o v -H o »-* m * o> c

•* c -rt * ^ c c xi-xt.xt-x«-c x x x 0 »9o>e> ee

m a ft m -rt c e c <-• •« m v ft «*- ft s- oi«».ft-*mm«^mm«^mm«4.mm fc a c eee 9-H *

c o o. c x t. «. «. x o C-MV -<-! * * » •< * x ft c * ft c •1 ft c -* ft c -^ c •* * * * c in m

O'H a o <»• ft ft ft «^ « 0"»»n-»*<H***-»*-*<^Q.o*>ao*»ao*»ao •»« c ee c •-< m m

»1 t> U ^ -* U U U "* O «-I C •- C «- C *- C •- •* X »-< C X ^ C X •-• C X <-l O" »••< *•»< •* » ft ft

a. > tn a. » m m mmoa.pu puftu«u«*»-ft. PH-O. ti»- a. p»-a. eu*t.i»>v*^ uwm

ft c -.i it .1 m -o "o TJ "o c v -a T> -«iftppftpftinm

• o • • ft a A e» o • •« ••* * ** • —i »p • »i^ • •»* • ••* • • e^*AAAA<-i<c

O 00*0 rHOO o o o o -o o o 00 00 O O '« • ft

• •••M.* . o • ••• ••• . , . 1-1 . . ••* • • • •••«h.cn» » » . en • -

CM O •« CN OOO »itMOrtO'-'O'HO»-« •MCMO'-<(NO'>-iCNO'-"Mjj OOOO aOO

oo«4«40ooooooo«HorMoi*)Ovoooo»4<Hor4Monro 00000000

cs^r5r5o^rjMO^V^Vc^*ro^<r*oir)in^ioincNioiOfo«iooo-iiNioVo^ri

r)Nf*»f*>orowp)Or»»o*n^'ro*»*)^,MOror»irtMroinwwiOMrooo^^«*«*oriiN

^n^^MM»^M^^^M-tM^rt^rn^«i^^ro»H-^ro'<M^ro»H-MO-t^«r'r<rrsi*^

• ••••• • • * • • * • • • ••• ••

• • • • • •

»< »M »^ ^ »< »^

A-6

^
w

i 1

c

o

•4

§

•M

W ft U

ft O H c

(. w •-« «• 4» «- -H 0 .*> o

o. ft w c 0 3 > M u IH

ft o> w

«- •^ ft O. a u <- o 4^

3 5 » 14 W W ft 0 ft <• o

C C ft ^

^ ft u ( 4* a. u

r< i-l ». s (• ft - ft ft O JC O. iH ••« ^ ^ «•- C 3 U (• •H 4« C C <- u «. <*. «^ oi

t- CO m r* "O o f; M (*. k W (- ». Q. O **- U O •M c

3 v> ooo» k

u a. oi ft ft 4 O U H

O I-I » o fi. Q. w e £Oft7 UXWU * • *" ** ft It w (- ft *> o •« O ft >. c •-• »^ GL t. ** Oi

o

A k o> •5 W •'H a. 4* «- •4 * -4 U O C W 0

f« «^ a, C 0 w w c e 0 o o

5 ft ft » 4»

4* «^ 4* e x: ft oi

o> o K o o •»i « c 0 U ft 4* -H «. P4

c o o » a« C —i a, ft ft -H ft r-« U O» OH u O

•»< £ c c •»< ^ •H c C u «» •-« «^ v

:t Ä 4» W ft ft ^

c u ^ T» O -H o o 0 ft > ft u u

o u fc> f* o c o

rH H Q.rt o v a

•* to V ft -H e (. üK a. c «^ «*• v ft

o u •o** -^ •»> o o. 4* •r* 3 O »- O X» Q. fc. V<^4*9 ft«>w

«- >« c c •»< o e ft ^4 "O >«*-««

-o CX ft C •«4 ft

ft Q. O H 01 3 ft

OOO U r -

OO O 3 C W -O 3 C

ft w ft e -a H

O -4 ^ X ft W ft ft 0

5 UUWKW «*yi u.P0ewoft coo

»- i. (0 0) ft c (• X •*4 O

ooc rl > -^ > t. % u u

X> GLo' U c «*• ft w w 0

o a t* M 0» X c m •*» u •»4 % c ». » c

n 3 .H ** o «- c ft C ft a w ft O a. W 0 o> o>»-4 »Aftft«4W HC-O

1 (0 0 V W ft 3 «ft c C 4» = t. • e e

C U. C ft *. »4 ,4

niLV-Hwaw o.

Mftuwftowe *-

C A ft H O ft ftOCft

t4 •».(*. O *- H O Q.

O V0 04*34>HO

» o 01 ft e c o> *> m u * ft

JZ C £ w u -M e ••4 ft WO o c

«* o u 9 3 » > u ti ai «*- «- -S5£2££

1 •»« (0 Q. V W CPU. •M -»H «1 b ^4 • 1 gc-sstss g^^ •«H ft e •-• 4* 9 ft 3 *C4*094*4> -O O > «-

£ u o » A u er «- -o o> 4» »H ••4 •H u IH a.<-4 »I u ew w ft 3 o o.

'H c

^4 1-1 *l

»- » Ul

I o -^ u.

a. o n C

O ft 3 Ol C

wg.liti o ft a> •Hie

•o ft H w

4* 44

fl rift ^« ak Sk m i*

00>-H -O 0> »• U r4 3

W ft% ^ e Mh Al P Al O d m* Al ^F ^^ W W ^r" ^P

u c a. e e c e

•H0U.Oft05oA>-iW

• -« u •tH •t4 •»« O. -H • ^ -H -rH (• •P o> ^ o> c ft H O O Q. ^4 k ft u u ft

• r. c «-

-44» o u 4* -e -o

o e • 01 0) O c •*> e •** (. «* -H k 3 3 & W C -14 »4 tH

b. MflC U "O •-« rl -r* -r* <* 3O*»D.O>«-OV303 o. ft O 0 u > >

o 4* 5 « -H o p

U. C »-4 *. fc

O <*• 1-4 ft ft GL CL

1 «- ft c —• c ft -a (- 0 e * ft

«*•«•><• * o E r4 O •-• o 01 O c. or> ^ V U

O O f 3

o- X 01 w

O

(• a3 ««- ft o SUä-SS *2 7> O» C «-4 XI li. 3 A O

CeAk c «"4 -4 « •

O O OOOAOOO>0

ft -4 O-H- 4* -H > C *»

V X V •» u > 01 4» V » >- c ft c o •- a. U. « ».*.«.

> » > W c •M (. •o a. a. «- 14 v o c o OLI-I 4> e> e O O O

•»<•»< -H II -P 9 CD C O •o o. o c <*• g c o V A c e <«- «^ «^ <*• »iliili ft rl .-1 rt T3 t- "O => o » 0) Sow rT w H •4 W W 0

•-44*6 4* o> ft W -H

•»<•»< -^ O C 0 Ol u. -*» 3 H W TS 3 c -axw

^1 XJ XI C tl a u — m ^ a *••* ft c oi o» v > oi ft -^ «kA-OftftA-H^ 5 «4> ft

o o o 0 Q. a o n u u u f 0 n O > (- O 0 (. k 4* O-H-Hk-O-H^C-^HI.

u u u •H 0» 3 t. C 0 o 0«>V-«4-f4 3 V > 3 30 •O ^4 O 3 3 3 3 ^ k>9e(-Aft4*U3

i. E bP4 ft fc 2.

•^ •»< -H

^4 r4 rH u

•o W Q.«^ Il 0 ^4 *• -^ -W 4> > -O C

0 r-l rH C U ft ft S^^^

• u Z Z Z. u

o. Q. a 9

a. a a <-

c

1*

c er

•H c w ft a.

O

C 0 O 4> Ä «- «-

S^^

e e e •** •H ft a. 0 w «.(.(. a.

w or> i» e •« X ft ^^oao^^o.•B-ooa.o'oo^-HOo.

ft ft V -H ft«*. k «*.

•o "O o> (- -o o» o o

0 O 0 c c C tH «- * "D o

e e c •»< IH •rt a 9 •

(. ft -O 0> 9 0> 0<

•w -^ •« u o>v e 3 c c e c e c c c e e c w c w c c e c

W ft C H •H «4 -»4

eeweoicwc pwc

W M «I O" c C 4* ft <» e O •H •»« -H 'H ftfte-*44*fte-HfteH

WWW c «1 0« » o o w x e e c c c >> e e e > O 4» o c c c c «*4>Oe-H4*O>>4*0C

ft it oi •H s 3 0> o > ^ Oi 1*. .^ ^4 «4 .^1 ^ (^ •t4 •»< « «^ ee«H-Hk c »H «^ 0I-I-I4

www c a- Or C o co a. oc •H « .«».««. V «»• -«4 «^ «*•«». 'H O •-• 0. <^ <«-<*• V

• • • a o e o

•-4i-4a.«*.u>Ha.Hua.v 111 s

mm-*

CO (0 c • • • • ä£SSS£ä £££* • • »A* • • ft • • O

• • • •

ooo• •

O

• Ä Ä . . .^^^^ . . • .

OOO 0 0 0 0.00

o oo <?<>Ä ^ N W * o o oo o ooo »4 r* ro o o o o ^NWO^NM »^NO

o ooo o o OOO oooooooooooooooooooir^voinininoooo

mvmo^rMoooo A A A A A A A A A A

OOxCMrO^TMlO -« (V ro O r*> m n «4 CM n « wMro«roMroo ^«r^

V V V V • • #••

ir)0-H-H»H^-HO(M(N(MOin»nr>mroMMinioiomo«,)irjoioio«r

» «r ^w * * o v« V4 «4 •^ «HinmnviinMinvtnn «4 «4 v4 m m in in «444«4|n«4«4*44r «4<Nin • • • • * • • • • • • • • 4 • • • •

v4 «1 v4 ^«

«4 «-•O n «4 «4 «S •£ «IS n

T4 «4 *4 in «4 ,4 «>4 «4 «4 n **

•H IO H *4 «1 •4

A-7
'S 'S «i o eu -HO «9 t r-i o

Z > -> O W> «- *>'* * r* > > U

S-SuS U «159 So u •** JS -> »- -«4 -X^U V L» •> * -^ Ottt>

5 o «S » 9^3 C«c •> * f« a o -o 9 e a. o >

O><B«*>H u9e«i**oeft9 « c v o- *

x^ «i x> CC(--H9-MCCO«I-P e> «• u otic a £Cc

• » -• «»o o-rt«-*.^**B»«.o c '**•* H 6 o**8

> ä o o-*« •^e9(-90iii.iiu «-tin«* O » v c »» o i •

•MVU -H f-< «*-HU9u«tii^eH iivwotu -I-I »n -»4 *< « «* k w

o o c » a. «- 9 »- cr k^ti o -^ c titi •*»••« x * c «- oe»u i -« v •«

«triO-^o <-• * c o» *• o o«fl(pH^r^4a*)^-«».».titJ p^ee « ^ *. >

x» u » H a o > «4- c — u * — 9«K9 »(-ttcSco a. c •-> o. > t»

o o o ä av tio-»4e woo.»o-tikuf«*ft«o^^ • ••« c «^ v -^ a.

ooe« o-*4 9 M £ o v ** v c o» > «» -rt » u. 0 o •- 3 ä » r

<». o« •«< •-««-oca»*>«o* c -^ > o> -rt * M «- c u «i ou^-s Ö«*0>HW«* 9 O ^* «- -^ O > T» • -H • -««CO on« o > a. * -5 o * o

o -P • -o (- •-<> •*> o. A o -o « > «i «4*-H4*a-o« «* e e >"9

c H «. «. O 3*0 O *> "O O >•> O u C 9>40D> *» •* 41«*?*. ^ t. •• eo. o => CD -< • •- c -o * a. o a» - »^ -^ .c c -o o o • c r -o •»

§a. c *> -o o. •<-«cii.>tt9o>oort»-i»-i -3 0e9o«*-p4 • ^ o< "^ Ä 1« -. «

o o c c 9 «.^«u.>^c<4.u-oi«tija«.«.u> a, -o - c »gsft«*

«.•^ieio »- ni-iti9-4 c>> So*»*«* oo ea.H x u a- o xi

-O O.*» • > 9W «"OU-PKI*0«-0"-- i i •- u » ^ S>«> u c • ^ c

Sow»*» ue» •^»nO(>u*>c*4i-P*>«-.^ » » M -M O «I ^J o H C

5eu« o OX»»I.««H e-*40UMn9(.at> ecu x • x o » o

•*< • M x o>->4 «*e«k4*ov«tt4***eoou(-eii •»<•«*» sC9*^ p -

• x H « » -HOW ucp » o o 9 •» i o» u » • »ip •- v • H > a.

•i S eu •H--«-H90«^u.ci(i«ij.c*ou.üL-H«».fcc »H H u. « i»p*»o<o«.in

a. * a * o •» e«».«*>H«»>o»H-^w»i-«0(-v»H»-iQ.o«>-»4 9 »» I-I o. H c > o •-

SrSwt-uo. «- H o o c&*t(*»flaua«-««o « u«« m » »»•« c »« t>

S 9 «• *» -« 9 > w w •»< ». H r» ^«o>4 ^O - JO 9eM«ie

» w» » » o 0'H**^ov>a«n«^x»«-Q.^.Pt»-o>> •- » » UOOO-H^O«

w t* * c ■♦> v ago »ooiifcrtoc o u > o u. » u <J*I.->*»O*-^~

SoO-«4 * > O- > O O X «- »- 9 O «^ O» 90XiH(. 9 O 9 U 9 O X «- •- C tf»

» » «o u. * *> x> v c >v p-c occ-o-o^av u^-oouo'-et.-poooi

o o u > »-• -o oo o>«^ fiOD>-»<oe o e> u. e .o e «- •- •«

e>«* v** • •a»co«oo-««eok->«eo>c-HOoi«^e «».«o,o«^-««iOÄ*>*»

e e a o» s ovO£x<**-^ ti-PHc^-Puxxo-^ooaufeOPwuuoo«

•>4«l-*400e^<«*- >*»*9*>0««-9XiHtl3 «•« » » o oo -H^uO

(. )«.& -H 9 o o £ o «* x a c Q-x» (*..*» u .o <*• «»jaooo^Mt^Of-i«*«»«^^««

O (• O (• ( w u u u u o> C H Tl H -rt H o c H o> C M to -o «> O «>0I03 vo

^l3^90HH»c0 0C*t.-HV0«.u00«-UC0«--»< C U C C <- *> 9 W-HK

oooo-HO«-e-M(.^oScoo eo. ea.*oeopi'H5.-Hioe o«o^

»U*»U0T»9Ufc.O.0-H^0>P*>0*O'*0*:<^0>fcO'0*«»0>Ua.f<^0Q.Ä

B 5«* uooo. > • u eu ecu > o u »- o «- fii o »mo

OO.OQ.OOO'BX •(••D-aottH-oOe-<->'oot--e'oo v-oo Noeooox

o> cr •' T) Ok o»« -o 0 a.-o o 9 o O>TI -a c -P x> e ••* > * -P -> *

oeoeoo e«^e9eeco^oeok^eweeco-Mueo-Muoo^o^ ** CHC-HCCo>»«fC o •« o o c i 90c*»u«[c^o SeeOOeeOe^OjBX3>

eeoeooe-*>ooo**-**<**o>t)n<»*oi C-PO-P-PVOH av o-^ao •£•»»«»• a»«-

^,^^^rt«.rtc**pH ooe c^t- UIC^POC^OC c^o «e<H o * « F ^ -. "Ö "T *

OO o -H e e ••< 4 4 o • N e a» c

o o oo^oooooooo ooo ooooooo ooe oo S-^S • «••••O*** •9* ••••••••••9* ••• ••• ••••WO****

«i o ^ o «^ N > «i M n « •< r* M o 4-« CM « o ^ w ro ^ *< n w o ^ N w o •* w o • e #oo

«4or*onn ^ooooooood^^^orirvirviooooo^^^oNNey ^ooo

^N»M*^oooooo»H^^^^^^N^'H^oc5r5c5*lNr5f3r5Nc5n2oo»2r5

iow«<rin^O'0-o«o<oO'€-«-o^-0'0>o»H'«-o-oo<o>o^CM"0««'Or*«0'OCN ^oo»^^

•iio»<irt^^o»x^«-i*<^-<^^^»H^^'0^^^eM*4»H^.o^iH^^^»4'«oo»«^4-i

* • • • • • » • • ••••••

«4*4 O -O «4 «4 -O «4 «4 »H O O -« fI n

• • • • • •

«O «4 •-• O ^4 CN • • •

«4 O CM

o

A-8
rw

I

!

I 1 .

u > 9 «I c t» «i

c -^ o c-»* ^w •*» *

-^ > XJ e o> c IAOO. om

•S,eO i-l ri -rt CO.«- ex

^ «fl <* -J >s «O O «- 3 •>

w «- c in »i «- rn 3 a ai c

c c ai *• «- -o »* aa. «- o

e e>w>c ov uot

o> «no»*» intnuio» ev-nSc

•M -o «- -M oi «- c •»< 3 » -M

M a*- > > oi -^ x >

0 > o» •» «e S o» «n •MW-^Mfl

•-»

?

VXO VO» ^ «-O u-HtO-M«.

MIOOr Eft X tc O. C <»• A >>^>

O. * C «- «OWf «OO^OI

•«4 (. 0> "O 0>0. -HOU U <- O "O 'O

> co in in oi >».in*>3i«inc<j-*»coi

e v-> e -o *- »o»oi**t-oi-«-^ooo(-»*

g

> c w T» o JC c «>> u x: «n o >> c T> «. u

»* 3 * MO)«- * «n »i «- u J: O £ «- c «I^COIS

i/i «i e v > * o oi «- 3 o u >*»OOIOI-»>U>»-O

«I0rl/t W 3 in 0* O-^XVkuOOlO 0» IflkwO c

01 > if C C 0> U > > O.« 0»C3«Oin0i3*»«*O

« M Ifl O tfl O "D -MOO<*-0l-O0ll- > *> 01 3 -D OU

eve •»< u -rt <»-'** o<.oi«-o»-u."o c • c

o» o » * ti v oi m «co» *v •« <»-o xe*»in-»>oo*»oi

«. > o o. o u •»» u» u • c c »«». x «•> floecvout- o*

co» u in u c evtiin-H o -« «. x o o x «* a. «n

a>0i£ n no oica.Oiui »H » u 0i » -o 0i TJ > JC » in *. x

cinv «^ .« «». « x> Oi in «- 3 » *. ot > « o> x: oi oi •*> ups^

HOI e •»< ^i H «- c in • A e u oi a n 3 -o v-nr^oimwoiD.c

»(.(•-o u 3 u o o t- 3 * a oi «n c u m -< o o» •-• oi «-o x c e

3 & O -. 0t • 0i <*- H 0) U 3 U (- C 3C 0» « 01 O Q. «- O -• rt u o

oi -p a. H a «- »* T3cuc O-P«^OOI-O««-C ^at-fjeoa-H

» «- -n a. *> m «> m oi Q. c-teooi-MC u x: «^ o-<o*»c «- >!«:

»i -< .« t. m Q.-<-I e -n • c v oi o » -o •»< c o> -o «- oi a oi x u>

-H t. 3 «- m o-H -M (- in • -o «-no« e evoeiM-e««-! * -^ o

>-« oi c u «- x:'-«»«'-«*-o«-«0)"ooinuo.in-Hx:o -n H •« -o o JCOIO«*

^ H m t, u9 30inoio*>oi««-3-«oo » o>**c»-«oco>«-i«-oc

*>*<***>0ie 3 • -P • oi c u o » »- «». ^ o«*»c«j-^eLx:-»»coo oo

* in O I« "O Of OH«.i-i^<-OO>Xt.00l*n0lO<H3C-H3OCt<»*0i-n<^*»0> r>

•no V-H •H«*o«*i-i'>v-Htfiiko.uu>Cfc oiefc«-ooi«(n «ou.

m r^ v «7 -o c m «««-»«in-Hc««-" IVOIOIOIHO»OI« *- > > i w -o o *>•-

<*> ecooi v o JCOOIOX:«^Q.Q.,OC*»UQ.O« m •» •HUSO aa

*> c^tt-HOH-oinc^*: 3

i

o<A-H'oifle>3>M«* •H.HCOO«C«-'öCO • -o I^ »* -^

»

j

oi m > m » v Oi m «n o c ** -H-HO-H o'Ot-O'-i^HOOOi oi»-« c oi

8x>OH30)o«cx»oor-io«^o«>£^je0i0)C«-u"o0i vx:ft0inaiA-Dox:

« > »* uuflioia-*»-»*«^ ouoit--**uu-H*' «->«».«». »> 3 •*>•»> c oi > »»

«3 o *» c H > x) 3 •** H > oi cccvtooioo-poivce Oi m 01

• u0l*>in0i«^U<«u0l0l>«-^<»-tf<C3L3 30l«OH».-ooO OlAOlOlOlOuCOr-)««-

3C>CH3-.>.3C>>-»*O0lHX3a<0l>«^«OO0iU 0» 0i » Ä««Wa0iO

ij-H-HO)^o-uinuH-HHC£>oinu»HHH«-t.«-«^o»oo'Oio«uu o» Oi o o. «-

C > V OlOlVC > > > u Oi Oi \ > > •»* C O •»* C C(»(-il-H«'Hf-i-0«*>9(-lll

-HJ30 jeincxx»-«ÄOO 3<-«Q.äXI«-«-O «^ « C-HO-OI-I-PCOIO» o m H m

ovoiin mo o > > oi o »noooiOi*»o»«-»-^o CT-H eüvvxov's.v o*

XJ -3 c * - o oi -3 ja -3 c c > r-< m -3-5.»»*>c*»o»oo>c H-o-om^Hinmooiininc

O 0i O » ^ Ä O 01 01 O «^ -C x» CC0lOQ.«^*U#O OWV O. «- «- b

-» 3 > u »0 3-3 3-.J*»O »»O3 3H^I*>U c*»j<«-o»0iincouoinooo*i

Oi -H 01 -3 Oi 01 HOl -3 OI OI X V HO»« « *'ee«£O-*ftiftU***>0l

0i ^ 01 *> •»> 01 -H 01 H 01 01 ••' »» 0i HHOIOIOIVV^OICO H H ri» >Q.Q.O.inuu—•

>>(-coc3>>>(-keoc3>>>>kce«c o *» oi««oa<»inuix ooo. «-«-O0;TJ-HSt«-COO0i-O-0l«-*-«-«-O0lT>OH^ *»fc.«.**inOC«H-0lU.tL« 0i0iQ.x:H«.HO0i0ia.Q.£H«H0i0i0i0lO.Ä-H*»«0»0iO0lti O £ o

«ivni4»Hik>v«)«> oiOi-P>H(->-PVMinoi-»*rH u > «> u »* -P oi »- • • • • i~ • • t.i

«t.-3O0l0iCXJCt-»-3O0l0lCe£ja»-3O«0»*oOJü*N ^CMW^- — (M

tH<L<^.»*flei-i5«Ha.£LC>«*o:*HiHoBa.<>Mt«*AflC>-*oax • • • • • • • • •

•• Olli ** ^©•^•^«-•'-•OCNMO

• -••• • o • ••• • • • • G • • • •••Ho • • <; • • o ••* • • • • • •

ooooo oooooooo OOOOOOOOM oo ooc»-»ooootNooro

ooooooooooooooooooooooo oooooo ooooooooo

ro^-io-cfxO»Hr^i*)<rm«orvtDO-.{Nto<rio-«rvaaoo-<fMO^rMOOMnrtrooror>)0

^^^^Mor^r4Nr^(Nr^r4Nonnnronnnnoo<H«4or<ifNOptNNr4r4nr<icNto

« cJ ri n n « n n rJ rj M rj n r^ n rw n n n rj « « o N ci n CN r-i w ö

• • • • • •

CM CM NtM CM n

• •

A-9
o

o oo

•.••

nnon

••••

NN row

CM

A-10
mmi*mimmmm

«-

•

M •** *

Sc *

^ 0*0*

m c * 9 *» »

O b ^4 «* o o

^5,3 £•',^

lllll1

«^ * Q.-4

Fo «- iS >.-*«

e o •< «» u

i u •« «

EC «^ c x

ago Sx:

«• « S (. ft *

* ft -o -e O--D

f^ fc *« 'H ft H

»HO. «-

•»4 *» » O

^ -o o o -• >

7 -x e e o

^*« «i-S?

e -r* f* •** JZ r*

u ^ >H •-• £

- O O O » tfl

v v •*> ** •«

iiiisi

« • • • oc

• • • • «: •

O CMN N O

St

o o

ft ft

*» >

* I* *> ill

m * «^

MM f

•H rH *

•H -H C

I«

ft * o

£ £ 0> a a. e

«». -o S

0 •»« «

o o •

•

H

^*

c ft ft i

^ o o

ft «*•

«. »

•o ••«

ft 0> 9

t c F

55 2

»So l?|

< «•-"

•« N PO

IM WW • • •

ro w w

n n w row w

• • • • •

M M CM n n n

• • •

CM n (M

WWW • » •

CM CMCM

POMWWMMMrOW • •• •••*

MCMCNCMrOrttMCMtM

• • •

CM «CM

CM

A-ll
nnn

• •

«MM • • •

ro o

n

tN

• •

• •

n CM

» CM

• • •

* • •

n ro w • •

CMNf»

CM

nnn

• •

www • • •

o o

• • •

O O «4

» •

O O «4

• • •

o »H <r

• • •

O M

• •

* CM

o o o

• •

ooo • • •

CMNM

o o • * *

ooo

• • »

O «^ M

• • •

O CM M

• • •

CM *

• • •

* N CM

CM

O O

OOO

O «4 CM

o n ro

w V «r

N CM

CM

O O O O O

• •••••

o oo o o o

• •••••

o «4 CM n « in

o« • •

e o o

c • • •

ooo • • • •

O O «^ CM

• • • •

OO «4 «4

• • • •

o -«m m

o o

# •

oo

VCMNCMCMNÖIÖCMNCMCM

04

o o • •

ooo

. • •

n o «i

• • •

*4 O CM

• • •

« CM in

• « »

N in CM

CM

A-12
8

«•

* s

t

uC•

or

cm•- *- _

It* 7 S-" I - •

-^^^ I ^S^ H S 8 S

2 S^oC^^; : g-5^15 jS% !^ I * J° ^ «^.si^ , 5s§fe.cg '^] 2° ; tz*\

si? mu: 1 ls|i|| 111 Ei I iJii

2.5» c«.Vcu o c -o o *» o -« «- «u iÄ2?

|il i |J 15 II If lit. I n 1 ]

ua>in « H •* ***>£ * «ixftw •»- u e eo«*oe ooft<*--»4ftw>-<^i>«'

£S5 ^I^S'8: i-s^s^x?? ^^s-gSft .ssxi&c^t^

«^ * ° C O » •»c;:*»2-oSuUO'«»*>>*^^^0ft3 0«tW3ft3p0ftCft

'

S

l1l

&

äSi5'ftl*Sn^^''22^^5lg55lxh5SSt2

25|^2i:-"

,

2*

,

^Si5xxx

v

^.?«^ü:5:ft'5-55:ftiiSSSftS;5ft

000 5 00 00 o© g 00 00 u 5 0000 000 OOOOV^OOCN

«r<N©,«MN"«riNNfj©r^«NCMr»>«©0'^nMr*)M^rnnn^r*»wnn©

• •• •• •••^ J! ^ «a ^WtM fs.W CMO^M M W fj

M r* ©w w

o

A-13
c •- «a v c

-3 S e (• o 1(1 «> o

4i we *- •> c o * -• — c win eu*

o m •> ** a* w ^ *

fc -^ •»* > o t«tti ot-

•i eoc *» ^ f* ff »^ 41 C «* ^a-^ .MH ,4 at

«- -H «K *C»»>«'>»~«« «-«OO -•'-' > 01 O •« "< U(-

o. u * t-i v o em^vtaa-tDU. * x> ja ai üCC-^-M IP «

tl > t( o *- i>cc U > «-i 3 ^ O O » C O O .O .O ft-O«)^

U. «*0«*«l > c • H « O O * KH £ Ä O 0lC-»*Ot**»a.Q. t»3£«l«>

«-' OAu«* •-• c •-•>£ 0» a» o •- « <^ * eu * u «- • u u 0 o «> x) «1 ai

«b O 01 O 3 •HcilU><-*WV<l O Q. CC^«»-«OOUU 9'HbOlC

•1 -5 «. ti »n -^ -•- o ai * c <- o tf» o -o o o Vv o v A <- ^ c

e> U **- A 9 > 01 *» O £ •»^OCO/t-OWVt-uOl -^jct. «-«-«- •-«••«•- "< » 93

C-9C(-OUHI. (.(.u(-jr>oi(.o -»* oi is vcc<^atceoo «- v ^ u o«

••-• -o o oi u *> e**0«»uu.fe»**eoie'Dc UK» c «- *- *>Ouoi

<**OCO.OI0UMI k c o •-« 0» •-• c o> ^ (• 0) 01 0 +* c -* > * •**   o ;: >.

> «^ o > c -^ -5 ti •« 01 0 t oiumioiooicc ?-** -o9 o»J? cp • 01 c

i(.i-iawo>e5w(-'Hco.cc«>«ix's** •^ca.ttvu.k.Sai c > o ^ c

«*

?

O. 0i 0C. (- 9 tlMU.(-0 (. UC-COl^Ott-COl «»•Ofl(-AMM>> OO««*>-HO

CL OOp-<*<»-»-ICil(P0l*00C*£ 0» •< •»» >4-»4f4 « A 0« 0) >^H J-H

» »-voi4i£0ie&«-»-i^o»-^ ot s ^ u m c 0 u -P I-IU -^ •»« • 01 «- *

u.(90ue**u-o »-< ii.f-«coi»^wuot-coi«- 0t9i-i—•o»aa££ ^«-owu

o>-i^**tiO0e ** a*- t v & 0 01 3 •** ^ ao^-^xoouo ft t. «- o

't- GO u. 9 a •- > oiOo^A(-c^o>x o c o> o m ••o»r<»o>«-«-0o <- 01 <- 01

»- o (.mot- :> u. 0(0rt-£x>o-»>b «- c x 3 v<^ 1- m c **-

c 01 A ^ o 01 ^c-ou.'-'^cyu-»* «* o > v o « - 01 •> i« 93 »9 «- o

*-•*> »> r> e o» 11 « o c oi m > ^Su.«itfio c 0 c •*' -^ o o •- w 4J o

O««-«.Li.O>u-O*>*>0Q 3rt0rU»C*>O^«-iC0iC »-O.eO'H'S-?* 90CO»C >*>OO»-iCO0lOO V»»ej«ÄT9 0i0lC-rt nO-DO>>C^«*M99M)tfl W Ol 01 'H O

o u> (xa-^oiv^oi^oc u -rt « 0 > c f ** *» L. *» «•£•' 0ii *<

e «0 c u »> a. o O^0O4JO»>COXO(-O.»*«^0*»O*O»**O«»IOI 0» £ »*

ov(^<*> ••«(•» x 0-Hu. ««•r«>iiOiH«-e^u^J>u0t^O'e«*i>»>.u9 tf<4>>cooooa.wioive«*«-iO ** tt "* »** 9 0 >« <- t. «1 c u «HS-^U

* • V W- X V tiO-«l«i**-«»<.-i0l»UC*>O «•l«»-9«»>t*0 V6>.«*tf£0*

S-o * * c » c v «« M c 0Ax:(-oi0OO3<B0e««ee,ao0"3>HU03x

01 0« *> -P O tfl *r-(uV-n««--PH0o rt C fi>C«0 C T3 UT A •»> « Ot ft

v«n**ccuoiOic*««H<*.oc-*'Q.a*>-0Oi*> W90ftft > ^ «- 9 o w »» *> o u »- « c

-^ &- 01 (- Q. C ft O C 0 Oft^9O0(<.ftO> 01 -P O 0 »* ft-* r. ».

» '•u>uo>oxH-oa«>»u»-c-»<ÄU-o£i-iw 3'^^'oje0«i-o(-'O(^ o

-> > «• * 0» > o ft * * v c 0» ui o <«- o 01 v u « o> o x) 9 »jiftftHcom«^

ä 0 vet-Kt-HW ftco-HSfu«»-T3->x-»* ftoc.co*«»^*»'*»-»*^-»« e

OflftCeft9(-£ftft'ef4-*4ftft-»« ft u «• 01 o -a a. •* »- ft 0 w o <*. c o -o

(•«oviuftuuC «*•- *< "o H oi oi fftiii**>e*^9k«4*M«t400o-Hft vo-»'(-inftiJC0ft0C«<r4(--o0ft<«-o.in«-.»>-4 o x o tf> » y ft •»< u **•»*« > ^

c«*. <». » c t- e f44*c-'4C«-*4-H«<ao <*• 01 0 o m ft » «- ft v « ««-

ft «•> ft0C0(> > C «H 0 «». U 0<^l«rt£O,«-O'ftO*"»9-PO.**>9'0-«

e.COO>£'»*-«4ftft**0Wft<^ft ft'rtft.*» C U -P C •*> C ft PXJXJ-PKI'O-OÄCU

•H c > in ^co^-Pftftt-O'fua.oc-oOft -H«-* 13 O O •* O oo-xoft

«». m -* «*. t-c^cft-DOcpftincoft-n o,^. •*> c -P o ft ft (- <* JC JC t. u a.

ommu><*-->o+>o * ~» -•* 0. n v m o o o 0 •*OV**'0O«>«>-P in

ftftwo« ooiin^ftftOi"»*in«».inin«»vvu (-u(.<*-<*-.oft*>ft ft ft m <»

• ^1 c * o c £ ^ > > ft £0«o ft ft o -^ o » x o * o * 0 -rt >< • • n o m

ft 01 ft • x 03icc^ft*'x:-* * •* * u o P ^ «- % y *»- «^ * -0 ft

«-x v U*^-H (- ft ai ft •** •»»«cft**-»*c*'oc-*inoincft'O'«'0-*«»H-i eu

ft^ti^ftk^xftic ec o->4-'4-HOu.ft ^«-«».(.oaftyftu^ooftoc

«* ^ »J (.fra^cftoipt.—<inin>i««FHi-t-i>«inMin>4ft ft •* m -^ ft -< ft -.< •»« * ,H -H o

^ca.c90.^*>c>>u9 «.fc^» •* »<•* » (^CASini^^o.ttoä.ja*'«*«-»*-**

* «n « » u ku9ftft0uwooo^AAeft-HO0ft«*fteft-^9-H 9OBe00in

cmoiu^uo Df« * xu »-•»* * u .-1 0 ouc u •*> •< ft PHU-PU 1 u 1 *- 01 01 .c u •

MftUfteb.inX ££ft0OUU->4«O. O.- Oft*»«->0->-»*UftCftC9-»**>0-*9

me f-i )-HOioi>uucc-»<00<»000(«--HQ.uftftftft«^ftQ.Oo.oinoo(.(*.u

tu M • ui ft A a •-i«<M«-4uu.b.-»4 Q.U u -H «* m « ft ac cc ac r* -»en a: tn x 0 0L a. a. * t.

x: ^ ft üL 0 u 0 uu « y« ft u *

• .^.•^— ••en* • • u. • • ft o • »ftc ••o***fto* * ••£** • ft c_)

^«01 •-• ^wn M rsi r> «r ~*C4 a. ^CMO. ^N ^MWO, »«CMM* — CM W Q.

• •••••••^••«•* »••</>••• o) ••••*•• en ••••••••«(/j«

CMCMOrOO*^ftO-<^^l'-«OfM04 0^*4 O ^ ^ O CM (N CM ©»H^^^OCMNrM O

o o ro o «r o o «^oooonooo^^i-io^cMCMrMrMrMNOv^tomnnMronroo«-!

ooooooooo«4«4«4«-i ©•^•-•©•i»*«^ocM»**^tM«^^»« orn****»*»^in"<^«-«o*

f^WONOfMNO^NNCMCM-^tMfNI^^PilCilN^CMCNI^CMCMWM'-tWWCMW^fViriir**^

mMCMrnrMMMOfMWMrjro fMMn»-«rMWr»^MnMCMMwro ^wwwwwcMmMw^n

• • • * • • • ,*• • • •

w n ^ro w CM ro CMW n wr» n CM n

• • • • •

CM ro M w n

A-14
5 ö o % * S

» * © V O » •• PC

U U > Uil> » -P C •« T> ft

O O • 0) •»» ^ > C O V fli U

~>u*» •M4wc>o

Ä V O 01 »> t. «I UCl^U-*

9 > -O TD +» •-

Oi »> t. Oi * u «

>0"»»'"> S S 4i .

u r« c c c * m * * * xt -3 o c -^ u *« o»

5 c c «^ OOfi Q. u •>* -5 c o ^» ooo>«e

x ooouui c U £. r* -

0i,

'

<

£«

,

'l^ «S-M-H ttiietio x: »«•POti'PV

C C O O C n — O OOi-ift O «• i3 £ f 9 'l-C^4?'**"4!''4

^«j^tivoos ^H -H » u ti *• xz Q> c «: «ots^tüos

o-oeso Sevto 0 o. o «- «^ « • 8 ••* '

CCOjDÄXO»-t-3UI OlO-O^ "^P-,

ouf-ivooai^ w> o •»> *- £ «*• o» > «wii.

uSM-^-^ft-feSJui u«. 5 c x: u ^ u: l

6

?

JD .^ ». «utt» Mtti«t- eo-oo • y o •* » • o # oi

•»»

C U

" 9

E

i

ceo jOAxot.«.*« iio-o^ 0'

<P--e0

liE

JD .^ ». «Utt» Mttl«t- eo-oo • u o •* **• o .

««osoi-t-^wo'O'o-o J «i«».«^ *» S-.^^*1?!; 00-9COO«. C C 0» O J3 -O «.F^ C •« O * X » W O

•-3-3 *i«.0l0(0l-33«*>O0) ecoo o ,*» o — ^^2

f-i i- o «- *- c K» «». i- * o a.<*- u t^^o-co^g

o u o o» ti (.kpo-HXt*-*« .c •<-• o» o*a.o.o***o

•^ «HC^<-«-O*50'U<«.O3 «> ^> (. 01 -* ** »- O O tt-H

•*><*>(>(-^0l0l0lCCC«ltfl^-O-O>'4UQ.X> O •» -P >^ TO

ww o <- "o x> « »- •« •»< a-H »ess S wen«,

•^ -^ t- 0» «» c c OOOM«*>>(>-H (• oi o o » «> t. v u v o a

rHfHtl^C99Cfr(-(- OU C -O > X> *» •»* • «- O C H »I ««. -P

8o TJ c o^-**4*w«e««-«4 0i«i » > 3 oi • * o» *

0«C0l3«»t.Ö.V«<W*> o -o o -o c o «> u -^ -o ^ V U * o f

«-».3x:o«-»^3«-ccc#>3-P«-3-^-»»i» ? • • S«^*96t

> «43 •oo-*4«a,tn#*f-4 > *- •« u»e(>»su-H fi

« (-•t C7>oo-ocuuo3 0) uee<«4 -^ 9 p c u a. .o < > •

•l0l«*«*C«-<'->«C ^C-OlAOlCiiO-P * a *- »< 01 -^ £ Q.

^ "O « C •*< x ^ "« 3 ^ *- » O > .C -rt tH U C 01 M&k^r^f-lKIUU -H

CCSOl-t'OOOO'OOW«* «* ««»«I •*« «- « W 00-^1-10 0^03

33««3>>Q. C-^CUS-'O • 01 O -3 0» • » «»-JCLC-MHÄ^tlOlCT

-4 a. v o a m ** >o9O0i*-tt-»4 9« • (- a c •*4oroxa«>e -o oi

o>o> -««o* •-« a »- o u T» •-< o M > «- o -c OL «»« uS^'*! £ ^ S* ^

C C •• 3 X > -P O £ o«lVV<P->4>P » W Q.*» etpkOO.VtflWfOUOlC

•««•«4 O-aiOCMU-^0((*-t--O''4 -rt r-l I« < - kUtlMOVtl-"« u «-«- » «^ •« c o

XKMS C0l-»*ft>AOGL*-U«e^-^CO *t • »« "P O.I»X-P«-30l»-*»X -rt t-«Mt:C • C X O O 0» «- M £ ^ O0lCQ.*in«»<0lOOfcCC»O3-P*»

o.o.o»t.-iH*»Q.«0(x-ooio»xio.OÄO o« *s9..'l*Sy^P^€ss P.F0

eoo -oiisckACb^o I «oc o«*«i ^H oifvfcoioto-poio

«• «t « « cr »> a HO» *> O • -H > «i •*iu«-<C'-»je»-«XOt*«oi^rH^oi* a. «^

*»* ^ naoi«i^-ooiou«'-*oo'n0)3 «*•» o-ii«oo»^c«. -o c c -

Ot-»0<»>-r*>tC •(•-•Ort t.fc-«4«*uO- O -3 "O rt 4t C A ^ 9 0) 0« • «03^

cooio-rtswo-DC'** (.inoi(*-«>t>coi ox>o>*>*'-i-«o»o TI £ x: £ o -xero

•P3vuo-^o*rtincoio*» coSw« -xoio o-oui-**- it***»-*'-»»'*-** itt

rU »-< »OlO •HOCti«* •-• C 01 Q. 3 **- 3«-4U3rtQ.OO O O U

OOXQ.VOX«^«-0*«-0«-O«t0.cr0i -< 0» w »- O •« > * O 3 3 3 3 U«»

o«»->c»nu«vc-»i'»*u3«-o»u-Mwoo»£ uj:«<o**w-^u*>u«-ooooyO£Q.

4* o -• o u ¥> » u H > •» w «» > •» >o<«-cx*»o**o..cx:.c.c*-i-»*tfl

u TO c 0.0*0 • c -o * «^ c o a. c ot £•»*(- o» o. a

O0l'4^.-rtO0l(4.Q.O0lA(».rt-rHN<«>(*><«-«^ •« «^ ««- •- « a. o a.*» ff«».«».«»«*.«».*«*-««-«»

o 3 » rt o * u n ou -rt a o o o oo oooor'ooo

or«*- «t-wti^c • «..«». «^ ««. «^ v 0» •»

gOl-O^cWOHcCO-COcCCC PCC000000-PCCCCC*»CCC

•»UO*»*UOOI*>00>0000 * 0 O »COOOObOOO

«4 Ol <- -fi -O •-• 0) ti C -rt •) ^-i -n -M (. -ri -IH -n -rt M-fi-fiCCCCCC

<^a..«j«»c**>a.«*-M*^o.«>«*o-«>o«*«*4*«* t»-o.*»*>c**-a.-*>'*i*'0.**-**a-**Ot**»*»-*> o.*»*»oooooo O.*»*»OOOOOOO.*»P*»*>-P9.*»*»**

»«WOOOrtUloOrtWoOrtO O O O O «OO-rtrt-rt-^rt^^OOOOOfOOO

Ul«««U Ul UkUl>HU(--'«CUUUU*OUU«*4*«*«*«**'OUUUUU0UUU 01 C > i-i ¥> 0< C -rt > Ol C > ^-i Ol > O •»< --< H •»* C U •< -rt O O O O O O U -^ -rt it ••< •-« U •»< -rt rt

Q.o«i<*.-<o.O(*.tna.ooi(<-<*>a) u. u. <*.**. o «4-<*-NNNNNN V«^«^€^«»..«^V.V

cnxo-n o en x n c en x a M • -• A o« -ii r« •« -« •-• •»•«•^•»if«-»«^'»*'»« oi •« •« »4 --i ^ o» •»< •< H

uo uO u(- 'ouwuuo.'ouyffffff'oyuuuyvuyy

•HtNraa »^NQ. »*r<Mo. «-">Q.Q.O.Q.C> a a-»* »***» »*» > a. a, CL o. a > ,9>£>£-

• • »en • • • tn • • • »en • •owmmviOO<n(n»^^*^t-*>^*-*omm(Jitnwotntntf>

^<*H»t ©1-1»^ o»-«»-'»^ ©'-<•- »» ^ »- •-

4r«>«o<->inmo<H<0'<'«o«-<rs. oooo« oooooooo ooooo ooo

»t^i^oin«-<^0-4}'^^*-iOrv«-iOOOOO ooooooooooooooooooo

wc>iNio«-«wN'«^r>iNNiC-tc>io^ww*oo^Nn»«>orvmo»tfMM*ioo»jfviM

winio^Nnro^Nnnw»tf5Mof5c3f5f5oo»t-4»t^^^^^of5r«c5p4NonwM

fC w eCw fin CM ci f5 CN f5 o »< w w to M ro w ro r» w ro w w « M w w ro M

MM M CN M M M M O W M MM M M M MM r» M M M M Mn M M M • • • • •

M M M M n

A-15
M

9

•o

1 S > Y

» -^ « 9

M« V «• * «

ft ft ft ft Q. HI i» *- > > «- a. •

ft 9 «< •« O O S- ft

•M ft y y ft ** ft -e -a

in u ftft«-yQ.Oft-D*

o (>Q. OOft ft > O.T> o a c

mc o c..c Oftftaaio

( o -^ - • ft o «J \ « «.(.ftt..^

o» H 0v uuo3fta.o(>cx.»>

a.** «-e ccwft-a A a. a. o

Ou •*> ft o o *» ft u ft ft » u

^3 »«•** IA • « IT «- (- O "C ». t «/ t. M

ft(> ft H c « t«-ftOo«i c e ex c- o <4-

> > -< c o ft oo<*»«*>ai«o ox

O |fl fttf^-XU U « • <« ft O WftMiH ftV -D C Q. • O bbc«*«. «t V «- «I «» O

•pe H o.-o - (.«ftftu^a.(-#»wo<»*«i9

ft ai * 90«« Q.ftaa.-r4WO»ft*ftar

ftft «ftblAft «- X«*ft-«4ft«*Wft«*

Cft ft-HO « > •0(9C93k>«* «>«*ft

(. > >«». T3 ft M x: a> > x: o 0 JZ tf> JC c

~* ** m M n c •»> ** «>ftu.u.*>u£>>uftOu*>

0» 3 «- «'-•ft o a« u m «-•-••-lOOwftjaMVCrt

CO- O *'-> *> Kftft P^O.MMVft-t'Xt dC x<^

gft«« «* 3 u c -»J "« -7 »* c «> 'VSxxrso (•(» 0 O-O«ICÄrt*trtftXO(*(*».ft0ft UU

* ft 4» ft<*-i fto>d««v(> «- w c • * o.»» ft » ». -^ .< «. *>

( oic -^ Cftftc-M«*e9 09« •«4«iO£eo ft -H (- i-l ftOb O.-^ «-4 3 C ft "O ftOV'Xfb U «- <». 3. 3 <*•

O.9 0 >* «."^x-rtCOO-ftft «*4*ft<^uO-«4ft

•o ft u «a. 9i»9-Mc:ftlft« 0« (.-M^nV ^i c ^ b «. m

«xft I-« 0 ft inar^eroot-ftuft «Mft-s^« ft «- o o o u» «>

ox: V ut-cftoftb-Mftc-t*-* «* ft ** u ^ 3 xi 09«^<^eft

8u o eoot.9 » »«-•-« 9 » «. o M v • > ft o- o *

0 V Q W ft V •1 0O> U0.9O--M « -O -I r-l 9 *- (. 0 »-I 00 -H«

•M -P c«.ei*»^-xc0 9 o ft > <** -o • a. 00 x: 1. c c *> e

«» D 0 O0l*»OO>«*»»-t.ft«.rt 00*- o.v «^ o. ft ft ft o o o

ue ft ft •*•»••>• uube » o» «. «* ft 00 **&** £ * <* ut90 u eu » 01 « '< x -rt •« >•» c 0u «•4*4* 0 «- v 0k«»**>MO

t- «- oc ob-*4<*>0,00fcc(»^4fte 0O90(-&0cxiooe«*><*> 4*ft 9 -« O O U « X 9 t- O -x O ft -H (. £ £ C 0> O ft ft ••-• ft U U n

00 O -»»TJ^Mt-OUXO-PO. ftCVol ft«*«*-*4e-H04*B«^-««-*«f-iT)

c •- 0 0u-Hc<«-o<4-ftD. o. ** -* c -i <~> e *> H > » 3 e 1 H- <«- e oi

-x » ft tt990-H«* a (.C.90 0*40 C000.*»00ft«»01.-r4-H9(.

«- ui-a>i-iu0<^Mvoo0 «^ 0 •»< o*>'**fto.c:ft.eftuft~4i-io-o

s- «^ c * o.ft«*o o «4- w- ftoi-uv ft*«i*»«-ft*»«>Ä e o o > a

O O <-• ft 0 >-• OL-H •-> t-IC ftOOvntlft A. -o c 9 9 Ol 00 £.eocMi^0O000O-H0a.<^ !«*«*>.« «•-«•. 0 ft 9 o &•*» t.

gc •*> * ti-x-HO •xe-Hccc^^e c •- b0uooe^e» ca OC«* 0«c VW*»uO-»»0 00*»*>0«^«^0 Ol TJ -O ft «- ft ov > » o

^-1 1-1 01 c ftftve000He-H-»4-HC9MOO-*4 cftftefte9ft-*«->4ftec>in

«*«*•« O(.OV0lftW.«*ft«*«*«*ftO-*» » C<-»-c3CCU*»»H«»OO0l**

0O90 0 •- 0t-««* O0OOO0 O00O 9009ft9-COOO>>^in

uuu0 «: 9 (•> 01 0 «^u0uuu0«uftft4* o>a.ä. o>- o><r>o-x9-o«ftaiai

•X-^Oft O 0> O «H «I <« <*- O H ft X -W M ft ft '>4 X M C ft ft > 0>>O'«^— »-•»*

«* «•-'« O 9 OO **- •* «*-«•- ^««.bfeE >«-».>ft>.«ftft -Q ft Ol k

•»< M 01 «1 A x <A -e ft 0 -rt 01 « -^ « 01 w •« o o i u.a.ouu.ccu._j'-ieftcccac>-ioi

y u vo v^e-ovoi'oiAtti'Pu'ouuu'oouoag »H M IH ft bo x

ftftc-rixu30«-«4v<**0ft-'iftftft"4«*ftS0y • * • « * « *flc • ••-• o • • . Ho.a.x>tio-**-'4u>tiftMQ.>o.o.a.>fto.9 90 «-IN «* «^ «^9 ^(Mn (ncncoi-3cn_i(nO(n(n_i(nO(ntn(noci(ntn(nAtfi * * • • • • • • •er» • • • •

• • O CL • • • .•^.••••CL* ••&. ••« •••• •••_• •••<

OO«- OOOOO OOOO OOO OOOOOft^OOfMOnOVO « O O O M

• •!»••••••••••••••••••• ••!>•••••••••••••••

OO OOOOOOOOOOOOOOOOOOOOO O0000000000*4—«o

lOOO^fjMV^o^r^M^rO^cNtoo-^cNrnVinooioiooiooiooioo-Hirtioio-H

tomoo«-i*4«4V4^ocN(NNNOnrono^4r«'«>4roin(ominninr*)mnoinmnnin

• •••••••• •»••• • »••••*• • • • • • *

i«»wo»**^r*0'*'N**0'*n0>0'^0,v<r*<r4rOM M M W •< M M

ro n o «• w ro n mro 0-M row n * ro w n ro n w n roo 10

• • • • • • . •

w ro

A-16
• c *»

> o u

o ii u «i

•O <*» "9

o -M e ft ot

ft "O -o ^ «-

«- c -^

•H o -o O tl 9 u e v «

Or 0

ft -o u «• O

I- ft » v

X <*- ft ft IP

0

ft

ft

o o -

9 3 ^

> or O -^

U C 9 O

0 9 O- U

^ o

e > — ft

e A ia -M u ft ft

cr-o -o u > ** o «-

ft ft t" » m *

m ^ in c- ti

O U« r-> ft U «*4*M

*- e o * c

Q.«*. Kl tfl ft O «*

• «i ft o e

e -H ft (- (. «*

o*4 ** a

•M U & •» >.

9 ft X > O •»*

O- ft «• A » -w

ft ft ft •»

t •»< C ^ *» V •»<

^ > ft IP > JD

o> a. u v ft H o

e • o ^ •»* -« e

e <o -4 a A

«- C ft rH 0 O ft 8 5«* ft (- ft

•-« 4* o in ft «^

•H O 1-4 o •-< <-l

"* c

or*>

§xo

t »^ *

o. ft a. x

to*»

o **

O >H

c » -^

O ja

•«H >. o

ft

•-<s*

o

9 O

e >

er-«

c e o>

•*4 -H e

e o k

•H •- »

o *>

«- ft «* O ft

ft 9

tt 9

> »»

52

e e

u u

o> a>

££

O O

o »-•

•*« ft o> o

$A ft e u

o M -n

^ «»ft

ft w a. «ft

v ft • ft ft -H .

ft -v« •v« «» *» (H r4 ft 'H ft a a.

*» o. c * e ft •

• O » -^ •»< 'H

o

u

ftft

ft

»

o o

«4 ft O

2.5"

» ^ *

E •

-

o H ^

o» » -^

~ e

ft ft

* £

ft cr«*

T£o

O «» "O

coo

ft O X

«• 4*

a» *< •

c ^ ^ ft

•H -M -M «*

CA .fl O

••4 o o -o

ft ft C -M

*» k ft -M -ft

8«» ft ^ c

4» t-l rl ft u

C 9 ^-O

I I

070

a. ft Q.

o.fra

ft «. ft (. a. (•

o e

ft ft k w

» o >

* f

• ft ft

»*» *

ft •O ft o

ft > c

E

e « £

^» u u

WO U £ ft-*i O S C » uki-i

• ••• • o o • • < • » o • *ou ••• —•• tJ

,

**oc/>»» • on**

-« CN M ^ IO ft »«CM *4tMM «-iriWft «eMW ^NM *- CM M » «i CM W o.^ «-• CM

w w n (N CN o^^ofMNNonnn o^^^ofMNNOWMn ö ^ ^ «^ oo«^^

oooooo«i*4«-iCM«H«4«-«r)«4v4*4o«4rMr4CMOirMr4tNnoi(MrMO«4m**no «400

n'roronn<Minmninnnroinnnn^Mmtoninnmroinnmn«Hinronro«40innr)

• • • • • • • * * •••

10 M n r> «r» n ro «w ncvin

* • • *

ro w w mo

ro

A-17

»^ • * ,1 «_•-« ^-m." 1 tmm^mmmmmmma »iwf Af-iu-»-^ »jn ^- •.» •_ • »-t ^_-»n nILT »_n 1
ti ** 6 •** o *» JO

>.«»u fi-n-».** -^F^ ox»

» •< S 4* Ä y 43 •*« ~9 O C».

•n f c M:UO«OS 0>-it9Ax *-> «^ c

r-l > > U» O T» C -> r» O > O »> II k» •«4(9 to o o T3 ja o> u. -^ « ä c «i a

e b. c w e-««»«*« p aa t> * --» « ^ m

•-i-H (- ti c »> u *- -o t< c ••« c v o^aocfcci «it* « okotfte o- »- -^ >>ce net* •-> o -> *

C «<- 0 -X O »> TJ U «- C X 9>,«*>,f-l •« «I t.

tO X .-. •»> "D •« «- Or 9 -^ r-l TO 0«-^ ••* -H JZ « ** O TJ

•o «-• a. «-• • 0» «• XJ «- o o •> x o. * x a 6^0 0 a »-• (-

?4> cr«* -^ O «•*• «•«H (-( 01 -• O 3 »H O H CSV o

•Mfeoxtt- »A c ««o.a. ucx a. ^ JO x og-oi'Q Q. -O

«» «- c o a -^ o » o w oi > ca.*» t* a-f 0 0 »-< « XJ 01 01 01

0 a. -^ c u 01 -P 0 -P 000 900 9 a. ^ -o 0 ,4 «- «-

b«*aiM«e -^ c -Hft-^uv e *> c c «ec u -^ 0 9 •*•* a

Z -w s ot « »^o ••* c 0.-1 «n x 0 • o o ««00 00. vw •-• 0* 01

o. c •»* o « ä > uE m ~4 01 > u 9uu uu _ a«* u • « 0 •- «-

0-^«Oc 0 • OM^ii**-* U » WOO 0 ••* Oi 3 OCk

01 •- -4 W O. M »-I «Htfl Uk ».>(.!.> c rt ** at- > "4 > 01 •• « c •« • •« 0» —ecu» c c c c o

1

f 9-'-« »" f

30£X^«)<P0k V H •I O «- O -< ^4 ^4 * 01 ^4 »H 01 U U W -4 » «10

öl •- 3 3 3 Oi W > •-< w .-4 > -4 ««. a. « 0» -rt •»< tt« -M-M 0«

» » «1 M ti 01 fto.u*> •> o u» w > «^ * w v «. i- > TS m 3 > w

«iMOinmoi-»*-»»'-«*« » « o o o» 01 -^ 3 01 ot -»4 01 w -»4 c c t -o m *

ftCtk(.».>H 01 '* O C »4 W«*-M-f4aiUf4-H Uk(.CO>0(--OO> £«

«* -^ <»- e c P4|I(»<** x v M -H •-? «*U<-4^£üI^*M «r4t-*eev0«*e u 0»

«. (- »M -^ £ t- c ja c 01 o» »* a« «100. 0.«* a. Q. a w OL •»«•HC'D«* •H«*

we«-fti-4^^x><**'4 0 o > x: ^4 • o « « « « JT«» « * « »«-<? c ^4 ^

u o » «o > u->4f4«*3-»4 «* m -^ »4 «. -*4 -^ in «»oo^oato 3£

i4i-i0XX3 x x o» ^««u m •»* > o a. OI«»»4MUI-«I»*»COI U

JC -* f* JO & ^ Ä •- fl rl U ^ • Ol «> m Of «*«>«*'»4«*4*4***'4'4**-*« I« -»4 TS t- -«

S0> • oi coi 01 o ^ o «- u o» -c-pff 3in«nco.o.«^0"ox Of

VOI-OTJC-O-OO»-« oit- o»*>in**oioioi-oa»oto»^¥«-^**oi«»'«^*. «^s

«. -o « oi o c oi «* «i -o «4. in •-> oi -H^voiv^vxro^f«^ •MCWO

DO «-«««.aitfi-p^incoiu > XJ -o •- t <». •^v * «- o o« **- x -4 «^ m (-

«^v-pm^c^tfimmotOPOiPu 003 O-HO O>«^ W O> o» •PO>O»3X:HX*» m > o

oo0oicoioia>**»4«x»-to» £«<*-uee-»4»-4eei-*ee o-*» u v o w «* e «*

-T)£Q.«^Q.t-i*--P*» «*U0l3^«*aLfi 01 »4 .4 o o »4 H o c -»4 •4 •- m « ot c 01 m m •*> -a o» u -M m A in u'ocLin-40«ow>-4<»-5Mii-«o>«.a.-'4 cfiim

oi«« oiO<-p(*--o-»*-*>£0'40oicoioooiWHtf4a.vinM u#*eom-**»3m**>

«*« ii J: Ji c e o. <*• c a. <* c a«^ i t o.«». e c » c 01 > 0» 01

•»4inair-tf-i4^tno<«*-»>xoi<^oiooi<*-oi -«4 01 » o» -* 9 -^ "o 01 u * -4 •

^-1 w » -4 ^4 de äOIO«.V«IO«.CUOI-P*»CO»***»«^ ••»» » ooto»-»»*»©^ £0i

f40l -»4 -^ in «- »4 W 1» 3 3 3 3OOi3O0Oi30O'4 300«-rt300«.£o>

^o in 0 3 3» 3 cv-«J-o*»ur-i-o*>-o-4.4 Q.4->j:j;:c*>x:£U«»££«*<^V££a.unoi

o •> 0» «.*fcin»io»o»Oi-^ot«iO-4**inoi-*>*»Q.o»*>*>o»o»-p**i-»4 0»*>*» 0 »4

0 •»'-*'C0OO(O)-*>AXO-DA«-00 A -—A a.o 01 U -O <».«-£

Vinfkaii-^^-H Oi VK c a.«*, u w x x 3 x x * x x » oi x x 0 w 01 u

-»>ooai0il-4««-4 u w 01 m .«oi«n*»**0-in*>.p ««*«*a.a.i«i^«* «eo

c** *»*> o>o>-»4C4**»**C'P*»^uc-»<-^ttC'H'HWC'H-^ m c -^ -M c •- «-

0 •» jao>«^cc»-iOoocoooa.cO"-4»-i Oi-i"44JO<-4i-i«». o »-4 ^4 o 01 0 *>

> in •*> O 0 O C O-«4-«40l>«<££-H-H££CkiO''4'*

4

'«

4

V-H-«4-HCaHf4->4 O tfl -«4 ^ -»4 ll COlC

Oi <P C Oi Oi t- -»4 t. (. ***>«* 4J*>*>0-»»-»*ääC4Jää»4*»^»JO p<**A«>»>(-r-i0

« in oi a. c m o o w 0 * a in00O0o00O00oin«-0O0O0i>

»- »Soo«o-»«»4-^o>oou>ooi(3>ppin>po4*>ppfc-o>po-»4-«oi«-«

«. •-«i-(>=i(.0oo-^oi<-(.oioi(-fc-(-uoi(-(.^ioi(-t.woi(>k0coi«-t><*.^ c oi

M oi UJ a. a. u *> i»-»-^*OA.OL-»ao.a.9 «-oo-o.ocia.«Lcoft.a.,oocio.a.-'4 oi c cc

• ^••**iin**C***0***OU***H-* ••U***O(O***0l3(9*

»*> -* fM rO 4r *- -^CM »^ M M -4 fsl M Oi ^OtM •- CM M -4 CM M ** -4 fM f*> o. m »4

• ••• • »a. ••• •••••• • »£ ••• ••• ••• • . . tn • • ••oTo»» «4 O N CM N M O *4 «H O (MN NO n ro n O ^ •J »^ O P* CM CM O W W M o «4 «^ «^ 5®t

orMOoooo^v^^cM^M^n^^vHÖ^rMCMrMCMfMCMCMncMMrMO^nmno *40

CMOCMCMNCMÖ^rMCM^NNN^NrMCMOrMCMrMNCMNCMNNNCMNOmNCMCMÖoon

incMioininn^CMinincMininincMnininNNioiniocMininincMinininmNininin^on

fOlDWMMnfM«fOM«nnn«rOMWCMIrt«MMIOWWWIOWMfntMIOMfOtOtMOinM

•• • • • * • * • •

n m n n n « ro r*> n tnn «MM

• • • • •

n n w w io

rn

A-18
lllill ?ii 111 « ?8 .5 21« Hin j ll| °ss

: IJ !s in

IsslsS'? il Ilo5S?

v

SE'tSSS^.f iS J5 fi J K * p4 «-o So-»» #

"'So* ÜSS ^jae •o«o«^ «iS-S

x"$«-ooo-o ^.5 5OHV«!I J fli« killl. IISJsJJ 1

i. •-o« S 5 o-»« " * * »..* ^ •» ' • S^"S ^•2H'***,S 2 Rl'S^'S "i^ ^ilhiiJillis^niiiJi^iliiillHll!!!

e5C fcKfc«»^5<Sfl 3^S o» «- *•»« uoa>o>oc f 1 9 . 8* • •

*» o a « ^ ^ H i c «^ li^ 0^I^CO^»HO.C o^^^B^^-Si--^; -2--2^

• t » * "25 5 > o u u " m 11 2 a» *- «•Zu oi «- «- rn o» «- <-•»•» ( «- o c Jf »• *- «^ »*

• • fi • ..• . 'S o »»^ • • • B • • »ou • • • *s • • »o» ••©«••••c MM^^NM^IOO^«M*-.NrO-.Mw8-<NrO ^f^M .^WMj» .^^W«}-?

, » , » »,• • ä » ••• ••• ••• • C * • • • ••• •••••io ••• •ine

• ••• ••• ••• • •• •••••• ••• ••• ••• ••• ••• • _i * " ^ •

n»n«MnrowwM«wnioronMinnMnMioMMMinroMMioWMwMinwwnMO

n ion w w «M M w «n wo

n rn w MO

V

A-19
us

o

• * C 0

U «t "< V

1

^ i

sr E s« £

;i i Ji 1

ill I tls I

4^ k xi in *> »- * £

u .»*««. a o-evo 9

«- T> U C X i-IO* 3

^* -^ c «- c *> a

•••OHO*» *»* P •• •• •

uo O-»JC+>O«I*»H -H i

v -j «> i. o H «I -H a. o. - -H

On o^-oc-oS^ew <" (-

> 0» O -H ^i «. o * tl «I

»••H>X'O*P»I f a-o -o w o.

«-o*>*»c»*»v»fc *» -{»O •>

OVO M £ t( V- -D * • H

•Pe^do«*»*«* * • o « » «-

S» c a »> £01*0« o o

«iftt-AC'**«. vtioti o> 0» u n

V «» * V O O ««- 1*1 C X) XI a« x M (

i-tt **-o-«4 v**-a.

voatiM «-liiiiMe-nMiix » fnc

o.«.OT»u*H"ocu>*«-»*o>»aio-H

»«•.«- O C V «.Otl>C-H0i>«UC ««-KIAOSMOX t- •« O TD "O H JO «» H

fcTQ .H«-H •»«IT» »«-•H > H O

ofi-o«Stfi'ow>-«fi-4y>P< • o u ** -o *-

•H5»4«*- > ©•-• r«». c -"5 c x »-» -9 o. •»*

o»(.ecbxefic«*-o -o o o o «- o

5 « -««ta ^•^«^•oo ••HO

•J » O O tl -H O jQ tl C H t» 3 C Ott

•H •Ot.C-H>tlCC-Ht«--»>tl«-CU

» » o » 00**0«*0(-0Z-H«».

«« **• HOttf-H^O«»' *- 3 < «- O

OttWttVUV tl > -P U *• C tt«*(- o •»*

--? » t> v u-HO**ouHtt-H aoti.*>tttt

ti x i «-•« > ti «- -H etttt ti e

•OI..»>«J4>uV)V»>f> u •• o »* &•• u <* tt

fi^ttinuittcw(>iiitt<**-H • m c o w

>CQaiCQ.OtlOCQ.C*>JCtl*>tl JO eu}ü.i-^tn«»-o»-icntto o-»- c «. ti «

«- ti • •- H n * *

a. • • • ••o • • • •ettX*0£3*

ooooo oooopa o «- -o o

O O O O O O O O O O O •£ e o «> oo

• •#•••• • • • > • • • C • * *

o^Nm^rmo^cMrov eoo«-iOoo«4

•i2«««4 «« J« O NCMM N O O «^ o O «4

«H'«0«0'0«0"«M'0«0'0'000«-"^00^'H

••wwMMiO'or*»r»roMO©^Voo^in • • •••,••«

• • •

O^ O V)

A-20
APPENDIX B

Training Enabling Subsystem Data Base

B-l
I

II

t

$

X)

3

n

9

C

•H

H

C

atou

I»

HO

X

(- *£

000

E(M<H

E 4»

3X0

04>

3«

(-3

CDH

^ |H

iLtLH

4*

0

H

0

tn

»-

b.

en

e

o

r-t

O

• n -5

01

10

0

a

«.

3

ü.

4>

c

I

E

I

(.

3

t

0

I

£

LJ £

c ^

H u.

c n

••< Km

u *» a>

»- c e

e -w

<* i-i

^

H£E

U I

EEÜ

IIX

XX3

000

i I

o .1

w CM I It a» « •*« -H

L f~t > *»

* -H C -.

C X> •» -*

fc- -^ S o «i 0 e

• C I» u -o u <^

c k ^ -H

«

<• a «^ •> e «t ^

ft o w >- o

-^

f

in ti e a. u

c B I c »

•*« o <- o tt a

CO -n £ w

f » S «* ••*

«^ a« -^ M u •**

•*« (> 9 *»• C

ft o> ft > * O »• O O

ft c u ^ a.-*»-> •*> > ft

um •" e » x c ft -^ ti ft

£ft ccftuftotfteOfH u

O«* c o -^ i-«*- u -^ a. •*> i. o« cr> 3

ft -^ oftftft*H-««e-<H«*e eeOft

ft > ft ft-Hf-* ft u a. ••< «^ E 9 c o <*. ft-*i->4ftft

.. «- ri ft > > ft V X XI -HOO-OUO <- C C 01 O

v > ft go«- wftt« e u ft ft > ft t. u «- c

o •»» o •^Mfta.o a.o -o o ft ^4 ft ft c coo ft

co cft-H-m«*- x *» a> e * a. v^«*o •- ti ft o> »<

ft ft Oft^U ftftC OVft^CftO-^ ftM'HCt-

>H ft o< £ "< U H >> •»<•»« O OOb-H«* ft n a«

ja ft c uc •*> ** «^ "o c ft o «- o ••4<*<*>ea.

o o< -« ft o o>«- -4 o *- ft(.^>ftft(*.o.<> o o •- x

Q.OctflftCCO^-*' O" O •*< O UftOOOft O Oft

OCt-C O -^ Q. •»< U ft C ft U <•> ft C -^ »- O. ft ** C * •

uOooo>ftc jOftS-M^ftecftft^fta «• -H o c -« •»*

ft-ncftkk-HQ.-Bc «i ft -^ -^ 9 -^ ft a. i. «* c -** -^ ft c

• -*** * oftftftftt-ftfto« » o-^ 9 o a-»« t. c > • o

JOO> O-t-^ftCCOlftOfbuft-^ft-^ > O ft O ft o> >

c«^ooo^(-0(>(.ft«*g>.««>>«uftftft>HftVu>eft

«» •^O * «- o a. c*-!-! o*^-*«-«^ ocJ:cf>-icnfti-<rH

ft c o>-^-»>ftftft£ 0>«.ft***>-o«^^»*»*>u O M M «^ ft

2^ftccc>^oi-^*.xica-oouc o o u a .c o <-

o i-i -rt o o « «••Mfte*^ k o o o> c c c > a o c i

(- 3 ^ ft u u • 3(-ox« a> ft c w -^ -^ o> e»oo-Ho>

ftvftft«io*ek->4t.»4oer-«eftftft->4k>»«* « o» ^

VKfft'-i^c-H <- «. ti -o ft o> -Mg-Xf^^PicftCftftftc

ft O U£9 90X ftllO'OOe «-XC"O0»H»H^i«. c »- I- kfttXT»»U-0"OUO*Cft«-ft«.ftft«.ft*.-M-»«-«0».Oftft^O

ftf «tftft flcb^o eiMO£«*^4<Mftoftc:e'Oft

oo ££0 tie ft-oftft ft u mmmrim-^uu **

ft » o>uu »ftsa^-HC^ a«i-«ftft •-• o o "o

£ i o> c m « a> c f-ic-oo c c f o> o> e> o» e>ftftco>

»- x c -^ c fH o« -^ o>e-«4a>o>«*eeeea>eFH^oc

0->4XO>0>-MO>CO>0>0<OKC'«>>CC »i -rt H »< c •« -^

•• -o JC v c eco-^co c e-«« ftv->4-Me>a,a a«* -H ** o> o> a* r->

8 ifi^i-^-ninu^-^«--«! •-•(- o -H «- c c o o o e kce CCJO

D>-»4«*ft ft-«49-'4«*9,Ok9 c »-« O O •»«•-• •-•'H ft O ••«••"< -^ ft

^ c -• c •-< M -« o u o r n o « o> a. >^ •-«-** ft ft ft • a «• •© •»» »- c

» -^ ä ft > > JO u •»< * < o » o o ft -«4 M o > > > ft ftft-M ft-HMI

•*4-DO-oftftoe^*'C>-^ft-H-MOOkftftft^4£(-oft9ft

«•« »^•oQ^uitüv-iü« exouiH-»- •ooea.o.ft.OH- o-<

«<Plil •••III •••••«•• •••o»» •«••••*•

» fc- O o o ooooo ooooo o o o o o o o o

ft ooooooooooooooooooooooooooo

ooo^Nno*4Nn«ino^r4n4rino*4Nme<4Nrn«o«4

oo^^^owf^MCNcConMrororno^^^roinioioioO'«

O

o

B-2
.5

I

I I

c c •*

> c

sg i I I :

*.£(*.• »» -H X T» 5 . »-9 • C

» o> coc ^e

oi w -o > cuco> « «» «n * *» c

wo. ft c er c o -^ •- c c «< «- > O 001

^x uooeo *! «* C • '"• P'lS"!tE a

w

ooi^c *» *4 .* «* u 0l u c «- * c » o » cc

•»* «i Oi vt c •»> o o -< H «- .^ «. «. o -rt oti

fc.>t>-*Oi*(>0 tit. flint- -x u o v«-»-

» CL tl U

»o»o

U 01 CO. >0ip^«*0l <«-> o

y.^.M ft »>*><.

*>ft4J-0»-»>-PftC "* UO-HK-I ft X •-• fi O. •»< c c ft

S>«JftCX.rtft^ft C I- C «- > ft C •« » ft ft Q.

•wftftXi-i in oouoauin cr>^ ft-^o**r««**x

rt -3 c ft •HKOlOl « <^ o JC oii o> eft -c •« «> ft > ft

ft ft JO A IA U <- ft -MCCt-t. C -HU •** T» > c ft -^ uo

2 is «. o »ft ftu * c u o o 9 9 •* itu p > *

SK-I 0>00 * ft «^ ft «- > ft -neOflCO b O -O > Q. Ul *> O <*>M(>

o c *4*c.c 3 u< ft JC > t. > m .4 * a ft ^ * MC c t- ft

ft o cm« ft o ft ft x: or c m ^ in m -« e> e oca c ** ft c •-

t.-^ (•t.t- eat-c UU-MC 0cfto<-in ft *< •— o«-o«c»-uo

> o ft ft t. fi o e c ^ ^ -^ u ft » ft t. M x: a> c » o c c ft p ft

o>u ft c c 0»o t. ft x: •- c (• c t- o m »* c t. omwoks-oft'H

c 3 »Ht-t. ft in «> v o. u x oxft <• cr <• ft 'S ? *• w e ft c ^

fi i~ 0O ^H ft t-i «- ft -H o.i-i •-< o> o o t. c c c ft -^ ft -o c ft v e ft

***> ft ft ft a- m ft ftO<**ftfto.ftt-eft(- mo"^*-*-*-4o-t-co^-^ox

o m -r4 x: 'H ri o o ft > J: «* >uo>ft-Ht-4o. no oo • «* o> o H u m •*> «*

u c —• •»> -P c in •i *» m ** * -n a. c ft«-inftft«*i«0O ** *o x i

~* *> 9 oo •MC«*« ft u -^ * e>«* Xrttyftinfta.ft'^^oo kftccftiAo>o

t- ««.«««> * ftuftciflcuucuftocin<*>c -H •»« « a. U ft O o c »

jt **. * o Celine«-« oooo^o «»•««ew % u «»• o ft « « c » «t-x:*

o o * wiDOos 9 OL « «- » ftc<-ftftumcO'**00k%ft-»>ino a,-a a«

<*- if>inftft'HV4«».ftinva.'-(-iAQ.0inuHOifl'**-«4fcft -« ^ ft in -^ o c fee

X t->>+>**o>i-<«-ftft90*- ftfe-MO«* «*.ftn>>mo-*>Ct.«J O ft ft O ii

t.«*9ft-^-*<o o c a. ft u •*> o. <*- ft (- *- ft f • ? m •- »4 c w * »- > c »- ft o «• .4 m o. c

o»-cc*J**«*>iirtco»n t-eftueu **«-ftu«-.^^ftfcoeo.«*»c«-tfit.

S.f-l->4f.UUkee<Pt-t. ft ft (- C t- O T> ftCft0««.'-«Cft ft c t- O ft o ft o

•o^eoftfto » o r-< o o-e JBO.0 «- x» e «- c -o c «- a. o» ft -« »-ÄO»^pwft3in««. <- ft

CjO«-ft-3->^-inft3» U •** ft0CftQ.OCt-0«n^T>Ä0-»*C ft C C •** >M

O O O •-• ^1 JO C ft • « rH 0> O O"-« ft O rl 00ft 0 ft fl a>r-1 -M t H O O'-O

•-I ft oe-iHk c ft o> c <-< e>o> ftf-ia>vo>'-i<-*o|«*e * o m c c ft

o> -^ ri or< o.90>e>-*4 e •>-• o> o< o> c c «x ^ c e c •-• c 0 »< o> 0 ft a»-*0jc

eo eo>o,o> ece^c-Hvecee-^-^e o»'H-H-«0o»^bo>cb^o>cN •*

••< > o> •«< c c c o

1

•'< -^ -»< 0 c xin-'4C-»4-»4uo.-«HO>cininio> ex«* 0 •« > e -^ -^ cr

> • C • «S •«* •*< « 4» N » ft ^ ^ S > ^ «• > 1 • > •-M 1 8 8 • <M ^ C & 9 Q »-H >• V e o> •*4 fi«.v*>-pHUH^«*in-4«^'-<*,

ur-«ft«».-H*»ccc » «eat-cc-pftc'-rtc 9o>>>oo0e(-fti-iOMO(--«4OuftO9ftfta.uo>e>o>cpp(-QOoOHO>atn-M

ereft<^>H^ft0i-4-H>eaoe>9^>e->ui^9o00e>oiusi«fte0c<»*

u-»<>cftftu<ftft*>cox-<oc"oftcftft.oft,o-.t-»4-««-'4«-<fteftft3«-o»-fto

« i-c^<cocftOtn3i-<ftuiox:i-i ccn«waoieciQO«--gLJöujocci u o o o en --.

• inc>***0u****o* •••(j******u***m2*****ib****fic

o c 000 0000 0000 000000 000c 00000 0000 * M • • • * • •••••• ••••••• •»••• *H • • •••••••••

O OOOOOOOOOOOOOOOOOOOOOOOOO OOOOOOOOOOOO

NÖo»Hf5Mo^Nw«ro»2Nrt*o^f5rn*«'«oJlNWOO^rviM^rinO'Hf>iw»o

«ooo»i-<^OfCNNNOMr«iroMO^r*4r4r«r*oioioiooo^^^-<*«ocNiMrvitMO

^O^NWNNNNNWrONNMN^WfSNWNrilrtMWNO^WWrOtOMNWWWMM

^ocC^^^CN^^^J^NvH^^^iN^^^^^^N^ivH^oro^^^^^ni-i^^^ri • •• * • • • • • •

B-3
i 2 r

•1 2 ^ £

> c v o> oi a. m

» c w w u c «-

«ecc »n * -HM ^wti

01 > 01 o oi -? u> c 01 <- c

•-« 0» > •-< •- Ä 0» •- C X> -O Ol (•

c —• oi •»> 90-HOO o» «ooce

s oi -^ o v +> a, .* c o -->«<• oi

V fc- ^: •- Ot o »* ^i > 0>-xo>J3'^ 10^4 ui

IT i u oi o -rt > o eceoiiiioi k

I (- O OL O 0< «^ •»< «- > •*••*•»< ->D.Ci-«i-t** 0«

0)0l O > »- c -^ > ••< -rt C O C f«*OC c

•< C (- C O. ^ U U UI 01 -^ * U m* It *> (- tfl » «-

o t. n o oi w c oi o OJ .c o o > o x: <«- a «- o

UO«^«*^ (. C «- Q. »< -P « «.»- »«».OXO'- i

Oi O 0) O O IA O»-** *» 01 > O Ä H 3 i-l

«i oc «- «*^0'4>co«oivoioinoier>oi «*

So-*«-'«« o O-PC-^-H m JZ J: ^«-ck«*. o

* 4»c«i oi 9(.e ».*> >« e-*><^«*'oi«iiiH (-a.o

•*4 O

1

'*«- "-< ^O-^ O U U C t O 01 JC O 9 •-< K»«

u» er «- O Oi 0« O <«. Oioo>O0<^**-ctf>»>cr X » Oi Oi

WC -r* *- C O U > «A »-* <*>->4 01 O «I o *- o tt -o A 01 > >

3 — o.**^ «* e m v> x» o> v » «^ > o » ». «- o» u H u

u c m o s • TJ oi (> c o o oi «umü o o>-o -o c v ü

«(• o • oi x Oi oi c «••oio» •M-'« «•üioi*>Oi-P'^0(-P«ieeoiO( e>ua

•»< o o > u o u «- -H o» c 0(CiAHM^uau>ooi'4 0*Mi'4 o> c o> x

•o oi * - «- 0 cep oi c .c •- • n.*» oiino»oiM«T3c^»3«.«>.^-9 0(

ri O V « W O O <*- *> T» «*O0ll«l<-OQ.Oa.J3-**O9t-C 01 9 0) C ^

o><^ r-i t- «- o» £t.oi xSm oi •-« A ^ a. et-ooeu-x« iix9xo oi

90 0) 0« > U O O ex « O O N O 0» X 10 O «'O'X^^OlUOlUUOlOlx V

0 >**-C0l ««- V Q. C ^i«.«.f-i0»H0»*»0l O «« » JC C •»«O'-iu«* 01

«- c vwoe •-«. * a «. o> C-HQ.« >o>«»uo.o*»**5*»«- c -• »-

JZ O r-i C u 0( 0» C O-HO OOl t.(.£-Hi-4->4 ill >riekO«».0~4 Oi

«*••» 0 e o. a. Oi ••# oi -• O«X:J:00IU*»0*» •OiOix«».fc.09-«*O«H >

<** » <- -o o, •« c -o x: TJ «- * »••»>•»< •'<£ux:x:x:£ii»ooi>u «- JO o

mo * ** c *. s ti o 0» c ^* o -H ijicuoio*»^*^* q. o« -e w o o u

01 o «- 03 «»«««io 01 «*3Ol03a>>>*^-'40*5<-<OlOl^«*.

U - 01 01 (. A *«- «> II«* -oc *i O » »«- £ O» O- ST ClttlX-OOb» 01

Cot C XT -O •• Ol O 01 H» Q. Ill O C%0>->4 •PUQ.03CCCOO fc*>eO0l^ J3

OI ca. «- > c »/i e *- Ti oi o o -i o « c 0 > 0 I-I ,t H «* .< o> 0i o> a. -H

-xo. e oi «- «- -^ «pteui o»rt».uco»oi0i"»»«>^>*» ••> e ** x: •** ui o

«- o • o>*> «i 0 ui cr m o» C3> c c -i »^ oi**£££00oeiW0-.-io«»ct)-o -« > >

0i otCXC 0i Oi C 0 > O O 00.00 "9 ^ »^JC.-i^^OuC •**•*** 3 e

a.**- •xtib ri -o •H i oi ^ *> ^H • oi -P A <*- »99 9<-ixu«a -e -a 9- o> in

X O O»«» 0 N«*.«T»#» O-x -O OO^**1^—"""»-^?* i»BCO«*U oi e0Ooiw<*>ar>-H^ # m oo09-H-Mv4flo.ieiAkSoike-«4

>. ^*>«*>H»-OCU0lDOO o o c <- m e«*«*«*oo^o*-o«***OiOo.

«u C'X o tH»««««»«*«» «»«»..Hev^iiioiA-xkkkXo «o e e «- u o

Ml O -rti-«*.o«C0U*> (.OC-»*V«>«'0OOV ^U«*0-M-H0 «*

( «< o^oetiik-Mbnuiiii w ui tn ^ Seceo ^ o ^ 0 ^ o>

0» -O >o**H0l>IO«-Ol«.».t.l*»t><-«. >^t0l0l0lk-OT3-aOO-«40l«*0l0l0»«lC0l

eo c0C9OiovocoioiOit-oiOiOix:**-o««f-^ccc>uu«»HC«x:J:^x: CiO-«4«*>|!U<-«CC «.CCCOiCCC-«J-<OlOlOlOll«l0O0»H-»<-»4C'-««-*»*»**C*>

Or. OC-Q oOOO'-4OO0O(-oOO3HO>«««tOlOO>O>O>OUO uoo>o>o>0e>

•H c eee«*-x(-Or40ioooooo xi c (eect-oevo^ccet-c

0> kT3 O O T3 > rtf>4i-«Ootr4rtO>0-HOOOO->«-p«->4«»t.iie<^ ^ •«•-< » •»<

o>coe m * v H o> rt c -• £X!£'e«*«*-**oo.j>-«« -^ -o -o "o -o

C-HO0O>OO>Ue>l-l«*CO>a»O> O>O,O>»*'<4i*>*»*»"S.0C0 9 «*90CCCOc

•*4N»4 C COCOIU-^CCCC^CCC-^OX O>«-(.«.^o>»0C«00OJ=O

«*•«« e>-Ho>fi-a-*> a •*> *--> * c -> * ** u > » o>o>o»c-p***»'He C-PHSV-P «>«*«>

••>iiiio>e<**e*>i>»-Hi-o«»*»«*fi.*»-**«*O0 eec-^ooo-ft^-H-MN OOM O

OOC-»40itoO>O«*«*4*l|l|lllllt-i||llll||lk 0> M ii -^ • C C C ^ >^HO'».*-*-o<«-

r4f-x«-H>S C>ftl|l>»4^«.M-HO->i-'4-»4 Oi O« C 3>1*-S6 OO»0 0»<«-COiOl OCOl

o.o£-«4<oi-o,*4eo c •-• in «iino m * in *> c -« coo o • • •Ci-i^uo-xwv-x'o

xouqook-oooo-xoinnciiiiiioc^seec^oSo-xooO'XleBcae

ui ui o oc x (n a. oi u a. u u««c 9€<CM «- o ^ ^ ^ t-ocao^ocKu.«^ 555 5 03

o oi o o s c o it H c e

* • O • • « It • • • It • • O » • « * W ^ » » (t • « »(Jaavfl^** •}£ •

00 0000 000 000 ooooc 000 000 0000 000 o

• •••••••«•••••••••••»^••••••••••••ik******

00000000000000000000 000000000000 000000

J«NO«Hr4n<ro*Hr^nO'<4Nno«tCMm«roö«tr4roo<Hr^roo*tNnoo<tNroo<H

rono««-4r«oioinno<o<«<oo^,rKrsrsoo«to4*torMCMCMor«>ronoo«4»4«tO(M

roro*MwwMiororoi«»'OnnMr>«.wwrnno»t^*«,«vi*V«'r»«,**o»<»ioioNin

• • • * •# • •

*t «•« «t ^4 ^<-i »-i «t in«^ «t

D-4
0> 9 • «I

C C (- c «- o

^ O •»> O u c •> «^«*

C ** tfi * * O O k

O U « O O. »» O CL U

•w • » 3 u o. c e <•* o

S

> r-l r-t «- «4- H O O O •* "O O- *>

«> -o -4 m o. c f> <-< -H on«

u * ^ c x ex o» o a. u o» c »

0 a«i-^(-o>u a. o ** -M

~ -a C«-HH a <*- t«. o a

» O Ol T3 0> •( IA «^ > X 9>H O t.

U CllOtl'Oe > »- O -rt fc. Q. K O ** TJ

«itt * 3 m * ^ c o >-4 at o a. (.«i m I/I •«

•-> o "O 3 3 «• c •« » X a; > O ti > «)«cuai*>

jac c« * 3 «- «> u *- -o -^ «^ «n > c o «n H > o

o ai x> »A» o JT o 3 * ^ o c ii-H.F« o » ^ M

9 oi xs is oi oi > > u > N oi ai > -t o oi *> u» > u

o» er 0i TJ o x) i-i o» ^» ^ in -o «« > -M oi »- > -HUO

U 01 -D C 01 O «> O C^tfl^b-e-M «* -D «J (> 01 ( 01 if)

c »n oi -o »* »> o o> oiccoiocoio» u in oi £ oi «•- u»

r-<oiOicoi«nfli* •« •**> 0>eo -9 S5 uOi

(.0 -p 01 O C 01 (- 01 IA «- » U» 01 £ * > H US > U •• XO 0IUt

OU ilC U 3 •»< 01 3 C i-i • * » n C O«««. O •>< *> ^«* t. -»» 0i

«^ H ft 01 "O 3 X> > t- <0 O "D 01 »i •- -^ CO O» U Ul 01 A OlA>

t.o. c "o —i c t» oi ^ 3 ••« c «- » o -H o -H « w ^ *» H -^ «> m JC o M

Oi o c x> o oi u > -a "o *> o > u cm ii in o *»<*. c » vu^cuu«*

O. -P M0O U O -M 0* S 0OlO»OO«»-«-T) W 0. 0i C M H H 0« 01 0 N. u

•-iii o (- •** u •-• ooi«--i*»<*-oooi i«c><^'<-4U0> «- a. o m 3 oi

o m oi oi -^ «-• kQ.uo rH •M^40^a-f4 oiOtin>0-HOooi*-"4 u m v ai <- -*?

S* •>4I-H0A a. o »- o »«x -*4-4o*in^coi-••<£••* <> a. ** «* « oi •>«£«

«* 4B A > o -o a, •.ina.v*>v-'<uo >*»OU«**»*«*UJ^,O*-OI*>-»»O

8 0001-4 "o c -o «ica.«no0ucoio »-H y m *» * c o * »^ x: -H

M^ -^ c o c -0 coo Äa.oooi*>c*»oi*>-5cc**o 0 3 x: *> ^ -o •

u > •«4-*4«no 0 oe O-H • •»» • «fc Ä C -H u "9 u .o -•< o u w »«* o •> 0 w «t.

•MC 0 a oi > m m 0 u »» x ^ Q C veoi«vOOufi e u -x o 3 <»> «•

0.0 >>U0-Binoiw •^t-».inu0ioio«c«--5 0^ (- ^» u -H 0» 3 «* a- u 01

00 00«- o 01 >-• 01 m ti x) 01 o 01 u coo« «•• ««»»«»••Max«* 01 o.

* 3 in £ -PI •»> <• 01 <ac><**«>tteo-H>frov*iw-»4Ci«*.o«*uxI**w(-

0* ¥»«nooi*>-pH3.rt o^ac(-o^coi>-i o»<-0«c«fc«-»««-i 01 r. u •*> •- x

enx: «ioiinuoiH>-o<*> o u -< •»* oi 0 £ •- »^ -o o» c o •« u'iao.ooimo««.

c » uu0»t.«>H0iM «* 01 •-« -M cinr-«iiooieo0uuoo*oc«^aoiuin

i*^ 3 3 o TJ •»* o o •H 01 v> ••-• 01 0 ou ex: uua. » v «- •-• 01 -^ 01 v

00 oo0wieuo»-**int.oi<<u o.£ O-^-HWOUW -XW« * t- o.« (• e A 0

• *-MO»0O auüOÄ^'OX^^fc^occ « »v -o •»* -c 01 0 m

we oi 01 "o (- c 0 u > «^ v oi 0i'xcou«*oux-»>ou<*'a.c>-4cfl-o

eo «.«-41 weoe cm -H m m «* U > u 4: ai u u x: 0« 0» 0 •* -^ o

o H m «in4;0Hoco c^uoitnoi-«ooioifl»voi*»f-«otoi*»i-i*>o>-r«i-*>3*>

•*>0u«ie*->-H3<**0*»*M*-Ht.a.«'o<*>o «nooioi^soi^^ifto -etc iux

0Ntcfto»'««'**o** «

>

^

-,

-.**•-t?<»,^ *•< P *--? c^^-*o«*'.** T4-'4 3c«X'4Oii>cu03-<-ie o 0 > 01 c o v 1«. «. -Q o c "o x » 0 x c m » •»<

ocOoo.«--«uooirt3U'ooo*>*>rt-Ort**oinoi eo«* e o «- ••* «. »* o o -•

oomm-MOirioain^—i(>cuv •»< > e c 01 1- o 0 •** O0.*>*-<-IäOIC«-£.<

ino<oi(-3'*>u(-oioia.0xo ••-«U'H«-I«-P> > • » 0 •»« u O.H & ** «

in«-*-OlO-OOQ.I.«.o.>OuO«3-x00<«.3 *« o «MIS wwuxi-««xo 0

00 a. 01 • «4- aaoo 4»3r-iu «-oi'Hin^vinoic'xwoie 0 x 01 •- £ m

0> 0> 0> » >^30O>3(-3OlU ilkOSOlbO-** 3 *« U 01 in

0i0lC0l0t0l0lC0l0l0l0lCO33U(^c-O U>fi4*y9-HUU9-«4MlM M *••* »

££ *«££££ •MC^fX-HfOU-»« •»<0»fcHX-3c«-"O*>X«-'O**3- t'» «n £ U u

«*«>e4*«*«*v e«*-**-P**e3£-M *• c c u o •> •»> ia 01 9 01 ^ «- 3 0i •»< •*> o *> *« 3 c 01

••4 -w -tH *-<-O(>O'S.«-UO-*>Ou-0<-OU'O O <»• U U 01 O.

9a>Oo>90to>oo>o>o>e«oo>o>«-9 0 •- -0 3 0« c«oe3woef'** » 01 o H m

cc(-cccc(-cccc*-cc3uo>oia.cu-->30Ci«-outii>oa*oino.a.-»>uo

•« «•«4«*-H-|X-H-r««*-H-H-fl-*44*-*4f4U C <-• 0 XI 0« U <- Q. U «-CLUV^iCinin »««

«V » "O "O TJ ^ T3 -O T3 -0 "O •••I 9 Ol O C 01 Ht-HltOÖ 0>**- O.

ceOecceoiceeeOi CCOI£NO>CO«£ o>0>o>o>x:0>a<o>Xi-«0«.i.oiOi

0O£0000xO0OO£00-C4*»4e>4e*>o>9eeee«*eee«-«44*fta.-o-ooi

mm «i in in m m m in m mm o>0vo«>a>-H'H>,xxx9XXXti0XmiAwx.*j

«. «. o- «- «- «- «. 0ib«. t. b 0> «. «. 0>c**u'-«-xc**c<«-«».«^«^ c<*>«»>V£<*.<«>mm m<^

OlOlCOlOlOlOlcOlOlOiAC 01 0» C --i -x 01 01 3 x •» H rt ^H ^ .^i -M •»< -^ .-« >- -x « 0l Ol ^ •>'

0-0<IXI'O'OT3'4'O,0'OXI>-I "O"«»*** o.'-« > o

1** »^«^^^^^4»^«^ •4»mmm*»m

cc3cccc3cccc3 cct-a.0oioiuo.«<oioooo a. o oo*-Ncmmmcoi

SSOSSSS 033330350 outnoJEoäocxcs OXCZ mx0lC<<0ile e c »-• -o TJ -o Oi «^ -o -o

o o oooo oooo o o o oooo oooooo o o o m C ooo o

• ••*••••••••• ••^•••••••••••••••••c******

ooooooooooooooo oooooooooooooooou OOOOOO

wwo^wn«ro^rjro*o^c^oo»HCN»o«ro^Nn*io^oo*-ic5w«.oo»-iMroo^

CM'CM onnrorooY^r^^onnoo^^^^ocMCMMrMCMCMonmro ©O^»<»40(M

iniotoinioio«^«ioininio«^0'H'0-oy)-o(M^<0'0>0"««ow<o-«'HOOo»-i-H^^fviw

*4«4m*4«4«^«4|n«H«4*4«4in«4«HO'<0<Hv4*1«H'«*4«4«4«-lv4«4«0*4V4«4OO«4CM(MrMl-<r4

• • • • • • • ••• *

•1 «4 ^^0»H »4 «H Ov4(M CM

• • •

-• OCM

CM

B-5
I > * Cf> > *

^, « ^ «> c > «* *> Co •»*

«t a %* •••« p« ••« | o '* g

*; o S <** -^ <* o> u w (-ai ti o- w

-7 t- • w > o rs d »^ -o V •**- Oo. •«•-•

40 Q. 01 > U ••< V ^ * * tl 9 U O ri

f tfl O. .«J I-XCUUKQI «4 ft ft o >-• X <- •»»

cuftcuo<veooa«ft -^ t» ft •- c a.k -o t»

VO-H-MOCCI-Mft«) C ^ ft C O Q. > O T3 w

(.« *>>>. xS^VOft W-O ^H 0»H p M

ft «Hfto-ce »« >»- u«> ft to o > , ^ "o «- -i -<

o.(->e>c(-ftftftea. c - o -H ft ft o eu o > o a. o o -H

0->4 0(>0^>9ft O •HOC'* H 03 "»^tC -H C •* X

«*-<«--^cr-<aio.^cTCx: • a- xi c » ^o H t- •© • ft •« -o «^ o © x> in

O U O »-!« > ft U «- C ftOW-H ft > V ft > > ft ft O ^ ft ft

o< o • o> -4 u c <-< o OM 1-H>H> a U« • H -M "O * "O C ft

Sc c^vft-^aft^c »"? 3= •S01* •»•Hft u

c o> x •« o *- «4- v-H -o u o> •*' xi «- •»< -a "oft ft "o •-• -o ft x> e

tH-^crH^* or<<^x-H(. ft o c 9 c u e * ft c TJ ««. c c H -o c e o

> •HAcmaftvvo o. «- a u r* o e MMTI e o 000 0

uftc-Hftft r-*ft<* ** "o •** e ^ e ft . * 9 "tf * s

o f- *- "o m a.-o ** > v o ft m in I-I m m -n • ai * o m ft «i ft a> a* 0 o

vooftftxom-i-iooi Oftxi -o c o * ^ 0C ^. »•H ^ OC-* « « •

ifivft^.^^oo^ec -> u ft o-Hkc o MPH o m c •*' o '* ** o «-

•««k^uo. o- u u o H w c ft £ > ft £• a m J: >> ft w £ ^ a.-^ x: ft

«*ft «ft o • c ft v o c » «»- > > 3 • u • ** c i > *»-oo> v a

oo.ft'Oitinfts H-HOtt ft o (- ft a» rn t. H H ft ft ft M ft -a ^4 «4 01

«I .C^-»vO.O«-C7>OOU »-in fi Q • 3 »- t. »* ft--»' l0ft4* JO

.K«>9O>.C £e«>(-einOft w »> o> u s <- x W-HU >U O

ftin o c > o > «- *>ftrH«t--f4 ft>H0C ft ^ u s > ft *- 0 o ft m ft o ft •-»

> o » u -< «J ^H v H ft «. » c ft »* ••-• £(- u-H ä H o- ^ .c a -o v JC

«1 ** c o>o TI u «- u > ft rt «> > ^ c «>(.« ** » 0 ft m > o m » v

•-• ft*»o*>«Pft*»ft0^ftca.> a» -H •- 0e£U art-«* ^i «1 «> o

XftSUC U » C Q. > V i-i M (^^•HO «t- u t. ^> 0 «toft C «*- ft (- C «to

u-eftftoftfiuftmaft JO-^ O oft o o -^ v o (- £ ft o> ft oinm

0-'4r*a«>«*ftao**«ftOu ft 0 -« ft ft s m . » > ft m > ft**

> a u> •- m «. ft «. H > •-» o m «> <*- mx:«-« v m m ^ « m -o ft ft m •- ft

ooao>t**-r<(.F4a.f4*-«« v CH ft c > xc Sfs C > c > >

««<-nl.U0l.OQ. M ^ I« X: in O in £ £ O t. » ft O ^ 0> O »- -M 0> O-Hft

o.va.o»«ftuCftU*--Hu-** —1 n u » •»««toO-H> n 0 x c -«4 o «* c •H>f-i

X ft > ft > n (- a O 0 n c > 0 I-I * o «* »- ft «* ft > n «*«iu-*4 > H

t.*>o>oin»-uftv,0««.-*»axft cxcrjex a. 00 a.^-*4e o. 0 ft e o.*» x

OOO •*« O X> ft O. 0 ft 3 > n ft 3 «to o n c » H a- o « t. ». 0 n «* -5 (. o n u «-

m c c £ £ o* <- a« ft o>*> «- ft 0 »- ». i-i -IH «. o I-I i-i c n t- *- 0 0 •-« ». •!-• ^ 0 •« «. o o

inXQU C O -O O »- C •< (- H u (.«»^OUH^n-H-OUOnft-pu-lOft-OU v

ftft«rtr-«.-£-^cD.M»-io>oo»"o in o- o c ft m «* n • c ft * v ^ •-1 ft 0 H ^H ft m o> u

U-0 f O C U 3 O •OHC«toCftftC«toft«ftOUO«-«fcOn ft ft U ft • ft c o

ft -< O 3 n .H \ O « O «H U H -M •(-•*< •O^t.«»o«toOX*-*»«ftVt.0o"OV«-'H«*.

c > > * o in u «>>0>-»e*No.>*-»-*«iiia.o.«to ft0a.-H0£0a*toc-'4Kia.ein

o mct-ft o o «to ft .0 t ^ a* A "o 0 ft «-' TO •-• «to «> "0 -0 -o (-•**

o>t-xftftv->«jrvo«<- Hooo-onooo-oTJÄ«- o "o -H o-o^aoo-ooo

CCX«*->4M «*u c a. v x: ft£ft£ •»> x: ft o u -OÄftuftTa^ftdi-^Äftftin

»4 «««»(.k-M-MX-H uuflr>4«*to4ufl fj —• O-JH»*»HO«..M«>»-<X:O *»>***

co^«naio-4jc*>cftfto«to ftn0vinftn3ftJ3 ft •*« «». • n «* <*. m ft n ti

£

•H«'-««>o.M03n •>«(.». (- ft a 0 t- ft • 0 o ooao-»oio t- ft • 0 m -P

A -I x *- 9 1-10 •- ft ft x: > ft ft > ««^ft •»» » ft ^ *> «*ftft> v » o

ino«*fta.o-in*4t-inot-a.44o>ftk&-»4o>ft<-ika 0> ft *- o 9ft*>&p«p>ft(-k

» C U ft .0 -*> C U 3 C-09 >C-003«-inC"O3 « c -o 9 «fCVOV

oinommm-Ho o mu-o-H m u u n m o ft n m a * n m u u *> » m

«^ n % « % « U « .-iOC-<«*>0ic-i-«ft**ftOcV>*>ftC«to>*»ftCi-ift*»ftOC

o m «- a> > -I-» * cm o ui*»-H«to c^-i*<«to-">c«-«t*-»4fc.-Kc«-***--H e^-H^-jc^-t'O

xOeuuuS« -H •HOft.Q -n^iftA ft*>ftÄ^ft«»ft^i '*i«Sjo-Hi

«*u«>noiftft->4«««*uinou«<ftaouoaa0O(vufpo&yp9Ouo#a<-«ft

inftoco.a.ci.uu0ftft4>ft ft ft *> ft ft ft ft > ft ft ft » ft ft ft •*> ft ft ft n -o o^3«><-ininini-iftO«-H a.-9^0 ex ft <-• m H u • ><-< 0 u ^ «-• m 0. a« •-« m u

u •-ioono«toQ.u «>00Aa.000<cp.0«*0-'4«a.0M^Ao.«0 m ^ o. 0 o in

ftp4ft(.(.i-ftin »-He o « 0 c >0o»«c«toO«oc«toO0oc «* a d <«- <-

ft«>Hi-ia.a.Q.0Oft#>>0x -«»vox •»«"^>OH n "s o n n %. 0 x -HV O

> O U (. * O n ^ «to ft ft .1 «to ft ft-HF-IUft ft o« U ft ft ^ «to ft ft 0» >

o<-ioxin00 xo-O-H^a-i-iN** «- 0.1 N«*«-*»Q.ftN«»«-a.ftN*>».o.HN*> (-£0

«3«to«to*Ulin«to i3U »> X U o VXUOU CX X U 0 Q. X U 0 ** X U O -P -P

** m nftaiftH«>naaftc-ifta.ftc-4ftQ.oftin>Hfta.ft0>HftQ.ftc-4fto. H

««(.ft«>in0in<*'0«>(- mftoriaimfto^ft 0 0^00 a—tt^tia^oi-*»-^ inONC(ninircft*oc»4-ocft«--H,

0Cft«-CHXCft«-H-0cft*-H-ocft*-uH

tu u. xft««c ft»-uJU.o>>>-ic(na. > «- < a» a. o« > w < (n a. > M « (n a. > M « en a. 3 u

«TS « •«• ft -Hft 0 ft -co

• •0to4***to4**»0Bi Q** **0O l:4** **O*** • C U.

ooe 000 o o o ft 0000 ooooft 0000 ^oooo 00000

00 00000000 0000000000 ooooooooooodooo o

r5Moo»if5?oo»if5noo^r3M*o»HMn0-oo^Nr»«ro»4r4W0'O»iwin0'Oo

(M'Noo^«4iONrMr400«4«H<-i«4or4rMNrMoo<i «-i^^otMr^rMr^oror^n ro o o

•i»HO-4c5NCNWf5NCNO»4W«t*)Wf5MrOWrOO«H«r0,0

,

0'CVJ«'*0'*»«»»^0'0'©^

wrCowrjtNrif^r4r<rjor4r^(^cNr^Mnwr^f^O'rr>iNCNirNi<-rir>irifi0'(\irJrI(Noif)

• • • • • • • • * • •

W.N CM MN M »(M W M lOfM

• - . • • •

e^ w r* CM

B-6
MN CM M

B-7
u

11ii

CB ft

*-^Fe

oc ft

e^ o.^

ft -^ -o

£'55

t-l -« •»<U

3H

OW «-

«>•(»•

uc r« «-c o> *> e (- u ft e

>

ee o4e>c -^

o -o ••>E ft5 o>

> «-VCO -H Q.

ftOSftftftSft ^ u i ^ k =

ftUV ft3»

C -H 0>*» ^» ft

ft* ftO Oft

«*ftC«*<!-B£9

ecu* ft ^ ft ~t * "O » tt

•>4 ft -^ ft ft ft0

Y t***U>U•

«*o.ftbft4*ft«*

ftKftlslft •"Mliso _ _ o

Jou^-Pft.^

Sr >-Si£ ft ? 8'

^ ft OS'*

K Q. (.

o

«>

c a.o ftuo

ft^ofteftriS u . -^ ft ft u •»* c «- » o> e (- ** ?jD q ft goo

oftS^softft «*-efto.u«^ew ( Oft • « (.Oft ft^CO(-fti-i(-

CL a. a.u (- a. a. OL

o ooo ooo

oooooooo • ••• • » » .

•^iMro*o^fMn • •••••••

^^4^ -<ow tvi ra

• ••• ••••

MCMNNrMOIMN • ••••»«.

B-8
APPENDIX C

Training Delivery Subsystem Data Base

C-l
e

i r

-« •

X

o

u

t

o •»H

»H £

1

(9 5

>

M

U H

0 II e -^

ano •H Ü.

e r-i

ICO •^ VIM

flC^<f »- c «-

HO 2$

X H*S

< "C H XÄ

0QDO «^1

ENH. d"?»»

i V HÄ6

3X0 U 1

rtt- ♦♦4)

d4) £i«

H3» IIIX

dts 4)4»«

CIJH • »£

•H |H W3

iLlkH I 0)0)0)

in B

«

-^ ti > <*• 5

•»< «* > o -ox

•»< > •* J -o o

•^1 » .»< u c c <*-

> U •»* CO» tl

iH O U » ** "O

> o o> •*> C *

•*» ce>E Bib«!

U «- C I-* tfl -^ c 01 O C

-* C C •< C »• C > •

> £ — *> -< «1 «•• a« b .c >

•K > c »^ a» •»< -^cc**!» 01

•-I 3 -W »- * CO > «- * "O C ••1 U JO -Hlff

3 OCO-HO-HUO •» xoi

a: "o w •> c o> *> <*- > -o m m *» -a u -^

«« 0) 0» x> ^ o c W •WCbll*^ 6 o -t->

^ > (-01 inoioo>«>ii»<.e «- •»* •«

«1 -HU-ooinj:0(Xo>0» o c 13 0» * 5u-r4

•o <- uoiOi-Ht.4*a0ca • -• x 01 *- »« • c >

1*01 ocuuo« • •-• a a>«*u0 c «t o u

** c m c o o c c «1 c o ** c c -»< o o m > £ o

o t- in ino«.in«-.4xi««-uc'<-^>«-* **0ic

** a ~ 0aia.iA0 »> *• 0 Se -«J a. «- e^Oio» ^ u 01 o r^ 0011-1 c 0« -o c <- ^4 u ft o •£ e

SOi->-H «I <•> ^iOlO>(--<-4CO0O>«C 4* puox

m t- Oioio^ c c •-< sow-«« «**- 0» o » c

• moift <-£ oioie-H CO(>^«-O>-HOMO« > «-

cox-** 9«*cc£Oin£-»«3-*4 01 c > •< «• m iioi«0

o ** o v o c *> tn in u >'o-**t-M'-oc -rt u ai CK

•• *. «n • m a* o -< o cti^oivcftOrH^ ^ 0 JSCHI-I

<*. H ^1 o (.u-»>4Jinoo*u£0<oo»'oa u*t>a u o »*

o>0t'OiOiO 0t>4*au30x: e «000111 ox-««*.

c w xe*-«i«*oi 0 o><**oio»9«* x 0» c > m

otoi(-in-»>(-o.a*c a.^ <*- c c x«*c-^o>fw>> * * ^ «-

r* »r-i 0 t-oi oiot-^aimtJCMset-x * ** 0»

joa^^ott-ttai^motc o •»* -^ c Co e »-owe oco-xo^x-oOixeiA^eoex k*»**-** -«4 »«»0t.

Q.-« it. ** * -4 *> o ti o »* o» » c 0 u >w e 0

o c 01 01 01 o a. ui ^ d -w •»«•'«•««»Ot.« u «n a> Hi

ut-o-^inxoocobo-H^m-xs ^ t. c x o* 0» 0 o» c ^

8*>O3V-^«.'<*»0ii.,oo«--o 01 • • • c :<-) ü-H-H

• a. a. •>3O»»c0'Oi*»«xt.*» ^4 v c <*.

x 1-1 m xoi» x w x>«*eo.o0i*cv 0 •»< -Hko

8o>*>*>ino«*»ino»*» •**> •x o> 9 *< ^1 -e 0« > c > o

c «H Oie->4Oic-*.c*0-M«*e o. 01 ^H •« m »H H * x

«n£u>4 3tfiuM3UM3-p 01 N. u 0« m « l-o *> m v »H o

2v u c i-iuc uc •« 0» <-H in 0 f.(.M4CcuoiiAu c

0t.Mo0t.tfi0t.n3x c t. w 01 o oot.oiOFHOi

x ot.^« 0 t. o «. -»»t-ooit-ccm** o« -• 0-x

J-**X0l0lt.X0l0lX0lU« O-H«*0)t.|LkCM»OUOiCU

rt * rt c 01 > <-« c ••-> r-« ct.o<*.«*ee0o c -^ » c »- c eo-* 3 -H t. «* •*« £ •»< t. * •» u -^ t. oi 0» t. u M a. o> H .4 «*.

W 30IOO30IO30IOC 0(O O •-< ^ ^ 01 kl C « c » «^

x in x ot • xOi x«t.tfiit.%ü x<».t.vu(-M0i

«Ik tfi«*rH in.»*<^in«>r^oin'4V«i>-4«>«*«>«'<^ aw-* o -o

• t. o> «. t- b at •»> in t. ooo t» i t. *- ti -o m

m c eo>e>ece>o>ce> o"-« u -^ o» -^ x x x ** «-IOIOI^OX

xt-t-cc-^t-cct-cc uo><oee<**<****oa» c "o »

h-Or4-»4->4«*>-i-H-Hi-).>4-«4o>0e Ex xea»E o>o>

m x*> s x > x »* c •«4»e«>e>e>a>«'-»«eoe>eea>

•• i-i ox*, c x o»««- c e>«^ e-*40>**c« eee fc. •- o» c r4 .4 c

«n c 'H -^ ••-• e -H -M c -«4 -#4 is c o H _i 9 -H -^ •«« cr o «* >H •*< o TJ -^ tie>H«*0t.-W«>0-'4^0-H-HU> C t. t- t. C 4^ tf» » C -4 N

^CT>C3.»*"OC3T>C 3 > "O O O 0> -^ 0 9 0 -H .^ 0 o. O 01 > -«

«*«'<0(O'in'<0»O--i-'0l0r0»4r^j|c«-lAUItflt.c-9C0»4JO«

•*4T> >-OU-t-<>-OU>'0 U«->i-«*-»4 0CC C 0 O "O H t. X »- n

-i « OMCoo»H<Oi-4ca.o<:a:t.tfiiH»-iMo£<o>t.uja. x

o > «- t. t. u s c e o o 0

«4 o ex. • • » 0L • • a. « • a. • « m «-i « « . I-I » • c u • • a: ••*(. OOO O O OOO OOe ooo ooo oo Sft. • • • •••••• • • • »«M** • • • • »t» •• •

oooooooooooooo eoooooo oooo «**««••• .... • • • •••••• • • • ••• • • •

eoo«4Nr*)Oi-irMO»4r4mo<-4r4oo«4r4mo~«(Moo»4r4o C •••••• ^ ••••• • •••••••••••••••

oo^4-<-40(NCNOror)«o**oo^4-4-40f>iMoo-4-40

oo^«^*4«HCM«^<Nro4H«4*4«r«t^o«4r4r4NrMr4Mo^nnr4

oo^vH^^^^v^^-i^^^^vH^ocM^^^M^^oro^t^ro

• •• • • • 4* 4 • •

• • 4 4

O •"• " *4 «4

4

o

C-2
a

» -> IP M

••?«-*-» -j «- «i m «in u

x> c o « « o > -o •-? «latint^«

m ot- o o <«- -Mtt f« 4 •H->4a«r4«t e

«- > *i »-• t» x (- » oi * o •PiJ.HaQ.-^iAi/iai-^

« ai u • > w o c "o c -? o« « on «- o • •-» ••-i > a ««»««»eve

> tl & O. • • C O X3 «Cft IP > C Ql '•*«««* ••4U>wAr4-Ht.|pO

Xll -3 U •> I- 0> •> « O O (- »* I- >* 9) O.U- tfl U u ** * t- S > > >-< JO \l\

ja * ja iP1-i-»<*>oiP**t.-^ —< ip o> »-H a« -4 •»> cxoou(->cn<-iM su

£ o i- o ^< ip «^ co «i xcaiQ.aiip<*-aiwi ok^ «*«*«J(/)>I

er u ceo. «. > — «^ in u c w* •-• c -o x o Q. JQ 9 o> 3 -o *> u u <- o.

co u «- M Q. cr- • x (-0 cS^ccc IP 3 c c o> u COOOIP

•* c '-<«OCO-O»IPQ) oo9iHoato>A««(^cnt-(.e ef o. u «

cm * *- >- c o a.u m a)o«P*»c3co ^-((ffiflcrcra.Hai

(.«* «•• o «i m r-t * 3 •*< u «v u t~ # * in m o> ^ »»>oicc3*>+>

c «- *- *>*»*» v t. r-< > o o TS -*J (b > c > » c > * •x-Hut-kipipin

•IO •> c * o 0 o> o <- <». e exw ^CCCFI »-«- » 3o«0'""i-i •-• >

riQ. cr Q. oi»««*>rt««.cca/ ipucoo oo>u«^oo»- -< o o» IP

a. c •-• o> o -o »- FH u xiaioa-»'CO(-'rta>^aa.o(-oa.v-*'4->o

»* 3 ^a(inx>t-a»-e<aiip'^« * Q. ^ « ai «>-«-> «P x:oa.a.o.o-»>at-(. _» o

«.« c-»>-ooaiiPxic« (>«- a» * o -o > ato^ucsoo,'« uioo« -»J

0 .-« H o» u > ai (- «i f ai o c-ouuai«Ai3t-acoui^iP9—'"O o. a ai •

a.x oinat-^cMCoieu«'«*. no vxxo -H UI -•« II C O. O.V O -O

Q.I-1 C-^C-I-H c 4J -o H cr£«^a«*ip-o(*-a<**o>aoo »ossHi.ai

3 ** > 3 Q. -3 Oi C II C <<- ^ C IP IP O H -H 0(>CO-**VOeOt-WM><».-Dai

IP C O-eT'Q.««-i3*»rtt.3 0O »-,HiH«.«O>>"O«-0i-«fc. «* (*. ^1 4^. O -MU

ll CÜCOOOC O O »H»-i*>«^T3C0*<4-Q.+>««.»l-O -«4 o O I- X > C

x-M o i- '< 1 <*• 0. i- *> a- JQO o^t-tj a. «- inaii3«iox^>«>o.boc>

rlU w *) c • » O» O 01 O C O« O IP X Ca.-'4X30X-nV«|l-*>t- >(-e

** rt «>c.^(-cco t. <* ft ip c * kip'H >(-IPQ.(>V>I-OO > IP -o m ^ QL a»

c<^ oo-ootiu^j^Q. aw •Hw«*.>i.oino> a.>(->-M «-4 «1 oi -o a« »

•l«^ U «- « ri c .O «1 01 Xt(0'-<C»---i*-^l-3^i»)0>IPrttlt-<-Ort'0+»C

-^ 0) r« 4^ > -^ 01 «* •) « O > IP «^ D(-^>oa.-DOl/«'T3Q.kO<- -o »••-• O b-x

U * - O 0>OC«lP-3tlin IP «^ IP 1-4 O <*• X CX (• O • »« > »DO

•^ * x •-<£*•-««> i» ••-10 ^ c c cr t. v x CP u c» m Q.*»3o,i-«o«-ca«'

v- IP ^VO-**-H ai c IP 01 c > 0 c <«- «1 tie«*oe * o^c •-»- > «a.«

«^ o o c -^ JCOUUIP£UO 0iP-H(>o^:<^c(-e«*.i.xwii-»*3«-ua.iPi-9

01 m ra-ocT-uvcoo**©*» yi > > * *> o o *> ai «> c ^ «p -^ u -^ o C^IPIP

cr 0) 0/ C O O CO. CO m > U >«*'P4»-I-H (^MIX OI

«*<•* ^ > • «- «n > ai 0 cr> »• IP c • ti (- o x «.co«->-i«-<j-«t.«- -• «. u «

Ml O ^-"^O)—'«- >'OC*>CwV '4>(jO->-<<*-U<»-OO(-O'O0l(-UkO«^aVOHf

oc o^t. -HCO-H •»< O» C ^ t. >»- *> o e o <*• u Q.U. a.t»03<»-o-'«o<*>«*'«

« -^> «x: **-oiii • IP o f 01 01 u ti 0 9 (- x a. <*> u *• w ^

-ouuOj:ai.(.«*-o-^>c uca.«>u—•o-x.*>o>o>-*«4>tix * x ti x » « •«

»IP tio^-^ttoaie > c «--«otio*-cc«-iP •cc«-o-»*u«-o'o

O 01 • «^ IP > <^ V «- ti C OvlPOS-9-OOOi.UOC«- UUOOOOOO0

£-^ Oi (- ^ 01 ti 01 0l>OCUcilA^ Q.-io-0 9 0.-»**»O.Hti*'*iiQ.9«9cX<-t«»-

»» > «-^u>c.-tip-Dc«^«too*-»-«-Q.'-* er Q. a 4*u««&o- cro.

•*< ti ti ^ — £ C U • •^0ec90»tltl9«-*^9«>-«>eS39il^09**-<*-

IP > (-x:a-»>CuoooiP(-o>-^><«-k(.iP9x:c-DiPooiPOoo IP -o o x. « o o

• ^ >O«*IPU(.0-«4^» c ti c 9 «- M r-4 cr > •*> o • -o o o o o

•rt > C <*• tl tl 4* 01 O •»> b ^H 01 <*• (^ X X (^ X ^4«*«*<*- X C^ X X

-P U 01 aiCJ"5*>O>0lPlP»4tl-»*0Q.O'O>0O>O«0>O,0 0U0U-»*9 0'9UO»OUU

.-<o«iPc'-«J3ticut.».*»'oi> ec cccc 00 o u o> •« ti eoc 00

> tl -O H O -O (- -r4 c 9 O «4- 01 * »H -^ X «^ »4 H FH X 9 9 X 9 t» «OlX •«4 9-^X99

•w «^ > Oi « o •»> < •> o u o o • ccuccccucraruo-"?o«(-(.uccrcuo-oJ O tl tl «- » T3 OI .1 rH o «H * I C • (.(. Ok(.U(.OtitlO«IAC O» O« O (-tlk O Ol 01

U -* C tl C<*>fto1 I«*- T30ia<tltl9a<tt0l«l9-OT39'D9(-0l0l9tl-O0l9^-D

0 t) -C > IP O (-O<«.c-^lPC.»>-»>UUO-UUUUO'OOO'OIP>-l-D'OCruOUO'O0

U U * t> tl (PQ.90i1'H(.ailplPCCtlCCCCtl § tl C C tl

CTCOIPTl-wiPk a- 0O XXOO-OOOOO-DOltlVOlOltttlOl-OOtiO-OOlti

C tl «- U > tl IP CO- 9 C - «IIPUUOUUUU0(-kObkkkk0Uku0kk

•»49» tl O C C C - » C IP O- (> IP

CO-tPC-^tltlt-klP-Hk Ok(.tt00tl00O0il0Oit00OO09lQ0Ot»O0

»tlkk MfiOCt-VC» ttcO *-*>•** I-*»****-* *' ***» *-•*> **•**••*'**'- **»*> *- *>**

0ipai0«iuo>ait>c0kKif-4i.v 00 0000 00 00000 000 00

01 Ctlk^tl^rHkOli-lk -t O-O'OO-OTS'O-OO^TJO'O-O-OTJ'OOTD'O'OO-OTJ

r-«0k<-4c<*-vi

8

•-« » c cr u» * •*» •*» «* » »

(. t> 0>9 9tc9tl0O>O

>0O>CP99O9e»09aPO>9O>0 9O*O>0O>a«

O»^-OO'CC0«OlC-4-«Clp-t>CC'OCCCC-OCC,OCCC c c o c e c -o c c

C C >-• C C -i-i -i-l C IP -r« V -H O -»4 •M •!-« -rt •« ••-• 'M 't* -r« .M •»« MH -F» -«I r4 -H •« -H

^ -1 -«4 cr IP -H * IP -1 (- IP o> IP ip A o> -^ *» o><*>«*«*<**9<«'«*ei«*«*«*«»4*9'**'**«* er > »

•»'-oo>ccoctfioc9ocoio cuucuuuueyucuuuuueuuueuu

u-ic^-^<c'-ioici4oe'4.»>co<'«oiOi>4tiQitioi'4aiti-r4 0itititi»'4tttiOi'>-4aiai

0l>-«0*>O>«*>Cr«UO<«**O»CT3»^<-4X)'-4'H'H<-<"Oi-irtTDi-lr-iFH^4^4-O»^«-li-i'o«-4'^

r-lOV«- IP O »- Ol O *- I 0*-|PO-H-»4,^»^..<.^»-liH«-«^»4fHF^.»4M»1^»^.H-»4^^»-«-»4«H«^

oi <- 0 ti ti «i ti k-^tic-Htio-M-e > 0 0 >oooo>oo>ooo oo> eoo >oo

cno. 9-^t-o-pa.ovt-ia-^Q.o^ o o u ouuuu o u o ouuouuoouu ouu

— tl 01 tl •!>*-«- k f. k k

• *OA • Ci **0**0**0^**C^ ^** ft>*** • t •*• ^ • •

O O > OO OO OO OOk 00 0000 00 00000 000 00

. . UJ • ••••••• . .a.»*» •••••••••••• ••• ••• • • •

00 000000000000 000000000000000000000000

*4r400«4rMO»4(Mo>4r40i-iMoo«4rMO«^cMto«-o«4r4o*4r4n^no«^r4no*4rM

c>ii>400«4«-i0r>ir4onr«>o«'*>oo«4«-iociir>iryic>ior')roo*,<r>.r««oioiftioo''0<<o

inMO^»»cvi*4rw**4r4r^o^wionjioioio«Minin4r«ioioioininirtioin<oioin

<-IMO«>4-l*4^<4«4«-^v4«^^OinJ^in*H«4*4«-4in«4»4|n<H^j4^«4lO<-««4*4in«H»4

r

C-3 J
: l I !

c C •** -^ -p u

^^ ^ t * * t \

3 C • U > O •*<-H fi»)

(ewe * * ä > > -^ »* -* u

.-«

> -^

«-

c -o

tt»c

c o w

>

H

> .^

Q. o

a mou

u *> «ix

* »-I •HOI

em

o a «- H- ato«»ftu*>-owi oio -«> « «>

mco c v u a» oi o in * .« a> in«*o -H

€^ c «- >- Mif-ie « -3 m > •-• > c o« tt-HM» m *»

t, o •« -' -< o 900 ftxili»« •** -fHt-e •'« >u> -H y

u o ib Oofl i- o «. -o -f-« «-^ -^ > »-« «. > •« •»< »^ ti

etfl»0>> <-< * -O *- » -^ Q. C >KI (A-H -• f* *>«». > c **?

U«*>Cil C •-< O •© 3 <4- -^ -I w w v» t- > U-MU O J3

0 u c •« ~H o > <». 0) er «- o> -^ «- u u o <«- -^ o * • -H O

C-OOIN» e k-iikaiaiticti uo a* o <*- o *« .^"^ **

01 0 X-** 3*»T)»C<-aH> O * c U "O 0) O 43 o •

v «- c •-' c • "o x c a ca» o i.c7>-*>4) o 0) ä over a>

ecktoCtft «H OI-H fH^m-M-H e>«> 'N o c • "o > *» «- « M -O

•*< ^ c > au >o> w o c » o J: c -< «i«*«>Q.ti o> o» o«»>>u •*

OM Of-r« -»«CXkCSai^Hf-U •*<•-• <^ f-< .-i 3 01 xi C-HO O «I H -H

m tr u * -O-^^OH-OAO-^O «- K c o- c ti •»<£ * «-« H «> tn 3

•» c »- o « c c r« ««> c M a* 3 3 u OMDOf > eu o<**«(-ato

*> U H O > ••-••H4* •H>>T3*0 "DO » v x: * «-o*»*»cuo^c

c o -o « •> •-< octfion *•«* > <«. o ^ ** *» •»* o eeSoa.-*>K c

at.-* w > ft-aia*9<D-'•>'-• c » u« ** -^ £ ••aifia*i •»* o

« o. > -o > u U-»>-O>SC-CHOT9 OX -Ho^eiAa o-orH^at-o» ««»>>«» .^

a. oai-*>o e •»< H «- H u t» 3 o» .< xs -i t. • ai o » • H > o ^ IH > «*

•H » «. > o «i o. > o c -a T» t» -M o a.** tt VO-GäOäIä«-*** O

3 u> Q. o -5 c e«*>uxk •« -rt * u in uv-<4«*u •luevecf« • u -^ i

O- * •• 3 X» C X H 01 <^ O tf> > 0i Otl O O 3 •'-i C «9-x -«••-•£ X C O JC t.

01 -4 (- X> O •<< Ä 3 U» -> H «4. > •• -rt C •- »- «^ > CT- ri O -OflW U "O «- U O

> o c - a o oi x» »> -< xi »3 •>4 0i-M'« ocoaio>o«oooo>o «».

«» •M<^OU«O D'-i>ocinou)cu«ifi'p-o Mt-i vik «>ou>(-b feie e

Or-« U 01 —' «•-4-H ai-»>3X^C0lC0l •fiMOO'B UfxOOO «>-« •- O -H

OK) •M'-i co^ot'oc'O'*' o H n u -» u «*. u o« tn *» > » »i c »

X "4 0i «.< »> O •r<«^UUHSf4>ar-4«» O «lOW UC0l0««-U**4J Otlb 0l

u^-^X"««*- 3 ficii>voaH(-(. w«*.oi<*.efc>.eotioo» Ji 9 "9 f

OC«***> O- * -? O I 0» -^ O «- > 0» Q. t. * O O H «^ ^«£«*(.4«VW »

30l'4>^0lin0l-OJ3CtA-*>-OO(*-X>4C o» c -P c oi »> o ««*•** e oi —• o» w

O'«'^4«>vx:ioi.ftot4*oc » >«-•• COI^««>H»HO^*OIO ^e«* c «- o

oi o »4 o u » a» • oe^'HoiM^ue t. ai >-4 tn oi £ KI in •* (• o c oi *>

T> o.43 oo o« ** c o* <4> » m c x > o o» m o3-i0i-»>oi-?in>**v-»>«>feox:oinc

o o •»* c c u »- « »- c *» -rt «-icmai-^ucaiaxiot O-H-M OI ** ^ <- «. C

«^ o. v v 3 «(.eo(Oiuio>-4>.**o>vo^>-)ai0'4a. O-HC v «». -4 <-« » on o o

o>oaoooin>oiOo.>o«<*- *» u c «- ^4 o .0 <*• x 3 o •*»•**» m 9t 1-4' •- o c a •1

e u «oies 01 > v u o -1 o o. »4 c o cr > 0» ^ «*ee-«4«»ii(-t>^«»

•rt x >.xX(>^(.».p-t-Hf4inoo c > (-S*>o>iiOi £>oirtOio»»oc-»JO c

cu<*-uu(-0i£OO«£-»>0i o>(-oxoioiuct- in ** 1-1 e><»> JO & « t. «1 01 oi o«

«-oooo«-cuOi«*-v>uu-^xeT>co-»aV4J3o*«- M-H »> O a* «t 3 o c »^ £ *>

• 9 3 30»-o-<t-F4 00l*»*>C«-Ol<t^40*>Q.'M«« <*>UO«eXXtt 01 o > »

uo-crcro-uo oi-o •-» ^ ,4 •- ^1 rH ^4 > « 01 # wvoeeSee-^Ot-ifle o

gOlOiOlOl 0(»j-4Q.-oO>Xl >»-«-< i-t -^ x> 0; v x ft o fe.ooo*»«* Soo

Ot3TJ"O*>^4^0 O C O -^ H r* f. U *> Oi (_ O •** * £ *- *•- -»4 tlV-H^J«

UOOOOin <*- 3 T5 f* »13WOOOU>*» •«4«*30lOO«*><*-V->«X>^'*> w

SU'4-oaiQ.e<*>u->4 3c « • o * o -4 c -4 -^ e «*ooo-M«i«*eook

"Mu^^O-HOO-POOc »OOO'H'-i«^«-!« » -»4 O A (^ Oi

4*(.kt.(. <« 01 > O rl O 0*'4 0a»«»>t-**3r4XJ'4 00«-«-t.».«-0*>0»4*-CC

e • •<i a. -4 «*oi<-4*o>ae<^->4> c o c •»«•»> a u OioibOiftoiOM^^o^t.

•ooooofl/ijin-oin>*»oic«OH*»r4».Q.o30-to*-^cccec > o<*- o

«««««'«»«»ii c o»i«n-'<o«-"oo*>'-<Q.u«u-H<».o» »-'-»-fc-«-3Hii^«coioi

^ooooincxoi-^oi-oo c a *> -o t. u 3 « o co»ooopoo*«*»c^x:^4

cts-o-ouxinjc f^btt«- xoo«i<4-in.c.c-4>Q((-x:oiOiotiiti.cuoo« »

M «n »»x-^xa^ooiOi o. «4. o mmo-ot-o-p-^»-"-""-'»-« o (- m 01 ai

*>o>o>o>o> V «*. ««. o o; -»J a. 01 «4. Oi ^ »4 a. 3 oi x *> * JO >. ^

yece c 01 o x^m^in^Hr-iaoi o •- o> xt-i-t^-iox»^« • 01 • m m u. o« m <- «> o «*

«•n «*«>«*£ «^ ^» J> > 10 3 3 -»*'-< Q. 9t ** ^ & «4- -^ (.«. >«.».». .^ »co. r^

-•T3UUU»-«n-.4C0<C0l«*-rt0iai 0l-4 3eO0l»4<»-«OOOOOO«-OO XO.3

rn-M^i,-),-) •»4«*iiino(in(>oo'-«>'*A'-

,

>o--PV<Hb.e(><*-**->*^*^*»>oiHi«*tiin0

o>»-<»-i»-4.. wt:,oin-o»nOi-«>0(OiuÄ*oin«nx»o»oocc^ccc»-io»Oju>»4i.

U OUUU * >. 0l«-««»-i<U.-^UJQO 3 OflCCUiUJ e»UVl-i>-i«*Ml-(MUflCO 9 COO t- OII-IT) o xie eco -o o

• o_ • • • in o i~4 • * »•«>— • »CUJ *• • • ixj • • •-• • • 1 • ••• «»co* •

o o o o m c 00000 0000 0000 00 00 OOOOOOO 00

••••••« ••••••••••«• ••••••••••••• ••••Ö*»« OOOOOU 0000000000 000000000000000000 000 • • • o • • • ••• • *•• ••• *««**• ••*••• • • • «#•

w o »4 r4 r> (-oo^cNrovinovHCMfnoo ^tMtovooiNO^rMO^Nm^rin^oo^tM

«oor^rvrv oo»H-«^^4»40MCMCMoo^»-^-40CM<Noror«)o^'<r^-4r4r'*oo-4-4

ift^ioioiooo^4i^^^^M«^«o^f^r^r5r^rJrNNWrNN*cNWryiNWtCo'HMro

<Hn4H^^004*r^CNNNN<Hr<|NNONr*r4NNNNNNNNNNr4NNNNOnriN

4^ 0*401 fM fM« CM CM CM MW

• • • •

O W IM C4

o*

CM

C-4
f I

O *» «I M

c c «nm aieo« «* o * •»< c -^

« OcOc^fcCXOcA-^V^-POO

O a ** 0 •*> tt tJ » O > O «fl • C > *- O T3

ft * (>ii-SftftWftkO«*9>^-i-ifVb x ui a>

> "o otot-'sxe-Hft UO-O-H .»>.»* co ft ^9

H 9 «I «^o(*-o«ftOipQ.-Bau(-j<u->«-'4<*.i(«>«n^

«> > »- c c «^ c «^ • -^ ft ft o o. m o 3 cfteoo

U -^ C ft O -^C-tC^-O-tJUTlXV ft ft*90.O>

ft »* o c -H •H-*4^euo>c|»o-o-H««px>^u

ft -5 *> * I. •»>-O<fOOOC0>H t. -« H 01 O •-< V

co o c ft o>ft x o fta.in^p •> o «- im-cfc-« o «

O ft iOi-l -Q <- C ft C V fft*-4AUOOeill»-ftO«*ft t- *

m ~* ft > «- •< •oo>ftOftftftx-*4v (^vo-Hft-^ctf-^ft-^u

t. »i Oft O V ft ft pH«*..MCin->4>>i-4ftftC<^k •HUCLlACXO-*'U c

ft-*« C H w «^ o £ u 3C«*094*C'Hd>£ft0ft«V( -^ 9 a. ec0

««. «: ft 0x:(.ei«*e o -^ o u cOOOC«*-* accftointfia.<«-oo-p

«.ft c m « u ft ^ »- o e i •• ft u «i «- o -oft oftx«*e 9 o -»* a

«i o «- «- o c o o « ^ -a u <«. 4* a.u-ftu*«H«>ft OMW amft

ex ft -x »o (.•<***(. ft o o c ft <*• o Iftke » ft ft u ft ft ai u

> «>ftc*»-oo>e o -ex«^ O(-o<**ft>>eo0cukftft ftMuu9u

X> ft D.i.«. «. »•) i-l-xft«^ ft ft C ft ^ <-•'-« O C ft 0 0 «. rt ft ft CUr-lO

O-H M ft 0 ft « > C «- X > -rt ft » ^O C ft Q. 3 H H t. -H •eOV>9ft000

-»XT (.CftGL** (.»Oft ft C 9 ••< H O 0> ft ft » O-^ft^c^HO^ft^» Sft

u u «- ^ a ft o > »« o. > o -o fto-f-4-0Ocu»<*.-Hb0i.>-i.c09a.'o >

ft O ftO XJ ft £ Q. »< * CUftM-H «* ft ft 0ft(.ftO ft»«ft>«0l«.*>0

«. ftftftOu.»»Q.*>0.o o >» 90er-t ft eftHt^oa.«^ aufto.*-

9 o T» •-« JT •-? u 9 <- u o uo>ftco,**ft3efto>oa9t>*> 0 oSuSoa.

«>-** «* 0OftO-<*-3 I C > O U » O x 9 C H 0ft O Oft 0O-Oft

9 Cft ft » I a-H >. H c ^ O W ft C •^ >«*UO.ft**ft*'>."0 *> O M

«^ ft 0£O*>0 ft a. a. ft ^eo«* *> «. « ft •• c u 0 c > ft ft (-

«- ^ » *9*>C99JI«. 3 «- O -»i ft C 0> •- O C O5ft0«*5x-Hft00

«tvO V «* 0^«f49 »ft ft-Qftftft-Oftft 5ft«*-****- Cftft-O«***

«. u> v u o iA 9 -o •« 0 «* cuftH-Ha>ftft>uft-f4Ti «*ea.fto>x:-«4

ft«.»- o*>c<^ft*>>ft^9 e£99ft09^0c£«*-«4ftuiSftue4*a,fte

c ft ft ft 5 ft 0 <e0<*- ft o > o- o« t- «- 3»-o*»-4 3*»0«.r4 9 5 u ^> o

fccGk<.ftft-ii«eift«*t ft u u •MQ.ooa.u ftoe -H *-• c u ft e-*

8«- tt £ C *» » *~ £. O*- 9 O'0 09«*»Cft 9-x i«^9O0O T> O ft »*

ooc«*Ou(-«o«><*--Hft «e «* or »< J< •« ft c 9 o •> o 0-»* > »H «> *. a,

(MftVt. -M0Cifc>«*. O > «. MCHft ft ft c •1 0 4> X ft ft c 9 -HO

^ o o •»» cceo 0ft 5cx:ftt>o«*<^>-Ocu 9pk«*ec-»ie>ft9»

«t vft^uftct.^^jftfc cHk«*^ «>coe<4k0fta't o •< < o c t. er o

£ ft ft >-« O x: 0 — Cft«- P« •> ft ftM ft •-. >«, fc.ftOM90->4>>f-lftft

4>£ft C «»ft ft C O ^ O O ft U ft 3 4Cl-ft0ftUftftt-4*- 0«**»ft^3«.^

«*cftoft « o > o H ft «*ftc,oftftt-9ft-***'e«3>«»-ftcftftu-xftN.0

O £-HX:V «*- -x •»< » « ^« ft90i-<cft0 09ft 90'HUI«lftft5«-ftC3 Cft x

«>Oft«*«>«*Ofte4*.**oc (-0090x3 ft •-o-u9cfta.'Hu 90 oft*-

g

«*(. C jC-XkCUOftft o> «* k* 0 0 -x »* •• 0 o » x >

ft o ft <*- ft *> o ft x £ u •> > «n «-n«Vft»«ft » » -x <. ON«*

c c > v o o fto.«*>Hft«>-HftfteOftcft c «. ft ftcoftcftcftftftw« ft c ft

•*o(- «« c o > o.** a ft «*(.(.o«*(-9(-9e(-(-o«>t-9t.9fc-*-ke««t-9(-

«>f40ft0ftO««-X90l9OU-HC-H c » c •»>-«-—' c x C-PC-PC-XCO e«*c

0«*ftC U ft -P ft •-•-i**09«-«*ft«.«.«-«.'-«9«-"»*ft«-t.«-».«.9«.xft«.«-«-

COdO« CCft^l« 0 (-VHftC^O^'O 0-^iftC'HOt-tOi-iO-»-««*C«-iO'-«

i •xinoocOftftft>fto.u ft o a. o. ft o ft o a. a u wo o. O «- m > t. « ««- O Q. 9 (.9 ftV0«*9-xft0.ft Q.9 0*»9-xftO.ftaftO-»»ftx ft Q. ft

vocuftkb-x a>x ft ^ ft 9 c a o-*> 909009. a«* 9O909 a 9 «* 9 o =< cw>ooeOft«>90e0«^ooft ftcr a- -HO» «10 0 o- o ft o- ft o- o-

*4 C-* (-(«.Q.Uft> (.>•*' OX ^>U> ft-»4ft^lft«*«*Uftft-Hft-XftX«*U ft-Xft-X

••X«*ft0l- 0N. 0 > > uU>9«»«.«*«--xo»9-»»«-«»*.'»» U ft 9 •»* «- *

ft u JC ft ft «1 ft •> ft ft ft 0ft0fc.or-xe->4C*-ftoi.o>-Hc-xe-xft0(.o--xc-x

£ft0«»^«o.£ftftx:^je«»ftub e ». i. «- t. o fc- c cEbEbfe e «.«.«-

» £ «*£9«* «*0£-XftMfc.ftUx4U<X ft*4l.ftUr-«u>-IUft*«(.ftU'XU

«>ftftftft«*^ftC«>>HC^k 9» K w* U e-<«-

C £ X> £ X C O X £ C » a «- V CMftMft (•«* CMftMftME«* CMftM

•xft«*-H^>«*^4ft>0«*HftftO.0-x(.(. -Q T) £ O x «. *. -o t» O x t. (. TJ

0«* «- o«* »x 0c«>oftuft'-ix:xx:xuftuftix£x£xx:ftuft>xjcx£

•-iO>>U33iX0Xo.3'X5o .x -x 3 u>U>0'X-x3.u>U>UrXX3 U>U

o,^4ftft0Oo.MftftoaiiX4* ^ftaeftoooo 'Xftj^ooooo •* w JC 0 o o

x ft > e «. J: x ft>x(.xfti»ua.ocftO«-o «.uo.ocftO(-ot.oo.ocft o •- o

B 5 -oft •• ft

00 coooo 000000 0000000 • 00000000 000000

00000000000000 00000000 «oooooooooooooooo

r)ft>o<xrMnVino*4rMm«-inoon«iO'crvao»«HO*4Mm«,in<orvfBOv4r4m,ft'm>o

«4>?or4(Nrvir4C>ionnnronoo«4««^«4«H»4«4'4orir>ic>iiNNr4i><r4onroronrom

«MCMWMWfowwwi*)WWMO»H*<r*<r<r* ^• CM^r^^^^r^^^ro*^» *

• • • • * •

W N *• M W ft«

fvi

C-5
*) » > > W > in

! 1«! | 55 ll^il

c •» > 1 c o> x: > c t- x:

* in

J

0 > c m

•^ cd > i-^ uoifou u

> in 0 > g

i 11 * * i ^ •** m u .c •» *- ft ft •M

» > 0 £ «n H 0 -o ». u -^ 01 x •H O»^ »

0 ai 0 «t um £ «- oo£aa> •*> in e £

C-fl * O ttl u -o 0 <«. u * •« > t- 0 »

• E

«EjCt-C'O (- OCX» X) O *> JZ »H 3^9

u O Or Oi X» 0> 01 c ifi «1 C v. in •« 0 •>

M > *» O ««- « "O fll O <». > O C XI TJ 01 U

C 0» 0» X> «- O C » O C 01 > C

0» x> X9 > a> c a. tHiA -^ 01 -0 01

.1 U K (». o> i

k

ll a 0 ( 0 •** 0

• *«•. » »- •14

* soioioicoi u > -P c a t. fc- in a • H >

N. •• ^ X) x) -^ > «n c «- 0 ocbgo

in*»0(*u0(Hciflt.<»- •< cutft M 0> C 3

s

0

4l «» M c 0 m £ elec once

9 -H > K u ft

> tf> *> c 0 x) (-«*o « a x ^ 01 fl ft 9 •• -s c > £

1 0 « u> x> v 0 tikOiOt-aiAVOi

U r-lO»«<^C««l.1««-ai 'H^-Ht.

•- v 0 -< c > x (• ^tOit-v.e-^^-H

c 0 3 in ••< 0

•• C X X) 0 «»- ft

« •** ot- 1. in c >

0. in in -P m a»» in •»4 0

1 O Ol 0« k «1 k ti ' •m

' u

• w 0

•HO<»J^ 0lO<-«*H4*UaO-H »•**•»>( iH ft o> -rt XJ > > «t k

9 U»«* 0<*>9eOl * XJ 0 «- Oi •* a c ••> «- 0 -H 0 m a

O-UK c-pwuct-ao^oomoia •M -l-l -«H a «> *» 0

1 • •*(..*J.,4C*llO«***+> « > ^ U 3 > in k > o> y

•* x 9 -^ ft •• ft

m e

•H ft

«* >

k

9

* (-^oc 01 0« ««- fc. w» (- 0 «1 a

^ > a*5«.3uo» cm-oo-oio-^o»

^a»3 O'Pt-OVQ'OOl«*-***) .x>«^

4J m -o «J k -^ a

0 0 0 u m a

0 o< o< O -1 ft 0

O in

> «- c a **»<-* c cm*-"* 9 * * 0

•l-VU^ > ^«OlOl-OWlflOVOlC

• IT, * J X(n*O«T>CC'0l.C>«O0i

*nctf>«-C-OC0l 0 «*-«»« -P ' U -H

> => coo«--»«>o*> *» U> £

in k

a. C C ft -H O ^ k «^ 0

>. m o< M ^ -o C ** o» i5 ft 0 *

f-« »< c 0 k 01 e > c e M >

1 *

k k k C O *» i

- * «= k W k O

' a 0 «<cruiowu0"00(*>inin^>00*»««« »1 » 3 c m u c ¥» C -• 0 k ft £

0 *> ^ u XJ u «n x) 3 .* <- > x: u n. U O k O k s

i^

ft ft C ft

cot. «icc*>jCtfix:crtawoi*><j «» in u — ft 0

0 *- O OS-UOO-^U«!^«-«* C "»'U 0 > 0 a ae> ft Sit; k k 0

» 0 O O X7 O Oi « » * O C fi ^ v i«it-o>c

• c c 0» 0 * is ocrHcio«

•-oioin -D 0 «- t. £ c *->(•

> e ^4 X O O ft ft

•r4 x c u» «* o> k m 8 Xt "I ft -1 »

! W > «J ft «* «* M (*. •• e > ^H 0

> O -« X> > • tf» £ •*4-HUUO>k-»><-)0lO •H m c ^4 a 0 v ft c 0 ft -^

e ^ •» •tCX>O*'i0l3C-<C^C ••< ^4 ft ft 0 - ««- m m m O £ O £ k

• 01 £ O «( <** '«•-)JIO'C£-H f. **• JC — •»« k -w 3 0 0 #* k e ••4 ft «* 4J Q.

«- A *•*£«- e J.H*»»)«.JC>.<^OU £ a-* ** -rt c ft«* e> •** £ O

•M N^ > M -c- U» AC «-o « «».^- Oir. oftuu^Wftin

inkeokfti-**

c 0 e O M X k

3 «» "ou»-»<3c*>*,

«<*-v«4-«no^.-ic > 2* "5 a« c £ a

a ot *- > 5 o- 0 C 0» OW«.*>fll-»>X>OxiC -mo a

O C 0» > « -O - OlOlC^C -^ 41 «

O «- 0) X (- *> o> O -P ft 0( -o -^ C ft

0 9 •»« 0 e ft »H

«1 u > o> c a > > 3 sst U k -r« m 0

«- •>< 0 o> c «H 01 «* • in

e — c ft <« u H •

-<«no30ft£k

* >- * a. •4 ft «J k

N, O •D •* ft ^s P c 0 0 ^

« £ > ,1 ^H in£-«4-H^-HCTI > «-•HrH-»» ft .4 ft •H k k k •>• m

» U 3»ft*>r.*»0l**3ftft3 • ft V £ C in 3 H » v L» U *» 0 a 0 0 0 > > k

-M «1 0-0-F«-^ o- -p f-« u * > H > 0 x> a c u a ft m 0 ft

lftftC*> aftO»OfcO£3V -oft««

O in -P 01 in ft 0 v 9 fi v

> Ifl •»< • in -^ £ v w

|

k

> w x a- -< C £ c

c «- ft *- j< C «- 0 <«- 0 «- u v o<- •»» m •*> k c M « ft

in3uin«ftin*>

•H *» k C ft 0 ft k

> c i £ 0

- 0 ft ft

> c *-x •**Mv,'ocina«*o u> xx e » e 0 -x

(. t. »»Met-r-itfltt^-HOUt * *»-«-««- V 3 00 01 m in k a u ft in »

0 ~< c OftOO^Ol«* *» c XI > FH rt ^ w *> •H x e -H 01 m m-*» 0

ft ft 0

» X> ft ^4

a •M «- ««a-^-rtOuococo «199-H-M O kkoomocs*» urn £

a 01 •*> a*>ftadc-*><>->4 *.*>«•'* oioro-«*«* > 0 » em k 0 * o> > m k 0 -P ft «> <*-

e r> HI C X> O ¥> C > «^ Hft*>«.»-ftftOO ft «*- m 0 0 O 0 x

o>*» a v. •»> ft k

in c k 7 « ^ »

w M 0 k > 01

i

e •H 1 > u •»» 0

o- 01 » > •** '* *3-oca «- «- m «1 *• O-H w V ft • «^

M -^ *> «(-«lUtMbCUM -^ 0 * ft £

y ** •* 0 «*. ^ 0 •*>

(• *> (-Ot>b«>OQ0l(-ttl«l t.MOOOO

eacc ca«ack».uaft££££

(> a «- t- *. aft «fl (-c c-Ha«>3333

rH O »-1 •-•.-• O O i-l 1. «. «^ O

> m c~ -O «^ a c

tt tOi •P

m a a u c m » ft ft

0 k ft m So 3 c -P S ** U t«* £

O X»** t. -M O £ 0«* m «^ 4 O .4 ft U O ft

>. ^ M ^1 at w u* M m T3 O X X 4* > > 4J 14 U

•I M

S

ftftftftftftft<^U u ft w •-»-*. (. t. £ • » ft £ k » m z0>H ft 0 0 •• «> k

TJ e-o(.-o (.-Dt-^c-ot-ftc occce O m u ^-« x> c m .P u •^ O U > M .^ O

•H £ f-4 H-f-l0-r4 0 -^ O ^ •< 14 0 a« c «- *- «- *- «». •M 9 -X k -H -^ ft ft 0 k u e

> u • >a> a > a c • > a ^ • er><^ **->**

ft 2 8° 0 ° 0 -S * 2 onftSaooo

•-• T> a 0 0 — *» -< £ H u a 0 ft c

•— 0 ft a —

if »* ft £ c fl U -P £ in -H »- k

u (•

^

^4 k ^ 01 01

a. 0 :><*>o.ua.ooLUi-i«*ft.ui-<*>a>-i**»4t-i •»< •* a-i < uj au cc oc

• •

0

g Ä J[. . .I 9 W W

O* L&J * • • * UJ •

k

a*. . • ^1 • • •

0 0

• •

^ooooooo OOO OOOOO ^

0000 00 •••••••• u

9 ^

OOO OOO

0 0 oooooooooooooöoooo 00000000

k

a.

00000000

^oooo«Hr4n4rin'OCDO»4r4P)0«4r<4n*-in O o^Nro*o»ir3 OO^fMMO^tMM

nhioo^^^^^^^oMoKNomronr-ono OO«^ x •« O N CM <N

«r * 0 •^inmnmmirmcMinininroinininmino ^•^oy)0>OCN«0'00 0 «4 «4 *4 •^ <M r4 -4 «-t

(MCNOin(M(Nr4rsirM(MrMinr4CMCNinrymtMr4CMO<oM(MCMr4«otMMO 0 «MMWW-HMMM

«W fM M •O IN M 0 V* M M

f

N N 0

•

0

•

n

C-6
» t

> >

I •£ »

ft u e

o> in x H o> c u

•x Sftue -x ft ui • ^ «^ H a rt ^ 45^5^ «>«i Mft^ftou o--»»

S ^»uu-ss oft •F'i'tf10^^*'^

" C ft -S -So o> ou •«• »•••««.•ft«»-

C ft U) ft «* -H C «- ft •»* U C 3 rH -rt 5 !*> H»»«»- -• • •-» as0

? 2 '*.!: *.: ft>-H TiftuO) ce > x> 9ft eft ««»«in ft S.2.3ftT»"c"S u t ft o o •»•-»•"i^S-^': •-* c > u i o u o oo u ^ ftcVe^Vftiu«*

e> lia9'Sft|Cu ^i-« ofto— u^^^-feftj^-H

C T3 " Ä * O x O C flic-** U OO «ft U

r ftc w-g

ö

5 üT i oo * 51 >•«• «^ •«••*

e "OH «- ft --9 in v ft i- «^ > u -H •»< o» c -P <-•»-

> ftfiS 5*o'5c-o^ oo S-'u-HtcaOftu^jin

«, l^» JÄSoftftoa oo © o »tl^ S fc 2

0

u. .i a ft iiftjc ** i 9 c i- > •* ec ft c u O u • «n <». •. ^

^

ft «1- 5cftO« ^ r>««***fcVlftCft — Cftfts^fto*»«^^ >>ftftt-9C&^ft>XfteO ffFF-SS f±-t^

"S^ ««•I ^ISu jri.-«4^a 9ai**4<**k(.fteii sccftfikCScoo AftftS'H»^ 1 If i l<s.% f

i. iin fy

j£«* «-•aw

u-^ ut« 5 o- »* >«»« 35>rHeftftOKea, c

M eft.. •• rill r

8

"^-:;^^ >.ss.5 gfc5?|ftg0

o.-o-of'.ojti ftttSSft^^^ftcc i-aa-.^ ^5^ *^ 5 2&ü Sftcoe^moincxri-HCinMiwvtift fteo«* "** ^ ^ £ ^ 5 fr

I«ift««<?e9uuw9ftftftftfti ^"L-L.t^fUtST^fS-0'

•«« «> Z ••« w o i&tiaoofteec >> •-*I/I*>»***--**>£U^9I

• •oSx-^#«S"*EEtfftft • S:i-S!^Sa2?5E^2

k 1 e a£0ftftO->4->4-«4OfHa«>N«uu «^ *. u o *!: "^ _ a.l>

ee^-SoÄ-oooo-o^^^ft ft ft c o<«» *» «* .-^-^ * «

wpE^toftfo-SjEA^^ooo »

0

'^'!l^-,'

OftO»0-OjC«-0>

<5üüil"2'2üi:^ >999WCCOC CÜ^i^STÜtlff

• •ooco-x^ftcftSSc^^v^oooftft^-o-o-oo-o*»«»

Sols5ft'3

,

S^^-ftfttSSScSg>v-550:5*55,:35CC

UJcÜowSa:aI!n»-i-i-3ftoooo<<o*j»->-iico"S-S->^änäc»

OfcoOOOiJ» ooo ooo ooo ooooooo oooo

.j ..• .. <? ...« .... .. .u. JZ^sZJZJZJZ^JZ oooooooooooooooooo ooooooooooooo

*OO^NO^NOO^NMO^NMO^NOO^CjW*«-ft^O-NM*

• ***«#•««•••••*#*••••* * ft • ft * ft ft ft • ft

.• •«• •*.•..••• •* •**••••••• •

.. • »...• . • • ^r

. wn rn M w w n * n "

• • J:

C-7