5.

.search Note 84-78 1%

EARLY TRAINING ESTIMATION SYSTEM (ETES)

FINAL REPORT

Lawrence H. O'Brien

Dynamics Research Corporation

Charles Jorgensen, Contracting Officer's Representative

-' Submitted by

Michael H. Strub, Chief

ARI FIELD UNIT AT FORT BLISS, TEXAS *

and

Jerrold M. Levine, Director

SYSTEMS RESEARCH LABORATORY

r 7

U. S. Army

Researct- Institute for the Behavioral and Social Sciences

June 1984 NATIONAL TECHNICAL '"

INFORMATION SERVICE

US. DEPARIMUIN Of COMMERCE

Approved for Public release; distribution unlimited. SPRINflILi, VA. 22161 I This report, as submitted by th' contractor. has been cleared for release to Defense Technical Information Center

(DTICI to comply with regulatrry requirements. It has been given no primary distribution other than to DTIC

and will be available only through DTIC or other reference services such as the National Technical Information

Service (NTIS). The views. oplniens, and/or findings contained in this report are those of the author(s) and

should not be construed as an ificial D'epartment of the Army position, policy, or decision, unless so designated

by other official documerintatioi.

26 157~ gi14
·•· 

THIS DOCUMENT IS BEST 

QUALITY AVAILABLE. THE COPY 

FURNISHED TO DTIC CONTAINED 

A SIGNIFICANT NUMBER OF 

PAGES WHICH DO NOT 

REPRODUCE LEGIBLYo 
SECURITY CtASSiFICAT'CN 6F THIS PAGE (%7)S. D..-4 Enlt Pro v

REPORT DOCUMENTATION PAGE READ INSTRUCTIONS

BEFORE COM IPLETING FORM

[I. REPORT.NUMBER 12. GOVT ACCESSION NO. 3. RECIPIENT'S CATALOG NUMBER

Research Note 84-78 A(l/ 4. TITLE(-dSubtitle) S. TYPE OF REPORT & PERIOD COVERED

EARLY TRAINING ESTIMATION SYSTEM: FINAL REPORT FINAL

6. PERFORMING ORG. REPORT NUMBER

7. AUTHOR(a) 8. CONTRACT OR GRANT NUMBER(*)

Lawrence H. O'Brien MDA-903-80-C-0525

9. PERFORMING ORGANIZATION NAME AND ADDRESS 10. PROGRAM ELEMENT, PROJECT, TASK

AREA & WORK UNIT NUMBERS

DYNAMICS RESEARCH CORPORATION

60 Concord Street

Wilmington, Massachusetts 01887 2Q162722A791

II. CONTROLLING OFFICE NAME AND ADDRESS 12. REPORT DATE

Army Research Institute June 1984

5001 Eisenhower Avenue 13. NUMBEROF PAGES

Alexandria, Virginia 22333 237

14. MONITORING AGENCY NAME & ADDRESS(If different from Controlling Office) 15. SECURITY CLASS. (it thle report)

US Army Research Institute for the Behavioral UNCLASSIFIED

and Social Sciences UNCLASSIFIED

Fort Bliss Field Unit, P.O. Box 6057 ISA. OECLASSIFICATION/OWNGRAOING

Fort Bliss, TX 79916_'-" 16. DISTRIBUTION STATEMENT (of thle Report)

Approved for public release; distribution unlimited

".

17. OSTRiBUTION STATEMENT (of the abetract entered In Block 20, It different from Report)

18. SUPPLEMENTARY NOTES

Charles JorgensEn,Contracting Officers Representative

19. KEY WORDS (Continue on reverse aide If necesar'y md identify by block number)

Training Training Estimation

Instructional System Development Simulation

Data Base Management Training Effectiveness

Task Analysis Training Effectiveness Analysis

Front End Analysis 2 Ak;S7--AACr Czatlme om rev srse siit it noee.e.a . ead ldet!y by block number)

This report describes the research and development activities conducted under

the Early Training Estimation System (ETES) development project. The Early

Training Estimation System (ETES) is an integrated set of procedures and

automated tools for estimating training requirements during the earliest

phases of the weapon system acquisition process. The ETES has three major

components; a System Description Technology (SDT), Early Training Estimation

Aids and Procedures (TEAP), and Evaluative Technology. The SDT is a data

base management system f-or storing and tracking task and training-related

,."DD DD FO" 1473 EDITION OF I NOV 65 IS OBSOLETE JAN UNCLASSIFIED

SECURITY CLASSIFICATION OF THIS PAGE (Wheri Des Entered)

r.
Unclassified

SECURITY CLASSIFICATION OF THIS PAGE(-ha Data Enre..d)

*data. The data in the SDT is used in the TEAP to estimate training requirements for a new system. These training requirements include estimates of

task requirements, course requirements, and resource requirements as well as

estimates of training costs, training efficiency, and training effectiveness.

In the Evaluative Technology, the integrated impacts of training requirements

are assessed, training alternatives are evaluated,, tradeoff and sensitivity

analyses of key parameters are conducted, and the relationships between ETES

outputs and key Army acquisition documents and processes are specified.

This report provides an overview of the components of ETES, describes the

research conducted under each of the five ETES study tasks; and outlines

future directions for improving ETES.

The final report and Appendixes are published as separate volumes as follows:

Final Report: ARI Research Note 84-78 (includes Appendixes A through E)

Appendix F, User's Guide: ARI Research Note 84-79

Appendix G, User's Guide, System Description Technology: ARI Research Note

84-80

Appendix H, User's Guide, Media Selection Program: ARI Research Note 84-81

Appendix 1, User's Guide, Automated Resource and Cost Estimation Technique:

ARI Research Note 84-82

Appendix J, User's Guide, Automated Planning and Scheduling Technique for

Individual and Collective Training Plan: ARI Research Note 84-83

UNCLASSIFIED

A SECURITY CL ASSIFICATION OF THIS PAG E(I7 n Dire Enerd)
S ' . .. . . . . . . . . - . - 1.R. - W .- S -

TABLE OF CONTENTS

Section Page

1.0 Introduction and Summary 1-1

1.1 Objectives 1-1

1.2 Organization 1-1

1.3 Need for Early Training Estimation 1-3

1.3.1 Requirements for An Early Training

Estimation System 1-6

1.3.2 Intended Users of ETES 1-7

1.4 ETES Study Tasks 1-8

1.4.1 Original ETES Concept 1-8

1.4.2 Task 1: Assess Existing Concept

Development Procedures 1-11

1.4.3 Task 2: Develop Method for System

Concept Description 1-12

1.4.4 Task 3: Develop Training Estimation

Aids/Procedures 1-13

1.4.5 Task 4: Develop Evaluative Technology 1-14

1.4.6 Task 5: Develop User's Guides 1-15

1.5 Current ETES Components 1-15

1.5.1 The SDT 1-17

1.5.2 Training Estimation Aids/Procedures 1-26

1.5.3 Evaluative Technology 1-37

1.6 Relationships Between ETES and Other ARI

Products 1-42

2.0 Task 1: Assess Existing Concept Development

Procedures 2-1

2.1 Assess LCSMM Interfaces 2-1

2. 1. 1 OICTP/ICTP 2-2

2.1.2 CTEA 2-6

2.1.3 Identification of LCSMM Reports 2-10

2.2 Interview Acquisition Participants 2-17

2.3 Review Psychological Aspects of Design Process 2-21

/ F T-,

iii I

". ."5 :"' :".%".".i :" < . ' -- " ; L ".-" "."- . ,". ." . ','- " ";- -'. -; - '; f.. . ..- -'' .'t ' •
TABLE OF CONTENTS (continued)

j Section Page

3.0 Task 2: Develop Method for System Concept

Description 3-1

3.1 Review Requirements Analyses Tools 3-2

3.2 Review Data Base Tools 3-6

3.2.1 Overview of DBMS Concepts 3-7

3.2.2 Process and Results of Reviewing Data

Bdse Management Techniques 3-11

3.3 Review Research Related To Human Computer Interaction 3-37

3.3.1 SDT Requirements Related To Human

Computer Interact ion 3-38

3.3.2 Human-Data Base Interaction Literature

Review 3-39

3.3.3 Summary of Implications for SDT 3-53

3.4 Develop SDT 3-55

3.4.1 Identify SDT Requirements 3-55

3.4.2 Identify Data Elements 3-55

3.4.3 Identify Data Structure 3-57

3.4.4 Identify Frames and Program Logic 3-60

3.4.5 Develop Software 3-60

3.4.6 Test/Revise Software 3-60

3.5 Apply SDT to Army Weapon System 3-62

4.0 Task 3: Develop Training Estimation Aids

and Procedures 4-1

4.1 Identify Functional Requirements 4-1

4.2 Identify Functions and Tools To Be Included

In TEAP 4-8

4.3 Select Tools To Be Automated 4-12 4.4 Develop TEAP 4-13

5.0 Task 4: Develop Evaluative Technology 5-1 L

5.1 Review/Examine Simulation Techniques 5-1

5.2 Develop Evaluative Technology 5-3

iv U
TABLE OF CONTENTS (continued)

Section Page.

5.2.1 Identify Functional Requirements 5-5

5.2.2 Identify Functions and Tools To Be

Included In Evaluative Technology 5-5

5.2.3 Select Tools To Be Automated 5-9

5.2.4 Develop Evaluative Technology 5-10

6.0 Task 5: Develop User's Guide 6-1

7.0 Potential Improvements 7-1

7.1 SDT 7-1

7.2 Training Estimation Aids and Procedures 7-3

7.3 Evaluative Technology 7-5

7.4 General Areas of Improvement 7-6

References R-1

Appendix A: Overview of Media Selection Program A-1

Appendix B: Overview of Automated Resource and

Cost Estimation Technique B-1

Appendix C: Overview of Automated Training

Planning and Scheduling Technique C-1

Appendix D: Description of ARI Projects Related

to ETES D-1

Appendix E: Documents Reviewed During Examination

of LCSMM Process E-1

o' '" S

iV
LIST OF FIGURES

Figure Page

1-1 System Definition As A Function of Acquisition

Phase 1-5

1-2 ETES Components 1-16

1-3 ETES Architecture 1-22

1-4 System Development Process for SDT 1-24

1-5 Training Estimation Aids/Procedures 1-28

1-6 Relationship Between ETES and ISD 1-38

1-7 Evaluative Technology 1-40

3-1 An Architecture for a Data Base System 3-10

3-2 Overview of Processes in SDT Development 3-56

3-3 Relationships Among SDT Entities 3-59

3-4 Example of SDT Menu 3-61

4-1 Procedures for Developing Training Estimation

Aids/Procedures 4-2

4-2 Procedures for Developing Automated Training

Estimation Aids/Procedures 4-14

5-1 Procedures for Developing Evaluative Technology 5-4

A-I Overview of Media Selection Program A-4

B-I Overview of RCET Procedures B-5

C-i Overview of Procedures for Using Automated

Planning and Scheduling Techniques C-4

Vi

'S' ' , , '_ *, . j , S'S ,, " , " , ,, % ''', '" ' , .-. S% .% - . . . . t .< * .~. * - .. , " .•
r Ar .r. 7- W

LIST OF TABLES

Table Page

1-1 ETES Study Tasks 1-9

1-2 Comparison of Initial and Current ETES Concept 1-10

1-3 SDT Data Elements 1-19

1-4 Major Army Acquisition Processes and Documents

Related to Early Training Estimation 1-43

1-5 Relationships Between Projects in ARI Thrust 1-44

2-1 LCSMM Documents and Processes Related to Early

Training Estimation 2-3

2-2 Relationship Between OICTP and ETES Functions 2-7 v

2-3 Relationship Between CTEA and ETES Functions 2-11

2-4 Summary of Standardized Reports Related to

Early Training Estimation 2-13

2-5 Organizations Interviewed During Task 1 2-18

2-6 Potential Problem Solving Aids 2-23

3-1 Requirements Methodologies and Results of

Preliminary Assessment 3-5

3-2 Characteristics of Commercially Available

Mainframe/Minicomputer Based DBMSs 3-13

3-3 SDT Hardware Configuration 3-20

3-4 Past Efforts At Human Resource Data Base

Development 3-22

3-5 -Overview of LSAR and Its Major Weaknesses 3-24

3-6 Limitations of UDB 3-28

3-7 Data Elements Contained in CDB 3-31 , 3-8 SAT Data Elements and Limitations 3-33

3-9 NEPDISS Data Limitations 3-36

3-10 Martin's Dialogue Types and Their Applicability

to SDT 3-41

3-11 Martin Categorization of Input-Output Devices

And Their Applicability to SDT 3-42

3-12 Ramsey and Atwood's Scheme for Classifying

Human Computer Interaction Information 3-44

3-13 Ramsey and Atwood's Scheme for Classifying

Dialogue Types 3-46

3-14 Smith's Scheme for Classifying Human-Computer

Interaction Information 3-48

3-15 Smith's Scheme for Classifying Dialogue Types 3-50

3-16 Sidorsky's and Parrish's Scheme For Classifying

Human-Computer Interaction Information 3-52

3-17 Sidorsky's and Parrish's Scheme for

Highlighting Information 3-54

3-18 Example of SDT Data Dictionary Page 3-58

Vi

vii

S. - . ."

.. . . =.*.***.*.**.***~***.*****.** .
LIST OF TABLES (continued)

Table Page

4-1 Functional Requirements for Training

Aids/Procedures 4-3

4-2 Focus of ETES Training Estimation

Aids/Procedures 4-11

5-1 Simulation Techniques Reviewed During Task 4 5-2

5-2 Functional Requirements for Evaluative

Technology 5-7

' 6-i ETES Documentation 6-2

7-1 Potential Areas of Improvement 7-2

A-I Possible Assignment Strategies A-6

B-I Example Cost Elements B-3

%V i

'-

04

viii *5

.4 - . " . -' ' .2 . - . - ' - ' ' .. . i - .. . - ... . ..: - .- -' -' -' ... - .-: • . - . -
SECTION 1 - INTRODUCTION AND SUMMARY

4

1.1 OBJECTIVES

This report summarizes the research and development

activities conducted during the Early Training Estimation

System (ETES) project. The ETES project was a three year

effort sponsored by the Army Research Institute. The

objective of ETES was to produce an integrated set of

procedures and automated tools for estimating training

requirements for emerging Army weapons systems during the

earliest phases of the acquisition process (Mission Area

Analysis, Concept Exploration, and Demonstration and

Validation).

1.2 ORGANIZATION

The remainder of this section describes (a) the problem

areas that ETES was designed to address, (b) the intended

users of ETES, (c) the ETES study tasks, (d) the components

of ETES, and (e) the relationship between ETES and other ARI

projects designed to assess manpower, personnel, and

training requirements for developing Army systems.

Sections 2 thru 6 provide a detailed description of the

research conducted under each of the five ETES development

tasks.

Section 7 describes potential areas for improving the

current ETES. Appendices A, B, and C describe the

algorithms and software in the three automated ETES training

r-.

" o ... . " - . "' . . . ° . * t "." - " ° ° ° ' . . °

. .. K .. '. " " .. , % °"• %
estimation tools -- Media Selection Program, Resource and

Cost Estimation Technique and Automated Training, Planning,

and Scheduling Technique. Appendix D describes other ARI

projects related to ETES. Appendix E lists the Life Cycle

System Management Model documents reviewed during the ETES

study.

Detailed descriptions of ETES automated and procedural tools

are available in the following documents:

o ETES Procedural Guide (approximately 300 pages,

o User's Guide: ETES System Description Technology

(approximately 300 pages)

o User's Guide: ETES Media Selection Program

(approximately 200 pages)

o User's Guide: ETES Resource and Cost Estimation

Technique (approximately 100 pages)

o User's Guide: ETES Automated Training, Planning

and Scheduling Technique (approximately 100 pages)

Additional descriptions of ETES tools are provided in the

following technical papers:

Jorgensen, C., and O'Brien, L. The Early Training Estimation

System: An automated training needs assessment technique.

(Paper to be published in the first issue of the Training ""

Technology Journal).

1-2
O'Brien, T Automated System Description Technology. Paper

presented at the Second Annual Conference on Microcomputers

in Education, Washington, June 1982.

Boylston, D. An automated decision aid for the assignment

of tasks to training media in early training estimation.

Paper to be presented at the 51st Symposium of the Military

Operations Research Society, September 27-29, 1983.

O'Brien, L. Early Training Estimation System (ETES).

Proceedings of the TRADOC Developments Institute Chiefs of

Analysis Seminar, October 21-13, 1981.

1.3 NEED FOR EARLY TRAINING ESTIMATION

The Early Training Estimation System provides a capability

for systematically estimating training requirements for

developing Army weapon systems during the earliest phases of

the acquisition process (Mission Area Analysts, Concept

Exploration - Phase I, and Demonstration and Validation -

Phase II). These estimates of training requirements include

specification of the system's task requirements, training

course requirements, resource requirements, and estimates of

training cost and "effectiveness." I There are two major

reasons why such early estimates of training requirements

are needed.

ETES only provides gross high level estimates of

training effectiveness. These estimates are only

appropriate during the earliest phases of the acquisition

process.

1-3
16

First, by developing earlier and more accurate estimates of

training requirements, the training planning process can

9beg in earlier. As a result, the training products

*associated with a system, many of which require a long lead

time, are more likely to be available when the system is

fielded.

Second, by developing estimates of training requirements for

the various design alternatives which are considered during

early phases of the acquisition process, the training

developer can provide the information needed to effectively

influence system design.

The importance of obtaining training projections during the

earliest phases of system acquisition cannot be

overestimated. Most of the major design decisions related to

a new system are made during the early phases of the

acquisition process (see Figure 1-1). Thus, if training is

to influence design, it must impact these early design

decisions. And there is good reason for ensuring that

training-related considerations do, in fact, impact

design.

In most weapon systems, manpower costs, including training

costs, are the largest component of the system's operation

and support costs. Because these costs are the result of

demands generated by the design characteristics of a system,

acquisition policies have been established by the Federal

Government to ensure that support requirements are

accurately determined and evaluated in conjunction with

system development (for example, the DoDD 5000 series on

majo~r systems acquisition). ETES is specifically designed to

provide the Army with the capability fo r meet ing the

1-.

1-4

'- k D~L S' *.*5.* .- .
CC

h6~

z>

-6

2 w

ac

3 2

0 ae 

'

1-
training-related requirements specified in these acquisition

policies.

1.3.1 Requirements for an Early Training Estimation System

As part of the first task of the ETES development project,

an extensive review of existing Army capabilities for

conducting early training estimation was conducted. The

review identified three major problem areas which limited

the Army's ability to develop early estimates of training

requirements. These problem areas are:

(1) Lack of Systematic Flow of Information Among

Participants in the Acquisition Process - Under

current practices, training developers often do

not receive information on system functional

requirements and design concepts in any systematic

format, nor are they kept abreast of changes and

updates to the design concepts. In addition,

there is also a large amount of redundant data

development and data collection among training

organizations, particularly during the early

phases of the acquisition process. Army system

developers and training development organizations

do not currently have the full capacity to share

information and this can lead to unnecessary

duplication of effort. As a result, different

training-related organizations often are not

synchronized with each other or the prime

equipment developer.

1-6

..1
(2) Lack of Estimation Procedures/Aids Approp)riate to

the Early Design Process - Even if training

developers receive accurate and timely information

on early system concepts, deficiencies in training

( stimation aids and procedures preclude the early

estimation of training resources. Most current

training technologies are designed to be applied

with the type of detailed data and the types of

analytical questions which are more relevant to

later phases of the acquisition process.

(3) Lack of Systematic Technology for Rapidly

Evaluating Training Alternatives - Currently,

there are no systematic procedures for rapidly

assessing the training resources, cost, and

efficiency/effectiveness of training alternatives. In addition, there are no techniques for

quickly conducting tradeoff and sensitivity

analyses, and asking "what if" types of questions.

1.3.2 Intended User of ETES

As part of the review of existing Army procedures conducted

during Task 1 of the ETES development study, potential users

for an early training estimation system were identified.

The primary user organizations of ETES are expected to be

(1) the training developers in the Army schools associated

with the development of new systems, (2) program management

offices (PMOs) for new systems, particularly those individuals concerned with training development or Integrated

Logistics Support, (3) the TRADOC System Manager (TSM), (4)

other Army organizations concerned with training development

1-7

:n:

" " . ., . ", .. . . . . . .. . . . . . ° ", .. . " " " • ". "2"
•iI

such as the TRADOC System Analysis Activity (TRASANA) and PM

TRADE, and (5) contractors who must develop training

requirements for new systems.

1.4 ETES STUDY TASKS

The ETES development study involved five tasks. A summary

of these five tasks is presented in Table 1-1. This table

lists the subtasks under each task and the major products

that were produced under each task. An overview of these o

five tasks and the original ETES concept is provided in the

sections which follow. More detailed descriptions of the S.

research conducted under each of the five study tasks is

presented in Sections 2 thru 6.

ft.

1.4.1 Original ETES Concept

In the original ETES concept outlined in the statement of

work, ETES was to have three components; a system

description technology, a task generation procedure, and an

early system simulation model (see Table 1-2). The system

description technology was to consist of an "optimal

language for describing developing system concepts" and an

automated aid to store emerging system concepts described by

the language. The task generation procedure was to provide a

tool for generating task descriptions fram the early system

descriptions stored in the system description technology.

The early system simulation model was to provide a technique

for identifying and evaluating critical tasks. More

specifically, the simulation model would be used to

determine how system performance could be expected to be

1-8

* .71 . . . . . . . . . .
0) U) U

Orf r-4 .4U -4 " 0 0 1-4- 01

0-4-,0 O U 0- - '

04 0 --) 4 0 0 )

O-+ C,. 4 i 

0 0" 0 . 0 

-1-4

0 0

0 (0 0

-a'- ,4 C -1 C r 0 4 -0 0 r4 .

>r (a U 0 > 4j 4 (a

0 . -- - 0 43 Ell.

fo U) (a ull w W,-

LV 0 "

Em -42 -4~ - >40 >*-4 0(00

12. (a)L EU24 LAJ

00

w cn Cw4 0- 

(D to, as0 6 -

0 w 00 4 4E -

-4~-- C 1 a4 - )0 Q

4) >CO ALC~ -4 -4 Q U)

00) 0 to~ 0~ a O I Z>1E0>, ~ 0 4

°=o

0 . 0- . S1- 0 r 0

41 0 a r_

CPOd ) l -4

r 0 --4C. Q) V -4J a.4 tr u -N-4(fl0f

WU~ 0 0-4 0,w 0U0 CI)C wL~ 4.)4 to -m14(n U)r- i

( U En w 0 ( a U ()(n r 1

F--

E1-9
O~ON

0 cp t W C -4

4 U) W 4) 66.m -M -.4 C .j m 414-j (a 61 t'- ) E4-14.c E. C~ 4j .4 i C 9) 6 (a c toc Un.4 '-MG 14. - to go -4 4) 41 a 6 0'~' 3c -4 u 4-' Mci' 66

4.).A 44 * -4- 44 4 c 4j U 4) G) LnG) U) 14 (ai 61 U) 0a0 w44 1 14 4) 6 .0 0 6, C 'a (a .. 0) M C M w W 0 '4-1 t 0 cV ) -4 C I

C.C -~ Wi U = 1 41u 0 -M 4 ) 4 to U2 m 0 to'.C 3 4-4 ) L~ a) Iu) 41. m

ri r *41 V w L = C C U) 4J W0 (L) 904J c ). 02 0) (a2a n0 -4 ua)0C > . u C C

0 0 4. -M a4j u 4) 3: t2c (ia0..U20 11 .- tU - 02Z C (n 0 U) .j G4) '-M 0 c M mU) (a 41 >4En it0 LWi a inU26i wi(a 'O3-4 . w -M0 w 41

-4 .0 aU to ~ ad L>i'aU $W L-i..14 wi ) U 0 U) o 1.)C 0 0 rE -02 u) X: -V cr 41 ~ : 44 >04 4 ) 4

43 o'a i6 -xr q1 -M V) ( dJ2 V :3 to 4 o41 >

(a44A U C )( V m02. 0)- 0) 4 cO %Du 0 4 V U~ A t 10c EaL4UG' 0.. U- 41 P4J 0 -.4. to U)d a 1 4

E- E-4 *01jWw Li Cl- (a Li A 0 P4U~0 .4)0 V r -. 0 C M O >, wV 0 r- to 'a 4j -4 0 0 -0) 0 C2to "4jC c -4 -4 Q) 1 ) (2a u0 c._C 4> O 0 M CiU w 0- i m %W4 toa 0 44 0 Ai E24 a0) z )~- Z .I *- o 'a M -M4.-i r=J 4.' 04.j 10 - c C. 0 > r_0)N 0).-4 &L 0 -4 a

E-0 L 0 t WJ rzi04a 'a ) 'C 4 U ( 4- 'a W -4 'a '=X> -M to -4 w 4j -4W~0 4) AJ1-2f OiI 4 4) 0 -- 4Vfl0 W .4'a

Li M.0UOG *41 U)~ (a 410 f ) - 0D ) -c04 4 w0U) , --2I 4

uz u 41J ~ 0 t w2 0 JU 0Ca 14.0 41 0 . z E-'U ~LU, :3 4u VOCU200 r_ u 0) V - LiC oC 0 4) 01 M ' t C) ) 00 > > a) 0~ 0-4-C -UQC u c i4 Li w44G~L U2'- a), *.) j $ a%-4 a) >io 0)a --f 0- 0wic (a C '0 014 2C1. .J '0 M C 4J-4 Ad 4-40 U 4 W S 2 u r= -M m 4 m . 4 L J~c A (a'4 'a 61 UW2.4 to 41 -14.4 o>1 to w W0) 0 w 4~ c C p 4t 4OLC) :3 O' s2Ca-4 41.4W EA 4.1 W 4. U 4J10 m ) 02 -4 g -M '44 -4 0 2)--'14 0 C .0~- 4* w 02CL4-4 'a .)VO t to 2 A' 4 a 4 'a J ' 0 (IJ 4J4J 4) )-40 'a > >0L0 Li a-4) a) w w ) > C-M0L0 U) w VC LiC0 > C)A E-' a4 (a2 to1-IJ.I' r-a~V0.- 0J- O X 0''~ 44.) z

z

-4

LO 001

0 L -40

0 ul U '0 'a (a -4 02L 1~

4)ty U C ()a L64 V 0 E -4 0

0~. 4. teU pA

U20 0202 -40

-40 E 0 u

u 00 z 0 Ad m m

04V- CO r-4 2004j

-4 'a 0 I '-4 w Li -4

0 i-) a ~ w 6

C CVC 'aC C)LC- I E En C W L 0 0) M M i

'a *'3 0) CU2 0 Qa 

EI L2a E' V 0> a C C fl

C Q) az ) ~ E2 -M M.- :3- U 0))

Z n -4t40( 4'a4I 00 C - .02 4 0 O >- C 4.4L M .Cv 0 > m 'a

e1 Li 4.4 E- a. 44 0- 4d'V 0

~ U '1-10
impacted by human task performance and how human task

performance would, in turn, be impacted by system design

concepts.

V.

V'

Table 1-2 describes the current ETES concept and camares it

to the original concept. More details in the current ETES

concept are provided in Section 1.5. More details on the

development of the current concept are provided in the

sections which follow.

1.4.2 Task 1: Assess Existing Concept Development

Procedures.

During this task, three major activities were accanplished.

First, Army Life Cycle Management Model processes were

examined to identify the key products of an early training

estimation system and to identify the likely users of these

products. As a result of this effort, a detailed listing of

the documents, processes and output products related to

early training estimation was developed.

Second, interviews were conducted with likely users of ETES

to more precisely define current early training practices

and needs. These interviews indicaced that early estimates

of training requirements (that is, estimates prior to

Milestone 1) were seldcm, if ever, developed. The

interviews were further analyzed to identify the technical

problems which contributed to the lack of early training

estimation. Three major gaps were identified: (1) lack of a

tool for describing and storing task and training

. information, (2) lack of tools for estimating what a

training program should look like during the early phases of

1-i
the acquisition process, and (3) lack of tools for

evaluating early training program concepts.

Third, the psychological aspects of the system design

process were examined to identify the types of problem

solving aids that might be appropriate for inclusion in any

automated system design aids that might be developed under

ETES.

1.4.3 Task 2: Develop Method for System Concept

Description

During this task, five major activities were accaplished.

First, automated software requirements analyses and other

related tools were reviewed. The purpose of this review was

to determine if any of these tools would be used as vehicle

for the system description technology. The results of this

review indicated that none of these tools were appropriate

because (a) they were too canplex to use, (b) they could not

handle the frequent changes which occurred during the early

phases of the acquisition process, and (c) they could not

describe all of the types of data elements and data element

relationships that were required for early training

estimation.

When the investigation of requirements analysis tools proved

to be unfruitful, a second review was undertaken to

determine if an automated data base management technique

could provide the technology needed for early system

description. After reviewing off-the-shelf data base

management systems for mainframe computers, mini-computers,

and microcomputers, it was determined that a comprehensive

1-12
early system description capability could best be achieved

by developing a new microcomupter-based data base management

system that was specifically tailored for early system

design and training program estimation. After examining

existing Army microcanputer usage and the expected memory

requirements for an ETES data base management system, the

Apple III microcanputer was selected as the hardware vehicle

for this tailored data base.

In the third subtask, recent research in the area of human

computer interaction was examined to identify general

guidelines for guiding the construction of SDT screens,

human computer dialogue techniques and program logic.

In the fourth subtask, the SDT was developed in accordance

with the general requirements for early training estimation

identified in Task I and the human camputer guidelines

developed in the previous subtask.

In the fifth subtask, the SDT was applied to the Single

Channel Ground Airborne Radio System (SINCGARS).

1.4.4 Task 3: Develop Training Estimation Aids/Procedures

One of the major needs identified during Task 1 was the need

for the development of a set of tools for estimating

training program requirements (tasks to be trained, course

requirements, resource requirements, and costs) for

developing systems. With this in mind, Task 3, which was

originally intended to only develop procedures for task

generation was expanded to include procedures for estimating

(1) system functions, (2) system hardware/software and

i-13

[-* . . ...... - -°*'~*-..... . . %.................................
design concepts, (3) tasks, (4) training program

requirements, (5) training resources, (6) training costs,

and (7) training efficiency/effectiveness.

Development of the Training Estimation Aids/Procedures began

with construction of a detailed listing of the functions

that were required to estimate each of these seven system

elements. Existing tools which could be modified to perform

these early training program estimation functions were then

identified. Finally, a camprehensive set of Training

Estimation Aids and Procedures consisting of both modified

existing tools and newly developed tools was then

constructed.

1.4.5 Task 4: Develop Evaluative Technology

The task was divided into two subtasks. During the first

subtask, existing simulation techniques were reviewed to

determine if these techniques could provide a systematic

tool for evaluating emerging training concepts. The results

of the review indicated that current simulation technologies

were not appropriate for early training estimation since (a)

they required data that was unavailable during the early

phases of the acquisition process and (b) they could not

provide output on several of the criteria (e.g., resources,

requirements, costs) that were needed to evaluate emerging

training concepts.

In the second subtask, a listing of the functions which

must be performed in the early evaluation of training

program concepts was constructed. Exi sting tools which

could be modified to perform these functions were

1-14
identified. An integrated Evaluative Technology consisting

of both modified existing tools and newly developed tools

was then constructed.

1.4.6 Task 5: Develop User's Guides

Separate user's guides were developed for each of the ETES

automated tools. In addition, the manual ETES procedures

were consolidated into a single guide.

1.5 CURRENT ETES COMPONENTS

The current ETES has three major canponents; a System

Description Technology (SDT), Early Training Estimation Aids

and Procedures (TEAP), and Evaluative Technology (see Figure

1-2). The SDT is a data base management system for storing

and tracking task and training-related data. The data in

the SDT is used in the TEAP to estimate training

requirements for a new system. These training requirements

include estimates of tasks requirements, course

requirements, and resource requirements as well as estimates

of training costs, training efficiency, and training

"effectiveness". In the Evaluative Technology, the

integrated impacts of training requirements are assessed,

training alternatives are evaluated, trade of fs and

sensitivity analyses of key parameters are conducted, and

the relationships between ETES outputs and key Army

acquisition documents and processes are specified.

More details on the three canponents of ETES are provided in

the sections which follow. A detailed description of the

.11

1-15

Im
2

0

2

aa

P 2 : c u

RE 2

> ,

.A>00

61 0-.

U'U

U, z

2U U1 2 zo 

W, fa ito 

.j ZU 'A. u. CL 0 0

IC *

UA

UJ

z zz a Q Z uj (r u

30 > 1- 1
SDT is provided in User's Guide: ETES SDT. Detailed

descriptions of the Training Estimation Aids and Procedures

and Evaluative Technology are provided in the ETES

Procedural Guide. Detailed descriptions of three automated

tools associated with the Training Estimation Aids and

Procedures and Evaluative Technology are provided in User's

Guide: Media Selection Program; User's Guide: Resource and

Cost Estimation Technique; and User's Guide: Automated

Planning and Scheduling Technique.

1.5.1 The SDT

The SDT is a microccmputer-based data base management system

for (1) describing actual and projected system elements

including system functional requirements, system hardware

and software concepts, tasks, skills, and training program

elements and their associated resource and cost

requirements, (2) for storing the above information, (3) for

changing and updating this information, and (4) for

transmitting the information among the participants in the

acquisition process. The SDT data base management system is

designed to allow training developers to keep track of the

numerous system changes which occur during the early phases

of the acquisition process. In addition, it provides a

centralized data base, thus eliminating the redundant data

collection ef forts which typically take place among the

numerous training development organizations within the Army.

The computerized SDT also facilitates the use of automated

aids for estimating training program elements and their

associated resource and cost requirements. These automated

aids allow the training developer to quickly develop

I_--".
training requirements estimates for alternative training and

system concepts. This capability greatly increases the

probability that training requirements will be effectively

considered during the early phases of the weapon system

development process.

1.5.1.1 SDT Data Elements

To provide an effective conmunication vehicle for training

developers and other participants in the acquisition

process, the SDT has been designed to describe (a) training

programs and their associated resources, (b) the tasks which

drive these training programs, (c) the personnel who will be

required to perform the tasks, (d) the system designs which

generate the task requirements, and (e) the functional

requirements for which the system designs have been

developed. An overview of the major data elements currently

included in the SDT is provided in Table 1-3. Although the

data elements in the SDT are designed to remain fixed for

any particular application, the SDT software allows the SDT

data base manager to quickly change the data elements to

* mneet the needs of particular users and/or models.

1.5.1.2 Characteristics of SDT Data Base

Data base management systems such as the SDT use a

specialized "language" to describe the relationships among

data elements. Unlike many other data base management

systems, the SDT does not require the user to explicitly use

or even know about this specialized data language. In the

SDT, the data language is made "invisible" to the user

I - 18
LU

Lu Lu

z02 

LU

LU--

LUC6

U) z L z u zL LU LU ccj

U) U L 

E_ z z2 LLU I.ca 0U LU 0

Eu o = - I- wj0 2

Cu''.~4 > ~ 0N z

m 0 > -AOC m- 

* 0 -

2 ~ u LU L L

rzu LUJ LU ca LU 0

00 2 0 to LU -

CA U) 4 r

-C >- - : 0 cc LU _j CC 4 U) 0 z~ in 0 c( LU LU Q 0 uj CC 0 -002 i 0 L 0 0

LU C ca %~ Z L 0ZL t

2-i ~~~c n 0 U. 0j M - r U c UJ qILU ~.uj C0 Z j0 UV

*L 0 0 0 0 0a

U) , z) c U -J m : U)> I- U j > o= LU LU 0 0 2 U ca C 24- 0 j C2 hC L u w

f4 (a

LUz UL LU u 0

.. J W LU cc LU. 0 

U. Ck LU 2 Zj 0 LU rA ca

LUA ~z U~ 0 2 CL 0 ~ UZ- ) LUUx g-- 0 U Z 

CC U. Z ,- LU 0 g. 20 zo a L. 0 0 0 J a z oV4~4 )~ ZL Z ZU U

U.. *

1-19
L. V, 4. X 

-. . 7 - . . . -.

through "user-friendly" human-computer dialogue techniques

such as menu selection and question-and answering.

There are four major types of variables in the SDT data

language:

o Entities - Major system elements. Entities are

roughly equivalent to nouns in the English

language. The entities in the SDT are functional

requirements, system missions, equipment, tasks,

courses, training media, and personnel.

o Subentities - Lower level system elements.

Subentities are linked to entities in a

hierarchical fashion. For example, "task

conditions" are subentities of "tasks."

o Attributes - Descriptors that delimit or specify

important properties of entities. Each attribute

has a set of values associated with it.

Attributes are used to describe both entities and

subentities. For example, one attribute for the

entity "task" is "task frequency."

o Pointer Variables - Variables used to specify the

relationship which exist between different

entities, between entities and subentities and

between entities, subentities, and attributes. The

relationships specified by the pointer variables

determine the SDT data structure. (The SDT uses

elements of both hierarchical and relational data

base structures.)

1-20
-- -. V. 0 TW

1.5.1.3 SDT Configuration

When fully implemented, the SDT will utilize a distributed

processing architecture (see Figure 1-3). A centralized

data base for each weapon system (or weapon system

alternative) will be stored on a mainframe computer. At

periodic intervals, users will transfer a copy of the data

base from the mainframe to a local microcomputer. Once on

the micro, users can perform standard data base management

functions (input, output, modify). Thus, all major data

base management functions can be performed independently.

Once users have completed their activities, they can

transfer the updated version to the mainframe. A detailed

"audit trail" will be kept for each weapon system so that

users can systematically track and assess system changes.

The current version of the SDT is designed to operate on (1)

an APPLE III microcomputer with a modem, printer, monitor,

floppy disk drive, and a 5 megabyte hard disk and (2) a

Honeywell DPS 8/32 mainframe computer. The mainframe

computer is actually only needed when there is more than one

user and these users are located at several different

sites. The SDT has built-in security features which allow

the SDT manager to restrict data input, modify, and output

capabilities to a limited set of users.

1.5.1.4 SDT Operation

To provide a user-friendly interface, dialogue on the micro

computer relies on menu selection techniques for data item

and command selection and data output. Form-filling and

question and answer dialogue techniques are used for data

input. A "help" key is provided to allow users to obtain

1-21

72:• - . . . . . o .. /. . -. 
a..

a.L

4i

L41

UJU

LLILU >b

0 .

1--
guidance on all SDT commands and data elements. This "help"

capability can be activated at any time during the operation

of the SDT by pressing the escape key on the APPLE keyboard.

1.5.1.5 Generation of Data for SDT

In order to provide a capability for early training

requirements estimation, the SDT will be used to describe

system elements during the earliest phases of the

acquisition process. To generate data during these early

phases, comparability analysis procedures, which are part of

the ETES Training Estimation Aids/Procedures, will be

employed.

During the early phases of the system acquisition, when only

information on a system's functional requirements is

available, comparability analysis techniques will be used to

identify existing subsystems which are similar to those of

the new system. Historical data for these subsystems will

then be collected and modified to (1) meet the differential

characteristics of the new system and (2) correct any

inherent deficiencies in the historical data base. By

utilizing design and task data from comparable, existing

systems, meaningful early estimates of training requirements

can be made when only functional information on the

projected system is available (see Figure 1-4). Later, as

actual design concepts are developed, comparability analysis

can be used to develop estimates of tasks and training

program elements. Still later, when the actual system tasks

are available, only the training program elements must be

estimated.

1-23

S - - . " ": ' . "'" "" S -..'.- . ." . ". .-. . .. . .", ..-. ° , . . ..-. .." * "- " """. o ''- , - "-"-°; - "*'
C LL

*A 9

~. C . c

~ 00'- - CL

C o ____~ . n ~~ Q3

_ 4J

C- < 

..-. LLL_

6uS

1-24

07
Thus, by adding comparability analysis procedures to SDT

data base management capabilities, the SDT will be capable

of (1) describing alternative system concepts during the

earliest phases of the acquisition process, (2) describing

projected system elements, (3) relating alternative system

concepts to a common framework so that meaningful

comparisons can be made, and (4) refining system information

as more accurate and more detailed data are developed.

1.5.1.6 Application of SDT in the System Acquisition

Process

In its initial application to a particular weapon system

development process, the SDT can be used to describe the

system functional requirements which are generated during

functional analysis. These requirements specify the

functions which must be performed if the system is to

satisfy its designated mission need. The SDT can be

employed in a functional analysis as soon as the need for a

particular system has been specified. Formally, this

activity occurs after the approval of the requirements

document at Milestone 0, which initiates the Conceptual

phase of the acquisition process. However, the SDT could

probably be used to describe functional requirements even

prior to Milestone 0 if alternative system concepts were

identified earlier.

Once the functional requirements for a system have been

developed and described via the SDT, the SDT can be used to

generate and describe system designs. These designs specify

possible mechanisms for performing the desired functions.

These mechanisms include equipment, personnel, and

1-25
I

Once developed, the system design(s) can also be described

with the SDT.

Once the mechanisms for accomplishing the functions have

* been identified in the design concepts, the human tasks

which must be performed to utilize the system designs can be

specified. These tasks, which are the key building blocks of

training development, can be documented in the SDT. Once

the tasks are identified and specified in the SDT, training

estimation aids and procedures can be used to determine

training program elements, estimate training resources, and

develop training products. The resulting training program

and its associated resources can then be documented in the

SDT.

The SDT, like the other components of ETES, is primarily

designed for applications during the Concept Exploration

phase of the acquisition process, which runs from Milestone

0 to Milestone 1. However, the SDT can also be used during

subsequent phases of the acquisition process. The primary

uses of the SDT applications during these later phases are

to (1) make more detailed estimates of task and training

resource requirements, (2) determine the impact of

subsequent design changes on task and training requirements

via the data base management capabilities of the SDT, and

(3) conduct trade-off studies of proposed solutions to

identified training problems.

1.5.2 Training Estimation Aids/Procedures (TEAP)

The Training Estimations Aids/Procedures are an integrated

set of procedures and automated aids for performing six key

1-26
early training estimation funct ions: (1) Functional

Requirements Analysis - Systematic description of the

functions which the system must perform and, where

necessary, estimation of the hardware/software design

concepts needed to achieve these functions, (2) Task

Generation - identification of the tasks required to operate

or maintain the system, (3) Training Program Estimation -

estimation of where and how the tasks should be trained, (4)

Training Resource Estimation - estimation of the training

resources needed to implement the training program, (5)

Training Cost Estimation, and (6) Training Efficiency/

"Effectiveness" Estimation - estimation of the adequacy with

which the training program can be expected to train

personnel.

A listing of the ETES Training Estimation Aids and

procedures is contained in Figure 1-5.

The TEAP includes new techniques as well as existing

methodologies from other ARI research projects such as the

Training Efficiency Estimation Model (Jorgensen, Kubala, and

Atlas, 1981), recent work on cost and training effectiveness

analysis by Dawdy, Chapman, and Frederickson (1981), and the

Hardware Procurement - Military Manpower (HARDMAN)

methodology (O'Brien, 1983; Mannle, 1981). These existing

methodologies were modified to meet the specific

requirements of early training estimation. Brief

descriptions of the Training Estimation Aids and Procedures

follow.

1-27
a 2 .a Z" 4* z

I m z

00.

Z.U

*C 00

16 -C

-, I2220c

4o 4

us.

IC

. a - z. z
F:

1.5.2.1 Functional Requirements Analysis

In order to estimate training requirements for a new system,

the training developer must have information on the system's

hardware/software design, and its functional requirements.

Unfortunately, this information is often not available to

training analysts during the early phases of the acquisition

process. Consequently, it is often the case that no

systematic estimates of training requirements are developed

during these phases. The TEAP is designed to assist the

training analyst in overcoming these problems by providing

systematic techniques for estimating the system's functional

requirements and hardware/software design during the early

phases of system acquisition. The TEAP procedures for

estimating system functions, and hardware/software design

are at a general level. They are designed to provide the

minimum amount of information needed for early training

estimation. As the development of the system progresses,

and actual information on system functional requirements and

hardware/software design, are developed, these system

elements no longer need to be estimated by ETES procedures

but can be obtained directly fran the combat developer

(functional requirements) or materiel developer

(hardware/software design and manpower requirements).

ETES functional requirements analysis procedures provide a

description of the information which should be generated

during the functional requirements analysis and the steps

that one must go through to generate these elements.

Early estimates of system hardware/sof tware design are

generated via comparability analysis. Quite simply,

comparability analysis involves (1) identifying comparable

1-29

Im
-

existing systems, (2) collecting data on the comparable

systems, and (3) modifying these data to reflect the

*differences between the comparable existing system(s) and

the new system.

The most recent version of the DoD standard on Logistics

Support Analysis (LSA) (MIL-STD-1388) identifies comparability analysis as the preferred method for estimating key

system elements during the early phases of the acquisition

process.

Identification of comparable equipment in ETES takes place

in three major phases: (1) identification of the

Predecessor equipment system(s), (2) identification of the

Baseline Comparison System (BCS) equipment system, and (3)

identification of the New equipment system.2 The BCS system

is constructed by (a) adding subsystems to the Predecessor

subsystem to reflect additional capabilities required in the

New system (i.e., the system under development), and (b)

subtracting subsystems to reflect capabilities no longer

required in the New system. The BCS provides a baseline for

comparing the New system training requirements to other

similar systems. The BCS concept is directly congruent

with the new version of MIL-STD-1388 which specifically

calls for the construction of a baseline comparison system

during the early phases of the acquisition process.

2 The procedures described . in this section were derived

'rom the HARDMAN Methodology. A description of the HARDMAN

Methodology is presented in Appendix D.

1

1-30 .'

II "I
- -7 07

1.5.2.2 Task Generation Procedures

In the present version of ETES, comparability analysis is

used as the principal means of estimating task requirements.

In this approach, task data for the comparable existing

system(s) are collected and modified to reflect the

differences in design and/or employment between the New and

comparable system.

The task generation process begins with the collection of

task data for the equipments and MOSs in the BCS system. The

BCS is canposed of the existing subsystems (including the

Predecessor subsystems) which come closest to meeting the

New system functional requirements. The primary data source

for existing task data is the Soldier's Manual since it, by

definition, contains all tasks associated with an MOS. The

task data from existing systems are updated to reflect any

equipment or doctrinal changes that were made since the

source was published.

Once the BCS task data has been collected and updated, this

data is examined and compared to the New system hardware/

software design and functional requirements. BCS tasks

associated with equipments or functions which are no longer

needed are eliminated. New tasks are added to reflect new

eqaipment or functions. Descriptions of new tasks are

developed in accordance with existing Instructional Systems

Development (ISD) guidelines.

Identification of New system operator tasks is facilitated

by examining the subsystem functions identified during the

functional requirements analysis.

1-3

• .. . .., ., ... . • . . . .- . . 3.1. • . . - . < . . = . •..- . _
B~y adding and deleting BCS tasks, a New system task list can

be constructed. The reason for each deletion/addition is

documented using a systematic set of modification codes.

It is possible that a New subsystem may require the same

task as a BCS subsystem but that the essential characteristics of this task must be changed to reflect the New

system requirements. For example, the same task may be

required for both the BCS and New system but the frequency

with which the task is performed may differ.

To account for these changes to essential task

characteristics, all tasks with major modifications are

identified and the reason for each task modif ication is

documented. Tasks associated with major changes to skills

and knowledges are analyzed further to identify the specific

skill and knowledge differences between the BCS and New

system task. Tasks with minor changes in task elements 3re

not examined further (it is assumed that their skills and

knowledges have not changed). The reason for this strategy

is that the purpose of ETES is to estimate training

requirements rather than to develop training products (e.g.,

courses, manuals). Hence, only major changes to skills and

knowledges which would signif icantly impact training

requirements are determined.

Once tasks have been identified, their conditions and

standards can be developed using existing ISD procedures.

These same procedures are used to assess the adequacy of any

new task descriptions which are qenerated during

comparability analysis.

* IM

. . . .- < ,. << = . . .. . . . , . . . ,*.- .* . ." 7 - .* - i -

• .- - • - ,
1.5.2.3 Training Program Estimation3

Algorithms are provided to assist the training analyst in

determining the tasks to be trained and assigning these

tasks to training settings. To provide input to these

algorithms, tasks are rated on a series of task

characteristics (e.g., frequency, learning difficulty).

Several different algorithms are provided to meet the needs

of different phases of the acquisition process.

Quasi-programs of instruction (QPOI) are constructed by (1)

modifying or deleting modules from existing courses to

reflect the task deletions and modifications made during

task generation and (2) adding modules to reflect the unique

requirements of the New system. As part of the QPOI

construction process, the instructional methods and

curriculum hours which must be devoted to each module are

determined.

Media for the training program are selected by the

application of an automated aid, the Media Selection and

Efficiency Estimation Program. This program is an extension

of the Training Efficiency Es-imation Model (TEEM) produced

by Jorgensen et al. (1981). The Media Selection and

Efficiency Estimation Program significantly expands the

capabilities of TEEM by recasting media selection as a

dynamic programming problem and automating these procedures

on the Apple III microcomputer. This automated program

permits the user to employ the SDT to input and store the

3 The current version of ETES only contains procedures

for estimating training programs for individual

institutional training courses.

1-33

->~....-.............................. .. < .i.,< - - ........ .. . .- ...... ,
Ip

data needed to feed the media selection procedures. Using

dynamic programming techniques, the program can assign tasks

to media in a manner that optimizes efficiency, relative

cost, or efficiency and relative cost.

Efficiency is determined by camparing the stimulus,

response, and feedback characteristics of the individual

task to the stimulus, response, and feedback characteristics

of potential media categories. More specifically, a score

is calculated which describes the match between media and

task characteristics. Efficiency for each task-media

combination is calculated by dividing this score by the

maximum match that may be achieved for the task.

Total efficiency for a set of tasks is determined by

aggregating the efficiency score for individual tasks. An

additional efficiency measure can be calculated by weighting

the efficiency of each task by the task criticality score.

This task criticality score is calculated by aggregating the

task factors typically used in selecting tasks for training

(e.g., task frequency, percent members performing, task

delay tolerance, etc.).

1.5.2.4 Estimation of Training Resources

These procedures estimate the training resources needed to

implement the training program. The training resources

encompassed by these procedures include (a) the number of

students to be trained, (b) number of instructors and

support personnel, (c) facilities requirements, (d) training

device and training equipment requirements, and (e)

ammunition requirements. Included among the procedures are

1-34
techniques for using off-the-shelf automated worksheet

software (e.g., VisiCalc) for storing and applying several

of the resource determination algorithms.

1.5.2.5 Estimation of Training Costs

These procedures estimate the costs of the resource

requirements and aggregate these costs into a total course

cost. The procedures include techniques for using off-theshelf automated worksheet software to calculate course

costs. The procedures also describe how to use modified data

from comparable existing courses to assist in the cost

estimation process.

1.5.2.6 Training Efficiency/Eftectiveness Estimation

Procedures are provided for determining the training

efficiency of selected aspects of the training program

(e.g., media). Training efficiency is defined as a measure

of the extent to which the characteristics of the training

program element match the task characteristics of the New

system. For example, media training efficiency is

determined by comparing the stimulus, response, and feedback

characteristics of the tasks to the stimulus, response, and

feedback characteristics of potential media categories. An

additional variant of this efficiency measure may be

obtained by weighting each task by its "criticality" where

criticality is determined by aggregating weighted scores of

key task characteristics (e.g., frequency, consequences of

inadequate performance).

-1-3

....................... ...............
Training "effectiveness" is determined by using modified

versions of procedures outlined in Dawdy, et al (1981). More

specifically, a group of subject matter experts is presented

with the following information for each task: (a) the target

population description of the personnel who will perform the

task, (b) a description of the task including its associated

conditions, job performance standards, and general skills

and knowledges, (c) a description of the training program or

training program elements (e.g., course length, methods,

media) that will be used to train the task, and (d) the

criterion which must be achieved for the task (if different

from the job performance measure). Each subject matter

expert is then asked to estimate the expected percentage of

soldiers in the target population who will pass the

.. criterion given that training program.4

It should be noted that the ETES TEAP is designed to

estimate training requirements for a New system. The TEAP

is not designed to provide techniques for actually

* developing instructional materials or programs (e.g.,

course, instructor packages, student packages, training

literature, training devices). Thus, the TEAP will assist

training analysts in determining what instructional

materials/programs should be produced, what the content of

these materials/ programs should be, and in estimating the

cost of these materials/programs.

It is recognized that this measure of effectiveness,

may differ from other conceptualizations of training

effectiveness. It is also recognized that this approach

only provides a gross measure of "effectiveness" that should

only be used during the earliest phases of the acquisition

process.

1-36

-7-
74 w. _

This focus on training requirements estimation rather than

on training product development is one of the unique

characteristics of the TEAP. This focus on training

requirements distinguishes it from other existing training

development methodologies such as the Army's Instructional

Systems Development (ISD) process and the Army Research

Institute's Training Developer's Decision Aid (Hawley,

1979).

The training requirements produced by the TEAP provide

front-end information needed for the early planning and

analysis of training programs. The training requirements

information produced by the TEAP provides the foundation for

the actual construction of training products in ISD and

other related methodologies. Techniques and methodologies

which can be used to develop training products are provided

in TRADOC Pam 350-30, Schulz and Farrell (1980), Hawley

(1979), and Fink (1981). An overview of the relationship

between ISD and ETES is provided in Figure 1-6.

1.5.3 Evaluative Technology

The Evaluative Technology is an integrated set of procedures

and automated tools for (1) developing figures-of-merit for

assessing the integrated impacts of the training requirements developed in the Training Estimation Aids/Procedures,

(2) identifying potential problem areas for system training

and the likely sources of these problems, (3) identifying/

evaluating training problems, (4) developing trainingrelated input to key acquisition documents, and (5)

determining/evaluating training development schedules (see

1-37

-- - . . -- . ,. 
ISO BLOCKS IN EACH OF THE FIVE ISO PHASES

THE BLOCKS IN EACH PHASE ARE:

SEEC CNS hLCT ANALYZE SELECT PHASE I ALYZE > 0..OsMCI PS 0EXISTING INSTRUCTIONAL

Joe F~~UNCTIONSP44MNJ OR FTN

x x x x x

PHASE 11 DEVELP OEVELOP ENTRY QUENCE & OBETIE ESSBHAVIOR STUCTURAE

x x

k SPCF Y INST RqUCTON 4 5 i~r~clO t~gwfIvILP •VALIOAIE

NSTO| ITL

-- • I MU~ TIONALi•,NU'

PHASE IVI 0:""E Mct4GEMIT h

MLAGEMENT IN2 IoN

PHS V~i UTOA ONUC m 

PLAN

PHASEV V____ CNU EVS

PHNSV, i •'EXTENAL

EVALUATION VLATIO 1 J T

X = Function Covered by ETES

FIGURE 1-6. RELATIONSHIP BETWEEN ISD AND ETES

1-38

• " "I' -' " '+ " - " ,." " ",+ " "" ' ' - I" " I ' ' " " " " " " " ' " " +" . . . . ." ''' ' '''-"'"
Figure 1-7). A summary of the procedures in each of these

areas is provided below.

1.5.3.1 Development of Figures-of-Merit

Procedures are included for identifying figures-of-merit

which summarize the essential features of the training

requirements. Eight potential training figures-of-merit are

utilized including (1) cost, (2) training efficiency, (3)

training "effectiveness", (4) congruence with training

development guidelines, (5) congruence with program

requirements, (6) training complexity, (7) training

capacity, and (8) feasibility. In addition, a procedure is

provided for constructing a summary evaluation score which

aggregates the scores on the individual figures-of-merit.

This summary evaluation score provides a global measure of

the "goodness" of a training program.

1.5.3.2 Identification of Problem Areas

Procedures are included for identifying training problem

areas. These problem areas consist of the Army Military

Occupational Specialties (MOSs) which have high figure-ofmerit scores relative to (1) the Predecessor system, and/or

(2) the other MOSs in the New system. Procedures are also

provided for identifying the courses, equipments, and tasks

which contribute to the high figures-of-merit and

identifying the training program elements which are likely

causes of the high figure-of-merit scores.

1-39
as 3~

a'4

r-Oz

1-40
1.5.3.3 Identification and Evaluation of Alternatives

Guidelines are included for (1) identifying training

alternatives which can address the training problems, (2)

evaluating the training alternatives through selected

reapplication of TEAP and Evaluative Technology procedures,

(3) conducting sensitivity analyses of key parameters, and

(4) assessing the impact of non-training system changes

(e.g., hardware/software changes) on training.

1.5.3.4 Development and Evaluation of Training

Schedules/Plans

Construction of training development schedules for emerging

systems is a difficult process. Over 100 developmental

events are listed in TRADOC Reg 351-9, the Army regulation

governing training plan development. The sequential

relationships among these events are camplex and are not

described in any systematic and integrated manner in TRADOC

Reg 351-9. To assist users in developing training

schedules, procedures are provided for using automated, offthe-shelf scheduling software (e.g., VisiSchedule) to track

and monitor the training development schedule. By using

this software, the training developer can quickly and

efficiently respond to changes in the training development

schedule. Use of off-the-shelf scheduling software is

facilitated by the inclusion of an input data diskette which

(a) describes the events ir. the training development process

(as specified in TRADOC Reg 351-9), (2) describes the

temporal/sequential relationships among these events and key

acquisition milestones, and (3) lists the expected duration

of these events for a "typical" major Army weapons system.

This data diskette significantly reduces data input

requirements. In addition, it eliminates the need for an

1-41

S S ,a ..... S. ,
analysis of the complex sequential relationships among

training development events which are either implicitly or

explicitly specified in TRADOC 351-9.

1.5.3.5 Develop Inputs to Acquisition Processes and

Documents

Major Army acquisition processes and documents related to

early training estimation were identified and are summarized

in Table 1-4.

1.6 RELATIONSHIP BETWEEN ETES AND OTHER ARI PROJECTS

ETES is part of the Army Research Institute's thrust to

develop systematic techniques for determining manpower,

personnel, and training requirements for developing Army

systems. Other major projects currently in this thrust

include the Man-Integrated Systems Technology (MIST), the

Army Manpower and Personnel Requirements Estimation Program

(ARMPREP), the Army Hardware Procurement-Military Manpower

(HARDMAN) program, and the Human Resources Test and

Evaluation System (HRTES). MIST is the central project in

this thrust since it will integrate the other projects in

the thrust. Table 1-5 summarizes the data that each

component will contribute to the three phases of MIST

development. ETES will provide MIST with two critical

components: (1) techniques for estimating and evaluating

training requirements and (2) a prototype data base

management system (the SDT) for storing MPT information.

A description of the other projects in the ARI thrust is

provided in Appendix D.

a'.

1-42

v -7

€.1
Table 1-4 Major Army Acquisition Processes and Documents

Related to Early Training Estimation

" Cost and Training Effectiveness Analysis (CTEA)

. Outline Individual and Collective Training Plan (OICTP/ICTP)

" Logistics Support Analysis (LSA)

" Operational Testing (OT)

" Training Device Requirements, Documents, and Processes

- Training Device Letter of Agreement (TDLOA)

- Training Device Requirements (TDR)

- Training Device Letter Requirement (TDRL)

- Training Device Study (TDS)

* New Equipment Training

* System Requirements/Documents

- Justification for Major System New Start (JMSNS)

- Letter of Agreement (LOA)

* Request for Proposal (RFP) Development/Evaluation

* Personnel Documents/Processes

- Tentative Quantitative and Qualitative Personnel

Requirements Information (TQQPRI)

- Integrated Personnel Summary (IPS)

* System Level Documents/Processes

- Concept Formulation Package (CFP)

- Cost and Operational Effectiveness Analyses (COEA)

- Tradeoff Determination (TOD)

- Best Technical Approach (BTA)

- Decision Coordinating Paper (DCP)

- Defense System Acquisition Review Council (DSARC)

- Army System Acquisition Review Council (ASARC)

Mission Area Analysis

1-43

-- 7
rr rr~r ' - -~r~nz

~~~~~~- 3 ~~~~ i- r, ~ - x s- * x- *-

zi0

-31

3l 3 - <X'- 43 Z3 3 3 z

r iU3 3 wt ix. w

E- 4 .-' E- E- E-E- E- 444< C2.2( u.CU w w. w0 2:23 caz w wZ7, Z ZZ Z

ZI.

0 ~ w0))U In ~ 4 

E- E-' E3 E -< 3- <- 3 

14z z 4Z Z ZZ z z z

4- 2I) 0n c1 0o CD22 m3 a2 0 0 0 0 a a

F E, - E-FE-F-E- E-< < 44u 4 U < u Q U0) 3- xi 0 322322230 0 0 Oz z3 z 3 3

=7-

V;L3Ln2

En0p

F- E 0 ' ,3

0 , 4 ) L ) E - .

0 w3 w3 r . a)= ) c W 4. 0 UV ,aV

212 F- I 0xa o- )" =C - Cl) a0 a)4c 3

- .12 3 3 3 3: z

F- 0 ~ 0 ~ 12 ~ W WWO ~ 00 a) 0c0)

F- 4~ ~ 4 - : FFFF - 4 4 c u wuCo le 

>. rLcC1

0 4>

Lo -i > >1 a-'4 0-> - nu

SC C, W

-~~ ~~ 3 -440 oCOo~o oCCC
".7.

Iu 1

)

I WI E- U)

LOI~~0~ E, 14 U )W

-a) EZ 'U) U)

ZI E: 0, E- S ZU) m U)M

E- Wa))

~U) Q)E

( -.. E cca)a)T

w 0a

E- a. ~

all U)E 0FV

- ) UI U) E

U) Z-U U) 

- O -a - a) >- - : C w _ C

a) 0 :

E. 0 a

LI (.0

> V) ) a)t

Iw m- C -V2 a) >U

z ai u a) U" I-U , "

Q) w~ W- "-a

Z-r o>.a F-0 'E' 

7U c0 Z. In a) 

a. U)>.)C

a:a

1-45W
THIS PAGE INTENTIONALLY LEFT BLANK

46 --
SECTION 2 - TASK 1: ASSESS EXISTING CONCEPT

DEVELOPMENT PROCEDURES

During this task, three major activities were accomplished.

First, Army Life Cycle System Management Model processes and

r documents were examined to identify the products that must

be developed by an early training estimation system and to

identify the likely users of these products. Second,

interviews were conducted with likely users of the SDT to

more precisely define current early training estimation

practices and needs. Third, the psychological aspects of

the design process were examined to identify problem solving

aids which might be included in any automated aids developed

in ETES.

2.1 ASSESS LCSMM INTERFACES

During this tasK, a detailed review was conducted of

existing Army and DoD documents relating to system

acquisition or training development. The purpose of this

review was to (1) identify current requirements for early

training estimation, (2) identify likely data elements for 4-

inclusion in the SDT, and (3) identify any tools currently

being used for early training estimation. The review

focused on documents and processes related to Mission Area

Analyses (Pre-Milestone 0), Concept Exploration, and

Demonstration and Validation phases of the acquisition

process. The ETES review of training-related acquisition

processes was the major input for a later report describing

manpower, personnel and training interfaces in the Army

h 2-1

" . . ; 
~

L£

LCSMM produced by Wagner (1982) as part of the ManIntegrated Systems Technology (MIST) project.

Table 2-1 describes the documents and processes which were

identified as having the most relevance to early training

estimation. .4.

The two documents/processes which were determined to be the

most critical to early training estimation were (1) the

Outline Individual and Collective Training Plan (OICTP) and

(2) Cost and Training Effectiveness Analysis (CTEA).

2.1.1 OICTP/ICTP

The OICTP and ICTP are plans which support the development

and implementation of new or revised individual and

collective training programs at institutional and unit

levels. The OICTP and ICTP are the major resource and

planning documents for developing training for new Army

systems. An approved OICTP is sufficient justification to

enter, manpower and funding requirements into the Army's

programming and budgeting processes for inclusion of the

TRADOC Review of Manpower (TRM). As specified in existing

regulations (TRADOC Reg 351-9), the OICTP/ICTP contains a

proposed training concept and training strategy. The

concept and strategy are typically generated by the

proponent school responsible for the New system's training

requirements.

The OTCTP/ICTP is generally an evolving document that

increases in specificity (via appropriate updates) as the

2-2

...:. " ." <:.i~i> " -..:.: .:: i> .]-- :::. ' " " " " ' " " .. . " ' " " "" " " "
3t Z

C

0 -4 0>~:~

w ro U)1 -E- 1 C L 1)

c~ ~ Q)J c0nE 0' a U>-

QO .- W) C a) Q)) c

E cW AU)C

a)) > 'C , )(V U.- 0 V 41a) C- -! 1) C daO.)

(1) U. 0 c 0 ( V-4 ~ E ) a' w U) 

CO w.' C OU 2.- W0(a wC . 1j .

z ( 0 41 L. W1 rnDw C.' 0

U - 4 J (1)U U .j cn E .c

04 -W C -U c U)E () u 0

E- 41W* -C r_) . 4jU . )u -

a) U - 0 .(1 - 0 w a-~ (aEW 

E* - r(.C 41 *-0U1 e- (a Cw ) r4. 41 0

U)) 0 4-1 1' ul (1) 43 0

U) 04U) E-) >-J 4j z W c

tn U) .fU - 4-1 - W> ) C 

W 0 41 U. W O > ~ E (1) O 0W - %4(~ < -W C- WQ0 CC W W

* 4 M ))7f C4.d W0 > '~ 4J~ C 0 L, wU) 4

*0 E C4~ EC'V -~ C Q) cWl Cu

x >- < 0 E- ~0- E - U > E- 4

= - U3 CEl -0 L .

CJ (0C 0 -- . -4 %64 -4(a0 3 4. 41~ -..JW CWa ~ r 4jCr :3(n)~ 0

V) C < C >- 4d)U ~~- ZWW fj) 4 dJ 4: Z

l u 4-- 6a'-4 Q~ >-W E C 4. C UC

E--E C a4~ *- - E- qo 0 - ) jU Q)-C <:U U~- E- U) 41: 0 zC u CC -~ Z . Cp U)C- a (n U CC~ ~ U.04 'E--u) * * -aC~ 0 ~ 0)0)) CC * 4-)J)

z 'o ~ (1) '' r- (U0.. 

E- E-' U ,~ a) 4 *- 0 .. . 7.C 0w E- ( UI u - E E- ra) 0-J CLC E0) - 2 U) 1

U) -1 C.. U'zU) w WW)aj> E.z M E' ZC 01 02 ) 0 A . 41. -. 6.J (a 4 W)r C).V 03 U 4) 0>- ea U) &))U) U

3- SE E-'z)) Wl~. w:-3 E- 0 E- E-U)0 E-U))

0

I) 4:

LO0 (1) C

I Lo 0 - E w I -

I' 0C Q) ~ ~ a)- IV 

> E- C4: : E-C0' Z p

>).. CE-. C: WICa - ~ ~ ~ 0)' ~ :-a uIO 

* *.0 .-. 10 " .

-14>

>,Ul ''W a v~- C:-'

1O. I a' ,4~Z -

C- 0

-~u a'l

>- ~ a''

2 C- ~ -0 - I C7

C: u' ='4-U - ' = < I' -C >'' 0 C~r

71 J 7 1' 1- ''-j 3
0 w 3 

0 >, -44~ 1

c in ~ 60 C-I (a...4

C 'a w AD

OCC' 41 V '

U) = 24 a.c E 0E

a E- 0 U 40 0 W~4

nOo ACot

U (a ~ 0(nC a 4 3 44 0 > 0

1 > C -4-

W U4 3 a W 8 E3 a

4J C6. 4)

z 0rn 4) 'U ' XC C 0 n

4) 60,) 40 w0, Ad D (a-4

>, Lo f 0-Z c LW ~ . ur -.

(2) -4 'U -C .i> .. 0

~ U wW C..1 q

0-). 0 u InU. 0 w LJ-.- ow.- C 40. 102 4- j m cu

41C 3 (J2 4 -0' C 1

C u4) Ia.1) a) C ~ -4. M (n

C. 4 CL 02)4 Mi~ 4.)-

04 ~0 -. U = -4 .- a

-) 3t 0 E-1>E~.

~0 a w 4C- c~r >U

-~ ~ ~ EZC1 E 41~~,-~-.

0 -4I" r

o* t6 -410(

a) 0)-W 

cnI=ui)E C Cu (Z

19U

0n

60 u 0 En a (o

rntf 0 - -4 z

Z C- CL C:

uJ -'a ) 03 v !a. ; O, 'o - a, -

~~I U ) 13C 

A.~ 0JC0 U >,CZ

'O ~ C ~ E 0 E-, 00 -

* ZI r -. ~1C o-'O 2-4Z4
[ .-

system under development is further defined. The OICTP/ICTP

provides significant feeder data for the Tentative

Qualitative and Quantitative Personnel Requirements

Information (TQQPRI), the Training Effectiveness Analysis

(TEA), and the New Equipment Training Plan (NETP). After

LOA approval, the OICTP can be used by the contractor to

facilitate the identification of training requirements

conducted as part of the logistic support analysis.

The CTCTP/ICTP is intended to guide the development of

training subsystem requirements and to provide the general

framework for their future incorporation into the existing

training base. The regulations specifically emphasize the

intention to use the proposed training concept in the

OICTP/ICTP to "... identify the constraints which training

requirements and resources may impose on the design of the

material system" (TRADOC Reg 351-9, pg. 5). The regulation

also mandates use of the OICTP/ICTP as the vehicle to

describe "...the integration of the training subsystem into

the development of the total system and the integration of

the developing system into ongoing training systems."

TRADOC Reg 351-9 further stipulates that the OICTP/ICTP must

incorporate the principles of Army Training 1990 (AT 90)

into training for a new system, for both institutional and

unit training and at all skill levels for the MOS/SC

affected. The OICTP/ICTP is intended to develop and

describe a systematic and feasible strategy for training,

extending from the development of "initial qualification" to

continuing "sustainment of the proficiencies" needed for the

successful fielding and operational deployment of the system

being acquired.

2-5

* . * * .]
As specified in existing regulations, the OICTP/ICTP

provides information on the training required to integrate

replacements fran the training base into the unit, and to

qualify personnel for higher level tasks as they advance in

grade. The OICTP/ICTP is further directed to provide

information on the identification, quantification, and need

for training devices, simulators, documentation/

publications, training aids, support facilities,

instructors, costs, and all other support and logistic

considerations necessary for the implementation and test of

the proposed training plan.

Table 2-2 provides an overview of the elements in and

OICTP/ICTP and describes which ETES procedures can provide

input to the development of these elements.

2.1.2 The Cost and Training Effectiveness Analysis (CTEA)

The CTEA process is potentially the most critical LCSMM

process related to early training estimation since it

requires the user to not only estimate what the training

program will look like (as does the OICTP/ICTP), but also to

evaluate these training programs and to provide training

related input into the overall system development and

evaluation process.

The new TRADOC Reg 351-9, governing the OICTP process,

contains the most current definition of the CTEA. It

defines CTEA as "a methodology which involves documented

investigation of the comparative effectiveness and costs of

alternative training systems for attaining defined

performance objectives." The definition further specifies

2-5

~* - ~ ~ -*- * ** y~ '-. * * * ~ -..-.- **9** .... . .
'46

IIi

rLr

4, 4

'o IL

0 0 

to , U) w 

~ ' a 0,

- 16 

5 0o IL 4,

a I 4,I

* 71 4,.

~ 4, C .1. I

C 4,- e 2-7
4I

C 0

"" I S

0- 0

* '0

* S}

* mm-a C I , I!" * -. a

, 

in , I !

___,_______________ - , -1 -- - -

I-I '

I-I

I ~* 2 -~. . . . . . . * A ~°*
ft .V -. -. 7 %( 9. . . q r P! . W~ .

that a CTEA can focus on any one or a combination of the

following:

o Training concepts

o Training strategies

o Training equipment/devices

o Programs of instruction

o Training impacts of

- New materiel

- Organization

- Tactics

- Families of systems

Regardless of the specific focus of the CTEA, TRADOC

Regulation 351-9 stipulates that the CTEA should include an

analysis of the attainable levels of proficiency and the

costs associated with each alternative. In addition, a CTEA

should include a cost effectiveness trade-off analysis of

the feasible alternatives. This regulation further

specifies that a CTEA must:

[.~

o Ensure that training development is initiated

early in the life cycle of hardware systems and is

accomplished in coordination with combat

developments,

o Ootimize soldier-hardware subsystem interface,

o Ensure that all feasible training subsystem

alternatives are considered,

o Optimize soldier-training subsystem interface,

2-9

................
=0 Optimize soldier-training subsystem interface,

0 Recommend the preferred training alternative based

on cost and training effectiveness, and

0 Provide decision makers with more precise

information at critical points in the acquisition

process concerning the total system (comprising

the training, hardware, and other subsystems).

These objectives demonstrate that the CTEA, unlike the OICTP

and the QQPRI/BOIP, is an early training -related document

intended, in theory at least, to influence the hardware

system design. This is a very significant dif ference from

the OICTP which is, by regulation, primarily an MPIT planning

and resource document. The steps in the CTEA process and

the corresponding ETES procedures are detailed in Table 2-3.

2.1.3 Identification of LCSMM Reports

As part of the review of existing LC SMM processes, an

attempt was made to identify any report formats which might

be relevant to early training estimation, and in particular,

to the identification of data elements for the SDT. Table

2-4 summarizes the report formats that were identified

during this process. A listing of the documents that were

reviewed during this process is contained in Appendix E.

2-10

'a'-

'-1

S . ". ° o ° . ° . - ' . .° '
-~7 ov 'R T--7 f' "1 -7 -- 'Tc-w'r -

00

~0 0,

0'

-, c

SC 0

.2 4,0 -4 C

oo -0 io 0 0)

C0 V, 0 t

Ct> or C .0r0

00 4, 40 C

CO O.0

wa

2-' tj2 2 -11
I o I 0 0 I 

,, * 4,

.1 !

o 0

0 ' 10

L4I 0. 4 ,

0,4. L ,4

_- + ~ 4 4 ,4 ,=",",,

+ . ++ , 4, 4, 4). . _ + ,

0 0 w - + - U.

E-4 x

4+ 4, - + .

In I In 4,

ad ~ ~ 00, ~ Q , t; ~ I4 

2-12

+%+" t ' u ,-, 0. 0. ,

"

". ". 0. -" . 4, 0.4,- 4" : , "4,4 _o + , I . .. ', , +,'. . t +. ,, ,2 .
Table 2-4 Summary of Standardized Reports Related

To Early Training Estimation

Title Document Items

Task List, TRADOC CIR 351-8 Task Number, Task

Training Setting Description,

Assignments Training Materials,

Responsibility for

Training

MOS Training TRADOC CIR 350-3 Task No.,

Program Task Description,

Worksheet No. of Duty Positions,

Training Products, Supporting

Tasks

TCH Computation TRADOC CIR 351-12 Course Module, Instructional

Sheet Hours, Method, Instructor

Contact Hours

Task Analysis DI-H-0006 Equipment

For Manning (MIL-STD-XYZ) Nomenclature

Report DI-H-0004 Part Number, Task No.,

Task Description

Task Inventory DI--H-0001 Mission, Scenario,

Report Conditions, Function,

Job/Duty Position

-1, 3

NI

Fm

p..-].

" '- '.- , " -* "*. -* ' . -. " " - ' ' ' : ' . """ " " ."* " ' "" ""- ""-""* -" " - " a" - ' . ' ' ' '' ' '
Table 2-4 (continued)

Title Document Items

Task List MIL-M-63040 (TM) JPG/ETM, Equipment, MOS,

Item Code, Tasks/Subtasks,

Conditions, Standards,

Comments

Task Selection TRADOC PAM Task No; Task Description

Sheet 351-4(T) Task Characteristics/

Factors

Job and Task TRADOC PAM Task No., Task

Analysis 351-4(T) Description; Condition,

(Worksheet) TRADOC REG 351-4 Standard, Job Title,

Supervisory Job Reqs.,

Task Characteristics,

Enabling Skills and

Knowledge, Equipment Used

to Perform Task.

Task Analysis ARI Research MOS Skill Level, Task

WorKsheet Product 80-13* Number, Task, Initiating

Cues, Conditions, Standards,

Refer ence/Tips

Task Summary ARI Research Task, Conditions,

Sheet Product 80-13 Standards, References,

Performance Description

Task Information TRADOC CIR 351-12 Task Number, Title,

Sheet St aida r d

Co urse TRADOC CIR 351-12 Cour se, Hours

Summary

2- 14

. .. . . ...
Table 2-4 (continued)

Title Document Items

Task Cluster TRADOC CIR 351-12 Task Cluster, Hours,

- Annex Student/Instructor Ratio,

Lesson Objective, Lesson

Reference

Training Resource TRADOC CIR 351-12 Course Number, Title,

" Section Manpower Requirements,

Expense Elements

MOS Course TRADOC Cost Course Costs by

Costs Analysis Program Courses in MOS

Course ATRM-159 Course Title, Course

- Costs Number, MOS, Course Cost

Elements

Equipment MIL-M-63035(TM) FSN, Part Number, FSCM,

List Description

Tool 3nd MIL-M-63035(TM) Maintenance Level,

Tpst Equipment Nomenclature, National

Stock Number

%IairnancP MIL-M-63035 (TM) Group Number, Component,

Aj locat ion Maintenance Function,

Char Maintenance Level, Tools

and Test Equipment No.

2-?

- .. , ,: :i . . ., . _ : ... .i . >_: , <.?< , i k. .. /.-:, ., . ,_ M:
. . . . . -- 

Table 2-4 (continued)

Title Document Items

Preliminary MIL-M-63035(TM) Task, Equipment, Tools

Task Development and Test Equipment,

Worksheet Personnel Reqs., Equipment

Condition, References,

Preliminary Tasks, Follow-on

Task, Task Elements

Training TRADOC REG 351-9 System Data, Development

" Subsystem Schedules for Following

Summary Training Products - Technical

Sheet Manuals, SM/TG/JB, ARTEP,

Resident Training Program,

Resident Training Equipment,

Devices, Training Literature,

TEC, Audiovisual, ACCP,

Facilities/Ranges, Ammunition,

NET. Also, Resource

Requirements by Fiscal Year.

LSAR Input MIL STD 1388-1 End Item Maintenance

* Data Sheets Requirements, Reliability

A through J and Maintainability, Summary

of Maintenance Task Data,

Maintenance and Operator

Tasks, Training Materiel

Description, Facility

Description, Skill Evaluation,

Supply Support Requirements

BCIP Feeder AR 71--2 oescription of End Items to

Data Sheet be Entered into lnventory,

Items to be Replaced

QQPRI Report DARCOM P-750-16 Description of MOS, Qualified

Part 1 or to be Qualified Durina the

Development Process

2-]6
2.2 INTERVIEW ACQUISITION PARTICIPANTS

The review of the LCSMM documents identified prescribed

requirements pretaining to the area of early training

estimation. To identify what was actually done, interviews

were conducted with several of the likely participants in

early training estimation. Table 2-5 summarizes the

organizations that were contacted during these interviews.

The major finding of these interviews was that little, if

anything, is typically done in the area of early training

estimation prior to Milestone 1. The only activity which

takes place during this period is the inclusion of some pro

forma statements of training requirements which are included

in system requirements documents such as the Letter of

Agreement (LOA) despite the fact that several regulations,

particularly the newer regulations, such as TRADOC Reg 351-

9, indicated that several training-related activities should

take place during this time frame. Three reasons were

identified for this lack of early training estimation: 1

(1) Lack of a Sys tematic Flow of Information Among

Participants in the Acquisition Process - To develop

estimates of training requirements, training developers

require information on actual or estimated systems

functional requirements and design concepts as soon as

they are generated, and to maintain the accuracy of

these estimates, these same training developers must be

quickly informed of design changes and updates. A

1 A more recent review by Adelman et al. (1982) has

indicated that these problems still exist.

2-1.7

t - "." ,T I : -' I -' - - -" .- -'. ." -- - -" .I"
Table 2-5. Organizations Interviewed During Task 1

Location Organizations/Individuals Interviewed

Air Defense School, * Director of Training Developments

Fort Bliss, Texas 0 Director of Combat Developments • Division Chief, Short Range Air

Defense

0 Division & Assistant Division Chief,

Patriot and Nike Hercules

* Patriot Project Manager for Training

* Educational Specialist, ROLAND

* Educational Specialist, SPAs

* ROLAND Project Officer, DTD

* Associate TSM, Patriot

* Associate TSM, Roland

* Associate TSM, DIVAP

Armor School, a Associate TSM, XMI

Fort Knox, Kentucky • Staff of Directorate of Combat

Developments • Staff of Directorate of Training

Developments

Redstone Arsenal, • Chief of New Equipment Training,

Huntsville, Alabama Patriot

0 Chief of Training Developments,

Patriot

* Chief, New Equipment Training Branch

* Chief of Training Developments,

ROLAND

Army Training Support * Staff of ATSC

Center (ATSC)

Fort Eustis, Virginia

Training Developments * Training Specialist Overseeing

Institute, Transfer of Information on New

Fort Monroe, Virginia Training Technologies

Signal School, • Director of New Equipment Analysis

Fort Gordon, Georgia Division and Staff

• Director of Combat Developments

and Staff

* Training Analysts, Directorate of

Training Development

TRASANA, * Staff, Training Effectiveness

White Sands Missile Range Analysis Division

2-18

*. . . . .. . . . . . . . . . . . . . . . . . . ... .' .
Table 2-5 (continued)

Location Organizations/Individuals Interviewed

Army Logistics Managemient e Staff, LAMC

Center, Fort Lee, Virginia

DARCOM, HQ • Staff of Associated Directorage

Alexandria, Virginia for ILS

MRSA, Lexongton-Bluegrass • Staff of Logistics Engineering

Army Depot, Kentucky Branch

TRADOC, HQ e Director of Human Dimensions

Fort Monroe, Virginia and Staff

* Staff of Deputy Chief of Staff

for Training (DCST) • Staff of TRA

2-19
.

functional requirement defines what a particular design

must do to accomplish a specific mission need.

Unfortunately, under current practices and procedures,

training developers do not receive information on

system functional requirements and design concepts in

any systematic format, nor is there any formal

mechanism through which they can obtain information on

system updates.

In addition to the information flow problems which cut

across functional areas, there were additional information

flow problems among individual training organizations. More

specifically, there was a large amount of redundant data

development and data collection among training organizations, particularly during the early phases of the

acquisition process. System developers and TRADOC

organizations did not have the capacity to share

information. This led to unnecessary duplication of

effort. As a result, different training-related

organizations often were not "singing from the same sheet of

music."

(2) Lack of Estimation Procedures/Aids Appropriate to the

Design Process - Even if training developers were

receiving accurate and timely information on early

system concepts, current deficiencies in training

estimation aids and procedures preclude the early

estimation of training resources. Current training

technologies are designed to be applied with the type

of detailed data and the types of analytical questions

which are relevant to later phases of the acquisition

2-20
process. These technologies cannot address the special 

requirements of the early phases such as the 

:i.deu:ification of comparable existing equipment, the 

generation of tasks for systems whose hardware has not 

yet been built, and the rapid estimation of training 

resources and costs and other evaluation criteria. 

(3) Lack of Systematic Technology for Rapidly Evaluating 

Training Alternatives Currently, there are no 

systematic procedures for rapidly assessing the 

training resources, cost, and efficiency/effectiveness 

of training alternatives. In addition, there are no 

techniques for quickly conducting trade-off and 

sensitivity analyses, and asking "what if" types of ... . ques ~lons. 

2.3 REVIEW PSYCHOLOGICAL ASPECTS OF DESIGN PROCESS 

During this subtask, psychological research relating to 

engineering design was reviewed. The goal of the review was 

':o identify gener-al guidelines for SOT software development. 

The revie·. focused on the individual cognitive processes 

involved i~ system design and attempted to identify features 

that should be include<:! in the SDT to facilitate the system 

Jevelopmen~ process. 

The ·eview began with the search for a ge11eral cognitive 

mode'. which could be used to describe the design process. 

Ramsey and Atwood's (1979} scheme for describing the problem 

solving prc-·cess was determined to be the most suitable 

frame'·ork. Using the framework provided by this scheme, 

prob~em solving aids relevant to each stage of the problem 

2-21 

. 
solving process were identified through a review of the

problem solving literature. The implications that each of

these limitations had for the SDT were then assessed and

documented. A summary of this analysis is provided in Table

2-6. While it was possible to incorporate some of the

features of the problem solving aids into ETES automated

tools, in many cases it was found that the problem solving

aids were not relevant to the types of procedures which were

eventually included in ETES.

2-22

A .. .. . . . . 5-' - ..5 *
It

Table 2-8 PHOBLEM-SOLVING AIDS RELATED TO ETES*

Adingcpel Implications

Mechanism Oeariton Comments Refereencies for SOT

Alternative Thse adls may eithe" automata Except for aids that automata the Brown at al 119751 SOT must be capable

Evaluation the ow's evaluation clrera, require user's ealuation criteria, these task Hormann (117f of feading into these

the user to use established criteria. aids are cask-specific. Moot useful if Rapp (19721 altnertma evaluation

or simulate the results of actions the task is not well-defined or if a Smith, KI1 

T and aids.

that cop not have well established large number of evaluation criteria Crabtree (19751

evaluation criteria, need be considered.

Alternative Theg aids we primrily used to Except for well-defined task domains, Baldwin & Siklossy SOT must sdacibe

Generation generate alternatives that the user whe they may hae very little (197711 desgn options, HC eres.

would not normally consider or. impact, they c,.a difficult to Gagliardi at at SOT will not be directly

for extremaly well-defined tasks, construct. Can be cost-effective for (19661 irolved in generating

to preent algorithmscolly training applications. but generelly design options.

determined altarnatilnl. are of limited use in complex

problem-salving tasks.

Automatic Action Such aids Permit the user to same Most useful when the results of Carlson & Hodgion ETES application ,"-

Execution the delsired action without applying an anon do not impact 11977) pograms mast allow

explicitly carrying out the steps subsequent problem-solving actions. Hanes & Gebhard for estovetic action

involied in its exectution. If this is the cas. the user, may (1966) executions whenever

seed sophisticated alternatives Puifer (1972) maSlam.

envalution heuristics.

Automatic This Type of aid functions us an Although demonstrated to be Freedy at at (1972 SOT must se capable

Takeover automated decision maker that is effective in some, coneetat (e.g., Stels & Fredy of gererroting autometic

able to select altanetiva actions on control tasksl. the range of taks 11976) 'help" qu ies when

the bsis of prior observations of the in which this is approPriaell is ot user mke m aj"or errors.

human decision maker's behacim. well understood. User acceptance

Although allocation of control to may he low and should be

this aid occurs automatically, when- carefully examied.

ever omae criterton of correspondence between predicted and observed

human behavior is reached, voluntary

turnover of control is ales possible.

Backtracking Such an aid allows the problem Useful in tasks wheare it is Carlson & Hodgson SOT mest have capability

suiar to "undo" the effecta of possible to "undo" re t actions. (1977) of allowng user to return

recent actions and return to an Can Improve performance at Midyte ar al to provious framas.

earlier state of the problein'olsng relatively little develonment cost. (19681

procems without actually sattring Trritlman (19721

overir

Btter Weighting This aid re-code lIow-fidelity data Depenqi on the ability to Toomiller (1968) If possible, SOT must

of Unreliable into a form that is more redily accurately reaooe low-fidelity data. Howell & Gerttys cearly distingsuish betweeni

Data uable by the problem solver. 11966) animated and acoal- dataChinge ol Typical implemntations of this Most sinful in well-understood Chesler & Turn Not applicable to ETES.

Problem aid present problems us isomorpdhic tasks An Inappropriate (19671

Reoriesntatios iariations of more standard Problem representation may seriously Smith, H. T. (1974)

roresenttalons. It is intended that degrede performance. Newsycd & Wynne

this will aid the problem soler in (1976)

siectling an appropriate and

efficent problem formulation

Orcivion This type of aid assists the usrs Usefsl for .sPart problem solvers Davis et at (19751 Not aplicable to ETES

nitency applying their own decision in ,vell-dlfined tasks Including FreedV et at (1976)

icrseement lirategies consistently ,n cases in sufficwnt versatility to adapt to

which those strategies are rndlovdual userv may be difficult.

Decision Strategy Such aids assist the user -n Useful in well-definled tasks ,n Cruso 19701 Not apoliable to ETES.

iwprovrnnt applying proesolvoising which optimal, or neer optimal Gagliardi at at

tIeceues htha womld sot problem-solvingf techniques are 1,no-.r. 19651 1194

normally he considerded or oi s tasks in which general Rogers vi at 1964)

kmown heuristcs. such as problem ole (19691)

reduction. ar applicable. Requires

detailed knovdedge of the task

Dncompositton In his type of urd ltows hi user io iJseful only 4f a task can be Krotk 19711 ETES muest be cmoosed

Rronpoi1riyn div ?he rigirnal eroblem into tub decomposed into ndepandent a

t 

a eat of distinct aids

iroblews The solutions of the suborobems. Requires a food and procodurs.

icrious uborsbires rn then jndrrsta ln ,of d he task

oumbiset nt Sf ou t~ion is the

tr tfqnal larger problem

0,stuptOin s )f Such an id Is intended to disrupt Potentially usetol out iaSt.ewart -19761 hppo i,,Oable to ETES

Pv.hoioqicalt et ,y titus q, its i'hat tie Quset disrupt an apprtroiae met

may mpiov aid thereby itimulati!

cliinq a"tpts

Table l-eril vt Ramsey, ind Atwdod

2-23

Il

• ." •, .is. ., .. ' -. :- -i ,.- -. .. ... .. .-" .. .. -.. ' '. ,. . . . . ., . . , - .
Table 2-6 (contintw)

Aiding Principal Implirar ios

Mechanism Description Corment. Reference for SOT

Eo-ktus Meto e This aid allos the uses to str Very useul in lmost ai risks Bler & Shisev SOT, ba- a data and retrieve problim-retlaant Success sl related to the eam of (1968) be. meanagement information. This information nay reirienal from t ernory Nrewied & Wynn tysrem. must hae initially be generated by the .- a (19761 etintsiss attended

or by other problem-soltug aid. Smith, H T & memory capabilitines such s aide for afteneatie Craberee (19751

generation and elUation

Lockout In art interactitve problemsOlving Although domonstrated affect tea fooh. t. May he necessary in situation, this technique tasfricte in some onlexts user acceptance (19711 SOT in cans where two

the Problem solvers access to the was low The tradeoff between San it a1 or more users attempt to compute fo sona partid of tima US- acceptance should be (1971: access end modify the

lter the presertatlOn of the caefully contidred sne dote simultaneousy

rsults from the current request

for information

Rapid This aid allows the user to raidly Easily implemanted in melt defined Balt.i & Shirey Whenever possible. Trialand Error aend easily aoamn the consequences tasks. May offset inadequacies in (1968) mnsitvity analysis

of altenatve iacion by simulating decision strategy improvemant aids. Carlson & Hodgson procedures should be

their application 11977) included in SDT

Rapp (19721

Wilde (19691

Table derived from Ramsey and Atwood

'i.

z-24

.- . . . .. . . .-
a-V 11 7 .

SECTION 3 - TASK 2: DEVELOP METHOD FOR

SYSTEM CONCEPT DESCRIPTION

During this task, five major activities were accomplished.

First, as recommended in the statement of work, automated

soft" are requirements analysis tools were reviewed to

determine if any of these tools could be used to develop and

document early system descriptions. When this review proved

unfruitful, automated data base management tools were then

examined. This analysis did prove to be fruitful as

evidenced in the current SDT which is a microcomputer-based

data base management system.

Third, recent research in the area of human computer

interaction was examined to identify general guidelines

which could be used in the development of the SDT.

Fourth, the SDT was developed in accordance with the general

requirements for early training estimation identified in

Task 1 and the human-computer guidelines developed in the

third subtask.

Fifth, the SDT was applied to the Single Channel Ground

Activated Radar System (SINCGARS). More details on each of

these five activities are presented in the sections which

follow.

3-1

• , . . ..
3.1 REVIEW REQUIREMENTS ANALYSIS TOOLS

The review of requirements analysis tools was conducted in a

three-stage process by DRC's software engineering group.

During the first stage, DRC surveyed government reports,

IEEE Software Engineering Transactions, and other trade

publications to determine what tools were available in the

area of requirements analysis. Fortunately, a comprehensive

review of requirements analysis tools had just been

ccmpleted by Devorkin and Obenodorf (1979). Further

investigation indicated that this report contained all

requirements analysis tools with sufficient maturity for

possible use in the SDT.

During the second phase of the review, the methodologies

listed in Devorkin and Obendorf were reviewed in more

detail. Each review began with an examination of the

available literature on the methodology. Following the

literature review, individual users were interviewed by

phone. With the aid of user cornments and knowledge of the

SDT requirements, criteria were developed for identifying

methodologies with a nigh degree of potential application to

the SDT. The evaluation criteria were as follows:

I. Applicability. The methodology must be capable of

buildinq a data base of the conceptual information

normally available during the early phases of a

developing or evolving system. This data base must be

capable of refinement as more specific system

information becomes available, It must 5e capable of

describing requirements, design concepts, human tasks,

and training program elements.

3-2
I-.

2. Understandabilityt. The methodology must be capable of

being understood by the types of "personnel" who are

likely to use the SDT.

3. Demons tratabili ty. The methodology must have been

applied to a number of different types of projects.

4. Transportability. The methodology must be capable of

being implemented at a minimum of cost on standard

business processors used in military/ government

agencies.

5. Training. The methodology must have an existing formal

training program available to potential users.

6. Sponsorsiiip. The methodology must have a specific

government agency, university or industry committed to

enhancing the methodology to meet additional user needs

as they become known. The methodology must reside in

the public domain.

While investigating the first few methodologies, it became

evident that there were two main thrusts in the area of

requirements definition methodologies. One thrust

emphasized graphics representation, primarily through

functional flow block diagrams, as a means of specifying

relationships between system elements. Another thrust

emphasized a high level conceptual language as the mechanism

for specifying relationships between these system elements.

Because there was good deal of overlap between the tools

within each of these two thrusts, particularly among the

language-based tools which are all basically more advanced

3-3

. . .
t_77.7

derivatives of earlier work conducted by the ISDOS project

at the University of Michigan, it was decided that the tools

listed in Devorken and Obendorf would be evaluated in terms

of the six criteria listed above, and that the tool in each

of the two major thrust areas with the highest evaluations

on these criteria would be selected for further analysis in

the third stage of the review.

Table 3-1 displays the requirements analysis tools which

were evaluated during this stage and summarizes their

assessment.

The two tools selected for further analysis were the ICAM

Definition Language or IDEF, which was determined to be the

best graphics-based tool, and the Problem Statement

Language/Problem Statement Analyzer, which was selected as

the best language-based tool.

During the third stage of the review, these two tools were

examined in even greater detail. As this review was being

conducted, it became apparent that the requiranents analysis

could not provide the vehicle needed for the SDT. There

were two reasons for this.

First, the requirements tools did not appear to be suited

for the types of uninitiated users who would utilize the

SLYI'. This was not surprising when one considers that all of

these tools were specifically designed to describe software

requirements for large complex systems. Hence, they are

ces irne,, to he utilized by technical personnel who have

fi1rLy sophisticated backgrounds in software deve opnent.

The oo Is were des i jed by so tware p ec 1a list s to r

iof twa r sLecialIsts) .

-I
-w0 0 0 0 0 o, 0 0 a 0 0 0 0 0

0~0

W- - C,- Z

0 0 . V ~ ~ ~ ~ ~ C (A - ' - 224 : , 0

LL' c_ , >2 ' '

CL 0 , * , 04

U ,,0 0 CL

-U U -J > 4,w

0U a' C '

w w0 V I C ~ V 4

0C .0 0

~j 0 cc0 0 >0 - -

-L 4, - -<. C- 
At a slightly more conceptual level, another factor

contributing to the complexity of the requirements analysis

tools is that they are designed to be extremely flexible

tools which can be utilized to describe any type of

system. This type of flexibility necessitates a certain

degree of abstractness. This high degree of flexibility and

its associated abstractness may actually be a hindrance in

describing data for developing weapons systems since such

data tends to be similar across systems.

Second, while requirements analysis tools deal with an

important aspect of early training estimation (i.e., system

description), they are not really geared for dealing with

other important ETES related problems; namely, the update

and refinement of these system descriptions and their

communication to a wide range of participants in the

acquisition process

3.2 REVIEW DATA BASE TOOLS

4hen the investigation of requirements analysis tools proved

o be unfruitful, the focus turned to automated data base

anaglcbement systems (DBMSs), as a mechanism for storing and

lescrting developing system descriptions. Generally, DBMSs

fulfill the SDT evaluation criteria of applicability,

understandability, demonstratability, transportabili ty,

tr1ning, and sponsorship that were identified in Section

3. 1. En addition, most DBMS have the following advantaqes

t,r the :4DT:

.. .. .. . . .
: ,-21

1. The DBMSs are designed to store many data items

that are related to one another The SDT consists

of many data items with complex interrelationships. Therefore, DBMS technology facilitates the

development of the SDT.

2. DBMSs allow data to be centrally located and

controlled. This simplifies data sharing among

multiple users.

3. They can be fitted with data access aids that are

easy to use. These aids allow a user to input,

modify, delete, and output data using "userfriendly" techniques such as English-like phrases _

and commands.

4. Access to data items can be restricted.

unauthorized users cannot view, modify, delete, or

output restricted data items.

5. The format of a data item is independent of the

computer program that is accessing it. This is

significant if future software systems--other than

the SDT--wish to access the data in the SDT data

base.

3.2.1 Overview of DBMS Concepts

Biefore examining the results of the review of DBMSs, it may

he helpful to review definitions an,! teatures ty of the

DBMS concept.

-.... I
i

 : _ 

: v . W : v W 

U y 11 w q . p .,

 . . . \ ., . ; WT7 VV7

An automated data base management system may be defined as a

computerized and integrated collection of stored operational

data used by the applications groups of a particular

enterprisel. The key word in this definition is

"integrated." The data elements of a system are likely to

have relationships or associations which one could use to

link these elements to one another. A data base is

integrated when it incorporates information on these

relationships as well as information on the data elements

themselves. This information can be used to store and

retrieve data. It should be noted that, strictly speaking,

a data base need not be resident in a computer or its

associated media. However, all automated data bases will be

stored on a computer or related media and all modern DBMSs

are automated. It is clear that only an automated data base

can meet the storage and retrieval requirements of the

SDT. The term "operational data" is used to refer to data

which is pertinent to the ongoing activities of an

enterprise. Operational data excludes input data, work

queues, output data (such as messages or reports) or any

other form of temporary information.

o Advantages of a Data Base Management System

The major advantage of a data base management system (DBMS)

is that it provides the enterprise with integrated,

centralized control of its operational data. This

centralized control can, in turn, (1) reduce redundancy in

stored data, (2) avoid inconsistency in stored data, (3)

1This definition is a modifirti - t oi o -t an xit nq

definition by Engels (1971).

• . '- - .- / .- -, . , - - " ... 7- -. . 7 ..,. . .. . . - .- , . .. <. .--. . - <- . . ..-
allow for greater sharing of data, (4) permit standards to

be enforced, (5) permit security restrictions to be applied,

(6) permit a greater capability for checking and maintaining

data, and (7) provide "data independence." Data

independence is achieved by maintaining an internal

structure of the data which is independent of the individual

applications of the data and individual user viewpoints.

This data independence may be contrasted with data dependent

systems in which the way data is stored and the way it is

accessed are dictated by the structure of the applications.

o Data Base Management System Features

Perhaps the best way to describe the essential features of a

DBMS is to outline an "architecture" for a typical DBMS.

Such an architecture is displayed in Figure 3-1. This

architecture is taken directly from Date (1977). DBMS

architectures are typically divided into three general

levels: internal, conceptual, and external. The internal

level is concerned with the way in which the data is

actually stored physically. The external level reflects the

users' view of the data. The conceptual level provides the

medium for linking the internal and external views. The

conceptual model provides a genera, community view of the

data base since it contains an abstract representation of

the entire data base. This community view is to be

contrasted with the external views of individual users who

typically will only have a view of a portion of the data

base.

Perhaps another way to descrioe these three different levels

of a DBMS is to refer to the structures that psycholinguistics uses to describe human language. The external

3-9
a 

7

.1a

'p6
-0-~~ 7 ~ TI ~ ~ ;;,. ~ T- -%7 77 17 'O .7 . qF q

- -.

view of a DBMS can be construed as being roughly analogous

to what psycholinguistics describe as the "surface

structure" of language, while the conceptual level can be

construed as being analogous to the "deep structure" of

language and the internal structure can be construed as

being roughly analogous to the physical structures in the

brain for representing speech.

It is possible for each external user to have his own

"language" for utilizing the data base, although in many

cases, all or a large number of users can use the same

language.

The conceptual model is defined by a conceptual schema which

includes a definition of each of the various types of

conceptual information in terms of content only (storage or

access features are not described). Thus, the conceptual

model provides the definition of the total data base

content. The conceptual model is critical in that all other

aspects of the DBMS are affected by the conceptual model.

It has a major effect on the format and structure of the

data sublanguage which is used to store, update, and

retrieve information from the data base.

3.2.2 Process and Results of Reviewing Data Base Management

Techniques

Review of data base management systems for the SDT occurred

in a four step process. First, existing mainframe"

microcomputer data base management systems were reviewed.

This review did not identify a data base which could meet

ETES requirements. Second, existing micro-computer based

•-I

,'.1

.°
data base management systems were reviewed. While no offthe-shelf microcomputer based data base management system

was identified which could meet the ETES requirements, a

concept was developed for developing a tailored microcomputer-based data base management system which could

specifically meet ETES requirements. Third, hardware/

software was identified for implementing this concept.

Fourth, concurrent with the review of mainframe databases

existing military data bases containing training related

information was reviewed to assess their applicability to

the SDT. None of these existing data bases met ETES

requirements.

3.2.2.1 Review of Mainframe/Microcomputer Based DBMSs

Fifty-one (51) commercially-available mainframe/minicomputer based DBMSs were surveyed through a study of DATA

PRO reports (70E-01B-61a and D30-100-002) and other

literature. Table 3-2 summarizes the characteristics of the

51 DBMSs that were surveyed. Each data base is described in

terms of six basic characteristics: Vendor of the DBMS,

Supporting Hardware, Approximate Usage, Primary Data

Organization, Approximate Price, and Applicability to SDT.

"Vendor of the DBMS" refers to the company that distributes

the DBMS. "Supporting Hardware" refers to the computer

hardware configuration on which the DBMS will operate.

"Approximate Usage" is an estimate of the number of

installations of the DBMS. It is based on the following

3-12

-- -

- ., - - .-. .** *.*- -
Table 3-2 Characteristics of Commerciaiiy Available

Main-Frame/Mini-Computer Based DBMSs.

Primary

*Vendor of Supporting Approximate Data Approx. Applicability to

DBMS the DBMS Hardware Usage Organization Price SOT

-ADABAS Software AG of North IBM: 360,370, Moderate Network $2500/ Very high

America 303 x, 4300 month

S40-1 60 K

ADMINS/Il Admins, Inc. DEC: POP-il. Very low Relational N/A Low

VAX-l I

AMBASE Amcour Computer DEC: POP-li N/A** N/A S16.5K Moderate

Company

*BASIS Battelle. Columbus IBM: CDC; Cybr:r Very low Relational-like $38K Moderate

Laboratories Univac: DEC

*CREATE Complete Computer Data General: Very low N/A $18K Moderate

systems AiOVA & Eclipse

*DATACOM/OB Applied Oata Research, IBM:360,370 Low Relational S47-57K Very high

Inc.

DATA DEMON Gemini Information Perkin-Elmer; Very .3w N/A $1000/ Low

Systems. Inc, IBM: Series/I month

S17.5K

DOW- Condor Computer Corp. Cromenco: Very low N/A S10K Low

System/3

*DBMS Prime Computer, Inc. Prime: 400 & 500 N/A Network S20K Low

DBMS 2 EGS Systems. Inc. Modular Computer: Very low N/A N/A Very low

MOO COMP

DBMS-10 DEC DEC; Systemn-iD Low Network S30K High

DBMS-il1 DEC DEC: POP-li N/A Network $16.5K Low

*DBMS-20 DEC DEC: System-20 very low Network S30K High

OSDMS-300 Compudata Systerm Inc. DEC: 300 Series Very low N/A SiOD/mo Very low

S51K

DBMS-990 Texas Instruments, Inc. TI: 05990, N/A N/A S21K Vary low

Models 6 & 8

*DL/I DOS/VS IBM IBM! 370, High Hierarchical S4-34/mo High

303x, 4300

DMS I1 Burroughs Corp. Burroughs; B Low Network S23.25K High

700 or 800

DM5 90 Sperry Univac Univac: Series Vary low Network N/A Moderate

80 or 3

*DMS-170 Control Data Corp. COC:6000O, Cyber Very low Network S730/mo High

70, 170, 700

DMS-1 100 Sperry Univac Univac: 1100 Moderate Network N/A High

Series

DMS/1700 Dedicated Systems. Inc. Burroughs; 61700 N/A N/A S51K Low

DNA-Data Exact Systems & DG: Nova or Very low N/A N/A Very low

Base Manager Programming Corp. Eclipse

*DBMS selected for further study.

**Not available.

-i3
Table 3-2 (continued)

Primary

Vendor of Supporting Approximate Data Approx. Applicability to

DBMS the DBMS Hardware Usage Organization Price SDT

4DPI National Information DEC: System Low Hierarchical $966/mo High

Systems, Inc. 10 or 20 S38-47K

DRS/XBS A.R.A.P. IBM, DEC. Univac, Low Network S22-60K High

CDC

EASE DBMS Bloodstodc Computer DEC. PDP-li Very low N/A S8.5K Low

Services, Inc.

GIS/2 IBM IBM: 380 or N/A Hierarchical S520-970/ Moderate

370"t

18DB Tesseract Corp. IBM: Series/i Very low N/A S4.1lK Low

**IDS-I/Il Honeywell Info. Honeywell: Series High Network SAOO-S2K/ Very high

*(DM.-IV) Systems, Inc. 6000 & 60 Level 66 ma

* IDMS Cullinane Corp. IBM: 380,370, Moderate Network S50K/yr Very high

303x, & 4300

IDOL Science Management Wang: 2200; Low N/A $364/mno Moderate

Corporation IBM: Series 1 $9500

* MS IBM IBM: 360,370, Very high Hierarchical S1045/ High

303x, 4300 mo

Inrfof lax DBM Interactive Info. DEC: Datasystem Very low N/A S12K Low

Systems, Inc. 500 Series

INFOMEDIA Mead Technology IBM: 360 or 370 N/A N/A $20001 Low

Laboratories MID

Si140K

*INFOTRIEVE Educational Data EDS: Point 4; Very low N/A $2000 Moderate

Systems DG: Nova

*INGRES INGRES, Inc. DEC: POP-1l Low Relational N/A Low

10/NET Infodata Systems, Inc. IBM: 4300 N/A Network S40K Moderate

INQUIRE Infodata Systems. Inc. IBM: 360, 370 Low Network $70-150K High

MADMAN G.E. Company DEC- PDP-li N/A Relational-like $20K Low

*MIDMS National Technical IBM: 360 N/A N/A $450 Moderate

Info, Service

MINDS Minnesota Datasystemrs. BTI: 4000, Very low N/A S3-5K Low

Inc. 5000, or 8000

'MODEL 204 Computer Corp. of IBM- 360, 370, Very low Network S90-150K Very high

America 303x, or 4300

OASIS University of Windsor IBM: 360, 370. Very low N/A S30K/yr Low

or 303x

ORACLE Relational Software. Inc. DEC: POP1i or N/A Relational S48-96K 'A, !Prate

VAX

*OS 200 D8 Honeywell Honeywell- 200 Very low N/A Bundled Very ow

or 2000 free with

Hardware

PLUS/4 Century Analysis, Inc. NCR. 101 or N/A N/A 510< Low

above

OCRT The Management Group. IBM- 360 or 370; Very low N/A S25K Moderate

Inc- Honeywell: 6000
O Very high -greater than 1,500 installations,

o High - 1000 to 1,499 installations,

o Moderate - 500 to 999 installations,

o Low - 100 to 499 installations, and

o Very low - less than 100 installations.

It infers a rough measure of the popularity of a DBMS within

the computer-used community. However, this inference does

not imply that a greatly used system is better than one of

less usage.

"Primary Data Organization" is the logical structure of the

data base at the conceptual level. The structure can be

relational, network, hierarchical, or a combination of these

three. However, only the most commonly referenced structure

is listed.

"Price" refers to the basic system purchase price that was

quoted to DATA PRO in late 1980. The purchase price

generally includes an unspecified monthly maintenance

charge. Monthly or year- ly lease/rental plans are

occasionally included.

T-e column entitled "Applicability to SDT" is a composite of

the 12 SDT Requirements that were identified for a mainframe

data base management system. Each DBMS was examined to

determine how many )t these 12 requiremnents it fulfilled.

The scale ujsed was -as follows:

L.

- . . ( . ... . -
o Very high -All 12 requirements fulfilled,

o High - 9 to 11 requirements fulfilled,

o Moderate - 5 to 8 requirements fulfilled,

o Low - 3 to 4 requirements fulfilled, a nd

o Very low - 2 or fewer requirements fulfilled.

The applicability factor is the deciding factor for

selecting DBMS candidates for the SDT. Only those DBMSs

that ranked very high (fulfilled all 12 SDT requirements)

were selected for further analysis. These DBMSs are marked

with an asterisk (*). The seven DBMSs that were selected

for further analysis were ADABAS, DATACOM/DB, IDS-I/II (DMIV), IDMS, MODEL 204, RAMIS II and SEED. Each of these DBMSs

fulfilled the 12 SDT requirements.

At this point, an attempt was made to identify which

mainframe of microcomputer DBMS would be available to a wide

range of ETES users -- the intent being to select the data

base from the seven finalists which would run on most of the

hardware owned by ETES users. This attempt to identify a

common mainframe or minicomputer for ETES users was not

fruitful -- a variety of ditferent machines were used across

user sites. Many users did not have mainframe or

minicomputer nor did they have access to one. Furthermore,

since the cost of obtaining these machines was high, it

appeared unlikely that they could obtain one in the future.
., 

·, 

1 

·I 

3.2.2.2 Review of Microcomputer Based Data Base Management 

Systems 

As the limitations of the mainframe or mini based DBMSs 

became evident, alternative configurations for the SDT data 

base were examined. ,_one configuration which was identified 

as having considerable merit was the distributed processing 

configuration. In the distributed processing configuration, 

a centralized data base for each weapon system (or weapon 

system alternative) would be stored on a mainframe 

computer. At periodic intervals, users would transfer a 

copy of the data base from the mainframe to a local 

microcomputer. Once on the micro, users would perform 

standard data base management functions (input, output, 

modify). Thus, all major data base management functions 

could be performed. independently on the microcomputer. Once 

users haw~ completed their activities with the data base, 

they ~ould transfer the updated version to the mainframe. A 

detai!ed audit trail would be kept for each weapon system so 

that use~s can systematically track and assess system 

changes. The distributed architecture has several important 

advan~:ages over other configurations. Specifically, it (1) 

minimizes 

is only 

minjmizes 

computer 

required 

on-line 

::::-esource requirements (the average user 

to purchase the microcomputer), (2) 

computer charges, ( 3) allows users the 

f lexi b i 1 i ty of conducting their own independent analyses, 

and (4) provides capabilities for the maintenance of a 

centralized data base, thus maintaining data integrity. 

After discussions with ~he COTR, it was decided that the SOT 

would be designed to ~eat the requirements_for a distributed 

Jrocessing architecture and the search began for a 

3-17 
[ .

b'

4'4

microcomputer-based data base management system which would

support this configuration.

Several of the most popular and widely used microcomputer

based data base management systems (for example, dBASE II,

MDBS III) were examined to determine if any of these data

bases could meet the ETES requirements identified in Section

2 and 3.1. This review indicated there were two major

problems with existing off-the-shelf microcomputer based

data base management systems which limited their

applicability to ETES. First, all of the micro-computer

based data base management systems used a query language to

access the data and obtain output. It was felt that these

query languages, while considerably simpler than high-level

languages, were too complicated for the potential ETES user

to learn and/or use. Second, it was felt that these

generalized data bases would not take full advantage of the

common data elements which are used over and over again in

the development of each weapon system (One of the major

objectives of ETES was to identify a common set of data

elements which could be used across systems). In this

sense, the flexibility of these data bases, which would be

an advantage in most applications, was considered to be a

disadvantage for the ETES application. Recognizing these

limitations, it was decided to build a specialized

microcomputer-based data base management system which was

(1) tailored to meet the specific needs of early task

training estimation for Army weapon system and (2) used more

"user-friendly" human computer dialogue techniques such as

menu selection to access and output data.

3-18

! ,. , " I - J " ' • I M- " - .. .
3.2.2.3 Selection of SDT Hardware/Software

Once it was determined that a tailored microcomputer-based

management system would be used for the SDT, an

investigation was conducted to identify the microcomputer

and computer languages which would be used for SDT

development. As part of this investigation, an attempt was

made to identify the brand of microcamputer that was most

popular among ETES users in the Army. 2 This investigation

indicated that the Apple computers were most popular among

potential ETES users. For example, Apple computers were

being used by ARI and by several of the TRADOC schools.

Once it was decided that an Apple was to be used, the

specific hardware configuration that was to be used to

develop the SDT was identified. This configuration is

listed in Table 3-3. The Apple III, rather than the Apple

II, was selected as the microcomputer to be employed in the

SDT because the Apple III had the capability to be used with

a external hard disk system. Such a system could provide

the memory needed for the SDT data base management

functions 3

To provide a capability for testing the distributed

processing capability of the SDT, it was decided to use the

contractors (DRC's) mainframe canputer (the Honeywell DPS

2 A more formal attempt was made to identify any Army

wide plans to purchase a single brand of microcomputer. This investigation did not produce any fruitful results.

Since the SDT was developed, Apple has developed a

version of the Apple II, the Apple IIE which can be used

with the external hard disk.

3-19

-[ . - ' "] 

"'2 ". . ,""" "''.'' " ." . .,"•;• '' " , . , -.. -..." ". . .. -"-" ....-- . ." "-, . ."- . -" -".'-'.." '.€.' . .' ' -
Table 3-3 SDT Hardware Configuration.

MICROCOMPUJTER MA IN FRAME

CONFIGURATION_______

APPLE III DPS 8/52

MODEM

PRINTER

MON ITOR

FLOPPY DISK DRIVES

PROFILE MASS STORAGE (HARD DISK)

OR

<ADDITIONAL FLOPPY DISK DRIVES

3-20
040

8/32) since it provided a low cost mechanism for

demonstrating this capability. Because of resource

limitations, only a very limited distributed processing

capability was to be demonstrated during the ETES

development contract. It was decided that the full

distributed processing capability would either be achieved

in the Man Integrated System (MIST) contract, which is

currently integrating ETES with several other ARI MPT

projects, or in the ETES implementation contract which was

to follow the ETES development contract.

After the hardware conf iguration was identified, the

computer language to be used for the SDT software was

identified. PASCAL was selected as the language for

developing the SDT and other ETES automated tools because of

(1) its widespread usage on both microcomputers and

mainframes, and (2) its highly structured nature which

facilitates modular program cons truct ion and

transportability.

3.2.2.4 Past Efforts In Developing System-Specific Data

Bases

Concurrent with the examination of data base management

technologies, past efforts in developing system-specific

human resource data bases were examined. Table 3-4 lists

the major data bases that were examined during this

process. More details on each of these data bases are

provided in the section which follow.

3.2
n'1%7p 7r.1- FP W.. -wn' r lrJ F -I! w 'r'P TFd Pl F jw-pww Nv~yd . - -- W-~. JTable 3-4

PAST EFFORTS AT HUMAN RESOURCE DATA BASE DEVELOPMENT*

(1) Logistics Support Analysis Record (LSAR)

(2) Unified Data Base of Air Force Human Resource Lab

(3) Consolidated Data Base (CDB) of Navy/Army HARDMAN Projects

(4) Structured Approach to Training (SAT) Program for the BI-Bomber

(5) Navy Enlisted Professional Information Support System (NEPDISS)

*Efforts are listed in terms of their decreasing relevance to the ETES SDT.

Fe31

* 3-22

.... .*..*.**J'.~.*- ... ..-..... .. *- .-. . - . - . - - -

. . . . . . . . . .. . . . . . . . . . . . . . . . . . . . .
o Logistics Support Analysis Record

One major effort which is closely related to the objectives

and goals of the SDT is the Logistics Support Analysis

Recore (LSAR). MIL-STD-1388 states that the goal of the

LSAR is to be the "single source of validated, integrated,

design-related logistic data pertinent to the acquisition

program."

Table 3-5 lists the system elements that are described by

the LSAR and the major weaknesses of the current LSAR in

respect to the goals and objectives of the SDT. As Table 3-

4 indicates, the LSAR has several weaknesses which limit its

use as a canprehensive system description technology for

human resource assessment.

First, there are several important system elements (e.g.,

system functional requirements, collective tasks) which the

LSAR does not describe. Failure to describe the system

functional requirements is particularly distressing, since

these functional requirements provide the foundation on

which all other system elements depend. Lack of a

systematic description of functional requirements makes it

extremely difficult for training developers and others who

are tasked with relating their particular system elements to

overall mission performance and its associated functions.

For instance, it makes it extremely difficult to relate

human tasks to mission performance. Given its lack of a

capability for describing system functional requirements or

projected system elements, it is not surprising that the

LSAR is currently not applied during the concept exploration

phase of the acquisition process and seldom, if ever, 5-

applied during the validation and demonstration phase.

3-2

c •

-. "3-23

'I

% •. -, - . -,• - o . .. . . ..... . .. . . 5 . V
IW

T - - . -TT.-1vr P- M 7V T. F-I .P . - -

Table 3-5

OVERVIEW OF LSAR AND ITS MAJOR WEAKNESSES

System Elements Described by LSAR

* Equipment (work breakdown structure, work unit code, nomenclature,

reliability, maintainability, failure symptoms, failure effect and criticality,

maintenance concept)

* Tasks (task code, frequency, elapsed time, skill specialty, man hours,

requirements for training equipment, support equipment, tools, task elements,

aggregate maintenance man-hour requirements)

" Support and Test Equipment (physical characteristics, associated equipment,

associated tasks, associated training, special skill requirements)

* Facilities (associated equipment and tasks, general requirements, lead

times, type of construction, utilities, facility unit cost)

* Skills (associated task and equipments, specialty codes, aptitude, rank/rate,

special physical and mental requirements, educational requirements,

additional training requirements)

* Supply Support (part no. and nomenclature, physical description, associated

equipment, allowance quantity, distribution)

Major Weaknesses of LSAR*

" Does not describe system functional requirements

* Does not provide adequate description of operator tasks

* Does not describe task characteristics or performance information

• Does not describe collective tasks

" Does not adequately describe skills

" Does not adequately describe training program elements

* Does not provide mechanism for describing estimated or projected elements

* Is not applied in early phases

* Does not have data base management capability

* Cannot generate tasks or other input data

*Manv of these limitations are apparently being dealt within the present LSAR

improvement programs.

3-24
_,5

Hence, its value as a data base to support early human

resource assessment is very minimal indeed.

Second, there are a number of other systems elements which

are described by the LSAR but are not described adequately

or in enough detail (e.g., operator tasks, task

characteristics, training program elements skills). The

emphasis of the LSAR on maintenance and maintenance tasks is

quite obvious. This emphasis makes it extremely difficult

to develop or maintain adequate descriptions of operator

tasks. For all types of tasks, the LSAR does not fully

descrioe the task characteristics and performance

information that is needed by training and/or human factors

specialists to adequately assess their components of the

system. The training portion of the LSAR places an emphasis

on training equipment and devices and ignores other

important aspects of the training program (e.g., course

modules).

Third, at a more conceptual level, the LSAR does not provide

an adequate capability for describing estimated or projected

system elements. Such estimates are necessary during the

early phases of the acquisition process.

Fourth, the LSAR was not conceived as an automated data base

management system description -- that is, as an automated

system for describing, updating, and expanding system

concepts and communicating this information to system

users. It should be noted that the Army, through the

DARCOM Materiel Readiness Support Activity, has been a

leader in "automating the LSAR." However, this automation

apparently refers only to the use of computerized algorithms

for aggregating certain LSAR elements or for presenting

o'3

-. 3-25

- • - " - - d I d I -l d - -+ d ,- _ _ ° " " +"_ " = u + = • Z " " ' "- " _
Id

printed outputs of reports. It is not designed to be an I interactive system. More important, the automated LSAR does

not provide for the automated description of system 

concepts, updates, changes and expansions through a

comprehensive data base management system. This is due to

the fact that the LSAR does not have a systematic internal

structure linking the various system elements to one

another.

o Air Force Human Resource Lab Unified Data Base

p..

The Air Force Human Resource Lab (AFHRL) has initiated a

program to develop a Unified Data Base (UDB). The goals of

the UDB are very similar to the SDT (see Thomas, Newhouse

and Hankins, 1980; Thomas and Hankins, 1980). Ultimately,

the UDB is designed to provide "a centrally located data

base of human resource related information for utilization

in the weapon system acquisition process to influence

hardware concepts and design." The UDB is to be supported

by a Data Generating Technology Data Base (DGTB) which is

intended "to generate generic data to fill in the needs of

users where the data systems, and likewise the UDB, would

leave voids." Thus, the DGTB is somewhat similar to the

ETES training estimation aids and procedures.

At the time the ETES review was conducted, UEB development

efforts had focused on (1) an assessment of existing

historical data bases which would feed the UDB, particularly

the projected portions of the UDB, (2) a description of the

weapon system design process with respect to the potential

use of the UDB, (3) an assessment of user needs in terms of

3-2.

~~...,.;,-.............. ,.-.. ....... •....-.... ......-... .... ...-........ ........ ,,.... ..
adequacy of current technology and data4, and (4) the 4

development of a plan for UDB/DGTB development.

At the time the ETES review was conducted, a description of

the actual data elements to be included in the UDB was not

available (this was to be developed in future phases of the

study). However, by examining the types of historical data

bases which are projected to be used by the UDB, it was

possible to make some estimates of what it would contain and

to assess some of its potential limitations.

These limitations point out the differences between the UDB

and the ETES SDT. These differences are actually quite

significant despite the similarity in the goals of these two

systems (see Table 3-6).

The first limitation of the UDB is its emphasis on

maintenance tasks and personnel. The UDB, like the Air

Force Coordinated Human Resources Technology, emphasizes

maintenance behavior and the use of historical data bases

related to maintenance. There is little relevant discussion

of the procedures and mechanisms for developing or

describing operator tasks or training requirements.

This emphasis on maintenance tasks is closely related to a

second limitation of the UDB; namely, its emphasis on

aircraft systems and on Air Force data bases. In the Air

4 In the examination of the utilization of human resource

data in tradeoffs, it is interesting to n-te that lack of

information and lack of appropriate analytical t-)ols were

seen as two of the major types of limitations on the use of

human resource assessment.

3-27

71l
Table 3- 6

LIMITATIONS OF THE UDB

Focuses almost exclusively on maintenance tasks.

Emphasizes aircraft systems

Does not appear to adequately describe functional requirements,

collective or team tasks, task characteristic or performance data,

and training program elements

;. Is not based upon comprehensive data base management syster

structure

Is geared for use by sophisticated users

Cannot generate tasks and other input data

3.

-" ~ ~3-28"l°
Force, the role of enlisted operators is much less

significant than it is in the Army or Navy. Hence, it is

not surprising that the UDB has focused on the maintenance

of aircraft systems.

Third, there are number of other system elements which the

UDB would appear, at least at the present time, not to

describe. These elements include functional requirements,

collective or team tasks, task characteristics, and

performance data suitable for training and human factors

analytical activities, and training program elements. (This

failure to describe certain elements would not be critical

if the UDB had the proper data base management structure to

handle additional system elements. Unfortunately, it appears

that it does not have this capability).

Fourth, and perhaps most important, the UDB again does not

appear to be based upon a data base structure--that is, a

structure which represents the implicit relationships among

the various system elements. Such a data base management

structure would provide a mechanism for describing the basic

structure of a developing system which was independent of

the various user viewpoints of the data. This data

independence would increase the capability for relating

various descriptions of the system to one another, for

updating and refining the data, and for adding new elements

to the data base in a systematic modular fashion with

minimum destruction of existing programming thus providing

the basis for a true data base management capability.

Fifth, the UDB appears to be geared for use by technical

personnel who have sophisticated analytical and/or computer

programming experience--unlike the SDT which is geared for

3-29

-. ...

• " " ' ;,= " b ' - ' { | - ' " ' ' "' " '' :' ' ' ' -- " " ' " -".
use by personnel with little background in computers.

Because of this difference in emphasis, it is not surprising

that the UDB does not specify or deal with the human factors

of human-computer interactions as with the SDT, which is

specifically geared for utilization by uninitiated users and

will attempt to employ the latest guidelines on humancomputer interfaces. Because of its lack of consideration

of human factors issues, the UDB does not attempt to provide

procedures for assisting the user in generating tasks or

other input data elements.

o Consolidated Data Base (CDB) of Hardman

Methodology

The Navy has a program, called the HARDMAN program (hardware

versus manpower procurement), to develop a methodology to

systematically assess the manpower, personnel, and training

requirements of emerging weapon systems, with particular

emphasis on developing predictions for the early phases of

the acquisition process. The HARDMAN methodology has been

applied to a number of different Navy systems and has been

modified for use by the Army and applied to several Army

weapons systems. (See Appendix D for a discussion of

HARDMAN).

The application of the HARDMAN methodology is supported by

the development of a system-specific "data base" which is

designed to contain all of the inputs and outputs of each of

the steps in the HARDMAN methodology and provide an audit

trail for monitoring the data elements which are developed.

Table 3-7 lists the data elements described by the CDB. Like

the other current human resource data bases, the CDB

3-30

• ." ," .
Table 3-7

DATA ELEMENTS CONTAINED IN CDB

General System

, Requirements Documents

* Study Plans and Objectives

Technology Base Studies

Projected Operational Environment

System Functions and Performance Requirements

Program Constraints

• Minimal Essential Elements of Information List

• Audit Trail Files

* Worksheets

C3DB Index

0 Predecessor Equipment List and Related Data

0 Reference Equipment List and Related Data

0 Predecessor and Reference Reliability Data

Manpower*

* Workload Taxonomy

• Indirect Workload Factors

* Task Event Networks

• Manpower Model Data

* Manpower Metrics and Associated Values

i System Manning (MOS, Skill Level, Duty Positions)

Training*

* Task and Skill Data

* Course Catalogue

• Course Outlines

o Course Methods/Media

0 Course Costing Data

- Course Scenario Information

. Career Path Information

a Training Concept

. Training Device and Equipment

* Steady State Resource Requirements

Steady State Ccurse Costs

• Replacement Personnel Requirements

0 Task Selection and Assignment Algorithms

0 Facilities Requirements

Personnel*

" Career Path Data

" Career Path Statistics (Attrition. Promotion. Upgrade)

*All elements for predecessor, reference. and baseline systems except where noted.

3-31
q.17

has several limitations with respect to the SDT

requirements.

The major limitation of the CDB is that only parts of it are

automated. Thus, it cannot provide a computerized data base

management capability. Another major limitation of the CDB

is that, like the UDB, it does not contain a systematic

scheme for relating the various system elements to one

another, a scheme which would be independent of specific

input and output requirements. Thus, the CDB is not really

a true data base management system since it does not have an

automated capability for linking various system elements to

another for retrieving data elements.

Finally, the CDB does not provide any extensive automated

capabilities for generating input data formats or actual

input data elements.

o SAT Program for the B-I Bomber

The Structural Approach to Training (SAT) program for the B1 bomber represents a relatively early attempt to develop a

system-specific data base to support instructional systems

development (see Sugarman, Johnson, and Ring, 1975).

The SAT consisted of two major elements, a data base (the

contents of which are displayed in Table 3-8) and two

computerized aids. One aid is a sorting model for the

storage, retrieval, collating, and updating of mission/

function task analyses and supporting data; the other is an

analytical model for providing cost and training estimates

of the B-I bomber training system.

3-32

.,-. "-2"'2.'.'*. . ",- ..-."-",: i', ,-".-" -"- ." "-', - " '" '" " '

° " " " " " 'e".7" - " "C. ""t '
Table 3-8

SAT DATA ELEMENTS AND LIMITATIONS

System Elements Descrited by SAT Data Base

Tasks (title, task element number, operator behavior, task duration,

crew interaction, previous task element, task characteristics, and

performance data)

* Control/display information (associated system, synonyms)

* Behavioral objectives (title, initial conditions, concurrent behaviors,

performance criteria, enabling and ancillary objectives, operators,

interactions, task elements, objective criticality, objective difficulty)

Limitations of SAT Data Base

• Is geared for one specific system

* Is not designed to provide generic data base management capability

Does not systematically describe system functional requirements and

design concepts -

Does not include training program elements in automated portion

of the data base

• Is geared for sophisticated users

Cannot generate tasks and other input data

3- 3 3

I'° .

P°3- 33

5°-
The SAT data base is interesting in that it is probably the

only past effort which has attempted to (1) systematically

describe task characteristics in a format which is amenable

to the application of automated training aids for

determining the tasks to be trained and selecting methods

and media, and (2) systematically describe the task

performance characteristics of equipment (e.g., relationships of tasks to controls and displays). Such task

performance data is critical to human task performance

simulation models.

However, despite its desirable features, the SAT data base

also has several limitations which restrict its

applicability to the SDT. First, the SAT data base elements

and programs were specifically designed to fit one system--

the B-1 banber. Thus, all of its task and control/display

dictionar-ies and structures are only applicable to that

system. The SAT was not designed to be a generic data base

system which could be applied across a wide range of weapon

sys temns.

Second, the SAT does not describe several important system

elements such as functional requirements and design/hardware

elements.

Third, training program elements are described but not

included in the automated data base.

Fourth, the SAT is geared for very sophisticated users with

extensive computer experience.

Fi fth, the SAT is not s tructur ed to assist users in

developing input data formats or actual input data elements

such as tasks.

3-

dt

* 3-34
o Navy Enlisted Professional Development Information

Support System (NEPDISS)

The objectives of the NEPDISS are more limited than the

goals of the cther human resource data bases described

above. The NEPDISS is specifically designed to store and

retrieve data related to training program development (see .

Davis, 1977 for a description). Thus, it is primarily

designed to describe task and training data (see Table 3-

9). The only description of equipment-related concepts in

NEPDISS is in the task statements of the task portion of the

data base. Other major limitations of the NEPDISS are its

lack of capability for describing projected system elements,

its apparent lack of appropriateness for use by uninitiated

users, its lack of a capability for generating tasks and

other data impacts, and most important, its lack of a true

data base management capability for updating and refining

system elements.

Despite. the weaknesses, it is important to note that the

NEPDISS is especially strong in describing task and skill

related requirements which are appropriate for training and

personnel analysis.

o Other Data Bases

There are a number of other data bases which attempt to deal

with some of the issues related to the SDT. For instance,

the Consolidated Occupational Data Analysis Program (CODAP)

and the Training Developments Information System (TDIS) are

Army data bases which also deal with task description. The

CODAP focusses on tasks from the perspective of a single MOS

while the TDIS focusses on cmmon tasks which are applicable

3 -

ik_:..........

" ' .- -' " '" " - % ", " " " m" m "I " " '"" "" , ' '' "' . "' . .r '" el " " 
Table 3-9

NEPDISS DATA LIMITATIONS

* Does not describe system functional requirements. design concepts, training

program elements or collective tasks.

Is geared for use by sophisticated users.

" Cannot generate task and cther input data.

" Is not designed to describe projected system elements.

* Does not provide comprehensive data base management capability for updating

and refining system elements.

3.,

.k

'%

3- 36d;

. " ,_L..........--......"......"............................ . .... "-..".." ..... •......................."..
across MOS. However, neither one is geared for use in

describing the hardware/software design, tasks, and training

characteristics of a emerging weapon systems.

3.3 REVIEW RESEARCH RELATED TO HUMAN COMPUTER INTERACTIONS

One of the major problems surrounding the weapon system

design process is the communication and flow of information

among the participants in this more general process. The

ETES SDT is specifically designed to deal with these

communication problems by providing a centralized, automated

data base for describing and updating emerging system

concepts and providing direct access to this data base to

all participants in the weapon system design process. Thus,

the SDT provides a systematic vehicle through which the

participants in the weapon system design may communicate.

To provide a foundation for SDT development recent research

related to human-computer interaction was examined. This

section describes the results of that examination.

The section is divided into two subsections. The first

describes the SDT requirements related to human computer

interactions. The second subsection reviews psychological

research relating to the process of human computer

interactions and summarizes the implications that this

literature has for the SDT.

3--3

-"S

3- 37-

S.-
3.3.1 SDT Requirements Related To Human Computer

Interactions

Task 1.0 examined the problems in information flow and

communication which limited early training estimation. This

task indicated that training developers and other

participants in the acquisition process are not systematically receiving information on early system concepts and IF

are not being kept abreast of system changes and updates in

a timely and systematic fashion. This lack of systematic

communication makes it difficult if not impossible to

effectively assess training and other human resources, and

this lack of assessment, in turn, makes it difficult to

effectively manage and control these resources.

In order for the SDT to fill these communications

deficiencies, the SDT must itself be designed to facilitate

easy and rapid communication with the personnel who will use

it. Task 1 indicated that the primary users of the SDT will

be personnel from the staff of the training developers,

combat developers, and material developers. These personnel

are likely to have had little, if any, experience in

utilizing computers or computerized data bases. Interviews

with current personnel in these organizations indicated that

there is also likley to be very little time or resources to

train these personnel on ETES-related activities. Thus, it

was imperative that the SDT be designed to (1) be utilized

by uninitiated users who have no background in the use of

computers and computer languages and (2) have minimal

training requirements. Fortunately, in recent years there

has been a growing body of literature on human-computer

interactions and the types of interactions which are

3-38

i i . . . - . . . . ... i
'p

Y 7.

appropriate for uninitiated minimally trained users. This

literature is reviewed in the subsections which follow.

Before discussing this literature, it is important to point

out that the systematic study of human-computer interactions

is a relatively new area of research. Consequently, many of

the guidelines discussed in the next section are only

rational schemes for dealing with human-ccmputer

interactions -- empirical research to support these

guidelines is generally not available. However, the

conceptual schemes which have been developed do appear to

have a high degree of face validity.

3.3.2 Human-Data Base Interaction Literature Review S

There have been four major efforts to survey and categorize

literature relating to human-data base interactions. 5 More

details on these four efforts is presented in the

subsections which follow.

3.3.2.1 Martin's Work on Interactive Dialogues

The first comprehensive work in human-computer interactions

was conducted by Martin (1973) who documented his work in a

book on human-computer dialogues. Martin's book was the

first attempt to provide a systematic set of guidelines or

human computer interactions. Earlier work had focused on

This research area is also described as the "manmachine interface" (MMI) as well as the human-computer

interface or interaction.

3-39
the development of guidelines for canputer input devices or

output devices or computer programming practices but had not

systematical ly covered di al ogue or process-related

questions.

Martin's basic approach toward conceptualizing the humancomputer interactions was to divide human-computer interactions into 18 basic dialogue types and to outline the

advantages and disadvantages of each type in terms of types

of users and information characteristics. 6 Table 3-10

displays Martin's dialogues types and the estimated

applicability to the SDT based upon Martin description of

their advantages and disadvantages. As Table 3-10

indicates, the most likely dialogues types for inclusion in

the SDT are menu selection dialogues, form-filling, and

question and answer dialogues. These were the dialogues

which Martin indicated were (1) most appropriate for

uninitiated users and (2) would not require extensive

development costs or special terminals.

Martin also provides a series of guidelines to consider in

selecting input and output devices. Table 3-11 lists the

input and output devices covered by Martin. To reduce

implementation costs, it was decided that the SDT would

utilize existing hardware (see Section 3.2.2.3 for a

description of the SDT configuration). This requires that

the SDT interactive input device mechanisms be restricted to

a keyboard and SDT output devices mechanisms be restricted

to a printer and a CRT screen.

6!

6 Martin presents little empirical evidence to support

his concepts (very Little work has been done in this

area). However, his concepts appear loical and seem to

have 3 high degree of "face validity".
Table 3-10

MARTIN'S DIALOGUE TYPES AND THEIR APPLICABILITY TO SDT*

1. Programming languages

2. English-language dialogue 4.

3. Limit English input

*4. Question and answer dialogues (in which the computer asks the operator

a series of questions)

5. Dialogue using mnemonics

6. Dialogue with programming-like statements

7. Computer-initiated dialogues (in which the operator responds to the computer

rather than the computer responding to the operator)

*8. Form-filling (in which the operator fills out a "form" on a visual display)

*9. Menu-selection dialogues

10. Build dialogue features into special terminal hardware

11. Dialogues with a light pen for input (or other means of pointing to the

screen)

12. Fixed-panel responses (in which the computer responds with one of a standard

set of panels)

13. Modifiable-panel dialogues (in which the panels can be modified by the

programs)

14. Graphics using chart displays

15. Graphics using symbol manipulation

16. Dialogues with photographic frames

17. Voice answerback dialogues

i. Dialogue via a third party

*Items applicable to SDT.

3.4

3-41 .L

-
"S.

Table 3-I1

MARTIN'S CATEGORIZATION OF INPUT-OUTPUT DEVICES

AND THEIR APPLICABILITY TO THE SDT

I.

.. :

Input Output

*Keyboard *Typewriter or printer

Lever set or Alphanumeric screen

Rotary switches *Graphics screen

Push buttons Screen displaying film frames

Light pen for point at screen Light panel

Finger pointing at screen Graph plotter

Stylus for drawing Dials

Plate reader Voice answerback

Badge reader Facsimile machine

*Applicable to SDT.

. .
7r V. * 7 -V Tim~ V . ,i

3.3.2.2 Ramsey and Atwood's Work on Human Factors in

Computer Systems

Another major effort in systematically assessing humancomputer interactions was directed by H. Rudy Ramsey and

Michael Atwood in work sponsored by the Engineering

Psychology Programs of the Office of Naval Research. Ramsey

and Atwood (1979) have developed a conceptual scheme for

classifying different areas of research relating to humancomputer interactions. Table 3-12 presents this scheme and

also indicates which Atwood and Ramsey categories are most

relevant to the SDT development. It is important to point

out that input and output device questions were less

relevant to the SDT because the SDT utilizes existing Apple

III input/ output devices.

In discussing user characteristics, Ramsey and Atwood review

several past articles which have dealt with the requirements

and/or capabilities of uninitiated users. (e.g. Card et al,

1974; Eason, et al 1975; Evans, 1976; Martin, 1973;

Nickerson and Pew, 1971; and Thompson, 1971). Atwood and

Ramsey indicate that interactions by these users can be

facilitated if the computer-initiated or natural language

dialogues are used, and they point out that natural language

dialogues are very expensive to develop.

As was noted earlier, Ramsey and Atwood indicate that

canputer initiated dialogue would seem to a much more

effective means of communication with uninitiated users, who

are exactly the type of users which will utilize the SDT.

Ramsey and Atwood point out that computer-initiated dialogue

has several advances. First, this approach to dialogue

3-43 .
*

z

z ~i

- x

~ :3!

o

- -

~c

z

3 z * 0 L I ~) 0

Z C.) -~ - C.) C) ..

C.) C.) ~ ~ - - .~ - -V - - 0 C.)

L.C) 0 -V -D .-- o 3 - 0 = - - 0 L. C.) C.) ~, >. 0. * C.) -

~ En C.) C.) - - C) 0. - ~ 0 C) > 0. -- C.) - C.) -~ ~) C) C.) C.) Z C.) j~ -

C.) 0 V C.) C.) - C.) - C.) C.) 0 >~ -~ C.) C) 0 C.) C.) - 0 ~ *~ ~ * 0 *~ C.) ~ 0~V - -~ - ~ - C.) ~ * C) *V ~ C.) C.) -

~! -o - 3 ~ - ~- 3. V

* C) . 0 C.) - -~ 3 0 C/2 ~

- z - ~ 0

C.) 0 0- C.) C.) C.) * -. -~ - - - - -' fl - 0 3 * - 0. C.) ~ 0 C.) ~

* .~ ' 3 = - N - C) ~ C) - C.)

o 0 C) ~) V Z 0. - - - .- ~ 0 ~ ~) -

- C) *. *~ ~..z 0 - *'~.. V - - 0 - 0 - - .~. - -~

* - 3' - ~ 0.~ C.) .-. z -

P -~ C.) >' -

I p

- 3 C.) .. )

- - - -~ '- .7 - ~ ~ - z . 0 ~ ~ 0

* ~. 2 _ 0 ..fl -

-~

* ) -; ~ C) -~

-~ -~ -

0 -

Z

-. 3 -

* f

- - .. *: =, .- p - - - .- - p *-. p - *J**' - - -- - . -' S

- p p -~ -

- p - -. -

5-.

3-.' 4

-. *,..-....** ****5* '% -,-.........;...-. . -
I.

allows the system to rely on the passive vocabulary of the

user (the set of words which the user can recognize and

understand), which is typically much larger than the user's

active vocabulary (words which the user can generate and use

without prompting). Second, it allows the designer to

implicitly convey to the user a "mental model" of the

system's dialogue structure. The major disadvantage of

computer-initiated dialogue is the frequent delay it may

produce for experienced users.

Ramsey and Atwood also present a scheme for classifying

different types of interactive dialogues, and list the

advantages and disadvantages of each type. A summary of

this discussion is presented in Table 3-13. Two types of

dialogue seem especially applicable to the SDT because of

their emphasis on computer-initiated dialogue -- form

filling and menu-selection.

Form-filling is often used in situations in which the user's

input is dominated by parameter values, rather than

commands. Many system attributes involve this type of

data. hence, form-filling would appear to be particularly

useful as a data input mechanism for data attributes.

Menu selection is described by Ramsey and Atwood as the

"archetype of cmputer-initiated dialogue." Unlike question

and answer dialogue or form-filling, all of the times to be

selected appear on the screen, and thus the user need only

recognize the desired action. Also, a simple menu-selection

dialogue ordinarily requires only one user input (on the

keyboard or screen), rather than, for example, the series of

keystrokes required to type a whole word. Redsdale (1970)

reports a study which documented the effectiveness of menu

3-45
0

-n >, -Z

> :3 - --

V ;n

0,0 0 b 

0 51 Q = =-

0 a

z 3:0 zL 0

Zn -- o

-z ~~O~0 E ~ 00 ~~) -

CCL)

31 -Z - -to

-~ 0 ~'.~~3 
selection with naive users. The study indicated that menu

selection was especially effective when used as means of

obtaining answers to a set of branching questions.

It is especially important to note that menu selection is a

highly effective dialogue method for hierarchical search

because of its reliance on the user's passive vocabulary and

recognition memory. Hence, menu selection is particularly

applicable to information retrieval (see Thompson, 1969,

1979 for a more extensive discussion of menu selection as a

data retrieval mechanism).

3.3.2.3 Smith's Work on Man-Machine Interface

Another major effort related to the assessment of humancomputer interactions was Sidney Smith's (1980) work on the

development of guidelines for the man-machine interface in

C3 systems. This work was sponsored by the Air Force

Electronic Systems Command.

Like Ramsey and Atwood, Smith (1980) has developed a scheme

for categorizing topic areas related to human-computer

interactions. Table 3-14 displays Smith's schemes and the

categories of Smith's work which had the most applicability

to the SDT. Selected aspects of Smith's (1980) major report

in this area which were relevant to ETES are reviewed below.

o Dialogue Types

Smith, like many other investigators in this area, notes

that computer-initiated dialogue types (e.g., form-filling, .

menu selection) are more appropriate for uninitiated users

3-47

-.. ,.-.
Table 3-14

SMITH'S SCHEME FOR CLASSIFYING HUMAN-COMPUTER

INTERACTION INFORMATION*

1.0 DIALOGUE TYPE 4.0 SEQUENCE CONTROL

1.1 Question and Answer 4.1 Transaction Selection

1.2 Form Filling 4.2 Interrupt

1.3 Menu Selection 4.3 Context Definition

1.4 Function Keys 4.4 Error Management

1.5 Command Language 4.5 Alarms

1.6 Query Language

1.7 Natural Language 5.0 USER GUIDANCE

1.8 Graphic Interaction 5.1 Status Information

5.2 Routine Feedback

2.0 DATA ENTRY/INPUT 5.3 Error Feedback

2.1 Position Designation 5.4 Instructional Aids

2.2 Direction Designation

2.3 Data Type 6.0 DATA TRANSMISSION/

2.4 Entry Formats COMMUNICATION

2.5 Data Validation 6.1 Data Transfer

2.6 Data Processing 6.2 Data Type

6.3 Transmission Control

3.0 DATA DISPLAY/OUTPUT

3.1 Data Type

3.2 Data Density

3.3 Data Aggregation

3.4 Data Coding

3.5 Display Partitioning

3.6 Display Selection

3.7 Data Coverage

3.8 Display Update

3.9 Data Selection

*Table derived from Smith (1980).

3-40

-m , . . . nl u
than are user-initiated dialogues (e.g., programming 

languages). Table 3-15 displays Smith's categorization of 

the different dialogue types and his estimation of user 

tr-aining and response time- associated with eacn type. Based 

upo:c Smith's estimates question and answer, form-filling and 

menu selection would seem to be .the most appropriate 

dialogue formats for the types of uninitiated users who will 

use the SOT (response time is not a critical issue for the 

SOT). 

o Data Entry/Input 

In discussing data entry /input topics, Smith mentions some 

general guidelines that should be considered in the 

construct ion of data entry mechanisms. First, an operator 

should seldom be required to enter the same data twice or 

ente':- a data item already entered by another operator. He 

sugg.3sts that the way to do this is to program the computer 

to maintair. context (that is, the canputer should be able to 

access all data related to the user's input). Second, 

canp1t.er systems should be flexible. This means that the 

user- sh :>uld be able to set his own pace, cancel incomplete 

t r- ansa c t ions , e tc • 

o Data Display/Output 

Much of the material covered in Smith's discussion on 

display/ output is redundant with Ramsey and Atwood's 

wrr-k. Hence, it is not re.peated here. 

3-49 
Table 3-15

SMITH'S SCHEME FOR CLASSIFYING DIALOGUE TYPES*

-t

Required System

Dialogue Type User Training Response Time

Question and Answer Little/None Moderate

Form Filling Moderate/Little Slow

Menu Selection Little/None Very Fast

Function Keys with High/Moderate Fast

Command Language

User-Initiated Command High Fast

Language

Query Languages High/Moderate Moderate

Natural-Language Dialogues Moderate Fast

(potentially little)

Interactive Graphics High Very Fast

*Table taken from Smith (1980)

3- 5

.5

. . . .. . -S - :
o Sequence Control

Smith suggests that menu selection might be used as an

effective means of providing sequence control for

uninitiated users. He also indicates that flexibility is

important in sequence control, particularly in interactions

which involve the modification of stored data.

o User Guidance

Smith suggests that the fundamental rule in the area of user

guidance is that for every action by the user there should

be a response by the machine. Such feedback helps maintain

user orientation.

3.3.2.4 Sidorsky and Parrish Work on Battlefield Automated

Systems

A comprehensive assessment of human-ccmputer interaction is

currently being developed by Sidorsky and Parrish (1980) in

work conducted for the Army Research Institute. The goal of

the Sidorsky and Parrish work is to develop general

guidelines for describing and designing battlefield

automated systems so that ultimately the interoperatibility

of such systems can be increased, and, consequently, user

errors can be decreased. Like Smith (1979) and Ramsey and

Atwood (1979), Sidorsky and Parrish (1980) have developed a

conceptual scheme for classifying information related to

human-ccinputer interactions (see Table 3-16). They have

also developed a method for systematically assessing the

human-ccmputer transact ions which currently occur in

battlefield systems.

3-51

.. . ,.

• --;.i~v -L-" .- .-~ i ..' .. / -".-...- .- -- .i , - .i •V -•-i ..- "... .-. " . - '-;'-.'" ',i- .. -". -. ".".. ,. -. -. - - '--'.-
V.-

Table 3-16

SIDORSKY'S AND PARRISH'S SCHEME FOR

CLASSIFYING HUMAN-COMPUTER INTERACTION INFORMATION'

1. CONTROL METHODS 5. DATA RETRIEVAL ASSISTANCE

1.1 Command Languages 5.1 Query Method

1.2 Menus 5.2 Query Structure

1.3 Function Keys

1.4 Hybrid Methods 6. GLOSSARIES

1.5 Prompt/HELP 6.1 Standard Terms

6.2 Character Sets and Labels

2. DISPLAY FORMAT 6.3 Glossary Availability and Use

2.1 Fixed Alphanumeric Displays 6.4 Abbreviation and Coding

2.2 Variable-Length Alphanumeric

Displays 7. ERROR HANDLING

* 2.3 Graphic Displays 7.1 Prevention

2.4 Highlighting 7.2 Detection

7.3 Feedback

3. DATA ENTRY ASSISTANCE 7.4 Correction/Recovery

3.1 Information on Legal Entries

3.2 Unburdening of Input 8. USER/OPERATOR CONFIGURATIONS

3.3 Interrupts and Work Recovery 8.1 Operator(s) Only

8.2 Operator(s) and User(s)

4. MESSAGE COMPOSITION AIDS 8.3 Combined Operator/User

4.1 System Design Features 8.4 Operator and User Chains

4.2 Format for Alphanumeric

Messages

4.3 Graphic Messages

*Tabie derived from Sidorsky and Parrish (1980).

3-52

71

-d6
The study by Sidorsky and Parrish had just begun at the time

of the ETES review, hence they only had developed guidelines

I.L

in a few areas (e.g., data entry). Thus, their work .

3.3.3 Summary of Implications for SDT

As a result of the review of literature related to humancomputer interactions, two general guidelines were

identified and subsequently implemented in the SDT.

First, in selecting the type of dialogue which to be

implemented in the SDT, it was clear that some form of

computer-initiated dialogue should be utilized given the

types of uninitiated users who could be expected to employ

the SDT. As a result, three computer-initiated dialogue

types -- question and answer, form-filling and menu

selection were selected for inclusion in the SDT.

Second, coding of information on the SDT displays was

developed in accordance with the most current guidelines in

Smith (1979), Sidorsky and Parrish (1980), and Ramsey and

Atwood (1980) (see Table 3-17). However, only coding

schemes which could be implemented on the Apple III were

utilized (i.e., underlining, italics, character size

control, position displacement, and arrowing).

o .

.

3-53

I

.- v i .r 1 - 2 I - I I -. . .I I I - - I. . - m
71"

b

S.-

Table 3-17

SIDORSKY'S AND PARRISH'S SCHEME FOR HIGHLIGHTING INFORM, TION

Highlighting Method Methods Applicable to SDT

Brightness Control

Character Size Control x

All Upper Case x

Reverse Display

Underlining X

Different Font

Color Control

Blinking, Pulsating

Boxing x

Arrowing x

Symbolic Tagging

Alphanumeric Tagging

Position Displacement

3-54

I~
II

3.4 DEVELOP SDT

An overview of the process used to develop the SDT is

presented inn Figure 3-2. More details on the steps in this

process are provided in the sections which follow.

3.4.1 Identify SDT Requirements

Detailed functional requirements were developed describing

(1) the types of data elements which should be included in

the SDT, (2) the types of human-computer dialogue techniques

to be used in the SDT, (3) the estimated size and scope of

the SDT, and (4) the type of canmunication which could be

expected between the microcomputer and mainframe in the

SDT. Input for the development of these requirements was

provided by (1) the review of LSCMM documents and processes,

particularly the output reports generated by these processes

which are listed in Table 2-4 (see Section 2.1), (2) the

interviews with acquisition participants (see Section 2.2),

(3) the review of data base management techniques and

capabilities (see Section 3.2), and (4) the review of human

computer interaction guidelines (see Section 3.3).

The detailed SDT requirements were listed in the first

yearly report of the ETES development contract.

3.4.2 Identify Data Elements

A detailed data dictionary was developed describing (I) the

title of each data element, (2) the maximum length of the

data element, (3) the maximum number of the data elements,

(4) the type of data contained element (alphanumeric or

3-55

.................................................................. - . -
IDENTIFY

SDT

REQUI REMENTS

IDENTIFY

DATA

ELEMENTS

IDENTIFY

FRAMES AND

PROGRAM

LOGIC

DEVELOP

SOFTWARE

TEST/REVISE

SOFTWARE

Figure 3-2 Overview of Process Used to Develop SDT

. -. . . . . . . . .. . . . r
-~Z 70 Z- T. -~ -

numeric, (5) the category of the data element (entity,

subentity, and attribute) (6) the relationships between

other elements in the data structure and, (7) the definition

to be used to describe the data elements.

The data dictionary was automated to facilitate update. The

final version of the data dictionary is listed in Appendix A

of the SDT User's Guide. An example page from the data

dictionary is presented in Table 3-18.

3 .4. 3 Identify Data Structure

Concurrent with the identification of data elements, the SDT

data structure was identified. The data structure specifies

what data elements are entities, subentities, and attributes

(see Section 1 for a description of these concepts). Seven

major entities were identified: functions, missions, tasks,

equipments, duty positions (personnel), media, and courses.

Figure 3-3 describes the data structure that was used to

describe the relationships between these entities. A more

detailed description of the SDT data structure is provided

in Appendix A of the SDT User's Guide. As Figure 3-3,

indicates, the SDT employs a hierarchical data structure

(see Date 1971 for a description of different data base

structures). Originally, the intent was to use a rn 1-

hierarchical or relational data base structure which would

allow each entity to be associated with each of the other

entities. However, the relational data base structure

required internal memory requirements that were beyond that

which were available on a microcomputer.

3-, :9 -
- 7Nw

-7 In -2 a~ 

! - r- C5-

4A Q' r Z- 0- 0I

k- > 'G C- CL.- 

di 4 -: 'A

L- 04 >: j 0 a,

r- C- - ow cu -i 

l S - In -4 CL C t= Z

a, Z C. t:-L, -LM m4 ajC-n uU,1

1- - QA 0w.-~ ..-. CL n -4, Z. z Z4 C'4 C- CU > kA 0 04*F- ,- a, 

'-~ '- ~ ' >-.- - ~ - " "' ~ Ca 04 w a 0 - L

Or C.. - .0 , - - - ' - * 0 * ~-Z z04 ka f 0-. * . eC L. a. a 0 .

0004 0a C. 0 4L0U C ' C. a. C, a- a, ID.- 0': In 'Q~ 71 0 C D 00 *..' .- 04 .1

CM ' - -Z o C a 0404 'a z x. U -- r. L.Z 0. 04 r o .' a'.- 00 ZCI 'Z' L a w , w C. . -D a 4jC . ' *0 ~ . ~ . C.( 4 4 . ' C. 0 . . - 4 . - 0 0 ' 4 o 04 O ~ ~ 0 ~ U ~ ..a a ~ 0 ~ ,,,,D

- ~~C -. '. U' 0 = C'*- .- 4 . - ' C - M 0 O 4 -. .. C. U . . - ~

- 9.- -

04cmO CD

co Ll; -' .. = a -'a cn LO

.: C- La ,

Cr=

00 aD it Cc ~ a;. cd; *~~u r- I:_- . '

- -- 04 fl C

-04~~L 6f- - a;-_ :3(J

a; - -- a;
o Percent performing - The greater the proportion of

individuals in an MOS who perform a task on the

job, the greater the payoff for training them in

an institution. Thus, if a specific duty position

accounts for a substantial percentage of the

personnel within an MOS, then that duty position

is identified to be trained in the institution.

For example, for MOS 95B, Military Police, at

Skill Level 1, there are several different duty

positions, such as military police, desk clerk,

radio dispatch clerk, fingerprint clerk, machinegunner, investigator, prisoner-of-war processing

specialist, etc. The majority of these duty

positions account for a relatively small percent

of the total MOS strength. However, the training

assignment decision is easy because the position

of military police accounts for approximately 60

percent of the MOS for Skill Level 1.

o Time between training and required job performance

- If a task is complex and requires practice to

maintain proficiency then tasks which are not

performed by soldiers within 6 months after

training should be trained in the unit.

o Consequences of Inadequate Performance - This

criteria is important insofar as it indicates

which tasks may need concentrated or professional

instruction that can best be accomplished in an

institutional setting.

Data on these or other selected criteria can either be

collected in surveys conducted directly by the analyst or

3-59

. . . . . ..
',, F

through the Army Occupational Survey Program (AOSP) (more

details on procedures for collecting this data are provided 5-

in Section 3.1.) These data can be summarized on a

worksheet like that shown in Table 3-10. Data entries to

this worksheet should consist of mean scores/ratings for

each task on each criteria.

As part of the data collection process, the user should also

examine the Trainer's Guide for the MOS. This guide will

define the specific training settings currently used for the

MOS. If a new MOS is involved, training settings for other

ccmparable MOSs should be used (see Section 2.0 for

information on identifying comparable MOSs).

3.2.2.3 Make Training Setting Assignments

Training setting assignments must be made in a two step

process. First, tasks should be assigned to the either

institutional or unit training setting categories. Second,

specific settings within each of these general categories

must be determined. It should be noted that within the

institutional training setting category, there is a close

correspondence between specific training settings and skill

level. Thus, if you know the skill level of a task and have

made the assignment to the institutional training category

the assignment of tasks to specific institutional training

settings is largely determined. Listed below are the

relationships between skill level and institutional training

settings.

O Skill Level 1 - 5)..±c Training, Advanced

Individual Training (AIT), or One Station Unit

Training (OSUT).

3-60

. ............... .. . .
SDT MODE SELECTION

SELECT SDT OPERATION MODE:

-EXIT SDT- (Terminate the Program)

-AUDIT/UPDATE- (Examine Audit Trail/Update on Honeywell

at DRC)

-INPUT DATA- (Add Tabular Data into the SDT data files)

-CORRECT DATA- (Change Specific Attribute for a Particular

Entity) "

-OUTPUT DATA- (Display or Print Existing Data)

-SAVE/RESTORE- (Copy this system to or from diskette)

-APPLICATIONS- (Execute the Applications Programs)

-AUXILIARY- (Execute Auxiliary SDT Programs)

Figure 3-4 Example of SDT Menu

17-1

qU

...................................................... ................. .. .. . .
.1

Software Documentation. This documentation provides a

general description of the SDT programs, and subroutines

library functions and program logic.

3.5 APPLY SDT TO ARMY WEAPON SYSTEM

The original objective of this task was to validate the SDT

through a "simulated system development exercise."

Recognizing the limited relevancy associated with such an

exercise, an attempt was made to apply the SDT to a "real"

Army weapon system development project. With this in mind,

an extended set of discussions was held with the Ballistics

Missile Defense (BMD) office regarding the possible

application of the SDT and other ETES tools to the manned

BMD subsystems. Initial talks with this office were

extremely positive and provisions of significant additional

funds to expand the ETES application seemed likely. However,

the BMD office finally decided that an ETES application

would not be appropriate since (a) the BMD system concept

had been changing radically and had not yet been approved by

Congress, and (b) existing data on the system was considered

"sensitive" and not appropriate for release to outside

contractors.

The long yet unproductive discussions with the BMD office

when coupled with the tight ETES development schedule, make

it impossible to attempt to develop another extensive

application of the SDT. Instead, it was decided to examine

the SDT in a limited application to the Single Channel

Ground Activiated Radar System (SINCGARS). Since a HARDMAN

methodology was being conducted on SINCGARS by the same

3-62

-,-, .:_- _. i.; .:: -,: -' .:- ......-.- - . ,- - -' ",:. v : -) , -: . -:-,- . : -, ,-. - - .--- - ,.;. - , --- ; :> - :. .
contractor who was developing the SDT (DRC), SINCGARS

offered a low cost and rapid means of applying the SDT. i

During the SINCGARS application, the SDT was used to

describe system functional requirements, tasks, and

equipments. Three criteria were used to assess the validity

of the SDT during this application: ease of use, adequacy of

SDT as a Data Base Management tool, and meaningfulness of

results. Ease of use was assessed by interviewing the

personnel who applied the SDT. Meaningfulness of results

and data base adequacy was assessed by examining the data

input into the SDT. The results of the interviews indicated

that the SDT was easy to use and that it could adequately

perform the data base functions of storing, updating, and

outputting system data. The major user of the SDT was a

former Army training analyst who was able to obtain

meaningful output for an SDT entity within one week of his

initial session on the computer; - this despite the fact

that he was provided with no user guide; (the user's guide

was developed in a later task), and with only a twenty

minute briefing on SDT operation.

However, meaningfulness of the results produced during the

application appeared to be only fair. More specifically,

there appeared to be some confusion regarding the meaning

and use of several of the non-training related SDT data

elements, such as system functions and missions. This

problem seems reasonable since the ETES user guides did not

exist and there were no other documents describing the use

of several of these non-training related data elements.

In summary, the results of the SDT application clearly

indicated that (1) the SDT was indeed easy to use and (2)

3-63

. .1
.1'

successful usage of the SDT and all its associated data

elements would require users to either be familiar with ETES

procedures which generate the SDT data elements or to have

the user's guides for these procedures on-hand for reference

throughout the application of the SDT.

3-64

4-

4-

3-64

b'
SECTION 4 - TASK 3: DEVELOP TRAINING ESTIMATION AIDS

AND PROCEDURES L

h

Figure 4-1 displays the procedures that were used to develop

the Training Estimation Aids and Procedures. More details

on each of these steps are described in the sections which

follow.

4.1 IDENTIFY FUNCTIONAL REQUIREMENTS

Based upon the analysis of LCSMM requirements and user needs

which was conducted in Task 1, a detailed listing of the

functions which a comprehensive set of early training

estimation aids and procedures must achieve was identified.

This list is presented in Table 4-1. The original intent of

Task 4, as outlined in the statement of work, was to simply

develop aids and procedures for generating task

descriptions. However, the results of the Task 1 review of

existing Army procedures indicated that an early training

estimation must perform six basic functions. These

functions are:

(1) Estimate and describe a ccmplete list of system

functions. Without a complete list of system

functions and associated performance goals it is

extremely difficult to build a comprehensive

system description or to evaluate how well the

description meets requirements. Functional

--
TASK 1

RESULTS

*LCSMM INTERFACES

uSER PRACTICES AND NEEDS

IDENTIFY

FUNCTIONAL

REQUIREMENTS

TEAP

FUNCTIONAL

REOUI REENTS

4 Fgur 4. Prcedre fo DeeloingAutmaE ITN

Traiing stimtionAidsProcOLre

SEE-2OLST
I- 0 0 WT- , -

V)- W

E- a

- 2-

L0 U)

5!.Q Q) .u .- C

w 00w CLr

zi

z z z

z z z-z C4n

7 ,. C) V) C

i.'n C

7c-
' 

. . 

. . 

. 

. 

. . 

,. 

.. 

. . , 

J 

'; 

' 

·, 

. 

" 

, 

., 

, 

I 

, . 

• 

'· 

. 

• 

""' I 

..,. 

r------ ---------·-------------------

2.4 Develop New System Task List 

2.? Assign New System Tasks to MOS D~ty Position, and Skill Level 

3.0 Est:.illale Trnining Program 

3.1 Sei~ct Tasks for Training 

3.2 Assign Tasks To Training 

Settings 

3.3 Identify Skills and Knowledges 

3.4 Develop Target Po~ulation 

Oeser iption 

3.5 Sequence Tasks/Skills for 

Training 

3.6 Construct Quasi-Program of 

Instructions 

3.7 Assign Task/Skills to Media j 

3.8 Construct Training Paths 

4.0 Estimate Training Resources 

Table 4-l (Continued .. ) 

F~nctions Selected fur Jncl•1sion in Training Estimation Aids/Proce

Nrn OR 

~~!ST !~\LTQQb 

New 

liAR OMAN 

Job and Task 

Anal. Handbook 

New 

Existing lSD 

Procedures 

Job and Task 

~na t._J!andbook 

New 

HARDMAN 

TEEM : 

HARDMAN 

TYPE OF MODIFICATION (IF EXISTING) 

Extensive Proceduralization of 

IIAHDMAN Procedures 

Minor Modifications to Existing 

Procedures 

Extensive Modifications to Deal 

with Data Problems Associated 

with Early Phases 

Minor Modifications. Specifications 

of Input Data for Job and 'l'ask 

Analysis Handbook Procedures • 

Extensive Modification. Significant 

Proceduralization of HARDMAN Techniques 

Extensive Modification. Media Assignments Conceptualized as Dynamic Prog. 

Problem • 

Extensive Modification. General HARDMAN 

Concept Tailored for Army 
a) a)n

4 c- -) a,.

<C C

CrC a) EQ )'.)

cu - CC C~

XCC a)r 4

Q) 'V

w V)

vC C 0 C) I

x0x 0 a)

z a) 

Z, Cc C LJ IC

4-5
·'!'able ·1-1 (Continued .. ) 

functions Not Selected for Inclusion in Training Estimation Aids/Procedures 

.----------------------- --------------------·· ·-----

FUNCTION 

----·--

-----------'-------------- --------------. ·-------------- ----------------

Estimate Collective Training Requirements 

Generate Collective Tasks 

EEtimate Training Programs Estimat~ Training Resources 

Estimate Training Cost (O&S) 

Estimate Training Efficiency/Effectiveness 

Estimate Unit Training Requirements 

Estimate Training Programs 

Estimate ~raining Resources 

Estimate Training Costs IO&Sl 

Estimate Training Efficiency/Effectiveness 

Provide Training Input to Operation Testing 

Prov~de Input to Test Plan Development 

Assess Results df Training 

Estimate Acqui~ition Training Costs 

Develop Detailed Training Efficiency/ 

Effectiveness Measures 

Lack of 

Lack of 

Lack of 

Lack of 

Lack of 

Lack of 

Lack of 

Lack of 

Covered 

Covered 

Lack of 

Lack of 

Resources 

Resources 

Resources 

Resources 

Resources 

Resources 

Resources 

Resources 

in HRTES 

in HRTES 

ResourcesResources 
-- '

information items are identified as part of the QPOI

construction process (Procedure 3.6), identification of

course locations is relatively straightforward. Additional

guidance for course location identification can be provided

by examining (a) the course locations used in comparable

existing courses and (b) the projected locations where the

weapon system will be fielded.

4.1.2 Determine Course Frequency

Course frequency (i.e., the number of starts and sessions

per site) is determined by applying the algorithm listed in

Table 4-1. To estimate course frequency, you must determine

the number of students to be trained (Procedure 4.2). The

algorithm listed in Table 4-1 must be applied for each

location and year in which the course will be given.

4.1.3 Determine Usage Rates

Usage rates are determined for instructors and traininj

media. The usage rate for instructors is called tWo

student/instructor ratio. The usage rate for media is

called the student/media ratio.

Student/instructor ratios are determined for each

instructional method used in a course. Table 4-2 displays

the student/instructor ratios that are provided for each

type of instructional method in DA Pam 570-558 and TRAIN"T

Cir 351-12.

Student/media rates are determined by identifying th, ra,'

that were used for similar typeF, of media in coam ir 

courses. This information may be obtained by examlnri thn"

4-7

. . .°
TABLE 4-1. ALGORITHM FOR DETERMINING NUMBER OF COURSE

STARTS AND SESSIONS PER SITE.

1. Establish number of courses required per site

STUDIN NCOURSE

NSITE

STUDIN = Student Input Requirements

NSITE = Number of Training Sites

NMAX = Maximum Student-Instructor Ratio

for Course

NCOURSE = Number of Courses Required Per

Site

2. Determine maximum number of starts per year

(ClAX)

50 CMAX

CLENGTH

CLENGTH = Course Length in weeks

CMAX = Maximum Number of Course Starts

per year (NCON)

3. Select the smaller of NCOURSE or CMAX as the

number of course starts (NCON) required per

site.

4. Determine maximum number of sessions needed per

start (NSES)

If NCOURSE CMAX, NSES 1

If NCOURSE CMAX, OURSE -- NSES

KA X

Round NSES up to next highest unit.

4-8

S.. .j . IV . .U J.. . U, Ud .. 
.S.

between the Army HARDMAN applications and ETES. ETES

procedures for determining system functional requirements,

estimating training programs and estimating training

resources and costs were incorporated into HARDMAN. In many

cases, since HARDMAN was being continuously applied to Army

systems during ETES development ETES developed procedures

were applied in Army HARDMAN applications before these

procedures were fully completed and integrated in ETES.

TEEM provided two critical inputs to ETES. First, it

provided a media selection model, which with significant

modifications, provided an effective vehicle for early media

selection (see the description of the ETES Media Selection

Program in Appendix A). Second, the TEEM concept of

training efficiency provided a useful, albeit crude measure

for estimating the "effectiveness" or goodness of an early

concept for a training program element. More specifically,

in the training efficiency concept, the goodness of a

training program element (such as media) is measured by

assessing how well the characteristics of the tasks to be

trained match the characteristics of the training program

element.

After existing tools were selected for inclusion in the

training estimation aids and procedures, the functions were

examined to identify the areas where state of the art

technology could be most effectively applied to develop new

tools. In determining whether a new tool was appropriate,

three factors were considered: (a) the criticality of the

finction to early training estimation, (b) the technical

feasibility of developing the new tool/given the current

state of the art, and (c) the resources available under the

C;
IETES contract. Table 4-1 lists the functional areas where .4,

new tools were developed.

There were five major functional areas where state-of-theart technology and resource constraints prohibited the

development of a new training estimation aids and/or

procedure. First, it was not possible to develop techniques

for estimating collective training. Second, it was not

possible to develop procedures for estimating unit

training. Thus, the ETES Training Estimation Aids and

Procedures developed in this contract focused on individual

institutional training (see Table 4-2). It was decided that

it was more beneficial to develop a complete set of

procedures for dealing with individual institutional

training which covered the entire training estimation

process from task enumeration to training program evaluation

than to develop a partial set of procedures for the full

range of Army training settings. Third, procedures were not

developed for estimating acquisition costs for training

programs (thus, only procedures for estimating operations

and support costs were developed). The major reason for

this was that it was extremely difficult to obtain any kind

of systematic data on training acquisition costs. More

specitically, it was very difficult to obtain any kind of

useful data for existing training programs. It was

determined that while some data on acquisition costs is

available in informal data bases throughout the Army, the

cost of locating and analyzing this data would have expended

too much of available ETES development resources.

Fourth, procedures were not developed for providing input

lata to, or evaluating the results from, Operational Testinj

(OT) even though these functions were clearly related to

4-10

. .i .

,- -' -4 .-A " ." -, . , "/ . . . , "/ ."," :" :" ' -' -" -'- "- " " " ' """ " ' " ' " " "-,t > '
wfl r,'.r rjr 

. -7-Yr F 

. -~.*~--,-.*

o

4-

Cu

4- w

I.-

4-

Cu

1~ -~

I- 

-

C', w

I- A- w

4-

C

Cu

U,

ZU, Cu 

0

C X LL

0*-

LL< .2

4-

4-

N 4- h.

* U,

C 

U

_ II

.0

Cu _____ ______ 

x ICu

4-C

U

4-11

I
- - . - - ... * 7 . . -

training estimation during the Demonstration and Validation

phase of the acquisition process. The reason for this was

that a separate project, the Human Resource and Test

Evaluation System (HRTES), was designed to provide

procedures for providing manpower, personnel and training

related input into Operational Testing (OT). However,

guidelines were developed for providing input to HRTES.

HRTES and ETES and other manpower, personnel and training

tools are being integrated in the MIST contract. (see

Appendix D for a description of HRTES and MIST).

Fifth, while procedures were developed for estimating

training efficiency/effectiveness, development of a complete

capability for estimating early effectiveness was not

possible. More specifically, it was not possible to develop

measures of effectiveness that were appropriate for

inclusion in cost/benefit ratios. Such ratios require the

development of effectiveness criteria which can be measured

on a true ratio scale. It was not possible to develop

procedures which could produce effectiveness measures which

met all the theoretical requirements for ratio scale

development.

4.3 SELECT TOOLS TO BE AUTOMATED

Once potential tools were identified, an analysis was

conducted to identify which tools should be automated. Each

tool which was amenable to automation was identified and

assigned a priority, the intent being to automate tools

based on their priority until funds allotted for this task

were expended (No specific set of prncedures were mentioned

in the statement of work). Priority was based on three

4-12

I
1R.:-

factors (a) how critical the tool was to early training

estimation, (b) the amount of training analyst time that

would be expected to be saved by automating the tools and

(c) ease with which the tool could be automated, given the

hardware/ software configuration selected for the SDT (Apple

III/PASCAL). The overall goal in operating the tools was to

get the "biggest bang for the buck" while recognizing that

additional automation would take place in the ETES

implementation contract.

Two automated tools were developed; one for assigning tasks

to media based on efficiency of the training media and one

for estimating operating and support costs for institutional

training courses. An overview of these two tools is

provided in Appendices A and B. A more detailed description

of these two automated tools is provided in the User's Guide

Media Selection Program and User's Guide: Automated Resource

and Cost Estimation Technique.

4.4 DEVELOP TRAINING ESTIMATION AIDS AND PROCEDURES

Manual and automated training estimation aids and procedures

for the functional areas selected in the previous subtasks

were developed. The two automated aids were developed using

the procedures outlined in Figure 4-2. A description of the

Training Estimation Aids and Procedures which were developed

under this subtask is provided in Section 1.5.2.

4-

. . .
FUNCTIONAL

REQUIREMENTS

IDENTIFY

OUTPUT

REQUI REMENTS

IDENTIFY

ALGORITHMS

IDENTIFY INPUT

REQUIREMENTS AND

INTERFACES WITH SDT

AND OTHER SOFTWARE

IDENTIFY

FRAMES

AND PROGRAM

LOGIC

DEVELOP

SOFTWARE

TEST/REVISE

SOFTWARE

Figure 4-2 Procedures for Developing Automated

Training Estimation Aids/Procedures

4-14

.. . . . . . . . . . . . . . .. ".
SECTION 5 - TASK 4: DEVELOP EVALUATIVE TECHNOLOGY

Two major activities were accomplished under this task.

First, a review was conducted of existing simulation

techniques to dete -iine if any of these techniques could be

used in early training estimation. Second, the Evaluative

Technology component of ETES was constructed. More details

on these two activities are presented in the sections which

follow.

5.1 REVIEW/EXAMINE SIMULATION TECHNIQUES

A detailed review was made of existing simulation techniques

to determine if any of these techniques had applicability

for early training estimation.

The purpose of this review was to examine "low cost methods

for simulating interacting tasks together with combat

sc nscenarios." Ideally, the simulation technique would receive

as input system descriptions from the SDT and produce qs

outputs a listing of the criticality of each human task to

overall system performance. Thus, the simulation technique

would have the capability of relating both system design

concepts and training program concepts to task performance

and relating task performance to overall system performance.

Table 5-1 lists the simulation techniques which were

reviewed during this task. The two most promising

techniques identified during this review were the Systems

Analysis of Integrated Networks of Tasks (SAINT) developed

.

5-II

a a. . . . . . . .
.1.

0

ccwC W

w t- .0 -

0 C

z. E - 0

0 0

u, - (n . CI CLI

-g 0 . 0 C,, 0

0m 0__ _ _____ __ __

30

> E - 4- 

0 V)~ a Uc
by Pritsker and Human Operator Simulation (HOS) Technique

developed by Streob, Glenn and Whery (1978). However, an

examination of these two techniques indicated that they both

required detailed task data as input. This type of detailed

task data is generally not available during the early phases

of the acquisition process. Consideration was given to

modifying one of these techniques, SAINT, to accept more

generalized task input. However, this approach was not

taken for two reasons. First, advanced development work on

a human operator simulation using SAINT was being conducted

in ARI's MOPADs contract. It was felt that this work would

make previous SAINT work obsolete and that it made more

sense to wait until this more advanced MOPADs simulation

model was complete and then modify this model for early

simulation purposes than to modify a more primitive SAINT

model. Second, and most importantly, the review of existing

Army early training estimation needs conducted during Task 1

indicated that there were more real and pressing needs for

evaluating early training programs with respect to other key

oriteria such as resource requirements, cost, and training

efficiency/effectiveness than there were to evaluate these

programs with respect to task criticality (several less

sophisticated procedures for estimating task criticality

were readily available and were eventually incorporated into

the ETES Training Estimation Aids and Procedures).

5.2 DEVELOP EVALUATIVE TECHNOLOGY

-n overview of the procedures that were used to develop the

Evaluative Technology is provided in Figure 5-1. A more

Jetailed description of these procedures is provided in the

section which tollows.
TASK 1

RESULTS

*LCSMM INTERFACES

- USER PRACTICES AND NEEDS

IDENTIFY

FUNCTIONAL

REQUIREMENTS

FUNC. REOMNTS. TO

BE INCLUDED AND/OR

DEVELOPED IN ETES

PIOCEDURE

Figue 5- Proedurs fo Deelopng Ealuaive echogy

. TECHNIQUES

-.- - - - STUD - - - - - - - - -

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.2.1 Identify Functional Requirements

Based upon the analysis of LCSMM requirements and user needs

identified in Task 1, it was determined that an effective

Evaluative Technology for evaluating early training programs

must contain procedures for (1) developing figures-of-merit

for assessing the integrated impacts of the training

requirements developed in the Training Estimation

Aids/Procedures, (2) identifying potential problem areas for

system training and the likely sources of these problems,

(3) identifying/evaluating training alternatives, which can

be expected to reduce the training problems, (4) developing

training-related input to key acquisition documents, and (5)

determining/evaluating training development schedules (see

Figure 5-2)

5.2.2 Identify Functions and Tools to be Included in

Evaluative Technology

Once the functions required for evaluating early training

programs were identified, existing tools and technologies

were examined to (1) identify any existing tools which could

be modified to meet the early training evaluation functions,

and (2) identify the most promising areas for new

development.

In order to be selected for inclusion in ETES, a tool had to

meet two general requirements. First, it had to be capable

of producing or being modified to produce output that was

appropriate for the early training of the acquisition

process. Second, trie tool had to be capable of being

applied with the types of general data and information that

5-5"'
are available during the early phases of the acquisition

process.

Table 5-2 lists the tools which met the criteria for

inclusion in the Evaluative Technology.

After existing tools were investigated, the functions were

examined to identify the areas where state-of-the-art

technology could be effectively applied to develop the new

tools. In determining whether a new tool was feasible,

three factors were considered: (a) the criticality of the

function to early training estimation, (b) the technical

feasibility of developing the new tool given the current

state-of-the-art, and (c) the resources available under the

ETES contract. Table 5-2 lists the functional areas where

new tools were developed.

As Table 5-2 indicates, it was possible to develop a tool

for each of the functional areas related to training

evaluation. However, in two of the functional areas it was

not possible to develop tools which fully met all

requirements associated with these functions.

First, in the functional area involving the development of

figures-of-merit, procedures were identified for developing

estimates for eight figures-of-merit (training cost,

training efficiency, training effectiveness, congruence with

training development guidelines, congruence with program

requirements, training complexity, training capacity and

feasinility of the training program). However, only a very

31impl fled procedure was identified for aggrejating these

rcores. There are severa I more sophist iea ted techniques

siuch as multiattribute utility analysis which could he used

5-6C
Ul 

I 

-.1 

'l'abl e 5-2 FUNCTIONAL REQUIREMEN'l'S FOR EVALUATIVE TECHNOLOGY 

Functions Selected for Inclusion in Evaluative Technology ~---------------------,r----------.---------------------------------------.------

I 

I 

ilEW OH 

fUNC'l'_!ON EXISTING 'roOL 'rYPE OF MODIFICATION (IF EXI§:riNG) 

1---------------- -------------'----- ---------------1------------------------------t------

7.0 Evaluate :raining Program 

7.1 Identify/Construct Figuresof-Merit 

7.1.1 Identify Costa 

7.1.2 Identify Efficiency 

7.1.3 Identify ~FfectiV~fidSS 

7.1.4 Determine Congruence 

with Training Guidelines 

7.1.5 Determine Congruence 

with Program Reqs. 

7.1.6 Estimate Training 

c:;omple::i ty 

7.1.7 Estimate Feasibility 

7.1.8 Construct Summary Score 

7.2 Identify Likely Problem Areas 

7.3 Identify Likely Causes 

~.4 Identify Alternatives 

TEEM 

Rembas'l 

NeH 

New 

New 

New 

New 

New 

New 

New 

; 

Study 

I 

' 

Extensive Modifications, General 

HARDMAN Concepts Tailored for Army 

Gen. Concept Developed in TEEM for 

Media Extended to Other Training 

Program Elemento 

REHBA~S Procedures Simplified to 

Reduce Data Input Demands 

Autom

& Cos

nique

Autom

Selec

Provid

of F.f

Only C

Provid

Only C

Provid
Ul 

I 

':::D 

I 

7.5 

7.6 

I 6 0 

8.1 

a.2 

-·,---~-·· 

f---· 

--

Table 5-2 (Continued .. ) 

Functions Selected for Inclusion in Evaluative Technology 

OR I E~t!CTIO!! gXlSl'ING '1\ .. vl, 'l'YPJ:: Or' MODIFICATION ( 1 it f;;, .LiTI!!Ql 

f--- ~---

E·Ja lua te Al ternadves New -

Assess Impact of System New - Changes 

O(::i: ,~lop Input to Acquisition - - P::ocesses 

Develop/Monitor Training New 

Develc[?ment Sched:Jie 

Develop Inputs to Acquisition New 

Processes and Documents 

' ~-·---- -·-~--

Functions Not Selected For Inclualon In Eval II& t i VP. Technologl 

-·-·- --------

FUNCTION REASON WHY NOT SELECTED 

. Estimate Impact of Training Concept r.ack of Resources on System Performance 

. Assess Integrated Impacts of Manpower,, Not Within Scope of Study Personnel and Training 

l 

fOM

Au

Sch

Int

Pro

Ac
to aggregate these scores but these techniques r~quire more 

sophisticated subjective estimation methods. 

of such methods for use during the early 

acquisition process was not feasible within 

constraints of the ETES development, contract. 

Modification 

phases of the 

the resource 

Also within the same functional area of developing training 

figures-of-merit, it was not possible to develop a figure-of 

merit which would provide a measure of the expected impact 

of the training concept on system performance. During the 

early phases of the acquisition process, the only way to 

develop such a measure is to use a simulation model. As 

section S.l indicated, development of such a model was not 

possible within the resource constraints of the ETES 

development contract. 

Finally, it was only possible to develop crude procedures 

for identifying the likely causes of training problems. 

Development of more sophisticated procedures required the 

development of a quantitative mode~ (preferably a simulation 

model) for relating training program elements to the 

training figures of merit. Again, such a complicated model 

was beyond the scope of the present ETES development effort. 

5.2.3 Select Tools To Be Automated 

Once potential tools were identified, an analysis was 

conducted to :.dentify which tools should be automated. Each 

tool which was amenable to automation was identified and 

assi~·ned a priority,. the intent being to automate tools 

based on heir priority until funds available for this task 

wer9 expe dec. Priority was based on three functions: (a) 

5-9 
how critical the tool was to early training estimation, (b)

the amount of time that would be expected to be saved by

automating the tool, and (c) the ease with which the tool

could be automated given the hardware/software configuration

selected for the SDT (Apple III/PASCAL).

Within the resource constraints of the ETES contract, it was

only possible to select one additional tool for automation,

the Automated Training and Planning and Scheduling Technique

(APST). An overview of this program is provided in Appendix

C. However, procedures were also identified for using the

automated Resource and Cost Estimation Technique developed

as part of the ETES Training Estimation Aids/Procedures to

conduct sensitivity analysis.

5.2.4 Develop Evaluative Technology

Potential automated aids and procedures which were

identified in the previous step were developed. A

description of these aids and procedures is provided in

Section 1.5.3.

5-10

~,
SECTION 6 -TASK 5: DEVELOP USER'S GUIDE

During this task, user's guides were developed for the three

ETES components (SDT, Training Estimation Aids and

Procedures, and Evaluative Technology) and a final report

was constructed to describe the research conducted during

the ETES study.

Table 6-1 summarizes the documents developed during this

task.

To facilitate independent application of the four automated

ETES tools (SDT, Media Selection Program, Automated Resource

and Cost Estimation Technique, and Automated Training

Planning and Scheduling Technique), separate users' guides

were developed for each of these tools. In addition the

ETES manual procedures were consolidated in a single user's

guide (see Table 6-1). Each of users's guides was designed

to provide detailed step-by-step procedures for using the

ETES tools.

6- '

A ' A
Table 6-1 ETES Documentation

0 Final Report (150 pages)

* ETES Procedural Guide (300 pages)

0 User's Guide: System Description Technology (300 pages)

0 User's Guide: Media Selection Program (200 pages)

9 User's Guide: Resource and Cost Estimation Technique (100 pages)

* User's Guide: Automated Training Planning and Scheduling Technique

(100 pages)

0 ETES Software Documentation (300 pages)

4!

22

=p

6-2
,a.

SECTION 7- POTENTIAL IMPROVEMENTS TO CURRENT

ETES CONCEPT

The ETES development study was designed to provide an

initial capability for developing early estimates of

training requirements. As Sections 3 thru 6 have indicated,

resource constraints prohibited the full development of

tools in several key areas.

This section briefly describes the areas in which the

current ETES can be improved to (1) provide more accurate

estimates of training requirements or (2) provide estimates

of training requirements in a more timely and cost-effective

fashion.

Table 7-1 summarizes the major improvements which may be

made to each component of ETES. A more detailed description

of these improvements is provided in the sections which

follow.

7.1 IMPROVEMENTS TO SYSTEM DESCRIPTION TECHNOLOGY

There are four general areas in which the System Description

Technology could be improved. First, the mainframe

functions of the SDT should be expanded and refined to

provide a full capability for distributed processing of

data. These mainframe functions include techniques for (1)

storing a centralized copy of the SDT, (2) transmitting and

communicating data to various users, (3) providing security

for data base access, and (4) keeping an audit trail of

7-i
41- ta4 04)-4 2

" I3 C '-

0 4-) 2:0) 44 0 - id to c 0 (- U)- -4. 04 M

1 0 4C -- c a m a 0 w -

0 0 0 -. " c. 1- W U U, 4-4'

0 .. (n .4. E0 : . 0 0. 2

0 w. wr m, En A 1

m4- V4 . W4U -C 4 IC4) - 04 4)0Q

00 4J 1W 0Q 1 WM 0 0 Er (a fa ) . O W c a 4 Z 1 -. 0EW 0 0" O-' 1 0 U a .-.' E 4.) M 0 C D V"

- a -6 00W a WIC 4-U >c 2

W- -1 W0 C=- a .1 o I C ( -n -'Cz 'a VU)'. 0 30 n 0 0 c u 0

zC > C- WIC.C U 4-1 W 04D - U)

W4 'o c 0 WJ Z w )4/2

E- m r_ ( ) f -4 C C u > a) E - 4 ) z ) u) OUC, 4. VU) 0- -a404WU) 0 u)

0 a) 4j 0. W C) .

0 O 0" 0 0 u - FE 3 0C > WE- u-lW

414 ' 41 W 41 4 1 Q-i ~41 4- a U1

SW W6 .00 a= 'M ) 0- w. m m M a)

M 0 m .o 4 Uo w 0 C w E 0 0 0 -OC -C 4J O 0'1 0 0

uE-: C.-- w (0C -- U)Q)W) -C 0)0 Wa) -4 (-'d ) Q

• > a) > W V 4& -4(a4Q j 4 > ) w > (a

0-CC 0 w-- 0- 2 '-W----'-. 'CC, 0.WC 0-

a a) 0 0 0 0 C 0- 0 WV aa QE)U CC CO WO U- E

C

4. 0 >z S - ~ 

Zl Q) . V 

W- Q) Z

C CU) Wn 0 -4 0

a 1) a C -4 w 4 a.)

.W .Q . *. E 4-' IX V)

4 -u W ~ C U Cn w' 0Wo, >0 , ( 1 W - 0 f U)

c 4 4CC (0~- U) E- 4JE m 3-V ) U ( W- a-"- w ' .-

0' W 4- C- 0 0 Q) W -0' ( =' < 4-' 0 " F-a 0. -qC 4 - 0'C ( (3 U -= =

F- 0 0 0 --m'a w~ m' mA.

E- r, C V- 4 4.J 0) ) .Z C Z C

0 ~ ~ ~ ~ 0 U .0 V) 1-' 0W 0 C- )'

Ua 01 4.) 2 a) 00 EW a- - W

c ) -- 20 J) u- (3) .0~~ C . 4)C J 4J 4) -' '-F

WW Ln UC E- 0 u U) 4J 4-4

(1 W- 0 m~ 0-. E-. CL Q WU () )a 2 U) 00 m 0 0 V .0 <))C~ 0C 0 U CL 0 4- W

'-CL 4-' .- U) T O V CC > r)WWWW

U -C xC T.- c- 1 ) WV 2 -0C).- CCUCCW ~~CL 0- O 0

C C2-

".,........ -2--.:2.2--2'--.:'-:.r." -' - .V--.-::.'i:.-."-,.:-" VCC ;'-.4::-- - CC:-, i: ,:.- . ..- W.,.'. :,.; .. :-.: -
system changes. The current version of ETES only provides

relatively simple capabilities in each of these areas. More

sophisticated capabilities were not developed because it was

not possible to identify the "mainframe" computer on which

the SDT would be implemented. Since the development of the

SDT, however, the Army Research Institute (ARI) has selected

the VAX 11/780 as the mainframe comnputer on which the SDT

and other components of the ETES will be implemented. with

the identification of the mainframe, more sophisticated

software can be developed for the SDT mainframe functions.

Second, the current ETES only has limited capabilities for

searching and sorting data. To better meet the unique

demands of individual users, a more flexible searching and

sorting capability should be developed.

Third, more ccznprehensive graphics capabilities must be

added to the SDT. The current SOT can produce reports that

contain the the data elements needed in many key acquisition

documents. However, it can only output that data in formats

that "look like" the acquisition document.

Fourth, the SOT data elements should be expanded to meet the

needs of any addi tional tools which are developed.

Currently, for example, as part of the MIST project, the

prototype SOT is being expanded to handle the data required

to support manpower and personnel estimation tools.

7.2 IMPROVEMENTS TO TRAINING ESTIMATION AIDS/PROCEDUJRES

There are four general areas in 'which the ETES Training

Estimation Aids,/Procedures can be im pr ove d. First,

73

..... "e
procedures must be developed for estimating collective

tasks, collective training programs, collective training

resources, collective training costs, and collective

training effectiveness. Second, procedures must be

developed for estimating training programs, resources and

costs for individual unit training. These two improvements

will become increasingly important given the current trend

toward shifting more and more training to the unit. Third,

procedures must be developed for developing more detailed

and more accurate measures of training effectiveness. The

current ETES procedures only provide crude estimates of

training effectiveness. While appropriate for the very

earliest phases of the acquisition process, these procedures

do not provide the type of detailed data which is needed for

the refined training tradeoff analyses which occur in the

later phases of the acquisition process.

Fourth, resource constraints prohibited the development of

automated aids for several key functions. Automated aids

can assist the training analyst in dealing with the frequent

design changes which occur during the earliest phases of the

acquisition process. Some of the current TEAP functions

which are most amenable to automation are:

3.1 Select Tools for Training

3.4 Assign Tasks to Training Settings

4.2 Determine Number of Students to Be Trained

4.4 Determine Facilities Requirements

4.6 Determine Other Training Resources

5.2 Aggregate Course Costs

7-
.

'p

7.3 IMPROVEMENTS TO EVALUATIVE TECHNOLOGY

These are four general areas where the Evaluative Technology

may be improved. First, the current Evaluative Technology

only employs some rather simplified techniques for

aggregating individual training figures of merit. While

simple to use and appropriate for the very earliest phases

of the acquisition process, these techniques do not have the

accuracy needed for the detailed tradeoff and sensitivity

analyses which occur in later phases of the acquisition

process. Second, a more sophisticated tool must be

developed for identifying the causes of training problems.

One possible tool for performing this complex function is a

computer simulation model. As noted in Section 6,

development of such a model will not be easy since most

simulation models have extensive input data requirements. An

alternative approach is to use an artificial intelligence

technique to build an "expert system" for diagnosing

training problems similar to the MYCIN system which was

developed to diagnose medical problems (see Winston, 1977

for an overview of MYCIN).

Third, an automated worksheet was developed for estimating

selected training resources and course costs. This

worksheet makes it very easy to change data and immediately

assess impacts. The automated worksheet concept must be

expanded to cover other ETES functions and individual

automated aids must be linked together to provide a more

systematic capability for assessing impacts.

Fourth, procedures must be developed for interf3cing 0TES

with techniques which are used in the later phases of the

acquisition process such as instructional system developnent

(TSD).

7-5
7.4 GENERAL AREAS OF IMPROVEMENTS

* There are three general areas which, although outside the

*scope of the current ETES concept, coul1d s ig n if icantly

improve the use of ETES products. First, to facilitate the

transmission and carnmunication of data, the SDT data base

* management system must be incorporated into a Management

* Information System for Army manpower, personnel and training

analysts (this will in fact be done in later phases of the

MIST project).

* Second, a cainprehensive tool must be developed for assessing

the integrated impacts of manpower, personnel, and training

requirements and for conducting integrated tradeoffs among

* these three closely linked systom elements.

Third, a systematic tool must be developed for estimating

the impact of the alternat ive manpower, personnel, and

*training concepts on overall system performance. To be

useful, this tool must not have extensive input data

requirements. Otherwise, it will never be used during the

earliest phases of the acquisition process, when most major

* MPIT and system design decisions are made.

7-6
APPENDIX A: OVERVIEW OF MEDIA SELECTION PROGRAM

This appendix provides a brief overview of the Media

Selection Program. A more detailed description is provided

in the User's Guide: Media Selection Program. The Media

Selection Program is an automated tool for (1) assigning

tasks to media and (2) calculating the efficiency and

effectiveness of various task-media combinations. The Media

Selection Program allows users to assign tasks to media in a

manner that maximizes overall efficiency, maximizes overall

"effectiveness," or minimizes overall cost.' In addition,

it allows users to assign tasks to media in a manner that

optimizes various combinations of these variables, including

an overall "utility" measure which combines either cost and

efficiency or cost and effectiveness.

Efficiency, in the Media Selection Program, is determined by

comparing the stimulus, response, and feedback

characteristic of the tasks to the stimulus, response, and

feedback characteristics of potential media. More

specifically, a score is calculated which describes the

match between media and tasks on these characteristics.

Etficiency for each task-media combination is calculated by

dividing this score by the maximum match that may be

The program uses a relative cost and not actual cost to °"

measure the potential cost requirements of m edia. In

vidii i ;n, the pro r im does not use I measure )L

"f fe t i venes s" tha-t fits the most common usage of that

tr. R tner effect i eness is actua 1 1y efficiency we ignted

y Ksk r1 1ci I ty.

. ~.. .
achieved for the task. Total efficiency for a set of tasks

is the sum of the efficiency score for individual tasks. 2

"Effectiveness" is calculated by weighting the efficiency of

each task by a task criticality score. The task criticality

score is a user-defined weighted combination of the eight

task factors typically used in selecting tasks for training.

These eight factors are task frequency, percent members

performing, percent time performing, task delay tolerance,

consequences of inadequate performance, task learning

difficulty, probability of deficient performance, and time

between entry and performance.

A matrix of relative cost values is stored in the program

for each major media category. In addition, a built-in set

of algorithms is used to produce the utility measure which

combines cost with either effectiveness or efficiency.

PROCEDURAL OVERVIEW

The Media Selection Program is an interactive menu-driven

system. This means that users do not have to know or use a

computer language to run the program. Instead, they can run

through the program by selecting options from a series ot

menus.

2 This concept f me(Ia t I ,iency was org ni ]

developed by Jor rensen 1 1 II 4nd was Incorporated to

Traininj Etfti c iency Es t irTa t i rn t E M d descr ir e "

Jor-ensen, Kubul , ,ind ztlas < [ .

..........................--..
An overview of the procedures for using the Media Selection

Program is provided in Figure A-i. To begin the procedures,

users must rate each task on (1) the psychological variables

to be used to assess the match between tasks and media, and

(2) Khe variables to be used to assess task criticality (the

latter is only necessary if effectiveness is being

calculated). These scores must be then entered into the

ETES data base management system, the System Description

rechnology. When this is completed, users must enter the

Applications Program mode in the System Description

Technology, and select the Media Selection Program. Once in

the Media Selection Program, users must then select criteria

to be used in making media assignments. Seven options for

selecting criteria are provided: (1) efficiency, (2)

"effectiveness" (3) relative cost, (4) cost and efficiency,

(5) cost a ,d "effectiveness," (6) a utility measure,

combining efficiency and cost, and (7) a utility measure

combining "effectiveness" and cost.

Once the criteria have been selected, users must select the

tasks to be included in the analysis (only tasks already in

tne SDT may be selected). Typically, the tasks for single

course module will be selected for each analysis. With the

analysis criteria identified, the psychological variables to

he used in calcul-iting the match between tasks and media

must be selected. Ufsers may select from 12 variables

3ssessing stirmuls characteristics, six viri-ibles 3ssessingL

response characteristics anA t)ur variables assessing

feedback characteristics.3

3H

These psycholgic ,] vari s . r.1 te kien -i:ectI f r )III

noe TFEEIM ',cIe ( Jor:eson, Kuo u a, I At t V.s; 

A-- I
INPUT DATA

ON4 PSYCHOLOGICAL

VARIABLES iSOT)

SELECT

CRITERIA

SELECT

TASKS

SELECT

pSYCHOLOGICAL

VARIABLES

SELECT/WEIGHT

TASK

CRITICALITY~

FACTORS

COMPUTE] SCORES OF DISPLAY MqEDIA ON

LOGICAL

L VARIA1BLES

NEIGHT

CRITERIA

SELECT

OBJECTIVES/

CONSTRAlINTS

ASSIGN I DISPLAY 04 ;5UN

RELAT~vE ASKS -0 -IESUL-5 ANAL iSIS

j ST )F I MEDIA

ATr:GCR I ESI L - -

Figure A-i Overview of Media Selection Program
If the user has selected a set of criteria involving

effectiveness (effectiveness, cost and effectiveness, or

utility with effectiveness), the weights for the component

criticality variables must be entered so that a composite

task criticality score can be computed. Eight variables may

be used in the calculation of task criticality: (task

frequency, percent member performing, percent time

performing, task delay tolerance, consequences of inadequate

performance, task learning difficulty, probability of

deficient performance and time between entry and

performance)

At this point, the program has all of the information needed

to calculate efficiency and/or effectiveness. It will use

this information to calculate the relevant measure and it

will display the results.

If the user has selected one of the two utility measures

(combining either cost and efficiency or cost and

effectiveness) the user will then be required to enter the

weights to be used in the computation of utility. Following

this, the user must select the objectives and constraints to

be used in assigning tasks to media. A listing of the

possible combinations of objectives and constraints is

displayed in Table A-i. Once this information has been put

into the computer, the program will optimally assign the

tasks to training media. For instance, if the user selected

"maximum effectiveness" as an objective and "minimize cost"

as a constraint, the program would determine the assignment

of tasks to media which gives the highest overall score on

-effectiveness and still remains under a user-specified level 

'ft overall cost. Once the initial assignments have been

examined, the user can examine the effects of several

A ,.

-4 -A
x

4-T' "IT T1.O x xXx

x _ _ __ _ x __ __ __ _ _ _ x)

1507_ __ _ _ _ _ _ _

3S saoj~a; 0xU2 X 

saL u~atol;;s __ __ ____x___ ___ ___

3S Da;ga al,?AT' 

v4 131 0) 0) a) 0) c3 1)

r~1 > 0) C3 .43 ra)a 13 

- - -. 3 1 ~ 4 - 40343 - -- : a) U) -- Ld3-.4 0 1) 1" Lo3 L13 n -j U )U '3- 3 .1 -. 3 -130 4 =3 .131 C . .31 -131 -

m2 .1 w-i 11 Iz a 11 n ) -, 12 12 12 - .1. 13 2 C 0 ' cn C- - V - uI E3 V3 -

a3~. a)i4 v3 1301 3 1301 13)3V

a-0F-a U): >13 Z,~ 'o. E2 . 112 7 201 120- -n *-3x1 13 fl a) 1 - ' ) - ~ .3.33 .-34

- ) I J

13j1 0> 13 7) A 13> 31> 20 0> 0> .

131 ) 2 1 12 1 T3N I 'I

13~. -~ '13'3- 3' 1 13 -4 3 1

z333 -11 .~~ -233 44 ~ 444 ~ 4 31

71 3 V 1' 1- 1
7Isoo pUv

SSaUaAT0a;;2

- I

cr 0 V) V c

2) Q) -JO

2) -) a

~C) 0 0-

Q ) L w j

Cn V)C m) 

w) >, La- (V u 0 a 

..

-~ 20U) J)
alternatives including changes in (1) objective, (2)

constraint, (3) criteria, (4) task criticality variables,

(5) psychological variables, (6) task criticality variable

weights, and/or (7) utility weights.

After users have explored these alternatives, the task-media

assignments can be finalized and entered into the SDT.

A-8

A -"

- _ ; , t ; i _ " d " " , n - . • :_ - - . . -. . ... .
APPENDIX B: OVERVIEW OF AUTOMATED RESOURCE AND COST

ESTIMATION TECHNIQUE

This appendix provides detailed instructions on how to use

the Automated Resource and Cost Estimation Technique

(RCET). A more detailed description of RCET is contained in

the User's Guide: Resource and Cost Estimation Technique.

The purpose of RCET is to provide Army Training analysts

with an automated tool for estimating instructor

requirements and institutional training costs during the

earliest phases of the acquisition process.

The Automated Resource and Cost Estimation Technique (RCET)

is designed to use input data from the ETES data base

management system, the System Description Technology (SDT).

Actual calculation of instructors and institutional training

course cost in RCET is accomplished by using the VISICALC

automated worksheet software developed by Visicorp. The

VISICALC worksheet is also used to conduct sensitivity

analyses of key parameters.

B.1 CONCEPTUAL OVERVIEW

The Resource and Cost Estimation Technique has three

components:

(1) SDT Interface Software - this software is used to select

and remove data from the SDT and to format the data for use

in the VISICALC program. In addition, it is used to copy

the results of the VISICALC program back into the SDT.

B-o,

.-'7

4 44 4 . - 4 4 4

*. . . . . . .4 ' 4 4 - "t
(2) Tailored VISICALC Worksheet - this worksheet contains

the equations for determining number of instructors and

course costs. In addition, it contains all of the commands

needed to load and unload the SDT input file, and to conduct

sensitivity analyses. This tailored worksheet saves the

user from the somewhat tedious process of setting up a

VISICALC worksheet and command structure.

(3) Manual Procedures - these procedures describe how to

develop input data and how to use the SDT interface software

and the tailored VISICALC worksheet.

There are two major products of RCET: (1) a listing of the

number of instructors required in the course and (2) a

listing of projected costs for the course. An example of

the cost elements estimated by RCET is listed in Table B1. These are the same cost elements used in the Cost

Analysis Program of the Army TRADOC Resource Management

(ATRM) system.

B.1.1 Calculation of Course Costs

Costs for a new course are estimated by identifying a

comparable existing course, obtaining cost data from this

course data to reflect the differences in key resource

requirements (for example, number of students and number of

instructors) between the comparable course and the new

course.

This procedure provides estimates of course costs that are

(1) empirically based and (2) suitable for the types of higJh

level analyses which are conducted during the early phass

rof the acquisition process.

B-2

.. "--. .. .., .. -. -.-.. , ., .... - . . * < -... *. , . . .. . ,.. . . . , -.. ,.-. . .,' ".. " .,
- - -- - " - r- 7 -

tj L)

06 P

- . CL

ma z j J Iz a)l

-5 3
B.1.2 Calculation of Number of Instructors

The number of instructors required in a course is calculated

by an automated version of the algorithm listed in the

B.2 PROCEDURAL OVERVIEW

An overview of the procedures in the Resource and Cost

Estimation Technique is provided in Figure B-i.

The f irst step in the application of RCET is the

identification of the "reference"' course or the comparable

existing cousre, which most closely resembles the task and

target population requirements of the new course.

Procedures for identifying a reference course are contained

in the ETES User Guide Manual Procedures. Once the reference

course has been identified, cost data for this course is

obtained from the Cost Analysis Program CMOS Training Cost)

and entered into the SDT.

Reference course information is also used in the

construction of the quasi-program of instruction (QPOI) for

the new course. Included in the QPUI is a description of

the methods to be used in each module in the course, and the

student-instructor ratio and instructor contact hours

associated wirh each method. Procedures for constructing a

QPOI are contained in the ETES User's Guide. This same

.

Suide contains procedures for determining the number of

5'I

Estudetstio ehnine.iqaue prvie is cnritca facto in

thedetemiation ofth "rfn" course copaabs.U rte 

B- 4

exisingcouse, hichmos cloelyreseble thetas an

............................................ ... n.. .. th new5 core
IDENIFY ONSTUCTREFERENCE AND

4,

S.0

COPY SOT

DATA TO

LOAD

WORKSHEET AND

VISICALC

I NPUT FILES

1 SLECTED

RESOURCE

DATA

1 1 SOT PROCEDURE

fo~ Z ANUAL 'RCDR

EXAMINE

RESULTS VIICL OROCEDURE

r0 .0 ~ 107-.7

CONOUCT COP' RESULTS :READ SENSITIVIT'1 70 VISICALC qESU L-S

ANALYSES O UTPUT -ILE NTO SOT

Figure 3-1 Overview of Procedures in Resource and

Cost Estimation Technique RCET).
Once the QPOI has been instructed for the new course,

information from the QPOI on instructional methods, student/

instructor ratios, and contact hours must be entered into

the SDT. When this is completeu the the user must enter the

SDT, enter the Applications mode, select the Resource and

Cost Estimation Techniques (RCET), and copy the reference

cost data and new course QPOI data onto files which can be

read into the VISICALC program.

Once the VISICALC input files have been developed, the user

must remove the SDT software diskette, put in the VISICALC

software diskettes and enter the VISICALC program. Once into

the VISICALC software, a few simple commands may be used to

load the SDT input files and RCET worksheet into the

VISICALC. When this is completed, a small selected set of

resource data for the reference and new course must be

entered into the SDT. Course costs and instructor

requirements may then be calculated by executing a simple

command built into the RCET worksheet.

After examining the initial estimates of course costs and

instructor requirements, the user can then use a few

commands built into the RCET worksheet to conduct

sensitivity anilyses of key parameters. When these analyses

are complete, the final set of costs for the new course can

be copied onto a VISICALC output file. The user can then

exit the VISICALC software, enter the SDT software and copy

the output file into the SDT data base.

4.
APPENDIX C: OVERVIEW OF AUTOMATED TRAINING PLANNING

AND SCHEDULING TECHNIQUE

This appendix provides an overview of the Automated

Scheduling and Planning Techniques (APST). A more detailed

description of APST is provided in the User's Guide:

Automated Planning and Scheduling Technique. APST is

designed to assist training developers in describing and

monitoring the training development schedule for developing

Army weapon systems. APST is designed to be used with the

VisiSchedule software, which is an automated program for

describing, monitoring and reporting schedule information

and for conducting critical path analyses of schedule

events. A data input diskette, describing the events

required in trie Army's Individual and Collective Training

Plan (ICTP), is included in these techniques. This data

input diskette contains detailed information on the

sequential relationships among the events in the OICTP.

APST is designed to make it relatively easy for training

de:velopers to track and monitor the complex relationships

among the events in the training development schedule. In

addition, by providing an automated capability to monitor

tne training schedule, it should aid training developers in

responding quickly and efficiently to the frequent schedule

changes which occur during the development ot Army weapon

sys tem5.
C I CONCEPTUAL OVERVIEW

Construction of training development schedules for emerging

systems is a difficult task. over 100 developmental event:;

are listed in TRADOC Reg 351-9. Th e sequential

relationships among these events are complex and are not

described in any systematic and integrated manner in TRADOC

Reg 351-9.

Further, the training scheduling process, particularly

during the early phases of system development, is

characterized by trequent changes and updates. Determnination of the impact of these changes is a tedious and time

consuming process.

APST contains techniques for using automated VisiSchedule

software to track and monitor training development schedule.

kiusing Vis.-iSchedule, the training developer can (quickly

a nd efficiently respond to changes i n the training

development scheudle. Use of the VisiSchedule program is

facilitaced by the inclusion of an input data diskette which

(a) describes the events in the training development process

(as specified in TRADOC Reg 351-9, (2) describes th e

temporal,/sequential relationships among these events, a rid

(3) lists the expected duration of these events f -:r a

"typical" major Army weapon system. This data diskette

significantly reduces data input requiVoments. In additio)n,

it eliminates the need for an analysis of the complex

zequential relationships among training development event-,

wh ich -1r e either implicitly cr explicitly SpecIfI -d i n

TRADt )C 3351-9.

,..- %.- .A -A -,.
o Capabilities of VisiSchedule Software 

As applied to the 

VisiSchedule software 

following capabilities: 

training 

can be 

development 

viewed as 

process, 

providing 

the 

the 

(1) Allows users to systematically describe an 

integrated training development schedule including 

information on training development events, the 

sequential relationships among these events, the 

duration of these events, the manpower (by labor 

c3tegory) required to accomplish each event, and 

the costs (that is, salaries} of this manpower. 

( 2) Allows users to quickly determine the impact of 

changes to any of the above infor:ma t ion. 

(3) Allows users to identify the "critical path" in 

the training development schedule. A "critical" 

event is one whose delay would impact completion 

of the whole project. 

(4) Allows user to aggregate events to determine total 

manpower requirements (by paygrade or occupational 

specialty) and to determine total training 

d8velopment costs. 

C.2 PROCEDURAL OVERVIEW 

An overview of the procedures for using the automated 

planning and scheduling t&chniques is provided in Figure cl. 

C-3 
"a.

'3 PAG ~/I P

SIT-NTS

VI,.

ACOLtTIONML.

NUAIOII

*OOITIEOiAp

PiPJT'lOIP!MATION .. ,no

OA-~

"a-

. . . . . . . . . . . . . ..

p p~ . . -- . p . IP- p
The first procedure involves setting up the hardware and

so ftware needed to run the automated techniques. As part of

this procedure, the VisiSchedule software and the

accompanying input data diskette, describing the ICTP events

and their interrelationships are entered into the computer.

In the next three procedures (2.1 and 2.3), the data on the

input data diskette describing the ICTP events, their

interrelationships, and their durations is reviewed by the

user. If the user feels that the existing data is

acceptable and, thus, is an accurate description of his/her

training development schedule, the user can proceed directly

to the output report procedure (4.0). However, it is more

likely that the user will want to change the durations or

completion deadlines of some of the events; the events

themselves and their relationships are less likely to

require change.

If changes are required, these changes may be made using the

methods described in procedures 3.1 to 3.4. Procedure 3.4

allows users to enter and/or modify data on manpower

requirement and costs. Information on manpower requirements

must be entered by the user since this information is too

system specific to include in the input data diskette.

In procedure 4.0, the user may select from one of four

different output reports to describe the training

development schedule. After examining these outputs, the

user may wish to conduct tradeoff analyses or sensitivity

analyses of the schedule input variables. This can be

accomplished by changing the input parameters through

procedures 2.0 and 3.0.

C-5

% s R M
p

'ft

,.ft

THIS PAGE INTENTIONALLY LEFT BLANK

ft.

'ft

C-6

- . ft ~ - . - -. - ft~ft~%~%.::\~ *.
APPENDIX D: DESCRIPTION OF PROJECTS RELATED TO ETES

This appendix provides descriptions of the six ARI projects

which are directly related to ETES: (1) the Hardware

Procurement-Military Manpower (HARDMAN) Methodology, (2) the

Man Integrated System Technology (MIST), (3) the Training

Efficiency Estimation Model (TEEM), (4) the Training

Developer's Decision Aid (TDDA), (5) the Model for the

prediction of the effectiveness of Training Devices

(TRAINVICE), and (6) the Army Manpower and Personnel

Requirements Process (ARMPREP). More details on these six

projects are provided in the subsections which follow.

D.1 MILITARY MANPOWER VERSUS HARDWARE PROCUREMENT (HARDMAN)

The HARDMAN methodologies provide the Navy and the Army with

a systematic means of estimating human resource requirements

(manpower, personnel, and training) during the earliest

phases of the Weapon System Acquisition Process (WSAP). The

integrated models and data bases of the methodologies are

designed to (1) assess manpower, personnel, and training

requirements for proposed sytems; (2) determine the impacts

of manpower, personnel, and training requirements; and (3)

identify and evalute tradeoffs which would alleviate

unfavorable impacts. DRC has also extended the capability

of HARDMAN through increased automation and the development

of proprietary software packages. These advances have

significantly cut cost and response time and more closely

aligned the output products of HARDMAN with the information

needs of Program Managers, resource sponsors and their

staffs.

D-1

:: .: -.--. "--,"--, : - , ..- -'- . .- • • --- - . - - - - - -. . .*- - , - -. - .• -- * -- . . ., - ---- a. * ", a, , . - . ," , * ,- -- - ..-. ', . ,' '.-. , . - .. ,," .... * *.-, . .- .. ,. ', -- • , ,- - - -
In its present form, DRC's HARDMAN Methodology is composed

of six interrelated steps which can be iterated in a timely

fashion for the analysis of alternative design proposals. I

The first step is conerned with various aspects of data

collection, generation, formatting, and analysis. It also

includes the establishment of an automated audit trail or

program record of all input data, analyses and output

products. The final five steps involve data evaluation with

respect to acquisition program goals and constraints.

The establishment of the consolidated data base triggers the

methodology and ensures that all analyses pertaining to a

new weapon system use a common data source. Consequently,

consistent definitions and collection procedures are

employed in assembling data on operator and maintainer

functions, the weapon's operational mission/scenario and -*

maintenance concept, its cost factors, and its manpower and

training sytem support requirements. The proposed weapon

system's demand, in terms of manpower, personnel, and

training is determined in Steps 2 through 4 and compared to

the projected supply of such resources at system Initial

Operational Capability (IOC) in Step 5. Significant

shortfalls and equipment sources of high resource demand are

identified. In Step 6, the methodology is iterated to

examine alternative designs, training methods and media, and

total force tradeoffs. Thus the program manager can

participate in the design process with a heightened

awareness of the training and manpower demands of a new

weapon system, thereby ensuring its supportability as well

as its mission capability.

DRC's HARDMAN Methodology has been used on a wide range of

all three Military Services, including the Shipboard

Intermediate Range Combat (SIRCS); the ISD-41, a new class

of guided missile destroyer (DDG-51); the Submarine zkdvanced

D-2

................................... .*.. ..... 
Combat System (SUBACS) ; the Army's Divison Support Weapon

System (DSWS); the Corps Support Weapon System (CSWS) ; and

the Remotely Piloted Vehicle (RPV); and in modified form

with the Air Force's Combat Identification System (CIS).

These applications have fully exercised all of the analytic

capabilities of the methodology. Additionally, they have

demonstrated its capability to provide timely, accurate

support for program management and review. Specifically,

significant or problematic human resource demand generated

by equipment design has been targeted early in program

development; alternative system and subsystem configurations

have been evaluated through tradeoff studies; and the

methodology's consolidated data base, supported by carefully

documented resource design analyses, has been used to build

an audit trail of manpower, personnel, and training assessments and tradeoffs.

D.2 MAN INTEGRATED SYSTEMS TECHNOLOGY (MIST)

The United States Army Research Institute for the Behavioral

and Social Sciences (ARI) is developing a technology which

integrates manpower, personnel and training (MPT) considerations throughout the Weapon System Acquisition Process

(WSAP). Its goal is to ensure the effective planning for

and utilization of projected human resources for operational

readiness. The technical activity required to achieve this

objective has been organized into a major research and

development program entitled Man Integrated Systems

Technology (MIST). DRC has been selected as the contractor

for MIST which is an extensive (5 year, 32 man-years) multimillion dollar effort.

D-3
a

MIST will consist of the necessary technology and management

procedures to address the following considerations: (l) the

treatment of human resources as a performance and cost

factor during system concept formulation, (2) the planning

and forecasting of manpower information, (3) the parallel

development of associated training systems with weapon

systems, and finally (4) the specification of test and

evaluation issues to ensure human resource accountability,

and to support the ASARC/DSARC review at each of the major

development milestones. MIST in particular will integrate

and demonstrate the technical relationships among these

considerations, and provide the necessary methodology to

ensure their effective treatment during the design process

from a performance, manpower, training and cost point of

view. In addition, MIST will demonstrate how these

technical considerations are related and responsive to the

Army's Life Cycle System Management Model (LCSMM), and to

relevant Army regulations and responsible agencies. MIST

will further provide a comprehensive framework for the

assessment and integration of other relative technologies

and information as they are identified or developed.

When fully developed, MIST will serve as an integrating

mechanism providing efficient interfaces for information

exchange and feedback among concurrent activities and/or

processes involving industry and government, system design

engineers and support planners, Army acquisition managers,

and the Army's oversight bureaucracies in the Department of

Defense and the Congress. Hence, MIST will provide the

design and implementation of more comprehensive and

responsive data bases and technical/management procedures

employed in prime and support system development,

acquisition and operation.

0-4

* * . * . . - . ***\ . N * '... .. ~ . .\

.. - i- .-.- "- -" "" "" ' ** -" '- .. . .. . .-,-,.,- - ,.-.*',,.. . .- "- ", v ." - -'. - . "- -, -"- -- ," -, "., ,- ,
When completed, MIST will be composed of three principal

components:

o Analytical Tool Designed to Support Program

Managers, TRADOC System Managers, Acquisition

Managers (e.g., Weapon System Manager (WSM) in

DARCOM), and their staffs/supporting agencies; I-..

o A system-specific data base to provide a single

point source for all man-machine data (input to

the tools, output from them and decision support

information for program documentation and reviews)

collected for an emerging system. %

o A management information system to link MIST users

and the data base in an effective communication

network within the framework of the Life Cycle

System Management Model.

Hence, MIST will provide an important new capability within

the Army. This capability is embodied in a cost-and timeefficient system for human resource requirements analysis

early in weapon system development. MIST will capture the

most recent advances in both man-machine technology and

automated data processing to provide a system which is userfriendly at the operational level and responsive to the

analysis and information requirements of users at several

levels within the chain of command. Finally MIST will "fit

into the real world" by providing effective interfaces to

existing management information systems and data bases for

logistic support analysis, training system development and

personnel planning in DARCOM, TRADOC and the Army Staff.

D-54

S.°

Ii -

Ii
D.3 TRAINING EFFICIENCY ESTIMATION MODEL (TEEM)

TEEM is an interactive computer-based aid designed to assist

training developers in early evaluation and generation of

training program components. At the foundation of TEEM is a

set of psychological variables which reflect the training

information contained in a system task description. This is

in contrast to the normal textual form in which task

descriptions are usually developed. A representation by

variables was chosen because early task descriptions are

often not in standard formats, may change quickly, and may

require elaboration by design engineers or other subject

matter experts. A predetermined set of standard psychological variables provided a starting point which could be

depended upon for input into the fixed operational code of a

computer program.

Initial task groups corresponding to procedures that should

be trained together are generated through an algorithm that

clusters tasks based upon internal similarity along the

psychological variable dimensions. After clustering is

complete, a task group has maximum difference between task

clusters. The clusters serve as an automated estimate for

use during the initial formulation of a training program.

Task groups are then modified, if needed, based on the

expertise of the analyst or outside mitigating factors such

as management and cost constraints that would normally not

be included in the psychological variables.

Selection of concrete, costable components such as media and

method of training are based upon the similarity of a group

of task variable requirements to the ability of a media or

method to manipulate psychological variables during

D-6

. S,.
-N L- 6- -, -- lip 1:1 TO. - M

training. Selection is accomplished by representing each

training program component by a description in the same set

of psychological variables. A task is also coded as a .

vector of variables with an assocated variable value for

each dimension representing applicability or nonapplicability. Selection of a program component is

performed by automatically matching each task to each

potential component and selecting the component with the

higher match of critical variables.

If an already existing training component is being

evaluated, the matching process is utilized in reverse to

generate a figure of aggreement between the theoretical

maximum possible under the task and hardware conditions and

the actual match under a particular alternative. The

normalized numeric value of this match is called the

efficiency ratio.

After efficiency ratios have been generated for potential

training programs a final selection for the best training

approach is made based on pair values of gross dollar costs

and their efficiency ratios. Together the values represent

an early form of cost-effectiveness ratio for each approach.

The best candidate programs are selected for detailed cost

analysis through a much more complex accounting program and

given increased program specification.

TEEM was constructed with the realities of the early

training situation in mind. It does not assume extensive

and accurate information. It is "user friendly" with stepby-step guidance to individuals unfamiliar with microcomputers. It is designed to accommodate local school needs

and training development procedures without major program

D-7

............................ S. . . . . . - . . .. . . . *. 5 5 .. . . S
changes. It is also flexible enough to be quickly modified

based upon improvements in psychological research and field

experience.

D.4 TRAINING DEVELOPER'S DECISION AID (TDDA)*

The Training Developer's Decision Aid (TDDA) was designed to

assist training development specialists in applying the

Instructional Systems Design (ISD) process. The ISD process

is a comprehensive technique for the systems a -ach to

development training. Applied Science Associ ,'s Inc.

(ASA), working in collaboration with the Arr Research

Institute Field Unit at Fort Bliss, Texas, has d eloped,

tested, revised, computerized, and conducted tryouts of the

model at five Army schools.

The TDDA is intended to be an aid to training developers who

need technical assistance in designing or redesigning

training and to streamline the process of making training

decisions. The TDDA is procedural in nature; it leads

training developers through a series of procedures which

specify: (1) what decisions are to be made, (2) a rational

order for decisions, and (3) the nature and sources of

information required to make valid training decisions. The

model also serves to organize and retain information that is

gathered and processed during the course development

process.

Derived trom Frederickson, et al 1 i I

D- 8
-I JW WN __ V -r

*ihe TDDS model can be used with partial or complete task

lists or performance objectives. It is not necessary to

redo the job analysis in order to use the TDDA process.

This reflects a training development philosophy that a

thorough analysis of the job early in the training

development process is essential for making valid training

development decisions.

The Job Analysis Module requires the greatest input form

subject matter experts (SMEs). Those who hold the MOS being

analyzed and selected to serve as SMEs are expected to be

expert job performers with at least eighteen months of

recent on-the-job experience at an operational site. This

is an essential requirement for the successful application

of the model. The model is sensitive to quality of the

input data and information, since all three major decision

(what, where and how to train) are directly tied to input.

There are two separate parts to the SME input in the Job

Analysis Module. First, the initial task list is generated

by exercising function analyzers. Then, the task list is

verified as correct and complete. This should be carried

out by a second set of SMEs. At the time the tasks are

verified, the remainder of the input information is obtained

from the SMEs. The output of this module provides the

answer to the question, "What should be trained to prepare

someone to perform in their MOS?"

Th i- Functional Learning Requirements Module requires inputs

trom two sources, MOS SMEs and course development instructor

Cersonnel ,n order to make decisions of how to train. The

inf-rmation generated in this module is used by the course

developer to specify the functional characteristics of tne

D-9

' .""' "". " -/ ' "" "--" ' - -"-' •" ' -" ". "'" , "- ,'- ,- .- -- .. ' . - .' '" ". , . "- - '""- .. - , ' .- "" '" 
training program. The emphases in specifying these

characteristics are on how best to communicate the learning

objectives, job context and consequences of task

performance, and on providing for efficient practice for

acquiring special job skills.

Development of the course requires that the relationships

between tasks be determined. The last SME input provides

the information for describing these relationships. Tasks

are related in three ways. First, the same skills may be

required in the performance of different tasks. Knowing

which skills are redundant allows for more efficient

scheduling and sequencing of instructional elements.

Second, some tasks must be performed on the job before other

tasks can be performed. For example, a piece of equipment

cannot be repaired before the cause of the malfunction is

isolated. Or, an intelligence report cannot be prepared

before the intelligence information and data are analyzed.

The output of these tasks are inputs to the subsequent

tasks. The third relationship is merely chronological. One

task must be performed prior to a second one but there is

not direct output-input dependency. For example, a weapon

system has to be emplaced (a collective task) and made ready

for action, before it can be used to engage hostile targets,

which may or may not occur.

The dependency relationship information is provided by SMEs

and is then used to build a task dependency hierarchy. The

product is considered to be a general course structure upon

which the course map can be built. The task dependency

information is generated in the Functional Learning

Requirements Module, but is analyzed in the Structure

Designation Module.

D- lo
I

Information and data from several earlier inputs are used in

the last module to .,ke decisions as to where the training

should be conducted. The task criticality data is one of

the primary inputs to this decision. Additionally, the

instructional setting characteristics and school context

information are used in reaching these conclusions.

The TDDS model then is used to make three major decisions:

what to train, how to train, and where to train. Some

decisions are made algorithmically and others are

heuristically determined.

D.5 A MODEL FOR THE PREDICTION OF THE EFFECTIVENESS OF ,

TRAINING DEVICES

TRAINVICE is a methodology for the systematic assessment of

the characteristics of training devices under development.

The model is based on the assumption that certain attributes

to be assessed in the training situation will lead to

transfer of training to the operational situation. Therefore, the higher the rating on the assessment factors, the

higher the transfer that will take place and the more

effective the device. The model provides a framework for

the making of these judgments. The three variables entering

into the assessment are: (1) the transfer potential of the

device, (2) the learning deficit to oe overcome and (3)

instructional effectiveness. As with any model, its

effectiveness depends on the adequacy of the input data.

Inputs into the model consists of descriptions of tasks and

* Derived From Narva (1979).

D-11
subtasks represented in the operational situation, as

circumscribed by the training objective, and those

represented in the training device. The controls and

displays and their functions for both situations are listed.

In addition, the skills and knowledges involved in each

subtask in the operational situation are formulated for use

in the model. Using these inputs, judgments are made using

rating scales. The subtasks in the two situations,

operational and training, are compared to ascertain if

provision is made for representation of the subtasks in the

training device in the commonality analysis. Next, the

displays and the controls for both situations are compared

on physical and functional similarity. The more similar the

display or control in the training device is to the

operational situation, the higher the score. This is based

on the premise that the greater the physical or functional

similarity, the greater the transfer of training that will

result. Physical similarity refers to the appearance and

physical aspects of the displays and controls involved;

i.e., their "fidelity"; functional similarity involved in

the operation of control, in information processing terms.

The learning deficit analysis is based upon (1) the assessment of the level of proficiency in each skill or knowledge

for the students upon entering the training situation, (2)

the desired level of proficiency in each skill or knowledge

for the students upon leaving the training situation, and

(3) tne difficulty (in terms of training time) of training

in the skills or knowledges involved in a subtask. This

analysis yields a weighted learning deficit for each

subtask. The judgments concerning the level of each skill

or knowledge are made using scales adapted from Demaree

(1961). The last analysis involved in the TRAINVICE model

is an assessment of how well the training device adhers to

D-12
- -- -u 

"good" training techniques. In order to perform this

analysis, each of the subtasks is cast into one or more

categories of behavior. These categories are those of

Braby, et al (1972), which are derived for an earlier

behavioral categorization by Willis and Peterson (1961).

For each of the behavioral categories represented in the

subtask, a list of guidelines, also those of Braby, et al.

(1972), are consulted and judgments made of the degree to

which the guidelines are followed, or not followed, relative

to the manner in which the subtask is represented in the

training device. The guidelines are broken up into those

dealing with the stimulus, response, and feedback aspects of

the training situation. For each subtask, the lowest

obtained score on each of the three aspects is used to

derive an average training technique score. All of the

preceding rating, are then fed into an equation to formulate

an index of prediction of training effectiveness, ranging

from 0 to 1. This equation is as follows:

(Ci x Si x Ti x Di)

Di

where C is task commonality, S similarity, T training

techniques, and D the training deficit scores for each

subtask. The equation was derived from a transfer of

training equation of Gagne, Foster and Crowley (1943), which

was for use with empirical data, while the TRAINVICE extrapolation deals with judgments made concerning aspects of a

device assumed to bring about subsequent transfer of

t ra in i ng.

D-13

7...
A validation study has been performed on the model,

utilizing data obtained during the course of two field

studies as criteria against which to compare the predictions

derived from use of the model (Wheaton, et al., 1976). The

devices were tank gunnery trainers involved with burst-ontarget techniques and tracking with the main gun of the

M60Al tank. In each case, the prediction of no differences

between the training devices involved was found to be

consistent with the equivalence in transfer actually found

in utilization of the various devices. This was felt to be

a promising but not definitive finding.

In order to obtain additional validation data on the model,

and also to obtain experience in utilization of the model to

determine if there were aspects that might be changed in

order to enhance the practicality of utilization of the

model, the Army Research Institute personnel applied the

model to two maintenance trainers undergoing evaluation at

the Army Ordnance Center and School. This afforded the

opportunity to obtain data within a different context than

that dealt with by gunnery trainers.

These trainers were concerned with automative troubleshooting. No difference in training effectiveness was

predicted for the two trainers, which agreed with the

results of the empirical evaluation. Various aspects of the

model which caused difficulty in its utilization were noted

and influenced the development of the modified version . In

addition, ARI conducted a three-day workshop, in which the

developers of the original model and individuals who had

utilized the model or had an interest in its utilization

participated; this furnished further ideas for possible

modi f ica t ion.

D-14

m

.' " ":''',,-'' - ' "]''' ,'' ..-i -," '', '" .'., ' -- , "-. ... . . .-. .-.. -.-.. '- " : ... .-.. ' , ,-.. -
D. 6 ARMY MANPOWER AND PERSONNEL REQUIREMENTS PROCESS

(ARMPREP)

The Systems Manning Technical Area of the Systems Research

Laboratory, US Army Research Institute, is promoting a

research thrust known as Manned System Integration. One

major component of this program is the development of

improved procedures for the manpower planning process. The

major objective of this proposed research is the development

of new, innovative and effective techniques to support the

determination of manpower and personnel requirements.

ARMPREP will include: methodologies to accurately estimate

manpower quantity and skill level of personnel through

derivation of behavioral requirements and subsequent

translation into Military Occupational Specialty (MOS) and

other relationships; techniques to aggregate manpower demand

of new systems for comparison with available supply;

requirements and attributes of a manpower requirements

management information system; and finally, the specification for a computer interactive system. ARMPREP is intended

to provide tools for potential integration into the current

manpower planning process by cognizant Army agencies. In

essence, ARMPREP has four principal components each with its

own identifiable research products. These four components

are:

o Manpower and Personnel Requirements Determination

Methodologies (MANPERS).

o Aggregation Procedures of Manpower Demand (TOTAL

MANPERS).

D-15.

"- '. " : / " " • ": , - =: " - . . / " " ". _ . '-. , " _ . ._' V .', ,: ''_ ." ' '- -", ." "
o Requirements for a Manpower Requirements

Management Information System (MARMIS).

o Computer Interactive System for determination of

Manpower and Personnel Requirements (AUTO

MANPERS).

MANPERS

The MANPERS component will provide the techniques and

methodologies to formally quantify manpower and personnel

requirements. These formal methodologies will aid in the

timely and accurate determination of manpower and personnel

requirements. In addition, these methodologies are intended

to standardize the manpower and personnel requirements

determination process to provide information of a consistent

nature of use to Army personnel subsystem developers.

Specific aspects of the MANPERS component will consist of

the following attributes:

(a) The development of a taxonomic model to derive

behavioral requirements based upon new system Task

Descriptive Data (TDD) and translation of these

behavioral requirements into MOS and associated

decisions. The model will define the requisite

data to be included. This input data will be

obtained from sources such as LSA, Task and Skill

Analyses (TASA), system requirements and doctrine

documents, engineering design data and training

plans. The model will specify the level of

specificity of data congruent with each phase of

the LCSMM. In addition, the model will identify

the process by which the behavioral requirements

D-16

• . I

...................
are derived (i.e., the taxonomy) and the means by

which the level of data specifity is embraced by

the taxonomy. Finally, the MANPERS methodologies

will guide the selection of personnel for a given

job in a new system based upon the behavior (or

performance) expected of him, as opposed to the

current method which involves the preselection of

MOS category, previous to and/or without rigorous

documentation to support such a choice.

(b) Integration with events and activities of the

LCSMM, ILS (LSA) and ISD. The MANPERS process

will provide the necessary manpower input to

decision makers at critical system development

milestones to encourage optimization of manpower

planning. Where possible it will support the

determination of manpower related life cycle and

support costs relative to specific systems.

Through involvement with the ISD model, the

MANPERS component will support a margin of

requirements for new equipment training (of the

Material Developer) with training requirements (of

the Combat Developer) as identified by specific

schools. Finally, the MANPERS component will

serve to provide increased accuracy of information

used in development of draft plan TOEs (as well as

throughout the TOE planning process).

o.

(c) A final aspect of the MANPERS component is the

development of a MANPERS manual to include: job

aids, examples, procedural models, instrument

formats and presentation methods for thorough

manpower and personnel requirements determination.

D-17
The MANPERS manual will provide a tool for quick

responses to needs to generate quality information

on a timely and economical basis. The MANPERS

manual will facilitate the standardized determination and presentaiton of requirements to increase

the utilization of information by decision makers.

Finally, the MANPERS manual will provide a

competent training tool to producers of manpower

and personnel requirements (especially for those

new to the job and content area).

TOTAL MANPERS

The TOTAL MANPERS component will provide the techniques for

aggregation of manpower demand for comparison with available

supply data. Manpower demand is defined as quantity of

manpower by quality of personnel (e.g., skill level) both

within and across systems. In short, it is a quantity by

quality of manpower comparison between demand and supply.

Specific aspects of the TOTAL MANPERS component include:

(a) Development of technqies which can provide

quantity by quality loadings of manpower demand

for a specific system. The data used by TOTAL

MANPERS procedures will be an extension of the

basic MANPERS tools by relating some paradigm of

personnel performance or behavioral requirements.

(b) Another aspect of this component will be to

provide techniques which aggregate information for

quantity by quality of manpower across systems

through comparison of manpower demand loadings

D-18

-r.:-..7
held in common with personnel performance or

behavioral requirements.

(c) A final aspect of the TOTAL MANPERS component will

be its coordination with events and activities of

the LCSMM, the PPBS (through SACS and TOE

planning) and ILS to insure the timely determina- .,

tion of requisite manpower information, and

authorizations, as well as changes to force

structure and composition.

MARMIS

The MARMIS component will provide the manpower management

procedures to integrate the MANPERS and TOTAL MANPERS

components with the Army manpower related documentation

process. In addition, current manpower management and

documentation procedui-al deficiencies and recommended fixes

will be identified. The most significant activity in this

component is to support current Army actions in management

information systems developments relating to the manpower

planning process through the specification of the requirements and attributes of MARMIS. Specific aspects of the

MARMIS component include:

(a) Determination of manpower management and

documentation procedures to facilitate the

utlilzation ot manpower and personnel requirements

methodologies and aggregation techniqes so as to

improve the timeliness and quality of requirements

generated.

D-19

| -- , = = I i i- 4 - . -4 .. .. .. * . . . .
(b) Identification of means to improve accountability

of manpower and personnel requirements documentation to identify potential economies in management

and documentation flows. This will take into

account organizational and technical complexities

which serve to work against desired results. In

addition, these efforts will facilitate improved

manpower planning as well as to focus upon

deficiencies for possible resolution.

(c) Means to control manpower related information will

be developed so that many activities concerned

with other interests (e.g., LCSMM, SACS and TOE

planning, and ILS) can be fully coordinated with,

such that appropriate milestone are achieved on a

timely basis and the requirements of other

activities are met as they relate to manpower

concerns.

AUTO MANPERS

The AUTO MANPERS component will provide the basis for

development of a computer interactive system for determining

manpower and personnel requirements by developing the

requirements, attributes and specification for AUTO MANPERS.

AUTO MANPERS is intended to provide a basis for MANPERS an

TOTAL MANPERS methodologies. Specific aspects of the AUTO

MANPERS component will support development of a proposed

computer interactive system by addressing the following:

(a) Ease of documentation, update and edit of requirements through use of an online, interactive ADP

system which provides: continuous availability to

D - 2 "A

"°

.... . .............. *2.** '~ -s; v*.*~* *- .
,8M

authorized users, embedded training or job aids,

and quick response cross-referencing capabilities

with related manpower documentation. These

capabilities should lead to increased accuracy and

timeliness of requirements production and

utilization as well as potential economies in

administrative support.

(b) Another potential aspect of this component is the

development of a built-in analytic model and

mathematical capability to increase the rigor of

manpower requirements determination. For example,

whenever mathematical calculations or transformations are required the formulas should be readily

accessed on the system.

It should be noted that a training device can be considered

to be a large system in and of itself. Thus, it has its own

associated resource requirements. Estimation of these

additional resource requirements is beyond the current scope

of ETES. However, it is expected that these resource

requirements could be estimated by comparability analysis.

Input to the determination of training device resource

requirements is provided by the task requirements associated

with each device which are determined in Procedure 2.0 and

the instructional programs developed during Procedure 4.5.

D-2

i

o•

--- .-..- ~ -
.p.

V.,

-'V

-4.

-V.

THIS PAGE INTENTIONALLY LEFT BLANK

4..

.4

-V.

-V.

'.4

-'4

4-

D-22

-'V 9

, . . . . . *...................-.... . - - '.4 .-------.......--- '-'V. . . -V *.V..'~ * - -V V 4 -
J

APPENDIX E: REFERENCES

ARMY REGULATION TITLE

1-i Planning, Programming, and Budgeting

within the Department of the Army (MAY

76)

10-4 Organization and Functions, United States

Training and Doctrine Command

10-38 United States Army Concepts Analysis

Agency (CAA)

11-18 The Cost Analysis Program (DEC 75)

15-14 Systems Acquisition Review Council

Procedures (MAY 78)

70-1 Army Research, Development and

Acquisition (FEB 77)

70-8 Personnel Performance and Training

Program (PPTP) (OCT 78)

70-10 Test and Evaluation During Development

and Acquisition of Materiel (JAN 76)

70-15 Product Improvement of Materiel (AUG 80)

70-27 Outline Development Plan/Development

Plan/ Army Program Memorandum/Defense

Program Memorandum/Decision Coordinating

Paper (MAR 75)

7i-1 Army Combat Developments (SEP 68)

71-2 Basis of Issue Plan (JUN 76)

71-3 User Testing

-1-5 Introduction of New or Modified Systems,'

Equipment

71-6 Type Classification/Reclassificat in of

Army Materiel

E-L

41- .- r--.
REFERENCES (continued)

ARMY REGULATION TITLE

71-7 Military Training Aids and Army Training

Aids (OCT 73)

71-9 Materiel Objectives and Requirements (APR

75)

71-11 Total Army Analysis (MAY 80)

310-1 Publications, Blank Forms, and Printing

Management

310-3 Preparation, Coordination, and Approval

of Department of the Army Publications

310-34 Equipment Authorization Policies and

Criteria, and Common Tables of Allowances

310-49 The Army Authorization Documents Systems

(TAADS)

350-1 Army Training (JUL 78)

350-35 New Equipment Training and Introduction

(NOV 81)

351-1 Individual Military Education and

Training (MAR 81)

351-183 Service School Training Reports Control

Symbol (JULY 76)

570-2 Organization and Equipment Authorization

Tables - Personnel (SEPT 78)

600-200 Enlisted Personnel Management System (FEB

81)

602-1 Human Factors Engineering Program (JUN

76)

611-1 Military Occupational Classification

Structure Development and Implementat ion

(JUN 76)

S - - " " : -' , '.- ---- '* . - ' . " " " " " ' " ' "" "" "" . " "- ".... ."" ""
,.X . . - .- .. , --- .. 

77.-

REFERENCES (continued)

ARMY REGULATION TITLE

611-201 Enlisted Career Management Fields and

Military Occupational Specialties (JUNE

82)

700-5 Total Logistics Readiness/Sustainability

Analysis (TLR/S) (MAY 78)

700-35 Product Improvement of Materiel

700-78 Production and Post-Production Testing of

Army Materiel

700-90 Army Industrial Preparedness Program

700-127 Integrated Logistic Support (ILS) (APR

81)

702-3 Army Materiel Reliability, Availability

and Maintainability (RAM) (JAN 77)

715-6 Proposal Evaluation and Source Selection

(SEP 70)

725-1 Special Authorization and Procedures for

Issues, Sales, and Loans

750-1 Army Material Maintenance Concepts and

Policies (JUN 72)

750-4 The Army Materiel Plan-Part II Depot

Materiel Maintenance and Support

Activities

1000-1 Basic Policies for Systems Acquisition

(JUN 81)

TRADOC REGULATION TITLE

11-1 Manpower Analysis and Force Structuring

in Combat Development Forces (OCT 78)

11-5 Cost Analysis Program (MOS Tr3ining

Costs) (NOV 77)

11-7 Operational Concepts and Army Doctrine

(DEC 80)

.- F:- 3 "

S S°. ." "S .," ,,
REFERENCES (continued)

TRADOC REGULATION TITLE

11-8 Combat Development Studies (FEB 81)

71-4 TRADOC Standard Scenarios for Combat

Developments (MAR 77)

71-5 Scenario Oriented Recurring Evaluation

System (SCORES) (MAR 77)

71-10 Integration of the TOE development

Process and the Scenario Oriented

Recurring Evaluation System (SEP 77)

71-12 Total System Management - TRADOC System

Manager (SEP 78)

310-2 Development, Preparation and Management

of Army Training and Evaluation Program

(ARTEP) (DEC 79)

350-2 Development, Implementation and

Evaluation of Individual Training (FEB

79)

350-4 The TRADOC Training Effectiveness

Analysis (TEA) System (June 79)

350-7 A Systems Approach to Training

351-4 Job and Task Analysis (MAR 79)

351-9 Individual and Collective Training Plan

for Developing Systems: Policy and

Procedures

600-4 Integrated Personnel Support (IPS) (JUN

78)

700-1 Integrated Logistic Support (ILS) (JUL

77)

F- 4

• . . -...
REFERENCES (continued)

TRADOC HANDBOOK TITLE

Mission Area Analysis (DEC 79) (Draft)

Training Device Requirements Documents

Guide (MARCH 79)

Soldier Analysis (JULY 81)

TRADOC Training Effectiveness Analysis

Handbook

TRADOC PAMPHLET TITLE

11-8 Cost and Operational Effectiveness

Analysis Handbook

71-8 Analyzing Training Effectiveness (FEB 76)

310-8 Collective Front-End Analysis (CFEA) for

Development of Army Training and

Evaluation Program (ARTEP) (Draft)

350-30 Interservice Procedures for Instructional

Systems Development (ISD) (AUG 75)

351-4 Job and Task Analysis Handbook

70-1 Training Device Development (FEB 79)

(Expired FEB 80)

350-2 Officer Job/Task Analysis and Training

Development (MAR 79) (Expired MAR 80)

350-3 Individual/Collective Training and

Development Glossary (Dec 79) (Expired

DEC 80)

351-1 Common Job and Task Management (JAN 80)

(Expired JAN 81)

351-3 Training Requirements Analysis System

(TRAS)/Individual Training Plan (iUP)

(DEC 79) (Expired Dec 80,-

351-7 Job Training Program (JTP) (APR 80)

351-8 Individual and Collective Tr3ining Plan

for Developing Systems - Policy and

Procedures (MAY 80)

E-5
REFERENCES (continued)

TRADOC CIRCULAR TITLE

351-12 Format for Programs of Instruction (POI)

(Expired APR 81)

351-28 Soldier's Manuals, Commanders Manuals and

Job Books: Policy and Procedures

(Expired Dec 79)

DARCOM/TRADOC PAMPHLETS TITLE

PAM 70-2 DARCOM/TRADOC Materiel Acquisition

Handbook (JAN 80)

* - DARCOM CIRCULAR TITLE

700-4 Logistics-Ta iloring Procedures

Guide (FEB 80)

DA PAMPHLETS TITLE

11-25 Life Cycle System Management Model

for Army Systems (May, 1975)

570-558 Staffing Guide for US Army Service

Schools (DEC 79)

MILITARY STANDARDS TITLE

MIL-STD XYZ Task Analysis (JAN 80) (Proposed)

MIL-STD 1388A Weapon System and Equipment Support

Analysis (Nov 81) (Proposed)

DEPARTMENT OF

DEFENSE DIRECTIVES TITLE

DODD 5000.1 (Draft) Major System Acquisitions (DEC 81)

DODD 5000.1 Major System Acquisitions (MAR 82)

DODD 5000.2 Major System Acquisition Procedures

(MAR 80)

DODD 5000.39 Acquisition and Management of

Integrated Logistic Support for

Systems and Equipment (JAN 80)

E-6

• .,'" :....v ...-...-..".-.-....".",'.' --...-.'-.'-"."-.-.-.-.v.. .".. . . . .'-.,,. --.- , - . -"." - . " "".,'- S• ~ .. . . :. ,..... -- ... . ,- . .. .p. • , -. ..- **, ,"... l.':i-...::l;i' . . -
REFRENCES (continued)

DA USASC & FG PAMPHLETS TITLE

US ARMY SIGNAL CENTER Skill Performance Aids (SPA)

FT. GORDON Management Plan (FEB 79)

UNDERSECRETARY OF TITLE

DEFENSE MEMO

USD (R&E) Memo Major Defense System Acquisition

Program Documentation Format (APR 82)

DEPARTEMENT OF DEFENSE TITLE

SYSTEMS MANAGEMENT DOD Acquistion Improvement Program

COLLEGE (OCT 81)

E--

'I

F-
REFERENCES (continued)

TECHNICAL REPORTS

AIRMICS R&D Bulletin No. 1: Management Decision Support

Systems, April 1980.

M. Alford, Software requirements engineering methodology

(SREM) at the age of two. IEEE Transactions, 1978, pp. 332-

338.

J. Amkreutz, Cybernetic model of the design process.

Computer Aided Design, 1976, 8(3), 182-192.

Army Training and Evaluation system. Task 3 Report.

Softech Contract No. 1026, June 1977.

M. Atwood, and R. Jeffries, Studies in plan construction

II: Novice design behavior, (TR-SAI-80-154-DEN).

M. Atwood, R. Jeffries, A. Turner, and P. Polson, The

processes involved in design software. Science Applications

Inc. Technical Report. August 1980.

C. Bagge, and D. Rothman, A first-order methodology for

calculating probability of mission success. Defense Nuclear

Agency DNA 4843F, January 1979. (ADA-088910)

R. Balzer, N. Goldman, and D. Wile, Informality in program

specification (ISI/RR-77-59). Information Sciences

Institute, April 1977.

H. Barina, W. Cobey, J. Rosenbaum, and S. White, Automated

software design. IEEE Transactions, 1979, pp. 384-391.

S. Baron, and W. Levison, the optimal control model: Status

and future, Proceedings of the International Conference on

Cybernetics and Society, 1980, pp. 90-100.

M. Berkowitz, and H. O'Neil, An annotated bibliography for

instructional system development, ARI Technical Report 426,

August 1979.

R. Bonczek, C. Holsapple, and A. Whunston, Future directions

for decision support. Purdue University Technical Report,

June 1980. (AD-A087355)

E-8
-HUM

REFERENCES (continued)

S. Bonder, A review of Army force modernization and

associated manpower, personnel and training processes

(Technical Report 81-6). Alexandria, VA: U.S. Army

Research Institute for the Behavioral and Social Sciences,

January, 1981.

D. Boyd, and A. Pizzarello, Introduction to the WELLMADE

Design Methodology. IEEE transactions on software

engineering, Vol. SE-4, No. 4, July 1978.

L. Boydstan, D. Techroew, S. Spewak, Y. Yaamamoto, and G.

Stamer, Computer aided modeling of information systems.

Paper presented at IEEE Computer Society Conference, 1980.

D. Boylston, An automated decision aid for the assignment of

tasks to training media in early training estimation. Paper

presented at the 51st Symposium of the Military Operations

Research Society, September 27-29, 1983.

J. Braby, H. Parrish, and W. Swope, A technique for choosing

cost-effective delivery systems, Navy Training and Analysis

Group Report No. 16, April 1975.

R. Brant, and D. Taffs, A user-oriented approach to control

languages. Software-practice and experience, 1976, Vol. 6,

pp. 93-108.

R. Brooks and M. Garnet, Modern programming practices:

implications for human factors research. ARI Research Note

80-18, July 1979.

G. Brown, Top down design using HRTRAN. Proceedings of the

IEEE, 1980, 1154-1157.

R. Bulfer, A. Goldman, D. Wile, Informality in program

specification. Information Sciences Institute Research

Report. 77-59. April 1977. (AD A041667)

J. Burns, from flow diagrams to simulation model.

Proceedings of the Interntional Conference on Cybernetics

and Society, 1980, pp. 7-12.

R. Cordes, Software-user interface evaluation: Methodology

and tools. Proceedings of the Human Factors Society - 24

Annual Meeting, 1980, pp. 395-339.

E-9

A I-
REFERENCES (continued)

3. Cory, C. Johnson, A. Korotkin and R. Stephenson, Duty

modules: an approach to the identifcation and classifiction

of personnel resources and requirements. ARI Technical

Paper 367, June 1979.

L. Cheng, URL-URA, An Installation Guide. ESD-TR-75-36.

Air Force Electronics Systems Division, January 1976.

A. Crolotte, Selecting a decision aid. Proceedings of the

International Conference on Cybernetics and Society, 1980,

pp. 1067-1974.

J. David, J. Price, Successful communication in full scale

engineering development statements of work. Air Force

Institute of Technology, August 1980. (AD A087497)

E. Dawdy, W. Chapman, and E. Frederickson, REMBASS final

report. El Paso, Texas: Applied Science Associates, June

1981.

Demonstration of Input/Output Requirements Language (IORL)

for AIRMICS: Final Report. September 1980.

F. Detrich, Let's return to the fundamentals of the

acquisition process. AIAA aircraft systems and technology

meeting, August 4-6, 1980.

R. Dick, E. Koehler, Engineer's guide to he use of human

resources in electronic system design: an evaluation.

NPRDC Special Report 81-3. November 1980.

D. Dieterly, Problem solving and decisionmaking: an

integration. NASA Technical Memorandum 81191, April 1980.

M. Donnell, L. Adelman, J. Patterson, Development of a

computerized training requirements and cost evaluation

system for the U.S. Marine Corps. Defense Advanced Research

Projects Agency, Technical Report 80-5-315-3, November 1980.

W. Dzida, S. Herda, W. Itzfeldt, User-perceived quality of

interactive systems. IEEEE transactions on software

engineering, Vol. SE-4, July 1978.

E. Erbe, R. Hastwig, H. Kehman, G. Mueller, V. Schauer,

Integrated data analysis and management for the problem

solving environment. Information systems, 1980 (5), 273-

185.

E-10

4
..

REFERENCES (continued)

D. Evershed, G. Rippon, High level languages for low level

users. Computer journal, 1970, 14 (1), pp. 87-90.

E. Feigenbaum, Knowledge engineering: The applied side of

artifical intelligence. Department of Computer Science,

Stanford University Report STAN-80-812 (HPP-80-21).

C. Fink, W. Carswell, Integrated personnel and training for

TRADOC system manager (TSM): technological gaps. ARI

Research Report 1238, February 1980.

C. Fink, and W. Carswell, Integrated personnel and training

information for TRADOC systems managers (TSM): technological

gaps (ARI Research Report 1238). Alexandria, VA: U.S. Army

Research Institute for the Behavioral and Social Sciences,

February 1980.

C. Fink, and W. Carswell, Handbook for action officers and

training developers for new materiel systems. Alexandria,

VA. U.S. Army Research Institute for the Behavioral and

Social Sciences, June, 1981.

J. Foley, and V. Wallace, The art of natural graphic manmachine conversation. Proceedings of the IEEE, 1974, 61(4),

462-471.

J. Foley, V. Wallace, The art of natural graphic man-machine

conversations. Proceedings of the IEEE, Vol. 62, No. 4,

April 1974.

E. Frederickson, J. Hawley, P. Whitmore, M. Wood, Air

defense training developer decision aid: model extension

research requirements. Alexandria, VA: U.S. Army Research

Institute for the Behavioral and Social Sciences, November

1980.

R. Gagne, H. Foster, and M. Crowley, The Measurement of

Transfer of Training, Psychological Bulletin, 1948, 43, 97-

130.

J. Gannon, An experiment for the evaluation of language

features. International Journal of Man-Machine Studies,

1976, pp. 61-73.

GAO Report, Effectiveness of U.S. torces can be increased

through improved weapon system design. Washington, D.C.:

Comptroller General of the United States, January, 1981.

E-11

-9
REFERENCES (continued)

E. Garrison, Factors in mangement information system

failures. MILPERCEN Alexandria, VA Final Report, December

1980.

R. Geiselman, M. Samet, Summarizing military information:

An application of schema theory. Human Factors, 1980,

22(6) , 693-705.

H. Ghazealh, A. Nelson, A foundation for an international

logistics language, Air Force Journal of Logistics, 1981,

5(l), pp. 21-28.

T. Green, Conditional program statements and their

comprehensibility to professional programmers. Journal of

Occupational Psychology, 1977, 50, pp. 93-109.

J. Greenstein, The use of models of human decision making to

enhance human computer interaction. Proceedings of the

International Conference on Cybernetics, 1980, pp. 968-970.

M. Halpern, Foundations of the case for natural-language

programming. IEEE Spectrum. 1977 (March, pp. 140-144.

M. Hamilton, S. Zelchin, Requirements definition within

acquisition and its relationship to post-deployment

software. Higher Order Software Inc. November 1979.

Handbook for Human resources test and evaluation systems

Vols 1 and 2. (In Press)

Hanson, L. Vernon, and Purifoy, R. George, TSM guide to

training development and acquisition for major systems, (ARI

TR-78-A7). Alexandria, VA: U.S. Army Research Institute

for the Behavioral and Social Sciences, March 1978. (TSM

Guide)

J. Hawley, Evaluation of a training developer's decision aid

(TDDA) for optimizing performance-based training in machine

MOS. Alexandria, VA: U.S. Army Research Institute for the

Behavioral and Social Sciences, November 1979;

G. Heidorn, Automatic programming through natural language

dialogue: A survey, IBM Journal of Research and

Development, 1976, July pp. 302-313.

C. Herot, A spatial graphical man-machine interface.

Information processing, 1980, 1039-1043.

E-12

? W
REFERENCES (continued)

H. Hill, G. Kress, Development of a methodology for

measuring transfer of training effects for tactical training

systems. ARI Research Note 80-6, September 1979.

? I. Hill, Wouldn't it be nice if we could write computer

programs in ordinary English--or would it? Computer

Bulletin, 1972, June pp. 306-312.

J. Hoc, Role of mental representation in learning a

programming language. Journal of man-machine studies, 1977,

9, pp. 87-105.

S. Huff, Preliminary design for complex software systems

using graphic decomposition. Proceedings of the

international conference on cybernetics and society, 1980,

pp. 479-484.

Human resources test and evaluation system: prototype

handbook. ARI Draft Report, 1978.

E. Hurst, M. Koehner, Optimization in interactive planning

systems. Paper presented at NYU Symposium on decision

support systems, May 21-22, 1981.

W. Johnson, A. Rouse, W. Rouse, An annotated selective

bibliography on human performance in fault diagnoses

tasks. ARI Technical Report 435, January 1980. (AD

A086592)

R. Johnson, Information Requirements: the major challenges

part III: retrieving information from the data group. AIAA

international meeting and technical display on global

technology 2000.

C. Jorgensen, A methodolgoy and analysis for cost-effective

training in the AN/TSQ-73 missile minder (Research

Memorandum 77-26). Alexandria, VA: U.S. Army Research

Institute for the Behavioral and Social Sciences, February

1977.

C. Jorgensen, P. Hoffer, Prediction of training programs for

use in cost training effectiveness analysis (Technical

Report 472). Alexandria, VA: U.S. Army Research Institute

for the Behavioral and Social Sciences, January 1978.

E-13

"... -" - .- ..- -" . . '' . . -. ':' < '? 4 ', -- ". .. 1. .- " -,i"." - . . ., .- .-... - " "'., : .. ' . " . ; .'.' ' ! . • { t , ,- ', " - _ - , " - z - - • '- _, ,'. , -' _; -° , '. ,.. :
-f - ,. , .. - i i . . ~ 4 . P . , . | C -;,.. _

* AI

REFERENCES (continued)

C. Jorgensen, S. Kubala, J. Davis, and M. Atlans, How to use

the training effectiveness estimation model (Working Paper

01-81). Ft. Bliss, TX: U.S. Army Research Institute for

the Behavioral and Social Sciences, January 1981.

C. Jorgensen and L. O'Brien, The early training estimation

system: an automated training needs assessment technique.

(Paper to be published in the first issue of the Training

Technology Journal).

J. Kane, Personnel and training subsystem integration in

an armor system, (ARI Research Report 1303). Alexandria,

VA: U.S. Army Research Institute for the Behavioral and

Social Sciences, January 1981.

A. Kamal, Eid , S. Aly, & M. Mahmoud, Multi-stage

Clustering: An Efficient Technique in Socioeconomic Field

Experiments. IEEE Transactions on Systems, Man, and

Cybernetics, December 1981, 11.

W. T. Kerwin, and G. S. Blanchard, Man/Machine interface -

a growing crisis. Army Material Systems Analysis ActivityArmy Top Problem Areas Discussion Paper Number 2, Aberdeen,

ME), August, 1980.

Kinton Incorporated, Handbook for action officers and

training developers for new material systems. Alexandria,

VA: Kinton Inc., June 1981.

F. Kopstein, A. Siegel, L. Wilson, H. Ozkapitan, Human

performance in continuous operations: Volume II, management

guide, ARI Research Product 80-4, December 1979.

H. P. Lenzycki, and D. L. Finley, How to determine training

device requirements and characteristics: A handbook for

training developers. Alexandria, VA: U.S. Army Research

Institute for the Behavioral and Social Sciences, May 1980.

J. Levine, S. Mallamad, E. Fleishman, Decision aids in

estimating personnel requirements. Advanced Research

Resources Organizations, March 1979.

S. Lipha, Some issues in requirements definition. IEEE

Proceedings, 1980, pp. 56-58.

E-14

- . .- 

"* " -'" ' ." """"."'"' .-. -' - " .... " ;" """ a" ."..-.. '" ' ' " " '"- ."... " "". " .."".. .- .".. . . . .-... . . . . " .. '. ' . " ''. ' """. _ ' '. " " , * . "'. " " * " ","
REFERENCES (continued)

S. Madlin, Combat operations training effectiveness analyses

model: 1979 perspective. ARI Technical Report 393, July

1979.

A. Mahlhotra, J. Thomas, J. Carroll, L. Miller, Cognitive

processes in design. IBM Research Division Report, 1978.

T. Mannle, & J. Balcom, Estimating the manpower, personnel

and training requirements of the Army's corps support weapon

system using the HARDMAN methodology. Wilmington, MA:

Dynamics Research Corporation, September 1982.

R. K. Matlick, D. C. Berger, M. Knerr, and J. R. Chiorini,

Cost and training effectiveness in the Army life cycle

system management model (Volume I and II). Alexandria,

VA: U.S. Army Research Institute for the Behavioral and

Social Sciences, May, 1980.

R. Matlick, D. Berger, & M. Rosen, Cost and training

effectiveness analysis performance guide (Research Product

81-1). Alexandria, VA: US Army Research Institute for the

Behavioral and Social sciences, July 1980.

R. McAfee, and A. Whinston, International journal of policy

analyses and information systems. 4(3), 1980, 285-295.

P. Merrill, Task analysis - An information processing

approach. Paper sponsored by Office of Naval Research,

Personnel and Training Research Programs, Psychological

Sciences Division, Arlington, VA (Project No. NR154-280).

L. Miller, et al. Programming in Natural English. IBM

Research Report, 15 November 1974. (AD/A-3923)

R. Miller, Psychology for a man-machine problem-solving

system. IBM Technical Report TR 00.1246, February 1965.

R. Mittermeir, Enhanced system analysis for requirements

analysis, IEEEE Transaction, 1979, pp. 300-305.

R. Moss, G. Sexton, J. Kearns, Design and Evaluation of

C-mpiex Man-Machine Systems (Brieting).

. Mumy, R. Winger, .. bob-itt, J. Cunman, and W. Van

Newhirle, Handbook for the integration of countermeasures

counter-countermeasure considerations into the mat- riel

develolment/acquisition life cycle C>iCCM-HB-80-1 ) . March

1980.

E-1I
REFERENCES (continued)

B. Mullegan, J. Funaro, Front-end analysis: generic and

rongeneric models. Technical Report: NAVTRAEQUIPCEH IH-325,

September 1980.

D. Neves, J. Anderson, Knowledge compilation: mechanism for

the aatomation of cognitive skills. Personnel and Training

Programs, office of Naval Research, Arlington, VA 22217,

Technical Report 80-4, 1980.

D. Nor-man, Errors in human performance. Center for Human

Information Processing, University of California, San Diego,

August 1980, (Report No. 8004).

Northrup Services, Inc., Study to improve the development of

U.S. army maintenance manpower authorization criteria

(MACRIT) Data: Office of the Army Deputy Chief of Staff for

Logistics, Washington, D.C., December 1981,

*L. O'Brien, Early training estimation system (ETES)

Proceedings of the TRADOC Developments Institute Chiefs of

Analysis Seminar, October 21-12, 1981.

L. O'Brien, Automated system description technology. Paper

presented at' the Second Annual Conference on Microcomputers

in Education, Washington, June 1982.

office of Management and the Budget Circular A-109. Major

system acquisition, April 1976.

L. Osterer, A user's guide for GRAPEL - graph fo r

enierrQ agae Brookhaven National Laboratory, July

B. Ostrovofsky, Morphology of design of aerospace systems

with inclusion of human factors. Final Report. University

of Houston, August 1977.

F. Petrie, The utilization of requirement statement

rrethodoogies in the U.S. Navy and their impact on systems

acquisition. Master's Thesis Naval Postgraduate- School,

eMarcn 1980.

R. Pew, C. Freehrer, S. D. Miller, Critical oarrnn, revieo

and analysis of perfo-ermance models applicale to man-machine

systems evaluation, (AFOSR-TR-77-0520). Ari 19i7.

4MCI)Dt:Ofc fteAm euyCifo tf o

Logitic, DC.,Decmbe Wshigto, 19.o
REFERENCES (continued)

W. Pieper, T. Elliot, & J. Hawley, Training developers

decision aid tor optimizing performance based training in

machine ascendant MOS (ARI Research Report 1223).

Alexandria, VA: U.S. Army Research Institute for the

Behavioral and Social Sciences, November 1978.

W. Pieper, T. Elliot, & J. Hawley, Tryout of a training

developer's decision aid for optimizing performance-based

training in machine ascendant MOS. Alexandria, VA: U.S.

Army Research Institute for the Behavioral and Social

Sciences, September 1979.

H. E. Price, M. Fiorello, J. C. Lowry, M. G. Smith, and J.

S. Kidd, The contribution of human factors in military

system development: methodological considerations

(Technical Report 476). Alexandria, VA: U.S. Army Research

Institute for the Behavioral and Social Sciences, July,

1980.

Proceedings of the TRADOC Chiefs of Analysis Seminar.

Hampton, VA July 16-18, 1980.

H. Ramsey, M. Atwood, A comparative study of flowcharts and

program design languages for the detailed specification of

computer programs. ARI Technical Report TR-78-A22,

September 1978.

H. Ramsey, M. Atwood, Cognitive structures in the

comprehension and memory of computer programs: an

investigation of computer program debugging. ARI Technical

Report TR-78-A21, August 1978.

H. Ramsey, M. Atwood, Man-machine interface design guidance:

State-of-the-art. Proceedings of the international

conference on cybernetics and society, 1980, pp. 579-582.

A. S. Rhode, F. L. Friedman, F. E. O'Connor, Integration of

- manpower, personnel and training issues from the material

system acquisition process into the planning programming and

budgeting system, (ARI TR 526). Alexandria, VA: U.S. Army

Research Institute for the Behavioral aand Social Sciences,

4arch, 1981.

A. S. Rhode, B. B. Skinner, J. L. Mullin, F. L. Friedman,

4 and M. M. Franco, Manpower, personnel and training r quirements for materiel system acquisition. Alexandria,

VA: U.S. Army Research Institute for the Behavioral and

Social Sciences, February 1980. (ARI MPT Study)

7-17

- E-!7- -. . - 9..-.. -. *t . . ' S -

. .. .. . . . .. . . . . . . . . . . . . . . . . .
*IP*

REFERENCES (continued)

J. Rivlin, M. Hsu, and P. Marial, Knowledge based

consultation for finite element structural analysis. Air

Force Wright Aeronautical Labs, May 1980. AFWAL-TR-80-3069.

R. Robinson, Objective measurement of training readiness.

USAWC Military Studies Program Paper. US Army War College,

Carlisle, Barracks, PA, 16 May 1980.

H. Rome, and J. Matuszewski, Applications of decision-tree

oriented methodology for large-scale systems design.

Proceedings of the international conference on cybernetics

and society, 1980, pp. 258-271.

D. Ross, K. Schoman, Structured analysis for requirements

definition. IEEE transactions on software engineerinj, Vol.

SE-3, No. 1, January 1977.

W. Rouse, S. Rouse, Measures of complexity of fault

diagnosis tasks, IEEE transactions on systems, man, and

cybernetics, Vol. SMC-9, No. 11, November 1979, pp. 720-72.

W. Rouse, System engineering models of human-machine

interaction. Urbana, Illinois, North Holland, 1980.

A. Sage, A methodology for system design. Proceedings of

the international conference on cybernetics and society,

1980, pp. 272-277.

B. Schneiderman, A. Shapiro, Toward a theory of encoded data

structures and data translation. International journal of

computer and information sciences, 1976, S(1), pp. 33-42.

J. Schneider, D. Combs, T. Folsin, NETGRAF: a computer

graphics aid to the operation and interpretation of NETSIM,

a traffic simulation model. Department of Transportation

Special Report DOT-RSPA-DPB-50/80/10, March 1980.

D. Schofield, A. Hillman, J. Rodgers, MM/I, A man-machine

interface. Software-practice and experience, 1980, 10, pp.

751-763.

R. Schulz, and J. Farrell, Job ._s: descriptive authoring

flowcharts for phase IV analyze of the instructional systems

development model, Research Product 80-13. Alexandria,

VA; U.S. Army Research Institute for Behavioral and Social

Sciences, May, 1980.

E-18
1

REFERENCES (continued)

D. Seng, Automation of task analysis data: the LSAR/human

factors engineering enhancement proposal. Technical

Memorandum 24-80, 1980, US Army Engineering Laboratory.

S. Shrier, Algorithms for system design. Proceedings of the

international conference on cybernetics and society, 1980,

pp. 278-283.

R. Sidbrsky, and N. Parrish, Guidelines and criteria for

human-computer interface design of battlefield automated

sytems. Proceedings of the international conference on

cybernetics and society, 1980, pp. 590-595.

G. Siebold, The applicability of the ISD 4-factor model of

job analysis in identifying task training priority in nine

technical military occupational specialties. ARI Technical

Report 432, October 1979.

A. Siegel, W. Leahy, and J. Wolf, A computer model for

simulation of message processing in military exercise

control and evaluation systems, (ARI TR-77-A22), October

1977. -

B. B. Skinner, and F. L. Friedman, A study to identify and

consolidate manpower, personnel and training requirements

for materiel systems at ASARC milestones. Alexandria, VA: .

U.S. Army Research Institute for the Behavioral and Social

Sciences, August, 1981. A

B. B. Skinner, F. L. Friedman, E. H. Mueller, A study to

validate a sample set of questions and the general approach

in their development for an Army systems acquisition review

council (ASARC) III system, (Draft ARI Technical Report).

Alexandria, VA: U.S. Army Research Institute for the

Behavioral and Social Sciences, June 1981.

S. Smith, Man-machine interface requirements definition:

task demands and functional requirements. Proceedings of

the international conference on cybernetics and soceity,

1980, pp. 585-590.

S. Smith, Requirements definition and design for the manmachine interface in c3 system acquisition. The MITRE

Corporation Report No. M80-10, 15 April 1980.

E'

E-19 I.

U1
REFERENCES (continued)

A. Stavely, Design feedback and its use in software design

and sytems. Proceedings of the softwre quality and

assurance workshop. November 15-17, 1978.

E. Stohr, and M. Tannivu, A data base for operations

research models. International journal of policy analysis

and information systems, Vol. 4(1), 1980.

G. Sussman, J. Holloway, and T. Knight, Design aids for

digital integrated systems, an artificial intelligence

approach. Proceedings of the IEEE, 1980. pp. 612-615.

J. Taylor, Computer language and natural language taught

comparatively. Computer and people, 1977, p. 7-22.

D. Teichroew, and E. Hershey, PSL/PSA: a computer-aided

technique for structured documentation and analysis of

information processing systems.

D. Teichroew, P. Macasovic, E. Hershey, and Y. Yamamoto,

Application of the entity-relationship approach to

information processing systems modeling. In P. P. Chen

(ed.), Entity - relationship approach to system analysis and

design. North-Holland Publishing Co., 1980.

E. Thomas, R. Hankins, Use of human resource data in weapon

system design: identification of data/data systems and

related technologies (AFHRL-TR-79-36), January 1980.

E. Thomas, and D. Newhouse, Human resource data in weapon

system design: an initial plan for development of a unified

data base (AFHRL-TR-80-25), November 1980.

Pei-Ti Tung, A systematic approach to complex system design:

an application to printed circuit board test system

division. MIT Techical Report, 13 May 1980.

W. Van Cleemput, A structural design anguage for computer

aided design of digital systems. Stanford Electronics

Laboratories, Techical Report No. 136, April 1977.

VisiCalc: User's Guide for the Apple III, Visi Corp, 1982.

VisiSchedule: User's Guide for the Apple III, Visi Corp,

1982.

E-20

.. E-.2 0. .
rOw

REFERENCES (continued)

P. Weddle, & G. Fulkerson, Forecasting the Human Resource

Costs of Navy Weapon Systems. Defense Management Journal,

1980, 16, 6-12.

G. Wheaton, P. Fingerman, A. Rose, and R. Leonard,

Evaluation of the effectiveness of training devices:

eluboration and application of the preditive model.

Research Memorandum 76-16. Alexandria, VA: U.S. Army

Research Institute for the Behavioral and Social Sciences,

July 1976.

J. Wheeler, Embedded system design with ADA as the system

design language, Software Technology Division, US Army

Communications Research and Development Comand.

D. Whitmore, Requirements Specification (ASD-TR-78-45). 2

January 1979.

G. Willison, The battalion commander's guide to successful

training management. Master Thesis, US Command and General

Staff College, 1980 (AS A094436)

W. Woods, Multiple theory formation in high level

perception. Bolt, Beranek, and Newman, Inc. Technical

Report No. 38, April 1977.

W. Woodson, and W. Coburn, Human engineering design criteria

for modern control/display components and standard parts.

US Army Human Engineering Development Lab, US Army Missile

Lab, May 1980 (ADA091191).

W. Woodson, and C. Coburn, Recommended modifications to MILSTD-1472B, selected sections on controls, displays, and

related hardware. US Army Human Engineering Lab Detachment,

US Army Missile Lab (DAAK40-79-C-0145), May 1980

(ADA091477).

R. Zampora, et al. Tutorial in the use of the SRI TREE

language system. Standford Research Institute, December

1974 (AD-A008 927).

E-21

...-.... ....-
p.

h

I

I

r.

I..

4.

4.

4 44

E-22 4'

4

4.

J , .4*~ -. .- .* .* * .4 .**44*.4*4**.* *q* .'..~.' .- ... .*.~..- - -. ~ - 4 . 4 ~ S S * . 4 . . -