- -• o •.. . ,.- . , .,.-- .- • .- • : i,.e , l. .

LEVEL

Technical Report 503

COST AND TRAINING EFFECTIVENESS ANALYSIS

IN THE ARMY LIFE CYCLE SYSTEMS

MANAGEMENT MODEL

or• Richard K. Matlick, Doris C. Berger, C. Mazie Knerr

and

John R. Chiorini

Litton Mellonics

ARI FIELD UNIT AT FORT BLISS, TEXAS

SDTIC

Q. Pm 0ELECTE 1 SJAN 4

U. S. Army D

Research Institute for the Behavioral and Social Sciences

September 1960

Approved for public release; distribution unlimited.

•a,••.,•- ... il ...... . . , mI - m-.. ...-- !--"
U. S. ARMY RESEARCH INSTITUTE

FOR THE BEHAVIORAL AND SOCIAL SCIENCES

A "lield Operating Agency under the Jurisdiction of the'

Deputy Chief of Staff for Personnel

L. NEALE COSBY

JOSEPH ZEIDNER Colonel, IN

Technical Director Commander

Research accomplished under contract

to the Department of the Army

Litton Mellonics System Development Division

NOTICES

DISTRIBUTION: Primary distribution of this report has been made by ARI.

Please address correspondence concerning distribution of reports to: U.S.

Army Research Institute for the Behavioral and Social Sciences, ATTN:

PERI-TST, 5001 Eisenhower Avenue, Alexandria, Virginia 22333.

FINAL DISPOSITION: This report may be destroyed when It Is no, longer

needed. Please do not return It to the U.S. Army Research Institute for

the Behavioral and Social Sciences.

NOTE: The findings In this report are not to be construed as an official

Department of the Army position, unless so designated by other authorized

document3.

-4
.i- I

UNCLASSIFIED SECURITY CLASSIFICATION OF THIS PAGE (tefn Dat. Entered)

REPORT DOCUMENTATION PAGE RE COMPLESRCTIONS

1. REPORT NUMNSER [Z. GOVT ACCESSION NO. S. R"eCIPIENt's CATALOG NUMBER

-Technical Report 503 IA' " /'

4. TITLE (and Subtitle) 5. TYPE Of REPORT PERIOD COVERED

v•. Final Report ' ~~~~Cost and Training Effectiveness Analysis 2 pi -2 uy8

in the Army Life Cycle Systems Management Model ,. April O23 -- 23 July 80

PERFORMING ORO. REPORT NUMBER

}!7.' AUTHG.R(a, 'It. CONTRACT OR GRANT NUMBER,'e)

Richard K. Marlick John R. Chiorini

Doris C. Berger MDA903-79-C-0326

C. Mazie Knerr

. PERFORMING ORGANIZATION NAME AND ADDRESS 10. PROGRAM ELEMF.,T. PROJECT, TASK

AREA & WORK UNIT NUMBERS

Litton Mellonics Systems Development Division

Washington Scientific Support Office 2Q263742A794

5265-1 Port Royal Road, Springfield, VA 22151

I I. CONTROLLING OFFICE NAME AND ADDRESS 12. REPORT DATE

U. S. Army Research Institute for the Behavioral 23 July 1980

and Social Sciences, Fort Bliss Field Unit I1. NUMBER OF PAGES , '

P-O. Box 6057. Fort Bliss. Texas 79916 160

14. MONITORING AGENCY NAME ' ADDRESS(I dilfferent from Controlling Offlce) 1S. SECURITY CLASS. (of th otf)

UNCLASSIFIED

I"~~~~~~~~ a'o O E CM!•Irtf44l•DWNGRAO

s NA

16. DISTRIBUTION STATEMENT (of the Report)

Approved for public release; distribution unlimited

17. DISTRIBUTION STATEMENT (of the abstract entered In Block 20, It different from Report)

"IS. .1UPPLEMENTARY NOTES

19. KEY WORDS (Continteo on revere. side If neceessar and Identify by block number)

Cost and Training Effectiveness Instructional Systems Development iv Analysis (CTEA) Training program prediction

Training effectiveness Task list generation

Training Costs Trainability K Training in the LCSMM Training alternatives tj 20. ABSTRACT (COARM 010 rweves sie IV nemeeary md i dentify by block numnber)

This report examines the requirements and purposes of cost and training

effectiveness analysis (CTEA) in the context of the Army's Life Cycle Systems

Management Model. Current CTEA methodologies are examined. Methods to

augment existing models are developed and a general synthesized model for use

by CTEA analysts proposed.

DD S '^N 173 1 W mTtOM OF I •oSu 5 IS OBSOLETF UNCLASSIFIED

± SECURITY CLASSIFICATIO14 OF THIS PAGE Cl1ren Data Entered)

p-rn--

L • . •:•, .:..•.•,;.. . • .. . ... z. ... .. .. .." .
ToculelReport 503

COST AND TRAINING EFFECTIVENESS ANALYSIS

INTHE ARMY LIFE CYCLE SYSTEMS

MANAGEMENT MODEL

Richard K. Matlick, Doris C. Berger, C. Mazie Knerr

and

~ I John R. Chiorini

Litton Mellonics

HWI tA3 Ab Submitted by:

13 Michael H. Strub, Chief K ARI FIELD UNIT AT FORT BLISS, TEXAS

Avai~a111t~Approved by: Avi i/er E. Ralph Dusek, Director lt speci.al PERSONNEL AND TRAINING

RESEARCH LABORATORY

U.S. ARMY RESEARCH INSTITUTE FOR THE BEHAVIORAL AND SOALSECS

5001 Eisenhower Avenue. Alexandria, Virginia 22333

P', Office, Deputy Chief of Staff for Personnel

I Department of the Army

September 1980

Army Project Number Individual Training and Combat Unit Training

20263742A794

Approovd for public milene: distribution unllmluid.

Mi ILM__
ARI 

R&D 

Research Reports and Technical Reports are intended for sponsors of tasks and for other research and military agencies. Any findings ready

of 

for implementation at the time of publication are presented in the last pairt

mendations 

the Brief. Upon completion of a major phase of the task, formal recoin- for official action normally are conveyed to appropriate military agencies by briefing or Disposition Form.

I4ioa t

tev

to q;;

iv~IL~~tt
FOREWORD

The Army is aware of the increasing importance of training and manpower

estimation throughout the life cycle process. Dollar and manpower require- r ments are fixed very early in system development but their full impact is

r often not apparent until years later. For this reason, improved cost and

training effectiveness analysis (CTEA) has become a major Army need.

Guidance exists for what analyses are desired, but the state-of-the-art in

supporting technology has not kept up with demands for increasingly precise

estimates. As a result, ARI initiated a research program on CTEA in 1976

which was concerned with the early estimation of training media, methods,

and costs. In 1978 the program was expanded to include training estimation

problems during a weapon life cycle. Because the developmental history of

no two systems is identical, the present contract was designed too pull

together methods developed to deal with individual cases and produce a

systematic procedural guide for an Army analyst tasked witn CTEA performance.

j The result of the effort is in two volumes. The first examines CTEA

requirements in the life cycle, the available techniques for each analysis

point, and the R&D deficiencies. The sacond is a user's guide implementing

the first volume into a step-by-step procedure for CTEA performance using

actual field data sources and examples. This research is in response to

Army project 2Q263742A794 and special needs of the Directorate of Training

Developments, Ft Bliss, Texas.

JdSEPH I ER

'-T<chnical Director

LI

L'I'

i;V
BRIEF

Requirement:

During the acquisition of Army materiel systems as defined by the Life Cycle Systems Management Model (LCSMM) there is a continuing requirement for information

on the cost and effectiveness of training personnel to operate and maintain the

system. At a number of key points, decision makers need reports, the result of

formal cost and training analyses (CTEA), These CTEA evaluate both the development of training systems to support the materiel system and the hardware to assess the

impact of the system on personnel capabilities. Army analysts currently charged with conduct of CTEA require workable methods suitable to the particular

situation to perform these analyses in a timely manner and to obtain reasonable,

accurate recommendations consistently. Therefore, this research was undertaken

with the objective of providing Army analysts with a performance guide for the conduct of CTEA at each stage of the LCSMM.

Procedure:

Available methods were identified for performance of CTEA components. The methods

were catagorized using a general CTEA model derived from key analyst questions.

If the answers to the questions are negative, the analyst is routed to a rank

ordered set of methods to generate the missing information. If the answers are

positive the analyst is routed to the next operation requiree4 by CTEA regulations.

The process is continued until a complete analysis is perform.'. New methods

were generated where existing techniques proved inadequate and the complete process

incorporated in a user guide.

Findings:

Systems may be fielded with most, some, little, or none of the data described by

Army doctrine. Six basic data conditions and sequences were found as follows:

(I) no task list and no training program, (2) task list but no training program,

(3) training program but no alternatives and no effectiveness data, (4) training

program with effectiveness data but no alternatives, (5) alternative training

programs but no effectiveness data for all alternatives, and (6) training program

alternatives and effectiveness data. Methods in current use were found that coulc

contribute useful information covering one or more of these situations and were

within the state-of-the-art of CTEA conducted in the Army

Three methodological deficiencies were found. Use of historical data is

hampered because there is no consolidated data bank or base that describes tasks

functionally, behaviorally, and in terms of training. Second, existing methodology does not thoroughly address trainability (e.g., Can the available personnel

be treined to perform the tasks at the required level of proficiency, given the

proposed hardware configuration?). The third gap was in cost models. Costs of

unit training were omitted in even the most comprehensive methods examined.

Utilization of Findings:

A CTEA Performance Guide, a fully proceduralized manual, was prepared to provide

guidance for CTEA analysts. The Performance Guide contains step-by-step instructions for the selection, application, and implementation of the process method(s).

It includes guidance for synthesizing methods on the basis of inputs available,

issues, and other constraints as well as strategies for dealing with the various

situations. Instructions for using the processes are complete making it

unnecessary to procure source documents. The procedures are illustrated with

comprehensive examples of their use. The guide is currently being tested in the

performance of two Army CTEAs.

vii
TABLE OF CONTENTS

Section Page

I Introduction I- H

II Cost and Training Effectiveness Analysis in the

Life Cycle System Management Model II-1

A. Conceptual Phase 11-3

B. Demonstration and Validation Phase 11-8

C. Full-Scale Engineering Development Phase II-10

D. Production and Deployment Phase 11-12

E. Constraints on CTEA Methods 11-13

1. CTEA Location 11-13

2. Availability and Quality of Data I1-14

3. Available Personnel 11-14

4. Available Time 11-14

F. Summary 11-15 N

III Cost Training Effectiveness Anialysis (CTEA) Methods III-1

1. TEEM 111-2

2. Training Consonance Analysis III-5

•'' :3 . CH T II: -8

4. TECEP III-11

5. The BDM/CARAF Method III-11

6. lMVDIA 111-16

7. DAIS/TRAMOD Method ,II-17I

8. TDDA 111-20

9. B-i SAT III-26 10. Army CTEA Methods in Current Use 111-31

A. The DIVAD Gun CTEA 111-32

B. Improved Hawk (Hawk PIP) Training Development 111-32

C. Roland Training Development 111-32

D. ITV CTEA III-34' r7 E. DRIMS, CTEA 111-34

F. Other Current CTE A Methods II..-36

11. MethodS for the Analysis of Training Devices/Simulators I11-36

IV CTEA Methods Devised by Litton IV-i

A. The Analogous Task Method (ATM) IV-I

1..I Step 1: Task Definition IV-I

: 2. Step 2: Task Classification IV-4

S3. Ste 3: Analogous Task Identification 1-_4

4. Step 4: Training Asseswient for the Analogous Task r7-5

5. Step 5: Training Assessment for the Target Task IV-5

ix

#.f-

: ; :•. . .. .. • . .. ". .. . .." . . - • i . + ' .. . . i •
LIST OF TABLES

Table Pae

II-I CTEA in the LCSMf 11-4

°II-1 Calculation of Totalk Weighted Effectiveness Scores (E) 111-14

111-2 Cost Model 111-15

111-3 Ranking Factors and Ranking of Training Alternatives 111-35

V-1 Required Analytical Processes and Candidate Methods V-5

B-1 Data Quality Estimates 6-1I

"8-2 Method Cost Classes Ranked by Square Feet B-21

8-3 Cost Ranking B-23

B-4 Examples of Standards of Measurement of Cost and Relevant Cost

Elements for 8DM CTEA Methodology's Eight Training System Types 8-29

B-35 Suggested eost Data Sources., MODIA B-39

8-6 Data Requirements for SXC -•42

!ix

ix

* r~
LIST OF FIGURES

Figur Palpe

1-i Overview of CTEA Research 1-3

I-1- CTEA in the System Acquisition Process II-2

Ill-i TEEM Overview 111-4

111-2 Plot of Efficiency Ratios (the loss function) 111-6.

111-3 CDB Structure TII-10

III-4 Training System Development Model 111-12

111-5 Process Flow in the TECEP Tenhnique TII-12

111-6 Reference Materials Supporting the TECEP Process Flow 111-13

111-7 Sample of Matching Training Objective Characteristics

with a Type of Learning Algorithm 111-13

111-8 Relationships Among'MODIA Elements and the User 111-16

111-9 Training Requirements Analysis Model 111-19

III-10 The Training Developer's Decision Aid IiI-10

III-11 Representative TDDA Cost-Rating Matrix 111-25

111-12 Systems Approach to Training 111-27

111-13 Example of Task Element Behavior 11I-28

III-14 Behavioral Objective Illustrating a Maneuver 111-29

111-15 Example of Personnel Qualification Matrix 111-30

111-16 Reproduction of Part of Matrix Task Inventory Sheet 111-33

IV-1 Logistical Support Analysis Record Sheet D IV-2

IV-2 Logistical Support Analysis Record Sheet B IV-3

IV-3 Analogous Task Method IV-7

IV-4 Task Trainability Analysis Matrix IV-8

IV-5 Army FYDP Programs IV-O10

IV-6 The Arm.y Training System IV-11

V-1 General CTEA Model V-2

B-i Army Life Cycle Cost Matrix B-3

B-2 R&D Training Costs B-4

B-3 Investment Training Costs B-4

B-4 Cost Per Man Year Methodology B-5

B-5 Operating Phase, Unit Ammunition Costs B-6

B-6 Operating Phase, Personnel Replacement Costs B-6

B-7 Dcmumentation Formats Illustrating Documentation

of an C&S Cost Element B-7

B-8 A Basic Need in Life Cycle Cost Analysis B-8

B-9 CTEA Flowchart B-12

B-10 .ost Methocology B-13

B-11 Training System Assessment Worksheet B-26

B-12 CTFA Flow Model B-27

B-13 System Ownership Cost Flow Diagram B-44

B-I11 The System Ownership Cost Model for CHRT B-45

Xii
- " -.

COST AND TRAINING EFFECTIVENESS ANALYSIS IN THE ARMY

LIFE CYCLE SYSTEM MANAGEMfWT MODEL

The Army's Life Cycle System Management Model (LCSMM) defines the process

by which Army materiel systems are acquired. It was designed to ensure that all

aspects of the system are considered during the acquisition process including personnel and training requizments. Cost and training effectiveness Pnalysis (CTEA) provides to decision makers information about how training is influenced

by and influences the characteristics of the developing system. CTEA evaluates

the development of training systems to support the materiel system and the evolution of a materiel system compatible with personnel and training capabilities.

OBJECTIVE

The objective of this research was to provide Army analysts with a performance guide for CTEA at each stage of the LCSMM. The attainment of this objective required:

o Determination of points in the LCS•M at which CTEA are needed.

o Assessment of the utility of existing methods for CTEA at each point.

o Adaptation of methods to each LCSMM stage where CTEA are needed.

o Development of methods for areas where existing ones are not adequate.

o Irdentification of input information required and available at

each point.

o Estimation of the impacts of missing or degraded information on

the CTEA.

o Synthesis of a general model for CTEA in the LCSMM.

o Production of a CTEA Performance Guide for Army analysts.

CTEA SITUATIONS AND STRATEGIES IN THE LCSMM

Several LCSMt events require CTEA information. For example, four cost and

operational effectiveness analyses (COEA) are conducted during the LCSMM. CTEA

support COEA by providing assessments of alternative ways to train to achieve

the desired operational effectiveness of the system as well as proviiing the

cost of each proposed alternative. At least four CTEA are needed to support

the OOEA and additional CTEA are beneficial to support training development

decisions.

Our examination of the LCSMM decision requirements revealed the need for

a more detailed analysis of the CTEA job to be performed by the Army analyst.

S • ...... .. . .. - - = - .. . . '" ' '12 1 L i• . .. .. .. '-1
Materiel systems do not progress through the LCSIM in look-step fashion. Systams may be fielded with most, some, little, o,ý none of the data described in

Army aoctrile, and the data are not necessarily generated in the prescribed

sequence. If we wished to give the Army analyst the wherewithal to perform a

CTEA, we had to use as guidance (1) the decision to be rendered on the basis of

the CTEA information and (2) the data available to conduct the CMEA. These two

factors are the boundaries for CTEA methodology regardless of the point in the

LCS,4•.

Some data conditions and sequences are more likely to occur than others.

For example, the lack of a task list or training program implies the lack of

training effectiveness data. Litton identified six input-data situations as:

1. No task list and no training program,

2. Task list but no training program,

3. Training program but no alternatives und no effectiveness data,

4. Training program with effectiveness data but no alternatives,

5. Alternative training programs but no effectiveness data for all

alternatives, and

6. Training program alternatives and effectiveness data for all

alternatives.

Depending upon the data situation the analyst needs a different strategy

to perform each CTEA. None of th. CTEA methods unearthed in Litton's literature

review provided all information required for decisions at any point in the LCSkl,.

However, detailed examination of the techniques, steps, or processes within the

methods (i.e.. the elements from which the methods are built) showed that each

method contributed to the CTEA job to be performed. Some methods contributed

more elements within some data situations than others. For example, informal,

expert-judgment methods in current use generate task lists if none are available. Formal an~lytic models for prediction of training programs, however,

are superior tc the judgmental models if task lists already exist.

Litton devised a gtmeral model for CTEA to guide the analyst in assessing

the data situation and in selecting CTEA processes. The model leads the analyst

through questions concerning the availability of task lists and training programs,

including alternative training programs. Following these are the estimation of

effectiveness, cost analysis of training program alternatives, cost effectiveness

comparisons of training program alternatives, and resolution of issues. Each

situation identifies a set of required CTEA processes and suggests an approach

to meet the CTEA objectives. Litton designed strategies to guide the selection

of the processes and conduct of the CEA.

SELECTION OF 0TEA PROCESS METHODS

Powerr.al methods applicable to CTEA and the whole manpower, personnel, and

training side of acquisition via the LCSMM 1ave been developed and await broad

application and refinement.

Li -2- __________
An examination of the set of CTEA methods, the constraints on them, and

the GTEA strategies led to the following criteria for the selection of the

methods to be included in the CTEA Performance Guide:

o The method must have either demonstrated or face validity.

o The method selected must be usable in the present rather than

at some unspecified time in the future.

o Because all six strategies are required to meet the requirements identified by the general CTEA model, methods must be

selected or developed to fully implement each strategy.

In addition, alternative methods were selected for each process required

by a strategy. Automated methods were selected whenever possible to meet constraints that arise from personnel resources available to perform CTEA.

The methods that have processes useful in the LCSM are:

o Training Efficiency Estimation Model (TEEM; Jorgensen and Hoffer,

1978),

o Training Consonance Anaysis (TCA; Hawley and Tha~mason, 1978),

o Training Effectiveness, Cost Effectiveness Prediction (TECEP, also

known as TAEG No. 16; Braby et al., 1975),

o Army CTEA Methods in Current Use:

DIVAD Gun CTEA,

Improved Hawk (Hawk PIP) Training Development,

Roland Training Development,

Improved TOW Vehicle (ITV) CTEA,

Diagnostic Rifle Marksmanship Simulators (DRIMS) CTEA, and

Methods for the Analysis of Training Devices and Simulators

(TRAINVICE).

Extant CTEA methodology contained three weaknesses. First, use of historical data was hampered by inadequate definition of analogous tasks (i.e., tasks in

a fielded system ftinctionally and behaviorally similar to tasks in the proposed system). Second, the existing methodology did not thoroughly address the issue

of trainability, one of the estimations needed in the LCS141. Third, the most

thorough cost analysis model omitted costs of training in units.

Litton devised methods to meet these methodological needs. First, our

analogous task method (ATM) was developed to take advantage of information obtained in the process of training soldiers on fielded weapons systems. The

metnod is applied on a task-by-task basis with an overall estimate of training

effectiveness. The analogous task method has six steps: (1) definition of the

-3-
critical tasks to be performed on the developing system (the target tasks); (2)

classificatior of the target tasks to provide a basis for finding the analogous

tasks; (3) analogous task identification and selection; (4) assessment of training for the analogous tasks; (5) generation of estmates of training for the target tasks; and (6) aggregation of the effectiveness and cost measures across all

tasks to obtain a picture of training for the developing system as a whole. ATM

is illustrated in Figure 11-2. Each rectangle represents a step in the process, while the processes indicated in the circles show the relationship of the process

to other processes in a general CTEA model to be presented later.

The second methodology devised by Litton, trainability analysis, is needed

to examine the interactions among tasks, training program alternatives, and

personnel characteristics. Once tasks have been identified and alternative

means of training those tasks have been predicted or developed, it is necessary

to determine that, given the characteristics of the personnel who will man the

system, the tasks Wan be trained to required levels of proficiency. Our literature review unearthed no explicit rethod for trainability analysis, therefore

we proposed a process to use as an interim technique until further research is

conducted.

Litton's third methodological development expanded the TECEP cost model to

cover costs of institutional field training and training in Army units. The

Litton cost model is intended to aid the CTEA analyst in preparing recommendations to the decision maker regarding choices among alternatives. It is not

intended for budgetary purposes. The Litton model does, however, capture pertinent institution and unit costs.

Four methods developed for the Air Force were considered but not incorporated

in the model at present. They require substantial revision to make them suitable

for Army use. These methods are automated in total or in part and use consolidated data bases. They are:

o Coordinated Human Resources Technology (CHRT; Goclowski et al.,

1978), i

c Method of Designing Instructional Alternative (MODIA: Carpenter- I Huffman et al., 1977),

o Digital Avionics Information System: Training Requirements Analysis

Model (DAIS/TRAMOD); Czuchry et al., 1978),

o B-I Bomber Systems Approach to Training (B-I SAT; Sugarman et al.,

1975).

The Training Developer's Decision Aid (Pieper et al., 1979) method was developed for the Army and it is generally applicable to CTEA. It has not yet

been fully developed, however, and scme problems in implementation remain,

especially the generation of task lists. Because of the extent of arbitration

in the logic of the method, an extensive validation effort is necessary to

demcnstrate the usefulness and precision of its outputs. Available descriptions of the method are not complete enough to permit its inclusion in the

CTEA Performance Guide.

-4-
SPECIFICATION FOR THE CTEA PERFORMANCE GUIDE

The means for synthesizing CTEA methods from available process methods

have been provided, but guidance for the application of those means as well as

for the implementation of the selected process methods must also be provided.

The CTEA Performance Guide (a separate volume) provides such guidance.

The CTEA Performance Guide:

o Contains guidance for conducting CTEA on the basis of inputs available for analysis, issues to be resolved, and other constraints on

the analysis, such as the skills and numbers of CTEA personnel and

the time available for performance.

o Contains strategies for dealing with the various input-data situations that are a function of both the phasing of LCSMM itself and

the deviations from the LCSMM that any particular acquisition pro- gram exhibits.

o Contains complete instructions for each process method so that the unavailability of source documents will not, impede the conduct of

a CTEA.

o Provides examples or illustrations of the applications of process

methods.

GENERAL RECOMMENDATIONS

Results of the CTEA literature review, synthesis of methods and guidebook

preparation lead to the following recommendations:

o Program TEEM, TCA, and the Litton Cost Model. before the CTEA

Performance Guide is released to CTEA analysts. Be sure that

programs, suitable machines, and reasonable interfaces with CTEA

analysts are provided.

o Develop and adapt one or more of the automated Air Force methods

discussed above for Army Training Developments and CTEA.

o Develop means, including explicit contractual requirements, to

assure that all developmental data needed by CTEA analysts become I •available in timely fashion.

o In CTEA study directives, include specifications of CTEA timing

or phasing in conformity with the CTEA locations described in

* Section iI of this report.

o in CTEA study directives, specify CTEA requirements on the basis

of a realistic appraisal of the actual status of the acquisition

program of interest rather than oy reference to the LCSý14.

.4

S~-5-

t7
are realized 

These powerful 

by 

CTEA and acquisition process methods of the future, it they

promises 

the Army, will probably all be computer based. This feature

but aliust 

not only greatly enhanced efficiency in the processing of infomation certainly a degree of precision not now achievable. It is possible that 

could 

such 

both 

an 

increase 

increase 

the 

in 

effectiveness 

the efficiency and precision of information prooeming of systems and reduce the time required to acquire them.

41

A

-6I- !j

-6-
SECTION I

INTRODUCTION

The acquisition process for military weapon systems moves through several

phases beginning with the formulation and examination of alternative concepts

for achieving desired ends and finishing with deployment of a system with demonstrated capability to achieve those ends at an acceptable cost. That is, the

acquisition process ends with such a system if at every step the right decisions

are made. Making the right decisions depends on timely and valid information

including information concerning cost and effectiveness of training programs to

support weapon system operation and maintenance. Cost and training effectiveness

analysis (CTEA) provides that training system information.

CTEA have long been performed for existing weapon systems toward the same A

purpose: organization and presentation of information to enable decision makers

to select from among alternative training approaches those that increase manmachine system performance within cost constraints. Weapon system training ef-

'fectiveness analysis identifies training problems and proposes solutions to

those problems.

The Arm,'s Life Cycle System Management Model (LCStM), the process by which

the Army materiel systems are acquired, was designed to ensure that all aspects

of the weapcn system are considered during the acquisition process. These aspects include integration of user and trainer requirements and developments with

the parent weapon system development. Moreover, it was intended to allow the

trainer to consider cost-effective training options before final weapon system

fielding. However, the emphasis in the LCSMM has been on events concerned with

the parent system.

It is essential that the personnel and training impacts be known earl-.

Therefore, questions of estimated training program effectiveness and estimated

cost need early answers. The answers are highly sensitive to changes in the

development of the parent system. As with the parent system, testing is needed

for trade-offs between training on operational equipment, training simulators

and devices, and other training issues. Frequently no such testing of simulation, training equipment, or training programs is accomplished. When testing

is done, it occurs late in the cycle or after the system is fielded. Late

testing fails to ensure cost-effective training: the user should be confirming

the effectiveness of training during late stages, not starting its assessment.

The training developer lacks guidelines and procedures for the concurrent

develo;ment of training programs with the development of the parent system. The

problem, then, is to.furnish a complete, detailed roadmap on the best way to integrate training requirements into the materiel development cycle and provide

tools for trade-off decisions at each critical training program decision point.

The purpose of this research is to provide the CTEA analyst with tools to organize, select, and analyze data for decision making.

C'EA methodology in the LCS*1 must adapt to an array of data types that vary

in precision, accuracy, ana completeness. Some CTEA methods require empirical

performance data, others human faotors information, and still others require only task descriptions. These three CTEA styles represent a continuum in data

I-

.................- .. • •
requirements. CTEA methodology applicable in the LCSMM cannot be drawn exclusively 

types of 

from 

data 

any 

available 

one of these 

throughout 

styles. 

the 

It 

LCSMM 

must 

and 

synthesize 

responsive 

methods 

to information 

suited to the 

needed

1

for decision making. MTEA in the LCSMM needs methodology capable of replacing

input data at successive stages as more definite data become available. The

CTEA methodology must be iterative and usable as a tool in cost-effectiveness

estimation and in hypothesis testing. At each application, tra&.ning requirements must be forecast, alternatives that offer probable cost-effectiveness

selected, and results of prior estimations tested to gradually refine and improve training.

CTEA is a method for defining training alternatives, comparing those alternatives, and defining the proficiency obtainable with each training-hardware

combination. That is, CTEA must be:

o prescriptive - identify training program elements likely to be

effective;

o diagnostic - identify weaknesses in training programs; and

o predictive - define the probable effectiveness and cost of training

programs.

CTEA methodology, therefore, is more than a means for differentiating among

alternatives (the hypothesis testing furction). At earl,, stages of the acquisition cycle, CTEA methods need to identify existing data to define training options. In this forec.asting role, CTEA predicts an array of training alternatives.

CTEA must reveal training deficiencies and recommend solutions. In late stages,

CTEA methods must continue forecasting and diagnosis and must differentiate among

alternatives. Changes in threat, tactics, and operations continually influence

system performance. CTEA methodology mu,,st be capable of continually assessing

these impacts on effectiveness, identifying corrective options, and guiding

modifications.

In summary, the objectives of CTEA in the LCSMM are to:

o Ensure that training processes are initiated early in the LCSt9

and are accomplished in parallel with combat developments

processes;

o Ensure that a training subsystem is developed wi(h the same

degree of scientific application as is the parent hardware

system;

Ensure that training system alternatives are considered; andSo 

o Provide decision makers with information at critical points

concerning the training and hardware system.

"Research Objective and Overview

The objective of this research was to provide Army analysts with a performance guide for CTEA at each stage of the LCSS$. Figure I-1 represents an overview of the conduct of this research. A description of the acquisition process,

1-2

I _, .. 772.i
Description of Aev10w Of existing Acquaitiuon eeowtf -- Methods applicable vro efc

Determination

* '~tCOf

locations in

Identification Deveiorxlent or Developmn~t of

inputs/propess method meho (sodtesis)oy(aayss

DemonstrationM stccdrateies~.n

of process of Process wethods -methodis and (aul

strategies

4 ~Figure 1-i1

OVERVIEW OF CTE.A ESEA.RCH

1-3
the Army's LCSMM, was first developed. From the description of the LCSWt were

derived locations of CTFA. The CTEA locations imply the probable input and output information of the CTBA, such as CTS output required for Coat and Opera- ticnal Effectiveness Analysis (COEA). Existing CTEA methods, or methods applioable to CTEA objectives, were reviewed to determine their suitability for CTEA

during weapon system developuent. In addition, CTEA methods were developed for I

areas that existing methodology failed to cover adequately. The result is a

general CTEA model that synthesizes a family of CTEA methods, each of which is

applicable at a different point in the LCSMM. The methods that appeared most

applicable were selected for inclusion in the user's manual. The selected

methods were written in procedural terms and were demonstrated using examples

gleaned from Army analyses of developing weapon systems.

Organization of the Report

Section II of this report contains an overview of the LCSMM. an examination

of the requirements and purposes of CTEA within the LC1MM, and recommended locations for CTEA in the LCSIM. This section defines the problems of CTEA for

which solutions are proposed in the following section.

Section III summarizes existing CTEA and CTEA-applicable methods.

Section IV presents methods devised by Litton during the present research

to aurment existing methods.

Section V discusses a general CTEA model for use in the LCSMM and synthesizes the material selected for the user's manual. It examines the constraints

on employment of CTEA methods, and explains CTEA strategies.

Section VI offers conclusions and recommendations. It introduces a companion document, The Cost and Training Effectiveness Analysis Performance Guide, which contains procedures for CTEA in the LCSM and examples of their use.

'1

1-4:
SECTION II

COST AND TRAINING EFFECTIVENESS ANALYSIS IN THE

LIFE CYCLE SYSTEM MANAGEMENT MODEL

The process through which Army materiel systems are acquired is described

by the Life Cycle System Management Model (LCSMM), expressed in a 119 event

flowchart (Department of the Army, 1975). LCSMt outlines procedures for the

development and acquisition of Army systems from concept investigation through

ultimate disposal of obsolete systems. It covers coordination of combat development, research and development, production and logistic support, training and

personnel requirements, and actions required to develop and maintain the system.

The Integrated Personnel Support (IPS) system is the process by which personnel considerations are integrated into the LCSMM (Department of the Army,

1'978). The goal of IPS is ,o ensure that personnel resources are planned, developed, acquired, tested, and deployed in conjunction with the materiel acquisition process. Personnel resources include the number and characteristics of

personnel required to operate, support, and maintain the system, their training,

interface with the hardware, human resources development, and other personnel

factors.

The military services conduct continuing analyses of the threat and associated friendly mission areas, including military mission needs, capabilities,

resources, and technology. _ased on their analyses, the services prepare mission

element need statements (MENS) for approval by the Secretary of Defense to Justify major new system acquisition. MEN4S contain a "constraints" section for the

manpower and resources for the proposed system. Minimization of the number and

skill requirements of people needed to operate and support the proposed system

is a goal of the planning. Thus, personnel estimates such as the number, skills,

training, and occupations required are part of the MENS.

The number and characteristics of personnel estimated for the system influence task and training subsystem design. The training must accommodate the

number of personnel, their aptitude levels, and prior training. The type of

training depends upon the nature of the personnel who are to be trained.

Approval of the MENS by the Secretary of Defense authorizes the Army to

initiate the LCSMM. The process consists of four major phases: Conceptual,

Demonstration and Validation, Full-Scale Development, and Production and Deployment. Specificity of the materiel design, personnel considerations, and training vary with the LCSMM phases. Early in the cycle, such as the Conceptual

phase, personnel estimates and information on training effectiveness are gleaned

from engineering data and general principles of human factors and instructional

design. In later phases empirical data are available from developmental and

operational tests. During deployment, cost and effectiveness data are collected

in field tests. The data increase in accuracy, objectivity, scope, and completeness as the acquisition cycle proceeds.

Several LCSMM events require CTEA information and are influenced by CTEA.

Figure Ii6- shows the LCSMM locations that produce new or revised input data for

CTEA and that require CTEA. For example, four cost and operational effectiveness analyses (COEA) are conducted, one in each Tajor phase. CTEA support COEA

i !_'I-1 i
04.C 4141 svot" COIITOM INVtCW9

"lbTm ewt 4111414" 111,1o

111,10111" f9^LT0 Otw L ot R a4G496te~ gig".i

Ae gYO 191391111=~ gqt~u&. gs fayS4 Witha ~ ASA l~ tE A OVhTn SISC C Ii

*ISAMP ?1 AI~I?& t.I*"O.I

piI' R

40R~tA

- 7 A' AO 111 tv

£01DA'. IRDNTN8~ 

IN L FT%% s£

Figue ~~fl lIi. he SstemAcqusitin P OCes

C144 ~ ~ Ott ~ __ i
by providing assessments of alternative ways to train to achieve the desired

operational effectiveness of the system (Department of the Army, 1977). At -*

least four CTEA are needed to support the COEA, and, according to the Army regu- lations cited in this section, additional CTEA are beneficial to support training development decisions.

If CTEA locations in the LCSW are bound by events that provide new or

revised data to M.1A and by events that require, or are influenced by CTEA, then

six CTEA are recoeuended, as shown in Table II-1. Two CTEA are recommended

during each of the first two phases, Conceptual and Demonstration and Validation. One CTEA is reaxwmended during each of the two later phases, Full-Scale

Development and Production and Deployment. The events that bound the CTEA and

reasons for the recommended CTEA looations are described in the remainder of

this section.

A. Conceptual Phase

Approval of the MENS by the Secretary of Defense authorizes the Army to

initiate the Conceptual Phase. This phase establishes the technical, military,

and economic bases for the proposed system and formulates concepts through research and the development of experimental prototypes. It includes the solicitation, evaluation, and competitive exploration of alternative means of achieving

the mission.

Several requiremerts documents are prepared during this phase that need

information concerning personnel and training. An early training requirements

document is the Task and Skill Analysis (TASA) that describes roughly what the

people need to do to operate and maintain the system. Ideally, the TASA contains three categories of tasks: machine functions, human functions, and shared

functions. Critical tasks are identified in the human and shared function categories. The critical tasks are likely to require formal training and they guide

the training support plan. At this early stage the training support plan focuses

on training factors that influence hardware and training device designs to anticipate potential trade-offs. The Outline Individual-Collective Training Plan

(OICTP), an element of the training support plan, summarizes expected courses

of instruction, changes in existing courses, and training and performance aids

for the new system.

Personnel requirements are based on the rough TASA. They include estimates

of the number of personnel required, the skills needed, unique mental and phy-ical requirements, and human resources development factors (e.g., morale, organizational climate). Personnel requirements and savings are included as part of

the prospective operational effectiveness and costs in the Letter of Agreement

(LOA).

The LOA is a document jointly prepared and authenticated by the combat developer and the materiel developer. It outlines their basic agreements for advanced development investigations of the proposed system. Agreements and investigations cover operational, technical, personnel, training, and logistic

support concspts. Army regulatioyis demand quantitative estimates in the LOA of

operational effectiveness and cost of the new system (Department of the Army,

1979). Thus, cost and effectiveness irformation is required preceding the LOA.

11-3 I- _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Table 11-1. CTEA in the LCSMLM

PRIMARY LCS?44 EVENTS ISSUES REQUIRING

LCSl'E4 YIELDING DATA TO RESOLUTION

PHASE CTEA CTEA

"M L4NS e TRAINABILITY OF

" INITIATION OF TRAIN- BASIC CONCEPT

I ING PLANNING o COST OF TRAINING

" INITIATION OF LOGISTICS * TRAINING PROGRAM

SUPPORT PLANNING ELEMENTS TO BE INCONCEPTUAL CLUDED OR STUDIED

9 LOA o TRAINABILITY OF

o ORGANIZATIONAL AND ALTERNATIVE CONCEPTS

I A OPERATIONAL CONCEPTS e RELATIVE COST

(update) EFFECTIVENESS OF

TRAINING PROGRAMS

OF ALTERNATIVE

._ _CONCEPTS _ 

• OAP a TRAINABILITY

D DT/OT I o NECESSARY REVISIONS

II OF TRAINING PROGRAMS

T)EMONSTRATION o RELATIVE COST

AND EFFECTIVENESS

VALIDATION

k e PQQPRI * TRAINABILITY

L o DCP(IPS) 9 TDR

o RELATIVE COST

II A EFFECTIVENESS OF

(update) REVISED TRAINING

.PROGRAMS

9 NET PLAN 9 PERFORMANCE VERSUS

FULL-SCALE * AP STANDARDS

ENGINEERING III o DT/OT II e PERFORMANCE VERSUS

DEIELOPMENT HARDWARE AND TRAINING

PROGRAM DESIGN,

PERSONNEL SELECTION,

'__ETC.

PRODUCTION 9 TRAINING PLAN UPDATE I COST EFFECTIVESS OF

AND IV * DRAFT TRAINING REVISED TRAINING

DEPLOYMENT (upda e) PROGRAM PROGRAM

e AP

- e DT/OT III

!-1-

1 - -
Information in the LOA serves as a basis for ensuing documents including

Organizational and Operational Concepts, Outline Acquisition Plan, Qualitative

and Quantitative Personnel Requirements Information, and Basis of Issue Plan.

c The Organizational and Operational Concepts document describes

trade-offs between organizational and operational equipment and

personnel.

o The Qualitative and Quantitative Personnel Requirements Information

is issued first in tentative or provisional form (TQPRI or PQQPRI).

It provides information concerning the number and skills of personnel

required for the operation, support, and maintenance of the proposed

system.

o The Basis of Issue Plan (BOIP) indicates the quantity of new or

modified equipment for each type of organizaticn and changes in

personnel and supporting equipment.

The ncxt major document is the Concept Formulation Package (CFP) which

documents that the objectives of the system connept are met. The CFP describes

the technical approach, trade-offs, risks, personnel requirements, costs, schedules, and logistic support requirements. It includes the results of a COEA.

Thus, a CTEA is needew to provide information assessing alternative ways of

training to achieve the desired operational objective. Baseline Cost Estimates

(BCE) that document the cost of acquisition and operation, including personnel

costs, are developed for the CFP and are refined throughout the acquisition

cycle.

The Outline Acquisition Plan (OAP), formerly called the Development Plan,

contains plans for the management of the advanced development of the system.

The materiel developer prepares the OAP using input from the LOA, CF, and

ICTP. TRADOC, ADMINCEN, and LOGCEN assist with personnel issues that require

testing and to identify training requirements. Research on personnel tradeoffs and requirements is conducted and the results are incorporated into the

OAP.

The Conceptual phase culminates with Milestone I, the first In-Process

Review or Army Systems Acquisition Review Council and Defense System Acquisition

Review Council (ASARC/DSARC I) authority to proceed with the Demonstration and

Validation phase.

CTEA Implications

CTEA in the Conceptual phase need to provide a deliberate assessmert and

analysis of alternative means of training to achieve the operational effectiveness of materiel systems within a force. Tasks essential to mission acccmplishment with alternative system designs vary in the difficulty with which they are

performed and trained. Overall operational effectiveness of the system varies

as a result. An accurate determinai!on of operational effectiveness must be

based on the proficiency likely to be obtained with each design and the cost of

training necessary to achieve that proficiency. The CTEA should provide that

information for eacn of the materiel system design alternatives.

k L-3
A preliminary cost and operational effectiveness analysis is a portion of

the LOA. This analysis should "demonstrate that there is a reasonable chance of providing a significant advance in operational capability at an acceptable

cost" and should "indicate probable improvement in force effectiveness, probable

relative costs and the military need for the proposed system" (Department of the

Army, 1977). Moreover, it should "outline the methods and costs involved in obtaining information on critical issues for the CFP COEA" (ibid).

This preliminary CTEA is important in defining the system concept. It compares the training effectiveness and 6osts of high-risk training elements for

each of the alternative materiel systfta being explored. It identifies the

high-risk and high-cost components as early as possible in the LCSMM. It ensures a complete OICTP by providing known training concepts, strategies, costs,

and requirements. This CTEA is prescriptive, stating:

o Unique performance requirements that limit system effectiveness

because of low probability of trainability;

o Training program elements that must be included in the training

system to enable fielding of a fully effective weapon system- and

o Potential impact of training program omissions because of cost

and developnent risk.

The .TEA should project the consequences of various outcomes in a "trainability risk analysis" and their costs. Where alternative systems am, being

%considered, the cost and trainability risk figure into comparison of the alternatives. Trainability issues are:

o What task groups are exceptionally complex and constitute high

training development risk?

o What are the standards of performance for the high risk tasks?

0 Which high risk tasks require above average mental and physical ability?

o What are the difficulties in training the target population for

each alternative technical approach?

o What training approaches offer the greatest potential for solving

each of the high risk training probleo•?

o What are the high risk training support requirements for each

alternative system?

o What are the costs of the high risk training support requirements?

A variety of input data for the CTFA is likely to be available. The t-5S

specifies the military need and constraints on the manpower and resources. A

basic syste•i concept, or alternative system concepts, has oeen expressed and

the TASA and OICTP provide a rough estimate of the tasks and training summary,

including the identification of high-risk -asks, decisions on the locations for a

training, ano general statements about the type of training to te'Conducted.

TI-6
The output of this preliminary CTEt needs to define the trainability of

each alternative given the outline tra.naing program, provide the cost of training for each BCE, and provide recommendations for training program elements to

be included in the OICTF or examined in research. It should state the training

issues to be examined and provide guidance for detailed front end analysis for

training program refinement.

An update of the CTEA is recommended as input to the COEA that supports

the CFP. The CFP is the culmination of research designed to identify alternative means of satisfying the needs set forth in the MENS. The materiel developer

performs engineering analyses, identifies technical approaches and their support,

technical and logistical concents including the provisional QQPRI. The combat

developer formulates the BOIP, and together the materiel and combat developers

formulate the Organizational and Operational Concepts. The CFP is a compilation

of these documents that consider materiel and personnel issues, screening of

alternative systems concepts, trade-off analyses, and a COEA.

The CTEA. at the CFP stage addresses the relative effectiveness of alternative training programs. That is, given the proposed training program for each

alternative materiel approach, is it likely that desired weapon system operational capability can be achieved? The alternative with the highest trainability

estimate is, from the point of view of training effectiveness, the best technical alternative.

The CFP identifies those system concepts deemed suitable for demonstration

and validation. The concepts represent a single option, alternative systems, or

alternative subsystems. The options may include program improvement recommendations for existing systems, purchase of systems from other services or other I

countries, or new development. The CTEA methodology must deal with all of these

contingenc;.-es.

Training effectiveness analysis at the CFP Stage must consider the probability of training success given current knowledge concerning the efficacy of

various training methods for various types of tasks. The CTEA should appraise

the probable effectiveness of the training programs proposed for alternative

system concepts, identify the strengths and weaknesses of the programs, and

supply cost information for comparisons.

Data available include the background information for the CFP: the MES,

Organizational and Operational Concepts, CAP, QQPRI, LOIP, LOA, and input on

operational costs and effectiveness for the COEA. The data may include results

of research on issues, front-end analysis of prototypes, and an update of system functional requirements. The TASA way oe updated, and high-risk tasks may

be defined. Required output from the CTEA includes information on the cost and

trainabil.ity of alternative systems to support the selection of the best technical approach and recommendations of ways to enhance training effectiveness

of selected system alternatives.

In the Conceptual phase, two CTEA have been recommended. The f.rst precedes

the LOA and provides initial estimates of costs and effectiveness of training based on very rough task statements and persomnel estimates. The second uses

the information compiled for the LCA and additional information prepared for the

CFP. These CTEA are required in the ideaJ. case. In reality personnel and training considerati3ns lag behind hardware development. Thus, task and training data

II-7 "
may not be updated between the LOA and CFP and as a result there may be no new

data for the second CTEA. This discussion, however, presents the ideal case

since the goal is to provide tools to perform the CTEA as they should be done.

B. Demonstration and Validation Phase

A positive Milestone I decision allows the Army to proceed with the Demonstration and Validation phase. This phase consists of the steps needed to

resolve conflicts identified during the Conceptual phase, verify preliminary -. designs and engineering, accomplish necessary planning, fully analyze trade-off

proposals, and prepare the contracts required for fill-scale development. This

phase includes the use of developmental prototypes in development and operational

tests (DT and OT).

Advanced development is conducted by one or more contractors. Personnel

constraints determined in the Conceptual phase are a required part of the contracts. The goal of including these constraints is to develop hardware that

average soldiers can operate and maintain. The contractors are required to provide a Task and Skill Analysis (TASA) for each operator and maintainer.

The system proponents determine critical tasks from the contractor's TASA.

The proponents also evaluate training and training device requirements and estimate whether the operators and maintainers will require new military occupational

specialities (MOS), new additional skill identifier (ASI), or modification of

existing MOS or ASI.

The personnel task and skill evaluation information is used to develop the

personnel and training factors and criteria for DT/OT I, including the skill requirements, mental and physical demands, behaviors, high risk tasks, needs for

training development and training materials, and human resource development.

The results of DT/OT I are used to update the Outline Individual and Collective Training Plan (OICTP), Provisional Qualitative and Quantitative Personnel

Requirements Information (PQQPRI), initial unit structures, and tentative Basis

of Issue Plan (BOIPT). Update of the training support plan includcs descrip- tions of training devices, training methods and media, training extension courses,

soldiers' manuals and trainer's guides, field manuals, job performance aids, and

other performance guides.

The results of DT/OT I are also analyzed to determine logistic support

problems and to assess total life cycle costs. The integrated logistic support (ILS) requirements are revised and alternative logistic support concepts are

examined. The estimates of life cycle costs for each alternative system and

the resource requirements of the preferred alternative are presented in annexes

to the Decision Coordinating Paper (DCP) prepared for ASARC/DSARC II.

One of the documents that supports the DCP, the integrated program suni.ary

contains manpower and training implications. The manpower implications cover

trade-offs among hardware designs, differences in manpower for old and new systems, manoower sources, new occupations, job task identification, and planned

tests and evaluations to verify the manpower estimates. The training implica- tions cover plans for reaching and sustaining the required operator and maintainer proficiency, plans for formal, on-the-job, and urnit training, number of

personnel to be trained, training costs, and training requireiernt surniaries by

fiscal year.

. .- *
The TRADOC propc ent develops the Required Operational Capability (ROC) or

Letter Requirements (ER) thst include personnel considerations believed to have

impact on further devE lopment of the materiel system and personnel support.

These considerations :nolude training and training device requirements, personnel

interface with existiiLg and proposed equipment, system safety, and human engineering factors.

The Acquisition ?lan, prepared by DARCOM in this phase, covers such personnel and training requirements as new skills, individual and crew training requirements, trainirg devices, training facilities, and technical documentation.

I

Personnel estizwtes are used to develop the Initial Recruiting and Training Plan which indicates personnel and training activities that must precede

deployment.

The DCP and other documents are reviewed in ASARC/DSARC II upon completing

advanced development.

CTEA Implications

Two CTEA are recommended during the Demonstration and Validation phase, both following DT/OT I and before the ROC. The second is a revision based on information gathered for the PQQPRI.

During DT/OT I prototype systems are demonstrated to show that all technical risks have been minimized and that a working prototype exists. Prior to

DT/OT I the training• program for high-risk tasks is used to train the test,

participants. If ii: is not used or is used so that insufficient data can be

derived about effectiveness (e.g., special troops used, no control over training,

etc.) then this CTEA resembles a CTEA in the Conceptual phase. If field data

do exist, however, the CTEA serves as an analysis of the training programs to

verify estimates of trainability based on the effectiveness of the training.

Estimates of system effectiveness degradation caused by training problems are

also identified.

The COEA in this phase is designed to support the ROC by assessing the

attainability of operational effectiveness for the alternative system designs.

The CTEA has that fanction but in addition diagnoses and prescribes training.

It uses the lessons learned during DT/OT I to revise projections of training

programs, training device requirements, and the OICTP.

The issues add.ressed at this phase involve the high risk tasks and more

detail about personnel and training:

o What are the relative merits of the alternative system approaches

to training on the high-risk tasks?

j What is the estimated frequency of training needed to maintain C proficiency on the high-risk tasks?

o What devices are needed to support institutional and unit training

on high-risk tasks?

11-9

A. ... . y I.•t••.+ I ,
o What are the system training support requirements and associated

costs?

o What hardware/training trade-offs are required to reduce training

difficulties?

o What is the impact on current training and evaluation literature?

o What are the test and analysis requirements for the engineering

development phase?

Ideally, input data from DT/OT I indicate operator and maintainer performance cn prototype systems after training and include performance errors, prob-.

lems in the training programs, time to train, and training program costs. Outputs of the CTEA, given those input data, include the training program strengths

and weaknesses, recommendations for improvements, and an update of the cost and

trainability estimates.

The second CTEA recommended in the demonstration and validation phase folDT/OT lows the PQQPRI and precedes ýhe ROC. Personnel requirements are revised after I and the requirements document is staffed through appropriate Army headquarters. The CTEA at this point considers the test results plus training program revisions made as a result of DT/OT I. The ROC that it supports through

the COEA, is the final statement of the system capability the military wishes

to acquire. The COEA states the best cost estimate and reaffirms the force

effectiveness to be achieved given the DT!OT I results.

The CTEA should provide the training requirements data that form part of

the ROC. It should address those issues resolved by test data and provide a

final assessment of the trainability of the system concept. It should also

provide specific training program goals to be met in the final system. The goals include statements of program elements (e.g., simulator design character- •i

istics, media) as well as statements of the desired levels of training effec- tiveness (e.g., transfer of training, minimum acceptable levels of post-training

skill reteation).

In addition to information in the PQQPRI, data available to be used in the

CTEA incude DT/OT I data and revisions to training programs. Outputs are training requirements to ensure effective training (for the ROC) and assessments of

training program effectiveness. 4

The second CTEA in the Demonstration and Validation phase is a direct update of the preceding CTEA. The analyst needs only to alter input to the CTEA *1

model that represents revision in training programs after DT/OT I.

C. Full-Scale Engineering Development Phase

During the Full-Scale Engineering Development phase a system, with all

items necessary for its support, is fully developed ard engineered, fabricated,

tested (DT/OT II), and initially type-classified. Concurrently, non-materiel

aspects required to field an integrated system are refined and finalized.

DT II evaluates the technical capabi lity of the materiel and support systems including personnel and training requirements. DT II results are used in I

%the _=ecision to initiate full production.

11-10:
OT II tests the engineering prototype to estimate its operational effectiveness and suitability in a realistic military environment. OT II is designed

to employ personnel of the MOS required by the system and to test the technical

manuals, training devices, other training material, personnel requirements, and

support concept. Instructors trained by the contractor train operators and support personnel for OT II using the training programs developed for the system.

LCSMM documents are revised using the test results. The training proponent

determines the changes in the Army Training and Evaluation Program (ARTEP) for

the new system. Also, the trainer drafts field manuals containing doctrine,

organizational material, and training evaluations.

in The p-,posed system, with prototypes, is analyzed to determine enhancements

in reliability, maintainability, support concepts, and logistics. The logistic , support analysis contains qualitative and quantitative requirements for support

and test equipment, personnel, training, and facilities. The logistic support

information is issued in equipment-oriented publications such as technical manuals. Proponent service schools coordinate logistic support analysis records

(LSAR) summary sheets, training and personnel doctrine, draft equipment publications, and organizational structures.

Documents revised on the basis of DT/OT II results are the training plan,

the Basis of Issue Plan - Complete (BOIPC), and QQPRI. Considerations at the

stage of the final QQPRI are:

o Sufficiency of training to provide required proficiency.

o Appropriate MOS and ASI.

Flo Inclusion of all system components and subcomponents in the

*o Adequate numzber of trained personnel to support the system.

•QPRI documientat ion.

At Milestone III a draft decision coordinating paper (DCP) indicates the

status of the system after DT/OT II and refinements of requirements. The DCP

includes the Integrated Personnel Support (IPS) annex that summarizes the Implementation plan for the accuisition cycle of the system. It covers changes in

manpower estimates from Milestone II, effects on manpower of reliability and

maintainability levels, shortfalls in meeting manpower requirements, new occupations not yet approved and programmed, and future plans for evaluating manpower requirements. The IPS also presents summaries of forma' training requirements

by fiscal year and occupation, and plans for the following: attaining ard maintaining proficiency required to operate and maintain the system, additional resources required to train the initial set of personnel, training reserve compo- SI " nents, and validation of proficiency and performance.

ASAC/DSARC III authorizes full-scale development or limited production of

major systems. The item is type-classified as standard if all critzcal issues

wer'e resolved in developnent and operational testing. Limited production is

directed it' further testing is necessary.

U-11

~ ~ - **-..- 11*
CTEA Implications

The CTEA in the Full-Scale Engineering Development phase is recommended

after DT/OT II and preceding documentation for ASARC/DSARC III. This CTEA

assesses training costs to update earlier cost estimates, this time based on

experience with total training requirements. In addition, it evaluates the

match between training requirements and performance. It is also diagnostic

and prescriptive to improve weak training programs. 4

The issues to be addressed are:

o How effective was the training program in teaching the required tasks?

o Which tasks were not performed adequately during OT I! because

of poorly designed hardware, poorly designed training, insufficient

training, or improper personnel selection? "

o What are the impacts of training related deficiencies on operational effectiveness?

o What additional training resources are required to attain the

system performance objectives?

o What are the requirements for additional training development

and testing prior to full production?

Data input for the CTEA at this stage are from DTiOT II, and the output

is the CTEA assessment of the training package. This CTEA provides information

for the COEA conducted in preparation for ASARC/DSARC III.

D. Production and'Deployment Phase j

During Production and Deployment equipment is prciured and distributed,

operational units are trained, and logistics support is provided. Acquiring

and training the personnel demands that the final MOS decision be rendered and

the manpower secured, the TOE be approved, personnel requirements and training

schedule, new equipment training, including technical documentation, be com- I

pleted, and the ICTP be approved. Training of personnel needs to be started

in sufficient time to support the ncw system in the field. Resident training 3

is based on revisions of training plans. It requires the training equipment,

devices, and cther training materials.

Unit training begins when the units receive the system and it must be com-r

pleted to achieve the initial operational ,,apability. The training is guided by the ARTEP to include critical tasks for the system's wartime mission and is

conducted by the cadre who received new equipment training. To achieve opera- tional reaciness the operator and maintenance personnel in the unit need to be

fully trained. When the unit has the capability to perform the mission the goal

of' initial operational capability is achieved.

The Army LCS•kN allows the possibility that a Milestone III decision authorizes limited production with a follow-on review for full production and de-;-

ploymont. Thus, the Army has procedure to review the system at a later date,

11-12
ASARC IIIA and DSARC 11A. Reviews beyond DSARC III require individual oonsideration and the reviews vary with the situation.

DT III is conducted on initial production-run items to verify that the

items comply with specifications. DT III determines if transition from an

engineering prototype to production succeeded.

OT III is conducted on initial production-run items to estimate their

operational suitability. The test covers tactics, training, supportability,

and organization.

The final QQPRI is prepared eighteen months prior to deployment. It reflects all changes in design and organization and the results of the DT and OT.

It supports the final MOS decision.

BOIP II describes the TOE organization; support equipment for the system,

where and hcw much will be deployed, and the personnel implications. The BOIP

is used to justify the distribution of the system until the new TOE and IDA

4 • are prepared.

The acquisition plan (AP) records program decisions and analyses of technical options and life cycle costs for development, testing, production, training support, and logistic support of the system incorporating the DT and OT

results.

The Decision Coordinating Paper (DCP) at the IIIA stage contains inaformation for the optimum production quantities of the system that are consistent

with readiness and modernization. The annexes, including the Integrated Program

Summary (IPS) are prepared for the ASARC and DSARC. This IPS contains all updated information from tests and experience including changes in manpower and hi training requirements. The resources funding profile provides the costs of

*) operation and maintenance, personnel, and support for each alternative.

CTEA Implications

Given DT/OT III data, the final CTEA, conducted to update estimates, supplies information for ASARC/DSARC IIIA.

E. Constraints on CTEA Methods

Constraints on CTEA methods derive from the location of the CTEA within

the LCSI, the availability and quality of data, the background, skills, and

numbers of personnel available to perform CTEA, and the length of the timeframe within which a CTEA must be performed.

'1. CTEA Location. The principal constraints imposed by virtue of CTEA

location aris-efrom the absence of empirical effectiveness data on new system

tasks before DT/OT I. Tasks of other (fielded) systems may be used to estimate the effectiveness of the training of the tasks of the developing system,

but this approach lacks the power to discriminate among training program alternatives that may be obtained through empirical effectiveness data for the new

system. Thus, CTEA performed before DT/OT I must have methods for estimating

effectiveness. If they are performed before training program alternatives have

11-13
been described, they must also have methods for predicting training program alternatives. Since the prediction of training program alternatives usually requires some sort of a task list, the CTEA must have methods for generating task

lists if none are available. After DT/OT I the empirical effectiveness data

situation changes. Because the concern of DT/OT I is the demonstration of the

validity of alternative materiel concepts, there will likely be no training program and thus no comparative effectiveness data. Thus, even after the Demonstration and Validation phase has begun, the CTEA must have the methods of the

earlier Conceptual phase CTEA. After DT/OT II there still may not be enough

empirical effectiveness data to answer questions; it may be necessary, for example, to consider a training program alternative not predicted in advance of

DT/OT II and therefore not tested. Thus, even after DT/OT II CTEA may require

the methods of the earlier concept phase CTEA. Variability in CTEA location or

timing results from variability from one acquisition program to another. Data

other than task lists, training program alternatives, and effectiveness data

are also available at different phases and events of the LCSI1.

2. Availability and Quality of Data. Because any particular acquisition

program can be expected to conform to LCSMM only in an approximate way, it can- I

not be expected to consistently yield to CTEA the data implied by the LCSM4.

Thus, while the LCSM as described by Rhode et al. (1980) yields Organizational

and Operational Concepts in time to provide input to a CTEA located to provide

input to the CF?, an actual acquisition program may not yield such information

ýil much later. Likewise, an actual acquisition may not even yield such

fundamental CTEA input data as task lists until very late in the acquisition.

!At least one such situation has occurred.) When data identified or implied

oy the LCSMM are available at the times indicated by the LCSMM, they may not

be complete or may otherwise represent an inadequate quality. Task lists, for

example, may be only for maintenance tasks, requiring the CTEA to have methods

for the generation of operator task lists. Task lists may be grossly in error,

making i.lt necessary for the CTEA to proceed as if there were no task lists at all.

The essential point of constraints arising from the way in which actual

ac,•,l Uion programs are implemented is that the selection or development of

CTEA ;hods is driven essentially by the available input data.

. Available Personnel. In the course of normal career assignments, senior enlisted personnel, warrant officers, and officers can all be expected

to be tass' ", with the conduct of CTEA. Personnel available for CTEA must therefore be e. )ected to vary a great deal in background and skills. CTEA methods

vary in complexity, required expertise, and difficulty. Some are not complex

methods require considerable expertise in training development for successful application. Others require less expertise but are much more complex.

4. Available Time. The time available for the conduct of a CTEA appears

to be on the order of about 120 days. The importance of this constraint would

to some extent vary with the number of persons available to perform a CTEA, but

the usual CTEA staff seems to consist of about two or three. Some CTEA methods

may not be consistent with such constraints. The use of an analogous task, for

example, is not complex and does not require a great deal of expertise, but because analogous tasks must be located (perhaps through an extensive search of

the documientation of fielded systems), it could require more time than is available. Methods available for the conduct of CTEA must also recognize these

constraints.

:1-14
Because of the nature of constraints acting on CTEA, the process

methods that any particular CTEA must make use of cannot be predicted on the

basis of the location of the CTEA within the LCSW. Rather, methods must be

tentatively selected on the basis of the situation which a particular CTEA encounters with regard to available data. Tentative selections must be further

constrained by considerations of available personnel and time limitations.

F. Sumuary

To the extent that training produces or maintains %3ssion-oriented skills

required system effectiveness is obtained. Tasks essential to mission accomplishment with alternative designs or concepts vary in the difficulty with

which they are trained, so overall operational effectiveness also varies. An

accurate determination of operational effectiveness must be based on the maximum proficiency obtained with each design and the cost of training necessary

to achieve that proficiency. The CTEA should provide that information. The

CTEA methodology is iterative and is updated as new information becomes available. The CTEA results are increasingly detailed and accurate as the life

"cycle of the system progresses. IA

1 i

I -J

Iii

;[•{'¢'•'• '5 '• •11- 
SFCTION III

COST AND TRAINING EFFECTIVENESS ANALYSIS METHODS

The literature of training development describes a number of methods that

are either CTEA methods r 30 or applicable to sume aspects or processes of

CTEA in the LCSMM. While some of these methods have been developed for application to developing systems, others assume fielded systems, and none embraces

all of the requirements of CTEA in the LCSW. A purPose of this research was

to organize the varieties of CTEA and related methods so that the analytical

needs of each CTEA location in the LCSMM could be met. What was envisioned was

the synthesis of a family of CTEA methods applicable to the LCSMM rather than

the development of a new method which would meet all needs.

To establish a basis for the synthesis of CTEA methods suitable for the

variou& stages of the LCSWM, all seemingly relevant CTEA methods described in

the current literature have been e'Camingd. This examination of existing methods

has attempted to answer three principal questions:

o To what extent does the analysis described represent a true

CTEA method in the sense that given precisely defined inputs

another analyst could derive precisely defined effectiveness

and cost output?

o To what extent does the analysis, or the purported CTEA method,

embody processes usable within the contexts of a variety of

CTEA designs?

o For which stage of the LCSMM, if any, would the purported

method or its embodied processes be most suitable?

Each method was examined in light of the needs of CTEA in the Azrny LCSM4

regardless of its purported purposes or established uses. The focus was on the

details of processes rather than the overall conceptual scheme. The sumnaries

of methods thus developed established the basis for the effort to synthesize a

family of CTEA methods.

Since the DOD directives containing guidance on the LCSMM were issued, many

studies have been published documenting from various perspectives the costs and

effectiveness of training systems for developing and operational systems. They have been variously approached as cost effectiveness analyses (C/E), cost and

operational effectiveness analyses (COEA), cost and training effectiveness analyses (CTEA), and weapon system training effectiveness analyses (WSTEA). These

assessments have been performed at different stages of the LCSWM, some fully

supported and iterated as required and others given considerably less time or

Sresources.

The summaries of CTFA and related methods that follow represent those that

appear most applicable to the CTEA now being conducted as an aspect of the Army's

materiel system acquisition processes but by no means do they constitute an ex- ¶

haustive catalog of methods that might be applied. Jorgensen (1979) has identified some innovative methods and techniques with PossLble useful applicatiors

within training developnern and CTFA that are not included here. Nor have all

S --
of the methods and approaches summarized below been included in the CTEA Performance Guide since some will require additional study and development (or revision)

to maI:o them useful and feasible as CTEA methods.

Most of the following methods and CTEA approaches are listed under coamonly

used names and acronyms. Formal bibliographic information is included within

parentheses.

1. TEEM (Jorgensen and Hoffer, Prediction of Training Programs for Use in

Cost Training Effectiveness Anaysis,= 19T97)

The current acquisition policy requires effectiveness comparisons of the

training alternatives of developing weapon systems. In the early stages of development, however, these systems have no training programs to serve as a basis

of comparison. In the case of truly innovative systems, furthermore, there are

no analogous systems with training programs that could provide base cases. A

predictive computerized model for CTEA use has thus been developed by Jorgensen

and Hoffer (1978). Their Training Efficiency Estimation Model (TEEM), begins

with task analysis, proceeds through selection of training media and methods,

identifies information content and structure, and generates a cost/effectiveness

ratio. Iterations of the model for various training system options produce cost/

effectiveness ratios for comparisons of the options.

In TEEM trade-offs are recorded during the generation of an estimated

training program. These trade-offs estimate the potential decision cost for

the chosen training program compared with an ideal program with no constraints.

That is, an ideal, unconstrained program is quantified and compared with a similarly quantified real-world (constrained) training program. This measure provides a standard of comparison for several training programs generated with the

same CTEA method and has potential advantages not only for the comparison of

alternative programs for a given weapon system but also for comparison across

weapon systems.

The measure, the efficiency ratio, represents a value composed of the

efficiency score of an estimated program with real world constraints divided by the efficiency score of an idealized program with no constraints. Outputs of

this method include a cost-analyzed training program. Input requirements are

task list plus sufficient knowledge of the weapon system to permit inferences

about the nature of stimuli, responses, and feedback.

The heart of TEEM is a set of variables -- called a "metalanguage" --

used to describe tasks to be learned arnd the means (media and methods and, perhaps in the future, content and content structure) of learning them. When each

variable in the description of the task is matched by the corresponding variable

in the descriptions of the training program, an ideal training program is identified. That is, for each stimulus, response, and feedback need presented by the

tasks there is a medium-method combination in the training program to meet the

need. Because of costs and other constraints such an ideal training program

cannot be achieved. Rather, as the result of designing a training program within

real-world constraints something less than an ideal program is actually achieved.

The difference between the real-world program and the ideal is expressed by the

efficis.ncy ratio.

ii:-: L
The metalanguago includes three main classes of variables: stimulus,

response, and feedback. In addition, there are method (functional context), content, and structure variables. These variables represent relationships between

task conditions and requirements and learning media and methods as identified

through a thorough review of the training literature (Figure III-0). Matching

tasks wih media-method combinations through this metalanguage assures that the

iv training program is responsive to research findings regarding the relationsnaps

between tasks and media/methods.

TEEM begins with the identification of task information in the form of

standard ISD task statements or, if the materiel system has not progressed to

the stage that would make these statements available, in the form of gross task

lists derived from materiel system descriptions (draft TMs, LSARs, etc.). Once

the analyst has identified the tasks, he/she describes the task stimulus, response, and feedback variables. To describe the task stimulus variables, for

example, the analyst scans a list of 39 stimulus variables and records those

that apply (e.g., audio cues, intensity, channels or sources). To describe the

response variables of the task the analyst repeats the process using the list of

16 response variables. Likewise, the list of 17 feedback variables is scanned

to describe the task feedback.

-A Unless a task is trained alone, it is assigned to a gropp of tasks according to its functional context (training method). Functional context variables apply to groups of tasks that are performed together on the job and are

thus trained together. In the case of tasks resulting from a formal ISD process,

the relationship of tasks to terminal learning objectives is given in the form

of learning hierarchies; in the case of gross task infcrmatiop it may be possible to estimate terminal learning objectives from system scerarios and then to

cluster tasks accordingly.

The functional context (training method) is described for clusters or

groups of tasks using 13 variables that describe the context in which a task

group or cluster is performed and should thus be trained. For example, the

task mray require individual performance (performed alone). It may be primarily

physical (overt bodily actions), unstable (not constant or regular but subject

to continued change), and characterized by a physical context of low impact (the physical environment has little effect on the performance of the task) and

a psychological environment of low impact (the psychological environment has

little effect on performance of the task).

The selection of media (training devices and muterials) is accomplished by the TEH computer program in which training devices and materials have been

described using the same variables used to describe the tasks. Media with the

highest nis.ber of matches with tasks become candidates for selection.

The media selection procedure results in stimulus, response, and feedback media for each task. Since an array of media is not supportable in the

real world, the media set for each functional group of tasks is reduced to a

supportable number derived from the matches between task descriptions and media

descriptions. The ideal, unconstrained training program estaolishes the base

case against which reduced media sets are compared. By definition, the efficiency of this media set is 1 .0. All reduced medis sets have an efficiency of

less than 1.0.

111-3

S.. . . .. . . • . w . . ... .. .. . ... .- .. .• . .. •- ...• - IT - • •' ' ' '•' •- = "• '. . .. . . . •' . ". " ' •
E PRMPARE TASK

DISCRIPtIoNs i

C H O O S E M E D IA F O R S ICR E P U N C T IO N AM. EACH TASK COeX OF TASK GOUP

RECORD 

FOR4 EACH EPPICIENCY 

TASK GOP IOPMTO

IRIOUCE MEDIA SET COST PROGRAM

FORM EFFICIENCY NORMALIZE COSTS FOR I

RATIO$ C1* RATIO

Figure 111-1. TEEM Overview

(from Jorgensen and Hoffer, 1978)

.IT 4

-L -- _ _ _ _ _
The TEEM computer program, the medium with the lowest number of matches

across all tasks froum the media set in each matrix (that is, the medium least

useful for stimulus presentation, receiving responses, and providing feedback).

Iteration of the procedure measures the fit of all tasks with all functions and

this measure divided by the measure of the ideal, unconstrained case expresses

the efficiency of the reduced media set. The medium in each matrix with the

lowest number of matches across all tasks is removed from consideration, and the

efficiency of the further reduced media set is calculated. Iterations of the

procedure continue until all media have been removed.

Efficiency ratios are plotted against the number of the iteration. This

plot shows the iteration where the efficiency begins to drop off steeply and the

analyst uses it to select a media set that provides the lowest acceptable efficiency (Figure 111-2).

The method selection procedure uses functional contexts and their appropriate instruction methods. Selection is based on the number of matches between

task and training approach. The method with the highest number of matches with

functional context variables is selected for each task group.

Costs are determined by describing media and methods in terms of 37 cost

variables and the cost data are analyzed by the computer program. Outputs are

program Costs.

Once costs have been obtained, a decision metric, a cost/effectiveness

ratio, is obtained by dividing the ost by the efficiency value. The analyst chooses alternative meth.ods, calculates their efficiency values and costs, and

obtains their cost/effectiveness ratios. The analyst could recomm.end the method

withn the lowest cost effectiveness ratio (the lowest co~t for at least an acceptable efficiency), or he might recommend a method with a slightly higher cost

effectiveness ratio to obtain a large increase in efficiency at a higher cost.

2. Training Consonance Analysis (Hawley and Thomason, Development of an

Air Defense Cost and Training Effectiveness Analysis Methodology,

2 vol., 1978)

Hawley and Thomason's Training Consonance Analysis (TCA) technique is

a modificatiQn of TEEM that compares training alternatives on the basis of task

descriptions and the methods and media employed to train the tasks. TCA uses

the same variables as TEEM to describe tasks and media-method combinations.

Unlike TEFM, TCA uses the descriptions to indicate how close the media-method

combinations come tothe task description; that is, TCA yields an indication of

the consonance of the task descriptions and the training programs. Hawley and Thomason further modified TEEM by adding the diagnostic concepts "training deficiency," "training excess," and "training redundancy."

o Training deficiency: A variable in the task description dces not

occur in the training description.

o Training excess: A variable in the training description does not

occur in the task description.

111-5

IMOM
ONN

ATIO ~ ATI VAUM E REPEETN

VALur e EfiinyRtoA(h A1-2 (from~~~ SINGL EFIIEC osfntio n ~ ~ ~ Jogne adHferi98
0 Training redundancy: A variable in the training description w~ich

is redundant. This occurs only if two or more training media and/or

methodologies are combined in the training description. The variables

common to two or more media/methodologies are redundant.

These diagnostics explain the differences in the efficiencies of alternative

training programs. The smaller the number of excesses and redundancies, the

more efficient a training program is in media and methods.

TCA describes each task according to the 85 psychological variables on

which TEEM is based. The task descriptions are coded on data cards for computer

processing. The next step is the identification of the media and methods for

training each task perhaps from a training prescription such as TEEM, TECEP, and

TDDA. Information in the following four categories is needed:

* o The stimulus medium (through which information about the task

is presented to the student).

o The response medium (through which the student demonstrates

what he has learned about the task on about any other information presented through the stimulus medium). [ o The feedback medium (through which the student receives information about the correctness of his response).

o The training method (which describes the pattern of interactions among students, instruction, and media).

The TCA program analyzes the media and method matrices to compare task

descriptions, variable by variable, with the descriptions of media and methods.

When a variable is present in both the task description and the des, ':iption of

the medium-method combination used to train the task, a training consonance is

scored. Otherwise, training deficiency, excess, or redundancy is scored. The

training consonance ratio is given by the total number of training consonances

scored for the task over the total number of variables in the task description.

If 36 training consonances are scored and 39 variables are in the task description, then the ratio for that task is .923.

The TCA program aggregates output over groups of tasks and over all tasks.

The output, therefore, may be used to evaluate groups of tasks (which correspond

to modules of instruction) as well as to evaluate individual tasks or the entire

program. The outputs of the TCA are summarized as follows:

o Outputs are given for each task, for each group of tasks, and

for all tasks.

o Outputs for each task incluae training consonance ratios

training deficiencies, training excesses, and training redundancies (see definitions of deficiencies, excesses, and redundancies above).

o Outputs for task groups and all tasks include the total number

of training consonances, the total number of variables in task

descriptions, and the training consonance ratio.

111-7
Training consonance ratios and diagnostics provide a basis for recommendation of a training program and for improving the program.

3. CHRT (Goclowski etal., Integration and Application of Human

Resource Technologies in eapon System Design: Coordination of

Five Human Resource Technologies, Vol. I, March 1978; Proces for

the Coordinated Application of Five Human Resource Technologies,

Vol. 2, March 1978; and Consolidated Data Base Specification,

May, 1978)

The Advanced Systems Division of the Air Force Human Resources Laboratory combined five technologies in the weapon system acquisition process: maintenance manpower mode.ling, instructional systems development, job guide development, system ownersh.Lp costing, and human resources in design trade-offs. All

five are applied individually during the materiel acquisition process and have

data requirements in common. Therefore, one objective wab to integrateo.and apply

the technologies to form a coordinated human resources technology (CT).

A second objective was to design a consolidated data base (CDB) to support CHRT application. The CDB establishes a common source of i formation for

human resource technologies early in the acquisition process. CDB is established for each weapon system in the acquisition process and grows as the

system develops. Each of the five coordinated human resouro technologies draws

on the CDB for input, and many of the outputs go back into t e CDB. Thus, one

data base contains information relevant to reliability, mai tainability, maintenance manpower, operations manpower, training, job guides, and system ownership

costs. It supports human resource planning during the acq iisition process

and continues to support operational and logistic planning after the system is

deployed.

It is useful to examine the five human resource t hnologies separately.

Maintenance Manpower Modeling (MMM) is a method for esti ting the manpower required to maintain a new system and the effects of certain trade-offs. Maintenance action networks are developed to describe necessary ntenance; these

consist of sequences of maintenance events required to com ete the action.

They describe the probability of occurrence, time to comple e, and support equipment and personnel needed for each maintenance event. The d scriptive data are

derived from the analysis of comparable existing systems. T maintenance action

networks are paired with mission scenarios to simulate the tenance of the

system under mission conditions. The simulation predicts main enance manpower

requirements.

Instructional System Development (ISD) within CHRT differ \ from ISD in

other contexts in that it is coordinated with other human resource technologies

and draws its data from a base common to these technologies. It res lts in

training concepts during the concept phase of system acquisition, a t ining plan during the validation phase, and a fully developed training progr during the full-scale development phase.

Job Guide Development (JGD) results in products that may substitute fr

or reduce the need for training. JGD at present appears to be concerned with

maintenance tasks although guides for operatinonal tasks are consistent with the\\

JGD concept.

1:1-8

S,~.* -
While a large component of' Life Cycle Costs (LCC) is the acquisition

cost (research and development and procurement), a comparable compcnent is the

cost of operating and supporting a system once it has been procured. Th~is

second component, system ownership cost (SOC) includes humani resource considerations such as personnel requirements for operations , -support, and training.

Every decision made as a result of Mt4, ISD, and JGD has an associated cost,

and it is through SOC that such costs are assimilated into the LCC.

Human Resources in Design Trade-Offs (HRDT) identifies points in systems development where the selection of alternatives has a large impact on human resources. The HRDT Design Option Decision Tree maps selections of systems alter-

*1natives that have occurred or will occur. The decision tree identifies tradeoffs, develops data, and studies to conduct.

CHRT integrates ?.tt., ISD, JGD, SOC, and HRDT. The individual technologies retain their distinct characteristics but they interact through a cow.,n

pool of' information, the CDB. A human resource trade-off identified by HRDT,

for example, is reflected in ISD, JOD, and SOC. Conversely, an excessive cost

flagged by SOC or a support problem identified by MMH will lead to a revision

in ISD.

CHRT consists of' four activities: (M) consolidated data base development, (2) the integrated requirements and task analysis, (3) ISD/JGD product

development, and (14) the impact analysis. The CDB is the key to the consolidat ion of the five technologies. The CDE consists of all data that the five

technologies develop separately, but through consolidation it eliminates redundancy and achieves efficiency and effectiveness in the use of data (Figure

111-3).

The CDB begins with historical data and assimilates updated data as they

are developed. It includes data on a reference system (an actual or conceptual

systems), one cr more alternative baseline systems, and current systems.

The integrated requirements and tasks analysis (IRTA) combines ISD and

JGD analyses for maintenance and operations with maintenance manpower modeling

(MMM*).onIRTA is predictive during the conceptual and validation phases, but it

bcmsproduct oriented during the full-scale development phase. It dctertnines

mapwrID and JGD requirements as well as the detailed task analyses needed

todveo ISD and JGD products. It is an iterative process, drawing heavily at

firs onhistorical data (comparability analyses, maintenance records, informnation on existing courses, characteristics of available personnel, etc.) and then

assimilating empirical data as the system evolves.

The ISD and JGD product developmnent activity is also iterative. As the

data become more detailed it produces the training and technical data (job guide)

concepts, the ISD and JGD plan, and the ISD and JGD program.

Impact analysis, the final CHRT activity, is the investigation of the

impacts on human resource costs of a variety of system alternative's. CHAT

assigns human resource and other systems ownership costs to system design,

ma~intk.aance, operations, and support alternatives so that these costs may be

fully considered during the early and critical acquisition decisions.

111-9

T O
TASK

MAINTNANC

DATAMANPOW

EOUIPMEN IAAlUIMN

TAS

OPERATION

g tDATA

GR~uDESUIGNOU

ALgERe NATIVE B trctr

AT
4. TECEP (Braby et al., A Technique for Choosing Cost-Effective

Instructional Delivery Systems, TAEG Report No. 16, 1975)

Of all available analytical methods with potential applications to CTEA,

the Training Effectiveness, Cost Effectiveness Prediction (TECEP) technique appears to be the most consistent with current ISD procedures: TECEP and ISD

employ the same learning guidelines and algorithms. TECEP is used during the

conceptual phase for training program development.

The TECEP technique begins with a list of training objectives, classifies those objectives according to the type of learning algorithm required, selects alternative media systems to support those algorithms, estimates the cost of each alternative delivery system, and identifies a cost-effective instructional delivery system. The technique is simple though its developers

caution that it is intended for use by experienced training system designers.

TECEP is applied within training systems development (Figure 111-4).

Once training objectives have been analytically derived TECEP identifies and

costs instructional delivery systems. The three-step procedure is given in Figure 111-5, and the reference materials needed to implement the procedure are shown in Figure 111-6.

Each training objective is matched with one of twelve learning algo- rithms. The matching is through a comparison of the task with the various

algorithms in terms of action verbs, behavioral attributes, and examples of

objectives (Figure 111-7). Once tasks and objectives have been classified

and grouped according to the learning algorithms a table is used to select

delivery systems for each group of tasks and objectives. Each system is

then analyzed for cost-effectiveness comparisons.

5. The BDt4CARAF Method (Cost and Training Effectiveness Analysis Handbook

for Action Officers, 1976)

The BDM Service Company Combined Arms Research and Analysis Facility (CARAF) guide for CTEA outlines the process rather than providing a procedure or

method %hereby inputs lead to precisely defined outputs. The guide suggests a

number of measures appropriate to the types of training systems identified. It

also requires an interval scale but does not indicate how to quantify on an

interval scale.

The BDM guide to CTEA describes eight procedures for each of eight types of training systems. The procedures are quite similar, however, and differ only

in measures of training effectiveness. The eight types of training systems are:

large group war games simulator, individualized or small group lessons, small

group tactical maneuver and deployment game, hands-on performance aid, large

weapon system practice firing adapter, small group combat engagement simulator,

trouble-shooting trainer simulator, and small weapon system practice firing

adapter. Suggested measures of training effectiveness for large group war games

simulators are: rates of movement, rates of attrition, combat power ratios,.

casualties, ammunition usage, battles won, missions accomplished, and others.

Measures of training effectiveness are suggested for each of the eight types

of training systems.

hI-

!-WAN
DEVELOP IMPLEMENT

INSTRUCTIONAL AND EVALUATE

PROGRAMS TRAINING

ANALYZE 0 SANeaifv Learning PROGRAMS ANAYZETRAINING Eventsl• Withnn

JOe REQUIREMENTS 0* Choose Course Feedbhck

INFORMATION AND DEVELOP Instructhon Fe eedback from

OBJECTIVES SENtl"E 0 Feeoack Fiold

* Develop Materials

* Validate Materials

DEVELOP

CRITERION

MEASURES

TECEP Technique used here

Figure 111-4. Training System Development Model

(from TAEG Report No. 16)

CHOOSE LEARNING IOENTIFY ESTIMATE THE COST

ALGORITHMS FOR INSTRUCTIONAL OF ALTERNATIVE

TRAINING DELIVERY SYSTEMS SYSTEMS

O1JECTIVES FOR EACH SET OC O

SIMILAR TRAINING Estimate Irct n.CH(• COSTt GIVEN Classify and group) 09JECTIVES of using each 9PPECTIVI

TRAINING _ training objectives 4 1alerative delivery INSTRUCTIONAL OIJECTIVES occording to the For eah group syltem to eELeVIRY train 

yVpe of learning of obtectives the required number SYSTEMS

a1IO9 ithtrl required idenitiy tlWO or more of students to meet Op M IX

it, achieve the types of intruction4i the obiectivel OF SYSTEMS

3zbleCtives delivery IsVtems that

Will 11,1r the law

Of the required

algorithm

Figure II-5. Process Flow in the TECEP Technique

(from TAEG Report No. 16)

1I$-12
NLiDENTIFY

N CHOOSE LEARNING INSRUCTIONAL ESTIMATE THI CHOOSe

TRAINING -. 4 RNALGORITHMS AAOIHSFR PON -•,•~ DELIVERY IYSTEMI L COST TERNATIVETON OP COeT-4PPSFICTIVI

OBJECTIViE TRAINING FOR EACH SIT OF ATRNAIVINSTUCTIONAL

OSJECTIVII SIMILAR TRAINING IYrTEM1 DlLIVINj I,"OBJECTIVII iIVITIM

TAILE OF TWELVE TYPES OP LEARNING

REQUIRED ALGORITHMS WITH REFERENCE THI CHARACTERISTICS INSTRUCTIONAL COST

MATERIALS OF TRAINING OILIVERY IMODEL

OBJECTIVES THEY SYSTEM PRACTICALITY SUPPORT SELECTION Tit?

CHARTS

LEARNINGI

GUIDELINES ALTERNATE ECONOMIC

ADOITIONAL ANO At. GORII4MS METHOD THEORY FORTRAN IV

REFERENCE FOR TWELVE TYPES FOR CHOOSING UNDERLYING P4OGRAM OF

DATA OF TRAINING INSTRUCTIONAL THE COST COST MODEL

ObJECTIVES DELIVERY SYSTEMS MODEL USE (APPENDIX C)

(TAIG REPORT (APPENOIX A) IAPPENOIX S)

NO. 23)

Figure 111-6. Reference Materials Supporting the TECEP Process Flow

(from TAEG Report No. 16)

CHARACTERISTICS OF TRAINING OBJECTIVES THAT CAN BE

ACHIEVED WITH SPECIFIC ALGORITHMS

NAMES OF

LEARNING ACTION

ALGORITHMS VEPRS BEHAVIORAL ATTRIBUTES EXAMPLES

9. RECALLING '.0%ivate V I. Concerns the chaining 1. Recalling equipment

PROCEDURES, Adjust or sequencing of events, assembly and disPOSITIONING Align V2. Includes both the cog- assembly procedures.

MOVEMENT Assemble nitive and motor V2. Recalling the operaCalibrate aspects of equipment tion and check-out

Disassemble set-up and uperating procedures for a

Inspect procedures. piece of equipment

V/perate V3. Procedural check lists (cockpit check lists).

Service are frequently used A3. Following equipment

as job aids. turn-on proceduresemphasis on motor

behavior.

Figure 111-7. Sample of Mat hing Training Objective Characteristics

with a Type earning Algorithm

(from TAEG . .rt No. 16)

S11- 13
As an illustration of the BDM/CARAF method for CTEA, the procedure for

individualized small group lesson delivery system is summarized below. Only

steps of same substance have been included, and the distinctions among these

have not necessarily been preserved.

After precisely defining the training system under investigation, the

analyst chooses the standard measure of effectiveness, which is an aggregate

of less comprehensive measures of training effectiveness. In this case such a

measure might be a summary index score of effectiveness. Next, the analyst

chooses measures with positive or negative relationships to the standard measure

of effectiveness. In the case of the standard measure identified above, these

might be measured achievement, measured retention, and completion time. In the

case of the procedure for small-group tactical maneuver and deployment instructional game, the standard measure of effectiveness might be a platoon leader

performance score, while the measure would include missions accomplished, number

of losses, and platoon leader evaluations. For each alternative system (including a base case system) data would then be col.lected for each measure. The

scores for each measure are" weighted and a total weighted effectiveness score

for the base case and each alternative is derived (Table III-1). Using a simple

cost model, the analyst estimates the Life Cycle Cost of the base case and each

alternative (Table 111-2).

Table I11-I. Calculation of Total Weighted Effectiveness Scores (E)

MOTEI MOTE 2 MOTE3 MOTE4 ZE

WEIGHTS-'* 80 90 85 75

BASE CASE 29.6 57.6 46.75 42 175.95 EB

Aj 25.6 32.4 38.25 33 129.26 EA,

A2 24.8 0 0 0 24.8 EA2

A3 -EA3

Each training system is compared with others through a cost benefit analysis of the general form:

V.

> Effectiveness

Cost < Cost Base Case

Base Case Alternative X Effectiveness

Alternative

111-14
Table 111-2. Cost Model

THIS REPLACES

NAME: "

COST AND TRAINING DATE: /. I /i

EFFECTIVENESS ANALYSIS PAGE: of of

PREPARED BY: .. .. ..

ORGANIZATION:

1. R&D COSTS BASE CASE- A1 A2 * A3

LAsson Development 637,000 3,000,000

Lesson Revision 733,000 1,000,000

SUBTOTAL 1,370,000 ' 4.000,000

2. INVESTMENI COSTS

Purchase of Lessons

Purchase of Audio.visual Eqpt.

SUBTOTAL 4,526,000 8,000,000

3. OPERATING AND SUPPORT COSTS

Lesson Distribution 1,376,000 4,000,000

Logistical Support 478,000 1,000,000

Administrative Support. 23,642,000 5,000.000

Instructional Support 10,708,000 50,473.000

SUBTOTAL 36,204,000 50,473,000 9,000.000

4. TOTAL LIFE CYCLE COSTS 42.100.000 50,473,000 "21,000,000

5. TOTAL LIFE CYCLE UTILIZATION 266,500 266,500 266,500

UNITS

6, LIFE CYCLE COST PER UNIT 157,97 189.39 78.80

*These data are vTur illustrative purposes only. + + 4,

CS CA, CA 2 CA 3

111- 15
When the two values are equal, the compared systems are equal in cost

benefits. When the first term is the smaller value, the base case is better

than the alternative. And when the first term is the larger value, the alternative is better than the base case.

6. MODIA (Method of Designing Instructional Alternatives). Reported in 5 volumes by carpenter-Huffban et al., 1977)

MODIA's developers explicitly characterized it as neutral with regard

to effectiveness of training designs. It does, however, reveal the impacts of

design decisions and thus encourages designers to consider alternatives. Considerations include implications of the subject matter for training resources

and strategies, effects of student characteristics on learni.2g, effects of

course management and teaching strategy on learning and resources, and how

changes in one course design element influences the others.

The rmethod is useful for comparing well-developed training program alternatives. Because of the level of detail required, however, it does not

seem applicable early in the LCSMM. Details required as input include whether

the course requires unusually expensive or scarce resources, course content

sequencing, maximum and minimum numbers of students, effects of tests on

student progress, how each unit of the course will be taught, time required,

attrition, and availability and unit cost of resources.

Major output from MODIA includes projection of average and peak student

loads, average time to graduation or attrition, student waiting time, percent of time each resource is used, resources required, and the start-up, annual,

and five-year costs.

MODIA consists of four elements: a description of options for course

design; a user interface; a resource utilization model (RUM), and a cost model.

The first describes the choices in the design of courses, the data required to

use the systems, and guidelines for making design choices. The second leads the

user step by step through a series of decisions that result in a course descrip- tion. The third simulates operation of the course to identify required resources.

The fourth estimates five-year investment and operating costs. The relationships

among the four elements of MODIA and the user are illustrated in Figure 111-8.

' OpFiONS FOR

IOURE DESIGN4

INTWA -TILIZATION

Figure 1-. Relationships Among MODIA Elements and the User

111-16
Before attempting to make use of the oomputer-based elements of MODIA

(the user interface, the resource utilization model, and the cost model), the

user studies the optiors for course design then gains access to the user interface (UI). The UI poses a series of questions that lead the user through the

design process. It asks where the user wants to begin since some of the design

work might have been completed during previous sessions. The UI begins the

queations relevant to the phase in which the user wants to begin. The eight

sequential phases of the UI are as follows:

o Select the planning phase.

o Describe objectives and tests.

o Describe student population and course diversification.

o Describe teaching policy.

o Describe test details.

o Describe resources.

o Describe resource constraints.

o Define resource utilization model (RUM) parameters. (These

tell the RUM when to stop the simulation.)

The program transforms the course design for input to RUM, and RUM

analyzes the performance of the course by simulating the flow of students through

it. Within the processes of RUM, for example, students enter the course at random or at prescribed intervals, progress through a sequence of learning events,

use different amounts and types of resources, may be matched with learning events

on the basis of their individual uharacteristics, may be temporarily held up

while they wait for resources or for other students, and may fail and then have

to repeat all or part of the course. The user probabl'r begins with very brief

simulations to determine if bottlenecks or other problems exist. As he/she works

out the problems, he/she orders longer and longer simulations to verify that the

course is indeed working as desired. Once satisfied with the operation of the

course, the user assembles available cost data along with RUM and UI o, tputs for

input to the cost model. The cost model then provides estimates of costs.

7. DAIS/TRAMOD Method (Czuchry et al., Digital Avionics Information

System: Training Requirements Analysis Model User's Guide, 1978)

The Training Requirements Analysis Model (TRAMOD) is one of a group of

computerized analytical models that make up a life cycle cost model. It is

similar to MODIA in that it is a computerized means for training design, but

unlike MODIA it makes design decisions rather than simply revealing the impacts

of decisions already made. TRAIAOD selects from an input list the .asks to be

trained. It generates a training plan consisting of place of training (school

or OJT), method of instruction (simulation, performance, lecture, etc.), and

media (simulator, mockup, etc.). Finally it determines possible schedules.

Task input includes values for a number of parameters: criticality; learning

difficulty; frequency; psychomotor level; cognitive level; and estimated time

required to accomplish training. In the early stages of design of new equipment, data values are obtained from comparable operational equi;ment.

111-17

S-, o -. - - -
TRAMOD considers cost but not effectiveness. Given adequate task data,

it would appear to be a valuable CTEA method for comparing training alternatives

resulting from various constraints (limits on training time, equipment shortages,

etc.). The general process of TRAMOD is illustrated by Figure IIT-9.

The preparation of the task data base required by TRAMOD is a manual

operation of considerable difficulty. The authors of the method suggest that

"... the values assigned to the task characteristic parameters should be based

on the judgments of engineers and technicians familiar with the equipment upon

which the tasks will be performed" (p. 12), but implementation of such a suggestion seems to require a change in the way hardware manufacturers currently per- form task analyses. The requirement for such a change constrains successful

use of the method.

Once the task data base has been prepared and entered the user interacts

on-line with the program by specifying constraints on the selection of tasks to

be trained, the generation of the training plan, and the generation of the training program (possible schedules, types and numbers of media required, number of

classes, etc.). The user specifies the lowest (threshold) value of a task characteristic parameter that selects a task for training. In the task data base

each task is evaluated for criticality, learning difficulty, frequency, psychomotor level, and cognitive level on a scale of 1 to 5. By setting the value for

each of these five parameters, the user places constraints on the selection of

tasks. The user then specifies which of five algorithms will be used to test

the values. One such algorithm, for example, requires each of the characteristics of each task to meet the threshold value set for it while another requires

the average of the characteristics to meet the average of threshold values. If

the user were to set a threshold value of 3 for each of the five characteristics

(triticality, learning difficulty, frequency, psychomotor level, and cogniý.ive -.vel) ' and were to choose the first algorithm described above to make the test,

!.ien the program would select for training a task evaluated as criticality M 5,

"earning difficulty = 3, frequency = 3, psychomotor level = 4, and cognitive a'".vel 3. If the user were to choose the second algorithm described above both

tisks would be selected for training since the average value of the characteristics is at least 3. The user could, in turn, select all five of the algorithms

to make the test and '.hen compare the results.

TRAMOD generates blocks of tasks to be trained in the school or on the

job. Tasks that must be trained together are assigned to the same block.

Once satisfied with the set of tasks selected for training, the user

proceeds to the generation of a training plan by answering a set of questions. The first question concerns how costs and training times are to be determined.

The first choice results in direct use of the time and cost data contained in

the input data set for each task; the second choice results in computation of

time and costs as functions sf variations in the levels of the value of tasks

characteristics, with the regression coefficients being provided by- the user;

the third choice results in computation of time and costs on the basis of regression coefficients stored in the computer (TRAMOD). The second question con-

"cerns how students are to be assigned to school and OJT. The options are termed

"mixed" and "non-mixed." In the "mixed" option (students are trained on some

task blocks in the school and some OJT) TRAMOD distributes task blocks between

the school and OJT on the basis of minimal cost within time constraints. In the

i +

I 'S I I I I - " '
P44

¶ ku

0 1- 1 

.

I ; I q

I ~Mal I 

0000 0

I !
"non-mixed" option (student is trained in the school or OJT but not both) TRAMOD

either calculates the split of students between the school and OJT or separately

tests school training and OJT against given time and cost constraints, depending

on choices made by the user.

The next questions ask how training objectives are derived from the task

data and how media and methods are assigned to training objectives. The user

can allow TRAMOD to make these decisions on the basis of information stored in

the computer or make certain alterations in the way the matches are to be made.

TRAMOD, for example, matches a task group evaluated as psychomotor level - 4 and

cognitive level = 2 with Training Objective 3, Understanding Principles and Relationships. For this training objective it then assigns simulation as a method

and simulator as medium to school training, and performance (hands-on) training

as a method and mock-up as media to OJT. The user can decide to match cognitive

level = 2 with Training Objective 4, Learning Procedural Sequence. Further, he

can decide th1at the methods/media will be discussions with transparencies in the

school and simulation with simulators in OJT.

To generate the training program the user sets minimum and maximum class

size for school training, identifies the task characteristics that serve as the

criteria for ordering tasks for training, and indicate which media are optimally

scheduled because of scarcity. TRAMOD prints schedules showing the order in

which tasks are to be trained, whether a task is to be trained in the school or

in OJT, days required to train each task, the cost of training each task, the

method for each task, and the medium for each.

After considering the results of the decisions he/she has made, the user

can rerun the program, alter his decisions, and compare the new results with

former results.

8. TDDA (Pieper et al., Training Developers Decision Dialogue for

Optimizing Perormance-Based Training in Machine-Ascendant MOS,

1978; and Tryout of a Training Developers Decision Aid for Optimizing Performance-Based Training in Machine-Ascendant MOS, 1979)

The TDDA is similar to TECEP with one new element added, Response Acceptance Mechanisms (i.e., ways of providing for student responses). Unlike

TECEP it was developed in both manual and computerized forms.

The four functional elements of TDDA are: task description; training

prescription; training hierarchy and sequence; and training cost. Tasks are

described using specific action verbs and the piece of equipment acted on.

Training is prescribed as to: learning algorithms; stimulus media; response

acceptance mechanisms; method of instruction; and learning setting. Training

hierarchy is the result of assigning tasks to resident training, on-the-jobtraining, or no training, while sequence is the specification of the order in

which tasks are to be trained. Relative costs of feasible training alternatives

are established through a cost-rating technique. Figure III-10 is a representation of TDDA.

The tryout of the task description element indicated that the use of

Soldier's Manua!Z may not result in adequate task lists and that the action

verb list may not be generalizable to all MOSs. The user of TDDA examines the

M OS documentation to identify the actions performed by an operator and searchea

IZI-20
SILICT I OOCVMINTAYIOAI

MOS Tht. SOLDIERSS

I OIISYSTEM COMPILI cowPlE SUPOV

SOIMNY LIST TASK LIST sawu T LIST

CREATE TASK 3

I STATEMENTS

L P~EELhININO~mEITO

* S EOU IJISI CHARACTERISTICS IRESTNTATION

STIMULT I STATUS

LL

TRAINIG OPTIN TRANINO OPTION

ONON TYP TELAININIM V IASI LITY SASE .

COMPI TV"SE - -1TWOOof COMPTC OPtIONALAL

NUIMBERA~ Op PIRFORMVPC I RNKNG TAIIN LCAIE

COU*SIIIO S*EI8A rON

Figure-- -- 11-1 -eeoe' T- Tr-nn --- Dei-o Aid 

111L..-2 ... ...

in LL1MNWT _ II -ll"%1AIH 111 0R~
the action verb list, a part of which is reproduced below, to find a verb that

accurately describes the action perforMed. The user may find it difficult to

decide which of several nearly synony ' ', verbs should be selected. The item

of equipment acted on is added to the task description.

Training prescriptions use the task action verb to select one of twelve

algorithms, which are models of instruction adapted from TAEG Report #16 (Braby

et al., 1975) and TAEG #23 (Aagard and Braby, 1976). They are nearly identical

to those in TRADOC Pamphlet 350-30, Interservice Procedures for Instruction Systems Development. Appendix A in Pieper et a]., 1979, lists action verbs with

the associated algorithms.

Having selected a learning algorithm, the user then selects the stimulus

meaium by identifying the general class of stimuli in the job environment and

then further characterizes the stimuli by answering certain questions. The

general classes of stimuli are verbal, audio, audio-visual, visual, and tactile.

Several of the questions answered to further characterize the stimuli are:

o Are the stimuli in color?

o Are the stimuli equipment indicators?

o Are the audio stimuli voice only?

o Are the stimuli visually distinct (not obscured or overshadowed by peripheral stimuli)?

Answers to these ouestions lead to the selection of one or more of the

following media types: audio tape, books with questions, microfiche film, mockup, movie, printed text, programmed text, real equipment, silent film strip,

silent slides, simulator, sound film strip, sound slide, or television.

The TDDA requires the specification of the response acceptance mechanism.

The general class of the response is first aetermined, and then the class is

further characterized through a list of questions. The classes of responses are I

equipment manipulation, voice, written, and body movement.

Questions askea to characterize the responses include:

o Is the task a maintenance task (one that requires access to the

interior of the system)?

0 Are the manipulations discrete (as opposed to continuous as in

tracking)?

o Are the control displays of the equipment nonlinear? I The available response acceptance mechanisms follow. These are what

the student acts on in training as he learns a task: audio tape recorder,

books with questions, group instructor, mock-up, question set, real equipment,

simulator, teaching machine, tutor, and video recorder.

4-
The 5s us medium and the response acceptance mechanism become the

basis for the sel tion of the method of instruction. A set of methods Compatible with both the st lus medium and response acceptanne mechanism is identified, and an optima:. m od (or methods) is selected by the number of matches

between methods and sever additional characterizations applied to both the

stimulus medium and the resp se acceptance mechanism. The additional characterizations applied to the stius mediun are as follows:

" Pacing controller - what bontrols the speed of application for

the presentation medium (program, student, or both)?

"o Stimulus content (visual, verbai," or audio)?

"o Next learning activity - what controls the presentations sequence

of items (program, instructor, student)?

The additional characterizations applied to the response acceptance

mechanisms are:

o Pacing controller (student, instructor, program).

o Next learning activity (program, instruction, student).

o Type of evaluation (individual vs. group; selected vs. constructed).

o Feedback (immediacy - immediate vs. delayed; source - intrinsic

vs. extrinsic).

The method is selected from the following list: case study, computeraided instruction, demonstration, games, group interview, guided discussion,

in-baseket exercise, peer tutor, programmed instruction, programmed practical

exercise, role playing, study assignment book, traditional classroom,

traditional practical exercise, and tutoring.

If, le, the pacing controller of the stimulus medium is the

student, then computer- n truction, programmed practical exercise, and

study assignment book are possib e-methods because all are compatible with this

characteristic of the stimulus medizn. The metho most compatible with the

characteristics of the stimulus medium and the response acceptance mechanism is

selected.

On the basis of three criteria one of five learning settings is chosen.

The three criteria are:

o The number of students involved (a small number vs. 5 or more).

o The nature of student interactions (individual vs. team).

o Whether or not equipment manipulations are required.

Learning settings currently employed in TDDA are small group site, large

group site, individual carrel, small group carrel, and traditional classroom.

111-23

I
Determination of the training hierarchy and sequence produces decisions

about whether a task is to be taught in resident training, in OJT, or not at

all. The decisions about the order in which tasks are to be taught are also

made within this element.

The task training matrix, which contains all relevant task data (systen equipment, support equipment, rank of task performer, task dependencies,

etc.), is used to make these decisions. All tasks within a particular cell of

the matrix are ordered hierarchically according to their interrelationships or

dependencies. Decisions about whether a task is to be taught in resident training, in OJT, or not at all are made on the basis of six additional characteristics. These are:

o Per~cent members performing - the relative numbers of persons •

in the MOS who perform the task.

o Percent time performing - the relative amount of time each job

performer spends performing the task.

o System readiness impact - the effect of task performance on

the system's ability to perform its mission.

o Time to application - the period of time after arrival on the

job before the task must be performed the first time.

o Task learning time - the number of hours required to learn

the task on the job with the supervisor acting as tutor.

o Rank of task performer - the rank of the soldier who will

perform the task.

TDDA training cost estimation is relative only, but detailed cost information may be developed after the selection of the most cost-effective option

or options. A relative cost indicator is computed for each'of the two training

categories (resident training and OJT). Values for the two categories are then 7

sunned to yield a training option cost indicator. This indicator is to form a

ratio that reveals the relative costs of the training options.

The training option cost indicator (TOCI) was developed by classifying

all training methods and all direct and indirect training costs to derive a

mean rating of each method. Each of the training methods was assigned to a cost

class as follows;

111Z-24

- - _ I II1[ . . ...... . . . . . .... ... . .. .. - - : r ; . . . .
ff

Method Cost Class Training Methods

1 Conventional

Demonstration

Case study Guided discussion

2 Peer tutor

3 Tutor

4 Programed instruction

5 Traditional practical exercise

b Programmed practical exercise

7 Computer-assisted instruction

All direct and indirect training cost variables were assigned to one

of the following classes: square footage (i. e., space required), instructorto-student ratio, system equipment, furnishings, expendable supplies, training

aid development, and training materials development.

The various classes were combined in the cost rating matrix illustrated

in Figure III-11. This matrix shows how costs vary as a function of training

method and provides both a mean rating of each method and a cost multiplier. Hours spent in training in any method are multiplied by the cost multiplier to

give the cost of training in that method. The sum of the method cost indicators

summarizes costs for resident training and for OJT.

II

COST VARIABLE CLASS

SO INST/ SYS FURN EXPEN TNG TNO COST

SO STU E R EXPE AID MAT MEAN MULTIFT STU EO SUPP OEV EV DVRATING DEV RAIG PLR PLIER

1. 1 3 1.5 3 3.5 2 3 2.4 1.23

2. 3.5 4.5 3.5 3 3.5 2 1.5 3.1 1.35

0 3. 3.5 7 3.5 3 3.5 2 1.5 3.4 1.40

4. 3.3 1.5 1.5 5 3.5 5.5 5 3.6 1,43

5. 6.5 6 5.5 3.5 4 3.5 4.3 1.55

8. 6.5 4.5 S.5 6 3.5 5.5 6 5.4 1.73

7. 1- 3,5 1,5 7 7 7 7 7 5.7 1.78

Figure 111-11. Representative TDDA Cost-Rating Matrix

1_-T- 2 5
9. B-1 SAT (Sugarman et al., B-1 Systems Approach to Taning

Final Report , Vol. I, July 19-75)

The B-1 system~s approach to training (SAT) developed a training programI

for the air-crew of the B-1 strategic bomber. It applied the techniques of

systems analysis to instructional system development to assure that the entire

training system would be considered within an orderly and complete process.

ISD application is rarely as rigorous as in the B-1 SAT study.

SAT begins with analysis of missions into the successively more detailed I

levele of mission segment, function, task, and task element. The description of the task element includes the actual behavioral initiation cue, action verb,

control, and completion cue. All task elements are then examined for comm~onalities, and a behavioral objective is developed to encompass each set of comm~on

task elements (Figure 111-12). Behavioral objectives include a set of comonnn

task elements and information about the behaviors as initial conditions, concur-I

rent behaviors, perfornmace criteria, enabling objectives, crewmeane responsible for the behavior and interactions among crewmen. Figure 111-13 represents a

task element while Figure 111-14 represents a behavioral objective that encompasses that task element. The task analysis is supported by a computer programi, called the sorting program, which includes a controls and displays catalog and 

'

an action verb thesaurus.

The controls and displays catalog contains such descriptive-information

as the names (and synonyms), locations, types, and subsystems of controls and

displays, while the action verb thesaurus contains synonyms for action verbs

~t I used to describe behaviors. The task analyst uses the sorting program to store ~ j task information and to retrieve necessary equipment-related information. The

use of the term "altimeter" or "rate of climb indicator" locates the information

about the display called altimeter/vertical velocity indicator and records the

"reply" to be retrieved or stored.

All behavioral objectives, inpluding enabling objectives, are arranged

hierarchically and examined in light of the qualifications of a probable entering student. The tool supporting this determination was the personnel qualifications catalog, an example of which is presented in Figure 111-15. The figure

shows, for example, that an FB-111 pilot coming into the B-1 training program as

a pilot would have been using a pitch and roll trim control identical to that of

the B-1. The pilot with such a background is not given training on that control".

B-i SAT identified training device requirements including method and 4 media selection. Guidance for the identification of training device requirements was taken from the Navy's Training Effectiveness, Cost Effectiveness Prediction Technique (TECEP) (Braby, Henry, and Morris, 1974), modified to reflect

the special requirements of the B-i SALT.

of courses, tracks, and instructional blocks. The tool employed to this end is

H the Training Resources Analytic Model (TRAM), a set of computer programs that

A examine proposed training system of resources, schedules, and costs. TRAM is

similar to MODIA (Carpenter-Huftbian, 1977), though it may not be as usable as

MQDIA, at least from the CTEA analyst's point of view.
C SORTING PRORAM

'CONTOLS ITASK ANALYSIS I AND DISPLAYS I DATA

*IGHAVIONAL OBJEC•'TIVES

r!

r PERSONNEL QUALIFICATIONS CATALOG

MINIMUM AVAILABLE i

• I ~INCOMING SKILLS •

S~AND KNOWLEDGES

L -

OIAGNOSTICS

r ------- 0APTITUDE SI METHODS/MEdIA CATALOG I _________ _ ACHIEVMENT

M ETHODSIMEDIA AAAO

LETMODSIEDIAN ENABLING .- CENON !

SLCINOBJECTIVES ERENCEE'S HIERARCHY

I.. - - - - L 

TRAINING REOUIREMENTS

TRAINING RESOURCES

ANALYTIC MODEL

., I EXTERNAL INFLUENCES INSTRUCTIONAL AND COURSEBLOCK

SI I ALLOCATION

INSTRUCTIONAL

SYSTEMIDESIGNS I

.INFORMATION FLOW

PROCESSES AND DATA

Figure 111-12. Systems Approach to Training

(from Sugarman et al., 1975)

TI.I-27
i3

U14AVMR

OP SITTING WING SWEEP ANGLE FOR CRUMS 4XINFISURATION

INITIATION CUI 1710" cutWE1

DISPLAY RELATION STATE VaEn _DISPLAY DISPLAY RELATION STATE

ALTIMETER 3= WING L

SWEEP

INDICATOR

- - - ADJUST WING SWEEPSAND POWER LEVEL so CONTROL

INDICATOR -

Figure 111-13. Example of Task Element Behavior

(from Sugarman et al., 1975)

I1 2

iII-2b 14
OBJECTIVE: Make a eeasdnesd trenisiton from climbing

to aruto configuration.

INITIAL CONDITIONS: VERTICAL VELOCITY: 2000 TEMP. AT DESIRED ALTITUDE. -655 IAIR POWER LEVEL: 100 DESIRED POWER.LEVEL: 90

SPEED: .6 ONSIRID AIR SPIED- .P ALTITUDE: 260= DESIRED ALTITUDE:300

WING SWEEP: Q ~ DESIRED WING SWEEP- 46

A-V TRIMMED POR CLIMB

CON4CURRENT1111EHAVION: Hulgeenenen

INITIATION CUE ACTION CONTROL ORt DISPLAY COMPLETION CUE

BEHAV IOR: ALTIMETER OR60 ADJUST THROTTLES POWER LEVEL IND. a 90

ALTIMETERn' MR60 TRACK PITCH IND. PITCH IND. a'0 _____________ ______ CONTROL STICK ALTIMETER a 30000

A LTIMETER , 30000 ADJUST WING SWEEP CONTROL WING SWEEP IND. me 45

POWER LEVEL IND. * O__________

ALTIMETER a 300M ADJUST TRIM PRtOPRIOCEPTION. NVUTI4AL

PRESSURE

AIR SPEED u .8

WING SWEEP IND. *45

PERFORMANCE: AIR SPEED * .8 (.*kts) at Cruise altitude.

ALTITUDE a '0000 (*ft) from desired altitude at cruise ITIME,< so0.

SUBJECTIVELY SMOOTH VERTICAL PLIGHT PATH lagq. REASONABLE 9 FORCES).

_______ WING SWEEP - 45 (*dag.I after adjustment (TIME < sal.

HEADING ERROR *0 degrees PROM DESIRED HEADING (*cdog).

ENABLING OBJECTIVES:

CALCULATE Necessary power level for cruise air speed.

altitude, temperature, desired true ait speed.I

CALCULATE Necessary altitude to Initiate power level change.

initial wert. velocity, deosired altitude. airspeed. desired airspeed, ANV characteristics. ICALCULATE Necessary altitude to initiate pitch change.

:vert. velocity desired altitude, airspeed, ANV characteristics.

COORDINATE Throttles and control stick to achtieve a rapidI transition having the g forces within a criterion

without undershoot or overshoot.

PREDICT Necessary pitch changes for level-off at desired

altitude from the vertical acceleration and the

rate of change In pitch,

TRACK Pitch indication with control stick to remain at zero pitch at level-off.

TRACK Heading indication with rudders and control stick

to remain at desired heading throughout Maneuver.

TASK ELEMENTS: 6.1.1.1 OPERATORS: Pilot land copilot)

6.1.1.2 INTERACTIONS: OSO provides hoading data

6.1.1.3 TIME: Indefinite, depending

upon conditions

6. 1. 1.4 CRITICALITY: 2

4 D IFFICULTY: 2

Figure 111-14. Behavioral Objective Illustrating a Maneuver

(from Sugarmnan et al.., 1975)

j 111-29
I

CONTROL USING Mill

ITEM DISPLAY CREW

CODE MIMBER P N

FLIGHT CONTROL STICK $1.1 P-CP A

SCAS PITCh SWITCH FLIGHT TEST ONLY SI-2.i (PIICP C

SCAS ROLL SWITCH FLIGHT TEST ONLY S1-2.2 (iP)-CP C

SCAS YAW SWITCH FLIGHT TEST ONLY S1.2.3 (P)-CP C

STICK SHAKER SWITCH S1-2A.4 (P)-CP 6

STANDBY PITCH TRIM CONTROL $1-3.1.1 P C

YAW TRIM CONTROL S1-3.1.2 P C

STANDBY PITCH TRIM CONTROL S1.3.2.1 CP C

YAW TRIM CONTROL .... _S_-3.2.2 .. CID C

TRIM FOR TAKEOFF SWITCH S1-3.2.3.1 (P)-CC c

TRIM FOR TAKEOFF LIGHT S1-3.2.3.2 P.VP C

PITCH & ROLL TRIM CONTROL S-3.43 P A

PITCH & ROLL TRIM CONTROL S1-3. CP A

PITCH TRIM SWITCH S14.1 P-CP C

ROLL TRIM SWITCH S14.2 P-CP C

YAW TRIM SWITCH 1,-4.3 P.CP C _

PITCH AUGMENTATION SWITCH S14.4 P-CP C _

\•ROLL AUG3MENTATION SWITCH S1-4.5 P-CP C

YAW AUGMENTATION SWITCH S14.6 P.CP C

FLIGHT CONTROL STICK DISCONNECT $1-5 P-CP D

CLASSIFICATION OF INCOMING ABILITIES

r * A - PRESENTLY USING IDENTICAL EQUIPMENT

• 8 - PRESENTLY USING EQUIPMENT WITH IDENTICAL

FUNCTION BUT DIFFERENT OFERATION

0 C - PRESENTLY USING EQUIPMENT FOR SAME PURPOSE

BUT FUNCTIONALLY DIFFERENT

* D -NEVER USED COMPARABLE EQUIPMENT

Figure :11-15. Example of Personnel Qualification Matrix

(adapted from Sugarman et al., 1975)

111-30
B-I SAT has three determinants for the scheduling of instruction: the

hierarchy of objectives, the instructional context, and resource management.

The hierarchy identifies dependencies within the content structure; that is,

what must be learned so that the next skill or knowledge may be acquired. The

instructional context refers to the system or the phase of flight with which

learning is concerned. The sorting program allows the analysts to search the

data base to determine all objectives associated with a particular system or

phasi of flight and then to group or not group the objectives on such a basis

depending on the outcome of tradeoff analyses. TRAM is the means of conducting the tradeoff analyses between resources and contexts. Grouping objectives within a certain instructional context may be indicated by the monalities

among objectives but contradicted by the cost of using training devices and

facilities in such a way.

Because students of B-i training programs were to come from different

sources, and would thus have different entering skills and knowledge, different tracks were developed for each distinct source of students, and were

examined by TRAM for total cost-effectiveness.

10. Army CTEA Methods in Current Use

Army analysts, such as those in the TRADOC schools having proponency

for the developing weapon systems, perform CTEA using a number of techniques. These techniques were developed in response to severe constraints on input

data. They represent what is feasible for the analysts at present and what

the present research aims to build upon and improve. Many of their methods

and processes are pertinent to a general model for CTEA in the LCSMM and are

reviewed here to identify and describe those processes.

Some of the current CTEA methods are applied to implemented systems

rather than systems still in the acquisition process, but their methods and

processes are relevant to CTEA conducted after DT/OT results are available.

Other of the CTEA apply to early stages in the acquisition process. The studies reviewed were:

o DIVAD Gun CTEA

o Improved Hawk (Hawk PIP) Training Development

o Roland Training Development

o Improved TOW Vehicle (ITV) CTEA

o Diagnostic Rifle Marksmanship Simulators (DRIM) CTEA

o Bridge '85 CTEA

o XM1 Tank Training Devices CTEA

o LAW WSTEA

o M16 'tifle WSTEA

111-31
Procedures in the DIVAD Gun CTEA are applicable as early in the LCSMM as

the conceptual phase, where they are used to generate task lists, and later through all stages until the comparison of training program alternatives. The other CTEA

applicable for generation of task lists are the Improved Hawk CTEA and Roland

Training Development method. The DRIMS CTEA contributed procedures for comparison of training program alternatives. Both DRIMS and the ITV CTEA contain methods

for the resolution of CTEA issues in the LCSMM.

A. The DIVAD Gun CTEA. When the DIVAD gun system was nearing DT/OT II

the data available for"CTEA corresponded to the conceptual phase. There were no

task data and few other forms of data. The CTEA was conducted on the basis of

knowledge about the class of weapon systems to which the DIVAD gun belongs.

There are certain tasks or types of tasks that are common to weapons of this

class, and there is a great deal of information about the training of these

tasks from prior research. On the basis of this general information, and without

task information, the CTEA analyst drew upon prior documentation and subject

matter experts to formulate three training program alterfatives and to estimate

the effectiveness and costs of the alternatives. One of the alternatives had

the highest effectiveness estimate at lower cost and was therefore selected.

The procedures used in this CTEA represent a "generic" type based on

prior research, prior or existing weapons similar to the developing system,

and subject matter expertise.

B. Improved Hawk (Hawk PIP) Training Development

One of the processes of the Improved Hawk training development

effort corresponds to a potential CTEA process, the generation of a task list.

Improved Hawk is a modification of the Hawk design rather than a redesign, so

many of the tasks of the former are identical to those of the latter, and the

training, training requirements, and training implications are also identical.

Where tasks have been added or changed new tasks were defined by inferring

tasks from the manufacturer's draft technical manual (TM). Figure 111-16 is

a reproduction of a task analysis sheet prepared by the training developers;

the inferred tasks are listed at the left, and task analysis data (MOS to be

trained and locations of training) are at the right. A source of this information is the page of the manufacturer's draft TM. Documenting information

was not always adequate so the training developers worked through them on

actual equipment.

Once the new tasks were defined they were used to develop new training. To the extent that similar tasks were known to exist in other systems, the

training of these similar tasks guided the development of training for the new

tasks. For other tasks they developed training through standard ISD procedures.

C. Roland Training Development

The Roland task analysis procedure represents what a CTEA analyst

might do given adequate data. Closely following standard ISD procedures and

using the manufacturer's Logistic Support Analysis Record (LSAR, D Sheets) and

draft TM, the task analyst developed a task list. Employing expert judgment

developed through the manufacturer's training course as well as broad experience in Air Defense, he/she made certain revisions in the task descriptions

provided b.Y the manufacturer. He/she then analyzed the task list to identify

critical tasks and to determine training locations of tasks.

111-32
MATRIX TASK INVENTORY

LMOS TRAINING

LOCATION TAS CRITICAL TASKC LIST

TAUIC/CATaGORY

DUTY. PERFORM CHECKS AND ADJUSTMENTS

TASKS:

16 24 24

S1. Check TAS opeantions E E G X X X

16 24 24

2. Set up TAS local control unit E E G X X X

16 24

3. Check TAS interface electronics SITE E E X X X

16 24 24

4. Check and adjust TAS video conditions E E G X X X

SUBTASKS:

16 24 24

4a. Check/adjust vertical roll E E G

16 24 24

4b.Check/adjust horizontal roll E E G

16 24 24

4c. Check/adjust humbars k E G

16 24 24

4d.Check/aidut ima•e focus a E 0

16 24 24

4@. Check/adjust rnir-or resolution E E G

Figure 111-16. Reproduction of Part of Matrix Task Inventory Sheet

111-33
I

D. ITV CTEA (TRADOC, Cost and Opeitional Effectiveness Analysis

for the Improved TOW Vehicle UNV), Part IV, Cost and Training

Effectiveness Analysis, June 1978)

This CTEA was originally conducted as a part of the COEA done, presumably during Phase III, Full-Scale Development, but was subsequently updated

to provide for training development needs. It is of interest methodologically

because it presents an examination of costs as a function of training location

(i.e., different Army training centers) rather than as a function of different

media-method combinations. It assumes that the training alternatives are equal

in effectiveness.

E. DRIMS CTEA (USAIS, Diagnostic Rifle Marksmanship Simulators

Cost and Training Effectiveness Analysis, August 1978)

This CTEA compared the costs and effectiveness of four approaches

to remedial rifle marksmanship training. Because of the high rate of failure

in rifle marksmanship courses, remediation is expensive and, conducted with

the M16 rifle and ball ammunition, possibly limited in effectiveness. Three

rifle marksmanship simulators were considered as alternative approaches to

remedial trai.'ing. These were the Weaponeer, the Lasertrain, and the Caliber

.22 Rimfire Adapter (RFA).

This study is of value to the development of general CTEA methods

primarily because of the procedure followed to rank the alternatives: the

rankings of the alternatives on four separate factors were combined to give

an overall, or final, ranking.

This procedure is illustrated in Table ITI-3. The four ranking factors are relative training effectiveness, relative cost, relative capability,

and relative user acceptance. Rankings on each factor were derived as follows:

o Relative training effectiveness was given in terms of the probability of a hit during record fire for soldiers given remedial training with

different alternatives. These data were obtained from previously conducted

studies. Relative training effect.'veness (RTE) is defined as the overall probability of hit (P ) associated with an alternative divided by the probability

of hit associated with the baseline case, which was the M16 rifle firing ball

ammunition. Thus, the RTE of the M16 rifle firing ball ammunition was 1.0,

while the RTE of the Weaponeer, for example, was 1.01 because the P associated

with it was slightly higher than the P., associated with the M16 firing ball ammunition.

o Relative costs are defined as the life-cycle costs of the alternative over the life-cycle nosts of the base case, which is the M16 rifle firing

ball ammunition. Because the RFA can simulate 25-meter fire but not record fire,

it6 life-cycle costs were compared only with the 25-meter component of M16-ball

anmnunition life-cycle costs.

o Relative capability is defined, simply, as the number of critical.

skills taught by an alternative over the total number cf critical skills of the

M16-ball ammunition base case. The relative capability of the M16-ball aninunition base case is 1.0, of course, while the three simulators have a somewhat

lesser capability for teaching critical skills.

71:-34
Table 111-3. Ranking Factors and Ranking of Training Alternatives

(from DRIMS CTEA, 1978)

BASELINE ALTERNATIVES

RANKING FACTORS M16 WPNEER LTRN RFA

RELATIVE TRAINING EFFECTIVENESS (RTE)

Record Fire PH 1.0 1.01 0.99 1.00

Relative Training Effectiveness Rank (1) (1) (1) (1)

RELATIVE COST (RC)

a. Table 4, Section V 1.0 0.08 0.05 0.10

b. Relative Cost Rank (4) (2) (1) (3)

R ELATIVE CAPABILITY (RCP)

a. Table IV-I 1.0 0.86 0.50 0.64

b. Relative Capability Rank (1) (2) (4) (3)

RELATIVE USER ACCEPTANCE (RUA)

a. CET of Weaponeer & Lasertrain 1.0 4.33 1.0 0.33

dated Oct 77

b. Relative User Acceptance Rank (2) (1) (2) (3)

TOTAL OF RANKINGS (LOWEST sýJRE (8) (6) (8) (10)

MOST COST AND TRAINING EFFECTIVE)

FINAL ALTERNATIVE RANKINGS (2) (1) (1 (2) (3)

111-35

< A I . .-
o Relative user acceptance is defined as the number of trainers

who preferred an alternative training device divided by the number who preferred

the M16 rifle-ball ammunition base case.

The rankings of the four alternatives of each o* the four factors

were summied to yield overall ranking scores such that the lower the score the

higher the overall ranking.

F. Other Current CTEA Methods

The remaining CTEA methods reviewed (Bridge '85, XM1 Tank, LAW,

and M16 Rifle) were of general interest for their processes, but did not contribute methods other than those already described. The LAW and M16 Rifle

weapon system training effectiveness studies represent methods applicable to A

fielded systems which are secondary to the present research.

11. Methods for the Analysis of Training Devices/Simulators

The process through which training devices/simulators ere selected and

developed for evolving materiel systems is complex and detailed. The entire

process does not fit within the limited scope of CTEA. However, CTEA may impact

on it at several points: first, to identify the need for a training device;

later to estimate the costs and effectiveness of proposed training devices alternatives; and compare alternatives with and without training devices with

various combinations of training devices. No special CTEA is necessary for the

identification of the need for a training device/simulator, nor for the comparison of training program alternatives with varying levels of training device use.

The estimation of the effectiveness of proposed training devices and simulators does require a special method. The prescription or development of training

device and simulator characteristics is clearly beyond the scope of CTEA prop- erly conceived.

Wheaton et al. (1976) reviewed a number of methods for estimating the

effectiveness of-pr;posed training devices and simulators and have found them

to be generally inadequate. They proposed a new method based on the assessment

of training variables (tasks, behaviors) and device variables (appropriateness,

efficiency, effectiveness). It has apparently been useful for estimating the

effectiveness of proposed devices, but its validity remains to be tested.i

The method developed by Wheaton et al. (1976) considers training device

effectiveness as "a function of the transferFpotential of the device, the learning deficit of the trainees, and the extent to which appropriate training techniques are utilized in the device (p. 7)." Transfer potential is given first as

a function of the comnonality of tasks in the training situation and the operational setting, and second as a function of the similarity of the training device

ana the operational equipment. The learning deficit is the difference between

entering proficiency and the proficiency required by the operational task. The

extent to whicn a aevice employs appropriate training techniques is determined

by an evaluation of each subtask trained by a device in terms of about one hundred principles and techniques; each subtask thus results in a rating of the

device as a potential means for overcoming the learning deficit represented by

the task. The transfer potential, learning deficit, and training techniques

analyses are combined to produce a device effectiveness rating for each subtask,

and these separate ratings are then combined to yield an overall effectiveness

rating for the device.

III-36

~I
---

TRAINVICE has been proceduralized for use by Army personnel with experience in task analysis and the development of training objectives. Five analyses

are necessary to apply the method: !

o Task conionality analysis

o Physical *imilarity analysis

o Functional similarity analysis

o Learning deficit analysi -

o Training technique assessment

The procedures for each of these analyses are given in the manual.

According to the authors of the method, these procedures represent an intermediate rather than an exhaustive level of detail and thus yield an acceptable degree of precision while avoiding the extensive efforts required by other

methods for estimating the effectiveness of training devices.

Training Device Requirements Document Guide: A Procedures Handbook for

Directorate of Training Development Project Offices for Devices (PM TRADE, 1979)

includes the TRAINVICE procedures for estimating the effectiveness of devices

as an aspect of developina training device requirements. It also includes procedures for media analyses to be employed within the process through which an

expressed training device need is analyzed to determine whether a training device requirement should be d'eveloped or the need should be met by alternative

media. If an expressed training device need is justified, other procedures are

undertaken to produce a training device requirement; these include the TRAINVICE

procedures. Other procedures of the training device requirement development

process are concerned with the formulation of training device concepts, cost

analysis of alternative training concepts, and validation of training device

concepts. In general, although in less complex form, the procedures specify a

LCSWt1 approach to the development of a training device and thus include as an

aspect of analysis separate CTEA.

g1-

111-37

:,• + ' . -- I _ J ] .... .. 1 • . . .. .. .. .. .
SECTION IV

CI'EA METHODS DEVISED 'BY LITTON

To meet recognized CTEA methodological needs, Litton has revised or de.-

veloped three methods: an empirical method for the prediction of training

proram -nd the estimation of their effectiveness; an interim method for

AMtakes advantage of information obtained in the process of training

soldiers for fielded weapons systems. This information is brought to bear on

training estimation and assessment questions early in the acquisition process

of a developing weapons system. The fundamental link is the identification of

a task or tasks on fielded weapons systems whose characteristics are comparable

or analogous to a task on the developing system. Having established the link,

the deta.As of training for the analogous task (fielded system) can be used to

estimate the requirements for the target task (developing system). Furthermore,

the performance of soldiers on the analogous task and the cost of training to

proficiency can bce used as estimates of corresponding quantities for the target task.

The method is applied on a task by task basis with an overall estimate of

training effectiveness calculated from the task estimates. When discrepancies

are found between proposed training plans and training for the analogous task,

the training plans are marked for further investigation. Likewise when the

assumed effectiveness of training or its cos~t differ markedly from measures of

analogous task training, training for those tasks 'is also marked for study.

ATM method has six steps: (1) definition of the critical tasks to be per-I formed on the developing system (the target tasks); (2) classification of the

target tasks to provide a basis for finding the analogous tasks; (3) analogous

task identification and selection; (J4) assessment of training for the analogous

tasks; (5) generation of estimates of training for the target. tasks; and (b)

aggregation of the effectiveness and cost measures across all tasks to obtain a

picture of training for the developing system as a whole.

1. Step 1: Task Definition

The basic data of the empirical method are the description of those

* operator and maintenance tasks essential to weapons system operation and mission accomplishment. This list is compiled initially eithur by the manufacturer

or proponent school early in the conceptual phase of the. LGSW. Without the

list of tasks, the analysis cannot proceed. The analyst, must either wait for

the task list to be produced or attempt to compile one - A critical task list

may be assembled by the analyst from a step by step des~-cipt~an of system oper- ation obtained from the manufacturer's technical personnel. Whee the concept

is sufficiently advanced, construction of a mock-up aid-- the exact description

of the tasks. When the Logistical Support Analysis Recocrd (LSAR) is available, the task descriptions on Sheet D (Figure IV-i) and the Failurt Analysis on

Sheet B (Figure IV-2) can be used to verify critical tasks.

IV-l
DRCOP-? 79lT-19

I I

II 46

c I w1 Iv ~~~ 

K7 I *

IV-2
I U

A Ug

aa

Li d~I Tr

T7 1i qT c

Lit

HS' IT 13 -K 9

T 4.

I L

2 4 1~ t.

- - ~±IV- 3 f~
- -------

The-verbs that describe the actions performed by the soldiers must

specify the nature of the action. These verbs will be used in the following

step as a basis for classifying the task. Verbs like "perform", "accomplish",

"continue", or "operate" are too general. to allow proper classification. As a

guide to the appropriate level of specificity, the analyst should compare the

verbs in his list with those in the task verb list (acceptable) and the general

verb list (not acceptable). Those description whose verbs are too general

should be rewritten with mnore specific verbs. It is likely that, on rewriting,

the "task" will prove to be a group of tasks, each of which should be considered

individually. The task verb list is presented in the.CTEA Performance Guide.

2. Step 2: Task Classification

The target task is classified by the action verb in the task description, the body of knowledge the soldier must understand as general background

to task performance, the stimuli provided by the equipment to initiate the task,

the response made to that stimulus, and the feedback provided by the system to

indicate proper or improper performance of the task. The task type is determined

by the action verb and the knowledge context for each task. The action verb is 'I compared with those in the task verb list. The list gives the task type to which

each verb meaning has been assigned. If the verb does not appear in the task

verb list, then the verb should be compared with the descriptions and examples A of each type which appear in the synonym list The verb may be an unlisted synonym for one of the verbs in the synonym" list. If not, the analyst should assign

the verb to that category which most closely describes it. The synonym list

appears in Annex C, Appendix 2 of the CTEA Performance Guide. LThe task categories, in some cases, include widely differing tasks

which cannot be distinguised solely on the basis of the verb. One basis for

distinguishing subcategories is the knowledge context of the task. For example,

computer tasks can be distinguished from electronics tasks or mechanical tasks

even though they all involve following procedures. (A list of knowledge contexts

is given at the end of the synonym l4st.) If 'the appropriate context does not

appear, the task type should be used without an associated context.

The determination of stimulus, response, and feedback types follows the

classification of Jorgensen and Hoffer (1976). The analyst examines the task

description and determines which stimulus, response, and feedback categories

best fit the task. The stimuli are characterized as to modality (visual, auditory, tactile, etc.), nature over time (static or dynamic; frequency of change), number, and organization (formatted or not; simple or complex patterns). Responses are characterized by mode of implementation (verbal, written, manipulating, etc.), nature over time (static or dynamic; frequency of change), number,

and organization. Feedback is characterized by modality (visual, aural, written,

etc.), time relationship to the preceding response and the following stimulus,

regularity, and frequency.

3. Step 3: Analogous Task Identification

As more CTEA are performed the resulting data may be stored in a

central training data base to facilitate future i~dentification of analogous

tasks. In the short term, the analyst must identify the analogous task intuitively. The first place to look for an analogous task is the previous generation weapons systems being replaced. If the weapon system is composed of maijor

IV-4
subsystems (e.g., radar tracking, Missile launcher), then a comparable subsystem on another weapons system may provide the analogous tasks needed. If

comparable subsystems cannot be founa, the analyst must look for individual

tasks.

To 'ýe an analogous tasik, the task on the fielded weapons system must

be in the same task type, have the same knowledge context (if any), and have

the same response type. In addition, the analogous task must have lesson plans9

or materials which describe the training for it in detail. If an analogous task

cannot be found, then an analytical procedure (e.g.*, TEEM or TLECEP) must be used

to identify necessary training elements. The assessment measures obtained from

these methods do not directly estimate or predict the performance of soldiers

w resulting from a training program.

If more than one potential analogous task is identified, the analyst

should attempt to choose among them. The choice 'is determined by the remaining

task information (i.e., other than the minimum criteria): the specific task

verb, the specific response, the stimulus t'ype, the necessary conditions.

These criteria should be considered in the order given. Only tasks identical

on a prior criterion should be evaluated on a later one.

4. Step 4: Training Assessment for the Analogous Task

For each analogous task identified, the analyst must obtain the details

of training. These details will be contained in the training programs and mate.-

rials and the specifications for training devices used in conjunction with 4 training.

The analyst must also obtain performance information to be used as a

measure of the effectiveness of training. The measure is defined as the propor- tion of soldiers achieving criterion performance on the task following training. If data from the hands-on portion of the Skill Qualification Test (SQT) are

available for that task, then they should be used to calculate effectiveness.I

If not, the written portion should be used in their place. Where no SQT data

are available, data from an end-of-course test should be used. In the absence

of all formal assessment tests, the instructor should be asked to estimate the

proportion of soliders who learn the task.

5. Step 5: Training Assessment for the Target Task

The training on the analogous task can be used as the training plan for the target task if no plan has been formulated. If there is a training *plan, then differences between the training plan and analogous task training

will indicate potential difficulties in training to be watched for in operational testing.

If the training for the target task is identical to that of the analogous

task, the effectiveness measure for the analogous task becomes the estimate of

effectiveness for target task.

If he raiingdiffers, the effectiveness in Step 4 becomes a criterion

aganstwhth heproposed effectiveness is compared. Care must be taken that

the ritrio diensonsfor the two are the same. If more than one analogous

IV-5

- - - - - ~ - - - -RIP. 
task has been identified, the data can be combined in the following manner. If

the training is roughly the same, the effectiveness measures should be averaged to obtain the estiwate of effectiveness for the target task. If the training

differs among the analogous tasks, that with the highest effectiveness Measure

should be chosen.

As data on trainin~g are obtained from OT I and OT II, the direct estimates for performance on the task should be substituted for the measures based

on analogous tasks.

Since training for different tasks may develop at different rates or analogous tasks may be found for only some of the critical tasks, it is entirely possible that some of the tasks will have only analytical measures while others

have performance measures. Therefore, combining measures into a single aggregate effectiveness measure will not be meaningful. Even if all measures are

performance measures, a reasonable set of weights for combining the measures

may be difficult to deduce.. In both cases, a verbal rather than numierical over-I view of training is suggested.

The method is illustrated in Figure IV-3. Each rectangle representsI

a step of the process, while the processes indicated in the circles show the

relationship of the process to other processes of a general CTEA model to be

described in Section V (Figure V-1).

B. Trainability Analysis

Once tasks have been identified and alternative means of training those

tasks have been predicted or developed, it is necessary to determine that,

given the characteristics of' the personnel who will man the system, the tasks 11 can be trained to required levels of proficiency. The purpose of trainability analysis, thus, is to examine the interactions among tasks (especially standards),. training program alternatives, and systemr personnel characteristics.

The review of the literature, however, has revealed no explicit method for

trainability analysis. According to a TRADOC official (Telephone Interview,

26 June 1980), however, such methods are now under developmient and will probably be available at some time in the future. This oftficial agreed that it

would be reasonable to apply the following stopgap method in the interim.

The objective of this method is to identify tasks that are of great or

J ~ moderate concern a:9 regards the trainability issue. Once those tasks have been

identified, certain trade-offs may be considered and ren~ouznended: the revision

I or reconsideration of certain training program alternatives; the alteration of

system concepts or designs; and the revision or establishment of personnel selection criteria. I The risk that a task cannot be adequately trained given an existing or

predicted training program and system personnel characteristics is judged to be

high, moderate, or low. Each system task is thus evaluated against each training program alternative. The criticality of each task is also judged to be high,A

moderate, or low. These twin evaluations are then used to form a trainability

analysis matrix for each task as trained by each training program alternative.

The matrix is illustrated in Figure IV-14. The cells indicate the level of the

iv-6
AN4ALOCOUS TASK IOINrIPICATI@N TRAINING

ASESSMINT

IDINTIOV FOR ANALOGOUS

ANTECEDIN? TASK

SYSTEMIoft

S4USS VTEMS

0DENT IFYV YNAININ(%

MWNWOD AMC)E

START CO 11MEDIA

AP TASTAS TYPEI II NOTASK DEINITI yesSTM TS

AVILOLPAE iAN

ACIO ANLGU @SIN PAN

IUSE LNAL

TTAO TAS

ST TF

PROCESS

TASKIN u0T

LOAININo EFFECTIVNES

ALTERNATVES ANA TASK

RESLUTON P TASK CLASSIICTION COSYTMPAEEFCTVNS

WTASKAS

TASYPEIL

KNOWLE ?,E

CONTEXT AN

RESONSPTPNSE.RNA

TYPES

VLSIRGIED A TSPEIICTRETTS

Figur IV-. Anlogo s Tas Metho

ANLOOU -7SN LN
analyst's concern. Where concern is great or moderate, the analyst must resolve

it by conducting trade-off analyses (above) or by calling attention to the concerr and recommending additional studies.

TRAM=N RISK

HIGH MODERATE LOW

Great Moderate Little

= onCOcer concern concern

Moderate Moderate Little

Sconcern concern concern

Little Little Little

Sconcern concern concern

Figure IV-4. TASK TRAINABILITY ANALYSIS MATRIX

C. The Cost Process Model

Because resources are limited and costly, choices among alternative systems

or programs must be based on the allocation of these resources among competing

demands. For example, if' more than one alternative satisfies the missionrequirement (equal effectiveness), the least nostly alternative is preferred.

Obviously, the cost analyses, whether it be absolute or incremental, in a CTEA

is essential.

A cost model orders and simulates the many actions involving the expendi-

"tures of resources (including time) for, in this case, A-my training. The cost

model allows the analyst to predict or recapture in a systematic manner I:hose

expenditures of resources that were or will be germane. The first requirement

of a model is that it be relevant to the need, i.e., be able to predict or, capture the costs of training Army tasks or systemo. Army training takes place in

the institution (schools) and in the units. Most of the institutions have base

or classroom inst,'uction and instruction in the field. Institutional base

training can be self-paced or instructor oriented. There is individual '.raining

and collective training in the Army. An Army training cost model shoula be Army oriented and have the capability of estimating training costs that consi_`,e.r all

pertinent variables.

During the literature review, Litton reviewed many cost models and approaches

to costing of training, eleven of which have been abstracted and appear in Appendix C. Litton sought a model that would provide the capabilities required

rather than to design a new one. None met all the requirements. For example,

the Army Life Cycle Cost Matrix, although Army oriented, is better structured

IV-8
for' and focused on budgetary, finance, and accounting purposes rather than CTEA.

The TECEP -model is focused on institutional base training but could be modified

to meet the other requirements. However, zoiiplete modification of the TECEP

model results in the Litton proposed cost model.

The C"1F.A analyst must retain a high degree of flexibility in relating the

form of the data available with the equations herein. For example, where the

equations contain variables on a "per' student" basis, these variables are usually then multiplied by the average number of students to obtain total costs. If,

however, total coat data are already available, then these should be entered

directly as the product of both of the variables.

The Litton cost model is designed to capture the significant costs of training a task or an entire program. If Cost elements are available on a training

program basis, then the model output will be the total cost estimuate of the training program. More likely, the CTEA analyst will have to deal with analogous tasks

and will have to iterate the model for each such task. The grand total of the

Costs of training all the tasks, of course, will give a cost estimate of the

training program.

The Litton cos t model is intended to aid the CTEA analyst to prepare recommendations to the decision maker regarding choices among alternatives. It is

not intended for budgetary purposes. The Litton i.iodel, however, captures per- sonnel and unit costs in order to convert training time and differences in grades of personnel involved into a measure compatible with the other factors in the

decision process: dollars.

It is necessary for the CTEA analyst to have some understanding of the Five

Year Defense Plan (FYDP) and its associated budget programs to assist in data

collection. Figure IV-5 shows the ten Army programs currently in use.

* Represented by the programs and subprograms are resources: people, money, materiel, and facilities. It would appear that all people, money, materiel, and

facilities required for training would be provided under Program 8T, training.

This, however, is not the case. Program 8T carries resources for almost all

institutional training but training resources are found also in Programs 2, 3, 5, 6, 7M and 8M. Program 2, for example, carries training resources for the

individual and the collective training of personnel in General Purpose Forces.

On the other hand, such activities as training developments, combat developments and direct assistance to unit training by institutions are not technically institutional training but are accounted for in Program 8T. Institutional training

of individual medical skills is accounted for in Program 6M.

The CTEA analysiý must also understand where various levels and types of

training are trained. Figure IV-b is a matrix which shows this. (It should beI noted that the term One Station Unit Training (OSUT) is not unit training but re- fers to the institutional training program that combines aspects of basic training

(BT) and advanced individual training (AlT) while the trainee is assigned to one

training unit.) The matrix portrays the usual situation. There are rare exceptions where, for a special system, some crews/teams will be given collective F training in the institution.

IV-9
Progam istrategic P Forces rograz 2 - General Purpose Forces Program 3 Intelligen and &4omwnications

3 C C o m un i oa t I M n

31 " IntellAlence 

i

30 " Other

Program 4 

-

1ift/elift "Program 5 Guard and Reserve Forces

"Program 6 Research and Development

Program 7 Central Supply and Maintenance 7S Supply

7M " Maintenance

Program 8 

Training, Medical, and Other Personnel Activities

t " Training

8M " Medical

0 Other

Progr• 9 Administration Activities and Associated Program 10 

Support of Other Nations

Figure IV-5. Army FYDP Prograns

17-10

4 4'.'
Institution Unit

Base Field G, L, M

Ynidividual OSUT Tasks not taught

rzin..ing to proficiency at

institution

or

Refresher training j

BT+AIT

*1 G, L,M

Collective Essentially all

Training collective tasks

OSUT - One Station Unit Training

BT - Basic Training

AIT -Advanced Individual Training

G - Garrison

L - Local Training Area

M - Vajor Training Area

I!i Figure IV-6. The ýLmy Training System

41

M,'vISM
The Litton cost model concentrates on the initial individual and collective

training required to attain proficiency on the system and operational capability.

Costs attributable to refresher training are omitted. Proficiency is assumied

when the soldier successfully completes the appropriate SQT and tne crew/team

successfully completes the appropriate section of the Army Training and Evaluation Program (ARTEP).

The costs of training can be either direct or indirect. Direct costs associated with institutional training consist generally of the following:

1. Pay and allowances of instructors and staff.

2. Cost of support rendered to the institution (including pay and

allowances of personnel) by T`OE units whose missiorn is to

support thL- institution (Program 8T).

3. Student pay and allowances.

~4. Student travel pay to the institution.

5. Cost of ammuunition expended fur institutional training.

6. Student per diem entitlements at the institueion. I7. Depreciation of equipment dedicated to institutional training.

8. Cost of consumable supplies and material.

9. Cost of contractual services.

10. Institutional overhead costs such as the pay and allowances of personnel i~n offices of t.-he Commandant, Security, Director of

Logistics and of School Training Brigades.

Indirect costs of institutional training are those costs not directly and I wholly attributable to training but all, or at least part, of which should be

charged to training. These costs include:

1. Cost of support rendered to the institution by Program 2 TIOE

units. Military pay, operations and maintenance, and TOE

A equipment depreciation are apportioned according to the mandays of support rendered. It is recognized that these units

are in the force structure to meet Joint Chiefs of Staff requirements and that pay and allowances, for example, would be

a cost whether or not they support institutional training.

However, time diverted to support institutional training

should be accounted for - especially when comparing systems.

This may be done by converting this time to dollars. Additionally, using pay and allowances to do this permits consideration of the different grades of personnel requirec by competing systems.

IV-12

-- '--
2. Proportionate share of the costs of base operations, family

hou.,ing administration, base communications, base medical

support.

The cost of initial proficiency training that takes place in Program 2

units is somewhat more complex. The niunber of hours the unit devotes to individual training on a particular system can be estimated from careful study of

the unit's trai 1i .ig schedule. A pro rata share of the unit's annual costs of

the following should be included:

1. Spare parts. Rather than using the moment of equipment failure

or repair as the basis for attributing total repair costs, it

would be more accurate to charge a pro rata share to training

based on the fraction of the use time allotted to training or,

if known, the fraction of the mileage attributable to training, in the case of vehicles.

2. POL.

3. Ammunition expended for training on the system. This can

probably be accounted for accurately without using the pro

rata distribution (except for costing by task).

4. Military pay and allowances - again based on the previous

discussion concerning converting to dollars the time diverted

from primary mission.

5. Construction of special ranges or training facilities (e.g., Redeye Moving Target Simulator (MTS)) for training on the

particular system. Here, it is important to cost only the

proportion of costs attributable to initial proficiency

training.

6. Acquisition costs of training devices charged to the unit and

specifically for training the particular system.

7. Operation and maintenance of training devices pertaining to

the system being studied. These may be costs incurred by the

Training Aids Service Center (TASC).

Costs may be divided further into variable costs and fixed costs. Variable

costs as used herein are, in general, those that vary with changes in the training load. Thus they can be used to estimate the changes in total cost as the

training load changes. Fixed costs, on the other hand, are, within certain

limits, insensitive to changes in the training load. Support by Program 8T TOE

units and equipment depreciation are examples of fixed costs. Ammunition expenditures, student pay and allowances, travel pay, and per diem are examples

of variable costs.

The Litton equations constitute the cost model for initial training of a

system. The equations are patterned after but extend those developed by the

Training Analysis and Evaluation Group (TAFG) of the U. S. Navy and reported in

TAEG Report Number 16, "A Technique for Choosing Cost-Effective listructional

iV-13
Delivery Systems," April 1975. The Litton model employs FORTRAN representation

for the names of the cost factors. The names and definitions are intentionally

close to the parallel factors employed oy TAEG. The L~asic inpuzt variable selection philosophy, also similar to TAEG, employs five genert c categories of cost

factors:

1. Facilities

2. Equipment

3. Instructional Materials

4. Personnel

5. Miscellaneous

There are significant differences between the 'two models. The Litton model

expands TECEP coverage of institutional training costs to include inter alia

institutional field training costs. The Litton model also adds cost factors for

the individual and collective initial proficiency training done in Program 2

(force) units. Thus, a number of TECEP' s input variables have been omitted,

and new variables have been introduced into the Litton model.

The Litton model iz based on certain assumptions. Some of these ar. discussed in the explanation of the cost equations, presented in the User's Guide

that accompanies this report (Cost and Training Effectiveness Analysis Performnance Guide).

The Litton model assumes:

1. Individual training beyond initial proficiency on the system is not included. Thus, refresher training is omitted.

2. Unit effectiveness training beyond initial, operational capability is not included.

3. Costs associated with loss of life, property, or time

occasioned by accidents are excluded.

More detailed assumptions, equations, and examples of use of the model are

presented in the CTEA Performance Guide published as a companion to this report.

IV- 14
SECTION V

SYNTHESIS OF CTEA METHODS

The LCSMM has been examined to determine the requirements for and purposes

of CTEA. This examination resulted in the identification of probable CTEA

locationi in the LCSMM. Methods applicable to CTEA have been identified

through a literature review and new methods have been devised to fill gaps in methodology.

The CTEA methods identified through the literature review and those developed to meet recognized needs have been analyzed to reveal their embedded

processes. The analysis was based on a general CTEA model that identifies

necessary CTEA processes and thus points to a set of input-data situations and

to a taxonomy of CTEA process methods. The general model and taxonomy produce

a synthesis of methods that better meet the needs and conditions of the acquisition process than any single method.

A. The General CTEA Model

The selection of CTEA methods in the LCSMM is controlled primarily by the

available data and the issues to be resolved by the CTEA. Method selection

begins with the identification of the basic analytical processes made necessary

by the relative sparseness or richness of the data and then proceeds to the

selection of ways to accomplish those processes. After these initial selections, the means of addressing the specific issues and questions of the CTEA

are considered.

This approach to the selection of CTEA methods is illustrated in Figure

V-1. The analyst first asks whether a training program actually exists. The

program, including tasks to be trained, methods to train tasks, personnel to

be trained, locations of training for various tasks, etc. may be in the form

of a training description not yet actualized or in the form of an ongoing

training activity.

Depending on the answer to this question, the analysi s proceeds along one

of two primary tracks. If there is no training program, the analyst next determines whether there is a task list complete enough to permit predictions or

estimations of alternative training programs. If there is no such task list,

or if it is not complete enough to permit the prediction or estimation of alternative training programs, he/she selects methods for the accomplishment of

the task-generation process. Once a task list has been generated, or it has

been established that a complete enough task list exists, he/she selects methods

for the training-prediction process. After selection of such methods, methods

are selected for the remaining processes: estimation of effectiveness, costing of training program alternatives, cost effectiveness comparisons of training

program alternatives, and resolution of issues.

If there is a training program, the analysis proceeds along the other primary track. In this event, the second question concerns the existence of training program alternatives. If there are no alternatives the analyst decides how

to predict or estimate alternatives and how then to estimate their effectiveness.

Once it has been establi3hed that alternatives do exist or methods have been

V-i
cc 5

V-24

ilkU
selected for the prediction of alternatives and the estimations of their effectiveness, the analyst next determines if comparable effectiveness data for all

alternatives exist. If there are no such comparable effectiveness data for all

alternatives (i.e.*, comparable performance measurements for all alternatives or

comparable estimates of effectiveness for all alternatives), then the effectiveness of alternatives for which there are no data or performance measures are

estimated.* This requirement is based on the assumption that, even when one or

more alternatives are actual training activities, the analyst will not be able

to obtain performance measures for the current CTEA by requesting the administrations of tests and that effectiveness comparisons of alternatives must

therefore include comparions of estimated effectiveness. Once it has been determined that comparable effectiveness data for all alternatives exist, or methods

;A. for estimating the effectiveness of all alternatives have been selected, the

analyst selects methods for the remaining processes: cost analysis of trainingJ

program alternatives, cost/effectiveness comparisons of training program alternatives, and resolutions of issues.

ii The six basic input-data situations in the general model are:

1. No Task iAst and No Training Program. In the worst case, function and

job analyses have n-ot be-en conducted or are not available to the CTEA analyst.

Task lists and training programs neel to be estimated in order conduct the CTEA.

2. Task List but No Training Program. If function and job analyses have

been completed but no training program has been designed, the training program

needs to be estimated or predicted for use in the CTEA. If the task list is

very rudimentary (i.e., does not contain complete task statements) then the

analyst. needs to use a method for estimating the training program that is not

overly sensitive to gross task information. A training program can be estimated

and analyzed from a rudimentary task list but the result will be a rudimentaryI

estimate. Since training developmient is iterated during acquisition, however,

a rudimentary estimate is adequate for a CTEA early in the LCSMM-I CTEA methods

used with early, rudimentary data need to be robust in the face of vague, incomplete, and inaccurate task information.

If this situation occurs in the demonstration and validation phase or

later, precise task information is highly desirable. Some task and training

data may be available from the manuffacturer, who develops and implements train- ing for DT/OT I. Task information may be available although a training program

may not be available in printed form. The CTEA analyst can predict a training

program based on the refined task information and the predicted program is a

* reasonable estimate; thus, CTEA estimates based on the predicted t~raining are

also reasonable.

3. Training Program but No Alternatives and No Effectveness Data. The

training program implies the existence of a detailed task list that may contain

complete task statements. The earliest training program is likely to be the

one produced by the manufacturer although it may be a service school product.

Since there are no alternative training programs, the relative merits and costs

of the extant program have not been estimated.

J4. Training Program with Effectiveness Data but No Alternatives. In this

instance there may be a detailed, complete task list. The training program is

likely to be that produced by the manufacturer, and the effectiveness data are

V-3
likely to be those from DT/OT I. These data are likely to be anecdotal and may

not be reported since training evaluation is not a requisite in that test. The

effectiveness data may be expert estimates or judgments. Since there are no

alternative training programs, there are nc comparative estimates of cost or

effectiveness.

5. Alternate Training Programs but No Effectiveness Data for All Alternatives. The prograns are likely to be those produced by the manufacturer and

by the service school. One or more may have effectiveness data of some sort

while others do not. For example, the manufacturer's training program way be

tested in DT/OT I while the service school program remains untested. Effectiveness data for one or more programs need to be produced, perhaps by estimation,

if all alternatives are to be compared for relative effectiveness and cost.

6. Training Program Alternatives and Effectiveness Data for All Alternatives. This situation implies a fully developed task list. One or more

training programs may have been tested to yield performance measures, and the

programs may have been tested in other research. The effectiveness measures

may differ in type and in whether they are empirical or judgmental. This situation is likely if the training programs were tested in DT/OT II during the

full-scale development phase of the LCSMM.

The general CTEA model thus identifies the total set of required CTEA

processes and relates these processes to specific CTEA as defined by their input data. Because each process may be carried out by means of a variety of

possible methods, each process suggests a category of a taxonomic scheme through a which those methods may be analyzed to reveal their embedded processes. ;

B. A Taxonomy of CTEA Processes

All of the CTEA-applicable methods reviewed and those devised by Litton

have been analyzed to determine the methods through whicn the required CTEA

processes have been or may be carried out. In Table V-1 the required general

CTEA processes are arranged along one dimension while the various CTEA-applicable

methods are arranged along the other. Each column constitutes a class of pro- J

cesses in which each method, depending on data conditions and other factors, may

be employed to carry out the process identified. Rows display the general appli- ]

cability of methods to CTEA.

Table V-I shows how the various CTEA and CTEA-related methods fit the requirements of the general CTFA model. Unmet needs are also revealed. The one

presently available automated cost model, the Navy's TECEP cost model, does not

cover the costs of institutional field training or of unit training. Available

process methods for t..a prediction of training programs do not include a formal.

method for predicting the training of new tasks from the established (and thus

validated) training of similar existing tasks; without such a method there can

be no formal application of the accumulated knowledge of how tasks of various

types can be effectively trained. Finally, there are no methods for resolving

the issue of tr.inability. How, for example, does the analyst determine that a

task may no* be trainable or may pose such a training risk that operational effectivenes5 may be jeopardized? Litton's trainability analysis is offered as

an interim process until trainability is more thoroughly explored.

v-4
;cnh

1.4I

cnn

CA.

Do 0 )a4

ll, .. CC.)ý- i. 1-4 uL LL) >

W..

EZ 1 -4 Qi

rCNýDTEMTHODS

V-5
C. CTEA Situations and Strategies: A Synthesis of riocesses

The general CTEA model and taxonomy cescribe a variety of poAsible CTEA

situations differing in input information and issues addressed. 'he situations

correspond roughly to. the phases of the LCSti but do not correspond exactl3 because systems do not progress througn the acquisition cycle in lock-step manner.

Potential input information includes task lists, training programs, and effectiveness data. In some cases the data may exist but may not be available to

the CTEA analyst and cannot be applied in the analysis.

The final challenge in the synthesis of CTEA methods was to devise general

plans or strategies to guide the selection of processes required in the inputdata situations. Each situation identifies a set of required processes and

suggests an approach to meet the CTFA objectives.

1. Generation of Task List

CTEA require knowledge of what is to be trained, but those conducted

early in the acquisition process may not have access to such knowledge, at least

not in the form of task lists. In this case an initial effort witnin the CTEA 12i must be to generate task lists. Such task lists need not be as detailed as the

formal task descriptions that will eventually be developed (for such documents

as the Soldier's Manual), but they should be detailed and comprehensive enough

to permit a reasonably reliable estimation (or prediction) of the training program that will support the tasks. But even when only the most general task

information can be obtained, training programs can be at least tentatively •! estimated.

The person responsible for an early CTEA may encounter data availability

conditions that range from no task list at all to existing formal task lists.

The worst case occurs when there is general information about the new system but

there are no TASA data. If a formal task list is available the analyst proceeds to the next strategy and starts with training generation.

a. The Army has developed these methods for generating task lists on

the basis of subject-matter expertise:

(1) DIVAD Gun CTEA Process for Generation of Task List. If there

are no TASA data, but general information about the new system and its training

devices and detailed knowledge about similar systems is available, the "generic

CTEA" processes used for the DIVAD Gun apply. Training program alternatives are

predicted on the basis of the established training practices of existing systems

of the same class and knowledge of the new system and its training devices. The

steps are:

o Analyze descriptions of new system and place new system in

class of similar systems.

o Identify generic tasks of class of systems. (These are

tasks or task types that tend to be conmon to all sy",tems of the class.)

o On the basis of general knowledge of new systems and its

training devices, revise generic task list to make it as specific as possible

to new system.

11-6
A task list generated in this manner is used to estimate

effectiveness of alternatives rather than to predict them; the alternatives

are based on training practices of the class of weapon syse•ms.

T(2) Hawk Product Improvement Program (PIP) Training Development.

This process is app-Icable if there are no TASA data but tasks are inferred

from the contractor's draft technical manual and from detailed knowledge about

the antecedent system. It was applied to the generation of task lists for a

PIP and may be limited to those situations. The steps are:

o Secure task lists of antecedent system.

o Analyze technical documentation (draft TM) and compare

with task lists of antecedent system.

tm oie o Revise (or add to) task lists of antecedent system to make

them consistent with technical documentation of new (or modified) system.

It may be necessary or desirable to develop some new tasks

through trial performance on new equipment.

(3) Roland Training Development. The Roland training development process is applicable if there are TASA data and descriptions of personnel

functions in the technical documentation, but no formal lists of tasks. It

assumes that detailed knowledge of the antecedent systcm is available. This

process represents a considerably higher level of analysis than the previous

two processes, and the prediction of training programs may therefore be more

precise. It identifies critical tasks and selects training sites. It does,

however, require an analyst who is a subject-matter expert with experience and

training on the system since precise judgments are required. The steps are:

o Secure contractor's (manufacturer's) LSAR D sheets and

draft TM.

o Compare tasks in LSAR with descriptions in TM and revise

to make consistent with LSAR, TM, and analyst's expert judgment.

o List tasks on Task Selection Worksheet.

b. Strategy for CTEA Lacking Task Lists. When there is no task list

the analyst must perform all processes from generation of the task list to

resolution of issues. In this situation the analyst uses expert judgment to

generate the task lists, therefore the analyst is also likely to use expert

judgment for prediction of alternative training programs and estimation of

training effectiveness. Formal analytical methods also can be used to predict

the training programs and then also used to estimate the effectiveness. The

estimates for effectiveness need to be in the same terms, i.e., produced by

1 the same method, to be comparable. Similarly, cost estimates for comparisons

need to be in the same terms.

2. Prediction of Training Programs

When there is a task list but no training prcgram the analyst proceeds

with the production of alternative training programs. Prediction of training

%1-7
programs is accomplished in several ways. First is exvoert knowledge of the

materiel system or expert knowledge of the class of weapons. Training programs

estimated early in the LCSMM need not be as precise as those estimated later.

Analytical methods such as TEEM, TECEP, and the Analogous Task Method predict

alternative training programs.

a. These methods are recommended in particular if there is one training program already designed but there are no alternative programs because

these methods are recommended for the comparison of alternative training

programs.

(1) TEEM Process for Predicting Training Programs. TEEM predicts

training programs by describing both tasks and certain elements of training pro-.

grams in terms of the same set of variables, called a "metalanguage." The variables, or metalanguage, were identified through a literature search and generally

represent research findings concerning relationships between task condition or

requirements and various types of instructional media and methods. Variables

relating task conditions or requirements to content and content structure have

also been tentatively identified, but these are not yet employed in the processes

of the method.

Tasks are first described in terms of variables classified as

stimulus variables, response variables, and feedback variables. They are then

logically grouped so that media selection may proceed. Each task results in the

selection of one or more media devices or forms ideally suited to that task, but

the total set of such devices or forms is reduced in response to real-world constraints so that only a few media devices or forms serve all tasks. The functional contexts of task groups are similarily described in terms of method variables (these relate task contexts to instructional methods), and instructional methods are thus selected.

The input of this training-program-prediction process is task.

information in various formats from which at least nouns, verbs, and modifiers

descriptive of the task may be derived. Briefly, the task information must indicate what is done to what within what limits or under what conditions. The

output is a set of media and instructional methods. Further development of the

method may result in the inclusion of specifications for content and content

structure.

(2) TECEP Process for Predicting Training Programs. TECEP predicts training programs in terms of suitable instructional algorithms and delivery systems. Each task or training objective of a training program is matched

with one of 12 learning algorithms. The matching is accomplished by comparing

the statement of the task or training objective, including the verbs, with the

lists of action verbs, behavioral attributes, and examples of training objectives included in the algorithm charts. Once each task has been matched with

an algorithm, all tasks are grouped or classified according to the algorithm

selected for it. Then, two or more types of delivery systems are selected for

each group of tasks. Such delivery systems are to be considered only candidates

for the developing training program, however, and must survive tests of practicality and other constraints.

(3) DIVAD Gun Process for Predicting Trainin Programs. This

process method might be termd the wwost-case method for the prediction of
training programr alternatives. Training program alternatives are formulated

on the basis of established training practices within the cla~ss of weapon

sYstems (or other materiel systems) to which the developing System belongs.

Alternatives are likely to consist of different arrangements of operational

equipment and various training devices (simulators, etc.).

(4) Analogoua Task Method for Predicting Training Programs. Tasks

of fielded systems that are analogous to the tsk for which training programs

are being predicted (target tasks) are identified. Once it has been verified

that an analogous task is similar enough to the target task to permit the inference that the same training program would be effective in training both

tasks, the training program of the analogous task becomes, in effect, the

training program of the target task.

b. Stratueies for CTEA U4sin Predicted Training PrograMs. Training

programs may beetimated by informal ,judgment or by formal analytical methods

such as TEEM, TECEP, and ATM. The formal analytical methods are recommiended to

predict training program alternatives to be compared with either those predicted

by expert judgment or with existing training programs. Large differences in the

training programs predicted by divergent methods signal areas that require closer

attention.

If at least one training program has been predicted, informal expert knowledge is not recommended for prediction of alternatives. Since the

tasks are known, the formal anal ytic methods can be employed. Whatever the

method employed, different levels of training devices and media are estimated

for comparison with the devices and media in the existing program.

3. Estimation of Training Effectiveness

Ideally, effectiveness is determined empirically, but because of the .1 impossibility of obtaining direct empirical data for systems not yet developed

to the prototype stage and the difficulties in obtaining it even for systemis

in prototype or later stages, estimation is often necessary.

a. Three kinds of methods for the estim~ation of effectiveness appear

to be possible: analys s of how well a proposed training program "fits" the

tasks to be trained; expert judgments about the relative effectiveness of a

set. of proposed programs; and generalization of the demonstrated effectiveness

of the training program of one task to the training program of all nearly idenitical (analogous) tasks.

* ~(1) TEM1 Process for Estimating Effectiveness. In this method

an efficiency metric that expresses how well a training program "fits" the tasks Li to be trained is taken as an estimate of effectiveness. The metric is the ratio

of measures, derived from the media selection processes, of constrained, realworld programs and unconstrained, ideal programs3. That is, each task results I ~ in the initial selection of a media type ideal for that task-, but then tasks

are logically grouped and the total set of media is reauced to an average, compromised media selection for each functional task group or training situation,

simply because real-world constraints would not perrit the costly and possibly

unmanageable ideal. Thus, the measures or scores associated with the sele-c-_

tion of the constrained, real-world set of media are divided y 'the measures
or scores associated with unconstrained, ideal set of media to produce an

"efficiency" ratio. The assumption is that if a training program were totally

unconstrained in the selection of media, then the media type selected for each

task would be ideal for that task and the effectiveness of the training would

be greater than if the program were constrained and media selections were less

than ideally suited to each task.

(2) DIVAD Gun CTEA Process for Estimating Effectiveness. This

process method for the estimation of effectiveness cannot be described in detail because of a lack of information, but it was clearly informal as applied

in the DIVAD Gun CTEA. One or more analysts, experts on the class of weapons

to which the DIVAD gun belongs, examined the proposed alternatives in light of

their knowledge of the "generic" tasks of the class of weapons and much knowleage as they had of the DIVAD Gun and the ranked them on the basis of judged effectiveness.

(3) Analogous Task Method for Estimating Effectiveness. Tasks of

fielded systems that are analogous to the tasks for which training programa are

being predicted (target tasks) are first identified. The training prgrams of

the analogous tasks are adopted as the training programs of the target tasks,

arid the empirical effectiveness data of the analogous tasks become estimates of

the effectiveness of the analogous task training program applied to the target tasks.

(4 ) TRAINVICE Method for Estimating Effectiveness. To estimate

the effectiveness or proposed training device alternatives, the CTEA analyst

applies the TRAINVICE procedures for the following analyses:

o Task commonality

o Physical similarity

o Functional similarity

o Learning deficit

o Training techniques

For each subtask to be trained by a device these yield an

estimate of device effectiveness, and the separate estimates are then combined

to yield an overall estimate of device effectiveness. (According to Training

Device Requirez.,ents Documents Guide, DARCOMI provides cost estimates for training devices though the device project office of the proponent school. DTD may

be required to provide some cost data. It is also important to note that,

since there is not yet a way to combine the eft'ctiveness estimation metrics

of both training devices and training programi to yield a single metric as an

estimate of the effectiveness of a training program with a training device,

the estimation of the effectiveness of trair.ang devices and the estimation of :[: ~~the effectiveness of training programs must be considered as -n'Z iz_--ues.

"Once the most effective training device Ulternatives have been selected, hcw-

,ever, it may e considered within the context of training program al-ternatives

as the costs and effectiveness of these alternatives are estimated. 'iven the

characteristics of the most effective trairinz device alternatives, -TA, fcr

exanple, cculd be used to ostinate the relative effectiveness Cf :-.ra.i.. - rczrazrs witr. and w-itcut the .e- . )

r-i
(5) TCA Process.for Estimating Effectiveness. In TCA the estimate

of effectiveness is the Training Consonar- Ratio (TCR). Tasks are described

in terms of b5 psychological variables a .ien each combination of medium and

method is described in terms of the same b5 variables. The goodness-of-fit

between the task descriptions and the descriptions of the combinations of me..._

and methods used to train the tasks is expressed in terms of the Training Consonance Ratio. The TCF is almost exactly analogous to the efficiency ratio oF

TEEM since the same lists of variables are used in essentially the same way.

b. Strategies for Estimation of Training Effectiveness. If Aere are

alternative training programs but not all of the programs have effectiveness

data, the analyst starts with effectiveness estimation.

It is reasonable to apply expert judgment to estimate effectiveness

of training programs that were predicted by expert judgment. If a task is similar to a task in an existing system about which the analyst has knowledge, and

if it is known that a certain type of training results in the required level of

proficiency on that task, then that training approach can be estimated for the

new training program.

If the analogous task method, a formal analytic procedure, was used

to predict a training program, then it is also used to estimate the training

effectiveness of the program. If TEEM was used to estimate two or more training

alternatives, the TEEM efficiency ratio is the estimate of effectiveness. Training consonance analysis (TCA) can be used to estimate effectiveness no matter

how the programs were predicted: the training consonance ratios produced by

TCA are the effectiveness estimates. TCA estimates effectiveness of both existing and predicted programs. It also diagnoses training deficiencies, excesses, and redundancies for revision.

The state of the task information (e.g., very rudimentary, refinec, . ij or fully developed) determines the precision of training program estimation and

training effectiveness estimation. Within a comparison, the levels of information concerning training effectiveness must be common for them to be comparable.

4. Cost Analysis of Training Programs

The estimation of the costs of training programs is a complex process

involving many variables, but estimation must be precise as possible since cost

effective decisions about competing alternatives are to be made. The one cost

process method selected for' inclusion in the manual appears to offer the possibilitiy of accurate cost analysis along with a reasonable approach to the complexity of the costing problem.

a. Litton Cost Model. The Litton cost model is a modification of the

TECEP cost model. It expands the coverage of training costs to include both

institutional field training and initial proficiency training in the units, and

it should therefore provide more precise analysis of training costs than TECEP

cost model. Because it is computerized the complexity of the problem from the

analyst's point of view is somewhat lessened. The Litton cost model is intended

for cost estimates only, not for budgetary requirements.

V-l1
b. Strategy for Cost Analysis of Traininc Programs. The modified

TECEP model is recotamended for all CTEA situations in the LCSMM. If one trairing program is estimated by expert judgruent, or if one program actually exists,

while the others are estimated by using the formal analytic methods, it is imapor' ant to note that the cost estimates must be comparable in level of detaail

and precision if valid cost comparisons among alternatives are to be made. .

5. Comparison of Alternative Training Programs j .

Once the effec..veness and costs of competing training program alternatives have been estimated or measured, the alternatives must be compared with

each other and with externally imposed standards or criteria. The purpose of

the comparison is the selection of the most cost effective of the alternatives

which meet the criteria. In order to assure that the conflicting demands of

adequate effectiveness and minimal costs are met, some sort of rational scheme

Smethod for comparison must be employed.

In established practice, schemes or methods for comparison range from

informal comparisons of rankings of alternatives on both effectiveness and

costs to formal cost benefit analyses. Further, because it may be necessary

or desirable to consider factors in addition to cost and effectiveness in the

selection of a training program, there are methods for incorporating these additional factors in the decision metric.

a. The following process methods by no means represent all possibilities methods for the comparison of alternatives, but they do represent established and provide for the requ~irements of most CTEA. J -

(e) TEEM( Comparison of Training Alternatives. In this method the

estimate of effectiveness, the efficiency ratio, is not a strong one but it is

the best available in a very early CTEA. Cost effectiveness (C/E) ratios are

f :rmed and compared. The analyst selects the efficiency ratio that represents

the lowest acceptable effectiveness and this step is difficult and judgmental.

The plot of the series of efficiency ratios (Figure 111-7) is closely examined

for pronounced decreases in efficiency as a result of the reduction of the media.

Efficiency may drop off slowly at first as medium after medium is removed from

the training description but then at some point it begins to drop off quite

sharply. At this point or sooner the analyst will probably decide that minimal

acceptable efficiency has been reached and that the media set can be reduced no

further. Then, because the relationship between cost and efficiency (and therefore effectiveness) is not likely to be linear, he/she will choose media sets

represented by several points higher than the minimal point and then compare the

media sets in terms of cost and efficiency. The alternative with the smallest

acceptable C/E value is chosen since it represents an acceptable efficiency

(effectiveness) at the lowest cost.

(2) TCA Comparison of Training Alternatives. The estimate of

effectiveness in Training Consonance Analysis (TCA) is the Training Consonance

Ratio (TCR). The TCR is similar to the efficiency ratio of TEEM, but it is

derived for training programs predicted by means other than formal analytic

methods. TCRs are derived by comparing task descriptions with training program

descriptions. The analyst encodes task descriptions and training program descriptions as input for the TCA computer program, and the computer program

I

V-12

a ..
•,•..' •2 ; •_• •_'• •• _ ' •-•.• • - • • ̧•:• •-•- . .. .... __ _____ ____ ___.. ..._

provides output in the form of TCRs and other data including diagnostic information. Training program alternatives (i.e., media sets and methods) are compared as TCRs combined with costs (C/E ratios).

When a predicted training program is being compared with an

actual (implemented) training program and the actual program has not demonstrated acceptable effectiveness, the predicted training programs may have both

a higher TCR and a lower C/E value. In this case, several approaches are open

to the analyst.. First, the predicted program could be recommended for replacement of the actual program. Second, the description of the predicted program

could be compared task by task with the description of the actual program to

form a basis for recommended revisions of the actual training program. Third,

the diagnostic information provided by the TCA program (training deficiencies,

excesses, and redundancies) could also be used to recommend revisions of the

actual training program. The second and third approaches would involve alternatives of training descriptions, recomputations of TCE's and costs, and further

comparisons of C/E values.

(3) BDWCARAF Comparison of Training Alternatives. The essence

of this method is the comparison of training program alternatives in terms of

multiple measures of training effectiveness (MOTE). Each MOTE selected is

weighted according to its perceived importance, and a total weighted effectiveness score (the standard measure of effectiveness) is derived. Costs are then

derived or estimated and C/E values compared. The method is suitable only for actual training programs when interval or ratio data are available.

(4) DIVAD Gun Comparison of Training Alternatives. This method

is appropriate when the choice in terms of cost and effectiveness is very clear

cut. tt is actually a very simple version of the DRIMS CTEA Method, but when

kJ• the choice is obviously very clear cut, it is completely adequate and avoids

unnccssary complexity.

(5) DRIMS CTEA Comparison of Training Alternatives. This method

of comparison considers factors in addition to cost and effectiveness that may

be considered important to the selection of a training program or training program element. While it appears to be most relevant to the selection of a training device (as it was applied in the DRIMS CTEA), it could be applicable to whole

training programs as well. Alternatives are ranked on all selection factors,

including one or more in addition to cost and effectiveness, and a total ranking score is derived. Selection is based on the total ranking score.

(6) Analogous Task Method Comparison of Training Alternatives.

When more than one analogous task is discovered, the effectiveness data associated with the training program of one analogous task is compared with the

effectiveness data of the training program of the others. The analogous task

training program that is most effective becomes, by inference, the training

prugram of the target task.

(7) TECEP Comparison of Training Alternatives. TECEP processes

lead directly to training alternative comparisons. Given the training objectives, TECEP selects learning algorithms and instructional delivery systems.

The next step of the TECEP Process (Figure 111-5) is estimation of the training

system cost, and the final step is selection of a cost-effective training de- livery system.

i :' ; V - 1 3

,••i • • .. . • .• • .:•, ?. .. ,•.,,;,.•• ,;••:,• • . .. ..=••,v.-i3..
b. Strategies for Comparison of Alternative Training Programs. If

some training programs have effectiveness data while others do not, the ones

that have effectiveness data are considered separately. Training programs that

have performance measures for each task are examined to select those tasks acceotable in cost and effectiveness. The decision is based on the cost effectiveness ratios showing the programs with the lowest values (lowest cost per

unit of effectiveness).

Cost effectiveness ratios are the estimated cost divided by the

estimated effectiveness. These ratios are used to compare the training programs. Analytic methods such as TEEM indicate which levels are acceptable and

which are eliminated. Final selection of a training program is deferred until

all issues have been resolved. For example, a training program with a higher

cost effectiveness ratio may be desirable because it is more effective in

training high-risk tasks.

If the effectiveness data are not performance measures or if they

are not based on standards in task and training objectives, the analyst uses

Training Consonance Ratios (TCR). TCR provide estimates for comparison of the

training programs on the basis of effectiveness and costs. Tentatively the

program with the lowest cost/effectiveness ratio is selected. The relationships among the cost/effectiveness ratios are informative. If the TCR of one

or more predicted alternatives is higher than the ratio for an existing program, the analyst compares the media and methods of the predicted alternatives

task by task with the media and methods of the existing program. Also, the

analyst examines the existing and predicted training program diagnostic information. This information indicates revisions of the existing program that will

increase its TCR and thus its estimated effectiveness. The values may warrant

adoption of one of the predicted alternative programs.

If some training programs exist and the others are estimated, and

one or more of the programs has measures based on task and training objectives,

the analyst uses the cost/effectiveness ratios differently. The training program or programs having effectiveness measures are compared to the objectives

to assess deficiencies. If standards or criteria indicate a deficiency, the

existing program is eliminated fron, selection in its unrevised state. The TCR

is computed for the program revised as recommended on the basis of the diag- nostic data, and the ratio is evaluated. The lowest cost/effectiveness ratio

is selected, with the caveat that the cost estimates for existing and predicted programs may not be comparable. Costs of predicted alternatives must include all development costs while the costs of existing programs include fewer developmental costs or none at all.

If the election is made of an existing program with excesses or

redundancies, the TCA diagnostic information is used to recomm.end revisions

and thus increase the cost/effectiveness.

If only some of the alternative training programs have effectiveness data, and the data are performance measures, the TCRs are used to compare

all alternatives. If a training program without performance measures has a

higher TCR than a program with performance measures, that program may be more

effective than the program with measures. The analyst might recommend collection of empirical data to verify the effectiveness. If the measures of per- formance include more than one measure of training effectiveness, the analyst

V-14 ,I v____________

- . . EE
can use BDM/CARAF to compute a total weighted effectiveness score and use

that score to form a cost effectiveness ratio.

When there are training program alternatives and data for all

alternatives, the first step is to eliminate from consideration any alternatives that have unacceptable effectiveness. TCRs can be used to diagnose prob- lems and recommend revisions. The acceptable programs are then compared in

cost/effectiveness. Programs with the lowest values are selected. If factors

other than performance measures are to be considered, the analyst uses the DRIVS

CTEA method.

Though the purpose of comparison is to select the most cost effective program, a CTEA may conclude with the recomaendation that final selection

be deferred until certain issues are resolved through further investigations.

Such investigations typically involve experimental determinations of the effects

on proficiency of training program alternatives.

b. Resolution of Issues

Most CTEA isues are resolved through processes from the generation of

task lists to the comparison of alternative training programs. Still, some will

not be resolved by these processes, and special methods for their resolution

must be selected or developed.

a. The typical issues requiring special methods are:

(1) Trainability. The issue of trainability requires that tasks

and training programs be compared with personnel characteristics to determine

if the personnel who are to man a system can be satisfactorily trained to man i

it through proposed or actual training programs. According to a TRADOC official

(Telephone Interview, 25 June 1980) precise methodology in this area is now

practitlly nonexistent but under development. This official endorsed the trainability analysis proposed by Litton (see Section IVB) as a satisfactory stopgap

until more precise methods are developed.

(2) Training Device Reuuirements. Several of the methods tested

above address training devices as elements of training programs, but none considers simulators or similar training devices on the basis of the several kinds

of fidelity. Because of the large number of uncertainties in the specifications

of training devices and the incomplete development of methods for resolutions

of training device questions, this issue is considerd to be suitable work for

a behavioral scientist rather than the typical CTEA analyst, and no training

device anaiyois method has been selected or recommended.

(3) Performance Versus Standards. This issue will be resolved

through the first five processes if complete task statements are available or

generated, but the development of standards is a lengthy and difficult process usually requiring threat analysis and therefore does not appear to be a suitable uncertaking for the typical CTEA analyst. No methods beyond those incluced

under Process A have been selected or included in the manual.

(4) Performance Versus Hardware. This issue is an aspect of the

trainability issue. If one or more tasks are found or estimated to be untrainable, the problem may be attributed to the hardware (faults in design, extremely

V-15
difficult tasks for personnel regardless of characteristics, eta.), but no particular method for the resolution of this issue has been selected or included

in the manual.

(5) Performance Versus Personnel Selection. This issue, too. is

an aspect of t1' trainability issue. It may ethAt one or more tasks found or

estimated to t untrainable givwn one set of personnel characteristics may indeed be traina. ;e given another set of personnel characteristics. Because of0

the undeveloped state of methodology in this area, no particular method for

considering personnel selection within a CTEA has been selected or included in

the manual.

(W) Other Possible Issues. These include such issues as user

acceptance of a training device, least-cost location of training program, and the extent of or impact of skill decay.

b. Methods that assist in the resolution of the issues are:

(1) ITV CTEA. The ITV CTEA resolved the issue of training location (e.g., which Army training center) through cost comparisons of various

locations. Then, under an equal-effectiveness-of-alternatives assumption,

alternatives are compared and a selection made on cost alone.

(2) DRIMS CTEA. The DRIMS CTEA resolved the issue of user acceptance by ranking each device and including the ranking in the overall effectiveness assessment,

(3) Trainability Analysis. Task descriptions and training programs are compared with available data on personnel characteristic. On the

basis of this informal (subjective) comparison, the analyst rates the risk that A a task cannot be adequately trained as (1) high, (2) moderate, or (3) low.

Then in the same way tasks are compared with mission analyses or operational

concepts and the criticality of each task is judged to be (1) high, (2) moderate,

or (3) low. Where high risk-high criticality; tasks coincide great concern for

trainability is indicated; where moderate risk-moderate criticality tasks coincide moderate concern is indicated; and where low risk-low criticality, high

risk-low criticality, or low risk-high criticality tasks coincide little or no

concern is indicated (Figure IV-4).

V-16

S* ; --< ,- .• ' " - =
SECTION VI

4. ~SUWtARY /J

-)" Powerful methods applicable to CTE and the whole manpower, personnel, and

training side of acquisition via the LCSMM have been developed and await broad

applicat4 .k.n and refinement. A number of these methods have-been.,escribed, in

Section III.

-1 Existing methods were weak in three areas. First, lack of adequate definition of analogous tasks degraded use of historical data from fielded weapon systems. Second, the issue of trainability (a critical question in the LCSMM) was

not adequately addressed. Third, TECEP, the most thorough cost model applicable

in the LCSMM, did not address training in Army units. Litton's methods, devised

to assist Army analysts in these three areas,tome-beewpresented, n Section IV.

Section V describes a general4model for - -

queries the data available regardless of the point in the acquisition cycle.

It i, designed to answer the questions that are pertinent at each stage in the

cycle. The model guided selection of methods for use by Army analysts and

strategies for application of ihe methods. The details concerning strategies

and procedures are presented in a companlon volume, the CTEA Performance Guide.

--- ethods that include the consolidated data base are especially interesting

since they provide a formal informational point of contact for thE many agencies

involved in an acquisition. When Army acquisitions routinely make use of a consolidated data base, both the timeliness and precision of the analyses will be

greatly enhanced. As the various agencies and personnel involved in an acquisition contributed data to and retrieved it from a consolidated data base, they

would necessarily become much more aware of the relatedness of the various acquisition processes. While data not yet produced would still not be available,

such data as had been produced would be available to all who need it. Another

possible benefit of the consolidated data base is that it could encourage all

involved in the acqusition process to speak the same technical language, thus

i asing the efficiency and precision of communications.

Powerful CTEA and acquisition process methods of the future, if they are

realized by the Army, will probably all be computer based. This feature promises

not only greatly enhanced efficiency in the processing of information but almost

certainly a degree of precision not now achievable. It is possible that such an

increase in the efficiency and precision of information processing could both

increase the effectiveness of systems and reduce the time required to acquire

them. '7

VI-1

-i.•i•

.. • .... "; . . " ,., - - : . - -" " •4
APPENDIX A

REFEMDCES

Aagard, J.A., and Braby, R. Learning guidelines and algorithms for twelve types

of training objectives, TAEG Report No. 23. Orlando, Florida: Dept. of the

Navy, Training Analysis and Evaluation Group, 1975.

Blair, A. B-1 systems aproach to train!n. final reort. Appendix S: iblio.rahy and adta ollecti on trios, SAT It Vol. 3, Wright-Patterson AFB, OH:

B-1 Systems Project ce, Data Configuration Division, July 1975.

Braby, R., Micheli, G. S., Morris, C.L., Jr., & Okraski, H.C. Staff study on

cost and training effectiveness of proposed trainin1 system (TAEG Rep. No. 1).

Orlando, FL: Dept. of the Navy, Training Analysis and Evaluation Group, June

1975.

Braby, R., Henry, J.M., Parris, W.F., Jr., & Swope, W.M. A technique for choosing cost-effective instructional delivery systems (TAEG Rep. No. 16). Orlando,

FL: Dept. of the Navy, Training Analysis and Evaluation Group, April 1975.

Carpenter-Huffman, P. MODIA: Vol. 1 Overview of a tool for elanning the use

of Air Force training r soUMs, R-1700, Project Air Force office (AF/RDQA),

Washington, DC: Hq USAF, July 1977.

Carpenter-Huffman, P. MODIA: Vol. 2, Options for course design. R-1701,

Project Air Force Office (AF/RDQA), Wasngton, DC: Hq USAF, July 1977.

Carpenter-Huffman, P., Pyles, R., & Fujisaki, M. MODIA: Vol. 3, Operation and

design of the user' interface, R-1702, Project Air Force Office (AF/RDQA),

Washington, DC: Hq USAF, July 1977.

Czuchry, A.J., Doyle, K.M., Frueh, J.T., Baran, H.A., & Dieterly, D.L. Digital

Avionics Information System (DAIS): Training Requirements Analysis Model;

Users Guide, AFHRL-TR-7'?-5b611, Wright-Patterson AFB, OH: Air Force Hman

Resources Laboratory, September 1978.

Czuchry, A.J., Glaser, J., Kistler, R., Bristol, M., Baran, H. & Dieterly, D.L.

Digital Avionics Informat(on SystemDAIS) Reliability and Maintainability

Model, AFHRL-TR-78-2,T1978.~

Department of Army, AR 1000-1, "Basic olicies for systems acquisition by the

Department of the Army", 1 April 197.

_____, AR 71-9, "Material objectives and requirements", 1979.

" "Cost , and operational effectiveness analysi.s; Improved TOW vehicle;

Vol.=V, Cost and training effectiveness analysis (CTEA)''. Director of Combat

Developments, Ft. Benning, GA: US Army Infantry School, June 1978.

tern". F"Cost and training effectiveness analysis: U.S. ROLAND missile system". Fort Monroe, VA: US rmy Training and Doct.~-ine Coirmand, July 1976.

A-i1
- , DA PAM 11-2, "Research and development cost guide for Army materiel

systems", May 1976

, DA PAM 1 i-3, "Investment cost guide for Army materiel systems,"

April I176.

DA PAM 11,.J4, "Operating and support cost guide for Army materiel

systems-•, April 1976.

, DA PAM 11-5, "Standard for presentation and documentation for life

cycle cost estimates for Army materiel systems," May 1976.

, DA PAM 11-25, "Life cVcle systems management model for Army systems," 1976.

, "Diagnostic rifle marksmanship cost and training effectiveness

analysis (DRIMS CTEA)." Directorate of Training Developments. Ft. Benning,

GA: US Army Infantry School, Augut 1978.

", "Modified preliminary cost and training effectiveness analysis (CTEA)

on the XM1 system training devices (less maintenance). Directorate of Training

Developments, Ft. Knox, KY: US Army Armor School, 1 December 1978.

________, TRADOC CIR 351-8, Individual and collective training plan for de-

--velopi1 g systems: Policy and procedures, 20 December 1W97.

, TRADOC PAM 350-30, "Interservice procedures for instructional systens development," 1978.

_ TRADOC REG 11-8, "Cost and operational effectiveness analysis in

the materiel acqU1sition process", 18 March 1977.

__ _ , TRADOC REG 350-4 (DRAFT), "The TRADOC Training Effectiveness

Analysis (TEA) System", May 1979.

, TRADOC REG 600-4, "Personnel-general: integrated personnel support

-- PS", June 1978.

TRADOC Training Effectiveness Analysis Handbook, First Draft.

US Army White Sands Missile Range, NM: US Army TRADOC Systems Analysis

Agency, 1979.

, Training Acquisition Handbook. Fort Monmouth, NJ: US Army Com- munications Research and Development Coand, 1 November 1978.

,TRASANA Tech. Rept. 12-79, "Division Air Defense (DIVAD) gun training costanalysis". White Sands Missile Range, NM: US Army TRADOC System Analysis Activity, October 1979.

" "Weapon _ systems training effectiveness analysis report: M16A1

basic rifle marksmanship". Ft. Benning, GA: US Army Infantry School,

30 August 1977.

A A-2K
4I

" "Weapo, _ systems training effectiveness analysis report: WT2A2 LAW

gunnery". Ft. Benning, GA: 30 August 1977.

Department of Defense Directive 5000. 1, "Major system acquisitions". Department

of Defense, Washington, DC: March 1980.

Department of Defense Directive 5000.2, "Major system acquisition process", Department of Defense, Washington, DC: March 1980.

!

Financial Systems Planners, Course of Instruction. Training device develornEnt

model (TDMW), Draft. Tallahassee, FL: Financial Systems Planners, Inc., 15 Apr=l 977.

Financial System Planners, Training device development management model user's

guide. Tallahassee, FL: Financial Systems Planners, Inc., 23 Sept I7M,.

Gallegos, M., MODIA: Vol. 4. The resource utilization model for instructional

course desg, R-1703, Project Air Force Office (AF/RDA), Washington, DC:

Hq USAF, July 1977.

Gay, R.M. & Nelson, G.R. Cost and efficiency in military specialty training,

Santa Monica, CA: The Rand Corporation, January 1974, P-5160.

Goclowaki, J.C., King, G.F., Ronco, P.G., and Askren, W.B. Interation and

Application of human resource technologies in wepons system design: coordination or five human resouroe technologies. AFHRL-TR-76-6. Brooks AFB, TX: US Air Force Systems Commando 1976.•

Goc~owski, J.C., King, G.F., Ronco, P.G., & Askren, W. Integration and appli- cation of human resource technologies in weapon system design: processes

for .the oordinated application of five human resources technolgies. AFURLTR-78-b (II). Brooks AFB, TX: US Air Force Systems Command, March 1978.

Goclowski, J.C., King, G.F., Ronco, P.G., & Askren, W.B. Integration and appli- cation of human resource technologies in weapon system design: consolidated

data base functional specifications. AFHRL-TR-7b-b (III). Brooks AFB, TX:

US Air Force Systems Command, May 1978.

Hanson, V.L. & Purifoy, G.R., Jr. TSM gLde to training and development and

acquisition for major systems. Technical Report TR-7H-AF, Alexandria, VA: US Army Research Institute for the Behavioral and Social Sciences, 1978.

Hawley, J.K. & Thomason, S.C. Development of an air defense cost and training

effectiveness analysis (CTEA) methodology (.for the .AN/TSQ-73):. Vol. I - CTEA

w -i~n the lire cycle- sste-m management moel. Alexandria, VA: USit AyRe- "se-arch institute for the Behavoa nd Social Sciences, in press.

,, ,Vol. II. Demonstration CTEA on the AN/TSQ-73. Alexandria, VA: USa

Research Institute for the Behavioral ndSocial Sciences, in press.

Hess, R. & Kantar, P. MODIA: Vol. 5. A user's guide to the coat model. R-1704,

Project Air Force Office (AF/RD'QA), Washington, DC: Hq USAF, Jualy 1977.

A-3
Jorgensen, C.C. & Hoffer, P.L. Prediction of trainin ro rams for use in cost

training effectiveness "analyi. Aloxandra VA: I rm esearc Institute

7or the Behavioral and Social Sciencues, 1978.

Johnson, S., Knight, J. & Sugarnan, R. Siuaintechnolog p~setrpr

(STAR), SAT-3, Wright-Patterson AFB, OR: BiSystem Project Ofice, Data

Cofgration Division, July 1975.

LaRochelle, D .Z * A cost eftectiveness analysis of an air comibat simulator,

AD-770-5T7. Wright-Patterson AFB, OH: AF Institute of Technology. June, 1973.

Menig, J. & Ranney, T. Sortin& model for B-1 aircrew training data, user's and

~grogame'3 UIde, SAT-4, Wright-Patterson 0F, OH: B-1 System Project

office, Data Co-nfiguration Division, July 1975.

Mitchell, J., Htnton, W., and Johnson, S. Behavioral obJectives for the pilot, coplo. ndoffensiVe svtmspe' NT-id, Vol. I & 2, Wright-.Patterson

APE, OH: 5-1 Systema Pr~ojetofice, Dta Configuration Division, July 1975.

Pieper, W.J., Guard, N.R., Michael, W.T., & Kordek, B.S. Trii develoere

decision aid for optimizing performance-based training in ahineascendant

1405. Valencia, PA: Applied Science Associatea, Inc., 1978.

Pieper, W.J., Elliott, T.K., Hawley, J.K. Tryout of a training developer's

decision aid for optimizing Performance-base training in machine-ascendant

mos.VaiI-ia, PA: Applied Science Associates, Inc., 1979.

PM TRADE, Training device requirements documients xde: A eroced~ures handbook

for Directorate of Trnn& Developments roct Ofces for Devices (MM.

Orlando, F: Nava Training Equipment Center,19.

Ranney, T. & Blair, A. Control/display catalog and action verb thesaurus, SAT-8,

Wright-Patterson AFB, MH: 5-1 Systems Project Office, Data Configuration

Division, July 1975.

Rhode, A.S., Skinner, B.B., Mullin, J.L., Friedman, F.L. & Franco, M.M. Mapoe

personnel and training requirements for materiel %YstuGM3 acquisition. Inrorrnatibn Spectrumn, Inc., Alexandria, VA: US Army Institute for the Behavioral

and Social Sciences, February 1980.

4J Ring, W. & Reif, H. B-1 systems approach to training, final report. Appendix A,

Cost details, SAT-i, Vol. 2, Wright-Patterson AF5, O:B-1 Systenis Pro-Ject

Office, Data Configuration Division, July 1975.

Ring, W., Gaidasz, G. & Menig, J. Training resources analytic model (TROlI).

Programmner's model, SAT-6, Wright-Patterson AFB, OH: EB-i Systems Project

OfficData on![guration Division, July 1975.

Ring, W. Gaidasz, G., Menig, J. & Stortz, W. Trbining resources analytic model

(TRAM). User's manual, SAT-5, Wright-Patterson AF5, OH: F-i Systems Project

Office, Data Configuration Division, July 1975.
f

S•Lith, S., Thompson, T.J., & Nicolini, A. Human factors and t ainirg evaluation

of the improved TOW vehicle (ITV) du ing OT/ I II. Draft Fesearn Memoranidum,

t. Benning Field Unit. Alexandria, VA: US Army Fesearch Institute for the

Behavioral and Social Sciences, May 1978.

Sugarman, H., Johnson, S., & Ring, W. B-1 systeMs approach to training, final

r , SAT-i, Vol. 1, Wright-Patterson AFB, OH: B-1 Systems Project Of'ie,

Via7Configuration Division, July 19r(5.

Vector Research, Inc. Cost and training effectiveness analysis (CTEA): Hand-, book for action officers. Leavenworth, KS: BDM Services Company, 15 January 1976.

Vector Research, Inc. Development of a cost and training effectiveness analytical modols/methodoloies for assessin trainina develovment and vroducts.

*Leavenworth, KS: BDM Services Company, 15 January 1976.

Vector Research, Inc., General model/methodology for corducting cost and trainini effectiveness analyses. Leavenworth, KS: EDVI Servi.es Company, 10 Dec 1975.

Wheaton, G.R., Fingerman, P.W., Rose, A.M., & Leonard, R.L., Jr. Evaluation

of the effectiveness of training devices: elaboration and aeplication of the

predictive model. Research Memorandum 76-16. Alexandria, VA: US Army Re-.

search Institute for the Behavioral and Social Sciences, July, 1976.

H

I _Mai

A-5V

!g '[
APPENDIX B

ABSTRACTS OF SELECTED COST OF TRAINING STUDIES

Several documents pertaining to cost analysis of training were reviewed,

The documents most relevant to CTEA in the LCSrei were abstracted and the abstracts are presented in this Appendix.

B

-- ,,-i "l
THE ARMY LIFE CYCLE COST MATRIX

(Department of the Army Pamphlets: No. 11-2, "Research and development cost

guide 

Materiel 

for 

Systems," 

Army materiel 

April Systems," 1976; No. May 11 -4, 

1976; 

"Operating 

"Investment 

and 

Cost 

Support 

Guide 

Cost 

for 

Guide Army for

1

Army Materiel Systems," April 1976; No. 11-5, "Standards for Presentation and

Documentation of Life Cycle Costs Estimates for Army Materiel Systems," May

1976. Washington, DC: Hq, Department of the Army)

In order to encompass all the costs incurred in the life cycle of a weapon

-system (or major component of the system), the Department of the Army has issued

a number of cost guides (DA Paros. 11-2, 11-3, 11-4, and 11-5) for life cycle

costing of materiel systems. The cost elements are as shown in Figure B-i, Army

Life Cycle Cost Matrix. Although a cost attributable to the requirements for

training may be costed in another category, this cost matrix treats training N costs as follows:

Category Budget

1 .0 Research and Development 1 I.0b Training RDTE, OMA

S2.0 Investment

2.08 Training PROC., OMA 3.0 Operating and Support Cost

3.023 Unit Training, Ammno & Missiles PROC I3.01 Personnel Replacements iA, OMA (depending i • on focus, assumptions,

etc.)

These are further defined by equations (Figs., B-2, B-3, B-4, B-5, B-6).

Supporting these equations (a gross, aggregate level) are detailed docuxnentations of each cost element, justifying the dolLar figure proposed (for format

see Fig. B-7). Although all life cycle costs (LCC) are covered by this matrix,

problems remain. A graphic display of the issues is shown in Fig. B-8. Another

issue, not shown in the table, is the inability of the analyst specializing in

one area to isolate and aggregate all the cost relevant to his particular focus.

Training is an outstanding example of such an area. For the purpose of forecasting for complete LCC of training for a given system the analyst must have

some cost data, such as ammunition and POL costs, that the weapon syMtem LCC

1⁄4+•i includes as separate categories. The training system cost analyst usu-aly takes

the 'bottoms up' costing approach shown in Fig. B-8. This is likely the case

when conducting a CTEA for a developing system even though the cost information

will be used as input to an analysis conducted by a system cost analyst as required at all major decision points in the acquisition process (usually a formal

COEA). This problem was recognized by Hawley and Thomason (1978), developers of

an Air Defense CTEA methodology especially fitted within the Army's LCSM. and

TSM concepts. Making recommendations for future research, they recognize requirements for:

B.-2
.1 t 11

IT - 1 1 1 1111 11 1 I'l I I111 111

H.IJ

B-36
UA a.

oto

- ww dc

0A )w Q L

-. jMI ~ L <.N ~ . J ;JX con (2

owci

1~5 J

(C II

B-44
04i I

jA 'o

2a. 

0 I. 

H

cc a.

I.-4

0 

0 

I

B-5
CONSUMPTION, 302

UNIT TRAINING, AMMUNITION AND MiSSILES, 3,023

I III

UNIT TNG TOTAL QUANTITY ANNUAL ( NUMBIR N.

AMMOIM1L - ( OPERATIONAL jX (AMMOIMSL COST) X OPERATING

3.023 EQUIPMENT PER EQUIP YEARS

Source: DA Pam 11--4. P. 6-3

Figure B-5. Operating Phase, Unit

Pimunition Costs

INDIRECT SUPPORT OPERATIONS, 3.06

PERSONNEL REPLACEMENT, 3.061

PERSONNEL TOTAL QUANTITY f TOTAL NUMBER

REPLACEMENT OPERATIONAL XI MILITARY PERS X

3.061 EQUIPMENT PER EQUIPMENT

AVERAGE AVEAG

ANNUAL AVRG NUMBER SATTRITION RATE REPLACEMENTIANNUAIN COST/MAN YEARS

Source: DA Pam 11--4. p. 6-7

Figure B-6. Operating Phase, Personnel Replacement Costs

B-b

• ,-,:... " -• . .,,.., . .. .....•.<, ;. • 7 .. , . I: . -..~.i-. . . - -. ,-- , •.-;'. * • . '
II 

I

I

I I

I

4

J

I a0 -U

14 0 u A. h41 a

a, - _ Id-i - 0 U 3 0

"4

'UI

4:1 'U

24

U)

--

-4

'-4

U)

- 'U

'a! ..

SM 

-

Iu1-

= 3. 

0

I 4'

-- z (U SM I 4'

1 LiK .4. ;,.q,

U ''I .

1--- I j SM. 24

A.

C3.4

I SM 4 

0 I SM . A

B-?
1I

J to

'au

I-I

o Q 4

i~ II-4Iii i

0.1-0 0. w 0.- U0 L6I

L_ __ -- - -
1. More definitive work on cost analysis and resource impact projection.

A major criticisn of the costing model (e.g., TECEP) used in the current dmonstration project is that the input cost categories do not match the output categories of the ArMy's accounting system. Future work should involve a resolution

of this problem. The oonnurrent and predictive validity of ousting procedures also has not been established.

2. More work should be done on establishing the organizational links

necessary to obtain information concerning RAM, maintenance cost data and maintenrice policy information for complex training devices. Equipsent RAM and

maintenance considerations often underlie assumptions comerning the operating characteristios of the materiel system and the training program. Therefore,

the correct estimation of such parameters is essential for valid cost estimation and later sensitivity analyses.

3. The investigation of cost analysis methods providing expected values

and confidence limits as output. In CTEA, it would be useful to obtain Mit only a point estimate of cost, but also a statement of the precision or reliability of the cost point estimate.

B-9
TRADOC TEA HANDBOOK

(Dept. of Army; '"RADOC Training Effectiveness Analysis Handbook, First Draft"

White Sand- Missile Range, NM: US Army TRADOC System Analysis Activity, undated. Received in 1979)

TRADOC includes CTFA as one type of a family of training effectiveness

analysia (TEA). In the recent TRADOC Training Effectiveness Analysis Handbook,

First Draft (1979) a CTEA is said to "evaluate the cost and effectiveness of

alternative training approaches as they are being forimlated to support the

developing total hardware oriented system" (p 1-7). The study further specifically defines a CTEA as "a study that assesses the variable effectiveness

and variable costs associated with a set of pre-defined alternative training

subsystems at each major decision point in the acquisition process." (p 2-2).

This definition forms the basis of the relation of costing to the CTEA

methodology. As shown in Fig. B-9, costing is specifically addressed at two

points. The first occasion is near the end of the preliminary phase after the 1

concept design of alternative training systems. Estimated concurrently with

training subsystems' effectiveness, the cost of each alternative system is input

to the decision process that chooses the best alternatives, The second costing effort takes place between OT I and OT II. At this time, a smaller set of 'best

alternatives' are selected and alternative training subsystems designed.

This set is costed and their effectiveness assessed and a still more

select set chosen and tested in OT II. Costing effo.-t, however, is implied --

either new or revised estimates -- throughout the system and/or training subsystem life. The TRADOC Hanobook suggests using the var able cost, variable

effectiveness model as the most appropriate for comparing alternative subsystems.

They state that the second cost analysis will be more detailed since both training and hardware subsystems cre ap)roaching final form making more precise estimates possible.

The cost analysis procesj is treated in a separate chapter, Chapter 10,

Cost Analysis.

The TRADOC-suggested ccst methodology i.-. shown in Fig. B-1O. It is applicable to both costing 'points'. Cost considcration and Iata collection are asst•med to begin at the front-end analysi2.

The generalized cost model is

TTC ITC + UTC

or

Total training cost institutLona.1 training cost plus unit training cost

The cost analyst is directed to wo-k with the rest of the study team to

define the essential elements of analysis and to identify cost subelements that

they require (e.g., awmio, cou.'ses, MOS to be trained). Once the TTC of each

alternative is calculated, the cost analyst is instructed to highlight in his

B-1O
analytical LMsentation subelements of high impact on total cost and those whose

incremental or decremental change will influence system effectiveness. Critical

cost considerations may outweigh total costs of the alternatives and may be the

deciding facors when TTC of the alternatives do not differ significantly. Of

further importance, this cost methodology implies that within the basic model,

subordinate equations will be written 4ased on the institutiorol training concepts, the unit training concepts, the training s•stem requiramnta specification and the life cycle cost estimates. LOC elements (LCC=) are those discussed

earlier. TRADOC identifies the LCM as the primary building block in the cost

analysis and the 'only tool' the analyst has to define the hardware oosts of the

training systems. Suggested sources (Army Aguncies) are also listed in Annex I.

Although the list of data for institutional training does not explicitly

state that field as well as base (school) training are included, field training

and its costs are not precluAded and they could be included in such categories

as equipment used, etc.

Unit training is examined by 'training event' (e.g., ARTEP event.' and the

cost of each event over the life of the system accumulated. This should amount

to a close approximation of Litton's model for LCC cost. No provision is explicitly stated for prorating costs or evaluating remaining assets at end of

costing period.

The TRADOC system rests on identifying critical tasks. The task list,

procured from the study proponent and/or contractor or devised by the CTEA

analysts, is established in the front-end analysis (Step 1 .0, Fig. B-9). This

list is used to design the first set of alternative training subsystems but

explicit directions for developing a task list are not specified.

B-11

S.... . u ...... ... .. . - •- , ,: • ., • _ '•, 

• '. , • , ' , . .. , . " /-•: " -

•:• :.., ,..• .:--.:.•:.: • . ,;•i• . . ...;• i -.• ,, ..•, ,l• : , ... ;,_ ..... • '- . .. .• .. .'

S....... • :" •,• •• -"•?• , • .. , • !••' •.•..:•,,,•.• -.-•u.:-, : ..•i .•: • ,,,•.
I~~ ~~ hm IM

Figur.B9 ""ItEAFowhr SA sota*1o

ottev 2

23 C-OST is
OR 3.0 TABLE I0.2

DEIINE TWIH ALTERNATIVES

DEFINE THU EISOUCEI IIMN •OF EACH ALTERNATIVE

0EPINI AISUMPTIONS18

0 ,AND OROUNO RULES

DECIDE ON ,

COST MEASURESH

.... _•__ SRIE NON-QUANTIFIABLE

ESTALISH COST C IF APPLICABLE

DATA SOURCES ...

AND

OBTAIN COST DATA I

_____

SYNTHESIZE DATA 1b COMPARE ALTERNATIVES

TO USABLE FORM "

I I 1 ISENSITIVITY ANALYSIS I

DETERMINE COST -____

OF EACH ALTERNATIVE K

DISPLAY RESULTS

INPUT TO BLOCK

2.3 OR 3,3 t TABLE 10,2

Sstaie: TRADOC TEA Handbook. 0. 10-11

Figure B-10. Cost Methodology

'II

B-13

- _ tZ 
!ii

SYSTEMS APPROACH TO TRAINING (SAT)/P-1 (1975)

(Ring, W. & Heif, HI. s-I sytems approach to trainina. final report. Apoendix

A: Cost details, SAT-1, Vol. 2, Wright-Patterson AFB, OH: B-1 Systems Project I

Office, Data Coffiguration Division, July 1975)

In 1975 Ring et al. applied the techniques of systems analysis to Instructional System Development (ISD) to ensure that the entire training system is

included in Training System Design (TSD). The SAT was used to Gevelop a recomumended alternative training system for the future B-1 bomber. A computerized

aid to the ISD was used -- the Traini•,6 Resources Analytic Model (TRAM). TRAM

provides time-phased LC costs and resource requirements for the training system

of the B-1. The SAT approach provides for trade-offs and sensitivity analyses

of such features as ratio of hours spent using various training media, centralization of training facilities, ratio of airborne to ground-based training hours,

and training media time phasing. Outputs of the program are descriptions of the

recommended and alternative training systems with a syllabus for each coursu, descriptions of required media and facilities, costs, and schedules. )

The initial source of information is considered to be the task analysis data base. The authors state that the quality of this data base establishes

the potential quality of the overall instructional system.

The SAT process was applied to the aircrew only. It is not known if maintenanca, ground crew or other training, also part of the overall B-i LCC were

similarly analyzed. The task analysis identified 1500 task elements.

The recommended system school setting augmented with training devices and

the alternative training systems were all costed. These costed systems were

identified by both TRAM and a quick-look version, TROLIE (Training Resources

Organized for Logical Integration of Expenses) model runs and subsequent data

development.

The cost moael was developed to fit a particular segnent of the B-i related personnel - the four airarew stations, and three particular training ,1

alternatives - three types of special trainers for each crew station. The

data identified as required to cost each of these special trainers by crew

station were:

Initial Acquisition Costs

PDT&E and other preproduction costs (includes cost of first unit).

These costs were all estimated through application of factors that

were percentages of the hardware acquisition cost. For example,

initial training = 1.5% of hardware acquisition cost (excluding familiarization trainer).

First unit production costs for trainers. These costs were first determined to lie within ranges and then estimated as a point within

the range. For example, the procedures for pi~ot/copilot station

are estimated to fall in a 0.10 - 0.25 ($ x 10 ) range and estimated to be 0.20 ($ x 10 ).

B-14
General purpose carrels and briefing rooms. These costs were determined to be a function of required multimedia A/V equipment and cost

of classroom new construction or conversion. Cost factors such as

$40/sq. ft. for new construction were used.

Facility requirements and costs. A similar costing approach to that

of the carrels and briefing rooms was used.

Instructional material. Representative costs for instructional

material preparation and production such as viewgraphs at $10 each;

programmed courses, self-paced, at $600 per instruction hour; and

technical data manuals at $125 per page were used.

Annual Operations and Maintenance Costs

Instructor personnel

Trainee TDY

B-I Aircraft (direct costs)

Equipment operations and maintenance

Equipment Upgrade

Instructional material

Facilities

All of these costs were estimated using factors for costs and aver-age lead

times.

A further feature of this costing effort was t'e assigment of a data quality designator to each cost element. Costs used were related on a scale of 1-3

depending on the quality (predicted accuracy) of the data source. (See Table B-I

for a list of all elements and quality estimates.) Among the cost data sources

were manufacturers; goverrmient facilities and simulation users; the Simulation

Technology Assessment Report (STAR); the Simulator Special Project Office; commercial handbooks and annuals of A/V and other media and construction costs; and

USAF Cost and Planning Factors (ArFR 173-10).

A baseline system, three modifications of a part mission trainer (a simulator) alternate baseline concepts, variations in crew ratio and variations in

the replacement model were costed.

The foregoing methodology and basic data were used to derive costs used

to evaluate, comparatively, the three training systems. Output was formulated

to display:

Non-Recurring Costs

RD&E

Acquisition Facilities

Facilities

SInstructional Material

B-15

4 ii
Recurring Support

Operation and MaintenanceI

for 

Costs were displayed as totals by year as appropriate in current dollar's the first year and inflJated dollars for succeeding years. Life cycle costs for ten years of operation were costed.

B-lb
11

Table 3-1. Data Quality Estimates

1GIOWNWACl•Irt~ VIV _- MTING.. " 

INITIAL ACQUISITION COST -

"OTIF AND OTHER PRIPWO)UCTION COST

AGE. DATA. OTHER 3

INITIAL TRAINING3

INITIAL PARIS 3

INTEGRATION AND INTERFACE 3

INSTALLATION AND CHECKOUT 3 PRP10IT 2m•

LEANNING RATE 2,

kPLIGIT STATION TRAINERS

FAMILIARIZATION 2

PROCEDURES 2

PART MISSION 2

010 TRAINERS

PROCEDURES 2 "PART MISSION 2

0SO TRAINERS

PROCEDURES 3

PART MISSION 3

ANCILLARY EQUIPMENT

MOTION BASE I

0 SEAT I

VISUAL DISPLAYS 2

IR 3

OTHER EQUIPMENT

• •*•SUBSYSTEM PROCEDURES TRAINER TRAINER 2I !

COMPUTER-AIDED INSTRUCTION (CAI) 3

GENERAL PURPOSE CARRELS AND CLASSROOMS

GENERAL PURPOSE CARRELS 2

CLASSROOMS 2

FACILITY REQUIREMENTS AND COSTS

REQUIREMENTS ISIZE) 2

CO0S'T 2

INSTRUCTIONA MATERIAL PREPARATION AND

PRODUCTION

LECTURE 3

VISUAL STILLS I

FILMS 2

AUDIO TAPE 2

AUDIO VISUAL PRESENTATIONS 2 A

TECHNICAL DATA MANUAL 3

CAI PROGRAM COURSES 3

ANNUAL OPERATIONS AND MAINTENANCE COSTS

INSTRUCTOR PERSONNEL I

TOY TRAINEEC I

S-1 AIRCRAFT 2

TRAINER EQUIPMENT 3

EQUIPMENT UPGRADE 3

F-A INSTRUCTIONAL MATERIAL

MANUALS AND TEXTBOOKS 2 SUPGR.*DE 3

FACILITY MAINTENANCE 2

B-17

Ii
A TECHNIQUE FOR CIOOSING COST-EFFECTIVE INSTRUCTIONAL

DELIVERY SYSTEMS, TAEG REPORT NO. 16 (1975)

(Braby, R., Henry, J.M., Parris, W.F., Jr. & Swope, W.M. A Technique for Choosing

Cost-Effective Instructional Delivery Systems, TAM Rep. No. 16). Orlando, FL.

Dept. ol the Navy, Training Analysis and Evaluation Group, April 1975)

One of a series of reports prepared for the Navy's Training Analysis and

Evaluation Group in Orlando, FL, TAEG Report No. 16 presents a technique for

choosing cost-effective delivery systems for proposed training programs. It is

not designed as a CTEA methodology although it could be used for conducting one.

The researchers (Braby et al., 1975) call the technique TECEP - training effectiveness, cost effectiveness prediction. It is intended for use by the training

system designer during the concept design phase. The methodology includes refinements of two earlier efforts - one with the same objective (TAEG Report No.

1, 1972) and a technique for choosing cost effective instructional media (1974).

TAEG Report No. 16 is designed to be used with two other reports - TAEG Report

No. 23, Learning Guidelines and Algorithms for Twelve Types of Training Objectives

and TAEG Report No. 24, Choosing Instructional Delivery Systems with the TECEP Technique.•

The report's system development model is earlier shown in Section III, Figure

111-5. It requires user expertise (unlike the BDM model designed for CTEA). The

position of the TECEP in the TSDM (a continuous, reflexive model) is as shown. A

diagram of the TECEP (an input-process-output) model is shown in Figure III-4.

As in other methodologies discussed, the costs are estimated after the

instructional systems have been designated. The authors stress the importance

of defining study objectives. For example, if the objective is to select the

most cost effective alternative from among several choices, the resources common

to all in similar amounts can be factored out. When absolute, life-cycle cost

is required, then all resources must be included and evaluated at their opportunity cost.

The basic output of the TECEP's cost module is the current cost of each

alternative proposed training system. Additional output was (at the time of

the study - may now be increased) formatted to show the total and average

yearly cost per student position, average cost per course graduate, and a

distribution of incidence of cost over the alternative's life.

The designers recognize that it will be necessary to access multiple data

sources to apply the cost model. They suggest:

1. past records of operational units

2. NAVPERS personnel data

Model limitations are seen to be: The model cannoa select the most efficient media; cannot forecast total system cost; assumes all variable cost functions are linear; does not provide for evaluating secondary effects.

B-18
The inputs are divided into seven categories:' ftacilities, equipment, inI

structional material, personnel, supplies, students, and misoellaneous (36 items in all). This model is computerized, written in FORTRAN IV. The entire output

or selected items may be chosen. A maximum 20-year life may be calculated.

I !I.

B1

ii

I ,I

B-i9 I._ ___ _
TRAINING DEVELOPERS DECISION AID (TDDA) FOR OPTIMIZING PERFORMANCE-BASED TRAINING IN MACHINE ASCENDANT MOS (1978)

(Pieper, W.J., Guard, N.R., Michael, W.T. & Kordek, R.S. Training Developrs

Decision Aid for Optimizing Performance-Based Training in Machine Ascendant MOS.

Valencia, PA: Applied Science Associated, Inc., 1976)

The TDDA is a process for generating training specificntions for developers

who must decide what, where and how tasks within an MOS must be trained. It was

developed in two phases - a manual process model based on logic charts and a

computerized model for which the developer inputs task data. Three outputs are

available - training prescriptions, training hierarchies and sequences, and

training costs. A single assumption, the availability of a usable task list is

made.

Cost analysis is the last module. The authors' (Pieper et al., 1978)

rationale is that this facilitates identification of beneficial cost tradeoffs and permits selection of only the necessary and relevant cost statistics.

Training material, course POI, criterion-referenced tests and OJT plans are

all considered.

This cost model differs from those previously discussed. Training methods

are assigned values derived through a cost rating technique. Pieper et al.

chose this method because: (1) actual dollar costs vary over time while relacive difference in costs remain essentially constant; (2) ratings can include

items such as student time without conversion to dollar amounts; and, (3) the

various methods of instruction can be ranked on relevant cost dimensions.

The tasks of any one MOS were treated as a single set -- some taught in

schools, some on-the-job, and a few given no formal training. In OJT two.. factors, number of hours in the learning cycle and number of hours to learn a

task, are their determinants. The two-year promotion cycle is set -ýs the learning cycle, with each year treated uniquely and time for training calculated

using factors for such items as non-productive duty and movement found in the

Army Regulations. This process was recommended for OJT calculation for each

MOS.

Training in schools was treated differently. Here Pieper et al. looked at

the direct and indirect costs of establishing and operating a resident training facility. They categorized training into seven method cost classes:

Class 1. Conventional classroom, demonstration, case study, and

guided discussion.

Class 2. Peer tutor.

Class 3. Tutoring.

Class 4. Progranmed instruction (student and program paced),

games, in-basket, and study assignment book.

B-20
Class 5. Traditional practical exercises,

Class 6. Programued practicol exercise.

Class 7. Computer assisted instruction.

Next, they placed the direct and indirect cost variables into seven classels.

variables used were:

1. Square footage

2. Instructor to student ratio

3. Systemn equipment

4. Furnishings

5. Expendable supplies

6. Training aid development

7. Training material development

Third4 , the method cost classes were ranked on each of' the cost variables.

An illustration, Table B-2, shows the method cost classes ranked on one cost vari- :1 able, value or square footage required for that instructional method. (This is

value or benefit not dollar cost.)

Method Cost Class t

2 3.5

3 3.5

4 3.5

5 b6.5

6 b6.5

7 3.5

Table B-2

As explained by the authors, "the cost variables were assigned ranks on

the banis of their estimated relative values on a scale of one to seven. In

the example (Table B-2), the square footage variable for traditional classroom

training was assigned a value of one, because this method can accoumodate more students per square foot that any of the other methods. The square footage

variable for peer tutor, tutor, programmed instruction, and computer assisted

instruction training method classes were each assigned a value of 3.5, because

there was no detectable difference in the amount of room needed for their use.

These four tied rankings consumed 2, 3, 4, and 5 on the one to seven scale.

The square footage variable for traditional practical exercise, and programmed

practical exercise training methods were assigned a value of 6.5, because therer

B-21 Ir
was no detectable difference in the amount of room needed to use them. These

two tied rankings consumed positions six and seven on the scale, and completed

the rankings for the square foot variable."

All the cost variables were assigned rank values using this procedure. The.

seven method cost classes and seven cost variables were then arreaged in a table

(Table B-3).

Using these seven rankings a mean rank for each method cost class can be

computed. The remaining calculations were:

"After the mean ranks were computed, the conditions for determining

the relation of the cost intervals to the rank intervals was set.

The lowest cost method was assigned a value of one, and the highest

cost method a value of two. Since the number of intervals within a

range of whole numbers is equal to the highest number in the range

minus the lowest number in the range, the cost interval for Table B-2

is 2-1=1, and the rank interval of the seven variable scale is 7-1=6.

The relationship of the one cost interval to the six rank intervals

is the ratio of 1/6 which is .167. This ratio was used as a constant

to compute the cos'l multiplier for the mean ranks of the method cost

classes.

The mean rank cost multiplier 'y' in Table B-2 was derived by subtracting one from the mean rank, multiplying the difference by the

cost ratio and adding one, (e.g., y = (rank - 1 Wx)) + 1). The

number of hours of each method within the training cost option was

multiplied by its cost factor to get the Method Cost Indicator (MCI)

for the methods. The Mf;¶ for all methods within the training option

were then summed yielding a Resident Option Cost Indicator (RMCI =

ROCI) ."

With the exception " .I, the cost method classes and cost variables were

also applied to OJT and az. aalogous on-the-job option cost indicator, the JOCI

was computed.

The two were then summed to obtain a cost indicator for each training opt-on (TOCI).

TOCI ROC. JOCI

or

Training option cost indicator Resident cost

option indicator + job option cost indicator

The TOCI is used to establish a ratio of one training cost option to another,'

giving the training jI manager as indication of relative cost. This enables the

manager to select the most promising option or options for a complete CTEA effort.

The procedures have been developed in both computer and manual mode.

B-22
Table 3-3. Cost Ranking

0 lWIT/ IVI PURN EXPIN AID AVIRMI

MIThO GOT CLAN PT STU Ig DIV MAT RANK I

1. TRADITIONAL

CLASSROOM 1,0 3.0 1.6 3.0 3.5 2.0 3.0 2.4 1.2

2. PIER TUTOR 3.5 4.6 3.5 3.0 3.5 2.0 11. 3.1 1.33

3. TUTOR 31, 70. 3.5 3.0 3.5 3.0 1.5 3.4 1.40

4.r PROGRAMMED

INSTRUCTION 3.5 15. 1.s 5.0 3.5 56. 6.0 3.6 1.42

S. TRADITIONAL PRACTICAL

EXERCISE 6.1 6.0 5.5 1.0 3.5 4.0 3.5 4.3 153

6. PROGRAMMED

PRACTICAL (XERCISE E.5 4.6 5.5 6.0 3.5 6.5 6.0 5.4 1.72

7. COMPUTER

•ASIlSTED

INSTRUCTION 3,5 1 .S 7.0 7.0 7.0 7.0 7.0 5.7 1.74

I L

Soure: Pieer at al.. p. l--46

3-23
PREDICTION OF TRANING PROGPAR FOR USE IN CTEA (1978)

(Jorgensen, C.C. & Hoffer, P.L. Prediction of Trainina Programs for Use in Cost

STrainins 19fct.ivos3 A sl nriap VA: U.S. Army Research Institute

for the Behavioral and Social Scienoes, 1978)

This. report describes a model developed by Jorgensen and Hoffer (1978) that

uses a systems approach to define training elements and their interactions within

a CTEA setting. Following a set of logic flowcharts, the steps in CTZA oond'iot

are depicted. Initial input is task and analysis data (ideally but not neoessarily in ISD form). The model has been computerized. The model output is a

cost-effectivene*4% ratio, obtained by dividing the estimated dollar coat of a

programm-enerated training program by the program's estimated effectiveness ("an

efficiency metric is 'ised to estimate training effectiveness. The efficiency

metric is an indication of the overall best fit between the training requirements of the task and the ability of various media and methods to satisfy these

demands.")

The system overview was shown earlier in Figure III-1. As mentioned previously, the methodology begins with a task list or description that states the

tasks and subtasks. After the tasks have been described, gr.ouped and appropriate

training methods and media selected (all by analyst/computer interaction) the

selected alternative training programs are (or single program is) costed.

A unique cost module was not developed for this methodology. The TAEG 16

cost model developed for the Navy (Braby, Henry, et al., 1975) was used to cost

the media and methods. As discussed earlier this model costs equipment and

methods in terms of 37 variables (input to this computerized model). This cost

model was, at the time of Jorgensen and Hoffer's work,in active use to cost

training programs at the Army Air Defense School. This model is Seared to institutional training (although adaptations might be possible) and does consider

depreciation and LCC.

B-24
I .1

BDM's GENERAL MODEL/METHODOLOGY FOR CONDUCTING COST AND TRAINING

EFFECTIVENESS ANALYSES (1976)

Vector Research, Inc., Cost and Training Effectiveness Analyais (CTEA): Hand ook

for Action Officers. Leavenworth, KS: BDM Services Company, 15 January 1976.>

Vector Research, Inc., Development of a Coat and Training Effectiveness Analytical iMeodels/Methodolis- for Assssaina Training Development & Products. Leavenworth, KS: EDM Services Company, 15 January 197b.

Vector Research, Inc., General Model/Methodology for Conducting Cost and Trainin6

Effectiveness Analyses. Leavenworth, KS: BDM services Company, 10 December 1975.

In 1976 Vector Research, Inc., working as a subcontractor to 5DM Services

Co. prepared a CTEA model/methodology for TRADOC. 5DM saw the training system

as a nznber of subsystems, each to be assessed and costed; and an evaluation

made of the aggregate total program (Fig. B-11).

As shown in Fig. B-1i, it was based on a list of critical, tasks and sug- gested that the CTEA study team consider individual and collective training in

both unit and institution and by several methods. However, when 5DM examin'.id

training for a number of Army systems they found a minimum of eight unique systems. These they identified as:

Large Group War Games Simulator

Individualized and/or Small Group Lesson Delivery System

Small Group Tactical Maneuver and Deployment Instructional Game

Program Directed Hands-On Job Performance Aid

Large Weapon System Practice Firing Adapter

Small Group Combat Engagement. Simulator

Trouble Shooting Training Simulator

Small Weapon System Practice Firing Adapter

They then proceeded to integrate these eight types into their general model.

The general model is shown in Fig. B-12. The authors of the methodology also

"prepared a CTEA handbook for action officers not formally trained for or experienced in performing CTEA. In spite of comprehensive overview shown in Fig. B-i 1, the guidebook offers no assistance to the action officer for the front-end analy- sis. He/she is directed to prepare a simple written desc,-iption of the system being discussed, its purpose and general requirements. Then the guidebook directs

the officer to the chpater discussing the system (of the eight) most like the one

he/she has identified.

Although the model (Fig. B-12) shows a concurrent costing and effectiveness

measurement effort, the description of detailed steps for each type system

B-25
II

Ir

1wU LU 1

8~~ 2___<_

(AA

00

CL.

I- I

a.U.~

iL i 4c-I

(AL 'e.

-9 __ ___ ___ ___B__26a
REQUEST CTEA

A' ouucmis SYSTEM

"STATE PURPON OP SYSTEM

ATE OPERATONALIC•ENARIO

0DECRIIE SPECIAL REOUIREMENTS

rsELECT COST ELE3MEN7TS SESLECT MOTES

COLLECT COST DATA COLLECT TRAINING EFFECTIVENESS DATA

DE1TERMINE COST PER UNIT MEASURE ASSIGN WEIGHTS

DEEMNE INDIVIDUAL SYSTEM'S EFFECTIVENESS

VI I

MAKE RECOMMENDATION

Source: 1II _OM, 

Figure B-12. CTEA Flow Model

B-27

i . . ...

i oT . o,..,o . oo., I 0"'' TT" '"''O' ":OT "'• " OA " i
- r •

directs the analyst to address cost analysis after the effectiveness of alternative systems has been determined. In each -ase the cost analysis method is

the sam. The analyst selects the sample system most similar to the one(s) to

be evaluated and then proceeds. Four forms and a checklist have beow, d-.gnaed

to ensure that all necessary steps in the CTEA have been completed.

In a method analogous to selection of a measure of training effectiveness

(MOTE), the analyst sets a standard measurement of. cost. (This always follows

the determination of weighted MOTEs for tte base case and all alternatives.)

Table B-4 displays examples of standard mesurement of cost and relevant cost

elzaents for each of the eight training system types. When the analyst has

defined the standard measure of cost, he/she enters this at the bottom of CTEA

Form #2 and enters identification information at top of CTEA Form #3. Next,

the appropriate cost elements are selected and their LCC cost estimates (also

entered on Form #3). Suggested sourms for these cost data are the Comptroller of the Army, the proponent school, or CTEA action officer estimation. Costs

are summed for each training system, and a per unit cost (based on 4xpected LC

number of units of system used) calculated. These per unit costs are then

multiplied by the effectiveness ratio and compared to the baseline cost and

each other in a cost-benefit analysis. This completes the CTEA.

B-26

.6n
&(7nA00

W- M

b., 44w 00 0 "40 4

to~~ "a 4 0 ~ W~u 0- 

(A C l AW0 1 6 00 4j ~4) 0.9 4.

> tot

W. CA 04 Ut8 4 4 t

0 >0 LU 'Sw CJ E-~to- w 4L )q w w i L e O 04-4 0~ 0 aefpefa~*~ Ee~ -

Q 4 t- 16000 440 0

cc 41 w 41w -1 44 CI41 ý

41AiC

-L v(

0)- 00 0Oc4d I0

$4E ýd LU t

S .0 Cf1 Lo 1 4 -

r. m wMV4

u0 " -4rw a

E.' Q 4 U U .0 0 -H fU

U00 41 fa LUa ý6 LU 3 'aJ~ ~ LU0I ~ (A -~ 4 c d U W J 4 r

en 4J t 6 p-

$4 A uu A )=-4W (

U4 000 pC 1A

.0 0 W 60 C

06

r4

B-29
42~~ 42'a-1 ~ U .~

444

((A

al 424J.. A

U 442(Ur-U) ~i 40

4.4 1 ~ " (42

0 rA m

40 41 0 C 4 0 0J .4 4 4 1 0 4.1 44~ m 4 o 8 m 0 t2Uj~. m

001 44 $4 j

U *0 0 i (A44

0U 44

(4.4 0 4

(IV 4110

C ' 4 Cr4-L4

41 0U 

-$ 0 

411

44(U

041 go j 414

41 M * VW

P-43
"4

0 (A

'3L 0 0) 4'I0

0 0- *'4" E-4 m 1. " "4 a 4

P4 41 Q 4 41 Q 4

0) :01% 41

'3 0 00 00 0

~~ 41 4141 to 1 14~0

""4 44 H 00 a"A 000

00d

4-4 03" 41 4 440 M

04I0

t 0 41 cd IA . E-4 C

10 w0 C W. 1-4 )P 1!4

'3~~' A "44 4 CO

V- 0- 9 0 0

0d 0 4W) ,004 .61

I0-4a 4 ) 

41~~ 0 w W"0. (Aj4pl< 

"-4 P4

~~o PC 410'~~o

"41 C:zH

E-4
0 cm C- m 4 .1 GJ

wI 0 to m. rn

40 co 0 Lj 4)

to1 41 P4H 4 c

u. 41u44a 4

(A P-4 Q j 411 Q.Q". 4 0 $

44 31-Pu 41

41 m.~4.Q

w '4 41 0 

I-0 001C0I4

01 0 01 4 . 4 4 0)

-4 P4 U

A 0

(A4-d, C 93 4jIi

0 01o

tw PC~ 1 ý4 mI JJ0

-4 4-1-4

02 4,4 VJ m 0 J .1 ' -A 

ý4 > 1 kv tr j

U3I

tnO

C,.,

B-32
C I

C6 >

0 sw U -e. to~ 41~. 4) to V j 'A IA 4 1 '0 4 ~

:0-1a 8 toaa 0 m 0 c o

Q -4a U 4 a~~4 ,- a "4 0 a .4

W4 00 a- 

"".- tol ItE a,0

4

000

u 4n

0 >0

co fa

a 0 to

0 .00P 0 4) to m 0 .41 u cc 0) 4

0 0UI >01 4) 4 C4 (

U) )4 A) P~3 UU (D

- -1

.6 01 s0 00U

21 0 0a

010

W -C

cn

0

U PI

ow

B-3
- :1

Ai 41

4.'

4.; u 4 hi4

M- o- 01ý 

0 ( A Do

Q Q0 'co

u 8 ,C '.4 

ca 'a 0

a. 4 -4 0 Q 0

~*) 1- CoU I) ~ U L

44 4

F-4 u

CdC

U) 1-4~- *y00j 4-A

0B 34
00 a1

'a em

A1 Waa r

1-d cc 0 1. 

0- E- A0 1- 00 (d u

ji 1 1. C: 414 44 0)

) - ON) 0.. 4) 4 A 0

u l. >a 0~ 03 0 W 0(Ac 0 00.94.4

4) 4) w 2 e%$4> 

400 

"41

to M )4

ca 0 W -wW3W

. a0 "4 - Wa Idc r 41- 0)U 1a :1 "4. a .

o~t 44 0~0.U'

0n 4., 0) 4 4a P4r.$ U a r4 '54U '~A' W

0 4 1u 0 o w

0)C 04 0 i . CO0)0 rh0 014) 4Jco

0 u HQ0=0w4

0jt LM L A4

0 w = 4.4 d)Q

U) -l40 P4 O W 4J-4r4P > 0 VA 00

0- 04 -4 .o 0 '0 ) 4-4' tor a ) - 0

j-) 0- 0 j (I-P4-H V A 04

U W ) CU I 4) I,4) 9O.) *,040

Id4 4 U) j4 ~ 0 " 0 C00

cc4 0,0 r400.1 w "4 u) 1..m

44 0av4OI. 41aG,'4.40a'

U) W) 0) Oh owat'00/U 4

00)

,44

) 00

0

c)

0 OJP4

B-35
V Q 94 U0.

ad1 to a ho .

a144.1A . 4 P 94J C6 L

.4t )0 a5

u 41

1., 0 1.0

0) IO)I4 6

to .-4 04 to

a~ ~ 4.a5. a4ac 

~a f A. j A

U 44

01 ~~~F 5Q 00..O 

0a 0 ".4LU

p.a

U3

".4 04

P44 u

$4 C

B-36
MODIA: A METHOD OF DESIGNING INSTRUCTIONAL ALTERNATIVES

FOR AIR FORCE TRAINING (1977)

(Carpenter-Huffman, P. MODIA: Vol. 1, Overview of a tool for planning the use

of Air Force training resources, R-1700, Project Air Force Offloe (AF-RDQA),

Washington, DC: Hq USAF, July 1977)

(Carpenter-Huffman, P. MODIA: Vol. 2, Options for course design, R-1701,

Project Air Force Office (AF-RDQA), Washington, DC: Hq USAF, July 1977)

In the early 1970s the Rand Corporation undertook an extensive effort for

the Air Force to develop a methodology for the design and cost analysis of an

instructional system. MODIA is designed as a tool to help the AF manage resources for formal training by "systematically and explicitly relating quantitative requirements for training resources to the details of course design and

course operation during the planning stage." (Carpenter-Huffman, 1977, p. 2)

MODIA has been computerized and is designed specifically for use in the five

Air Training Command (ATC) technical schools that account for 90% of the training load.

The computerized cost component of MODIA is called MODCOM. It operates

from input selected by the user based on course operation reports output from

the Resource Utilization Model (RUM). MODIA-designed alternatives will vary in

both cost and effectiveness. They may then be compared with absolute standards

such as budget constraints, minimum required effectiveness, and with each other.

Initial input to MODIA is the product of subject matter experts in planning

and developing courses. Their path in definition of a training plan is not

specified. In MODGOM, some of the requisite cost and manning factors are stored

in the program while others are supplied by the user or planner. Total course

costs for up to five years are included. Options on the included costs and their

method of computation are available. The specific manpower and cost categories

for which estimates are developed are:

Manpower

Students

Instructors

Curriculum Personnel

Hardware Maintenance Personnel

Facilities Maintenance Personnel

Trai•ing Administrative Personnel

Basc Operating Support Personnel

Medical Personnel

Cost Categories

Courseware Procurement At !Hardware Procurement

Facility Construction

B-37

+ _ .. . -II
Pay and Allowances

Students Instructors

$uppert Personnel Pemanent Change of Station (PCS)/Tumporary Duty (TDY)

Instructor Training Miscellaneous Operating Expenses

Alternatives that may be examined are changes in the number of entrants;

course duration; grade structure of the instructor force; the levels of staff

and base support; pay and allowance factors; PCS/TDY factors; and the types of

courseware and hardware used.

The author states that, although designed for use with MODIA, MOD(OM may

be operated independently. It produces five outputs:

graduates by student type

student and staff man-years

courseware, hardware, and facility characteristics

total course costs by functional element

total course costs by program and appropriation

MODCOM was designed to include 'relevant' costs, that is, only

those costs which are both incremental (lie in the future) and

variable (with respect to the decision to offer a particular course). Costs incurred as the result of past decisions and

costs unaffected by the existence or size of the course under

analysis are not considered relevant to the selection of an

optimum course design.

Since it is intended for comparative purposes, all costs are in constant dollars. Inhiprited and residual value of resources are considered. It is not

suitable for short-range budgetary planning, nonresident training, or courses given on an intermittent basis.

A large number of cost data elements are input by the user -- mostly average costs and rates or factors. Some such as personnel costs are calculated

from such input as grade and pay scale matrices. Sample sources are shown in Table B-5.

Output samples are shown at Annex IV. Course costs are displayed as investment

(courseware and/or hardware procurement and facility construction) and operating

(pay and allowances, military/civilian personnel; PCS; TDY; instructor training;

and miscellaneous. The final output translates costs into program and budget

appropriation cost categories as defined by their functional cost elements. i

B-3d
Table 3-5. SU~qqeted Cost Data SOUXC~SG, NOL~A

SOURCES:

1. Keesler Technical Training Center, Kessler AFB, Missippi

2. USAF Contand Planning Foctmr, APR 173.10, Department

of the Air Force, Headquarters USAF, Washington, D.C.

3. Rand-derived estimatels).

4. Rand astimates based on data in The Audio- VaW Equipmwt

0imectoty, National Audio-Visual Association, Inc.,A

Fairfax, Va.

5. Comptroller. Headquarters ATC, Randolph AFB, Texas

6. EPIEG RAM (newsletter of the Educational Products

Information Exchange Institute, N.Y.), No. 13,

April 1, 1973

7. OCS Plans. Directorate of Manpower end Organization.

Headquarters ATC, Randorlph AFB, Texas

8. A TC Cost Factors Summary, Department of the Air Force,

Headquarters ATC. Randolph AFB, Texas

9. 0MB3 Circular No. A-94, "Discount rates to be used in evaluating time-distributed coats and benefits," March 27, 1972,

* I Office of Management and Budget, Washington. D.C.

B-39
COORDINATION OF HUMAN RESOURCE TECHNOLOGIES (CHRT) (1978)

Goclowaki, J.C., King, G.F., Ronco, P.O., & Askren, W.B. Interation and ApPli-

"cation of Human Resoure Technologies in Weapon ystem Daeian: Coordination of'

Five Human -Resource Tec•noo•lies, AFHRL-TR-76-6 (I) Brooks AFB, TX: U.S. Air

Force Systems CoMmand, 1975.

Goclowski, J.C., King, G.F., Ronco, P.C., & Askren, W.B. Integration and Application of Human Resource Technologies in Weapon System Design: Processes for

the Coordinated Application of Five Human ResoUroe Technologies, AFHR.-TR-76

(11) Brooks AFB, TX: U.S. Air Force Systems Comnad, March 1978.

Goclowski, J.C., King, G.F., RonQo, P.O., & Askren, W.B. Integration and Application of Human Resource Technologies in Weapon System Design: Consolidated Data

ae Functional Specification. AFHRL-TR-76-b (III) Brooks AF5, TX: U.S. Air

Force Systems Command, May 1978.

Goclowski and his associates, in work conducted for the Air Force Human

Resources Laboratory at. Wright-PFtterson Air Force Base, OH, have developed

(1978) a methodology for coordinating five human resource technologies (maintenance manpower modeling (MM*), instructional system development (ISD), joh guide

development (JGD), system ownership costing (SOC), and human resources in design

trade-offs (HRDT). This methodology (CHRT) quantified the reliability, maintainability, manpower, training and JGD requirements for a weapon system. It was

designed for use by planners early in the system acquisition and brings the influence of the aforementioned quantified factors to bear on design, maintenance,

operations and support concepts and permits estimation of ownership costs.

baiCHRT, termed a predictive and product-oriented methodoLogy, depends on four basic activities. '

1. Development of a consolidated data base.

2. The integrated requirements and task analysis.

3. ISD and JGD product development.

4. The impact analysis.

Task requirements are to be developed very early; a traditional, integrated

task analysis, during full scale development. CHRT may be reiterated for alternative approaches. IS and JG products follow the lost task analysis. The cornputerized methodology was to be tested using data from the advanced medium STOL

transport (AMST) and the methodology refined. (A report on this work has not

been identified.)

An objective of this development was the production of weapon system operating and support costs. To this end, the method is subordinate to the ILS. The

ILS plan, properly executed, is cited as the primary data source for ten subordinate categories, including (inter alia) personnel and training. However, the

detailed data required for the logistic support analysis (LSA) do not become

available until the full-scale development phase. The CHRT process is planned

to fill a similar need for the conceptual and validation phases and support the

detailed design in later phases.

B-40
The costing portion of CHRT, the SOC, was not fully developed at the time

of this report. The authors state that more comprehensive models are needed to

identify the "real ownership cost drivers." As developed, the SOC technique

builds on data gathered for or derived from operation of the MM, ISD and JGD

models. A flow diagram of SOC is shown in Fig. B-13. When the trade-off portion of the methodology is exercised, costs may be constrained. The authors

recognize that equation and models for obtaining cost estimates are not standardized and that sources of data do not always adequately reflect the syntem under consideration.

Task is basic to the CHRT. It is defined as "an action or reaction related

to equipment, e.g., operate aircraft, remove radio, replace transistor. The

conclusion that should be drawn from the example is that a tawk may be either

general or very specific and that most tasks imply subtaska." (Goclowaki et al.,

1978 (a))

Task is more precisely defined in Vol. II as: "A composite of related activities (behaviors) performed by am individual and directed toward accomplishing a

specific amount of work within a specific work context. These activities usually

occur in temporal proximity with the same displays and controls and have a common

purpose. Each task has a goal."

General data requirements for the SOC are shown in Table b-6 along with other

essential data for exercising the CHRT. Forty-one items (of 42 data items and 27 sub-items) are shown applicable to the SOC. Most of these are derived from

MMM, ISD, and JGD (26 items) and others are input to HRD (9 items). Those unique

to SOC are unit cost goals, D-T-G goals, R/W consideration, and a number of costs

(manuals, LRU spares, aircrew, fuel, depot repairs, faciities, inventory technical records data, on/off equipment maintenance, and maintenance material. As

may be seen, SOC derives costs for more than the training system. SOC is designed to translate huizan resource data to cost data on a system or subsystem

basis.

In the conceptual and validation phase the task list is obtained from general maintenance requirements, a knowledge of equipment comparability, maintenance event data, maintenance activity data, interviews with maintenance personnel, comparable tech data and training resources, operations and mission concepts

and plans, operations requirements, comparable training courses and a review of

existing training resources. Data are separated into maintenance and operational categories and are stored in task condition matrices.

In the full scale development phase, a general task analysis is supposed to

be conducted for both maintenance and operations and following ISD and JGD, a

detailed task analysis.

It is anticipated that through experience cost estimating formulae for the

SOC will be developed. An example is shown in Annex VI(1). The cost model (a

LCCM) is shown in Fig. B-14. The authors caveat that the SOC must be revieweJ

for applicability to each weapon system where it is used and if necessary tailored to fit a particular system during the conceptual phase. it requires numerical data input for each element (Annex V(2)). The suggested sources are standard

B.-4
Table 3-6. Data Requirements for SOCj

HUMAN RISOURCI TECHNOLOGY DATA

DATA ITEM MCD1? MMMT JGMY ISOT ICCT

1, Viable Design Alternatives x

2. Other Alternatives x

a. Training X X X

b. Manuals x X x x ,-c,3 Kt K K K

d. Maintenance x X x X X

a. operations X x X

3. .upport Goals

a. Rqelalillty X

b, MMH/IFH X X X X

c, Avallability X

d, UOL X x X x

1. Spares X X X X

4. Unit Cost Goals x

S. O.".G Goats x 6. RIW Considerations x

7. MPulti.National Considerations X

I. Annual Flying Hoburs x x

S. Number of mas X X x

10, Number of Aircraft X x x

11. Crows Per Aircraft X X x X X

12. Crewmen per Crew X X x X X

13. Crew Makeup X Y x x X

14. Missions X K

IS. Misnion Essential Elements X

16. Performance X X

17. Configuration x x X x x

18, Construction X X X x x

19, Expected Operational Life X x

20. Maintenance Probabilities X x x X

21. Maintenance Times X K x x

22. Skill Category X X X x

23. Skill Level X X x X

24. Crew Size X X X x

25. SE Utilization X X X x
Table B-6. Data Requirements for SOC (Continued)

HUMAN RESOURCE TECHNOLOGY DATA

DATA ITEM HRDTr MMM" JODT ISOT aOCT

26, Safety Hazards X X X X

27. Available Personnel

a. Years of Service X X

b. Labor Rate X

c. Scores X X

d. Retention Rate X X a. Predictions X X!

26. Task Freauncy X X

29. Task Criticality X X

30. Task Difficulty X X

31, Degree of Proceduralization X X

32. Content of Task Information X X

33. Job Guide Concept X X

34. Job Guide Status X

36. Monual Covtent X

36. Training Concept X X ;

37. Training Status

38. Course Content X

39. Time to Train X

40. Quantity to Train x x

44. Training Resources X X

42. Cost

a. SE Investment X X

b. Manual Investment X

c. LRU Spares Investment x

d. Aircrew X

a. Fuel x

f. Depot Repairs x

g. Facilities X

h. Inventory X

i. Technical Record Data X

j. On/Off Equipment Maintenance x

k. Training X X

I. Maintenance Material X

4I

B-43

~ ~ ~oil 11
LLI

ul

wJ

.j.Ai

LEU

0 U0

L~i

2I- >.0U

>. U3r

U),

0101

2 .9

t~ I-

~CII

0 on C~I U.u 0 A

*Oz > ceOC? 0

A~~P~~P*~~-~. CL.* :.- m 4 -
SYSTEM OWNERSHIP COST*

SUPPORT INVESTMENT COSTS* + OPERATING COSTS- + SUPPORT COSTSSUPPORT INVESTMENT COSTS*-CE+CJ L

CSE - COST OF SUPPORT EQUIPMENT

CjG - COST OF JOB GUIDES

CLS -COST OF LRU SPARES

O&S OPERATING COSTS *CPC + CFC

CPC - COST OF AIRCREW

CFC - COST OF FUEL

O&S SUPPORT COSTS- CDR + CMM + CFA + CIM + CTR + CEM + CPT

CDR -COST OF DEPOT REPAIRS

CFA -COST OF FACILITIES

Cim- COST OF INVENTORY MANAGEMENT

CTR -COST OF TECHNICAL RECORD DATA

CEM -COST OF ON-OFF EQUIPMENT MAINTENANCE

CPT -COST OF PERSONNEL TRAINING -

*All costs expressed in annual dollars, i.e., dollars/year.

Figure B-14. The System Ownership Cost Model for CHRT

B-'45
government offioes/documents such as pay and allowance tables, cost factor manuals and regulaticns. Equations for each cost shown in Fig. B-14 are included in

Goclowski et al., 1978, (c). As outlined above, the task list creates the training program with its requirements for job guides, personnel training, media, etc. I

and these are costed along with all other support costs. Thus, althovgh the

life cycle training costs are included and calculated, they do not appear as

unique items in the final output.

B-46

Iii

I

., •I

I:,

i,• , ,..-: ... . ...;-= : .. ......... . .. . . . .. -h -:;. ,. • ,'. p

• • ., . . ;,• ,• ,,4 •,( , ; •, d •. . . • , .• . • .• , ." .- .• ._. , . - . .• ., . • i• , • • • -,. ,,••,•.•. ., . • N,.•• . _,.•,., . ...,•,..::. . .. •:•....B...-46• _
AN AIR DEFESE CTEA METHODOLOGY (1978)

Hawley, J.K. and Thomason, S.C. Development of an Air Defense Cost and Training

Effectiveness Analysis (CTEA) Methodology (for the AN/TSQ-73): Vol I - CTEA

Within the Life cXcle System Management Model. Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences, 18 December 1978.

__ ___, Vol. II, Demonstration CTEA on the AN/TSQ-73.

Earlier we referred to a methodology developed by Hawley and Thomason (1978).

This conceptual model was designed to perform CTEA at the points required by the

LCSMt. It was partially demonstrated on an existing system (Hawley and Thomason,

1978a). The research documentation caveats that detailed procedures and analysis

techniques must be developed for each of the points in the LCSMM to produce a

'full blown process' model from the conceptual model.

Hawley and Thomason prepared both a CTEA methodology and list of events/

stages of the LCSMM. Then they integrated these two series of activities since

they are concurrent processes. In their opinion this integration is necessary since hardware--related and training-related activities are so interactive that

'imbalances in development can easily cause CTEA to become a relatively futile

gesture'. These analysts also start with a list of tasks or activities generated

from analytical study of the materiel system hardware. They state that, for

conceptual systems, an activities list is developed as part of the training input

to the letter of agreement (LOA). After the development of the hardware prototypes, the task list is obtained from the Job/Task Analysis; the job conditions,

from the Mission Analysis; the standards, empirically or judgmentally derived.

Il

The next important step in this methodology is the formulation of EEA to

assess effectiveness of the training system. For each EEA, methods of training

effectiveness (MOTE) and measures of resource requirements (MTRR) are to be

specified. The MTRR, the resources recuired to reach the training objectives,

constitute the cost model input.

Examples of categories of MTRR are the following:

1. Operating requirements for a proposed training system or methodology:

o lnstructors

o Students

o Facilities

2. Support requirements:

o Personnel

J o Materiel

o Supplies consumed

o Parts expended

3. The reliability, availability, and maintainability (RAM) of the

training equipment. Included in this assessment is the maintenance

policy for complex training devices.

B-47

S . . .m _ I [ . - .. . . I I I . . . I - <t# • : .,.'. ,;
Suggested sources for existing data are:

1. Past studies and performance data (e.g., DT/OT, SQT, ARTEP, etc.)

on similar or ancestral systems.

2. Completed tests (e.g., DT/OT) on the subject system.

3. Task/function analysis data.

4. Training test scores.

5. SQT and ARTEP results.

6. Secondary data sources.

"o Literature

"o Institutional data bases

"o Expert opinion/judgment

The analyst is advised to be aware that:

I. Training methodology must be specified at a level of detail that

can be reliably costed.

2. Cost estimates should include initial (set-up or sunk) costs as

well as recurring costs.

3. Some COEA may require training LCCE (especially true if a large initial investment is to be made in equipment subject to depreciation and/or obsolescence).

Input cost data are categorized as falling into these categories:

1. Facilities - land, buildings, etc.

2. Equipment - training and support

3. Instructional material - development and acquisition

4. Personnel - instructors and training support

5. Students

6. Suppli.es

7. Miscellaneous (e.g., travel)

Hawley and Thomason also recognize the importance of the training concept

-co the cost. They advise the cost analyst to include in his cost data requests,

at a minimum, the following:

1. The training concept - a description of how the training will

be carriec out, instructional methods to be employed, and so

forth.

2. The make, model, quantity, and quality of materiel required.

3. The quantity, grade, and skill level of personnel required.

S~B-48
.~ ~ .... 7

4. The time requi~rements and materiel usage data -- the length of

course, facilities required, supplies expended, and so forth.

They state that training resources cost quotes should be expressed in constant year-dollar's and that DOD policy specifies that for CTEA the base costI

year is the fiscal year following the calendar year in which the study is

scheduled .o'be completed. That is, the base cost year is the year the fullscale or limited production of the materiel system is projected to begin (e.g.,

between ASARC III and I00). (A reference for ths information is not cited.)

Other Costs projections recommended are:

1. Sunk costs: Cost invested in tangible assets that can be

recovered only by use of the assets over their service life.

2. Fixed costs: Costs tnat are constant in total amount regardless

of production volumne, e.g., depreciation in building space

occupied.

3. Variable costs: Costs that tend to vary proportionately to the

activi.ty in the function involved, e.g., direct materials, direct

labor, operating supplies, and so forth.

nent suc asthe cost of maintenance and repair on a shut down

The alo sy tatprovision should be made for including CEtprjc

the ostof he raiingprogram over the life of the materiel system (or over

some reasonable portion of it). The life cycle cost methodology should calso

incudethecapbilty ordiscounting training cost over the life of the materiel system.

This research holds that the accounting and discounting requirements are

genraly aailbleinexisting cost models (e.g., the TECEP costing model presented in TAEG Report No. 1) and that in the absence of special cost analysis

requirements, one of the existing procedures be used. They warn, however, that

the nature of the available training cost data must i~iatch the input requirements

of the costing model. Obversely, if the EEA require features not available in

the "canned" model, it must be adapted or not be used. This methodology also

recommnends that specific comparisons can be made to highlight critical cost I considerations and cost-related issues that require further exploration in the form of sensitivity analyses identified. They advise sensitivity analyses, defined as investigations of the effect on training cost and resources if certain

assumed parameters take on different values. As examples of such parameters j they cite:

1. Training equipmient RAM characteristics.

2. Instructor, salaries (e.g., converting from military to civilI

service instructors or from enlisted personnel to officers).

3. Student selection factors (e.g., recruiting a larger proportion

of categor-y III or IV students resulting in a higher attrition

rate or more reeiltime).

* - B-49
The training cost sensitivity analyses are conducted to identify particularly sensitive parameters or assumed values so that special care can be taken

in estimating these parameters more closely. The objective is to select training methodologies that are cost-effective for most of the range of likely values

for these assumed parameters.

In a second volume reporting this research (Hawley and Thomason, 1978(b)),

the authors attempted to exercise the methodology using data from a developed

system (the AN/TSQ-73). A number of problems (believed typical) were found --

LCSW. not strictly adhered to: lack of coordination between hardware system

development (SD) and ISD (far behind HDS); formal task list/analysis non-existent;

TM in draft and full of errors. The HSD had passed through a third DT/OT and

it was too late for CTEA to be conducted for input to the final COEA. Other

attendant problems were present.

Because of the 'primitive' state of ISD, the researchers chose to use

Jorgensen and Hoffer's (1978) training consonance method (an estimate of compatibility of task and training, theoretically correlated with training effectiveness: two training courses - the Army's and the contractor's -- were

proposed and were the candidates for exercising the CTEA methodology.

The cost portion of the CTEA was not exercised. Programmed instruction

for one proposed system was the only suggested difference. Although found to

be of higher acquisition cost, this material was of insignificant cost when

amortized over the life of the system.

B5

5-50 .
APPE)IX C

LIST OF ABBREVIATIONS AND ACRONYMS

AD Advanced Development

AIT Advanced Indivicual rraining

AP Acquisition Plan

APM Army Program*Memorandum

AR Army Regulation

ARTEP Army Training and Evaluation Program

ASARC Army Systems Acquisition Review Council

ASI Additional Skill Identifier

ATM Analogous Task Method

1MDWCARAF The BDM Service Company Combined Arms Research and

Analysis Facility

BOIP Basis of Issue Plan

BOIPT Basis of Issue Plan - Tentative

BT Basic Training

CDB Consolidated Data Base

C/E Cost Effectiveness

CFP Concept Formulation Package

CHRT Coordinated Human Resources Technology

COEA Cost and Operational Effectiveness Analysis CTEA Cost and Training Effectiveness Analysis

DA Department of the Army DARCOM U.S. Army Materiel Development and Readiness Command

DCP Decision Coordinating Paper

DRIMS Diagnostic Rifle Marksmanship Simulators

DSARC Defense System Acquisition Review Council

DT Development Test

DT/OT Development Test/Operational Test

DTD Directorate of' Training Developments

FYDP Five Year Defense Plan

HRDT Human Resources in Design Trade-Offs

ILS Integrated Logistic Support

IPR In-Process Review

IPS Integrated Personnel Support

IRTA Integrated Requirements and Tasks Analysis

ISD Instructional Systems Development

ISP Integrated Support Plan

ITC Institutional Training Cost

ITDT Integrated Technical Documentation and Training

ITP Individual Training Plan

ITV Improved TOW Vehicle

JGD Job Guide Development

LCC Life Cycle Costs

LCCE Life Cycle Cost Elements

LCSMM Life Cycle System Management Model

LOA Letter of Agreement

LOGCEN Logistics Center

LR Letter Requirement

LSA Logistic Support Analysis

LSAR Logistic Support Analysis Record

MENS Mission Element Needs Statement

MILPERCEN Military Personnel Center

SMM4 Maintenance Manpower Modeling

MOS Military Occupational Specialty

i C-1

.::• ,,• ,,,.._ , . : . ...... ,,,.,,.-,1.. :.• •. •• 
MODIA Method of Designing Instruction Alternatives

MOD Modification

MOTE Measures of Training EffectivenessNET

MPA Military Personnel, Army

NET New Equipment Training

OAP Outline Acquisition Plan

ODP Outline .Development Plan

OICTP Outline Individual Collective Training Plan

OJT On-the-job Training

OMA Operations and Maintenance, Army

OSUT One Station Unit Training

OT Operational Test

P h Probability of Hit

PIP Product Improvement Program

PM Project Manager

POI Program of Instruction

PQQPRI Provisional Qualitative and Quantitative Personnel

Requirements Information

PROC Procurement

QQPRi Qualitative and Quantitative Personnel Requirements

Information

RAM Reliability, Availability, Maintainability

RDTE Research, Development, Test and Evaluation

RFA Rimfire Adapter

ROC Required Operational Capability

RTE Relative Training Effectiveness

RUM Resource Utilization Model

SAT Systems Approach to Training

SOC System Ownership Cost

SQT Skill Qualification Test

TASA Task and Skill Analysis

TAEG Training Analysis and Evaluation Group

TCA Training Consonance Analysis

TECEP Training Evaluation Cost Evaluation Program

TDDA Training Developers Decision Aid

TDIS Training Developments Information System

TEEM Training Efficiency Estimation Model

TM Technical Manual

TOCI Training Option Cost Indicator

TOE Table of' Organization and Equipnent

TQQPRI Tentative Qualitative and Quantitative Personnel

Requirements Information

TRADOC Training and Doctrine Command

TRAINVICE Training Device Effectiveness Model

TRAM Training Analysis Model

T .AMOD Training Requirements Analysis Model

TSD Training System Design

TSM TRADOC System Manager

TTC Total Training Cost

EU User Interface

UTC Unit Training Cost

WSTEA Weapon System Effectiveness Analysis

C-2
DISTRIBUTION

I US ARMY WESTERN COMNANU ATTNI APPt

I I)EPAmtIMEN1 OF IHE NAVY THAININU.ANALYSIS ANU.EVALUATION GROUP

I HQUA ATTNo UAA(3-EU

I NO* ICATA ATTNI ATCAT-OP'Q

P HQUA RESEARCH AND STUUIES OFC

I MILIIARY OCCUPA1I1NAL UDVLLOPMENT DIV DAPC-MSP.-O RM 852C HOFFMAN SLOG I

4 OASU (MRA AND L)

I HQUA OUCSPER

I HEADwUARTERS9 US 4ARINL CONPS AT7Nl COUE MPI-lO

2 US ARMY EUROPE AND SEVENTH ARMY

P HQ ImADOC TECHNICAL LIBRARY

I MILIIARY OCCUPATIONAL UEVLLOPMENT DIRECTORATE ATTN& ATZI-NCR-MS-9 M M, 3N33 HOFFMAN BLDG I1

I DATA ANALYSIS UIVISIUN AITNI ATLI-NCA-MDs HUFP1AN BLDG 1I

1 HI)A ARMY FORCE 4ODLRNIZATION CUORU14ATION OFFICE

1 1230 USARCOM HESERVE CENrLR

I US ARMY SOLDIER SJPPONT CtNTEk t

I OIRE0TORA1E OF THAININb AITNI ATZU-T

I DIRELTORATE OF C04BAT DEVELOPMENTS ATINs ATZLQD

I HGDAWCOM MARINE CORPS LIAISON OFC

I UEPANTMEN1 OF THE ARMY US ARMY INTELLIGFNCE $ SECURITY COMMAND

I US ARMY CECOM ATTN$ OHSEL-ATUU

1 USA PORCES COMMAND

I US MILITARY UIbTHICT OF WASHIN6TUN OFC OF EQUAL OPPORTUNITY

I NAVAL CIVILIAN PERSONNEL OMNU SOUTHERN FLU UIV

2? AR! LIAISON OFFICE

I 7TH ARMY TRAINING COMMAND

I HQODA OCS STUDY OFFICE

.1 U.S. NAVY TRAINI14 ANALYSIS EVALUATION GROUP

I USACuEC ATTNt A1EC-EA-E hUMAN FACTORS

I OASA (RUA) PEPUIY FUR SCIENCL AND TECHNOLOGY

I OFC JF NAVAL RESEARCH /

I AFHRL/LRT'

I AFHRL/LRLG

I NAVY PERSONNEL R AND 0 CENTER UIR.CTOR OF PROGRAMS

2 OFC uF NAVAL RLSEARCH PERSONNEL AND TRAINING RESEARCH PROGRAMS

I NAVAL PERSONNEL H + U CENILR /

I NAVAL AEROSPACE MEDICAL RSCH LAB AEROSPACE PSICHOLOGY DEPARTMENT

1 USA IRAUOC SYSTEMS ANALYSIS ACTIVITY ATTN: ATAA-TCA

I HEADwUARTERS, COAST GUARD CHIEF$ PSYCHOLOGICAL RSCH RR

I ATTNt ATTU-AIB-TA

I USA MATERIEL SYSTEMS ANALYSIS ACTIVITY ATTNI 0RXSY-C

I USA cOLD REGIONS TEST CEN ATTN' STECR-OP

1 USA RSCH DEVEL * STANDARUIZA GPv UeK.

I USAAi4L LIBRARY

I USA IRADOC SYSTEMS ANALYSIS ACTIVITY ATTN$ ATAA-SL (TECH LIBRARY)

1 UNIFuHMEU SERVICES UNIT OF TH. HEALTH SCI DEPARTMENT OF PSYCHIATRY

1 GRONINGER LIBRARY ATTNI ATZF-RS*L BLDG 1313

1 CENTER FOR NAVAL ANALYSIS

1 NAVAL PERSONNEL R AND U CLN LIBHARY ATTNS CODE Pl16

1 USA ACADEMY OF HEALTH SCIENCES STIMSON LIBRARY (DOCUMENTS)

I SCHOuL OF SYSTEMS AND LOGISTICS t

I DEPARTMENT OF THE NAVY IHAININb ANALYSIS ANU [VALUATION GP

I USHA DEPT OF bEHAVIORAL bCI AND LEADERSHIP

I USA LOMMAND AND GENERAL STAFF COLLLGE ATTNI LIBRARY

I USA IRANSPORTATIO4 SCHOOL USA THANSP TECH INFO AND RSCH CEN

I USA ADMINCEN TECHNICAL RESEARCH BRANCH LIBRARY

I USA PIELU ARTY BU t

I INSTITUTE FOR UEFENSE ANALYSES

I USA IHAINING SUPPORT CENTER ATTNI ATIC-OST-PA

I USA MOBILITY EUUIDMENT R AND U COMMAND ATTN% ORUME-fG

I DA U- ARMY RETRAIJINO BUE RESEARCH ÷EVALUATION DIVISION

D-1

""lo, ..-•0.

•.•:: 77.,.•' ••..s . .. " ,. .•. . , ... •,•• •
I US MiLITARY ACAf)L'4Y ULPI'. OF HISTUNY, HiLIG b0l

1 MAR4IA~ CORPJS INSTITUTE

I USAAYNC AND FT* WJCKLR AITNI ATZW-ES

) US MILITARY ACA0,L4Y DIRL1TOR ot INSTIfUTIONAL RSCH

I tISAAuS-LIlbRAHYL)OCUf4ENT!3

I USA iNTELLIGLN~-F CLN A14Ut S(..Ii AITNI Ars1-IuTSFL

I NAVAL POSTGkADUATE SCH AfTNI oUULLY KNOX LIbRARY (CODE 1424)

1 USA iRANSPORTATIOUJ SCHOOL Ofr~ulY ASST. COMMANDANT EUUCA. ILCIINOLOGY

1USA hIGNAL SCHOOL ANU Flo 60NUON ATP48 ATZH-ET

i USA ARMOR CENTý-H + Fr. F4NUX 0F11IIL. OF AwMOR FORCE M6T + STANDARDIZATION

I USA bI(3NAL SCHOOL 4 Fl. bUHDON 1WUCAfIONAL TLOHNOLOGY DIVISION

I HO AjC/xPTD TRAININb SYSTtMS OEVELPMEN'T

IUS Ant4Y ARMOR CENTER AT11NI ATIK-TU-PIO

I USA wUARTERMASTER SCHOOL DIRECIURAJE UF TRAININti DEVELOPMENTS

I. US CuAST tiUAWD ACADEMY/

I USA tHANSPORIAIIOUJ SCHOOL DIREFCIORATE OF TRAINING *ý DOCTRINE

I USA INFANTRY SL,1ODL LIbHANY

I USA MqP + CHEM SCH/ING CEN + FT, MCCLELLAN ATTN$ ATZN-PTS

1 USA i"P 4 CHEM ScLH/TNG CEN * FT. MCCLELLAN DiR4 COMBAT DEVELOPMENT

1 USA MP + CHEM SCH/TN(G CEN It FT. MCCLELLAN DIR. TRAINING DEVELOPMENT

t USA e4P + CHEM bCH/TN'i CLN + FT. MCCLELLAN ATTN: ATZN.MP-ACE

I U54 INSTITU-TE UF ADMINISTHATION ATTN: RFSIULNT TRAINING MANAGEMENT

I USA i-IELfl ARTILLERY SCHOOL MORHIS SWLIT L18HARY II USA INSTITUTE OF ADMINISTRATION ACADEMIC LIbRARY

I kiStk tNGINLER SCHO3L LIHNANY ANU LEAR1NING RESOURCES CENTER

1 USA ARMOR SCHOVL (USARmS) ATTN; LIbRA~y

I US AaRMY INTELLIGENCE CLNIE.P + SCHOOL ATTN: ATSI-TP

I US ARMY INTELLIGIEN~CE CLNTLN * SCHOOL ATTN: ATSI-ES

1 EPARTMENT OF 1HE AlIR IIRLt~ AIR UN1VE.HSITY LIWRARV WAC)

I HQ TmADOC TRAININJG DEVELOPMENT IWbITUTE:

2 CANAuIAN JOINT STAFF ENCSTF

I FENH AMYATTAC*k

IAUSTRIAN EMBASSiY DEFENSE. MILITARHY ANU AIR ATTACHE

3 CANAuIAN DU.FENLE LIAISON STAFF ATTN: COUNSELLOR, DEFENCE R AND D

I ROYAL NETHkRLANI)S LMbA$SY MILITARY ATTACHE

2 CANAUIAN FORCES PERSONNEL APPL RSCH UN-IT

I AWAY PERSONNEL RLSLAHCH LSTAbLISHMENT

6 LliiRARY Of CUNURLSS EXCHANGE ikN0 GIFT uiv

1 OEFEgiSE TECHNICAL INFORMATION CLN ATTN: I)TIC-L)DA-P

140 LIFIRARY OF CONURESs UNIT DOCUMENTS C.KPEOITING PROJECT

I US GuVERNMENT P~RINJTING OFC LIýJRARY9 PUB3LIC UOCUMENTS (DEPARTMENT

I US GOVERNMEN't PRNINING UFQ LIHkRAY AW4L STATUTORY, LIR DIV (SLL)

I THE ARMY LIIIRAHY ATTN; ARMY SrUDILS t~LC

NUHE F AL)URESSLEES 108 :

TOTAL NUMBER OF COPIES L-87