LW I S A -k & ~ . % t~-r~

Research Note 83-34

Implications for Development of the

Collective Training Information

System (CTIS)

Thomas D. Scott and Ana G. Ekstrom

',,

Submitted by

James A. Thomas, Chief

ARI FIELD UNIT AT PRESIDO OF MONTEREY, CALIFORNIA

and

Harold F. O'Neil, Jr., Director

TRAINING RESEARCH LABORATORY

4W

DTIC

U. S. Army E

Research Institute for the Behavioral and Social Sciences

September 1983

Approved for public release; distribution unlimited.

Th is report has been cleared for release to the Defense Technical Information Center (DTIC). It has been given

other primary distribution and will be available to requestors only through DTIC or other reference services such

* the National Technical Information Service (NTIS). The views, opinions. end/or findings contained in this

report ae those of the author(s) and should not be construed as an official Department of the Army position,

policy, or deeision, unless so designated by other official documentation.

,14
Iha~ Unclassified

SECURITY CLASSIFICATION Of THIS PAGE (When, Date Bntere4D

REPORT DOCUMENTATION PAGE READSRCTONS IREPORT NUMBER .GOTACCESSION NO. 3. RECIPIENT'S CATALOG NUMBER

ARI Research Note 83-347 ."v

TITLE (arid Subtitle) S. TYPE OF REPORT & PERIOD COVERED

IMPLICATIONS FOR DEVELOPMENT OF THE COLLECTIVE

TRAINING INFORMATION SYSTEM (CTIS): A SWRMARY

6. PERFORMING ORO. REPORT NUMBER

AUTHOR(&) S. CONTRACT OR GRANT NUMBER(&)

IThomas D. Scott, Ana G. Ekstrom

3-:. PERFORMING ORGANIZATION NAME AND ADDRESS 10. PROGRAM ELEMENT. PROJECT, TASK

ARI Field Unit - PERI-IO AREA A WORK UNIT NUMBERS

PO Box 5787 2234A9 Presidio of Monterey, CA 93940 Q674A

CONTROLLING OFFICE NAME AND ADDRESS 12. REPORT DATE

US Army Research Institute for the Behavioral September 1983

and Social Sciences IS. NUMBER OF PAGES

5001 Eisenhower Ave., Alexandria VA 22333 15

4. MONITORING AGENCY NAME & AODRESS~ll different from Controftin# Office) IS. SECURITY CLASS. (of tis report)

".5 *Unclassified

IS. DECL ASSI FICATI ON/ DOWNGRADING

SCHIEDULE

16. DISTRIBUTION STATEMENT (of title Report)

Approved for publ4 c release; distribution unlimited

7. DISTRIBUTION STATE14ENT (of the abstract uttered in Block 20. It different from, Report)

IS. SUPPLEMENTARY NOTES

5 KEY WORDS (Continueaon reverac aide it necoeee7 and identify by block nlumber)

Information System Performance Data

Collective Training Quality Control

Needs Assessment CTIS

Eval uation/Feed back

34L _SIRACT rCootfibu m tornm eb nufesemp' dW Ideaify by block nmer)

~In order to make a preliminary determination of components of a Collective 'I Training Information System, the Army Training Board requested that the Army

* Research Institute conduct an Information Needs Assessment. A series of semistructured interviews were conducted with training managers, executives, and VARTEP developers.,

Interviews revealed a definite need for feedback from field users of ARTEP

i-ocunents to the developers pertaining to the quality and utility of their

go147iIB 3 IOTIO10 OF INOV 6S IS OBOLETE Ucasfe

SECURITY CLASSIFICATION9 OF THIS PAGE (When Date Ene-ed)
v

r "wz r rr -c---- i , - - - . _ - .--& - -" - r . w- -..-. ... . ... . -"-""  '"

. CUMITY CLASSF.CATIO. OF .gS AQ(W , t eg&nto..E) ,.

" 20. Abstract (cont.)

products. Needed evaluation feedback by field units at various levels revealed a

"number of issues that a collection information system would need to take into

account. This report presents a summary of these issues and outlines information

system requirements and methods that could address the identified issues.

rAccession Po

NTIS GRA&I

DTIC TAB

Unannounced

, Justification

- ByDistribution/

Availability Codes

' ie Avail and/or

-. ~Dist S~a -'

b. ..

ii Unclassified

SECURITY CLASSIFICATION OF TMIS PAGE(11osn Date Entored)

<S~% *.* **V * 2 -.. - ..

. . MS. .... . # .. ~ S .... ... 5. 5**. . . . . . . . . .. 5. 2~*;
TABLE OF CONTENTS

Page

Introduction 1

General Information Requirements 2

a. Training Developers 2

b. Training Managers 3

a. Unit Executives 4

d. Additional Coriderations 4

Quality Control 5

Information Anal ys is and Use 5

Information Access 6

Manpower and Perannel 7

Devel opmen t Issues 8

a. System D evelopmen t 8

b. Configuration 9

Col 1usions 9

References 10

ti

l -- o r ., ' .. -. , , .- ," :,' , .. , . o .., . . .. .-. ... " ; '. . . -. . . . . , , . . ". .. . . " .. o.. -, . - . . . . .. . -. • . . . . .. .
INTRODUCTION

The purpose of this report is to summarize and extend the Collective

Training Information System (CTIS) work conducted for ATB over the past year.

The primary focus is on developing preliminary concepts for the CTIS, partly

based on information needs and operational considerations of three potential user

groups surveyed by All . It should be emphasized that in many cases this report

goes well beyond the data collected during the needs assessment survey. This

approach was taken in order to provide meaningful discussions of the implications

of the survey results in a systems development context. Also, some systems

development issues which did not emerge from the survey are covered here insofar

as they seemed useful in defining CTIS requirements. Wherever it seemed

advisable, potential solutions to development problems have been proposed, and

some topics in need of further examination have been identified. In contrast to

the survey reports, then, this document is relatively narrow in scope, comentrates on CTIS issues, and does not cover many of the job and ARTEP-related

topics included in the survey.

This summary report is intentionally brief and covers the more salient

development issues at a conceptual level. Detailed discussions of technical

issues have not been included, and indeed are not appropriate, until consensus on

system concepts has been reached. This report, then, should be viewed as a

preliminary step toward establishing requirements and priorities for the CTIS

development process.

Before proceeding to the main topic, let us briefly outline the needs

assessment effort. The assessment consisted of interviews with members of each

of three potential CTIS user groups: collective training developers, unit

training managers, and unit executives. Training developers consisted of service

school managers and action officers involved in developing ARTEP materials.

Training managers were battalion and company commanders. Executives consisted of

brigade commanders, G-3s, and one Commanding General. Interviews were semistructured and covered a variety of topics including opinions of the Army

Training and Evalation Program, perceptions of potential users' jobs, reporting

requirements, and needs for training-related information. Of special interest

here are the examples of training data computer printouts which were given to

each respondent for evaluation. These consisted of several ARTEP-derived

examples each in a somewhat different format, level of aggregation, or categorization scheme. The principal ones showed percentages of ARTEP standards achieved

aggregated into task/subtask, functional area (e.g., planning), or weapon system

categories. Examples were presented data by platoon, by battalion or by platoon

over time (trends). Respondents were also asked to comment on the usefulness of

normative data presentations. For example, data from a given unit might be

presented with division or even Army-wide averages for comparison purposes. Detailed oments by respondents are covered in the three previous reports and

will not be presented here. Finally, the caveats regarding the small sample size

and the representativeness of the sample, which were made in the three previous

needs assessment reports, are, of course, equally applicable to the inferences

made in this report.
GENERAL INFORMATION REWJIREMENTS

There appeared to be substantial support for the CTIS concept, but the

support was tempered by reservations. These reservations centered on the

respondents' prior experiences with innovations which have generated increased

burdens without yielding comparable benefits. Such concerns tended to be

pervasive, and if not effectively countered, they could develop into a significant implementation barrier. In order to maximize the potential for success,

early developments should concentrate on those aspects which help primary users

on their daily Jobs and which require the least amount of change or support

within the user organizations. Moreover, a close working relationship between

the user and the developers is mandatory, especially in the early stages of

development. A very good guideline for developing and implementing innovations

along these lines is contained in "A Guide to Implementation of Training

Products," by Gray and Roberts-Gray. The respondents' concerns notwithstanding,

most saw the CTIS as having significant potential for increasing the amounts and

types of information available to them, and for improving the efficiency of

operations.

Not surprisingly, reported information needs differed considerably between

the three user groups. However, the differences between operational unit users

(i.e., training managers and unit executives) differed primarily in level of

detail and data presentation formats. Training developers, on the other hand,

reported wanting different types of information than user units.

a. Training Developers

Respondents indicated a need for information onthe users perceptions of the

quality of the training products which are sent to the field. Without such

information, they contended, improvements to existing training products or

development of high quality new products is exceedingly difficult. Respondents'

coments indicate that highly detailed information is required. For example, if

unit personnel indicate that some training objectives are unclear, training

developers will need to know which specific tasks, conditions or standards are

problems, and the users' assessments of the sources of the problem in each case.

In order to handle this kind of textual information CTIS developers should

incorporate the capability for the input and analysis of free-format information

and should consider developing a scheme for using codes to flag and categorize

common types of problems.

There is, at present, no routine mechanism for obtaining information from

field user groups, and transmitting that information to the appropriate developers. It is apparent that such a mechanism is a key aspect of the projected Cris, and the information problem must be solved if the system is to yield substantial long-term benefits to the training development community.

Although training developers indicated no need for performance data per 8e,

some of those data could help to identify problems with fielded training objectives. For example, the frequency of "not evaluated," or similar entries

could give developers information about ARTOP standards which are difficult to

2
Apply in the field. Data on the frequencies of tasks and subtasks evaluated

during ARTEP exercises could provide important information on actual unit

training priorities and enable developers to focus their efforts better. Also,

these kinds of information should be easier to obtain than unit performance data,

as they are generally not considered sensitive, and impose no additional data

collection burden on the unit.

b. Training Managers.

Respondents reported that they need information which would help them

determine the strengths and weaknesses of their units, and of the leaders of

their units. Although ARTEP task-based displays of training data were regarded

favorably, respondents' comments suggest that the CTIS should incorporate software for displaying separately those data bearing on leader performance from

those data dealing with performance of the unit as a whole. Moreover, a

relatively fine level of detail is required to develop and manage effective unit

training program. Broad summaries are generally not very useful at the

battalion level except for reporting to higher echelons. Comments suggest that a

CTIS software which assisted managers in identifying comonalties of and reasons

for training deficiencies would be favorably received. In general, training

managers did not find displays of training data in terms of weapon-systems or

trends to be very useful. One of the reasons that trends were not viewed more

favorably was the impact of turbulence and turnover on training activities. Were

the personnel problems to become less severe, some trend data could be useful in

evaluating the success of unit training programs and possibly the soundness of

unit training management practices. Thus, CTIS developers should plan to develop

software to support trend analyses in the future.

Although respondents did not indicate a need for training data organized

into function categories, several noted that some data organization fully

consistent with National Training Center take-home package or after action report

organization was highly desirable. The NTC data information is organized into

"seven operating systems" which are similar in many respects to the functional

area categories, but there are currently several inconsistencies between the two

categorization schemes. These differences will need to be addressed by CTIS

developers and the NTC-TRADOC personnel in order to develop a common organizational scheme. An important area for work in the near-term is development of a

systematic method for translating NTC data into concrete training guidance at the

lower echelons (i.e., company and platoon levels). Currently there are no

explicit associations between ARTEP tasks, conditions and standards and the NTC

"operating systems." A "crosswalk" between these two types of training frameworks could enhance the usefulness and acceptance of the CTIS at the unit level.

Respondents consistently complained that extensive management related

paperwork adversely affects the time they are able to devote to training

activities. This finding suggests that the CTIS developers should examine the

requirements for and feasibility of establishing an interface between the CTIS

and an automated aid to training management.

3 o
 4 ... , .2 , . , . . . , . .  . . . . . . .. . _  . . . . . , .

c. Unit Executives.

Executive respondents reported desiring the same general types of information as the training managers. However, because of the broader nature of their

responsibilities, less detailed information is needed. For example, the executives found performance information organized into functional and subfunctional

area categories potentially useful but did not consistently indicate needs for

task/sub-task, or weapon-system aggregated data. Like the training managers,

these respondents also noted that some data organization consistent with National

Training Center will be needed. Although executive respondents generally

preferred broad summaries of training data, some indicated that they would prefer

to have the capability for reviewing detailed information. This finding suggests

that the unit executive component of the CTIS be structured hierarchically with

the broader summaries as the display "default option" and more detailed data

available upon specific query. That is, when an executive user requests

information, the highest level aggregation would be displayed first and progressively finer levels of detail would be available upon subsequent requests.

Respondents did not indicate a great need for trend data, basically for the

same reasons given by training managers, i.e., the effects of personnel turbulence and turnover. Some did, ho.ever, suggest that normative data could be

helpful in determining how realistic their expectations regarding the effects of

training are.

d. Additional Considerations

At the primary user level, trainers and training managers are usually amre

of their units' idiosyncracies. But, outside the unit, performance data often

becomes difficult to interpret. CTIS developers should consider incorporating

information into the data base which would help external users to accurately interpret and employ unit training information. Data on unit characteristics

(e.g., turbulence and turnover rates, percentage fill, and training history),

evaluator experience, exercise conditions, etc. could aid CTIS training development and unit executive user groups. In addition, information on individual

skill requirements for ARTEP training objectives, doctrinal literature bearing

on tactical tasks, alternative training methods for remedial training, etc. could

provide valuable assistance to training development users. Compilation of such

lists is not particularly complicated, but constructing a detailed crossreference indexing system is a painstaking, time consuming process. Fortunately,

some of the groundwork has been laid in the course of developing the Company

Training Plan with its fairly extensive cross-references. CTIS developers should

be able to apply much of this work directly in developing the CTLS data base file

structure. Provisions will need to be made for parallel efforts by each of the

Service Sdhools. Depending on the timing of a prototype CTIS implementation

effort in these institutions, software could be developed to support construction

of oross-reference indexes.

'41
4!

QUALITY CONTROL

Although the training development respondents did not raise quality control

issues with regard to their own operations, the CTIS could help to control the

quality of ARTEP products. The CTIS software to support both the Collective

Front-end Analysis (CFEA) management and the CFEA itself could incorporate the

capability to create audit trails for eagh collective task, thereby helping to

insure that critical development steps are not omitted. Also, criteria which

training objectives should meet and common deficiency cues could be incorporated

in the CTIS software to help developers improve the consistency and the quality

of their products.

Many respondents expressed concern over quality control issues, and were

especially skeptical about the quality of performance data resulting from

tactical field exercises. Among the reasons cited were lack of experienced

evaluators, limited mounts of time for evaluator training and variations in

exercise conditions (weather, force ratios, terrain, etc.). Although such issues

are mostly outside the scope of the CTIS development per se, if not ameliorated

they are likely to undermine the credibility of the system and adversely affect

the extent of utilization. In cases in which data is somewhat suspect, one

partial solution is to document probable sources of anomalies, thus providing

interpretive aids to the users. For example, if lack of evaluator experience is

a probable contributor to poor data quality, information on evaluator ranks, MOS,

time-in-grade, number of exercises previously evaluated and so on, can help the

user to interpret the data accurately. As noted earlier, other interpretative

aids might include data on the units recent training history, turbulence and

turnover, percentage fill, exercise conditions and training devices employed.

Because the performance data is not expected to be of exceptionally high quality,

CTIS developers should anticipate incorporation of as much capability for crosschecking data as possible.

Other ARI-POG efforts should also have a positive influence on some primary

quality control problems. For example, improved guidance on evaluator training

and evalution methods is being prepared to help units conduct better ARTEP

evaluations, and work in support of the USAIS is expected to result in more

objective, easily applied evaluation standards.

INFORMATION ANALYSIS AND USE

The data analysis requirements identified by respondents are mostly

straightforward sum, percentages and averages. Complicated analytic support

software is not required. However, as mentioned above, training developers will

probably require extensive cross-referencing capabilities, as well as capability

for manipulating other non-numerical information. There are currently available

a variety of relational data base management systems (DBMS) that could be used to

satisfy such requirements. Which of these is selected will depend mainly on the

hardware and operating system on which the CTIS is to be implemented, the

flexibility required, and cost considerations. In addition to the DBMS, training

developers may need other kinds of software to support their efforts, little of

5

.~U % * -4 '
wich will probably be commercially available. For example, managers of training

development activities may need management support software to help then keep

track of product developments, exercise some quality control functions, produce

periodic reports, estimate work requirements, and so on. Training developers may

need software to support improved CFEAs, up-date task files, identify development

problems and to facilitate production. Extensive coordination and interviews

with the Service School collective training developers will be needed to determine firm requirements.

In the longer term the CTIS may be used to support senior Army executives,

combat and doctrine developers, the combat modeling community and others. Some

of these users are likely to require very succinct summaries of complex data

sets. Meeting such needs is likely to require development of extensive and

rather complicated data analysis software. CTIS developers will need to investigate needs for such analyses as the users become identified over the next few

years. Currently, however, development of sophisticated mathematical data

analysis software is probably not a pressing requirement.

INFORMATUDN ACCESS

Because of the Army emphasis on combined arms tactical training, developers

from all service schools will probably require some information from sister

institutions. Thus, the CTIS will need to be designed to facilitate rapid

retrieval of data base contents in the developer's specialty area and in other

areas which are closely related. This raises the question of which users will

have access to various sections of the CTIS core data base, and whether portions

of the data base will require restricted access. It seems likely, for example,

that those developers concerned with intelligence, emerging systems, etc., may,

for security reasons, require restricted access to parts of the data base. In

addition, the CTIS core data base should be protected from unauthorized changes,

and some software will be needed to insure that changes have the proper

endorsements before the system will allow either deletions of existing information, or insertion of new information. One partial solution to the data base

integrity issue would be to acquire a data base management system capable of

creating "virtual file working structures," thus obviating requirements for

temporary changes to core files during day-to-day operations. However, an

authorization code system will need to be developed in any case, as changes to

core files are inevitable.

A major CTIS development consideration involves the clearly sensitive

nature of training data. Potential users at all levels expressed deep concern

that training data might be misinterpreted or misused. One solution, of course,

is to construct the system to safeguard the confidentiality of all training data

by removing any unit identifiers from data transmitted to higher echelons, and by

aggregating data into categories (e.g., functional areas) not traceable to

specific units. One proposal consistent with current procedures in many units

is to remove unit identifiers from data which is transmitted more than two

echelons above the uwit of origin. Thus, brigade headquarters would be able to

identify specific companies and the division could identify units only down to

6

S.
- .... _. .. i .. ... ... :. : . .. . . . t 4.. < . .. . . _ - * . _ . : , . . -_' , .. . . : . - .

.- 77

the battalion level. The central idea is to leave unit identifiers intact where

they materially assist managers in setting training priorities and in allocating

resources.

At the same time CTIS developers should be aware of significant moves in the

Army toward greater training accountability. On the one hand, safeguarding data

confidentiality drastically reduces the potential for misuse while, on the other

hand, it virtually eliminates the use of such data for accountability purposes.

For a number of technical reasons as well as the possibility of a shift in

training accountability policy in future years, CTIS developers should insure

that the relevant software is designed in modular formats to facilitate changes.

MANPOWER AND PERSONNEL

Respondents consistently reported personnel and manpower problems. In

training development activities, new personnel must frequently be trained,

leading to quality control and production problems. Also, operational units are

rarely full strength, and turnover and turbulence tend to be severe. Therefore,

the OTIS should be designed with a view toward minimizing the system personnel

requirements, both in terms of numbers of people required for operations, and in

terms of the data processing experience and expertise needed for effective use.

The CTIS software design should incorporate as many "user friendly" functional

routines as possible including menu operations, "help" commands, and so on.

Also, user instructional courseware could be developed both to help users

interact more effectively with the system, and to provide orientation to and

basic instruction on the requirements of their jobs.

In addition, the feasibility of automating some functions which are currently executed manually should be examined. For example, much of the training

deve'..qment production work could be aided by the CTIS. Software to support

formatting, printing camera-ready copy, final editing, preparation of production

status reports, etc., could be developed for the CTIS. Since many commercial

printers have changed from manual to cmputer-supported production systems, it is

likely that some off-the-shelf software to support document production may be

available. Likewise, the CTIS could be used to reduce units' manpower requirements by providing software and hardware to organize and print ARTEP exercise

data collection forms, to reduce and analyze training data, and to prepare

reports based on those data.

As noted earlier, the respondents complained consistently that they were

often overburdened with paperwork. The CTIS developers could examine the

feasibility of partially automating the field data collection process through the

use of hand-held data collection devices. One approach is to load such devices

with training objectives from a CTIS terminal interface, use the devices for

field data collection, and then off-load the information from the devices back

onto the CTIS terminal for analysis and report preparation. Such an extended

CTIS terminal could significantly reduce units' manpower requirements for handling training data. This is a long-term effort and exploratory work would need

to be undertaken in the near-term in order to realize the benefits 2-3 years from

now.

47

4,.',
7 - 7 1 

  l 4_ 

v . 7 

7

High rates of turbulence indicate that feedback must be immediate to be of

any use at squad, platoon, and company levels. At these echelons feedback should

be very specific, sub-task oriented, and presented in a readily understandable

format. In order to facilitate the incorporation of ARTEP feedback into

immediate planning for corrective training, CTIS should be integrated with

whatever system is developed to assist battalion training management.

DEVELOPMENT ISSUES

There are a variety of issues which should be addressed early in the CTIS

development. These are, for the mst part, beyond the scope of the user group

surveys, but are nevertheless central to developing a workable CTIS design

cor ept.

a. System Development. An integrated plan for CTIS development is central

to insuring that the operational system will meet user needs, and that it is

developed as cost-effectively as possible. Because of the broad scope of the

effort the current information suggests that the system should be developed and

implemented in phases. Initial efforts would focus on developing a system for

training developers, while later phases would focus on a CTIS for operational

units, senior executives, combat developers, doctrine developers and probably

other users not yet identified.

A considerable amount of software will be needed to implement the CTIS.

Most often, software developed for one computer system must be at least modified,

and occasionally rewritten entirely, in order to run on other systems. It is

likely this will be the case with the initial system developed for the Burroughs

and a later one developed for the AMDAHL. Because such software developments are

often time consuming and expensive, CTIS developers need to consider carefully

the effects of producing software not readily transferable to the "end-product"

system.

The initial tryout of the CTIS should, of course, be made with a limited

prototype. Careful consideration will need to be made of what portions of the

whole software package for each phase most require extensive user testing. In

addition, it may be valuable to develop some limited programs to support the user

tests. For example some simple programs which count the number of times each

display is retrieved, the duration of that display is held on the screen, the

sequences in which displays are retrieved, etc. could greatly assist in determining whether the system is being used as intended, and what kinds of changes would

be advisable in the final version.

At some point, the CTIS will probably be integrated with the other ATIS

subsystems. The nature of the interfaces and the requirements for transmitting

data across the "subsystem boundaries" have not yet been determined. While the

independence of the kTIS subsystems permits relatively rapid development of each

one, this approach raises fundamental questions regarding compatibility among

the subsystem. If developments for the ATIS subsystems proceed without unified

technical management, the ultimate integration could prove extremely awkward and

costly.

8
b. Configuration. It is expected that the system configuration will vary

somewhat with the various users. The number of terminals required, graphics

needs, maximum response times, and so on should be determined as early as

possible. In addition, the location or locations of system data base(s) can be

*: expected to have a major impact on requirements. For example, a centralized data

*: base would allow the colocation of many peripherals, (e.g., the disc-drive

*systems which tend to be expensive). Distributed data bases, on the other hand,

require distributed data processing equipment and would probably entail a

sbstantial amount of hardware duplication. The trade-offs between system costs,

response time requirements, access restrictions, etc., need to be thoroughly

examined in order to produce a viable system concept.

CONCLUSIDNS

There appears to be substantial support for the CTIS concept. The respon- dents, however, did express some reservations about the requirements for supporting suchn a system. CTIS developers will need to demonstrate to the users not only

the benefits to be derived from implementing the system, but also that the costs

(manpower, funds, etc.) born by the users are not excessive.

Respondents from operational units found the sample printouts to be potentially quite useful though they differed considerably in their ideas, what types

of data would be most useful, how best to organize them, and what levels of

aggregation would be most desirable. Training development respondents, on the

other hand, did not find the examples particularly useful, mainly because they

require information on fielded ARTEP materials rather than data on unit performance. Nevertheless, one might suggest that some performance-related data could

be analyzed in ways which might help in the identification of problems with

training objectives. This topic should be explored further.

Current plans call for the CTIS to be developed in three phases, each

servicing the needs of one primary user group. Phase I will concentrate on ARTEP

developers; Phase II, operational units; and Phase III, senior Army leaders,

combat and doctrine developers, DA staff elements, etc. The plan for phased CTIS

development was formulated after the survey was completed, and has not been a

focus of the present report. The CTIS Phase I will, however, be the principal

topic of the preliminary design concept report, currently scheduled to be

completed later this year.

9

**** ***,-.*.-*-* ,*.

** * '.. ~ ~ - -"~.- 
REFERENCES

Gmy, T., & Roberts-G ray, C. A Guide to Implementation of Training Products.

Perceptronis Report, October 1981.

Training Evaluation and Feedback Term. ARTEP-Related Information Needs:

Training Managers. ARI-POM Working Paper, August 1982.

Training Evaluation and Feedback Team. ARTEP-Related Information Needs:

ARTEP Developers. ARI-POM Working Paper, September 1982.

Training Evaluation and Feedback Term. ARTEP-Relation Information Needs:

Executives. ARI-POG Working Paper, February 1983.

Training Evaluation and Feedback Term. Collective Training Module Concept.

ARI-POK Working Paper, November 1982.

* 10

4%

*lO

4,. ----.. .- . . . -, , .; - .. . . ' .. . ... . .. . ., . . . . . . . . . . . . -.- . .. . , . - . , -. . ., , , ,