OTK ritt uxD

^

ARI Research Note 87-15

en

NATIONAL TRAINING CENTER LESSONS LEARNED: DATA REQUIREMENTS

Carol A. Johnson

ARI Field Unit at Presidio of Monterey, California

oo

5

i

O

<

TRAINING RESEARCH LABORATORY

Jack H. Hiller, Director

DTK2

ZLBCTE»^

MAY 1 3 198711

U. S. Army

Research Institute for the Behavioral and Social Sciences

February 1987

Approved for public release; distribution unlimited.
U. S. ARMY RESEARCH INSTITUTE

FOR THE BEHAVIORAL AND SOCIAL SCIENCES

A Field Operating Agency under the Jurisdiction of the

Deputy Chief of Staff for Personnel

EDGAR M. JOHNSON

Technical Director

WM. DARRYL HENDERSON

COL, IN

Commanding

Technical review by

Kent E. Harrison, Col. USA

Center for Army Lessons Learned

Combined Arms Training Activity

Jesse Orlansky

Institute for Defense Analysis

Douglas L. Rachford

U.S. Army Research Institute

Accession For_

NTIS GKA&I

DTIC TAB

Unannounced

JustificationDiitrlbutlon/^

¥

a

Availability Codea_

Avail and/or

IDDlst 1st I Special

Thii rtport hat bun cltarad for raluM to tht Dafamt Tachnical Information Cantar (DTIC) It hai baan givan no

othar primary distribution and will ba availabla to raquttton only through DTIC or othar rafarancc Mrvicas such

at tht National Tachnical Information Sarvict (NTISI. Tht viawi, opiniont, and/or findings oontainad in thii

raport art tho»t of tht author(i) and should not ba conttruad a« an official Dtpartmant of tht Army position,

policy, or dacision, unlass so dasignatad by othar official documantation.
UNCLASSIFIED

SECURITY CLASSIFICATION OF THIS PAGE (Whan Data Enfrmd)

REPORT DOCUMENTATION PAGE READ INSTRUCTIONS

BEFORE COMPLETING FORM

1. REPORT NUMBER

ARI Research Note 87-15

2. GOVT ACCESSION NO 3. RECIPIENT'S CAfALOGJUlM

4. TITLE fand Subtltl*.) 5. TYPE OF REPORT & PERIOD COVERED

National Trainin<] Center Lessons Learned:

Data Requirements

Interim Report

1931 - 1986

6. PERFORMING ORG. REPORT NUMBER

7. AUTHORC«;

Carol A. Johnson

8. CONTRACT OR GRANT NUMBERf«;

9 PERFORMING ORGANIZATION NAME AND ADDRESS

Army Research Institute Field Unit at Presidio of

Monterey, California P.O. Box 5787

Presidio of Monterey, CA 93944-5011

10. PROGRAM ELEMENT, PROJECT, TASK

AREA ft WORK UNIT NUMBERS

2Q263743A794

511

II. CONTROLLING OFFICE NAME AND ADDRESS

U.S. Army Research Institute for the Behavioral

and Social Sciences, 5001 Eisenhower Avenue,

Alexandria, VA 22333-5600

12. REPORT DATE

February 1987

IS. NUMBER OF PAGES

18

14. MONITORING AGENCY NAME ft ADDRESSf" <""»an( fro« ControUtnt Olllem) IS. SECURITY CLASS, (ol thlm nport)

Unclassified

ISa. DECLASSIFICATION/DOWNCRADING

SCHEDULE

1«. DISTRIBUTION STATEMENT (ol (h/a Rmport)

Approved for public release; distribution unlimited.

17. DISTRIBUTION STATEMENT (of th» abatracl mxltrad In Block 30, II dlllarmxt fron Roport)

It. SUPPLEMENTARY NOTES

It. KEY WORDS (Centlnuo on ravoraa a/<fa II nacaaaary and Idantlly by block numbarj

National Training Center (NIC)

Lessons Learned

Performance Measurement

20. ABSTRACT fCaofteia aa rovaraa «Mh ft mcaaoaqr and Idonllty by block nuaibarj

The Army Research Institute and the Combined Arms Training Activity (CATA) held

tho first of a planned series of working group meetings to improve National

Training Center (NTC) data on May 13 and 14, 1986. This research note summarizes

the results of that meeting, and lists previously identified user needs as well.

A comparison of user needs to the data currently collected shows that the data

are insufficient to answer many important research questions, and the note concludes with recommendations for improving data collection techniques, quality,

[and types.

DO , '22^» M/S eWTJOM OF • NOV 65 IS OBSOLETE UNCLASS I FI ED

SECURITY CLASSIFICATION OF THIS PAGE (Whan Data En(arad)
NATIONAL TRAINING CENTER LESSONS LEARNED: DATA REQUIREMENTS

CONTENTS

Page

INTRODUCTION 1

USER NEEDS 2

Definition of user 2

Initial Working Group Meeting 2

Identification of Needs 3

CURRENT SYSTEM 5

Instrumentation System 5

Take Home Packages 8

Communication Tapes 8

Data Availability 8

DISCREPANCY BETWEEN USER NEEDS AND CURRENT SYSTEM 9

HEQUIHEMENTS 10

Procedural 10

Hardware 10

Software 11

RECOMMENDATIONS 12

CONCLUSIONS 13

REFERENCES 15

ill
NOTE: Appendixes are too lengthy to be included in this document. Titles are

listed below, and may be requested through the Presidio of Monterey Field Unit.

APPENDIX A.

APPENDIX B.

APPENDIX C.

APPENDIX D.

APPENDIX E.

APPENDIX F.

APPENDIX G.

APPENDIX H.

APPENDIX I.

APPENDIX J.

APPENDIX K.

APPENDIX L.

APPENDIX M.

APPENDIX N.

APPENDIX 0.

APPENDIX P.

APPENDIX Q.

APPENDIX R.

APPENDIX S.

APPENDIX T.

APPENDIX U.

APPENDIX V.

ARI NTC-BASED RESEARCH PROGRAM OVERVIEW

LIST OF ATTENDEES WORKING GROUP TO IMPROVE THE QUALITY AND

UTILITY OF NTC DATA

LONG TERM/LOWER PRIORITY NEEDS FROM INITIAL WORKING GROUP MEETING

RESPONSES TO CAC REQMFST FOR INPUTS FOR "NTC DATA LIBRARY"

CAC NTC COMBAT MISSION ANALYSIS: 1500 QUESTIONS TO BE ANSWERED

FOR EACH UNIT

QUESTIONS FROM FIRING DATA WORKSHOP

POTENTIAL RESEARCH ISSUES

ATTENDEE ROSTER COMBINED ARMS CENTER LESSONS LEARNED WORKSHOP

ASSESSMENT PLANS FOR COMBINED ARMY ASSESSMENT TEAMS (CAATs)

U. S. ARMY NATIONAL TRAINING CENTER INSTRUMENTATION SYSTEM

DIGITIZED HISTORY

INGRES TABLE NAMES AND DEFINITIONS

DIGITAL DATA ANALYSIS AND DATA BASE DEVELOPMENT

TAKE HOME PACKAGE ANALYSIS

RESEARCH ON COMPANY TAKE HOME PACKAGES AND AFTER ACTION REVIEW

TAPES

AN EVALUATION OF THE ACOUSTIC QUALITY AND RESEARCH POTENTIAL OF

COMMUNICATIONS RECORDINGS FROM THE NATIONAL TRAINING CENTER

NTC DATA ACQUISITION AND STORAGE

OBSERVER/CONTROLLER PROCEDURES

STANDARDIZATION OF DATA COLLECTION PROCEDURES WITHIN THE CORE

INSTRUMENTATION SYSTEM (CIS)

MEASURING AND INTERPRETING UNIT PERFORMANCE AT THE NTC

ELECTRONIC CLIPBOARD

DETERMINANTS OF EFFECTIVE PERFORMANCE AT THE NATIONAL TRAINING

CENTER (NTC)

APPENDIX W. NATIONAL TRAINING CENTER (NTC) INSTRUMENTATION VALIDATION MEETING
Introduction

The National Training Center (NTC) was established in accordance with Army

Regulation (AR) 350-50 which defines NTC missions and responsibilities. The

objeciives of the NTC are to:

• Provide a facility where heavy battalion task forces, controlling

brigade headquarters, and supporting units can umlergo essential

combined arms training that cannot be accomplished at home stations

due to physical limitations and prohibitive cost of providing a

realistic training environment.

• Gather information to help improve doctrine, tactics, training

system, equipment and procedures. This information also assists the

Army in relating resources to readiness.

These purposes were reiterated in the Chief of Staff's NTC Policy

Statement in September, 1984, in which he emphasized the need and the

challenge to continue the tough, successful training for Battalion Task

Forces, while finding ways to expand NTC's capabilities to promote

innovation. One way to meet this challenge, he stated, was to "develop the

NTC range instrumentation and associated long-range plans to permit detailed

analysis and feedback. ... We must all work together to harness the NTC's

full potential and spread the NTC experience throughout the total Army."

The NTC has been extremely successful in meeting its training goals.

Units have been exposed to the most realistic, intense training environment

ever developed. Faced with a highly trained and motivated opposing force

(OPFOR), continuous training, and highly dedicated observer/controllers (OCs),

the training experience which units receive is unparalleled.

The development of Lessons Learned has been a slower process. According

to AR 350-50, "The training environment will be paramount at the NTC."

Primary emphasis has, therefore, been placed on the training mission of the

NTC. In August, 1985, the Center for Army Lessons Learned (CALL) was

established within the Combined Arms Training Activity (CATA). CALL'S mission

is to provide combat relevant Lessons Learned to the total Army. The Army

Research Institute (ARI) has a letter of agreement with CATA which includes

tasks directly relevant to this mission. Specifically, ARI has agreed to:

• Develop methodology for the use of NTC findings in doctrine,

organization, equipment, and training development.

• Develop methods to improve the utility and quality of NTC data.

On May 13-1^, 1986, the first meeting of a planned series of working

groups to improve the quality and utility of NTC data was held at ARI,

Presidio of Monterey. This report summarizes the results of that meeting. In

addition, this report includes a compilation of users' needs of NTC data which

have been gathered over a period of several years. The current status of the

NTC data collection system is described and its capability is compared to

users' needs. Requirements for meeting discrepancies are outlined and

recommendations for improving the data collection system are made.
User Needs

Definition of User

The agencies or groups interested in utilizing NTC data sources fall into

two categories. There are primary users of the raw data who have the

capability of analyzing that data and putting it into a format which secondary

users can interpret and utilize for decision making.

Among the primary users are CATA, the research community (e.g., ARI,

Arroyo Center), and the modeling and gaming community (e.g., TRADOC Analysis

Center). The analysts at the NTC are also primary users who use the data for

their After Action Reviews (AARs) and Take Home Packages (THPs).

The primary users put the data into a format which is usable by others.

These include the schools, computer assisted battle simulations, trainers at

home station, and policy makers at Department of the Army, Forces Command,

Army Materiel Command, and Training and Doctrine Command.

Initial Working Group Meeting

Purpose. With the establishment of the Lessons Learned program, the

applications of NTC data and the number of agencies and individuals involved

in data collection and analysis have Increased. Each agency has a partial

view of the data collection and processing system. ARI initiated the

establishment of a working group to, first, share information on current

activities and problems or questions. This exchange of information was

intended to provide a common broad-base description of issues.

The second purpose was to develop recommendations for near-term and

longer-term approaches to improving the quality and quantity of information

from the NTC. With these goals in mind, CATA invited the participants to

attend the May 13-11*, 1986, meeting held by ARI, at its Presidio of Monterey

Field Unit. Appendix A gives an overview of ARI's NTC Research Program, as

briefed at this meeting. ARI has resources in all of its' three laboratories

working on NTC-related projects and a number of research products have been

and are being developed.

Participants. The following agencies were represented at the meeting (see

Appendix B for list of individuals in attendance):

Combined A.-my Training Activity

TRADOC Analysis Center

AMEX Corporation

Army Training Support Center

Arroyo Center

Army Training Board

Combat Development Experimentation Command

United States Army Infantry Center

The BDM Corporation
Identification of Needs

After a series of briefings (which are integrated into this report and

included as appendixes), the working group generated a list of data of

interest to them. This list was divided into near and longer term based on

the difficulty of obtaining the data. The near-term data were prioritized

into three categories, based on the value of the information and the amount of

work required for NTC personnel to collect and provide it. Near-term, high

priority data needs are included in the Recommendations section of this

report. Longer-term or lower-priority needs are at Appendix C.

This meeting was the most recent of many efforts to collect similar

information. In 1982, the Combined Arras Center (CAC) solicited input on

requirements for an NTC Data Base Library. Most of the responses gave

detailed lists of the type of data various agencies would be interested in.

Responses are at Appendix D.

The information at Appendix D was distilled hy CAC through an NTC Combat

Mission Analysis. They developed 1500 questions to be answered for each unit

(see Appendix E).

In 1985, an NTC Firing Data Workshop was held to examine needs which could

be met with NTC data. Notes on this workshop are at Appendix F. ARI's

resident contractor at the Presidio of Monterey Field Unit, BDM, also

generated a list of potential research issues which might be addressed. These

are categorized by Operating System and are at Appendix G.

In November, 1985, CATA hosted a NTC Lessons Learned Workshop to extract

and disseminate lessons learned and increase awareness regarding the agencies

available to collect data, how they collect it, and what they produce. The

roster of those in attendance is at Appendix H. Most recently, the NTC was

used to collect data for testimony to Congress on the Bradley Fighting

Vehicle. The type of data which was collected and which would also be of

value for other training applications is at Appendix I.

A review of these appendixes makes it clear that there are a magnitude of

potential uses of NTC data. The following examples of questions which could

potentially be answered in each of the categories specified in AR 350-50

demonstrate the breadth of relevant issues.

Doctrine.

• What problems are presented by the J-series table of organization and

equipment (T04E), particularly with dismounted infantry?

• Where should Command Groups be positioned on the battlefield?

• How effective is air defense early warning doctrine?

• Does current doctrine adequately show units how to attack Soviet

strongpoint-type defense?
Tactics.

What is the contribution of Infantry anti-tank weapon systems to the

close-in anti-armor battle?

To what extent is combat power concentrated in conducting a

deliberate attack?

How can mortars be effectively utilized in battle?

Was the amount of terrain offering cover and concealment considered?

raining system.

How can the effectiveness of tank and tube-launched, opticallytracked, wire-guided missile system (TOW) crews be improved?

To what extent do inaccurate calls for fire cause direct fire

fratricides?

To what extent do task forces employ counterattacks in the defense?

Do engineers employ minefield equipment and techniques effectively in

obstacle construction?

Equipment.

How does the Bradley Fighting vehicle compare in performance to the

M-113?

Are current battalion communications equipment packages capable of

surviving on the battlefield?

What was the average down time for radios turned in for maintenance?

What is the contribution of VIPER and DRAGON to the anti-armor fight?

Procedures.

HoiV effective are cross-attached companies as compared to pure

companies?

How can the fire support officer (FSO) best synchronize fires?

What is the most effective procedure for passing early warning

information?

Do units react in accordance with the mission-oriented protection

posture (MOPP) level designated by the commander?

Many of these issues could potentially be answered with instrumented data,

e.g., position location and firing data. Others would require observational

data or information from the communication nets. It is, therefore, important

to look at the current data collection system to determine its capability.
Current System

There are three primary sources of NTC data available to potential

users: the instrumentation system, Take Home Packages (THPs), and 40-channel

communication tapes recorded during rotations. Policies and procedures at the

NTC are in a continual state of evolution, so this report will discuss the

capability of the current system, as known. Each of the three sources of NTC

data will be described and discussed. The actual use of the instrumentation

system differs from its design and potential utility, so these two topics will

be discussed separately.

Instrumentation System

Appendix J contains the briefing materials presented at the initial

working group meeting by AMEX, Inc., the developers of the instrumentation

system. Two subsystems are principally involved in the digital data

collection. The Range Data Measurement System (RDMS) collects raw field

data. Much of this data is transmitted to the Core Instrumentation Subsystem

(CIS), where data can also be input manually.

RDMS data. The RDMS collects data on trigger pull (fire event), laser

illumination (pairing), microphone key pressing, and the raw data from which

position/location is derived for instrumented players. During Live Fire

Exercises, data on target status (up, down, hit) is also collected. The

capacity of the system is limited so there are a large number of uninstrumented players for which position and event data cannot be collected. In

addition, none of the dismounted personnel are instrumented. For example, in

a sample of ten defensive missions, an average of 27% of the players were not

instrumented. The percent uninstrumented ranged from a low of 1751 to a high

of 38*.

According to Briscoe (1986):

RDMS data are as accurate as the reliability of the collecting/

transmitting hardware allows. The complexity of NTC field instrumentation

is conducive to a wide variety of error-producing conditions, including:

(1) Spurious radio frequency (RF) transmissions, leading to

erroneous events;

(2) "Noisy" laser sensors which generate spurious and/or inaccurate

pairing events;

(3) Hardware/electronic player instrumentation problems leading to

loss or duplication of valid events, and the generation of invalid events;

and

(4) Coverage problems resulting in the loss of track of instrumented

vehicles and the corresponding loss of position/location and event

retrieval capability.
Even in the case of perfect hardware performance, it is possible for

errors to be introduced by faulty initialization. If the proper B-unit code

is not associated with the right player identification, incoming events will

be improperly assigned or may be deleted as invalid. Such problems can

quickly lead to a serious loss of data integrity.

MILES. The force-on-force exercises use laser-based, engagement

simulation technology. MILES (Multiple Integrated Laser Engagement System) is

used on all principal weapons and results are assessed when a weapon fires and

the MILES laser hits a target. While this system provides a high degree of

realism to combat training, it is not without some deficiencies. For example,

the laser does not penetrate smoke and a target which may be sighted with a

thermal sight could not be "killed" with MILES.

Another problem is that in order for a TOW to kill a target, it has to

track the target for 15 seconds. This effectively simulates the requirements

for targets at long ranges, but not short ranges. Therefore, a TOW which

tracked a target for less than 15 seconds at short range would not be counted

as a kill, even though in actual combat the length of time may have been

sufficient. MILES also does not compensate for differences in requirements to

range to target or to employ a lead angle by actual weapon systems. The laser

technology instantaneously hits the target whereas actual ammunition would

take some period of time.

CIS data. The CIS logs data in real-time to provide the primary archival

source of NTC data. It also supports pre-exercise initialization in which

player information, control measures, task force organization, live fire

scenarios, and pre-planned artillery is entered. The CIS also pairs firers

with targets and calculates statistical measures.

According to Briscoc (1986):

It should be noted that many of the elements logged from the CIS are

manually input data elements. Reliability of manual data depends upon the

accuracy of the personnel entering the data and the verification

procedures that are employed, such as proofreading and consistency checks.

Digitized history database development. Appendix K shows the list of

statistical displays that the software is capable of supporting. Fobes (1984)

and Science Applications, Inc. documentation give detailed descriptions of

each of these displays. These data are displayed for a mission on a VT125

Graphics Monitor when a specific history tape is loaded. This means that data

in these tables can only be collected on a mission by mission basis. The

primary users of these displays are the analysts who prepare the AARs.

Science Applications, Inc. developed software for ARI intended to

translate the display data into a format which would facilitate analysis with

INGRES, a relational data base management system. Sixty-one INGRES tables are

created for each translated history segment from an NTC rotation, A

description of each of these tables is at Appendix L,

Instrumented database status. Software development to translate mission

data into INGRES tables la still on-going. Among the remaining problems, for

example, are the following. Position/location data are critical to most
research questions. However, the software takes a tremendous amount of time

(approximately one hour of position/location data takes an hour of computer

processing time) to process these data, so until new software is developed to

more efficiently handle the data, translation of position/location data will

be limited. Also, the current process creates an INGRES database for each

mission segment. Research clearly requires being able to aggregate data

across missions, but this is not easily done. The current database is

"inconvenient to use, is massively redundant, and contains superfluous data"

(Briscoe, 1986).

In addition, there is a great deal of missing data which are of potential

research value. On a sample of 71 missions, the number of empty tables ranged

from 13 to 44 of the 61 tables. Only an average of 35-36 tables contained any

data. Even those that contain data may be incomplete. For example, the table

SEGSUMRATINO is supposed to contain segment summary ratings provided by key

training operations group personnel (i.e., TAFO, EMCO, and Chief OC) at the

end of the segment. The collective judgement of this group on the performance

of the battr.lion is expressed in numbers from 0-9, using the following

scheme: 0 = no observation; 1 = very poor; 3 = poor performance; 5 = nominal

expected performance; 7 = good performance; and 9 = excellent performance.

These ratings are to be made on Overall Exercise Effectiveness; Target

Acquisition; Maneuver; Fire; Communication; Command and Control, and Logistics

and Administration. These data, although subjective, have potential research

value. However, this table contains only the date and time the mission

ended. All ratings are missing in the sample of missions which were examined.

There are two sources of data for the tables: the field instrumentation

system and operator-entered inputs. The tables which require manually entered

data are empty or sparse. Thirty-three of the 61 tables require some or all

data to be manually entered. It is clear, therefore, that the current system

could support the collection of a great deal more data than is currently

gathered. The utility of the data which is currently being archived is only

marginally useful for research on NTC performance, but the capability

obviously exists for significant improvement.

One problem which has impeded the database development process is a lack

of documentation of the SAI software. It would be advantageous, for example,

to have the format of the CIS logs. Without documentation, it is impossible

to know if the CIS tape contains everything on the ROMS tape.

ARI has recently developed software that allows for reading the RDMS log

tapes. The RDMS might contain additional information of importance for offline analyses, e.g., the MILES code for the firing weapon for each vehicle

which has been killed. This information might also be on the CIS tape, but

this has not yet been determined. The consistency between the CIS log and the

RDMS log is currently being examined. It may be possible to extract

additional relevant information from the RDMS tapes for research purposes.

Appendix M contains working group briefing slides on the status of the digital

data analysis and data base development.
Take Home Packages

Appendix N contains briefing slides from the initial working group meeting

which describe an analysis of the Maneuver Performance Trends section of 26

THFs from 1984 and 1985. The THPs potentially serve two purposes: to guide

liome station training and for development of Lessons Learned across

battalions.

The results of this analysis indicate that because the format of the THPs

is not related to the ARTEP, it may be difficult to translate the Performance

Trends into training guidance for home station. Due to the lack of

consistency within and among OC training teams regarding what they observe,

the data is also of little value for deriving Lessons Learned. Some trends

were identified, but of the 113 issues, relatively few were commented upon

consistently enough to draw conclusions.

The battalion THPs also contain After Action Review (AAR) summaries for

each mission. These data were not analyzed c :d are potentially of value.

Appendix 0 contains briefing slides describing an analysis of this section of

the company THPs for leadership Lessons Learned. Again, a great deal of

variation was found in the amount, quality, and format of information. The

key to THP data being of any value for Lessons Learned is standardization of

observations.

Communication Tapes

A recent analysis of the research potential of the 40-channel communication tapes found that the tapes are a rich source of detail and essential

contextual information (see Appendix P). They can be synchronized with the

workstation display of the battle which greatly increases understanding of

what is taking place. The OC control nets probably provide the best source of

data.

The major problem with using the tapes for research purposes is that it

is, currently, a very labor-intensive process considering that, with 40

communication channels, one rotation is the equivalent of 1280 days of

recordings. In addition, there is a problem with override of channels.

Electronic methods of processing the information would not be able to

determine whether a transmission was a legitimate one or if it was coming

through from another net.

There is a large discrepancy between the number of transmissions actually

counued in listening to the tapes and the number of transmissions reported in

the THPs. The THPs appear to report far fewer transmissions than were found

on the tapes. This makes the THP communication data misleading and

essentially worthless. This disparity needs to be eliminated.

Data Availability

Appendix Q contains the briefing material describing current and planned

NTC data acquisition and storage. ARI currently has digital tapes from

August, 1983 to the present; most of the THPs from January, 1982 to January,

8
1986; some operation plans; and a few AAR video tapes and unit After Action

Reports.

Coordination has recently been accomplished with the NIC Operations Group

to ensure routine acquisition of materials, including communication tapes,

exercise control logs, and operation orders by ARI. This clearly will result

in a requirement for storage space. To reduce this requirement, consideration

is being given to transmitting the THPs on floppy disks and to compressing the

communication tapes.

Discrepancy Between User Needs and Current System

The existence and the magnitude of potential uses of data for Lessons

Learned from the NTC was clearly demonstrated in the initial working group

meeting, as well as by Appendixes C through I of this report. The questions

raised by these sources cannot be answered with the data currently archived.

The reasons the data are insufficient to answer such questions include:

• The lack of standardization regarding observations made/recorded by

the OCs. Standardization allows quantification of information and

statistical manipulation and permits interpretation of results.

• The large amount of data which must be manually entered which is

missing. Much of these data would be valuable for research and

analysis.

• Limitations on the number of players which can be instrumented. The

current system supports 500 instrumented players. In reality,

usually only 400 to 425 players are actually instrumented. This

total must include not only battalion task forces, but 0PF0R, OCs,

and brigade players, so some percentage of players are not

instrumented and, therefore, not tracked. To the extent that there

is any systematic rationale behind decisions on who to instrument,

such as the fact that no dismounted personnel are ever instrumented,

the data which are collected may not be representative. There is

currently a Statement of Work for a contract to increase the system

to 1000 players.

• Coverage problems. Even if a vehicle is instrumented, the terrain at

the NTC can mask it so that its current position/location cannot be

tracked. This, again, results in missing data. If the missing data

were random, it might not be a major problem. However, if, for

example, there was a research question regarding the performance of

tanks in the defense and the better performing tanks were more

effectively hidden, any data collected on the less-well-hidden tanks

would not be representative. In a sample of ten defensive missions,

for example, there were a total of 327 instrumented tanks. Of these,

12 percent were never tracked and an additional 6 percent were only

partially tracked.
• Electronic hardware problems leading to spurious, inaccurate, or

duplication of events,

• The fact that a research data base does not exist. The current

structure of the software requires that data be accessed mission by

mission rather than across missions. Within the next six months to

one year, the data base will be redesigned. In addition, only a

small part of the information required for analysis and interpretation is in digital form. Merging of data from multiple sources is

manpower and time intensive.

The above issues differ in terms of the degree to which they can be

resolved. The next section of this report discusses requirements for

improving the quality of data from the NTC to support the development of

Lessons Learned.

Requirements

Procedural

The achievement of inter-rater reliability among the OCs is critical for

development of Lessons Learned. This is an evolving process and needs to

involve training programs. The system which is now in place was designed for

training and has been very effective for that purpose. However, a system

which will support both training and generation of Lessons Learned is

necessary. Appendix R contains briefing materials on OC procedures. It is

clear that the OCs are extremely busy. OCs are very competent and dedicated,

but they -^ed tools to assist them in making systematic observations, so that

they can 'work smarter, not harder." ARI has begun to assiat in this process

within the CIS. Training materials and SOPs have been and are being developed

for their use (see Appendix S).

It is clearly necessary to determine which aspects of performance need to

be systematically observed and measured. Careful analysis is needed which

will consider priorities for information, as well as what is desirable. The

ground rules regarding what needs to be observed have to be standardized and

OCs need to be re-calibrated after they have been in the field for a while.

ARI is currently working to design and develop a system for measuring and

interpreting the effectiveness of unit performance at the NTC. As part of

this effort, an NTC-specific ARTEP and NTC observer guides are being

developed. Briefing materials describing this effort are at Appendix T.

Hardware

One tool which would be of great utility to the OCs is the Electronic

Clipboard. Briefing materials on this are at Appendix U. This piece of

equipment would allow for downloading of performance observation guides into

the hand-held clipboard. Data on performance would be input into the

clipboard by the OCs in the field. The digital data in the clipboard could

then be radioed to the CIS or uploaded by an RS232 connector directly into the

10
CIS. This innovation will increase the reliability of observations by

providing standard observer guidance cues and automation for the recording and

processing of performance ratings. This will assist the OCs in doing their

jobs rather than adding to their responsibilities, and provide improved

information for AARs and THPr.,, and the research data base.

The Electronic Clipboard is also one of the components of the Integrated

Training Management System which is being developed in the 9th Infantry

Division. This division-level system is being developed to improve the

ability of units to prioritize, resource, conduct, and evaluate training

through management automation. It takes into account training priorities set

by leaders; prerequisite relationships among individual and collective tasks;

availability of resources; and unit training needs. A prototype system has

already been developed by ARI and successfully tested in a field artillery

battalion of the 9 ID (MTZ). This system has great potential for integrating

NTC experiences with home station training.

Related to this is a planned ARI contract on the determinants of effective

performance at the NTC. The purpose of this research will be to improve unit

training through better integration between NTC and home station and to

determine what unit training management practices, organizational processes,

leadership behavior, and individual abilities have an impact on performance at

the NTC. Briefing materials describing this effort are at Appendix V.

Software

As previously noted, there is currently a Statement of Work which solicits

proposals to upgrade the current NTC software. Appendix W contains briefing

slides and a memorandum on a meeting to validate the requirements of this

Statement of Work.

The upgraded system will be able to track 1000 players instead of 500. It

will incorporate new nucler, biological, chemical (NBC) control measures. Air

Defense, Helicopters, and Air Force jet aircraft will be instrumented with

MILES and will be able to be tracked and displayed. It will automatically

collect tactical fire direction system (TACFIRE) messages and have increased

fire mission storage.

A major improvement is the ability to assign each player an identification

code. Currently, pairings are established by time coincidence and player

identifications are made based on position at the time a firing event

occurs. The new system provides identification of the firing vehicle and

weapon fire for each MILES hit/near miss/kill event. It will provide

graphical display of individual weapons systems/player unit and hit/near

miss/kill data. Obtaining player and crew identification was identified by

the working group as a medium priority, near-term data collection need. It

would appear that this solicitation will supply this information.

The SOW also provides delivery of the NTC workstations to home station

units. This will give units the capability to replay NTC exercises. An

upgrade of the video and audio capabilities is also provided for. These

improvements are to be implemented incrementally, but should be completed two

years from award of contract.
Recommendations

The recommendations in this section were developed from the working group

meeting held in May, 1986. The most immediate requirement is that ARI be

provided detailed information on current procedures used at the NTC. The

training environment at the NTC is in a state of constant evolution. ARI will

have a researcher in residence at the NTC in July, 1986, which will facilitate

tracking of procedures.

It is also important that a concerted effort be made to get information

from units regarding their needs in the AARs and THPs. This will allow a

better integration of training and Lessons Learned needs. It would be of

value to know if and how units use the information provided to them and what

additional data would be of interest. This research effort is now in the

planning stage by ARI.

It is important to ensure that the NTC retains the tapes of the 40-channel

communication networks recorded for each rotation. The cost of the tapes has

led to reusing them after approximately six months. One of the briefings in

the working group meeting described an evaluation of the research potential of

the tapes and recommended that they be retained. The consensus of the group

was that the cost of the tapes within the context of the seven million dollar

cost of the rotation was relatively minor. Recent coordination with the NTC

has supported this action.

There is currently information being collected at the NTC which is not

being provided to ARI. Evidently, information on indirect fire currently

exists in a paper form, but is not retained after a rotation. Also, .ivision,

brigade, battalion, and OPFOR operations orders and graphics for each mission

exist, but are not collected or retained. Army Material Systems Analysis

Agency (AMSAA) currently collects data on Combat Service Support and

operational readiness to provide Lessons Learned to the Materiel Readiness

Support Agency. This data should be relatively easy to obtain to assess its

utility for other analyses.

The current software has the capacity to collect free format messages with

a maximum length of three lines of text of 80 characters per line. If the OCs

in the field would have the CIS analysts routinely enter information with

important research potential, it would be a relatively simple method of

collecting data for Lessons Learned. Of high priority to the working group

was to use this capability to collect information on the number or percent of

friendly and enemy kills, by weapon system, after every battle. The OCs know

this information, but it is not collected in a systematic fashion and was of

great interest to all of the meeting participants.

Also of high priority for collection with the free format message

capability was entering and time tagging verbal fragmentary orders (FRAGOs)

during the battle. Also of interest, were free format messages on:

• smoke conditions and major weather changes;

• dismounted infantry positions and casualties held some interest but

were regarded as a difficult data collection burden for the OCs;

i ?
• start and end times of the battle would be useful;

• the reasons for OC kills and resurrections are very important.

This method could also be used to collect information on informal,

spontaneous organizations that emerge on the battlefield. However, this was

of bottom priority for the group. It is important to emphasize that if

procedures are implemented to collect any or all of this data that it must be

done in a consistent fashion. Partial or inconsistent use of free format

messages would make the data uninterpretable.

Another useful set of data which could be supported by the current

software is the header data for each mission. This defines the scenario

number; intensity of fire support, artillery, and mortars; available planning

time; nuclear, biological, and smoke conditions; engineer, air defense, and

electronic warfare activities for both friendly and opposing forces. It also

tells visibility conditions and the number of times the unit has been through

the scenario. This data must be manually entered by the CIS analyst and is

not usually done. The working group determined that these data would be of

value, but the group was concerned about increasing the workload of the CIS

analysts.

Conclusions

The issue of the value of data for Lessons Learned and a recognition of

the shortage of personnel and current workload of OCs and CIS analysts was a

recurring theme in the working group meeting. Only a limited amount of

information can be collected through instrumentation. Much of the data of

value for Lessons Learned requires systematic observations collected in a

consistent manner. The current software would support much of this data

^collection, but it would be expensive in terms of personnel time. It was the

consensus of the working group that additional requirements could not be

generated without additional resources being supplied. For example,

additional personnel were assigned to collect data on the effectiveness of the

Bradley Fighting Vehicle. The Electronic Clipboard will be another valuable

training evaluation and research data gathering tool.

Array decision makers need to balance the potential utillty^f the National

Training Center for improvinghdoctrine,(\tactics, training »yat<mot"-equipment,

and^procedures against competing requirements for personnel resources

throughout the Army. The potential of the NTC for Lessons Learned has not

been realized. CATA, ARI and other primary users of the data for Lessons

Learned have utilized the available data as effectively as possible. However,

they recognize, and have described in this report, how much more could

potentially be achieved if additional resources were available.

n
THIS PAGE INTENTIONALLY LEFT BLANK

14
REFERENCES

Briscoe, J. A. (Iy8fi, May) Capability to Analyze NTC Data (ARI Research

Note). Alexandria, VA: U, S. Army Research Institute.

Fobes, J. L. (1984, July) National Training Center Data Handbook (AHI

Research Product QH-M). Alexandria, VA: U. S. Army Research Institute.

15