Training Superiority & Training Surprise

REPORT OF THE

DEFENSE SCIENCE BOARD

TASK FORCE

on

January 2001

OFFICE OF THE UNDER SECRETARY OF DEFENSE FOR

ACQUISITION, TECHNOLOGY & LOGISTICS

WASHINGTON, D.C. 20301-3140
This report is a product of the Defense Science Board (DSB). The DSB is a Federal

Advisory Committee established to provide independent advice to the Secretary of Defense.

Statements, opinions, conclusions, and recommendations in this report do not necessarily

represent the official position of the Department of Defense.

This report is UNCLASSIFIED



Table of Contents

Introduction/Members and Briefings 1

Summary of Findings and Recommendations 3

Detailed Overview 4

Recommendations 22

Conclusion 24

Appendix A: Terms of Reference

Appendix B: Acronyms Used
In late 1998 the Undersecretary of Defense (Personnel &

Readiness), the Director, Defense Research and Engineering, andthe Joint Chiefs of Staff requested the Defense Science Board tocreate a task force on training and education. Drs. Joe Braddockand Ralph Chatham were appointed co-chairmen. The task forcemet periodically throughout 1999 and early 2000. This documentis the report of our deliberations.

Much of what follows is more anecdotal and less

quantitative than we would have preferred. Unlike the other Title10 Service responsibilities (man and equip), training performanceand resulting military proficiency are not well measured. Trainingis therefore easier to ignore. Thus, many of the training issues weraise are structural rather than technological; we found no one inthe Pentagon with sufficient authority who is graded on forcewide training performance.

As we proceeded, our emphasis shifted away from

education to highlight training superiority and training surprise.We were struck not only by the achievement of the Serviceswhere they apply engagement simulation in combat trainingcenters (CTC) but by the failure of other nations to do this. Thisis, in part, due to a lack of resources.

This training revolution (CTC use) appears to be a

uniquely American institution and not well coupled to morehierarchical cultures. It has had as profound an impact on warfareproficiency as advocates hope that the revolution in militaryaffairs (RMA) will achieve in the future. Unfortunately, unlesswe give it more support than we have in the last few years, it maynot be here tomorrow.

A second training revolution is brewing. Without it the

RMA cannot be sustained. Thoroughly trained warriors arerequired to support concepts of massing fires, not forces, withwidely spaced units flawlessly connected to each other and totheir command structure. Future training must be delivered to theindividual, to units, and to joint forces, when it is needed, not inthe schoolhouse after which there is time for proficiency to decay.Training must be applied over and over again as the compositionof the units and joint forces changes and as skills erode over time.Training must also become an integral part of the acquisition ofhardware or we will fail to achieve the performance in ourweapons systems that our other superiority (technology) strives todeliver.

Fortunately, technology is emerging that will support this

and may save money in the process. Unfortunately, there is no

DSB Task Force on Training Superiority and Training Surprise

rec/DTI - dsb training tf/ 12/15/00/page 1

DSB Task Forc

Training Superio

&

Training Surpr

Final Report

Dr. Ralph Chatham

Dr. Joe Braddock

Co-Chairmen

MILITARY PERFORMANCE

This report can be read on three levels: viewgraph, caption (of which t

sketch above suggests the relation between performance of complex tasks 

curves that make for effective unit and individual training. Se

training laboratory, development establishment, or manager with sufficietraining revolution.

Training Surprise: In the last decade we surprised not only o

proficiency. There is evidence that the culture of our first training revenemy might also be able to capitalize on the new training revolution. In military proficiency built up in 1 year. Others could surprise us. Trainothers to gain.
The panel was composed of people with relevant

backgrounds in: military, defense acquisition, and training/learning experts. Some members had participated in previousDefense Science Board (DSB) training studies. Several wererecruited as well for an Army Science Board 2000 Summer StudyTraining Dominance Panel. In addition to those listed, observersand contributors from the Services, the Joint Staff, and theintelligence community attended out meetings.

We had eight meetings over approximately 1 year. The

organizations listed on the chart spoke, as well as others. Most ofour meetings were held in the Washington, D.C., area. Asubgroup of us visited new air combat trainers at the Air ForceResearch Lab in Mesa, Arizona and at Langley Air Force Base.We visited the air CTCs, at Nellis Air Force Base, and Naval AirStation, Fallon, in Nevada.

Recognizing the importance of training for future forces,

we chose to forgo a visit to an Army Combat Training Center andinstead visited the Army’s developing First Digitized Division,the 4th ID at Fort Hood - a critical part of the Army’s

transformation program. We held our final meetings at the newlynamed Joint Forces Command, in Norfolk, Virginia, and its JointTraining, Analysis, and Simulation Center (JTASC) in Suffolk,Virginia.

DSB Task Force on Training Superiority and Training Surprise

rec/DTI - dsb training tf/ 12/29/00/page 3

The People and the Pl

◆Co-Chairmen

Dr. Joe BraddockDr. Ralph Chatham

◆Task force members &

government advisors :

Dr. John ChristieDr. Paul ChatelierDr. Dexter FletcherLTG Bill Hilsman, USA (ret)Dr. Sung LeeRADM Fred Lewis USN (ret)Mr. Joe MarkowitzDr. Warren MorrisonDr. Harry O’NeilMG George Steiner USAR(ret)Dr. Gershon WeltmanVADM J.D. Williams, USN (ret)RADM Jerome Smith, USN (ret)CAPT Wayne Thornton, USN(ret)Ms. Sandra Wetzel-Smith◆Executive SecretaryMike Parmentier

& Dan Gardner

ODUSD(R)R&T,PPJoint Staff (J-7) (JV2010)USMC Combat Development CommandDefense Intelligence AgencyJoint Staff (J-7) (JPME)USA Training Doctrine CommandAF Directorate of C2 (XOC)

Dep. Dir. Naval Training (N-7B)USA Training Directorate (DAMO-TR)Dep.Chief NAVPERS (P&TI)USMC Dep COS, Manpower&Reserve AffairsUSAF Dep COS for Pers, Edu. & TrainingJoint Staff (J-7) (DOCNET demo)A Dep. Chief NAVPERS (pers. & tra. resource)USAF Edu. & Tra, Command (AETC)OUSD for Acquisition, Technology, & LogisticsODUSD for Program IntegrationODUSD(R) Readiness and TrainingGeneral MotorsHMT-303, FREST (maintenance monitoring)DUSD ReadinessDefense Acquisition University (DAU)USN Aegis Training & Readiness CenterNavy CVX Program Mgr. (PMS 378)DD 21 Program Office, Manning, HSI, &

 Training Manager (PMS 500)

Apache PMO, USA Aviation Training Cntr.Dr. J. Bruer; James S. McDonnell FoundationNav. Air Warfare Ctr. Training Systems Div.USAF Research Lab, Mesa, AZ.

Briefers, Contributing Org

This is a partial list of who we are and whom w
Summary of Findings

 &

Recommendations
This chart summarizes our findings and recommendations. Wewill address each element in more detail later and then return tothis chart at the end.We found:-The U.S. armed forces have a training superiority thatcomplements their technological superiority.-Some forms of training can deliver order of magnitude warfareproficiency gains in times as short as 2 weeks.-The process is currently conducted in specialized Army, Navy,and Air Force combat training centers (CTC) for some, but by nomeans all, Service forces.-The infrastructure of these centers is being neglected.-2010/20 warfare will require more training, not less.-Training is also neglected in acquisition and testing; littleattention is paid to how a weapon system will be provided withtrained operators and maintainers.-Inadequate & poorly timed training will negate the technicalsuperiority of our hardware.-A new training revolution is possible. It can pay for itself ifcultural and structural problems are solved.-Adversaries could use this against us, but so far have beenrestrained by cost and cultural impediments.We recommend that:-The Services restore the combat training centers.-Services and Joint Forces Command (JFCOM) recommend howto expand CTC training to new warfare areas.-Undersecretary of Defense for Acquisition, Technology, andLogistics (USD(AT&L)) make training a co-equal part ofacquisition and testing by insisting that each acquisition programhave a defined training subsystem.-Put Under Secretary of Defense for Personnel and Readiness(USD(P&R)) on the Defense Acquisition Board (DAB).-USD (P&R) develop Advanced Concept TechnologyDemonstration (ACTD)-like pilot programs for each Service to 1)make residential training self-paced, and 2) move as muchtraining as possible from the schoolhouse to just-in-time, justright training in the units.-Defense Advanced Research Project Agency (DARPA) establisha training technology research effort.-Charge someone at Assistant Secretary of Defense(ASD)/Deputy Under Secretary of Defense(DUSD) level with review andoversight of training performance and measurement thereof.

DSB Task Force on Training Superiority and Training Surprise

rec/DTI - dsb training tf/ 12/29/00/page 2

Summary

◆Our uniquely American Training Superio◆JV2010/2020 future will require more trai◆Training failure will negate hardware prom◆A second revolution in training is needed 

!This new revolution should be able to pay for it

"The incentive structure in the DoD won’t foster t

# A central cause is that training performance is not 

◆Training should take its Title 10 seat with 

!Restore & expand upon crown jewels of curren!Establish and test co-equal training subsystem !Raise OSD/Acquisition training conscience:

"Services & CINCs deliver annual training report "Designate ASD/DUSD to be held accountable fo

!Foster the second training revolution by establ

"ACTD-like pilot programs in computerized self-p"An advanced training research program element"DARPA office to develop high payoff training/hu

◆DoD & Intel Community act to detect & a

Training performance (versus process) is seldom measured. Because no 

on (unmeasured) training performance in units, in joint forces, or in acqu

“Man and Equip.” Consequences and recommended act

 -Deputy Secretary of Defense (DEPSECDEF) request that the intelligetraining report card on potential adversaries.And last, but perhaps most important, we recommend:- The Services and Commander in Chiefs (CINCs) report (with Jo

Secretary of Defense yearly on the state of force training, concentadequacy, not on process. The Service reports are to be on serviccover the state of joint training. The report’s format is not important, the kind of training readiness accounting that is needed to prevent thoverlooking tradeoffs between training and hardware.
Detailed Overview
The next three charts discuss some of the characteristics

of learning and training.

The word training has many meanings and is often used

as a synonym for education. For the purposes of this report,training differs from education in that training is geared todeveloping specific skills and delivering people who can performdefined tasks. Education is a more general process with a broadergoal.

Military training can be sorted in a number of ways. One

such sorting includes training to develop: Service/military culture,basic military skills, technical skills, specific weapon systemoperation skills, small unit warfighting skills, larger unit battleproficiency, combined arms and interoperability warfare skills,theater, joint and coalition warfare skills.

Another way of sorting the complex training picture is

suggested in the chart. The process is often viewed with the aid ofa learning curve: a plot of the skills achieved as a function of theinvestment in training. The slope is shallow at first. For example,in pilot training, flying proficiency remains minimal throughoutground school, climbs rapidly during the early flights, and thenflattens out again.

When measured far out in the learning curve it often

appears that training investments do little to improveperformance. For example, in a large sample of qualified Navypilots, a 20 percent change in average flight hours yielded only a 4percent improvement in carrier landing skills. Looking at singletask learning curves it is hard to see how CTC training can makesuch a dramatic improvement in already-trained pilots or Armyunits.

While considering this conundrum several years ago, one

of us (Thornton) pointed out that warfare is a complex task andtraining for it involves a stacked set of learning curves, eachspringboarding off the levels below it. Carefully done, trainingcan stay on the steep part of the learning curve until an entire jointor combined-arms force is trained for its warfare mission.

Currently the process stops cascading at the level of the

CTCs. Higher levels of training are performed in a detached anduncoordinated process; it is currently very expensive to conductmission-level training with the entire force. The new trainingrevolution may make this kind of training affordable.

The stacked learning curves are by no means the whole

story, however, for what is learned is often forgotten, as we willsee in the next chart.

DSB Task Force on Training Superiority and Training Surprise

rec/DTI - dsb training tf/ 12/15/00/page 4

Investment in Training 

(time, cost, days at sea, number of rounds fired, ...

INDIVIDUAL SKILLS/EVOLUTION TRA(carrier landings, torpedo loading,unopposed bombing accuracy, ...)

20% ∆

~4% ∆

COMBINED SKILLS TRAINING(Long deployments, complete attacks on

~90% ∆ cooperative target, multiple ship exercise

Military Performance (exchange ratios,

targets hit, surveillance performance, ...) COMBINED SKILLS TRAINING 

[Tired crews, bad weather, night

targets are allowed independeelectromagnetic countermeasu

But most feedback generated fro

 HIGH FIDELITY TRAININ

[Requires independent op

feedback. E.g., Naval FSchool, National Trainin

TRAINING 

[Large-scale e

High-fidelity net

A Hierarchy ofLearning Curves

Tolerance of and expectation

of failure during training

Anatomy of Effective T

Culture of frank, critical feedback

& dedicated OPFOR

When proficiency in simple tasks is viewed as a function of training investWarfighting is not, however, a simple task. Viewing training for war as a visualize why CTCs work. One lesson, for example, is that CTCs shouldn’been first trained into the unit. Integration of mission training (the dotted 

achieved.
After training, if complex skills are not constantly

exercised, proficiency will decay substantially in times as short asa few months. At that point some level of retained skill remains,and stays with the individual for years. For complex tasks such asflying, proficiency can be regained with refresher training in aslittle as a few weeks, even after several years of not exercising theskills. Over-training can slow the loss and improve the base levelto which skills decay.

The graph shows one case of this process. The Navy

patterned their Strike University (now combined with other airweapons courses at NAS Fallon) after the first CTC, Top Gun, toteach pilots air-to-ground combat skills. Pilots are well trainedbefore they go to Strike University in order to be well prepared togain the maximum value from CTC learning, and gain they do.

A 1990 Center for Naval Analyses review of 241

bombing runs concluded that after 14 flight hours of training, theaverage pilot’s bombing error deceased by a factor of 3.3. Thefirst factor of 2 improvement came in the first 4 hours. 45 dayslater, however, bombing accuracy had decayed to the initial level.

Note that it is not as easy to measure the forgetting curve

as it is the learning curve. If you test an individual several times,the very act of testing provides refresher training. Given thesteepness of the learning curve, one or two trials should deliversubstantial performance improvements. The forgetting curveshown here is only a guess at what happens between the twoendpoints.

Our current training and deployment schedules are ill

matched to a skill decay time of 2 to 3 months. Unless tacticalrefresher training is provided within the deployed units in thefield, the refresher training will occur in combat. We show laterevidence that this unfortunate situation may be the case today.

The new training revolution may help here. We saw the

Air Force’s new Distributed Mission Training (DMT) System atthe Air Force Human Resource Laboratory’s (AFHRL)Warfighter Training Research Division. The DMT allows fourpilots to fly together against a simulated adversary. Its fidelity isnot exact, but it can deliver realistic training on 80 to 85 percentof complex air-to-air warfare tasks that a CTC can deliver. It alsoallows some freedom to train in ways that safety considerationsdo not permit in any real aircraft.

These kinds of training devices should be an integral part

of equipment deployed with combat units, for example on aircraftcarriers in forward areas. System fidelity should grow quickly in

DSB Task Force on Training Superiority and Training Surprise

rec/DTI - dsb training tf/ 12/15/00/page 5

A Forgetting Curv

Air-to-Ground

training at NAS

Fallon, U.S. Navy

“Strike University”

(total 14 flying hours)

Weeks After Leaving NAS F

Bombing Accuracy of F/A-18 Pilot

200

150100

50

Bomb Miss Distance (feet)

Time (weeks)

Data on 241

bombing runs

Curve estimated fr

end points only

0

123456

Sketched in gray is a forgetting curve. The highest level of proficiency d

remains. Peak performance can often be restored quickly by refresher tra

predeployment training and combat during that deployment e

the future and their cost should drop, but care must be taken that they do n

We emphasize that the phenomenon of skill decay does not me

useless. For example, we will show that time to reacquire warfare proficlearned in a CTC. Consideration of skill decay times, does, however, sugtasks should be designed such that 1) the training occurs as close in tineeded, and 2) methods should be devised to deliver key features of the tr
Historical research by Weiss and others established the

understanding that in air combat the more successful engagementsa pilot had, the higher the probability that he would survive thenext one. An “ace” (pilot with five kills) had a 95 percentprobability of being the winner of his next decisive engagement(one in which somebody is shot down) as opposed to the novice,who had less than a 50 percent chance. The winner of 30 decisiveengagements was almost invulnerable. Weiss also showed thisgeneral trend was the case for other combat situations, forexample with submarine captains in World War II.

Weiss believed that this was a selection effect, that aces

were born, not made. The best pilots survived and the worst gotshot down. However, what the Navy’s “Top Gun” school (andlater the Air Force’s Red Flag Exercises and the Army’s NationalTraining Center) showed was that this was more than survival ofthe fittest; it can be the result of learning. Moreover, it is possibleto train to the ace level without bloodshed.

DSB Task Force on Training Superiority and Training Surprise

rec/DTI - dsb training tf/ 12/15/00/page 6

The Evolution of a 

0.10.20.30.4

Probability of being shot down

Number of “decisive combats” experie

0.5

5 10 20 30 4

Is this training or “survival of the fittest?”

Pilot Survivability in Air-to-Air Combat: WW

Analysis of air, submarine and other combat showed that individuals whkill was achieved were much more likely to win the next one. Until Top G

Darwinism. We now know that much of the effect is the 
Most U.S. combat forces enjoy a substantial training

superiority over potential adversaries. Much of this comes fromthe use of CTCs, an invention of the Navy tactical air forces over30 years ago. This new approach to training delivered a dramaticchange to their air-to-air combat proficiency over Viet Nam(discussed in the next chart). In the 1970s and early 80s the AirForce and Army adopted their own versions of the technique withthe Army’s CTCs created to train units as large as a brigade at onetime.

In a CTC trainees gain the kind of experience that Weiss

showed develops combat aces, but in a CTC there is no risk ofdying from enemy fire. Trainees are far better prepared forcombat than forces trained by other methods. In their Red FlagExercises, for example, the Air Force’s prime objective is to givethe “blue four” (novice pilot in a four airship formation) a chanceto get seven or eight combats under his/her belt so that he/shewon’t have to experience the dangerous part of the learning curveduring real combat.

Until 1991 the Army’s first battle of each war had been a

disaster. In Desert Storm, after a decade of CTC use and with theinsistence that every unit that went to war had to do well in theNational Training Center, the Army had an overwhelminglysuccessful first battle. A second battle was not needed. There islittle doubt that training was a prime contributor to that victory.

The key elements of the CTC process include: a highly

competent independent opposing force that uses the tactics andequipment of the potential enemy; careful post-exercisereconstruction enabled by the use of an instrumented range; anafter-action review, which consists of frank, objective feedback tothe trainee of what was done and not done in each engagement;and an expectation of failure in the trained unit.

The last two features in particular appear to be uniquely

coupled to American culture. We found no other armed forcesthat had been able to implement engagement simulation for theirgeneral forces. We found no training as effective as thatperformed in our CTCs except in a few foreign special force units.

The CTC process is used by most of the Army and the

tactical air forces of the Navy and Air Force. Its institution forthese forces amounted to a revolution in training. That revolutionhas not, however, expanded elsewhere within the Services, nor isit applied routinely for joint warfare training. Most of the Navy,for example, is not aware of the spectacular results that can beachieved by CTCs.

DSB Task Force on Training Superiority and Training Surprise

rec/DTI - dsb training tf/ 12/15/00/page 7

Our Second Superi

◆The superb performance of our military 

just a result of technological superiority 

TRAINING SUPERIORIT

◆New combat training approach invented 

develops, without bloodshed, individual

"Instrumented ranges at major Combat Training "Highly competent Red/Opposing Force uses “e

!Uniquely coupled to American culture

"Objective, no-holds-barred feedback/replay

# no longer does first person to blackboard win

"Expectation of failure in the trained unit and its 

!Used by Army & most of the air forces (USAF

◆A second training revolution is brewing

!It will be needed for future warfare!But there are impediments to its implementati

Since WW II we have claimed that we will win wars with technological s

that technology does not always bring victory, the Army and our air force

in training. It was a key factor in our Desert St

The Second Training Revolution:

There is an opportunity to create a second training revolution (the first Weapon, Top Gun, School). The new revolution will be fueled by acomputer technology. We may soon be able to export to many other parts the kind of training that engagement simulation currently brings to U.S. like other revolutions, there are institutional forces that stand in the wreport.
The air war over Viet Nam produced one of the best

warfare experiments (albeit an unintentional one) ever conducted.In the last few months of 1968 the Navy lost 10 aircraft whileshooting down only 9 MiGs and had fired over 50 air-to-airmissiles without achieving a single kill. In 1969 no planes wereshot down on either side because of a bombing halt. When the airwar resumed, the Navy's kill ratio was 12.5 to one while the AirForce's fell slightly to 2.0 to one. These ratios are based on theorder of 100 enemy aircraft shot down in the each of the threeyear periods (110 kills by U.S. pilots for 1965-1968 and 74 for1970 to 1973).

Therefore, while there is some room to argue about

details of aircraft types, weapons used, and personnel policydifferences between the Navy and the Air Force, the sample sizeis large enough to yield a degree of confidence in drawing theconclusion that the change in kill ratios was real and that it wascaused by the Navy delivering Top Gun trainees into the fleet.*

The results of the U.S. Army’s tactical engagement

simulations, as measured by changed performance at the trainingsite, are as spectacular as the Top Gun influence on air war overViet Nam. We would like to show examples from more CTCs butthere are only a few more, totaling three for the Army and oneeach for the Navy and the Air Force. Moreover, data from thecenters that do exist are sparse.

Other kinds of training can also produce spectacular

results. We show an example of a single training device thatchanges the behavior of sonar operators so that they achieve anorder-of-magnitude increase in submarine search area. TheInteractive Multi-Sensor Analysis Trainer (IMAT) is a PC-basedtool that allows a sonar operator and a submarine’s tacticians tovisualize a very complicated acoustic situation and determine howbest to use their sensors. An investment of a few million dollars inthis training research and development (R&D) project hasdemonstrated performance enhancements that far more expensiveprograms have not achieved.

Not so incidentally, the IMAT was developed by a

training psychologist who also became a technical domain expert(S. Wetzel-Smith, a task force member). Many training systemsare developed in the absence of one or the other of the twodisciplines. That is one of the reasons that, although there areoften more decibels (dB) per dollar in training than anywhere else,the training dB are not always realized.

DSB Task Force on Training Superiority and Training Surprise

rec/DTI - dsb training tf/ 12/15/00/page 8

◆Air-to-Air Combat Over Viet Nam◆National Training Center ~1987◆Submarine Tactical Sensor Employ

Effective Training Makes a 

Change in likelihood that well-trained unit

entering NTC wins an engagement

(incoming proficiency normalized)

USAF: 2:1

U

Result simulat

ONE

YEAR

of USN

CTC

operation

x5 f

TWO

WEEKS

at NTC

C

No CTC

Detect and track coverage Performa

TWO DAYS

The only changInteractive Mult

USN: 2:1 Exchange Ratios:

US (primarily F-4s) vs.MIGs

pre 1969

 before

training

Three examples are provided, spanning three decades, of order-of-ma

brought about by a very brief period of training. These are successe

consequences of training failures.

__________________________* Chatham, R.E., Training Assessment: a Critical Intelligence Deficiency. A Report on the ITraining, Exercises & Military Proficiency, Dynamics Technology Report DTW-9509.02.9-Gorman, P.F., The Military Value of Training, Institute for Defense Analysis 

"You Fight Like You Train," Armed Forces Journal International, May 1974 p 25, 26, 34.
The existing CTCs are not being supported as well as

they were 5 or 10 years ago. The chart shows a few, easilymeasured examples; there are more. CTCs are the crown jewels ofthe first training revolution and a central foundation of ourtraining superiority. Although the task force could not quantifythe proficiency consequences of the decay in CTCs infrastructure,we are concerned that, at least for air-to-air combat, we may be atthe edge of losing a substantial portion of the training value thatthese centers have offered in the past. Moreover, even if restored,the old infrastructure would not represent the current threats.Defining the new threats will not be easy.

The examples here are not parallel: the Army’s does not

describe the same kind of deficiency as those for the air CTCs.This partly reflects our choice to visit the Army’s developingDigital Division at Fort Hood instead of one of their three CTCs.That decision was driven by our concern for how future systemsand warfare concepts will influence training requirements. It isclear, for example, that the capabilities being developed in theDigital Division (4th Infantry) cannot be exercised properly in the

current CTCs. Nevertheless, the Army’s commitment to the CTCrevolution appears stronger than that of the Navy and USAF airforces.

For example, the Air Force decided several years ago to

forgo a substantial dedicated air opposing force (aggressorsquadrons). “Red” aircraft in Air Force Red Flag Exercises arenow manned mostly by active duty pilots who, with their aircraft,are borrowed from other squadrons. These pilots receive negativetraining for the time spent trying to imitate enemy tactics.Moreover, the aircraft used are not “dissimilar” (that is, they havethe same characteristics as the trainee’s aircraft). This seriouslydegrades the training experience.

The Navy still supports aggressor squadrons. The pilots

are mostly reserves who must formally qualify as opposing forcepilots within a week of flying. The aircraft used, however, are nolonger all dissimilar and most are reaching the end of their usefullives. The (unfunded) cost to buy 18 F-16s as operational force(OPFOR) for USN was $638 million in 1999. We saw otherindications of eroding infrastructure: many air crews get no liveordnance experience and the time between CTC visits islengthening. It might be argued that the major warfare threat weexperienced in the 1990s came not from aircraft but integrated airdefenses (IADS), mostly ground-based, but air CTCs are losingthese, too. An (unfunded) Electronic Warfare (EW) upgrade at

DSB Task Force on Training Superiority and Training Surprise

rec/DTI - dsb training tf/ 12/15/00/page 9

◆Army◆USAF CTC (Nellis AFB)◆Navy Air CTC (NAS Fallon)

1999 Cost to upgrade NAS Fallon EW and provide 18 dissimi

◆Infrastructure is eroding & does not repre◆No CTCs for non-air Navy, parts of USA 

& support forces & no CTC for joint/inter

Goal for NTC combat vehicle availability: 90%

State of the Combat Traini

8-10 dedicatedOPFOR F-16s.All old and unreliable

1989 Dec 1991994 Dec 199

1998

Proficiency consequences not quantified

Achieved during NTC 

40% vehicle commo

4 active duty aggressor squadrons with 24 dissimilar aircraft each

4 active duty adversary squadrons with 20+ dissimilaraircraft each

23 old dissimilaraircraft, reservepilots, limited to 3Gmaneuvers

The infrastructure of the CTCs is decaying. The Air Force chose to drop 

Navy’s cannot be sustained much longer. The Army is applying some res

to upgrade them to support modern weapons

NAS Fallon would cost $300 million over 10 years and leave the facility w

We emphasize that those manning the CTCs are superb warrior

they can with the resources provided, but that “can do” attitude may makthe possibility that, even with their heroic efforts, the infrastructure halonger support the kind of training we expect.

The foregoing only addresses issues with the current CTCs. We 

forces do not use CTC training. A key element missing from even thelsewhere in the Services is the notion of a dedicated opposing force that 

action.

We started out hoping to “bottle” CTC training and export it th

that we saw in the existing CTCs, we recommend first that they be rethreats and then funded to remain current.
Historical examples suggest that there is a substantial

risk that we won’t achieve the performance that our technologicalsuperiority promises. The top two examples in this chart showcases where lack of appropriate or adequate training reducedsubstantially or completely negated the gains from a weapondevelopment. The bottom two examples point out that even thebest training if not applied at the right time can rob us ofperformance early in a conflict; as John Byron pointed outrecently about Russian submariners, “people rust faster thanships.”

The anti-tank weapon TOW was designed to engage

targets at up to 4km range and showed that capability inoperational test and evaluation. A decade later the Army foundthat it was only used at less than half that range. The conclusionof their investigation was that this was caused by a failure to trainfor over-the-horizon use. Had we known that only 2km of themissile’s range would be used, we could have saved a largefraction of the development, production, and logistic cost of theweapon and designed it to fly only 2km.

The submarine force realized in the mid-1990s that the

price of living with legacy computing hardware in their acousticsystems had become intolerable. They started a highly innovativeprogram to replace all acoustic processors with commercial offthe-shelf computers for all attack submarines in a period of 4years. The first boat to receive the upgrade was said to have morecomputing power than the sum of that available to all previousand existing submarines in the fleet.

The first message from that boat, however, stated that the

new computers didn’t work. The few days of training at thefactory that the developers had thought would suffice wereentirely inadequate to deliver lasting proficiency either to operatethe hardware or to maintain it. (We will mention a paralleloccurrence in the Army later.) Ad hoc remedial training fixed thesubmarine’s problem, but the Navy will be hard pressed to deliversufficient training as the pace of installations speeds up. A major lesson learned by those charged with theremedial acoustic training program is that you can’t know there isa training problem until you have ways to measure proficiency.They developed a proficiency test disguised as a sonarman“survey.” Armed with knowledge of the test results they told me(Chatham) that the cheapest 10dB came from training, but theywere still worried about the skill decay time.

DSB Task Force on Training Superiority and Training Surprise

rec/DTI - dsb training tf/ 12/15/00/page 10

Risk That Hardware Performance W◆ Acquisition performance is usually predicated on perf◆TOW◆Submarine Acoustic Rapid COTS Insert◆Air-to-Ground Accuracy, USN Strike U◆USN Carrier Air Wing Air-to-Ground P

1972 performance in OT&E 1988-1990 D

Inadequate training 1999Skill DecayEntering performance 1/(200 feet) 

NAS Fallon 

First week at Fallon: 30% targets hit

Third week at Fallon:70% targets hit

~Four months between

Fallon & combat

performance after 14 flight hours training

1/(60 feet)

Message from sub “Not ready for sea

Max engagement range 3.75 km 1.5 km

Expected performance

(arbitrary units) Expected p

Failure adequately to consider training in acquisition can rob us of the 

much for. Even where training is well delivered, if it is not timely, s

That brings us to the lower half of the chart. We have already see

chart describing forgetting curves. Data on recent attacks against Iraqi targsuffered a similar decay, although the data are not as uncontaminated as foAfter a few missions the attacking force’s performance returned to the levin pre-deployment training at NAS Fallon over a period of 3 weeks.

Those at Fallon are trying to reduce the time between training an

high fidelity onboard multi-aircraft training devices should be deployed wDistributed Mission Trainers (DMT), which allow interactive simulation tare a start, but DMT is neither deployable nor embraced by the Navy.
The current acquisition system treats training as one of a

number of “ilities” that must be considered during the acquisitionprocess. Given its standing as one of nine (development,manufacturing, test and evaluation, verification, deployment,operations, support, training, and disposal), training is usuallyviewed as more of a nuisance or a block to be mechanicallychecked off than as a way to enhance performance by an order ofmagnitude (or conversely something that, if ignored, can reduceperformance by a similar large amount). In one briefing we foundtraining mixed on equal standing with crew privacy and foodservice.

Training should stand as one of only three (man, equip,

and train), rather than mixed up in the minds of acquisitionmanagers with things like crew privacy. Failure to so viewtraining leads to the kinds of performance failures discussedpreviously with the TOW missile and the submarine ARCIprogram.

We were reminded by task force member Bill Hilsman of

a parallel to the Acoustic Rapid COTS Insertion (ARCI) case thatoccurred during the deployment of the Improved Hawk AAWmissile in the late 1970s. Six months after it was deployed to theMiddle East the I-Hawk batteries were 90 percent notoperationally ready. Again the cause was a lack of training for theoperators and the maintainers.

The Army changed its acquisition policies to insist that

for each development program a training subsystem be formallydesignated and funded by acquisition dollars. If the trainingsubsystem was not ready, the whole weapon system would bedeclared not operationally ready and would not be deployed. Thatpolicy did not last. We recommend it be instituted again, this timeDoD-wide.

Consideration should be given at the beginning of a

program to how competent operators will be provided throughoutthe life of the system. Some of the issues that need to beaddressed include:- Can ordinary operators deal with the system?- Will the operators’ professional advancement be dependent upontheir proficiency with the new system, or will that not be tested?- Will training devices and training courses be available at initialoperational capability (IOC)?- How can we test the adequacy of the training subsystem duringthe operational test and evaluation process?

DSB Task Force on Training Superiority and Training Surprise

rec/DTI - dsb training tf/ 12/15/00/page 11

Required: A Formal Change to Ac

◆Acquisition Now◆Acquisition Future

OT&E

ility 1ility 2

training

ility n-1

Years

Develop & Produce 

Develop & Produce 

OT&E

ility 1ility 2ility n

...

...

E.g., our Task Fo

that he “will no

We were told by 

had become opbeen fielded anextended to De

Ad HocTraining

Td

Training developed, tested and deployed as a co-eq

The DoD acquisition instruction lists training second-to-last in a list of n

It stands only before “disposal”. Given the major impact training can 

equip and train” dictate of Title 10, training needs a new 
We heard a consistent lament during our deliberations:

the biggest change in the military of the 1990s was that eachService, each unit, and each Service member is being asked to domore for less. The funding squeeze seems to be on everywhere,and training, as a thing that is hard to measure, is one of the firstareas to be squeezed. Even if future warfare were not to change, alower cost approach to individual skills training, as well as unitwarfare training, will have to be found if we are to maintain ourtraining superiority.

Warfare in the future will not be the same as it was in the

past, yet the task force saw no plans anywhere, Service-based orjoint, for fundamentally altering the training infrastructure toaccommodate Joint Vision 2010/2020 warfare. As we found in theacquisition process, it appears that training is ignored whenplanning for the future in the tacit hope that it will solve itself.Training programs are, by and large, reactive, not proactive.

The characteristics of advanced weapons technology will

also require changes in the current training architecture. Acommercial anecdote illustrates this. General Motors foundseveral years ago that they were spending over $3 billion per yearon warrantee repairs. One third of the repairs were failures. They,therefore, instituted a comprehensive schoolhouse trainingprogram. After 4 years, half of their mechanics had received theschooling but GM then found that there was no differencebetween the repair performance of those with training and thosewithout.

The cause, GM believes, is that their systems are both so

reliable and so complicated that, after making a repair, severalmonths elapse before a mechanic sees a similar problem again.This is too long a time to expect her or him to retain thespecialized knowledge. We assert that this is true of militaryweapons maintenance and operations. It is a prime reason trainingmust be moved from the schoolhouse to the unit.

GM, capitalizing on technology started, but not initially

implemented by the DoD, began developing and testing the use ofan integrated electronic tech-manual that delivered troubleshooting knowledge at the point of use for Cadillac transmissions.We were pleased to see a few instances of these devices beingtried in the Services. The next few charts discuss what we call a secondtraining revolution: the application of computer technology andtraining research primarily to individual training. The promise ofthis revolution is that it will control the decay of skills by

DSB Task Force on Training Superiority and Training Surprise

rec/DTI - dsb training tf/ 12/15/00/page 12

A New Training Revolutio

◆ Even if warfare doesn’t change, budget pressure will re◆ Existing Service CTCs are not sufficient to train for fut

!CTCs do not/will not cover: joint warfare, deployment, gr

weapons, ships/submarines, interoperability, new threats, 

◆ Future weapons technology also appears to require mor

!E.g., the Digital Division must train for both old and new !Sophisticated maintenance & operational skills can’t be re

◆ Emerging manpower limitations will:

!Generate further personnel turbulence increasing the need !Demand shorter training pipelines!Decrease manpower that can be allotted to schoolhouses 

Individual

Training

Collective

training

% in School % in Unit

Now Future: more total 

Mass Forces Mass E

Individual

Training

Collective

training

% 

Trained people are not a commodity like fuel or weapons that can be deliv

is a serious detractor from operational and maintenance proficiency in 

move into the units where the right skills can be deliver

delivering training at the point of need and will enable complex trinexpensively.
Each of us has been educated and trained for a

significant fraction of our lives. In consequence, we all have anintuitive understanding of how the learning process works. Inmany cases we are wrong. The academic learning community, aswell, has its share of those who try to shoehorn all evidence intofavorite theories about how they would like people to be, ratherthan find out what works. There is, however, a well-supportedbody of knowledge about how people learn.

 It is not surprising that CTC training uses many of the

approaches now shown quantitatively to be more effective thanconventional schoolhouse training: direct feedback, collaborativelearning, and what amounts to individual tutoring. CTCs alsobenefit from the cognitive dissonance that comes from drivingpeople very hard. Success in this kind of environment, likesuccess in a demanding boot camp, can be shown to deliverpersistent attitude changes toward belief in oneself, theorganization, and the process.

We cannot rely on commercial training courses to give

us help. Consumer training packages can’t afford to use learningtheory; their prime goal is to keep the cost of the product on theshelf below $29.95. Moreover, much of the emphasis inuniversities is on education. Our emphasis must be deliveringpeople with a specific set of skills when and where they areneeded and to do that rapidly and inexpensively without regard tocampuses and tenure.

The graphs illustrate some characteristics (rate and

quality) of group and individual learning. Trained people cannotbe ordered in identical packages like weapons. We have alreadypointed out that skills, unused, decay more rapidly than steel rusts.A second difference between people and military hardware isvariability. Learning time can differ by as much as a factor of 7between the slowest learners and the fastest.

 Residential instruction must bias its course length

toward the slower student in a “one size fits all” approach. If thepace of a course can be matched to the learning rate of eachstudent, average learning times can easily be reduced by 30percent; in some cases the reduction has been as much as 80percent.

Tutoring (individualized instruction with feedback using

all the pathways of human-to-human interaction) does more thanreduce the time to learn. It greatly increases the level ofknowledge or skill in the student. The chart in the upper rightpoints out that a tutor, even one lacking effective learning

DSB Task Force on Training Superiority and Training Surprise

rec/DTI - dsb training tf/ 12/15/00/page 13

A New Training Revolutio

Individual

learning times Fastest 10% learn 5-7 times faster than slowest 10%

Average

learning time

Research findings on value of self paced learning

Addition of intelligent (human or automated) tutor

CMU/Indiana U. findings on collaborative, asynchronous

distributed learning

◆ We stand on the verge of a potential training revolution 

!Advanced computer learning, just-in-time/just-right training de

distributed learning environments, advanced embedded traininglearning, training administration and resource management (prcourseware), automated courseware development, automated a

◆ The new training can be cheaper, faster and there when ◆ New efficiencies (e.g., in training tailored to the individual) 

efforts critical to retaining and expanding our training su

Numbers of students Slowest 10% Median

Basiccourse

With self testingand collaboration

A range of electronic-aided learning tools is emerging. They are well c

course content and goals are easier to define than in education. Mor

courseware development hold great promise to reduce co

techniques, can improve student skills by two standard deviations over whthe next viewgraph we will show that the same benefits appear to be delivtutor.

We have discussed a few examples of results from the limited re

The slide lists other approaches that may deliver additional gains. Many oby technologists rather than those who understand learning processes. Reswell funded. Funding for research into how to use this to deliver skilled pemeasured in fractions of a percent of either the training or military R&D bpay enormous dividends.
 The University of Memphis had a problem. It required thatall students take a computer literacy course but they were runningout of instructors. Dr. A. Graesser, working under a NationalScience Foundation (NSF) grant, chose this domain area for thedevelopment of “Auto-Tutor.”

A student uses an ordinary personal computer to type in

responses to questions asked aloud by the program. (Keyboardinput was chosen because speech-to-text programs still have a 10percent error rate and the time and distraction needed to correctmistakes is unacceptable.) The tutor is also represented on themonitor by both text and an animated line drawing of a humanface (see the next chart). As a student types in her response shereceives instant feedback from changes in facial expression in theanimation. The student also gets an audible and textual response.

The Auto-Tutor guides the student through a series of

open-ended questions that, if answered correctly, demonstrate thedesired level of computer skills. It measures overall performanceas well as how the student is answering the specific question. Itautomatically determines whether the student needs additionalwork in an area and chooses other questions to exercise him or heruntil that area is understood.

Auto-Tutor and its cousins, the electronic technical

manuals, will only be affordable if new content on new subjectmatter can be acquired and inserted into the framework cheaply. Itappears that this can be done. The developers of Auto-Tutor havecreated a conversational and automated method to create a tutoron a new subject by asking a domain expert to type in a set ofquestions that she believes will cover the skill area of interest. Thedevelopment system elicits from the expert a set of seven or eightacceptable answers to each question. It also elicits potentialincorrect answers.

 In addition, review articles and other text on the subject

area are scanned and subjected to a process called latent semanticanalysis. It has been shown that automated sorting of theconnections among words in text can lead to a computer-basedessay grading system that evaluates student essays in the standardA through F system with a performance indistinguishable fromthat of human graders. Auto-Tutor uses the same technology tohelp it evaluate student responses to its questions.

Similarly, automated technical manuals can be generated

by scanning existing printed manuals. The connections among thewords and the structure are automatically analyzed and thenreformatted in a structure suited for troubleshooting. (One wishes

DSB Task Force on Training Superiority and Training Surprise

rec/DTI - dsb training tf/ 12/15/00/page 14

A Sample of the New Revolut

◆Human tutors evoke 2σ performance inc◆It appears that this kind of teaching can b

!U. of Memphis built Auto-Tutor to teach basi

"Personal computer based system"Line-drawing of human face asks questions (sig"Student responds on keyboard"Auto-tutor’s response to student comes as much 

spoken/written words

◆Developing new courseware can also be 

!Converting auto-tutor to new subject area req

"Scanning in background papers for latent seman

# Uses technology developed for automated essay g

"Set of questions & acceptable answers conversa"The rest can be automated

◆JFCOM exploring concept for joint task 

Auto-tutor is one of a number of new approaches to deliver training 

integrated electronic technical manuals (IETMs) are another. What is 

development can be automated, no longer requiring teams of cognit

to avoid a web-based structure, which is not well suited to troubleshootingin 3 weeks.Another month of work can animate the diagrams in the manual as well.

The military training value in these kinds of systems comes from

1) Rapid, cheap, automated generation of training content/courseware.2) Delivery of that content where and when needed.3) Training delivery systems that use the learning pathways wired into humtext or looking at pictures on a computer monitor.
 The words about Auto-tutor are on the last page. Perhaps thepicture here will be worth a proverbial thousand additional words. DSB Task Force on Training Superiority and Training Surprise

rec/DTI - dsb training tf/ 11/28/00/page 15

Auto-Tutor Auto-Tutor (2)

Incidentally, if this face does not inspireconfidence, a different one can bechosen to suit your cultural preferences.

A screen shot from Auto-Tutor. The face on the left delivers feedback by 

the student’s responses. The tutor’s words are spoken and displayed on t

expect that the student will be able to speak his or her answers i
Self-paced learning and tutoring are facets of a more

general concept of matching the instruction to the individual.Defining the individual in order to determine how best to trainhim or her leads to the issues of testing. The task force was struckby the concepts developed by Dr. R. Sternberg at Yale University.

He pointed out that what is measured by existing

intelligence tests is an incomplete predictor of future success. Forexample, scores on the Graduate Record Examination (GRE) wereknown to predict only about 10 percent of the success in the firstyear of graduate school. Sternberg asked: How do the scorespredict performance in the second year? The answer was that theywere not correlated at all. Because the GRE measures abilitiessimilar to those measured by our military entrance exams, thiswas disturbing.

Sternberg explored whether there are other measures that

can partially predict future performance. He settled on two newcharacteristics that he calls “creative intelligence” and “practicalintelligence.” He has developed repeatable and well-definedmeasures of these traits. These measures individually have aboutthe same predictive power as the currently used single measure(which he calls “analytic intelligence.”) The use of these three,independent predictors of success should give us a better way toselect applicants for entry into the military and help define theoptimum ways to tailor training to the individual.

We believe that Sternberg’s three intelligences are well

established and that there is merit in his contention that the currentstrong dependence upon analytic intelligence as a societalselection criteria is unjustified and may be wasteful of humanresources. The academic objectors to his combining the three intoa “successful intelligence” fall primarily into three camps. Onecamp believes that any kind of characterizing of individuals ismorally wrong, one thinks there is only one kind, and anotherdeclares that there are more than three kinds. Given this range ofviews, we believe that three is just about right. The payoff intraining and retention for utilizing these new measures is highenough to justify a pilot program to determine if the Services canmake better choices in recruiting.

Practical Intelligence measures the application of

knowledge. It is tested for by asking questions about how to solveproblems. In specific situations, it probes the ability to understandconsequences of actions beyond what the conventional (analytic)intelligence measures. (i.e., conventional intelligence tests theability to read, comprehend, and then compare and contrast.)

DSB Task Force on Training Superiority and Training Surprise

rec/DTI - dsb training tf/ 12/15/00/page 16

New Criteria for Predicting Ind

◆New research suggests that there are thre

!Analytic: ability to think abstractly, verbal abi

call intelligence)

!Practical: ability to adapt to a changing envir

in specific situations)

!Creative: dealing with unusual situations

◆Current (analytic) measures have only .30 

success

◆The others have a similar ~.30 correlation 

of each other

◆All three are well-defined and have repe◆Use of all three measures (Sternberg’s Succes

!Permit better coupling of training to the indivi!Improve the accuracy of recruiting assessmen

"Wider field of acceptable applicants; fewer drop

Current intelligence tests are only moderate predictors of success. Ot

repeatable measures of different kinds of “intelligence” have emerged. T

similar degree. Use of multiple measures should improve bo

Practical Intelligence might be tested for by describing a conflict situatiocourses of action would best resolve it.

Creative intelligence measures the flexibility of an individual t

for example, that there is a color called “grue” that is green before the yintelligence testing will ask what inferences one can draw from this count

Individual performance on any one of these measures is relative

other two. The military needs forces in which multiple kinds of capabilitie
The Army could save $114 million in per diem costs

alone simply from the reduction of instructional time inschoolhouses that would be delivered by the implementation ofcomputer-based self-paced learning. These are not the onlysavings to be had, merely the easiest to count. We estimate thatsavings DoD-wide from reduced learning time in residentialschools can easily amount to over a billion dollars per year; againmerely from the introduction of self-paced training as opposed toclassroom instruction. Indirect savings will be greater. Thesesavings are “low hanging fruit”. They can be grasped, however,only if the money saved in the personnel system can be deliveredto those who have to institute self-paced training in theschoolhouses.

We see a second future with even more payoff, one that

eliminates residential instruction for most technical courses andcreates skilled Service members via distributed learning, selfpaced courses, auto-tutors, electronic tech manuals, etc. With thepeople staying in their units instead of spending long times in theschoolhouse, this would help ameliorate the number one concernwe heard from every field commanders we visited or heard from:personnel turbulence.

This won’t come easily. The infrastructure to carry

advanced learning out to the units must be paid for in advance by

the training community before the savings accrue later in thepersonnel system. Moreover, many unit commanders will viewthis as shoving the burden of more training onto their unit, insteadof as a way to keep people in the unit where they are available forcontingencies. It is also a way to ensure that those people willhave the needed skills well honed while they in the unit ratherthan be at their peak when they are in the schoolhouse.Nevertheless care must be taken when moving training into theunit to ensure that it does not simply add another task to the unitcommander’s already over-filled plate, and the concept must becarefully and compellingly sold to unit commanders.

If the structural problems can be overcome, the payoff

from the second training revolution will free resources that will beneeded to expand training efforts to support new forces such asthe projected transformed army.

DSB Task Force on Training Superiority and Training Surprise

rec/DTI - dsb training tf/ 12/15/00/page 17

◆ Army Science Board found (1997)

!For 525 Army schoolhouse courses and 30% reduct

self paced learning alone:

"Potential >10,000 man-year savings and $114M per die

◆ A more extreme future: People stay in the units

!Over $3B direct savings DoD-wide, if personnel sys!Personnel turbulence reduced by 40%!Learn material in 1/2 to 1/4 the time when the know

◆ We can initiate and foster this revolution by:

!Emphasize collaborative asynchronous distributed l!Develop/apply (military & civilian) standards (being !Modernize & automate courseware development and !Institute a program of learning research for DoD-sp

Payoff from the Second Train

Current costs Possible fut

$4.4B/yr DoD-wide specialized training costs [those

that change with student load (1996) not including student pay]

Potential $1Bfrom self-pa

Now: schoolhouse fixed-time training costs A possible 

Self-paced learning in residential instruction can save over a billion d

personnel costs, if the personnel system can adjust to a variable course tim

delivered if people stay in their unit
When we made the case that effective training can

change proficiency faster and more cheaply than the development

of a new weapon system, we ran across the following argument.

“When money is tight and the threat is low, the military should

buy hardware, which endures longer than training, and hope to be

able to take care of training later.” This may, in fact, be a viable

strategy to husband limited resources, but it should only be

implemented after considering the consequences. It should not be

done by default simply because training has little voice in the

acquisition process. Unfortunately, the proficiency consequences

that might arise from neglecting training are not measured today,

nor could our task force find any existing tools that could be used

to make such measurements.

An additional factor militating against the “hardware

now, training later” approach is that the kind of infrastructure thatwill deliver training to the point of need is itself hardware. Therecan be no commercial off-the-shelf source to train a brigade inland warfare nor a pilot how to fight his or her aircraft.

In the 1990s America’s battles were come-as-you-are

events allowing little or no time to rebuild a training system. Wehad time to send our ground forces through the National TrainingCenter before they fought in Iraq, but we would not have had timeto build the training center as well.

 Our task force’s job would have been made easier had

there been a robust learning research community in the military,but there is none. We were shown by the Services trainingresearch managers mostly small projects with small budgets andsmall impacts, or we saw the tail end of formerly well-fundedprograms.

The acoustic training device, IMAT, was a notable

exception. It, too, was a project with a small budget, but its impactwas not small. Its success strongly suggests that there is greatleverage for additional advanced research in training technology.

In the Office of the Secretary of Defense (OSD) there is

a dynamic office of five people three layers down below the USD(Personnel and Readiness). They are the highest-rankingorganization that even has the word “training” in their name. Theyhave taken the lead in defining nation wide standards fordistributed learning. Still, they are too small and too far down inthe system to successfully remind the acquisition community thattheir weapon systems must have trained people to fight themeffectively.

DSB Task Force on Training Superiority and Training Surprise

rec/DTI - dsb training tf/ 12/15/00/page 18

Impediments to Training 

◆We found a perceived Training Resource 

Major Premise: We can’t pay for everythingMinor Premise: Training time-constants are much shoTherefore: Buy hardware now.

Fix training later.

"Unfortunately, training systems can not be crea

COTS source for military force/unit training

"This kind of misguided reasoning will prevail a

effective measuring sticks for training or profic

◆Most* training R&D today is ad hoc, loc

!There is no research to bind together the elem

"The training labs have been dispersed, disestab

!Schoolhouses currently resist major shifts to 

◆Different “colors of money” impede trai

!Start-up costs can’t be derived from future sa

_________* Major exception is OSD P&R setting of nationwide distributed le

The new training revolution may be able to pay for itself, but there are s

so. There is a general belief that training can be neglected when the fund

with sufficient authority in the Pentagon to coun
This task force was not established because a powerful

defense organization saw a problem that desperately neededreview. It was created in part because no organization was calling

for training reforms. A large contributor to the problems weidentified is the diffuse management of training. There areseparate barons for individual training, for unit training, fortraining certain warfare system operators, for logistics training,and for joint training. There is logistics training and pilot trainingand submarine training, etc. Even in the individual training arenathere is no consensus or leadership to identify or implement thetypes of changes that the training technology revolution willpermit and will require.

We saw reason to believe that the DoD can maintain

U.S. training superiority and make significant advances towardthe next training revolution within roughly the same amounts oftraining resources currently used. It cannot be done, however, ifthe DoD continues to spend in the same way they have in the past.The personnel system and the training systems do not cooperate.The acquisition system is oblivious to both.

Effective training systems could generate personnel,

acquisition, or operational savings. Conversely, expendituresduring acquisition or in the personnel system can pay dividends intraining savings, but there is no mechanism to make trade-offsamong those administrative stovepipes.

When it comes time to distribute money in the Pentagon

(or in Congress for that matter) there is no vocal constituencydemanding funding for training. This makes it all the moreimportant that structural changes be made to ensure that trainingissues receive sustained, continuing consideration throughout theDoD.

DSB Task Force on Training Superiority and Training Surprise

rec/DTI - dsb training tf/ 12/15/00/page 19

Impediments to Training Ch

◆Training management and resources are 

"Personnel policies/management are handled bpolicies/management for the same individuals bucares that the forces are trained."One set of people are concerned with technica"Unit training is the responsibility of yet differ"Logistics training resides elsewhere"Joint training is the responsibility of ..."Dollar resources are as diffuse as managemen"Tradeoffs among stovepipes are difficult belo

◆Therefore we are faced with a complex s

! How can we use savings from training efficienci

improved personnel management to fund more & im

training?

◆There is no “Military-Training Complex

systems

Training responsibilities are spread throughout the military and each o

ignoring the trades that might save money elsewhere. For example, up

useable/maintainable system might obviate a large training expense ov

incentive for an acquisition manager to pa
Lord Kelvin is said to have stated, “If you can’t measure

something or describe it numerically, you have no right to discussit.” With respect to this aphorism our task force has been placed ata severe disadvantage. Neither training nor warfightingproficiency is well measured. Worse, the few attempts to makesuch measurements have perished.

For example, a 1982 DSB summer study on training

recommended the establishment of a Training Performance DataCenter. It took almost 4 years to get it started and it waseliminated a few years later in a territorial dispute with thepersonnel establishment. The importance of measuring trainingwas not understood at a level high enough to protect it.

What is measured about training is throughput and

process (e.g., the number of students and the number ofclassrooms, colloquially: butts in seats). What should be measuredis whether the training delivered to individuals, to units, tocommanders, and to joint task forces was efficient and effective:how the student or unit or task force performs after training.

Even for process measures, each Service defines training

differently. Flight hours count as training for the Air Force. TheArmy has created a similar measure called “tank miles.” TheNavy considers ship time at sea as operations not training.Consequently, we could get nothing approaching a uniformaccounting of training costs from the Services. One tried toprovide such an accounting: a second Service tried when askedagain; we gave up on the third Service after asking three times.No one is graded on training performance. No one is charged withassessing, and evaluating different parts of DoD training to seewhere marginal dollars should go to and/or come from in order tomake the most productive improvements in force and unitcapabilities.

The Services are charged in Title 10 to man, equip, and

train the forces of our nation. There are well-establishedbureaucracies to oversee manning and equipping, but there is nocomparable establishment that covers training in the sensedescribed above.

Training development in the Services appears to be

reactive rather than proactive. That is, the forces train to use whatthey are given, rather than choosing weapons characteristics basedon whether people can be trained to operate them to good effect.

The first training revolution (and the IMAT acoustic

trainer) was the result of the actions of a few individuals who didunderstand training in this sense. They were not the result of a

DSB Task Force on Training Superiority and Training Surprise

rec/DTI - dsb training tf/ 12/15/00/page 20

Man, Equip and T

◆ Man and Equip are measured; Training outputs a

!If you don’t measure something, even the well-inte

"NTC OPFOR Commander: “We don’t measure our comb

accomplish our mission-essential tactics ... We measure it in soldiers we have, the amount of equipment we have, the mainavailable training resources.”

"Army Combined Arms Center developed data-collect

1995. Cost $2M/year. It was not funded.

"Training Performance Data Center, established as res

eliminated in early 1990s due to lack of high-level su

"If you only measure inputs (training loads), not outpu

individuals), there is no good basis for making tradeo

!This task force was unable even to get an accounti

money they allot to individual, unit and force level 

◆ There is no effective voice in the Pentagon who i

training performance

!Not in Personnel & Readiness, in Acquisition, in S!All major training achievements that we saw were 

extraordinary individuals exercising their existing for another one to appear spontaneously.

Title 10 of the U.S. Code directs the Services to deliver to the CINCS m

Manpower and equipment are easy to count. Training is not. Without a n

are training, even the well intentioned can ignore or miss 

training office exercising its charter (as is done for hardware) to identifythat would make changes to warfare proficiency. Training is seldom view

Our task force did not feel it wise to sit back and hope that anoth

spearhead the next training revolution. Structural changes are called for in have a better chance to be heard and be put in a position to insist, for examago, that training become co-equal with the other major factors that build 
We raised the following question. If the Navy could

change its air-to-air exchange ratios by a factor of 6 in one year bythe institution of new training techniques, is it possible for ourpotential enemies to do the same? In other words, should the U.S.be concerned about potential “training surprise” in the same waywe have been watching for technological surprise.

In pursuit of an answer we requested the Defense

Intelligence Agency to describe the state of training of potentialadversaries and whether they would recognize if one wereinstituting the equivalent of our CTC training. The answer wewere given was that the rest of the world is too destitute to dotraining well. We were left with the impression that, like theacquisition community, they did not view training as somethingthat could make order-of-magnitude performance changes.

We later raised the issue with the National Intelligence

Officer for Conventional Military Issues, and he convened agroup of senior intelligence analysts, explicitly tasking them toexamine the issue of possible training breakthroughs. Theycorroborated the Defense Intelligence Agency (DIA) estimate thatmilitary training in most of our potential adversaries is poor.

They did point out that small groups of special forces in

many countries are well trained and competent, even though thebulk of their forces are not. None of the special forces use theCTC approach to training, but by persistent and continual use ofmore conventional training they succeed in creating competentelite forces. North Korea’s use of South Korea as a trainingground for their commandos comes closest to the CTC paradigm,although, in this case, since the consequences of failure are morefatal than in a CTC, this comes closer to battlefield Darwinism.

The National Intelligence Office’s (NIO) ad hoc group

did, however, identify an additional instance of training surprise.(The U.S. experience with Top Gun and in Desert Storm areothers, although the devastating consequences of those surpriseswere felt by our enemies, not by us.) This example occurred in theCroatian armed forces in 1993 and 1994. In the space of 1 year,with the help of a U.S. consulting firm, Military ProfessionalResources, Inc. (MPRI), with unusually strong political supportfrom the top, and with adequate funding, the Croatians built aforce that drove the Serbs out of their territory. They surprised notonly their enemies but the rest of the world as well.

Croatia’s success was the result of exceptional

circumstances including not having an existing military to resistchanges that made for effective training.

DSB Task Force on Training Superiority and Training Surprise

rec/DTI - dsb training tf/ 12/15/00/page 21

Red Training & Training 

◆CTC training culture can be learned

!E.g., U.S.-trained Kuwaiti pilots benefit from Red F

◆Initial Intelligence Community (IC) persp

!Potential adversaries are destitute and cannot !DSB saw no initial evidence that IC would det

◆NIO (Conventional Military Issues) conv

assembly of senior intel analysts to exam

"They corroborated the ‘rest of the world is destit"Potential adversaries are not embracing CTC app"They identified a third example of training surpr

# (first example is Top Gun/Viet Nam; second is NT

"Their collective answer was: an NTC-like center "Not clear to us that they would see signs of the s"Export licenses for training technology and syste

◆The DoD should request a training breakt

We asked whether it would be possible for others to institute training 

proficiency changes our CTCs give us. The answer is yes: it has been do

cultural issues. The intelligence community does not , however, ro

Successes such as this are likely to be rare.The group assembled by the NIO would notice the implementation of a CTadversary. We worry, however, that since they were assembled on a one-twill fade with time. Nor were we convinced that the intelligence communwhat we have called in this report the second training revolution. We recoSECDEF should request a similar Training Surprise conclave annually to breakthroughs may be possible in training as well as in technology.
Recommendations
 We had hoped to capture the essence of CTC training andrecommend that it be bottled and exported electronically to otherparts of the Services and to joint forces. We still so recommend,but find more urgent the restoration of the existing CTCinfrastructure and its continual updating to meet the new threats.

We believe (and were told by CTC operators) that air-toair warfare training requires a robust force of “red” aircraft and

pilots. Temporary-duty pilots in “similar” aircraft will not suffice.The Air Force has virtually no dedicated red air force and theNavy’s will soon be worn out. We recommend that a joint force,possibly contractor supplied (aircraft & pilots), be created andadequately funded.

This will not be enough. The primary threat in America’s

recent air battles has been from integrated air defenses composedmostly of ground-based systems. The air CTCs no longer havethreat-representative “red” systems. A top Service priority shouldbe to keep all CTCs’ red forces current.

In acquisition we recommend that each development

program be required to designate a “training subsystem” chargedwith creating the infrastructure to ensure that the hardware willhave trained operators and maintainers throughout the life of thesystem. This training subsystem should be funded with acquisitiondollars and its development should have equal priority with anyother vital subsystem. The proficiency delivered by training andits long and short term costs should be traded against hardwareperformance.

The training subsystem must be tested as well. We

suggest that the operational test and evaluation (OT&E) processdetermine whether the training approaches developed by theacquisition program will work. One way is to take a unit that hasdone no special OT&E work-up, subject it to the training system,and evaluate its performance using the tested weapon. Thisapproach should prevent situations like the first submarine withARCI modifications reporting “not ready for sea” due to a failureto consider training during the development. It should also detectbefore-hand situations such as the one in which for 10 years amissile system was used at less than half its potential rangebecause of failure to provide for training during acquisition. USD(Personnel and Readiness) should sit on the Defense AcquisitionBoard to ensure that these issues are addressed.

Our final recommendations relate to raising training

consciousness throughout the military. Training performanceneeds to be reviewed at the highest levels.

DSB Task Force on Training Superiority and Training Surprise

rec/DTI - dsb training tf/ 12/15/00/page 22

Recommendation

◆ Services restore Air & Ground CTC Infrastr

"Upgrade opposing forces to meet new kinds of thr"Consider commercial supplier for joint air aggress

◆ Services & JFCOM report to DEPSECDEF ho

paradigm to additional forces and new/join

◆ DEPSECDEF task Services and CINCs (for j

yearly training assessment scorecard

"Covering training status both for deployed forces & "Endorsed by CINCs and, where appropriate, by a "Service-chosen format, but to include: performanc

◆ DEPSECDEF & CJCS request similar yearly 

training from Intelligence Community re: train

◆ For each new acquisition, define a Trainin

with other subsystems & funded with acquisit

"USD(AT&L), DEPSECDEF task DoD & Service O

Subsystem in final OT&E by training and testing a 

◆ USD(P&R) provide oversight on DAB for trai

The CTCs are the “crown jewels” of our first training revolution; they n

and future training can be made if a training subsystem is integral to 

deficiencies will be harder to ignore if a formal report 

The Pentagon decision-makers need to be reminded often of the leverage The Services should be held accountable for the third leg of their Titlreported upon by the CINCS who are charged with this function. Thimportant. What is important is that in the process of generating it, thmeasures of training effectiveness. We would hope that in the processwould migrate up through the units and into joint forces and that, with sucheld accountable in new ways for training performance.

In a similar way the Intelligence Community should report at a v

of training in the rest of the world with emphasis on all forms of training b
 The impending training revolution will take currentknowledge of learning behavior and apply to it the electronicrevolution in order to deliver the right skills at the right time andplace. It will not be enough to simply let electronic technologyproponents lead with new and amazing gadgets; an understandingof how people learn is vital. Computer technologists believe, asdo most of us, that they know what is needed to teach skills.Unfortunately, much of what we know is wrong. Both learningtheory and technology are required. This motivates ourrecommendation that the Defense Advanced Research ProjectsAgency create a new office and institute a research program inhigh-payoff training technologies.

Fomenting the next training revolution will require that

the DoD think about training in a new way: not reactive butproactive. Implementation of the first recommendation in theviewgraph would force the personnel and acquisitionestablishments (man and equip) to consider how to trade offhardware against training costs to deliver the most effective force.

We recommend a goal of moving 50 percent of

residential instruction out to the units in 5 years as well asconverting most of the remaining schoolhouse training tocomputer-based, self-paced, collaborative courses. To get thisstarted we recommend the creation of several Advanced ConceptTechnology Demonstration (ACTD)-like prototype programs.Perhaps they should be called Advanced Concept TrainingDemonstrations, although they would have a substantialtechnological content.

An ACTD joins together reasonably well understood

concepts into a pilot demonstration that has clear use to themilitary. It does not end with the demonstration, but leaves behinda residual capability that is supported for several more years. Thisgives the Service time to adopt the capability if desired. Onceinitiated, an ACTD should have stable funding throughout its life.Pilot programs created under this recommendation should havesimilar structural characteristics.

We recommend a second class of ACTD-like pilot

programs to demonstrate the value of the concept of several kindsof intelligence. This should improve the coupling of training tothe individual and allow better predictions of success in theServices, thus widening the pool of applicants. Each Serviceshould explore this approach, possibly with the aid of DARPA.Although not discussed elsewhere herein, we became concernedthat the issue of interoperability within and among the Services

DSB Task Force on Training Superiority and Training Surprise

rec/DTI - dsb training tf/ 12/15/00/page 23

Recommendation

◆ USDs (P&R) & (AT&L) foster the second trai

!Provide quantitative evaluation:

"USD P&R, AT&L recommend resource realloca

pers & training functions to achieve best trained f

!Support a goal to move 50% of schoolhouse 

training in 5 years. To initiate this change:

"USD(P&R) fund pilot program in each Service to 

from classroom-based to self-paced learning by 

"USD(P&R) fund 2nd pilot program in each Servic

programs from residential to unit-based instruct

"Services nominate courses. P&R fund & develo

!USDs (P&R)&(AT&L) establish (6.3) PE for tr!DARPA create a new office and research pro

training and human performance technologies

!Services institute ACTD-like pilot programs in 

development using multiple kinds of intellig

◆ SECDEF designate ASD or DUSD (existing 

Service & joint training performance. Service

The value of new training approaches needs to be demonstrated to the 

training effectiveness and retention can be enhanced by testing for more 

needs to be in charge; a standard DSB recommendation, perh

may become a downfall of future warfare. Innocent changes in one syperformance of the whole. This deserves a separate DSB review. 

Finally, there is the question of who is in charge. We hav

recommendation to all problems is: put someone in charge and give him this principle, cliched though it may be. Training of the kinds discussed current administrative structure; it will remain reactive unless there is a head of a new office or the recipient of a new tasking to an existing office
Conclusion
 This viewgraph is the same one you saw at the beginning,but the accompanying words are different.

Our task force’s principle finding is that the United

States military enjoys a huge training superiority over ourpotential adversaries. This second superiority is at least asimportant to warfare performance as is America’s betteradvertised technological superiority. We should not rely ontechnological superiority alone. It could not bring victory in VietNam, nor is technology alone likely to be sufficient for futurevictories. Since Viet Nam, actions by a few foresightfulindividuals caused our air forces and Army ground forces to adopta new form of warfare training that has created a trainingcompetence complementary to our technological competence and,in part, supported by it.

We cannot rest on our laurels. We would need to reduce

the cost of training even if nothing in warfare were to change.Warfare will change and training must change with it or we willbe unable to fight our future combat systems, our JV-2010/2020forces, or even maintain logistics systems that sustain our newagile and flexible forces.

Without a second revolution in training affairs, the

revolution in military affairs will not be supportable. That newtraining revolution is ripe for the picking; there is an emergingquantitative understanding of how to develop effective trainingapproaches, and the electronic revolution now makes affordabletheir widespread application. These factors include individualizedinstruction, direct feedback on performance, beating the forgettingcurve by delivering training at the time and point of need, andcollaborative and self-paced learning.

However, unless we make structural changes to the DoD,

the newest training revolution won’t succeed until long after it isneeded. If we fail to make those changes, training will remain anafterthought, something slapped together ad hoc to addressfailures like those that occurred in the I-HAWK or ARCIprograms. Such failures will be paid for by the Service memberswe send into harm’s way and will waste much of the hard-wonresources spent on acquisition of new (and old) weapon systems.

The structural changes we recommend hinge upon

making training issues routinely visible to those who write checksin the Pentagon. There is no COTS source for advanced militarytraining nor is there a large industrial lobby to remind decisionmakers about the importance of training. This lack of externalreminders makes structural change all the more important both to

DSB Task Force on Training Superiority and Training Surprise

rec/DTI - dsb training tf/ 12/15/00/page 24

Summary

◆Our uniquely American Training Superio◆JV2010/2020 future will require more trai◆Training failure will negate hardware prom◆A second revolution in training is needed 

!This new revolution should be able to pay for it

"The incentive structure in the DoD won’t foster th

# A central cause is that training performance is not 

◆Training should take its Title 10 seat with 

!Restore & expand upon crown jewels of curren!Establish and test co-equal training subsystem !Raise OSD/Acquisition training conscience:

"Services & CINCs deliver annual training report "Designate ASD/DUSD to be held accountable for 

!Foster the second training revolution by establ

"ACTD-like pilot programs in computerized self-p"An advanced training research program element"DARPA office to develop high payoff training/hu

◆DoD & Intel Community act to detect & a

The right kind of training can have electrifying effects on performance

Services and even that capability is eroding. We must do more if we wan

warfare. We won’t get there on the present course. Worse, there 

preserve our training superiority and to prevent training surprise from our 

The key recommendation for fixing the present is to devote more 

U.S.’s first training revolution, the CTCs, to permit JV2020 kinds of trainfixing the future is high-level training report cards. It doesn’t matter in whintelligence community tells the SECDEF about the state of training, whadelivered. The attention arising from these reports should aid implementatsustain recognition of the extraordinary value of training to winning wars.
DSB Task Force on Training Superiority and Training Surprise

rec/DTI - dsb training tf/ 12/15/00/page 25

Last Words

Training counts

Warfighting success is as dependent upon the proficiency of people as it i

upon the hardware with which they fight.

We need training superiority as much as we need technical superiority.

We don’t count training

We measure process, not proficiency,

and what you don’t measure or report, you can ignore.

Without structural changes in the DoD, training

won’t take its place at the table with man & equip

Unless it does, we will negate much of the promise of the Joint Vision

warfare transformation.

If it does, we will be able to maintain and expand our training superiority

without significant additional cost.

There is no military-training complex to force emphasis on training. Today that emphasis comes from the dedicated hardworking sol

marines in the field. Our key recommendations relate to making training visible at higher levels. If we are to restore the first training 

we are to upgrade them to meet the current threat, if we are to expand them to support JV2010/2020 warfare, and if we are to fo

revolution, the DoD must change.

This is the last chart. If, after all of the foregoing text, this chart doesn’t stand by itself, there is nothing here we can do 
Appendix A

Terms of Reference


Appendix B

Acronyms Used
ACTD Advanced Concept Technology Demonstration

AFB Air Force BaseAFHRL Air Force Human Resource LabARCI Acoustic Rapid COTS InsertionASD Assistant Secretary of DefenseCINC Commander in ChiefCJCS Chairman Joint Chiefs of StaffCOTS Commercial Off The ShelfCTCs Combat Training CentersDAB Defense Acquisition BoardDARPA Defense Advanced Research Project AgencydB DecibelsDEPSECDEF Deputy Secretary of DefenseDIA Defense Intelligence AgencyDMT Distributed Mission TrainingDoD Department of DefenseDSB Defense Science BoardDUSD Deputy Under Secretary of DefenseEW Electronic WarfareGM General MotorsGRE Graduate Record ExaminationIADS Integrated Air DefensesIC Intelligence CommunityIETMs Integrated Electronic Technical Manuals
IMAT Interactive Multi-Sensor Analysis Trainer

IOC Initial Operational CapabilityJFCOM Joint Forces CommandJTASC Joint Training, Analysis, and Simulation CenterJV 2010 Joint Vision 2010JV 2020 Joint Vision 2020MPRI Military Professional Resource, Inc.NAS Naval Air StationNIO National Intelligence OfficeNSF National Science FoundationNTC National Training CenterOPFOR Opposing ForceOSD Office of the Secretary of DefenseOT&E Operational Test and EvaluationR&D Research and DevelopmentRMA Revolution in Military AffairsSECDEF Secretary of DefenseTOW Tubular Optical WeaponUSA United States ArmyUSAF United States Air ForceUSD (AT&L) Under Secretary of Defense for Acquisition, Technology, and

Logistics

USD (P&R) Under Secretary of Defense for Personnel and ReadinessUSN United States Navy