# Presentation: 396429

## Models of Human Expertise as Engineering Blueprints:

- Department of Psychology
-  Carnegie Mellon University

- Cognitive Systems:  Human Cognitive Models in System Design
- June 29 - July 1, 2004

**Models of Human Expertise as Engineering Blueprints:**
** ****Applications to Landmine Detection**

**Notes:**
Goal:  Present evidence suggesting that expert models have practical value as blueprints for engineering human skills

## Overview

- Theme:  The utility of expert models for developing human skills
- Context for investigation: Landmine Detection
- 	The Landmine Problem
- 		Military & Humanitarian Perspectives
- 	Operator Training Development Projects (2)
- 		1.  PSS-12
- 			Expert Model
- 			CEBES Training Test Results
- 		2.  Handheld Stand-off Mine Detection System (HSTAMIDS now PSS-14)
- 	   		Expert Model
-     	   		CEBES Training Test Results
- Impact
- Conclusions

## Genome Map : Genetic Engineering  ::  Expert Model : Cognitive Engineering?

**Notes:**
Because expert models explain  “best practices” in a specific domain,
and knowledge acquired through experience is the foundation of expertise,
and the supply of top personnel in important domains is always too low,
consider the practical application of expert models suggested by this analogy:

## The Landmine Problem:  Military Perspective

- “Mines have always comprised a large part of the total [ground] threat... and the proportion of casualties due to mines has increased steadily in every armed conflict since World War I.”
- “Mines are a major threat in all types of combat and will be _*the*_ major threat in Operations Other Than War (OOTW), which are expected to be the most likely mission for US forces in the future, as well as humanitarian demining.”
- “The widespread employment of landmines threatens to neutralize US advantages in firepower and mobility by severely limiting our ability to maneuver and disrupting our tactical synchronization.”
- “The threat will worsen as the worldwide proliferation of land mines continues.”
- 	- Hambrick & Schneck, 1996

*Recent after-action reports from several theaters suggest that this assessment underestimated the landmine threat in the contemporary operational environment.*

## Basic Types of Landmines

- _Purpose_

- Anti-Personnel
- Anti-Tank

**Notes:**
Metal detection technologies have been used since WWII for landmine detection.  The advent and dispersion of mines with plastic bodies and
minimal metallic content in the early 1990’s has substantially increased the threat that these weapons pose - both to the military and afflicted civilian population.

## AN19/PSS-12 Mod 2

- Handheld metal detector fielded by U. S. Army, 1992
- Standard equipment used for mine detection, 19,000 in U.S. Army inventory
- Uses electromagnetic induction for sensing metal components of mines...
- ...also  responsive to metallic objects and conductive soils

**Notes:**
Mines with minimal metal content pose a major problem for the PSS-12.  US Army field tests conducted at Ft. Hood in 1991 prior to fielding showed that soldiers detected 3.38% of the M14 targets with what was considered state-of-the-art equipment.

## Performance of US Army Combat Engineers Ft. Leonard Wood, October ‘99

**Notes:**
Test results from 20 U.S. Army Combat Engineers using PSS-12.  All had recently completed AIT at USAES, which included training on use of the
PSS-12.  Green bar shows the performance standard for the PSS-12 specified in the ORD.  The red dotted line indicates the presumed standard for detection of low metal mines.
Error bars represent 95% Confidence Intervals.

Results indicate the locus and extent of the contemporary landmine threat - LM mines.

## Handheld Stand-off Mine Detection System (HSTAMIDS) Prototype

- High-tech, dual-sensor replacement for PSS-12
- Sensors:
-    1.   Electromagnetic Induction
- 2.  Ground Penetrating Radar
- $38M for tech development over 9 yrs.

**Notes:**
In 1998 the HSTAMIDS prototype represented the culmination of the technology development effort initiated in the early ‘90s to address the LM mine threat.

## HSTAMIDS Prototype, Summer 1998

**Notes:**
Results show that LM targets remain a challenge. Army and PM-MCD convened a “Red Team” of military, government, private sector, and academic personnel to evaluate the substandard  results.  Operator training was identified as a major contributor to the results. A CEBES approach to training development was 
funded and included in an Extended Product Risk Reduction Phase of HSTAMIDS development.

## Legacy of Mine Warfare:  Problem Dimensions  

-  2,500,000 emplaced per year/80,000 removed per year (1994 est.)
-  1:1 emplaced/removed ratio achieved in 1998, but later conflicts in Kosovo, Bosnia, Chechnya, Afghanistan, Colombia, Pakistan, India, Iraq have reversed progress
-  $300 - 1,000 removal cost per mine
-  $3 low-end cost for purchase and emplacement of AP mine
-  450 years estimated for clearance of 45-50,000 mines

- Legacy of Mine Warfare:  Problem Dimensions

**Notes:**
The longevity of landmines contributes to the medical, psychological, economic, social, and political problems that civilians encounter for decades after the conflicts that led to landmine deployment have ceased.

## 15-26,000 casualties per year

- 1/3 result in death
- 1/3-1/2 victims are children
- $100 - 3,000 prosthetic cost per victim
- 1-3 years medical care and rehab

- Legacy of Mine Warfare: Human Costs

## Considering that emphases on purely technological solutions had not

- solved the problem,
- it was plausible that efforts at developing better operators might improve detection capability.
- Basic research has shown that instruction based on expert models facilitates novices’ skill acquisition...but,
- ...is science on human expertise sufficiently mature to support engineering of human skill in a high-stakes domain?

**Notes:**
Why apply cognitive science?  Investments in applying advanced technologies to equipment development had not succeeded alone.  Anecdotal evidence for substantial individual differences in operator performance suggested skill as potential factor in detection performance.

## General Approach for Cognitive Engineering Based on Expert Skill (CEBES)

- 2.  Develop information-processing model of expert skill
- 3.  Use model to develop instruction for novices
- 4.  Test instructional program

## Sample of Expert PSS-12 Operation

- Detection of an M14  (AP-LM) @1”

## Model of PSS-12 Expert’s Procedures

**Notes:**
Model captures nearly all behavioral variability in test performance. Key differences between expert and novice military personnel in techniques and
Interpretation of output signals. Experts prospectively search for familiar spatially-distributed patterns of output signals , I.e., mine signatures, that they recognize as those produced by mines.

## M19	

- M19

- M14

- VAL69

- M14

**Notes:**
Patterns, I.e., signatures or “footprints” of AT-LM (M19) , 2 AP-LM (both M-14s), and AP-M (Valmara 69) mine produced by operator’s investigation during testing on handheld lanes, Range 71A, Ft. A. P. Hill.

Red marks identify approximate location of target center. Orange marks represent locations where onset or offset of PSS-12 signal occurs; markings are artifacts of the analysis process. Arcs are hand-fit to show contours defined by on-off points.  Areas inside of arcs yield constant PSS-12 output signals.

Note similar geometries and correlation between metallic content and size of signature.

Patterns generalize to other targets of same mine type under similar burial depths and media, however, metallic clutter can make distort and, in some cases, obscure mine signatures.

## Investigation & Decision:  Expert Pattern Acquisition and Recognition 

- Signature image
- synthesized in local investigation

- Category representation
- for high metal mines

**Investigation & Decision:  Expert Pattern Acquisition and Recognition**

## Training at USAES, Fort Leonard Wood, Fall ‘99

## Training Design Principles

- Content:  Train expert detection strategies and techniques
- Probability of Detection prioritized; minimal attention to target/clutter discrimination (FAR) and speed (ROA)
- Maximize hands-on practice with feedback (12-15 hrs/soldier)
- Hierarchical design, based on expert goal structure: - train basic subskills; when proficiency demonstrated, drill and practice more complex “subroutines” that integrate basic skill components
- Communicate content in the form of goal-based productions, i.e., condition-action rules

## Performance of Experimental & Control Groups Initial PSS-12 Training Test at Ft. Leonard Wood, October ‘99

**Notes:**
Aggregate Posttest PDs:  Exp =0.94, Control = 0.31, equiv FARs (0.09FA/m2, Exp ROA 80 sec/sq-m vs Control 25 sec/m2
 Exp group PD higher in all mine categories except AP-M. Diff not statistically reliable.

Performance relatively high for exp and control groups for M targets.

Considerable between-group gap for LM targets.

PD for control group a matter of concern.

PD for exp group meets PSS-12 Operational Requirements Document criterion.

##  Training Effects Against M14s (Hardest-to-find AP-LM) 

**Notes:**
Training effects magnified; roughly 6-fold improvement in PD. Note equivalence in groups’ pretest performance.

## AT & AP Mines Used in Testing 

- AT-M

- AT-LM

- AP-M

- AP-LM-S

- AP-LM-L

## *

**Notes:**
Note:  Iniital test and replication in BASIC body armor executed at USAES, Ft. Leonard Wood, against simulated mine targets. Testing at Range 16, Aberdeen Proving Ground, employed actual mines shown in the previous slide. 

## Individual Soldier Performance for 3 Tests

**Notes:**
Robust treatment effect across individual operators.  All soldiers tested at APG achieved detection rates exceeding the PSS-12 ORD standard.

## NCO Trainers on CEBES-spec design training grid,Joint Readiness Training Center, Ft. Polk, LA

**Notes:**
Training site at Ft. Polk designed for CEBES training program. The training grid was employed for instruction and drill. Separate lanes were developed for pre- and posttesting.  

## JRTC Training Area Tactical Lane

**Notes:**
More challenging tactical lanes were built for generalizing basic skills.  Training grid shown in background.

## JRTC, Aug ‘01: NCO Performance (with 3-4 hrs. drill & practice)

**Notes:**
Instruction and drill time reduced 66-75% due to “revised” orders and weather.  Posttest effects attenuated relative to performance of initial “full” training at USAES.

## Performance of  10th MTN DIV TF 27th EN, trained  by JRTC O/Cs, Sept ‘01

- (1 hr. instruction & drill per soldier)

- Before After

- Before After

- Before After

- Before After

- Before After

**Notes:**
41st En BN trained 6 platoons of combat engineers from 27 TF immediated following NCO trainer training.
Positive posttest effects persist despite major reduction in instruction/drill time and inexperienced trainers.

## HSTAMIDS Prototype Training

- _Dual sensor system_:
- Electromagnetic induction
- Ground penetrating radar
- Sensor-specific processing algorithms
- _Non-trivial operational complexity_:
- _Perceptual Req_: 2 independent auditory output channels operating simultaneously, con-tinuous visual monitoring
- _Motor Req_: Sensors performance sensitive to search head movement parameters (e.g., head height, sensor-specific speed_s_, speed variability,  roll)
- _Cognitive Req_:  Construct integrated mental picture of target signature; eval similarity to abstract conceptual patterns

## HSTAMIDS Expert vs. M14

**Notes:**
Synchronous audio visual display used to analyze expert operator behavior. Video data capture directed by automatic tracking of search head by computer guided cameras.

System display also used to provide feedback to trainees on sweep motions and functional area coverage. Also, used by trainers to monitor performance and diagnose problems with technique.

## Model of HSTAMIDS Expert’s Procedures

- Either EMI or GPR
- output provide alert

- (develop EMI signature,
- then GPR signature, &
- integrate)

**Notes:**
PSS-12 model of detection procedures generalizes to HSTAMIDS expert.  Key differences are that (1) either MD or GPR outputs could serve as alerting signals, (2) investigation processes proceeds first with one sensor (usually MD) and then the other, and (3) more complex patterns, synthesizing MD and GPR outputs, are constructed, represented, and interpreted for making mine declaration decisions.

## Signature image

- Signature image

- Categorical
- representation
- “AT-LM”

- Investigation & Decision:  Expert Pattern Acquisition and Recognition

**Notes:**
Pattern complexity increases with info from GPR, but additional info also adds constraint on operator’s interpretation of patterns.  Converging evidence from the two sensors yields increased mine/clutter discrimination, ergo fewer false alarms.

## M14

**M14**
**AP-LM-S**

**Notes:**
Ribbon marks mine center. White markers: MD edges.  Red markers: GPR edges -  Note North GPR edge and South GPR edge could not be resolved

## M14

**M14**
**(AP-LM-S)**

**Notes:**
Tee marks mine center. Another M14 @ 1” in low veg.  Same qualitative MD-GPR pattern.

## M15

**M15**
**(AT-M)**

**Notes:**
M15 target in on-road training environment:  American-made anti-tank mine.
Yellow mark indicated mine center.

## AT-M

- <date/time>

- AP-M
- (excluding VS50)

- AP-LM-L

- AP-LM-S

- AT-LM

- 2.63”

- 2.34”

-  2.26”

-        23.6”

-    10.21”

- Operator Heading:

**Mean expert target signatures aggregated by mine class**

**Notes:**
Aggregate Expert Target Signatures:  Distances to MD and GPR edge points averaged over multiple observations. Error bars are 95% Conf. Ints.
Mine edges shown by solid black circles, centers marked by crosshairs. Signatures drawn to scale. Green points = MD signal edge; Red points = GPR signal edge; Green shading marks area within which MD output signal occurs.

Relative invariance of target patterns by class for expert. Invariance in few target categories simplifies training novices’ pattern recognition.  

Signature development drill designed to help trainees induce general pattern related to each category.

## HSTAMIDS CEBES Training Test Results:  April & June 2000

**Notes:**
CEBES operator training successfully boosted performance levels against the -LM threat relative to those observed in testing of prototype. Asymptotic performance against AT-LM and AP-LM-S targets permitted diagnosis of prototype design weaknesses and guided re-engineering efforts. The performance gains relative to initial results and stable, interpretable performance limitations observed justified funding further development and EMD phase.  

Operational testing of system in 2003 configuration with operators trained using the CEBES training program against 1096 mines with high and low metallic content showed  overall 98.7% detection rate, .008 FA/m2 FAR, and 3.2 m2/min ROA.

## Impact

-   training of indigenous humanitarian deminers in Jordan, Eretria, Djbouti, 2000-2001
-  AMC-FAST sponsored installation of permanent JRTC training site and trainer training for 519th MP BN and 10th MTN DIV 41st  EN BN deploying to Kosovo, August 2001
-  41st Combat EN BN trains 6 sapper platoons  for Bosnia deployment, Sept. 2001
-  326th EN BN 101st Airborne trained prior to Iraqi Freedom deployment, January 2002
-  US Army adopts CEBES PSS-12 training and techniques, February 2002
-  Colombian Army adopt CEBES PSS-12 training and techniques, July 2002
-  Soldiers given CEBES training currently use the PSS-12 in countermine operations in Iraq, Afghanistan, and Colombia.

**Notes:**
CEBES approach developed highly effective training for operators of currently-fielded and emerging handheld landmine detection equipment and thus substantially improved landmine detection capability.

## Impact

- Pre-production model PSS-14 and training deployed to Afghanistan, January 2002
- Army distribution policy packages training with equipment, 2002
- Operational test of production units at USAES yields 98.7% detection rate
- PSS-14 and training deployed to support Operation Iraqi Freedom, January 2003
- U. S. soldiers given CEBES training currently use the PSS-14 in countermine operations in Iraq and Afghanistan
- US Humanitarian Demining Program employs CEBES approach to operator training for Mine Clearing Cultivator Angola Live Ordnance Field Test, Fall 2002

**Notes:**
CEBES approach developed highly effective training for operators of currently-fielded and emerging handheld landmine detection equipment and thus substantially improved landmine detection capability.

## Conclusion

- APG

- Information-processing models of human expertise
-   describe/explain “best practices” in a specific domain in communicable form
-   offer practical blueprints for engineering important skills via instruction
-   can support scientifically-principled instructional design and assessment

**Notes:**

Development of hardware/software  sub-components of complex man-machine systems require and receive rigorous science and engineering.
Too often and sometimes with significant cost, intuitions are acceptable for operator training development. This work suggests an alternative approach is available.  Cognitive science has sufficient theory, principles, and methods to bring similar rigor to the development of the human components and thereby improve total system performance cost-effectively.