Technical Report 1124 

Phase I Final Report on an Intelligent Tutoring System 

for Teaching Battlefield Command Reasoning Skills 

Eric A. Domeshek

Stottler Henke Associates, Inc.

March 2002 

United States Army Research Institute 

for the Behavioral and Social Sciences

 Approved for public release; distribution is unlimited

 
REPORT DOCUMENTATION PAGE

1. REPORT DATE (dd-mm-yy)

March 2002 

2. REPORT TYPE

Final 

3. DATES COVERED (from. . . to)

December 2000 to June 2001 

5a. CONTRACT OR GRANT NUMBER

DASW01-01-P-0355 

4. TITLE AND SUBTITLE

Intelligent Tutoring System for Teaching Battlefield Command

Reasoning Skills -- Phase I Final Report 5b. PROGRAM ELEMENT NUMBER

0602785A

5c. PROJECT NUMBER

A790 

6. AUTHOR(S) 

Eric A. Domeshek (Stottler Henke Associates, Inc.) 

5d. TASK NUMBER

211 

5e. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

Stottler Henke Associates, Inc.

1660 S. Amphlett Blvd/Suite 350 

San Mateo, Ca 94402 

8. PERFORMING ORGANIZATION REPORT NUMBER

10. MONITOR ACRONYM

ARI

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES)

11. MONITOR REPORT NUMBER

Technical Report 1124 

12. DISTRIBUTION/AVAILABILITY STATEMENT

Approved for public release; distribution is unlimited. 

13. SUPPLEMENTARY NOTES

This report documents Phase I efforts and Phase II plans to develop an Intelligent Tutoring System for Teaching Battlefield

Command Reasoning Skills.

14. ABSTRACT (Maximum 200 words): 

Report developed under a Small Business Innovation Research Program 2000.2 contract for topic OSD00-CR02. The research

reported here aimed at the design of a Socratic Intelligent Tutoring System (ITS) for high-level battlefield command reasoning

skills. The ultimate goal of this research is to develop new ITS techniques and technology for teaching skills that cannot be 

taught as simple methods and procedures to be followed. Achieving expert levels of proficiency in high-level command

reasoning skills—whether for battlefield commanders or for executives in industry—requires extensive practice, coaching, and

feedback. Learners must be given a chance to drill on detailed and situation-specific knowledge, as well as high-level thinking 

habits and skills applicable across diverse situations. We studied exemplary command reasoning mentoring by observing 

tutoring sessions centered on Tactical Decision Games (TDGs). We analyzed those sessions to produce a first draft general 

model of tutoring actions. We also built a limited proof-of-concept prototype that exhibited many of the key behaviors 

identified. Taken together, the conduct and products of our Phase I work, along with our proven strong team of collaborators,

position us well to carry through on the ambitious work plan outlined here for Phase II. 

15. SUBJECT TERMS

Intelligent Tutoring Systems, Socratic Dialog, Battlefield Command and Control, Simulation-based Training, Artificial 

Intelligence, SBIR Report

SECURITY CLASSIFICATION OF

16. REPORT

Unclassified

17. ABSTRACT

Unclassified

18. THIS PAGE

Unclassified

19. LIMITATION OF 

ABSTRACT

Unlimited

20. NUMBER

OF PAGES

93 

21. RESPONSIBLE PERSON

(Name and Telephone Number)

Dr. Carl Lickteig

DSN 464-6928

U.S. Army Research Institute for the Behavioral and Social Sciences

ATTN: TAPC-ARI-IK

5001 Eisenhower Avenue

Alexandria, VA 22333-5600

 i
 ii
Technical Report 1124 

Phase I Final Report on an Intelligent Tutoring System for 

Teaching Battlefield Command Reasoning Skills 

Eric A. Domeshek

Stottler Henke Associates, Inc. 

Armored Forces Research Unit 

Barbara A. Black, Chief 

 

U.S. Army Research Institute for the Behavioral and Social Sciences 

5001 Eisenhower Avenue, Alexandria, Virginia 22333-5600 

February 2002 

Army Project Number Personnel Performance 

20262785A790 and Training Technology

Approved for public release; distribution is unlimited 

 iii
 iv
FOREWORD

Improvements in Army training and evaluation are an enduring concern of the U.S. Army

Research Institute for the Behavioral and Social Sciences (ARI). A related concern addressed in 

this effort is that military officers may have insufficient opportunity to apply their battle command 

reasoning skills in realistic battle command situations. One way to provide sufficient opportunity

is through an intelligent tutoring system (ITS) for training battlefield command reasoning skills. 

This Phase I Small Business Innovation Research Program (SBIR) effort targeted the 

design and limited demonstration of a Socratic ITS for high-level battlefield command reasoning

skills. The research goal was to develop innovative training methods for conceptual skills, 

particularly new ITS techniques and technology for teaching skills that cannot be taught as simple 

methods and procedures to be followed. The product goal was an ITS for training battlefield 

command reasoning skills that could be hosted on an interactive web site. Overall, the objectives

for developing such an ITS prototype include: anytime, anywhere tutoring; deliberate practice 

opportunities; standardized instructional procedures; and at least a partial answer to the growing

problem of limited expert human tutors. 

This research was part of ARI’s Future Battlefield Conditions (FBC) team efforts to 

enhance soldier preparedness through development of training and evaluation methods to meet 

future battlefield conditions. This report represents efforts for Work Package 211, Techniques and 

Tools for Command, Control, Communications, Computer, Intelligence, Surveillance, and 

Reconnaissance (C4ISR) Training of Future Brigade Combat Team Commanders and Staffs 

(FUTURETRAIN). Results of this effort were briefed to the Director of Bio-Systems, Office of 

the Director, Defense Research and Engineering. As a result of the Phase I success, the Phase II

effort was awarded and an ITS for conceptual skill training should be available for commercial 

application by September 2003. 

 MICHAEL G. RUMSEY 

 Acting Technical Director 

 

 v
ACKNOWLEDGEMENTS

This report reflects work performed under a Small Business Innovation Research Program 

2000.2 contract for topic OSD00-CR02. The project aimed at design of a Socratic Intelligent 

Tutoring System (ITS) for high-level battlefield command reasoning skills. What is reported here

reflects the efforts of a large team. 

I would like to thank our contract monitors at the U.S. Army Research Institute for the

Behavioral and Social Sciences (ARI) Fort Knox Field Unit. Dr. James Lussier had the vision that 

led to this line of research. Dr. Carl Lickteig was immensely helpful in managing all of the details 

of the project. Dr. Ted Schlechter was also quite supportive of our work. 

Our subject matter experts, U.S. Marine Major John Schmitt (Ret.) and Brigadier General 

Keith Holcomb (Ret.) were both central to the development of this project. We quite literally

could not have done it without them. Their tactical expertise, combined with their enthusiasm for 

and experience with the Tactical Decision Games (TDG) format (including development of the 

electronic distributed version of TDGs) was invaluable. Equally critical was their ability to recruit 

active-duty Marine Captains to serve as subjects of tutoring observations. It is a tribute to their 

reputations and recognized skills that they were easily able to provide as many volunteers as we

could use. I would also like to thank the four Marine Captains who volunteered as subjects for the 

Phase I observations. 

Our colleagues at Klein Associates, Inc., Karol Ross, and Rob Hutton were also very

helpful in designing and carrying out the research reported here. They contributed most heavily in 

the instructional observations, helping to develop the format and schedule, serving as observers of 

the actual sessions, preparing session transcripts, and providing analysis of the results. 

Special thanks to Dick Stottler of Stottler Henke Associates, Inc. (SHAI) for conceiving

this project, setting it up, and providing direction. And last, but certainly not least, I gratefully

acknowledge the efforts of Eli Holman who implemented the Phase I prototype system that is 

described in this report. 

 vi
PHASE I FINAL REPORT ON AN INTELLIGENT TUTORING SYSTEM FOR TEACHING 

BATTLEFIELD COMMAND REASONING SKILLS

EXECUTIVE SUMMARY 

Research Requirement:

Under extreme stress, time pressure, and uncertainty, commanders must assess complex,

ambiguous tactical situations, develop and prioritize goals, and pursue these goals by issuing

orders, reports, and requests that consider the situation, own assets and capabilities, and the assets, 

capabilities, and intentions of the enemy. High performance decision-making requires the

commander to apply detailed and situation-specific knowledge as well as high-level thinking

habits and skills that are applicable across diverse tactical situations. These skills include modeling 

a thinking enemy, using all available assets, and considering how the commander's fight fits into 

the bigger picture from friendly and enemy perspectives. 

These reasoning skills cannot be taught simply as methods and procedures to be followed. 

Although guidelines can help commanders achieve baseline levels of performance, achieving an 

expert level of proficiency requires: 

Extensive practice of command reasoning skills in a variety of situations to acquire and 

reinforce skills until they can be applied flexibly and automatically. This experience can 

include a combination of actual combat, live training exercises, computer-based and paperbased simulations, and mental exercises. 

Coaching and feedback from instructors to identify strengths and weaknesses in 

performance, encourage reflection, stimulate the student's thought processes, and enhance 

the student's reasoning capabilities. 

Coaching and feedback are typically provided by expert instructors during dialogs with 

students. An example of this type of instruction includes Think Like a Commander (TLAC)

materials (U.S. Army Research Institute, 2001). These TLAC materials are currently used in 

officer courses at Fort Leavenworth and Fort Knox. Another example is Tactical Decision Games 

(TDGs) (e.g., Schmitt, 1994). In both these forms of training, tactical scenarios are presented to 

learners who must then reach conclusions and make decisions. If the learner fails to take an 

applicable theme into account, the instructor asks the student questions, at first indirect, to 

encourage the student to reflect upon his or her thinking about a theme. 

An inherent limitation of mentored instruction is that the expert mentors are an expensive

limited resource that is infrequently available; when available their attention must often be shared

among too many learners for truly effective personalized interaction. Classroom instruction, such 

as Army officer training, adds the constraint that instruction is only accessible to learners who can 

attend the course on site, requiring significant time and travel costs as well as scheduling

difficulties. Automated, computer-based training can provide instruction “any time, any place.” 

However, the types of expert coaching and mentoring required to teach these command reasoning

skills require far more sophisticated reasoning than is possible with current computer-based

 vii
training (CBT) and web-based training (WBT) technologies that typically assess learners based on 

simple multiple-choice and fill-in-the-blank questions. 

Intelligent Tutoring Systems (ITSs) are computer-based training systems that mimic human

instructors to provide automated, one-on-one instruction. Intelligent Tutoring Systems encode

subject matter and instructional expertise to assess each learner's performance, knowledge, and 

skills to provide individualized learning experiences, adapted to each learner's skills, background, 

and learning preferences. Most ITSs provide a simulation or problem-solving user interface that 

enables learners to solve problems or pursue goals by entering a sequence of actions or decisions 

selected from a wide range of possible choices. The ITS applies subject matter expertise to 

evaluate these actions and decisions to assess the learner's knowledge and skills and then select 

and deliver appropriate instructional interventions. 

However, no ITS system developed to date supports a domain nearly as complex as 

battlefield command which requires sophisticated reasoning about a large number of goals, 

potential plans, variables, and constraints. Evaluation of each learner's solution is extremely

challenging because there is no single "correct" solution to each scenario. As Gen. George S.

Patton wrote, “There is no approved solution to any tactical situation.” Thus, developing an ITS 

for teaching battlefield command reasoning skills will require "pushing the envelope" in ITS 

technology along a number of dimensions, such as: 

Representing solutions and their rationale to complex battlefield command problems, 

Assessing the learner's reasoning skills by comparing each learner's solution with multiple

expert solutions and their rationale, and 

Implementing learner-mentor interactions that enhance high-level thinking habits, based on 

an assessment of each learner's reasoning, as well as knowledge structures that represent 

each expert's reasoning. 

Procedure: 

During Phase I, our objectives were to: 

1. Identify the kinds of tactical analysis knowledge that should be represented by the 

system to support the evaluation of learner solutions to tactical scenarios, in order to refine the 

system's student model. 

2. Explore and evaluate candidate interaction techniques that can achieve the types of

instructional goals currently achieved in learner/mentor interactions with expert human instructors. 

3. Identify the types of automated tutoring strategies needed to select and generate

effective instructional interventions based on a student model. 

4. Develop methods for authoring ITS scenarios by entering good and bad sets of 

decisions and rationale, and then annotating these solutions with skills that are demonstrated 

(positively or negatively) by each solution. 

 viii
5. Develop a limited, proof-of-concept software prototype that illustrates key ideas

related to learner/mentor interaction methods, tactical analysis knowledge representation, and 

interactive tutoring strategies. 

Our core activities during the Phase I effort included: 

 a. Study of prior work related to the project goals (e.g., reasoning skills training, ITS 

techniques and Socratic Tutoring, and the existing Think Like a Commander course). 

 b. Conduct, observation, and analysis of a series of Tactical Decision Game tutoring

sessions between active-duty Marine Captains and acknowledged tactical (and tutoring) experts. 

 c. Design and construction of a proof-of-concept Command Mentoring Intelligent Tutoring

System (ComMentor) prototype based on a single TDG scenario covered in multiple tutoring

sessions. 

With respect to our original objectives, activities ‘a’ and ‘b’ directly addressed objectives 1, 

2, and 3. Our early work on activity ‘a’ helped frame the problem. The extensive observations 

and analysis in activity ‘b’ ended up forming the primary basis for most of our work; identification 

of knowledge form and content, interaction techniques, and tutoring strategies all fed forward into 

later objectives. Activity ‘c’ directly addressed objectives 4 and 5. Our work on objective 4 was 

not only a necessary precondition to success on objective 5, but forms the groundwork for Phase II

work on task-specific authoring tools. Likewise the prototype produced for objective 5 not only

demonstrates the feasibility of our ideas, but also serves as a design base and source of lessons 

learned for our proposed further system development during Phase II. 

Findings: 

Our major Phase I accomplishments fall into four categories: 

1. Developing a deep understanding of the battlefield command domain, and of existing

techniques for training battlefield command reasoning skills (see “Findings Based on ARI

Direction and Materials”). 

2. Observing, analyzing, and classifying Socratic tutoring techniques for battlefield 

command reasoning skills (see “Findings Based on TDG Observations and Analysis”). 

3. Designing and building a limited proof-of-concept prototype system called ComMentor

that demonstrates an automated implementation of many tutoring techniques observed and 

cataloged above (see “PHASE I PROTOTYPE SYSTEM”). 

4. Developing a system design and work plan for the Phase II implementation of a full 

ComMentor prototype (see “PHASE II SYSTEM DESIGN” and “FUTURE WORK”). 

 ix
Utilization of Findings: 

In this Phase I project, we have developed and documented a detailed understanding of

how expert human tutors Socratically guide junior commanders through training scenarios in order 

to drill and improve their battlefield command reasoning skills. Further, we have demonstrated, 

through construction of a limited proof-of-concept prototype, that many of the observed tutoring

behaviors can be duplicated by an ITS. Finally, we have developed a detailed plan for how to 

proceed with Phase II development of a complete, evaluable, ComMentor prototype. As the Phase 

II proposal was awarded, the findings will be used to complete development of a prototype ITS for 

battle command reasoning skills by September 2003.

 x
PHASE I FINAL REPORT ON AN INTELLIGENT TUTORING SYSTEM FOR TEACHING 

BATTLEFIELD COMMAND REASONING SKILLS

CONTENTS 

 Page

INTRODUCTION ........................................................................................................................ 1 

Overview of Problem and Solution............................................................................................ 1

 Benefits of our Approach........................................................................................................... 6 

 Innovations................................................................................................................................. 6 

RELEVANT METHODOLOGIES .............................................................................................. 7 

Scenario-Based Instruction ........................................................................................................ 7 

Socratic Tutoring Systems......................................................................................................... 7 

Natural Language Input Processing........................................................................................... 8

Case-Based Situation Interpretation/Diagnosis ......................................................................... 8 

Application-Specific Authoring Tools....................................................................................... 8 

PHASE I OBJECTIVES AND ACCOMPLISHMENTS............................................................. 9 

Phase I Objectives...................................................................................................................... 9 

PHASE I ACTIVITIES AND ACCOMPLISHMENTS............................................................... 9 

ARI Direction and Materials......................................................................................................10 

TDG Observations and Analysis ...............................................................................................11 

ComMentor Design and Implementation...................................................................................11 

PHASE I FINDINGS....................................................................................................................12 

Findings Based on ARI Direction and Materials.......................................................................12 

Findings Based on TDG Observations and Analysis.................................................................12 

Findings From ComMentor Design and Implementation..........................................................17 

PHASE I PROTOTYPE SYSTEM...............................................................................................17 

Prototype Goals..........................................................................................................................18 

GENERAL DISCUSSION OF PROTOTYPE SCENARIO ........................................................18 

POINTS ILLUSTRATED BY PHASE I PROTOTYPE..............................................................21 

 xi
CONTENTS (Continued) 

 Page

 Multi-modal input (graphics & text).......................................................................................21

 Free-form input (natural language).........................................................................................22 

Multi-part input (multiple sentences)......................................................................................22 

Student solution gap-filling.....................................................................................................22 

Student solution classification ................................................................................................23 

Case topic/theme scripting......................................................................................................23 

General scenario tutoring cycle ..............................................................................................24

Tutor goal management ..........................................................................................................25 

Prompts for reflection .............................................................................................................25 

PHASE II SYSTEM DESIGN......................................................................................................25 

Phase II Technical Objectives....................................................................................................25 

PHASE II ARCHITECTURE.......................................................................................................27 

PHASE II COMPONENTS..........................................................................................................28

The ITS User Interface............................................................................................................28 

The ITS Core Processes and Knowledge Bases .....................................................................29 

ComMentor Authoring Tools .................................................................................................31 

FUTURE WORK..........................................................................................................................32 

Phase II Plans.............................................................................................................................32 

PHASE III TRANSITIONAL PLAN ...........................................................................................37 

Possible Markets for High-Level Socratic Tutors .....................................................................37 

SHAI’s Market Development Capabilities................................................................................38 

CONCLUSIONS...........................................................................................................................39 

REFERENCES .............................................................................................................................41 

APPENDIX A. Related Work......................................................................................................A-1 

 B. “Tanks on the Farm” Sample Session Transcript ..............................................B-1 

LIST OF TABLES 

Table 1. Phase I Analysis – Representative Taxonomy of Tutoring Moves ...............................14 

 xii
CONTENTS (Continued) 

 Page

LIST OF FIGURES 

Figure 1. ComMentor Phase I Abstract System Architecture Design ......................................... 4 

 2. ComMentor Phase I Abstract System Architecture Implementation............................ 5 

 3. Tanks on the Farm Introduction....................................................................................19 

 4. ComMentor Screen Layout w. ToF Map and Force Structures....................................20 

 5. ComMentor General Goal Structure for Scenario Tutoring Session............................24 

 6. ComMentor Phase II System Architecture. ..................................................................28 

 xiii
PHASE I FINAL REPORT ON AN INTELLIGENT TUTORING SYSTEM FOR TEACHING 

BATTLEFIELD COMMAND REASONING SKILLS

Introduction 

Overview of Problem and Solution 

Under extreme stress, time pressure, and uncertainty, commanders must assess complex,

ambiguous tactical situations, develop and prioritize goals, and pursue these goals by issuing

orders, reports, and requests that consider the situation, own assets and capabilities, and the 

assets, capabilities, and intentions of the enemy. High performance decision-making requires the

commander to apply detailed and situation-specific knowledge as well as high-level thinking

habits and skills that are applicable across diverse tactical situations. These skills include

modeling a thinking enemy, using all available assets, and considering how the commander's 

fight fits into the bigger picture from friendly and enemy perspectives. 

Problem. These reasoning skills cannot be taught simply as methods and procedures to 

be followed. Although guidelines can help commanders achieve baseline levels of performance, 

achieving an expert level of proficiency requires: 

Extensive practice of command reasoning skills in a variety of situations to acquire and 

reinforce skills until they can be applied flexibly and automatically. This experience can 

include a combination of actual combat, live training exercises, computer-based and 

paper-based simulations, and mental exercises. 

Coaching and feedback from instructors to identify strengths and weaknesses in 

performance, encourage reflection, stimulate the student's thought processes, and enhance 

the student's reasoning capabilities. 

Coaching and feedback can be provided by expert instructors during dialogs with 

students. An example of this type of instruction includes Think Like a Commander (TLAC)

materials (U.S. Army Research Institute, 2001). These TLAC materials are currently used in 

officer courses at Fort Leavenworth and Fort Knox. Another example is Tactical Decision 

Games (TDGs) (e.g., Schmitt, 1994). In both examples, tactical scenarios are presented to 

learners who must then reach conclusions and make decisions. If the learner fails to take an 

applicable theme into account, the instructor asks the student questions, at first indirect, to 

encourage the student to reflect upon his or her thinking about a theme. 

An inherent limitation of mentored instruction is that the expert mentors are an expensive

limited resource that is infrequently available; when available their attention must often be

shared among too many learners for truly effective personalized interaction. Classroom 

instruction, such as Army officer training, adds the constraint that instruction is only accessible 

to learners who can attend the course on site, requiring significant time and travel costs as well as

scheduling difficulties. Automated, computer-based training can provide instruction "any time, 

any place." However, the types of expert coaching and mentoring required to teach these 

command reasoning skills require far more sophisticated reasoning than is possible with current 

 1 
computer-based training (CBT) and web-based training (WBT) technologies that typically assess 

learners based on simple multiple-choice and fill-in-the-blank questions. 

Intelligent Tutoring Systems (ITSs) are computer-based training systems that mimic

human instructors to provide automated, one-on-one instruction. The ITSs encode subject matter 

and instructional expertise to assess each learner's performance, knowledge, and skills to provide

individualized learning experiences, adapted to each learner's skills, background, and learning

preferences. Most ITSs provide a simulation or problem-solving user interface that enables 

learners to solve problems or pursue goals by entering a sequence of actions or decisions selected 

from a wide range of possible choices. The ITS applies subject matter expertise to evaluate these

actions and decisions to assess the learner's knowledge and skills and then select and deliver

appropriate instructional interventions. 

However, no ITS system developed to date supports a domain nearly as complex as 

battlefield command which requires sophisticated reasoning about a large number of goals, 

potential plans, variables, and constraints. Evaluation of each learner's solution is extremely

challenging because there is no single "correct" solution to each scenario. As Gen. George S.

Patton wrote, “There is no approved solution to any tactical situation.” Thus, developing an ITS 

for teaching battlefield command reasoning skills will require "pushing the envelope" in ITS 

technology along a number of dimensions, such as: 

Representing solutions and their rationale to complex battlefield command problems, 

Assessing the learner's reasoning skills by comparing each learner's solution with 

multiple expert solutions and their rationale, and 

Implementing learner-mentor interactions that enhance high-level thinking habits, based 

on an assessment of each learner's reasoning, as well as knowledge structures that 

represent each expert's reasoning. 

Opportunity. We propose to proceed with development of a Command Mentoring

Intelligent Tutoring System (ComMentor) based on the observations and designs generated 

during our Phase I effort. As demonstrated by our Phase I prototype, ComMentor will present 

TDG scenarios to the learner, and then manage a Socratic dialog discussing the scenarios with 

the learner. The system will interpret and evaluate the learner's solution, and then interact with 

the learner to stimulate his or her reasoning skills and refine its assessment of the learner's 

strengths and weaknesses. A combination of artificial intelligence and intelligent tutoring

system technologies provides enabling technologies for this system, such as: 

Scenario-Based Instruction – Stottler Henke Associates, Inc. (SHAI) has pursued 

scenario-based (or case-based) approaches in numerous tutoring systems in which the 

system's knowledge base contains a library of problems, or scenarios, to present to the 

learner. Each scenario has associated pattern-recognizers that evaluate the learner's 

decisions and actions and outcomes of the simulation to estimate the learner's mastery (or

lack of mastery) of knowledge or skills. These mastery level estimates then provide the 

basis for selecting instructional interventions (e.g., lessons, feedback, hints, questions, 

and stories) that prompt the learner to reflect on his or her thinking and search for and 

 2 
consider alternatives. This scenario-based approach differs from expert system/model-tracing

approaches that generate problems, solve them, and then compare these automatically generated 

solutions with those of the learner. Compared to model tracing tutoring systems, scenario-based

tutoring systems can teach skills in more complex domains because the system applies scenariospecific knowledge to interpret each student’s actions. Since the range of likely problems and 

plausible solutions is much narrower for a single scenario than for all possible scenarios, 

scenario-specific knowledge can be used with relatively simple pattern recognition algorithms to 

interpret and assess likely student actions accurately within a given scenario, even in complex

domains. 

Application-Specific Scenario Authoring Tools – Any knowledge-based software 

system is only as powerful and up-to-date as the set of concepts, rules, cases, and other 

objects in its knowledge base. In large, complex, and continually evolving domains, such 

as battlefield command, practical development and maintenance of a high-performance 

knowledge base requires the ability to capture most of the knowledge directly from 

subject matter experts, without programming. Some tutoring systems provide "generic 

authoring tools" that present a standard user interface by which the subject matter expert

must enter knowledge into the tutoring system. In many situations, however, subject 

matter experts can enter scenario knowledge much more quickly and easily by using

knowledge representations and knowledge editors that are targeted to a specific 

application domain, such as military tactical decision-making. For example, our Acoustic

Analysis Intelligent Tutoring System (AAITS) under development for the U.S. Navy

captures the reasoning processes of undersea acoustic analysis experts by enabling them

to annotate frequency-analyzed acoustic data images (LOFARGRAMs) with features, 

sources, and final classifications using a point-and-click user interface. Thus, AAITS 

enables experts to maintain the tutoring system's knowledge base of scenario-based 

problems and solutions by carrying out a familiar analysis activity in a natural and 

intuitive way.

Rationale Capture – When evaluating each learner's solutions, it's necessary to 

understand the learner's reasoning behind his or her decisions. Using scenario-based 

instruction, it is possible to capture the reasoning (or rationale) of experts for each 

scenario, ask the learner questions that elicit his or her reasoning, and compare this 

reasoning with those of experts. The rationale underlying expert solutions can also be 

used to stimulate the learner's reflection and reasoning by identifying issues considered

by the expert, and then asking the learner questions that lead the learner to think about 

those issues as well.

Multi-Modal, Mixed-Initiative Conversational Interaction – Our Phase I prototype 

illustrates the interaction of graphical and textual input and output modalities; in Phase II

we intend to explore the possibility of supporting speech input and output as well (an area 

of research under current development at SHAI’s Seattle office in the Navy funded 

TALK project). With regard to interaction initiative, while it is true that Socratic tutoring

puts most control into the hands of the instructor, that control is highly conditioned on the 

actions of the student; further, the instructor must always be ready to respond to a direct 

query or topic shift by the student (though clearly sometimes the right response is to 

dismiss or defer such student initiatives). Again, SHAI has already begun to develop the

technology for control of mixed-initiative tutorial dialogs. 

 3 
At SHAI, we have a team uniquely qualified to extend and integrate the technologies 

identified above. During this proposed project, we plan to draw upon our past experiences, and 

those of other researchers in these areas. 

Solution. The ComMentor’s assessment of the student’s understanding and intentions 

will be based on several subsystems, initially conceived as an analysis pipeline that feeds its

outputs back to the user interface, but actually implemented in Phase I as an executive making

calls on two subordinate modules (see Figure 1 and Figure 2 for abstracted architecture diagrams

reflecting the Phase I design and actual implementation):

1. A Graphic Input Analyzer (GIA) that extracts situation descriptions from icon 

placements, area highlights, and path indications the student draws on the scenario map;

2. A Text Input Analyzer (TIA) that attempts to categorize the student’s typed input into 

one of an expected set of utterances (e.g., orders, reports, requests, questions, answers, or 

assertions). Those utterance expectations are modulated by the state of the tutorial interaction 

(e.g., the question or statement the tutor just made) and the prior student assessment. 

3. A Domain Inference Module (DIM) that applies rules from a domain knowledge-base 

to answer a range of relatively concrete questions, and derive higher level situation descriptions 

from the outputs of the GIA and TIA.

4. A Case Assessment Module (CAM) that uses the results of input analysis and 

inference to retrieve situation-specific cases (prototypical good and bad solutions) that serve as 

hypotheses about student understanding and intentions and help direct the tutoring interaction. 

5. The Tutor Interaction Module (TIM) that uses the results of all the modules described 

above, plus its own session history, to choose a next instructor move, and bias the future analysis 

of the other modules. 

Case

Assessment

Module

Tutor

Interaction

Module

Domain

Inference

Graphic Module

Input

Analyzer

Graphic

Output

Generator

Text

Input

Analyzer

Text

Output

Generator

Figure 1. ComMentor Phase I Abstract System Architecture Design. 

 4 
Tutor Interaction Module

Case

Assessment

Module

Domain

Inference

Module

Graphic

Input

Analyzer

Graphic

Output

Generator

Text

Input

Analyzer

Text

Output

Generator

Figure 2. ComMentor Phase I Abstract System Architecture Implementation. 

The CAM is a critical distinguishing component that allows ComMentor to achieve

appropriate assessments of student input while limiting the amount of knowledge engineering

required. A library of "good" and "bad" (fragmentary) solutions specific to a scenario is much 

easier for subject matter experts to provide than a complete set of general purpose analysis and 

assessment rules. The ComMentor will use similarities between the learner's solution and these

good and bad solutions to identify strengths and weaknesses, respectively, in the learner's 

solution and, therefore, in the learner's underlying reasoning skills. This initial evaluation will 

by no means be "complete." Rather, this evaluation will result in an initial model of the learner

skills employed within the scenario. This model will be refined during later interactions with the

learner such as questions that probe into the learner's reasoning. 

The ongoing Socratic interaction (using a combination of graphical and textual input and 

output) will refine the system's assessment of the learner's reasoning skills and stimulate the

learner's thinking. In addition to the Phase I interaction style, in Phase II we will also explore

other types of interactions such as: 

War stories – The tactical scenario can provide a launching point for presenting selected 

stories or testimonials that illustrate important, relevant lessons. For example, when the

tutoring system identifies a questionable decision made by the learner, the system could 

identify the underlying skill or thinking habit that relates to the decision and then suggest 

a relevant war story to the learner that relates to the skill or decision. The key technical 

challenge is indexing these stories effectively so that the system can identify stories that

are relevant to the student's solution or to the current state of the student-mentor dialog. 

Speech Input/Output – Spoken language is generally an easier and more natural 

medium of information exchange for people than is typed text. It is, however, a more

difficult medium for computers to cope with. Nonetheless, efforts in this direction are 

under way, including a project at SHAI. We note that Forbus, et al. (2000) has already

produced a prototype system that combines sketching and speech input to capture 

proposed course of action (COAs). However, feedback from military personnel 

 5 
concerned with operational applications has led them to de-emphasize the speech 

component of their work (Forbus, personal communication, 2001). 

Again, a key characteristic of our approach is our reliance on multiple recognizably good 

and bad scenario solutions, or solution fragments, provided by experts, to support comparison 

with and evaluation of the learner's solution. Use of multiple solutions increases the probability

that the system can match a portion of the learner's solution with a portion of an expert's. In

Phase II we will devote substantial effort towards developing authoring tools that enable domain 

experts to enter good and bad solutions (decisions and rationale) intuitively, using an interface 

that is similar to the interface used by learners. The idea is that experts could then annotate such 

solutions by identifying skills that are/are not demonstrated by the learner if the learner's solution 

matches part of the expert's good or bad solution. This scenario-specific analysis will, as 

sketched above, then be combined with other types of knowledge, including generic instructional 

strategies, scenario-specific instructional strategies, and tactical "common sense." 

Benefits of our Approach 

Benefits of our approach include: 

Intelligent automated instruction of battlefield command reasoning – As with most 

automated instructional systems, our approach will enable learners to learn "any time, 

any where," at less expense and with more scheduling flexibility. However, unlike CBT, 

WBT, and simpler ITSs developed to date, our approach will support sophisticated 

learner-mentor interactions that stimulate the learner's thinking skills for an extremely

complex domain: battlefield command reasoning.

Feasible scenario-based knowledge acquisition – Our scenario-based approach will 

enable the intuitive entry of subject matter expertise without programming, using

rationale capture techniques and application-specific scenario authoring tools. By

making it possible to create new scenarios quickly, it will be feasible to create a rich

library of training scenarios that covers a wide range of situations and learning objectives. 

By making it feasible to enter many solutions for a given scenario, the system will be

able to draw upon the perspectives of many different experts to evaluate each learner's 

solutions more accurately.

Adaptive, flexible instruction – Our system will employ both generic and scenariospecific instructional strategies and principle-based instruction to support instructional 

interactions that are highly adaptive to the learning needs of each learner. By carefully

separating the system's scenario-based tactical analysis knowledge from the system's

instructional strategies, it will be possible for different instructional strategies to exploit 

the same tactical knowledge in many different ways, to achieve diverse instructional 

goals, depending upon each learner's knowledge, skill levels, and learning preferences. 

Innovations 

The ComMentor will be the first implementation of an ITS that supports the teaching of 

such complex and crucial high-level skills. Bringing together the scenario and rationale

 6 
representation, the user interfaces, and the domain knowledge will stretch the limits of what has 

been done in ITS, and for that matter, in artificial intelligence generally. We believe that the 

practical implementation of tutoring systems for large, complex domains will be enabled by

avoiding "deep," expert system representations of domain concepts in favor of scenario-specific 

(case-based) knowledge representations that are "deep enough." Specifically, we believe 

strategic combination of rule based and case based reasoning, annotation of cases with rationales, 

and inclusion of multiple good and bad solutions and stories in the case library, together form a 

cluster of techniques that will enable a practical solution to a currently unsolved problem. 

Relevant Methodologies 

Scenario-Based Instruction 

The validity of the scenario-based approach has both intuitive appeal and empirical

backing. As early as 1940, Gragg (1940) argued for scenario-based instruction. By presenting

scenarios that illustrate the important principles, the learner can see how principles are applied in 

operational contexts and tasks. It also defeats the well-known problem of inert knowledge first 

described by Whitehead, (1929) and frequently validated by other researchers. Inert knowledge 

is information or principles that a subject knows and can recall, but which he does not apply

when the situation clearly calls for it. Scenario-based instruction (and related concepts such as

example-based instruction, anchored instruction, case-based instruction, simulation-based 

instruction, and situated instruction) overcomes this problem by showing learners the application 

of principles in an operational setting and forcing them to apply them as well. 

Socratic Tutoring Systems 

Socratic tutoring is a recognized style of instruction, often associated with expert tutors 

(e.g., Glass, 1999), and known to produce effective outcomes (e.g., Rose, Moore, VanLehn &

Albritton, 2000). Adopting the Socratic style is not a simplistic tutoring “trick”— it is not about 

syntactically transforming all tutor interventions into question form. Rather, it is best thought of 

as a bias towards having the student construct as much of their knowledge as possible, while also 

encouraging reflection and analysis. While there has been ongoing interest in automating

Socratic tutoring over the last three decades, there are currently only a handful of leading

laboratories with a discernable focus in this direction (e.g., Martha Evens at IIT, Art Graesser at

Memphis, Johanna Moore at Edinburgh, and sometimes, Roger Schank late of Northwestern and 

now Cognitive Arts).

One observation to be drawn from past and current work is that most developers see 

unstructured language input as a useful complement to a Socratic tutoring approach. The general 

hypothesis is that much of the value in Socratic tutoring lies in having students work out for 

themselves what is going on in a given problem (thus leveraging the benefits of constructivist 

approaches and “self explanation”) which would be somewhat short-circuited by pre-structuring

lists of possible answers (or answer components) as in a menu-driven input system. For these, 

and other reasons related to tutoring behaviors observed in our sample sessions we chose to 

pursue a natural language interface in our Phase I effort. 

 7 
Natural Language Input Processing 

Natural Language Processing (NLP) has been an active area of artificial intelligence

research since the beginning of the discipline. Much work in the field draws on well-established 

linguistic traditions—adopting a multi-phased analysis mechanism that starts by emphasizing

syntactic parsing. However, much of the most effective work in the field has taken alternate

approaches—emphasizing semantic analysis tied to some particular context of language use. 

This often leads to faster, more useful processing, that is robust in the face of real-world input 

(ungrammatical, misspelled, or telegraphic input). In fact, the dominant trend in Socratic 

tutoring language components is to emphasize more effective semantic analysis techniques (e.g., 

Brown & Burton, 1975; Glass, 1999; Wiemer-Hastings, P., Wiemer-Hastings, K., & Graesser, 

1999). We note, however, that Glass (1999) has effectively critiqued the use of Latent Semantic 

Indexing (LSA) based approaches (as in AutoTutor) as unlikely to be very effective in domains 

that go far beyond general discussions of terminology and definitions, because LSA matching is 

primarily sensitive to overlaps in word meaning rather than to structured relationships among

actions and situations so critical to scenario-based instruction. Our language interpretation 

approach is able to make appropriate use of the structure typically available in language. 

Case-Based Situation Interpretation/Diagnosis 

Our aim, in fact, is to go far beyond instruction on terms and definitions. We aim to tutor 

students in effective application of high-level reasoning skills in the complex domain of dynamic

battlefield command. Tuning our language processing approach to extract structure descriptions 

is not enough; all of our student modeling and assessment must be capable of making relatively

fine situational distinctions. This is where we believe the technology of case-based situation 

interpretation and diagnosis can be usefully applied. Our experience in building ITSs for similar

domains shows that constellations of fragmentary cases can be used to effectively diagnose many

student successes and failures in complex tactical scenarios. Furthermore, a case-based approach 

to building such recognizers tends to make it easier for domain experts to manage and maintain a

tutoring system over time. 

Application-Specific Authoring Tools 

We hold, along with most of the ITS community, that a critical component of real-world 

success for ITS technology will be the ability to provide tools that let domain experts take more 

control of production and maintenance of training systems. We have already noted that any

knowledge-based software system is only as powerful and up-to-date as the set of concepts, 

rules, cases, and other objects in its knowledge base. Knowledge in such systems tends to have a 

limited shelf-life as the world changes (e.g., new weapons and vehicles, new operational 

concepts, new doctrine) and goals for the system evolve (e.g., shift from focusing on 

conventional conflict to focusing on missions other than war). To remain effective in the long

run, a system must be maintainable and extensible; to remain cost-effective, as much 

maintenance and authoring as possible should be supportable by available staff and those with 

first-hand expertise in the domain. 

 8 
Raw manipulation of the underlying data structures that enable an ITS to function tend to 

require detailed expertise in artificial intelligence (AI) and ITS technologies. To enable domain 

experts to take on maintenance and authoring tasks, manipulations of system structures must be 

translated into terms that are meaningful in the domain. Furthermore operations must be 

clustered, sequenced, and modeled so that appropriate system modifications become easy, and 

invalid modifications become essentially impossible. Achieving these ends requires creation of 

application-specific authoring tools. Since we see this as an important consideration in every

ITS we construct, this is an area where SHAI has had extensive experience. 

Phase I Objectives and Accomplishments 

Phase I Objectives 

During Phase I, our objectives were to: 

1. Identify the kinds of tactical analysis knowledge that should be represented by the 

system to support the evaluation of learner solutions to tactical scenarios, in order to refine the 

system's student model. 

2. Explore and evaluate candidate interaction techniques that can achieve the types of

instructional goals currently achieved in learner/mentor interactions with expert human 

instructors. 

3. Identify the types of automated tutoring strategies needed to select and generate

effective instructional interventions based on a student model. 

4. Develop methods for authoring ITS scenarios by entering good and bad sets of 

decisions and rationale, and then annotating these solutions with skills that are demonstrated 

(positively or negatively) by each solution. 

5. Develop a limited, proof-of-concept software prototype that illustrates key ideas

related to learner/mentor interaction methods, tactical analysis knowledge representation, and 

interactive tutoring strategies. 

Phase I Activities and Accomplishments 

Our core activities during the Phase I effort included: 

 a. Study of prior work related to the project goals (e.g., reasoning skills training, ITS 

techniques and Socratic Tutoring, and the existing Think Like a Commander course). 

 b. Conduct, observation, and analysis of a series of Tactical Decision Game tutoring sessions 

between active-duty Marine Captains and acknowledged tactical (and tutoring) experts. 

 c. Design and construction of a proof-of-concept ComMentor prototype based on a single 

TDG scenario covered in multiple tutoring sessions. 

 9 
With respect to our original objectives, activities ‘a’ and ‘b’ directly addressed objectives

1, 2, and 3. Our early work on activity ‘a’ helped frame the problem. The extensive 

observations and analysis in activity ‘b’ ended up forming the primary basis for most of our 

work; identification of knowledge form and content, interaction techniques, and tutoring

strategies all fed forward into later objectives. Activity ‘c’ directly addressed objectives 4 and 5. 

Our work on objective 4 was not only a necessary precondition to success on objective 5, but 

forms the groundwork for Phase II work on task-specific authoring tools. Likewise the prototype 

produced for objective 5 not only demonstrates the feasibility of our ideas, but also serves as a

design base and source of lessons learned for our proposed further system development during

Phase II. 

ARI Direction and Materials 

Discussions during the project kickoff meeting held at U.S. Army Research Institute for

the Behavioral and Social Sciences (ARI’s) Fort Knox office, in combination with review of the 

Lussier, Ross, & Mayes (2000) provided our initial direction on the project. 

Much discussion focused on ARI’s efforts to develop the Think Like a Commander 

TLAC course, as an example of what training in battlefield command reasoning skills might look 

like. We were given access to initial materials on TLAC, including preliminary instructors’

notes being compiled by Col. Fontenot under contract to ARI at Fort Leavenworth. It was 

suggested that we might want to base our analysis and development efforts on the TLAC 

materials, and that we might want to follow a new direction ARI was establishing to retarget 

TLAC materials towards training Captains (rather than Colonels). 

Following up on this initial meeting, we were provided with some of TLAC scenario and 

vignette briefing materials, as well as videotapes of TLAC class sessions run at Fort 

Leavenworth. Ultimately, we were invited to join a group of other contractors in visiting Fort 

Leavenworth to observe TLAC courses for a day. We observed two classes, each about 50 

minutes long. Interestingly (disappointingly), from the perspective of our goal to develop a 

tutoring system, the TLAC class sessions involved between 10 and 20 students at a time, and 

spent only about 15-20 minutes on each vignette. As a follow-up to this trip, we received a more 

extensive packet of material intended to provide TLAC students with a detailed background 

scenario supporting interpretation of the several TLAC vignettes. 

Finding the available TLAC observations (video and live) limited in their usefulness for

our tutoring analysis purposes, we had been planning to stage our own simulated TLAC sessions 

using our domain expert consultants as instructors. Our experts, however, were more

comfortable working with their own TDG materials, since they found the overarching TLAC 

scenario (with its staggering wealth of documentation) difficult to master and not significantly

more conducive to enabling effective training in battlefield command reasoning skills. We had 

also been studying TDG material (e.g., Schmitt, 1994) as part of our project effort. Fortunately, 

ARI had made it clear that TLAC was being offered as an example and a resource, rather than a 

dictate, so we were free to take what we had learned from our studies of TLAC and transfer it to 

the TDG framework. 

 10 
TDG Observations and Analysis 

Our experience with TDGs began with a sample session during which staff from SHAI

and our psychology subcontractor played “student” to one of our subject matter experts as 

“instructor”; we played the “Tanks on the Farm” (ToF) scenario. This was a valuable experience 

for several reasons. First, it introduced us to the on-line distributed format and infrastructure that

our subject matter experts had developed for electronic TDG training; we found this to be a 

convenient, cost-effective, and pedagogically effective approach. Second, it introduced us to the 

particular TDG scenario that would ultimately become the focus for the bulk of our Phase I

analysis and implementation. Third, it demonstrated convincingly that we needed real military

personnel to play the role of student if we wanted to observe more interesting tutoring behavior. 

Based on their expertise and reputation as outstanding tactical instructors, our subject 

matter experts were able to recruit active-duty Marine Captains (themselves all tactical

instructors) to serve as students in our tutoring observations. In all, we held six observed 

tutoring sessions, each about an hour and a half in length. Our two experts took turns as lead 

instructor: generally the lead instructor would handle the first 45 minutes or so of the session, 

then the alternate instructor would weigh in for about 30 minutes covering topics they thought 

had been missed, and finally the lead instructor would close out the session. In all cases, one of 

our subject matter experts was physically on site with the student, while all other participants 

were spread across the country, observing manipulations of a scenario map on a shared 

whiteboard, and listening to audio over a teleconference line. Four different students were 

involved. Three different scenarios were covered in these sessions, offering a range of

complexity. We devoted three sessions to the ToF scenario, two sessions to a somewhat more 

complex “Defense of Glosson” scenario, and one session to the very complicated “Evacuation 

from Enniottu City” scenario. 

For all six of the formal observational sessions we had two observers from our 

psychology subcontractor, and at least one from SHAI. Staff from ARI at Fort Knox joined us in 

observing the final session. Video and audio recordings were made of the computer screen and 

phone conversations in all cases. Rough on-the-fly transcripts were created in real time. 

Ultimately, we focused our analysis on the three ToF sessions, compiling detailed annotated 

transcripts of the proceedings. Samples of these are included as Appendix B at the end of this

report. 

ComMentor Design and Implementation 

As the tutoring sessions progressed, we began to refine our ideas on the design of the 

Phase I ComMentor prototype. We identified a finer set of distinctions in the knowledge 

required to support robust tutoring. We identified a modular breakdown appropriate to these 

interacting forms of knowledge. We began the process of implementing the identified 

knowledge bases and processing modules. 

 11 
Phase I Findings 

The findings from our Phase I effort can be categorized under three divisions, mirroring

the activities just described: (a) Study of ARI directions and Materials; (b) TDG observations 

and analysis; and (c) ComMentor design and implementation. 

Findings Based on ARI Direction and Materials 

We took away a small, but important set of lessons from our early survey of the problem 

and prior work: 

1. There are vanishingly few opportunities for officers—especially junior grade officers 

such as Captains—to get one-on-one instruction in battlefield command reasoning skills. TLAC 

is a very innovative course, but it is available on only a quite limited basis. At the time we were

doing our study, it was only available to quite senior officers, only in large group settings, and 

only for small amounts of time. 

2. It is unclear that the depth of background provided by the overarching TLAC scenario 

that frames the smaller TLAC vignettes is strictly necessary if the goal is to provide sufficient 

fodder for interesting battlefield reasoning exercises. The TDGs are exceptionally concise in 

presentation, and needed contextual information is either plausibly assumed, or introduced by the 

instructors on an as-needed basis. 

3. Prior work in the context of TLAC had identified an interesting set of analytic levels 

(or scaffolding levels) at which battlefield command reasoning skills could be drilled: (1)

Situation Analysis—i.e., “What do you know?” “What do you think you know?” and “What do 

you need to know?” (2) Situation Response—i.e., Role-playing the issuing of orders, reports, 

and requests with feedback of relatively immediate replies and consequences; and (3) Context 

Analysis—i.e., analyzing the longer-term and second-order effects of decisions made to cope 

with an immediate situation, including formation of future policies to avoid recurrence of 

problematic situations, or to ensure adequate flexibility to deal with them when they arise. Our

own observations suggested adding two more levels of interaction: at the low end (0) Fact

Filling—i.e., having the instructor fill in missing factual information (e.g., on weapons 

capabilities) and on the high end (4) Behavior Modeling—that is, having the instructor model 

effective reasoning and behavior after the students have made their attempts. 

4. When attempting to model and simulate the actions of expert tutors in mentoring

students at a particular level of expertise, there is no substitute for observing real tutoring

sessions between real experts and real students. Short group sessions will not provide the 

material for modeling extended one-on-one interactions. Simulations with fake students will not 

elicit the same kinds of tutoring behavior as will interactions with appropriate students. 

Findings Based on TDG Observations and Analysis 

The TDG tutoring session observations were central to the entire direction and outcome 

of our Phase I effort. Beyond the project’s basic purpose, and the projected system’s form, and 

 12 
core underlying technologies, essentially everything about the Phase I prototype was determined 

based on observations of these sessions. Here, as a main result, we focus on our preliminary

taxonomy of tutoring moves as shown in Table 1.

Table 1 is broken into four main blocks: 

1. Basic Scenario-Based Socratic Tutoring: This block organizes the core set of 

moves the instructor typically makes when guiding a student through thinking about a particular

scenario. 

We start by dividing essential factual information into that which is peculiar to the

scenario at hand, and that which is general to the domain. Key instructor moves related to such 

facts include four active moves: (a) presenting the relevant facts to the student (e.g., introducing

and setting up the scenario), (b) correcting student misstatement of facts (either directly or by

querying them), (c) clarifying slightly misstated or incomplete facts (again either by statement or

query), and (d) recapping a set of facts. This last variation on fact presentation is especially

interesting, because it can embed a focusing and framing intent—that is by choosing which facts 

to include or drop from the recap, the instructor gets to focus the student on essential 

constellations of facts. It is characteristic of Socratic tutors to often adopt a more passive role

with respect to fact statement—that is to ask questions instead of make assertions. We thus 

consider the following pair of moves critical in our analysis: (e) probing for the student to state a 

fact, and (f) refining the probe to get the student to clarify an incorrectly or incompletely stated 

fact. 

 13 
Table 1. 

Phase I Analysis – Representative Taxonomy of Tutoring Moves 

1. Basic Scenario-Based Socratic Tutoring 

Present/correct/clarify/recap scenario facts 

Present/correct/clarify/recap domain facts (respond to student questions) 

Probe/refine for domain fact understanding

Probe/refine for scenario fact understanding

Probe/refine/lead/clarify/recap for scenario situation assessment 

What is key missing information? 

 What are possible alternatives? 

Probe/refine/lead/clarify/recap for decision details 

Probe/refine/lead/clarify/recap for decision rationale 

 Direct student action

2. Topic Management

 Instructor-driven

 Explicit vs. implicit 

Shifting vs. digressing vs. abandoning

 Student-driven 

Accepting vs. deferring vs. rejecting

3. Theme-Specific Tutoring (examples)

Focus on Mission & Higher’s Intent

Does higher know what you know? What should they know? 

Has your mission changed? If so, why and how? 

 Thinking Enemy 

What are they doing there? What is their goal? 

What might their plans be?

 Big Picture

What don’t you know in your situational awareness (SA)? 

 Visualization 

What will happen after this engagement? 

How will you shape the future battlespace? 

Contingencies and Flexibility

Hypothetical variants on base scenario

4. Alternate Sub-Session Formats 

 Role-play

Play at self vs. commander vs. enemy, etc. (varying levels of competence/personality)

 Behavioral modeling

 Theoretical discussion 

 Anecdotes 

 Motivation 

 Reflection 

Ask student to assess own performance (best, worst)

Ask student to recap key learning points (what would do different?)

 14 
Moving beyond the facts, a tutor/student interaction starts to focus on situation 

assessments (inferences), decisions (intentions), and decision rationales (reasoning processes, 

and decision justifications). Some, but not all, of the moves just identified for facts apply to 

these higher-order constructs. In particular, tutors, for the most part, do not present or correct 

assessments, decisions, and rationales. They are frequently observed probing for student 

statements of these constructs and refining their probes to get at details (e.g., “Why did you do 

that?” or “What were you thinking?” vs. “Why are you worried about that [protecting your left 

flank]?”). The goal is to get the student to think, and articulate their thinking. One novel move 

that appears in this context is the (g) leading question1

—a tutoring move that stops short of 

being a presentation or correction (at least syntactically), but more or less strongly influences the 

student to focus on particular issues or facts, or to see things a particular way. Once the student 

makes some higher-level statements, tutors may feel free to make clarifying or recapping

statements as they would with facts. Again, recapping—as a kind of editing—can have a 

powerful influence on the future direction of the session. 

We have included two sample generic questions in this first block illustrating approaches

to scenario situation assessment. One interesting point is that these questions are very generic—

not tied at all to any particular scenario, or any particular aspect of a scenario. Another point is 

that these questions could equally well appear elsewhere in the table—in particular as examples 

of pursuing particular battlefield reasoning themes sampled in the third block of Table 1. 

Essentially the same instructor behavior can be motivated in different ways, and be intended to 

serve different ends, depending on circumstances.

The final kind of tutoring move we include in this section is the explicit directive to 

manage student action. Typical examples would be “Issue your orders” or “Draw me a picture 

of that.” 

2. Topic Management: The second block of Table 1 simply acknowledges that an 

extended conversation, such as a Socratic tutoring dialog, has an internal topic structure, and 

there are often (overt or covert) moves to control the topic. We start by noting that a dialog, 

inherently offers the possibility for mixed control over issues such as topic; however, we

continue to look at things from the instructor’s perspective. 

Accordingly, we suggest that the instructor can make the following moves based on their 

own assessment of conversational flow: (a) shifting the topic in an orderly planned way (either 

explicitly, e.g., “OK. Let’s take a different approach. Let’s take a look at your tactical plan.” or 

implicitly, e.g., “What do you think about the way you’ve got your forces arrayed to take care of 

his push from the East?”), (b) digressing to temporarily address a sub or side-topic while 

 

1

 We note that leading questions may be used for purposes other than getting the student to see a point the instructor 

wants them to see (or at least can end up having other effects). They can also be used for diagnosis, or for setting

traps for the student to fall into—that is, for verifying tutor hypotheses that it is worth spending time drilling in a 

particular area. For example: “So you think the idea of the Assembly Area in Viettiville is still viable at this point?” 

 15 
planning to resume the current topic later, and (c) abandoning a line of discussion to deal with 

some other, presumably more important topic opportunity that arose in mid-discussion.2

We also suggest that an instructor can respond to student attempts to change the topic by either 

(d) accepting the new topic, (e) deferring the new topic for later discussion, or (f) rejecting the

new topic outright. We actually have very few examples of students trying seriously to redirect 

their tutoring sessions—a likely mark of their acceptance of the instructors’ expertise and 

authority, and of their willingness to be guided. It is an open question as to whether that bias

will carry over to the automated tutoring context.

3. Theme-Specific Tutoring: The third block of Table 1 shows how a set of relatively

generic (for the domain) questions can be identified for some of the specific themes central to

battlefield command reasoning (e.g., a sampling of the eight themes identified as underlying

TLAC). In any given circumstance, most of the questions identified here could be classified as 

to their purpose according to the scheme laid out in Block 1 above (Basic Scenario-Based 

Socratic Tutoring). It is, of course, possible to get arbitrarily more scenario-specific in such

questions. The point, however, is that it is often possible to identify specific tutor moves with 

specific curricular goals (in this case, eliciting and reinforcing certain critical identified aspects 

of battlefield reasoning thinking skills), rather than with abstract tutoring strategies. 

4. Alternate Sub-Session Formats: The final block of Table 1 highlights that even in 

the context of a primarily Socratic tutoring session, there is room to use a wide variety of 

pedagogical techniques. We observed our instructors using all the approaches listed there. Most 

obviously, role-play was a significant and adaptable tool in their kit. Students were variously

asked to play themselves (that is, commander of the tank battalion at the center of the ToF

scenario), their superior commander, or various enemy commanders. An interesting technique 

used to get students to consider alternatives and contingencies, and to model a thinking enemy,

was the suggestion of specific ranges of competence or personalities for enemy commanders. 

Often the play was aimed at specific issues or answering particular questions. Sometimes, the

play was extended to consider possible temporal sequences, and thus grew into small-scale lowfidelity micro simulations. In such cases, preliminary inspection suggests it is plausible to base

such plays on local scenario-specific scripts. 

As noted in “Findings Based on ARI Direction and Materials,” we observed our 

instructors sometimes engaging in behavior modeling—that is showing the student how the 

expert would do something—generally after giving the student a chance, or several chances, to

do the job themselves. Likewise, we saw them devote limited time to theoretical discussions of 

such important concepts as shaping the battle. On occasion, they used personal anecdotes to 

make points (e.g., I’ve been in tanks under heavy artillery fire, and it’s amazing how tough even 

a little bit of armor is.). 

 

2

 It may be questionable how frequently it is useful or possible to distinguish between digressing and abandoning—

that is how often an instructor really plans to abort a line of discussion, versus how often they simply get distracted

and don’t happen to remember to come back to the topic. Discussion with one of our subject matter experts suggests

that often what appears as the instructor abandoning a topic is really a response to subtle cues (e.g., body language) 

indicating that the student is not benefiting from the current line of attack. That could be either because the student 

already gets the point (so the current tutoring goal is accomplished, and it is time to shift focus), or because the 

student is not following the line of discussion (and so it is time to shift strategies, while still addressing the same

tutoring goal). 

 16 
 One point, that has only been suggested so far, is that there is a definite large-scale 

structure to an entire tutoring session. Clearly, it starts with a presentation of the scenario. The 

session generally proceeds through an elicitation of the student’s initial response to the initial 

scenario challenge. There is a range of possible responses that trigger focus on a range of

possible discussion topics. For any given scenario, some topics tend to come up earlier in the 

discussion, and some topics tend to show up late, if at all. Eventually the session ends, and in all 

cases we observed, the instructors closed by encouraging a period of reflection. That is, just as 

they had a reasonably tight script to launch the session, they had a script to end it as well, by

asking questions such as “How do you think you did?” “What do you think you did best?” “What 

do you think you did worst?” “What did you learn?” “What would you do different next time?” 

The above is clearly a preliminary analysis, but reflects the breadth of our Phase I

observations. The value of this kind of taxonomy, and of accompanying observations, such as on 

the gross structure of tutoring sessions, is to begin to provide a way to think about structuring an 

automated tutor’s behavior. 

Findings From ComMentor Design and Implementation 

As we moved from observation and analysis into design, we identified several other

aspects of the tutoring sessions that drove us to evolve beyond the implementation ideas 

presented in our Phase I proposal: 

We observed that combining text and graphics was very important to duplicating the 

kinds of interactions we were observing: sometimes a map manipulation can clarify an

otherwise ambiguous utterance, and sometimes a description of rationale can clarify an 

otherwise mysterious set of moves on the map. Accordingly, we are devoting substantial 

resources to supporting multi-modal interaction in the Phase I ComMentor prototype. 

We observed the use of simple “role-play” snippets (e.g., issuing of orders, consideration 

of hypothetical follow-on scenarios) and felt that separating the tutoring interface from a

“role-play” interface would help to clarify (both for the student and the system) what kind 

of interaction was going on at any given moment. 

We observed the importance of having a basic competence to reason about simple 

quantity, time, distance, and force issues (what was often referred to as “battlefield 

calculus”); it appears that while Captains can benefit tremendously from tutoring in 

battlefield reasoning skills, they often need support in managing the basic facts of the 

situation. 

These and other points are illustrated by the Phase I software prototype.

Phase I Prototype System

As part of this Phase I effort, we built a limited proof-of-concept prototype Socratic ITS 

for battlefield command reasoning skills. We call this prototype “ComMentor”—the Command 

Mentoring system. This section provides a detailed discussion of the goals, structure, internals, 

and behavior of the ComMentor system. 

 17 
Prototype Goals 

Our goals for the Phase I prototype ComMentor system included the following:

1. Demonstrate the possibility of automated Socratic tutoring in a battlefield reasoning

application; 

2. Demonstrate the applicability of the Socratic tutoring principles identified above; 

3. Explore architectural options for an automated Socratic tutoring system

implementation; 

4. Help develop requirements and designs for a more complete Phase II prototype. 

General Discussion of Prototype Scenario 

Our Phase I prototype presents a clear image of the kind of system we propose to 

construct during this Phase II effort. A lesson is centered on a Tactical Decision Game (TDG) 

scenario presented by a combination of a situation map, a set of force structures, and a textual 

description. The description will generally contain discussion of the Blue unit that the learner 

commands, the learner’s orders, intelligence about the enemy, and often some critical incident 

that starts the action in motion (a “trigger event” that triggers the need for a decision). For 

example, as described earlier, in Phase I we developed a scenario around ToF TDG. Figure 3 is 

a version of the scenario’s introductory text. Figure 4 shows the ComMentor screen layout with 

the ToF situation map and force structures displayed. 

At this point, the student can be prompted for a decision. Based on extensive experience 

playing this game with a wide range of students, our domain experts indicate that, at a high level, 

students essentially always take one of three approaches:

1. The student decides to try evading the enemy force so as to continue on to the 

assigned position at Viettiville. 

2. The student decides to react to the presence of the enemy force in a cautious way that

involves significant preparation and maneuvering.

3. The student decides to take on the enemy force with a rapid race to the top of Balzer 

hill that preserves the element of surprise for Blue. 

Any of these responses provides an opening for interesting discussion and analysis. 

Typical issues that can be well addressed in the context of this scenario include: 

Interpretation of commander’s intent. 

Modeling a thinking enemy.

Assessment of the big picture. 

Visualization of an evolving battlefield. 

 18 
Recognizing contingencies and maintaining flexibility.

Reasoning about time. 

Exploitation of terrain. 

Basic “battlefield calculus” (time, distance, and force calculations).

You command a tank battalion of four companies of M1A1s, with a section of eight tubelaunched optically-tracked wire-guided (TOWs) mounted on high mobility multipurpose 

wheeled vehicle (HMMWVs). Your division has been fighting elements of the enemy 6th 

Division, a battle-hardened outfit of three maneuver brigades supported by an armored 

reconnaissance battalion and an artillery regiment. Typically, a reconnaissance company and 

artillery battalion will support each maneuver brigade. Intelligence estimates that 10 Brigade, 

the toughest formation, is at about 80% strength and that 20 Brigade is at about 50-75%. After 

having been pulled out of action some two weeks ago for refitting, your battalion is now once

more near full strength. You are making a night administrative move on improved roads to your

new assembly area in Viettiville, where you will be part of the division reserve. The front is 

about 10-12 km east of the east edge of the map. During the move you are instructed to observe 

radio listening silence. 

The enemy has been attacking from the east with armor and mechanized infantry, usually

leading with a battalion of tanks preceded by a reconnaissance company on one or more axes 

depending on the terrain. Cross-country mobility for armor is good, except in wooded areas, 

which are essentially impassable to vehicles. The division commander feels the time is about 

right for a counterstrike. Division intends to use your battalion to spearhead a counterattack out 

of Viettiville, direction and objective to be determined as the situation develops, in order to 

destroy the enemy’s armor, seize the initiative and transition to the offensive. You are expected

to be in Viettiville by 0630, about 90 minutes from now. 

At the head of the battalion march column, you approach Balzerton from the southwest as 

the day begins to dawn. It has been an uneventful road march. During the night you heard some 

traffic on the division command net about skirmishes with enemy infantry, but no major activity. 

But then you are contacted on the radio by a section of light armored vehicle (LAV)-25s 

performing rear-area security. They had passed you earlier on the road and are now in Balzerton. 

The light armor section leader makes the following report: 

I count about 25-30 T-72s––I say again, T-72s––at Vietti’s Farm. They are apparently 

laagering; I see dismounted crews. I see a pair of T-72s, apparently scouts, approaching Balzer 

Hill from the east; they’ll be in position to see you in less than five mikes. I spot a convoy of 

trucks––about a half dozen, including refuelers––approaching the farm on the Viettiville road, 

about three clicks from the farm. I am unobserved. 

The section leader’s voice is urgent, but not panicked. He is sure of what he sees. 

Figure 3. Tanks on the Farm Introduction. 

 19 
Figure 4. ComMentor Screen Layout w. ToF Map and Force Structures. 

Of course some responses make it easier or more natural to talk about some of these 

topics than others. For instance, if the student chooses the first option—avoiding the pending

confrontation in favor of following the original orders—it is natural to begin a discussion about 

the big picture and acting in concert with the commander’s intent. An important aspect of 

understanding this scenario is to see that the laagering tanks are likely to be the lead battalion of 

an enemy breakthrough from the East. Given the lack of radio traffic reporting such a major 

incursion into Blue territory, Division command is likely not yet aware of what is going on. 

Furthermore, enemy logistics trucks coming down the road from Viettiville suggest the intended

assembly area is already compromised. In light of all this, assembling at Viettiville in order to 

launch a counterstrike and destroy enemy armor is now beside the point. The student has an 

opportunity to seize the initiative and start destroying enemy armor right here and now. 

If the student seems to have chosen the second or third option, the interaction goes 

somewhat differently. It is still worth exploring the issues of situation assessment and 

commander’s intent, but the conversation now is oriented towards probing the student’s rationale 

in order to confirm that they took their actions for the right reason. If such probing confirms an 

appropriate rationale, then the discussion can move on to other topics. For instance, if the 

student has taken a slow cautious approach, the discussion can turn to appropriate use of time

 20 
and exploiting the element of surprise; among other possible directions, this may lead to 

reviewing issues of “battlefield calculus” (e.g., time, distance, and force calculations). If the

student has moved quickly and decisively to destroy the laagering tanks, then after another

rationale confirmation discussion, the tutor might focus on the specifics of their plan: e.g., the 

deployment of available forces, effective use of terrain, awareness of possible ambushes, and 

even the form in which orders were issued. 

Given enough guidance, pretty much any student can be led to an understanding of the 

big picture, and to an appreciation of the value and appropriateness of decisive action. But there 

is potentially still more to this scenario. An assessment of the big picture raises questions about 

the larger enemy element the laagering T-72s belong to, and that larger enemy’s likely

intentions. It raises questions about the fight after the current fight, and the relation of the 

current situation, not only to the commander’s original intent, but also to the commander’s 

situation awareness and likely future intent. This relatively simple scenario, thus serves as a

context in which to address a large number of battlefield reasoning themes. 

 

Points Illustrated by Phase I Prototype 

The Phase I prototype as demonstrated at ARI on 7 June 2001 illustrates many of the 

points central to our approach to building Socratic ITSs for high-level reasoning skills. In this 

section we will review those points and the prototype capabilities that relate to them. 

Our observations of tutoring sessions convinced us of the importance of supporting multimodal input and output. In those sessions, the primary modes were graphics (task organizations 

and manipulable maps) and speech. In our implementation we aimed to duplicate the graphics 

capabilities and substitute natural language text for the observed speech3

. The scenario

introductory sequence illustrates the ability of the system to output both text and graphics. The 

text is currently output as one large block into the “Tutor” pane (refer again to Figure 4). The 

system is capable of directing individual blocks of text to any of its text output panes (e.g., the 

role-play channels for “Battalion,” “Division,” or the “LAVs”). On the graphic side, the task 

organizations for both Red and Blue are displayed as (expandable/contractable) hierarchical trees 

with appropriate icons (the upper right-hand column in Figure 4). The underlying situation map 

is a prepared image file, but the placement of moveable items such as the Red and Blue units are 

under program control (in the upper left-hand block of Figure 4). 

Multi-modal input (graphics & text) 

The prototype illustrates an example of the multi-modal input capabilities we identified 

as important to a naturalistic tutor. On the text side, we showed that the system can parse orders 

to units and instantiate underlying assignment structures. On the graphics side, we showed that 

the student could manipulate the icons on the map, dragging them to new locations as desired to 

help illustrate a plan. The system is able to compute a range of simple spatial predicates to 

describe icon placement, and uses those to attempt to verify that the icon locations match up with

 

3

 As noted during our presentation, it would not have been tremendously hard to demonstrate speech output in our 

prototype, but resource limitations argued against it. The proposal for Phase II addresses the issue of speech input. 

 21 
the orders previously given in English. Thus the system can use information acquired from these 

two input modalities together, to build a more robust understanding of the student’s intentions. 

Free-form input (natural language)

When it came to inputting orders, we demonstrated the ability to parse rather varied and 

complicated expressions for the assignments that represent the student’s plan for his units. Our 

domain experts consider language a far superior form for initial input as compared, say to menu 

systems or map overlay manipulation, because it corresponds closely to the real task, and neither 

leads the student to a likely answer, nor promotes inappropriate abstraction of moving game 

tokens on a board. In our demonstration sequence we input a set of orders roughly as follows: 

Alpha go to Balzerton and observe laagering tanks. Bravo stay west of Balzer Hill and 

wait in reserve. Our tows should also stay west of Balzer and wait. Charlie move onto 

Balzer Hill and fire on enemy scouts when in range. Delta move south of the hill and 

observe laager formation.

While our parser is not yet a robust capability, it illustrates the promise of taking a 

situated semantics-first approach to input text interpretation. Similarly we were able to parse a 

range of factual and hypothetical statements about the situation, as well as various fragmentary

and elliptical utterances contextualized as answers to tutor questions. Many issues remain 

however, including more effective handling of conjunctions/compounds, as well as pronoun 

references and elisions. Some other specific features identified for inclusion in Phase II include:

(1) feedback to the student to clarify the system’s interpretation of their input; (2) on-demand 

hints to the student on the types of input that are currently expected; and (3) spelling correction 

on inputs to minimize misunderstandings based on flawed inputs. 

Multi-part input (multiple sentences)

One important aspect of the prototype’s free text input capability is that it is able to deal 

with fairly arbitrary blocks of input text. In particular it is not limited to one well-formed 

sentence at a time. When accepting the student’s orders to his battalion, the prototype can parse 

any number of sentences together, including sentences with more than one phrase (e.g., both a 

maneuver and a military objective). This is important since our observations of students indicate

that they will generate utterances of vastly differing lengths and completeness. Going forward 

we have planned extension mechanisms to deal with even more radically abbreviated and 

extended forms of input. 

Student solution gap-filling 

Student input can come in different sized chunks, and that implies different degrees of

completeness after one or several inputs. For instance, when the system has asked the student to 

enter a set of orders, it has an idea of what would constitute a reasonably complete input. So 

long as it is missing items it expects, it will prompt the user to fill in missing pieces at the end of

each received piece of input. In the demonstrated prototype, this is illustrated by prompts for 

orders to units that have not yet been given any instructions. It is also possible to prompt for 

 22 
missing details in orders to a particular unit—for instance if a unit is told to move somewhere,

but is not told what to do when it gets there. 

Student solution classification 

Once a set of orders are input as text and verified through graphical manipulations the 

system attempts to classify the student’s proposed response to the scenario situation. For 

example the orders shown above from our demonstration sequence can reasonably be classified 

as a cautious approach to the situation. The student is not running away from the enemy, but 

neither is he seizing the initiative and attacking the main formation. The tutoring that ensues 

addresses the issues of time and distance and initiative. In particular, what is going to happen 

when the enemy scouts get to the top of the hill and discover elements of the student’s battalion? 

Contrast this with the following alternate, more aggressive, orders that the student may enter on a 

later cycle through the solicit-ordersclassifytutor cycle:

Alpha go to Balzerton and observe. I also want my tows to move to Balzerton. Bravo you 

move to the top of Balzer Hill and open fire on enemy tank battalion. Charlie go to top of 

hill also and attack the enemy tanks. Delta head to south of Balzer Hill and observe. 

Here the student is planning to directly take on the main enemy formation. The system 

diagnoses this as fitting its bold case. In this case the system drills on how the user’s immediate

and future actions relate to the higher commander’s intent. 

Case topic/theme scripting 

As noted, the two cases just illustrated lead to tutoring on somewhat different topics or

themes. The important point is that each diagnostic case fragment suggests a set of topics worth 

covering. The cautious case leads to a line of discussion about the situation and the enemy’s 

intent that includes questions such as: 

What do you think is at Vietti’s Farm? 

What are they doing right now? 

What do you think they’ll do when they’re finished refueling? 

Where is the front line?

This line of questioning culminates in a recap of the essential facts: “So we’ve got an 

enemy tank battalion refueling deep behind our front line and you think they might be going on 

the offensive.” It further leads to discussion of time and distance issues with questions such as: 

How long before the 2 T-72s see you? 

How far can your M1A1 travel in 5 minutes? 

Who do you think will reach the top of the hill first? 

Where will your battalion be when the T-72s hit the top of the hill? 

What do you think is going to happen [back at the main laager site] when their T-72s up 

there start shooting and your battalion starts returning fire to the top of the hill? 

 23 
Again the line of question culminates in a recap: “So at this point it seems as though you 

have lost the initiative as far as engaging that enemy battalion.” It opens the way to revisiting

the initial orders generated by the student: 

What could you have done differently? 

Would you say time is most critical factor here? 

Do you want to try again? 

General scenario tutoring cycle 

The prototype illustrates a general goal structure for tutoring a scenario. The system sets 

up the graphical presentation of the scenario and prints the initial situation description. It solicits 

the student’s solution (a set of orders), and verifies those orders by soliciting updates to the 

situation map. It classifies the student’s solution to the current problem and tutors on a set of 

themes that follow from that case/diagnosis. It potentially offers the opportunity to cycle back 

and revise the orders, leading to a new solution classification and further tutoring on newly

relevant themes. It may move on through a series of decision points in the scenario, each with

their own cycle of alternate student solutions. Finally the scenario session ends with an explicit 

opportunity for reflection. Figure 5 illustrates the general tutorial session goal structure; the 

italicized lines represent processes that are explicitly part of the model, but were not 

implemented in the Phase I prototype.

Run scenario session 

 Set-up map 

 Introduce scenario

Check scenario clarity

Prompt for decision 

 Solicit initial orders

 Solicit map updates 

 Classify solution case 

 Check/solicit rationale 

 Tutor on verified case 

Possible loop back to prompt for new decision 

<Additional choice-points and student solution cycles 

 ...> 

Wrap-up session (reflection) 

Figure 5. ComMentor General Goal Structure for Scenario Tutoring Session. 

 24 
Tutor goal management 

In addition to a general goal structure for scenario-based tutoring sessions, the Phase I

prototype also illustrates a more general goal management mechanism. The system maintains a

goal agenda. The active goal is responsible for generating some output (generally a question) to 

prompt the user for additional input. That goal will also have a set of expectations to be used in 

interpreting whatever input is received. Depending on the input received and any other facts that 

have accumulated in the systems session memory, the goal may lead to the spawning of new 

goals, or may rearrange the contents of the goal agenda. All pending goals can have pending

expectations, which can be used to analyze the input should the current active goal not be able to 

arrive at a suitable interpretation. All together this flexible set of mechanisms allows for (1) 

sequenced behaviors, (2) branching behaviors, (3) context-sensitive behaviors, (4) varying levels 

of alternate or default behaviors. 

Prompts for reflection 

As suggested in “General scenario tutoring cycle,” the last part of any scenario session is

a wrap-up interaction aimed at encouraging reflection. A standard set of questions is used here 

including the following: 

What do you think you did well in this situation?

What do you think you could have done better?

Going forward, the main issue here is to have the system actually check student input 

against its record of student solution diagnoses generated during the session. However, even in 

the absence of a system-generated reality check, a simple canned question sequence begins to 

serve the purpose of encouraging student reflection on the lessons of the session. 

Phase II System Design

Phase II Technical Objectives 

During Phase II, we plan first to extend our work on the Phase I objectives: 

1. Continue to identify and encode tactical analysis knowledge that should be 

represented by the system to support the evaluation of learner solutions to tactical scenarios, in 

order to refine the system's student model. In Phase II, particular emphasis will be placed on (a)

getting reasonably complete coverage of attributes and inferences in basic areas such as units, 

vehicles, and weapons, complemented by authoring tools to ease additions and modifications to 

the accumulated knowledge-base; (b) characterization and prioritization of the space of possible 

extension sub-domains of tactical analysis knowledge; and (c) further exploration of ways to 

unify ComMentor’s tactical analysis knowledge with other efforts in the Department of Defense 

(DoD) community (e.g., adoption of proposed knowledge representation language standards and 

evolving ontologies such as those sponsored in DARPA research programs like high

performance knowledge base (HPKB) or command post of the future [CPOF]).

 25 
2. Continue to explore and evaluate candidate interaction techniques that achieve the 

same types of instructional goals achieved by learner-mentor interactions carried out by expert 

human instructors. In Phase II we will devote particular emphasis to (a) exploring the

extensibility of our current textual language processing approach versus other more structured 

input techniques; (b) exploring the applicability of evolving techniques for speech input and 

output; (c) pushing on the utility of, and coordination techniques for, multi-modal interactions 

that combine graphics and text (or speech) input and output. 

3. Continue to identify, classify, and implement the types of tutoring strategies that 

can be automated in order to select and generate effective instructional interventions based on 

the student model. Our Phase I work illustrates application of some of the tutoring strategies

already identified (e.g., see Table 1 and accompanying discussion). Many of the identified 

strategies remain to be implemented or even seriously investigated. Others likely remain to be 

discovered. Our psychology subcontractor is particularly eager, and well positioned to develop a 

model of Tutoring Instruction/Mentoring Strategies. Additional data collection and preliminary

development of such a theory during our Phase II effort should help to provide a firmer empirical 

and theoretical foundation for ongoing design and implementation efforts. 

4. Refine our methods for authoring ITS scenarios by entering good and bad sets of 

decisions and rationale, and then annotating these solutions with skills and knowledge that are 

demonstrated (positively or negatively) by each solution. In Phase II, emphasis will be placed on 

systematizing our approach to these tasks, developing custom authoring tools, and embedded 

guidance in application of those tools. Efforts to ease acquisition of this scenario-specific 

information complement the efforts in objective 1 to get a handle on the general (domainspecific) tactical analysis knowledge. 

5. Develop a fully functional software prototype that not only illustrates the key ideas 

related to tactical analysis knowledge representation, learner-mentor interaction methods, and 

interactive tutoring strategies, but also serves as an effective learning aid for Army officers. We

expect to observe, analyze and implement on the order of another six scenarios during Phase II. 

In concert with all the objectives listed above, this is a logical next step in efforts to scale-up the 

techniques identified and illustrated in Phase I.

In addition to the above-stated extensions of the Phase I objectives, we plan to address 

the following new Phase II objectives:

6. Demonstrate the effectiveness of the tutoring system prototype developed in Phase II. 

Evaluation will be an ongoing effort throughout the Phase II, starting with formative evaluations 

of the Phase I prototype, continuing through early user trials and observations of the evolving

Phase II system, and culminating in a controlled study of system effectiveness using tactical

reasoning skill evaluation metrics and instruments, most likely adapted from those developed by

ARI for use in the TLAC program. 

7. Develop guidelines and baseline data on scenario authoring. As a result of developing

a half-dozen scenarios, we will acquire a far better grasp of effective techniques, typical pitfalls, 

and likely costs involved in extending the system. In addition to authoring techniques 

 26 
concretized in authoring tools, and guidelines explicitly embedded in such tools’ interfaces 

and/or help systems, there is likely to be other “wisdom” gained about how to build effective 

scenarios, including the costs and typical tradeoffs inherent in such efforts. The results of 

objectives 6 and 7 together will feed into addressing our final objective number 8. 

8. Develop a preliminary analysis of the issues likely to bear on long-term acceptance,

effectiveness, and maintainability of an expanded, fully operational ComMentor. In addition, 

consider the portability of the techniques embedded in ComMentor to other, more commercial 

domains and applications. 

Phase II Architecture 

Figure 6 shows the architecture of the proposed ComMentor system, broken down at the 

top level into (1) the ITS User Interface, (2) the ITS Core Processes, (3) the ITS Authoring

Tools, and (4) a set of persistent knowledge bases. The main knowledge bases reflect and 

support much of the underlying structure of the system. For instance four distinct components of 

the Authoring Tool suite address each of the four major types of knowledge: Domain 

Knowledge, Scenario Knowledge, Curricular Knowledge, and Instructional Control Knowledge 

(or strategies). Likewise, those kinds of knowledge (as well as the individualized student model) 

enter into the main ITS operations primarily through distinct processing modules. 

One major effect of this design is to simplify knowledge acquisition, as subject matter 

experts will ultimately spend most of their time entering tactical analyses for each scenario, and 

will not need to concern themselves as frequently or deeply with other more stable sorts of

knowledge (for instance, determining how the newly entered scenario knowledge will be used). 

A related effect, then, is to allow for more flexible use of the tactical analyses, as the Case 

Assessment Module can employ a wide range of algorithms for comparing the student's solution 

with those of the experts to find significant similarities and differences, and the Tutor Interaction 

Module can likewise use a wide range of algorithms to select instructional interventions 

accordingly. Finally, this kind of separation allows the system to evolve more gracefully to

incorporate new and improved technologies, by supporting piece-wise replacement of the 

software modules. For example, in Phase II we intend to explore modification of the User 

Interface by introduction of a Speech Output Generator and Speech Input Analyzer, and expect 

minimal impact on the other major modules and knowledge bases. 

 27 
ITS Core Processes

ITS User

Interface

Student

Case

Assessment

Module

Tutor

Interaction Module

Student

Model

Scenario

Knowledge

Domain

Knowledge

Curriculum

Knowledge

Student

Model

Domain

Inference

Module

Map

Pallettes

Tutor

Role

Play

Situation Analysis

Facts & Rules Scenario Cases

Expert Model

Instructional

Strategies

Scenario

Authoring

Tool

Curriculum

Authoring

Tool

Domain

Authoring

Tool

ITS Authoring Tools

Subject Matter

Expert

Strategy

Authoring

Tool

Te

xt

GUI Graphic GUI

Graphic

Input

Analyzer

Graphic

Output

Generator

Text

Input

Analyzer

Text

Output

Generator

Session Model

Figure 6. ComMentor Phase II System Architecture. 

Phase II Components 

The ITS User Interface 

The ComMentor User Interface supports all interactions with the user. Specifically, as 

illustrated in Figure 4 and Figure 6, this module: 

Presents tactical scenarios to the learner in the form of a map display plus narrative. 

Accepts solutions from the learner in the form of texts (orders, reports, requests, 

augmented with tutorial questions and answers), supplemented by map manipulations 

(placement of icons, and annotation with regions and paths). 

Enables an interactive multimodal instructional dialog that combines graphical and 

textual input and output, while embodying a range of pedagogical strategies. 

The above is a fair characterization of the capabilities illustrated by the Phase I prototype. 

In addition, during Phase II, we intend to experiment with other forms of interaction. As noted 

earlier, the most likely candidates are:

Presentation of war stories that illustrate lessons related to the learner's solution and/or

reasoning. 

Extension of multimodal capabilities to include speech input and output. 

 28 
The ITS Core Processes and Knowledge Bases 

The core ITS shown in Figure 6 elaborates on the abstract version of the architecture 

originally shown in Figure 1. The basic pipeline structure—from the input modules, to the 

Domain Inference Module (DIM), to the Case Assessment Module (CAM), to the Tutor 

Interaction Module—is preserved. Remember, the idea is that successive modules raise the level 

of abstraction in the situation description to arrive at a useful tutoring-relevant characterization. 

For example, the input module might be able to generate the characterization: “Alpha tank 

company is north of Balzer Hill; Bravo and Charlie companies are on top of Balzer Hill; Delta 

company is south of Balzer hill.” The DIM might elaborate: “Alpha and Delta companies are 

not in range to attack or be attacked by the enemy tank formation, and Delta company cannot be

seen by that formation; Bravo and Charlie companies can attack the enemy formation.” The 

CAM might say: “The student might be proposing to use some tank companies, located on the 

key terrain of Balzer Hill, to attack the enemy tank formation; if so, that is a recognized solution, 

and it suggests some specific tutoring interactions...(e.g., verification of plausible rationale).” 

Figure 6 unpacks some additional details, primarily related to the knowledge bases that support 

the processing modules’ activities. 

For instance, Figure 6 calls out in detail how the Expert Model is composed of combined 

information from the Domain Knowledge Base and the Scenario Knowledge Base, and how that

combined expert model supports both the Domain Inference Module and the Case Assessment 

Module. In addition, Figure 6 shows that the Expert Model contributes lexical/conceptual 

information that configures the Text Input Analyzer to enable parsing of user utterances. 

The Domain Knowledge Base contributes baseline facts such as “an M1A1 has an 

effective range of 2km” or “the top speed of a T-72 over open country is 60 kmh” and rules such 

as “if an enemy target is in range of a weapon, then it can be attacked using that weapon.” It

may also contain tactically relevant knowledge about how such facts can be modified by

mitigating factors such as night operations, crew training, weather, etc. In such cases, capturing

possible ranges, and the contextual parameters they depend on is a useful technique. Over time

the Domain Knowledge Base can potentially be extended to support such forms of reasoning as 

trafficability analysis, in order to make more nuanced predictions about how long certain 

battlefield maneuvers might take. 

The Scenario Knowledge Base contributes facts such as “Blue has a tank battalion, 

composed of four companies with 14 M1A1 tanks apiece” and “Balzer Hill is 2km from Vietti’s 

Farm.” It also contains diagnostic case fragments that can recognize “the student positioning 

tanks on Balzer Hill is evidence in favor of an aggressive attack on the enemy position” and can 

recommend partial tutoring scripts such as “(1) verify the student’s situation assessment and 

rationale, then (2) dwell on time-sensitive aspects of (a) the tactical plan, (b) how the orders 

were issued communicating that plan, and (c) how the situation was reported to higher 

command, then (3) transition to discuss the longer-term situation assessment and possible 

plans.” These suggestions can be passed on to the Tutor Interaction Module. 

 29 
The Tutor Interaction Module gets an understanding of possible issues and ways of 

teaching about them from the Curriculum Knowledge Base. For instance, one curricular point 

might be: “Environmental issues put constraints on how effective orders will be.” The 

Curriculum Knowledge Base should know that ToF is one scenario that can be used to raise this 

point, and that in that scenario, the relevant environmental conditions are alertness of crews 

based on time of day and visibility, current position and activity. It should know that there are a 

range of general and specific points to make about this issue, including, potentially, anecdotes 

about how experienced officers have coped with such situations. 

One likely way to incorporate lesson-bearing personal anecdotes into the system is to

support capture, storage, and playback of desktop video recordings. While this clearly would 

require augmenting ComMentor authoring tools, from the perspective of our current focus on the 

knowledge bases, the interesting new requirements have to do with story indexing. Indexing is 

the problem of annotating stories with descriptors so that they can be found when they are 

relevant to interactions in the tutoring system. Attaching stories to curricular elements is a

plausible start at indexing, but some stories can be used in more than one context, while other 

stories only make sense when some very specific set of circumstances are met. Indexing is thus 

one of the key issues to be explored as we investigate the possible utility of incorporating war

stories into ComMentor.

The Tutor Interaction Module also gets guidance about the application and sequencing of 

pedagogical techniques from the Instructional Strategies Knowledge Base. This knowledge 

base can contain general sequencing suggestions such as “(1) bring student to understanding of a 

potential problem, (2) solicit possible solutions (3) evaluate and highlight shortcoming of 

solution (4) iterate on solution, and finally (5) offer model solution.” It can also contain topic 

nomination rules such as: "If the learner made a decision D1 in situation S, and an expert 

solution for situation S contains a decision D2, a relevant discussion topic is whether D1 or D2 

is a better response to situation S." 

Finally, Figure 6 introduces two feedback paths from the Tutorial Interaction Module 

back to the DIM. The first path goes through the Session Model, which maintains a history of

what has gone on in the current session. It knows what topics have been covered, what issues 

have been raised, what solutions have been attempted, what instructional interventions have been 

executed, and what responses have been received. This recent memory must inform the 

interpretation of student inputs (and down through the pipeline, must affect situation assessment 

and tutorial actions). 

The second feedback path goes through the Student Model, which maintains a longerterm view of what the student is believed to know and not know. For instance, it records in a 

persistent form what other scenarios the student has been through, what curricular topics were

covered (and how they were covered, e.g., were relevant expert stories presented?), and to what 

extent mastery of those topics was demonstrated. This information is critical not only to biasing

assessments of student performance during a scenario, but also to helping the system choose 

which scenarios would be of value to the particular student. 

 30 
ComMentor Authoring Tools 

We plan to develop an authoring tool for each of the four main knowledge bases as 

pictured in Figure 6. The Student Model does not need an authoring tool, as it’s content is 

entirely under program control, based primarily on the content of the Curriculum Knowledge

Base, with contributions from other knowledge bases as well. We will focus on the Scenario 

Authoring Tool initially, with a secondary emphasis on the Curriculum Authoring Tool. Over 

time, effort will shift to the Instructional Strategies Tool. The Domain Authoring Tool will be

de-emphasized; for the most part, we expect to provide simple functionality for defining

categories and filling in object templates (e.g., vehicles and weapons). 

The Scenario Authoring Tool will start by requiring a basic scenario specification, 

including a scenario name, map, task organizations (with any idiosyncratic supporting icons), 

and introductory text(s). This basic information provides enough context for the domain expert 

to begin specifying possible student configurations and sequences—the good and bad cases that 

will be used to help evaluate students’ proposed solutions when running the scenario. Following

the lead of other tactical decision ITS projects at SHAI each such case can be constructed using

an interface essentially equivalent to the basic ComMentor student interface. 

Once constructed, however, these cases must be evaluated, generalized, and annotated. 

The evaluations indicate the strong and weak points of the proposed partial student solution. 

Generalizations allow the expert to indicate some aspects of the case pattern that may be varied 

while still retaining the essential nature (and evaluation) of the situation4

. Annotations include 

citations of domain facts and higher-level situation assessments that play into justifying the

evaluation of the situation pattern. Among the annotations may be references to elements drawn 

from the Curriculum Knowledge Base—that is the expert may say that a given case fragment 

illustrates (or contravenes) a curricular principle. Finally, specific tutoring sketchy scripts—

(partially specified) sequences of tutoring actions—can be tied to these cases. 

The Curriculum Authoring Tool allows the domain expert to define curricular points (or 

principles) of the sort that can be referenced by individual scenarios and their diagnostic cases. 

As is typical of SHAI ITS’s, the curriculum can be structured to indicate dependencies (e.g., 

part/whole relationships as well as logical dependencies). For example, a plausible curricular 

principle is that you should keep your higher commander informed of important developments of 

which he may be unaware. Immediately, there arise the questions of what counts as an important 

development, and what constitutes evidence that the higher commander may be unaware of

something. A good understanding of the first principle clearly depends, logically, on an 

understanding of the later two principles. As with the scenario cases, curricular elements can 

also have tutoring scripts attached. One particularly important kind of tutoring information 

likely to be attached here are the war stories referred to earlier. 

The Instructional Strategies Authoring Tool is the user interface for specifying

generalized tutoring scripts and the recognition conditions that license their application. The 

basic idea is to pair sketchy tutoring scripts, with student and session conditions. As already

 

4

 Certain kinds of case sequence generalization were explored and implemented as a core technique in SHAI’s Task

Tutor Toolkit. 

 31 
suggested, some tutoring strategies can be triggered more or less directly on scenario conditions 

(e.g., case recognition), or on curricular conditions (e.g., the need to tutor on a particular point). 

Authoring of such strategies will be tightly bound to their contexts, and specification of the 

trigger conditions largely handled implicitly. But we believe there will also be more general 

triggers for more general strategies—as suggested, for instance by Jona (1995). Work on this 

tool will dovetail with the important project research focus on identifying and implementing

Socratic tutoring strategies (e.g., Task 4 in “Phase II Tasks and Milestones”). 

Finally, the Domain Knowledge Authoring Tool is likely to be largely limited to 

elaboration of conceptual taxonomies and the filling in of forms pre-designed to cover important 

classes of domain entities. For instance, a vehicle form might provide places to enter a class 

(from a pre-existing taxonomy), a range before refueling, a capacity (for people and cargo), a 

max speed (perhaps different over different classes of terrain), and so on. Exact schemas for 

classes like vehicles, weapons, etc. may plausibly be derived from existing sources such as the 

Military Intelligence Data Base (MIDB). We note again, however, that ranges on such values

(parameterized by relevant contextual factors) are often more useful and appropriate to support 

tactical reasoning.

One additional point remains to be made about the enterprise of authoring, and the 

construction of these authoring tools. Stottler Henke Associates, Inc. has extensive experience

with this problem, and an evolving tool set; however, the direct applicability of prior work varies

for the different kinds of knowledge. While curriculum authoring is a relatively settled matter, 

what it takes to author appropriate scenarios and domain knowledge for a given ITS (or for that 

matter any AI system) is, in part, learned through experience with early examples, and then 

through additional experience with early authoring tool users. The authoring of Socratic tutoring

strategies as already noted, is still a significant research problem, but one that SHAI, in 

combination with our subcontractor, is particularly well positioned to address. We have 

structured our Phase II work so that we will get early and frequent experience in both scenario 

encoding, and domain expert interaction with authoring tools; we plan four waves of authoring

tool implementation and observation. One concrete payoff of such experiences is that lessons 

learned can be rolled into the authoring tools, either through changes in interfaces, or through 

tailored help systems (or even special authoring tool training systems!). 

Future Work 

Phase II Plans 

Phase II Tasks and Milestones. We propose the following 16 tasks to achieve the Phase 

II objectives. 

1. Perform Formative Evaluation of Phase I Prototype 

We will begin our Phase II work by conducting a more complete formative evaluation of 

the Phase I prototype than was possible within the scope of the Phase I project itself. This 

evaluation should include hands-on sessions with the system, both for our consulting subject 

matter experts, as well as for more junior military personnel of the type who would constitute an 

 32 
appropriate user trainee audience. The object is to observe the sessions looking for aspects of the 

system that seem to work well, as well as for flaws and limitations. We will also collect the

users’ comments and opinions, and feed those into our ongoing design and implementation 

efforts. 

2. Identify and Adapt Phase II Scenarios and Plan Tutorial Observation Sessions 

Building on our successful approach to data collection on naturalistic battlefield 

reasoning tutoring, developed during Phase I, we will work with our consultants and 

subcontractors to select a set of six tactical scenarios, and arrange tutorial sessions with military

personnel as students and our consultants as expert instructors. We expect to observe 

approximately four tutoring sessions per scenario. The bulk of these will be with our subject

matter expert consultants acting as instructors. However to broaden our base of observations, we 

will also intend to run some sessions with alternate instructors. In those cases, we expect to have

our consultants act as observers, as a way of gaining additional insight into the session conduct, 

and as a way to encourage reflection on the ways they conduct their own sessions. The selection 

of scenarios and generation of a session and observation plan will constitute the first project 

milestone, to be completed by the end of Month 2.

3. Carry out Tutorial Observation Sessions and Analyze Results 

We expect that actual tutoring sessions will be conducted using the distributed approach 

piloted so successfully in our Phase I work. Members of both the project teams will participate

as observers. We expect to begin these sessions perhaps as early as the second month of the 

project, and project that all tutorial sessions and observations will be completed before the end of

year one of the contract. 

4. Develop Instructional Strategies Model 

One of the major outputs of the tutoring session analysis will be an Instructional 

Strategies Model. The goal is to establish a solid empirical and theoretical basis for the

Instructional Strategies Knowledge Base. We expect the final draft will be produced following

the final set of observed tutoring sessions. Generation of the final Instructional Strategies Model 

will constitute a project milestone, to be completed by the end of Month 13. 

5. Elaborate Instructional Strategies Knowledge Base

Elaboration of the Instructional Strategies Knowledge Base will occur in waves (as will 

be true for the other three knowledge based as well). The first two passes at augmenting this 

knowledge base will follow completion of the two drafts of the Instructional Strategies Model. 

Remember, the purpose of the Model in the context of this project is to help us systematically

identify and organize our tutoring control knowledge. We have also reserved a block of time and 

effort towards the end of the project (in Months 19-21) during which we can perform a final 

reorganization and extension of this knowledge base (and in fact all the knowledge bases) based 

on what we have learned during the first 3⁄4 of the project, in preparation for the final release and 

evaluation of the Phase II system. 

 33 
6. Elaborate Scenario Knowledge Base 

The development of scenario-specific knowledge is a top priority in this project. The 

central contents of the scenario knowledge base will be fragmentary scenario solutions 

representing good and bad approaches to the problem. Each such fragment will include a set of 

recognition conditions, as well as a set of belief and rationale statements with indications of how 

those statements relate to the facts of the scenario, to good domain practice, and to possible 

tutoring moves. Blocks of time and effort have been scheduled for this task following

completion of the three waves of tutoring session observations. As noted later, the second and 

third of these blocks will coincide with early efforts at developing end-user authoring tools, and 

will serve to guide those efforts (also note that exercises in the use of the developing authoring

tools will follow immediately on the heels of work on scenario knowledge base extension and 

authoring tool construction). As with all the knowledge bases, there is time reserved in Months 

19-21 for a final cleanup and overhaul prior to completing the final Phase II system release. 

7. Elaborate Curricular Knowledge Base 

This final knowledge base contains information about the principles, concepts, and skills 

to be taught by the ITS. That information includes interrelationships among curricular items, as 

well as possible ways to present information about, or probe for understanding of them. 

Ultimately, the individual student model is formed, in large part, as a reflection of this curricular

knowledge base—recording what topics the student has been exposed to, in what form, and with 

what success. Work on elaborating the curriculum will come in waves, the first three following

on efforts devoted to extending the scenario knowledge base. Of course, as usual, one extra 

block of time is reserved for a final cleanup and overhaul prior to completing the final Phase II

system release. 

8. Elaborate Domain Knowledge Base 

It is critical to have a basic competence for representing and reasoning about the facts of

the battlespace, but we intend to limit the effort devoted to stretching that competence. That is 

because the development of formal reasoning models for such a complex domain is known to be 

an open-ended (and quite difficult) problem; also, other research programs have focused, and are 

focusing, on developing such competences. From experience in developing other case-based 

ITSs, we believe the greatest payoff—in rapid system construction and end-user extensibility—

will be achieved by focusing relatively more effort on scenario-specific knowledge. That said, 

we plan to devote some effort early on (during Months 4 and 5) to getting our basic domain 

model established. We will also devote effort to this task as we approach completion of the

second major release of the Phase II ComMentor (during Months 16 and 17). Finally, as with all 

the knowledge bases, we have reserved a block of time before completing the third and final 

major release of the Phase II ComMentor system to one last reorganization and extension of the 

system’s domain knowledge. 

 34 
9. Refine Phase II System Design

From a primary concern with data collection and knowledge codification, we shift here to 

system design and implementation. Early in the project, based in part on the initial Phase I

prototype formative evaluation, we will review the proposed Phase II system design and look for 

newly uncovered requirements, or opportunities to improve the system. The ultimate result of 

this effort will be a Phase II design document that will guide subsequent implementation efforts. 

Generation of this document will constitute a milestone for the project, to be completed by the

end of Month 5. 

10. Implement ITS Core Extensions 

We propose to devote a substantial block of implementation time early in the project to 

enhance and refine the basic system code originally developed for the Phase I prototype, in 

keeping with the dictates of the newly refined Phase II system design. The object is to provide a 

firm foundation for a more functional, scalable and robust system. This overhauled system base, 

combined with expanded knowledge structures and a first pass the authoring tools will constitute 

the first Phase II release of ComMentor. 

11. Explore Alternate Interaction Mechanisms 

As part of the work on the second Phase II ComMentor release, we propose to 

experiment with a range of alternate interaction mechanisms. Two major examples cited above 

include: (1) implementing a capability to capture expert anecdotal “war stories,” and then

effectively deploy them during tutoring sessions; and (2) implementing a capability to manage

tutorial input and output in the form of speech. For the “war stories” effort, we intend to draw on 

earlier work, primarily at Northwestern’s Institute for the Learning Sciences, on conversational 

video archives. For the speech input/output work we intend to draw on ongoing efforts at 

SHAI’s Seattle office on robust speech processing for training applications. 

 

The introduction of new interaction mechanisms such as these has implications for the 

authoring process, and we expect to see that reflected in the later waves of authoring tool 

development. We class these extensions as experimental because they require substantial 

implementation (and ultimately authoring) effort, and there exists, as yet, relatively little

experience with how to most effectively integrate such capabilities with ComMentor’s style of

tutoring. Achieving adequate performance, appropriate forms of integration, and efficient 

authoring constitute technical research topics

12. Implement Authoring Tools 

The development of end-user authoring tools will constitute a significant thrust of this 

project and will be spread over all three releases of the ComMentor system. Early on, we expect 

to focus on helping to capture scenario information. Over time, we will aim to extend the

authoring tool’s coverage to include curricular knowledge (including war stories), and 

instructional strategies as well. Our domain expert consultants will contribute to this effort

 35 
through their comments on proposed design, and through participation in the authoring tool 

application exercises described in the next task. 

13. Observe subject matter expert (SME) Use of Authoring Tools 

Following each wave of authoring tool construction we will devote a slice of time to 

observation and reflection on how those tools are used by our consulting domain experts. The

idea is to ensure that the tools we build are usable by their intended audience, and to learn what 

is working and not working as quickly as possible. The final set of these observations is 

expected to occur just before we begin our final evaluation studies. Our intent is to have our 

domain experts make an effort to enter one or two new scenarios using the final versions of the 

authoring tools, and to include these new scenarios in the final evaluation to compare their 

effectiveness to the other, more painstakingly analyzed and encoded scenarios. 

14. Evaluate Phase II System Releases 

In keeping with our iterative development approach, we aim to interleave evaluation (and 

possible course corrections) into all the cycles of our work. To mark the end of the first two 

ComMentor release cycles we plan to perform interim evaluations of the system. These 

evaluations will involve use by representatives of the system’s intended audience. Initially

formative in character, the evaluations will transition to summative, culminating in a final formal

evaluation of the third release. The two interim evaluations under this Task will serve as 

milestones, marking the completion of release cycles 1 and 2 by the end of Months 10 and 17 

respectively.

To mark the end of the final ComMentor release cycle, we will perform a full, formal

evaluation of the system, as used by representative members of its intended user population. 

Planning for this evaluation will begin as early as Month 4. We expect to adapt and apply the

battlefield reasoning evaluation metrics we understand to be under development by ARI in 

support of TLAC. Completion of this study by the end of Month 23 will mark the project’s 

penultimate milestone. 

15. Analyze System’s Long-Term Deployment Issues 

In order to provide as complete a picture as possible of ComMentor’s utility, we propose 

to devote time at the very end of the project to identifying, and, as far as is possible, quantifying, 

the likely costs and benefits of deploying the system and maintaining it over the long haul. The

object is to get a clear vision of the road to Phase III adoption of and wide-spread use of this 

work. 

16. Prepare Final Technical Report 

In the last month of the project we will write up our results in a final technical report, 

documenting our efforts and accomplishments. Completion of this report by the end of Month 

24 will mark the final project milestone. 

 36 
Phase III Transitional Plan 

At the end of Phase II, we expect to have developed: 

A catalog of instructional strategies that can be employed to support Socratic interactions 

for battlefield command reasoning and for other similar learning domains. 

A knowledge representation design for tactical scenario facts, scenarios, and analysis that 

can be used to support an intelligent tutoring system. 

A scenario-based intelligent Socratic tutoring system that provides mentor-student 

interactions to teach battlefield command reasoning skills. 

An application-specific authoring tool emphasizing creation by subject matter experts of 

new teaching scenarios comprised of tactical scenarios along with "good" and "bad"

solutions. 

The Phase II research and development will provide a solid foundation for Phase III

commercialization in four ways. First, we will have a reasonably complete catalog of Socratic

tutoring strategies, and experience implementing many specific tutoring techniques. Second we

will have experience building and evaluating a complete Socratic tutor aimed at high level 

reasoning skills—a first in the field. Third, we will have a working system that actually teaches 

battlefield reasoning skills using a limited number of cases, and the authoring tools to help grow 

the system cost-effectively. Finally, we expect to have the interest and attention of relevant part 

of the DoD training community. We believe our Phase II system will be compelling to a wide

range of potential customers, not the least of which would be training organizations within DoD. 

Possible Markets for High-Level Socratic Tutors 

The proposed research presents enormous commercialization potential. Web-based 

training offers the promise of providing training "any-time, any-place." However, current webbased training technologies frequently focus on teaching facts and procedures rather than 

reasoning skills and thinking habits. We believe that enhanced ITS technologies that teach 

reasoning skills will be of tremendous value to both government and commercial customers. 

Consider a key example, already introduced above. Stottler Henke Associates, Inc. is 

currently working with Simulation, Training and Instrumentation Command (STRICOM) to 

develop the command, control, communications, computers, intelligent tutoring system (C4ITS) 

for the Armor Captain Career Course (a course, incidentally, open to many officers who are not 

strictly Armor Captains). We expect the reasoning skills orientation and Socratic tutoring

capabilities of ComMentor to form a tremendously useful adjunct to the more traditional 

scenario-enactment training orientation of C4ITS. We believe that a combined system would 

find a very receptive audience, and hope for a clear path to fielding a full-scale implementation. 

In general, we plan to use the reasoning skills ITS technology developed during this 

project to offer the following software products and services: 

Technology-enabled consulting services to help web-based education service providers 

offer reasoning skills tutoring systems. Potential targets include providers of paper-based 

 37 
case teaching materials who are starting to offer software tutors (e.g., Harvard Business 

School Press), and on-line providers of learning software for K-12, university, and 

graduate school-level programs. 

Technology-enabled consulting services that deliver turnkey ITS applications for large, 

end-user companies with sophisticated training needs. 

Tutoring system engines and authoring tools to consulting companies who provide

sophisticated training solutions to large companies. For example, Andersen Consulting

applies authoring tools developed by its partner, the Institute for the Learning Sciences, 

to develop intelligent tutoring systems for its clients. 

SHAI’s Market Development Capabilities 

We have identified intelligent tutoring systems as a strategic direction for the company. 

Although many companies are using the Web to facilitate communications among students and 

educators or to simplify access to traditional, electronic course materials, no companies yet offer

or provide a vision for web-enabled intelligent tutoring systems. We believe that intelligent 

tutoring systems are essential for enabling web-based learning systems to achieve the vision of 

helping students learn "any time, any place." To position ourselves for this important market we

are undertaking much of our development effort using the Web-friendly Java programming

language. 

We will focus our marketing efforts on consulting services provided to educational 

software providers, consulting companies, and early-adopter end user companies. We will sell 

these services via direct consultative sales to identify custom software solutions that support 

those customer's unique needs. These sales activities will be supported by a combination of 

marketing and lead-generation activities including direct mail, presentations at educational 

industry events, and articles. For example, during the year 2000, SHAI delivered presentations 

describing intelligent tutoring systems technologies and benefits at the Interservice/Industry,

Simulation and Education Conference (I/ITSEC) 2000 (three presentations), American Society

for Training and Development (ASTD) TechKnowledge 2000 conference, and the SmartSystems 

conference sponsored by NASA and other organizations. During the year 2000, we wrote (or 

helped write) articles that appeared in the ASTD LearningCircuits electronic magazine for

technical training (http://www.learningcircuits. org/feb2000/ong.html) and the San Jose Mercury

News. 

The STRICOM relationship highlighted above is an example of a marketing opportunity

that could be an early route to turning the results of this research and development project into a 

production system. Similar positive relationships with the Navy’s Surface Warfare Officers 

School (a legacy of the tactical action officer [TAO]-ITS project) could serve a similar end. 

Both would be examples of technology-enabled consulting that delivers turnkey ITS applications 

for end-users with sophisticated training needs. 

Through government-funded and commercial software development projects, SHAI has 

amassed significant expertise and software assets, which make us technologically competitive 

with the small number of potential ITS competitors. We currently have on-going software

consulting relationships with major educational software/content vendors, and we are confident 

 38 
of our ability to establish new relationships with a wider range of software/content vendors in 

other educational markets to disseminate SHAI's ITS technology through established (or rapidly

growing) product vendors. Likewise, we have a growing network of satisfied customers 

throughout the DoD world. 

Stottler Henke Associates, Inc. has the resources and expertise to support the sales and 

business development effort required to pursue direct sales of technology by enabling consulting

projects and licensing arrangements to end-user companies, other vendors, and consulting

companies. Stottler Henke Associates, Inc. continues to augment its business development staff

and budget to exploit commercial product and consulting opportunities. For example, James 

Ong serves in engineering management and business development roles at SHAI. Before 

coming to SHAI, Mr. Ong was the director of product marketing at Belmont Research, Inc. and 

led the commercialization of the TableTrans data transformation system, developed with 

partial funding from the National Institute of Standards and Technology (NIST) Advanced 

Technology Program (project 94-04-0024: "Voyager: Browsing and Automatically Extracting

Healthcare Data from Scattered Databases"). TableTrans was selected by NIST as a NIST/ATP

success story (http://www.atp.nist.gov/atp/success/belmont.htm) and is co-marketed by IBM to 

pharmaceutical companies to streamline clinical data handling. 

Conclusions 

During this Phase I SBIR project we accomplished several important goals. First, we 

developed a deep understanding of the battlefield command domain, and of existing techniques 

for training battlefield command reasoning skills. What we learned reinforced the importance of

making Socratic tutoring more widely available to commanders-in-training, and suggested that 

the TDG format was an effective context for such training. We identified a range of levels at

which such tutoring could be conducted, and we experienced first-hand the importance of getting

good data on which to base the design of an automated tutor. 

Driven by this initial survey, we conducted six sample tutorial observations. These 

observations served as the basis for developing an initial taxonomy of tutorial moves. Our

taxonomy encompassed scenario-based Socratic tutoring moves, general topic management 

moves, specific theme-related tutoring moves, and a variety of tutoring moves associated with 

styles of tutoring other than what is classically considered “Socratic.” 

Based on our domain understanding, and our tutoring observations and analysis, we 

designed and built a limited proof-of-concept prototype of the ComMentor automated Socratic

tutoring system. The point of the prototyping effort was to help advance our design ideas 

through experimentation, and to demonstrate some of the key ideas and technologies that would 

go into a complete Phase II prototype. We demonstrated a capability for multi-modal input and 

output (graphics and text) including parsing of moderately complex and variable natural 

language orders. We demonstrated an initial capability for the tutoring system to pursue gaps in 

student inputs, and to classify proposed student problem solutions into categories that could drive 

the tutoring session forward by working through thematically scripted interactions that

nonetheless maintained sensitivity to the student’s inputs. All of this was built to function in the 

context of a tutor goal-management mechanism, and a general scenario-based tutoring cycle that

 39 
showed how techniques other than Socratic tutoring could be combined (e.g., closing prompts 

intended to facilitate student reflection). 

Finally, we incorporated lessons learned from our Phase I prototyping effort into a project 

plan and system design for a Phase II effort. We believe we have developed a system design and 

a project work plan that will use the available resources to significantly advance the state of the

art with respect to automatically tutoring high-level command reasoning skills. That plan

includes full-scale prototype implementation of the ComMentor system with approximately six

scenarios, and an accompanying suite of authoring tools. It also includes evaluation studies to 

assess how well the system functions. We believe our Phase II plan, as well as our other 

corporate experiences and contacts, will position us to take the results of a Phase II effort on to 

successful Phase III commercialization. The technologies to be developed here will provide a 

significant increase in the capability of ITSs, while lowering the costs of fielding such systems. 

 40 
References 

Brown, J. S. & Burton, R. R. (1975). Multiple representations of knowledge for tutorial 

reasoning. In D. Bobrow & A. Collins (Eds.), Representation and Understanding, New 

York: Academic Press. 

Carbonell, J. R. (1970). Mixed-Initiative Man-Computer Instructional Dialogues. Unpublished 

Doctoral Dissertation, Massachusetts Institute of Technology. 

Chandrasekaran, B. (1987). Towards a functional architecture for intelligence based on generic

information processing tasks. In Proceedings of the Tenth International Joint Conference 

on Artificial Intelligence (IJCAI-87), 1183-1192. 

Clancey, W. J. (1987). Knowledge-based Tutoring: The GUIDON Program. Cambridge, MA: 

MIT Press. 

Collins, A. (1976). Processes in acquiring knowledge. In R. C. Anderson, R. J. Spiro, & W. E. 

Montague (Eds.), Schooling and the acquisition of knowledge. Hillsdale, N.J.: Lawrence

Erlbaum, 339-363. 

Edelson, D. C. (1992). When should a cheetah remind you of a bat? Reminding in case-based 

teaching. In Proceedings of the Tenth National Conference on Artificial Intelligence 

(AAAI-92), 667-672. 

Forbus, K., Usher, J. Lee, G., Mostek, T., Ferguson, R., Turmell, B., & Rasch.R. (2000). 

NuSketch Course of Action Creator User Guide. 

Glass, M. (1999). Broadening Input Understanding in a Language-Based Intelligent Tutoring 

System. Doctoral Dissertation, Illinois Institute of Technology. 

Gragg, C. I. (1940). Because wisdom can’t be told. Harvard Alumni Bulletin, October 19, 78-

84. 

Guralnik, D. (1996). An Authoring Tool for Procedural-Task Training, Doctoral Dissertation, 

Northwestern University, Institute for the Learning Sciences Technical Report 71. 

Jona, M. (1995). Representing and re-using general teaching strategies: A knowledge-rich 

approach to building authoring tools for tutoring systems. The Seventh World Conference 

on Artificial Intelligence in Education (AIED-95) Workshop on Authoring Shells for 

Intelligent Tutoring Systems. 

Lussier, J. W., Ross, K. G., & Mayes, R. (2000). Coaching techniques for adaptive thinking. In

Proceedings of the 2000 Interservice/Industry Training, Simulation, and Education 

Conference, Orlando, FL. 

 41 
Munro, A. and Q. A. Pizzini (1995). RIDES Reference Manual, Los Angeles: Behavioral 

Technology Laboratories, University of Southern California. 

Rose, C. P., Moore, J. D., VanLehn, K., & Allbritton, D. (2000). A Comparative Evaluation of 

Socratic versus Didactic Tutoring, University of Pittsburgh, LRDC Technical Report 

LRDC-BEE-1. 

Schank, R., Fano, A., Bell, B. & Jona, M. (1994). The Design of Goal-Based Scenarios. 

Journal of the Learning Sciences, 3(4). 

Schmitt, J. F. (1994). Mastering Tactics. Quantico, VA: Marine Corps Association. 

Steels, L. (1990). Components of expertise. AI Magazine, 11(2), 28-49. 

U.S. Army Research Institute (2001). Think Like A Commander [Computer CD-ROM and 

Materials]. (Available from the U.S. Army Research Institute for the Behavioral and 

Social Sciences, 851 McClellan Ave, Fort Leavenworth, KS 66072). 

Whitehead, A. N. (1929). The Aims of Education. New York: Macmillan. 

Wiemer-Hastings, P., Wiemer-Hastings, K., & Graesser, A. (1999). Improving an intelligent 

tutor's comprehension of students with Latent Semantic Analysis. In Proceedings of The

Ninth World Conference on Artificial Intelligence in Education (AI-ED-99), Amsterdam: 

IOS Press, 535-542.

 42 
Appendix A 

 Related Work

ITS Research at SHAI

AEGIS Tactical Action Officer ITS 

Stottler Henke Associates, Inc. has developed for the U.S. Navy a simulation-based 

Intelligent Tutoring System (ITS) which enables students to act as tactical action officer (TAOs) 

in tactical simulations. The simulation’s graphical user interface displays a geographical map of

the region and provides rapid access to sensor, weapon, and communication functions. After the 

student completes a scenario, the ITS evaluates the entire sequence of student actions to infer 

tactical principles that the student correctly applied or failed to apply. These principles are 

detected according to sophisticated pattern-matching algorithms defined by the instructor using

the System’s graphical user interface. The system is highly configurable within the domain of 

naval tactical simulations, and authoring tools enable the instructor to define new types of ships 

and aircraft, scenarios, and principles. The instructor can also define complex behaviors for each 

friendly and enemy ship and aircraft to create realistic, intelligent, multi-agent simulations. The

TAO ITS has proven an effective training tool. Tactical action officer students at the Surface 

Warfare Officers School are currently using it, and initial independent evaluation of the software

in use has been highly favorable. Simulation-based intelligent training systems complement 

traditional classroom or computer-based training by enabling students to practice the application 

of concepts and principles. Additional funding has been received to adapt the TAO ITS for fleet 

use on board ships. Contact: Joe Russell (703) 602-5959 x183. 

C4I ITS 

Under contract to STRICOM, SHAI is currently developing C4I ITS, a prototype 

intelligent tutoring system for armored and mechanized infantry company commanders. The C4I

ITS will teach tactical decision-making, command and control principles, and the use of the

Force XXI Battle Command, Brigade and Below (FBCB2) command and control system. Before 

each mission, students will issue pre-mission orders and graphical overlays that specify

movements. The tutoring system will assess these plans using symbolic pattern recognition 

techniques that compare each student's plan with annotated good and bad plans (or portions of 

plans) supplied by experts. Similarities between the student's plan and the good plans will 

identify specific proficiencies in high-level and low-level skills. Similarities between the

student's plan and bad plans will identify skill deficiencies. 

During the mission, the students will interact with FBCB2 and Spearhead, a military

game simulation program that is being adapted for use in military training and is being integrated

with FBCB2. Spearhead implements the simulation behaviors and presents a 3 dimensional "out 

the window" view of the world as seen from each vehicle. Spearhead will exchange real-time 

simulation data with FBCB2, and C4I ITS will intercept those packets. The tutoring system will 

construct scenarios and modify scenarios in progress to include situations that test various 

A-1 
situation assessment skills. For example, the tutoring system could place friendly forces in 

certain locations that the student might otherwise fire upon. If the student does fire upon those 

positions, the tutoring system can infer that the student failed to use FBCB2 to notice the 

presence of friendly forces in that area. As another example, the tutoring system could place 

enemy forces at a location such that the student should direct friendly forces to oppose them. 

Failure to perform this action indicates either a lack of situation awareness of the enemy forces, 

or a poor tactical decision. Contact: Rodney Long at STRICOM, (407) 384-3928. 

Internet Intelligent Tutoring System Authoring Tool 

In connection with a project to make ITS systems more easily distributed over the 

Internet, as well as allowing separate ITS systems to interoperate, SHAI is developing an 

authoring tool—IITSAT—that addresses several of the authoring tool needs identified for

ComMentor. In particular IITSAT provides for the authoring of curriculum knowledge as 

hierarchies of principles to be taught, and that can be interrelated, in addition, by prerequisite 

annotations. In IITSAT, principles have multimedia courseware attached, and can be linked to 

scenarios that rely on those principles, as well as to specific critiques that determine when a 

principle is being followed or violated. In ComMentor, curricular elements will be linked to 

scenarios and cases (critiques), but instead of multimedia courseware, we will be concerned with 

alternate forms of instruction, such as tutoring moves and expert stories. IITSAT provides an 

API to plug in custom scenario authoring tools (since scenario structure vary radically across 

domains), and provides a first instance of a critique authoring tool. The ComMentors scenarios, 

with their good and bad cases will fill these niches. Finally, IITSAT contains an Instructional 

Methods authoring component. However, unlike ComMentor’s proposed Socratic methods, 

IITSAT focuses more on simpler techniques to adapt instruction to individual students (e.g., how 

many exercises to give on any given topic, how quickly to give hints, how many, and what type 

of examples to provide, etc.). 

Speech Interaction for ITSs 

Stottler Henke Associates, Inc. has been tasked by the Navy, in a project sponsored by

the Office of the Secretary of Defense, to develop an innovative system that will advance the 

state-of-the-art in computer application support for natural speech dialogue. The design of this 

system ties together the best of the current research in speech recognition, natural language 

understanding, and dialogue modeling, together with an eclectic combination of supporting

Artificial Intelligence techniques, into a complete end-to-end design. In our Training

Application Language Kit (TALK) architecture, all components in the natural speech dialogue

system will provide disambiguation and validation for the output of the others. In addition, all 

components will operate using a shared set of resources, including a representation of the current 

dialogue context, generic and domain-specific dictionaries, and dialogue models. Sharing these 

resources between the components of the TALK model not only ensures consistency in the 

interpretation and generation of natural speech dialogue, but also provides an unprecedented 

capability for training the system for new domains and integrating dialogue capability into a

wide range of applications, such as instructional systems and automated customer support. 

Phase I of this Small Business Innovation Research (SBIR) program began on January 26, 2001. 

Client: NAWCTSD, Orlando, FL. Contact: Steve Slosser, (407) 380-4599.

A-2 
Dismounted Infantry Military Operations in Urban Terrain (MOUT) Intelligent Tutoring System 

Stottler Henke Associates, Inc. developed, in cooperation with Research Development 

Corporation, a simulation-based ITS (SITS) for training dismounted infantry, both as individuals 

and teams, that would be used with a virtual reality simulator. Included in the project was the

development of a generic automated training system (ATS) architecture that can interface with

existing and future simulators. The SITS diagnoses student learning needs, determines what 

instruction content and technology are most appropriate, and drives the presentation of that 

instruction. Key technologies are case-based reasoning (CBR), integrated knowledge structures 

for representing expert and student knowledge, automatic knowledge elicitation, and dynamic 

scenario selection and creation. The architecture supports automatic and semi-automatic 

knowledge engineering to update its knowledge base as the domain itself evolves. The system 

trains squad and fire team leaders in Military Operations in Urban Terrain. The ATS monitors

the student's actions in the virtual reality environment, assesses his deficiencies, and modifies the 

scenario or creates new ones to address those deficiencies. A CBR system also selects the most

appropriate instructional technique based on the student's individual requirements and past 

learning behavior. 

Constructivist Distance Learning System for Counter-Terrorist Intelligent Analysis 

Stottler Henke Associates, Inc. is developing for the U.S. Army at Fort Huachuca a

training system comprised of two parts. The first is a general framework that supports the 

creation of Constructivist distance learning (DL) courseware in a wide variety of areas. The 

second product is the Intelligence in Combating Terrorism (ICT) Tutor, a specific tutoring and 

scenario authoring system built using this general course creation framework. 

The purpose of the ICT Tutor is to give students extensive, hands-on training in the 

analysis of raw intelligence information about terrorist organizations and installation threat 

assessments, leading to a compact summary of a terrorist operation and an assessment of the 

current level of threat. The tutor uses Constructivist learning theory by supporting adaptive

learning, modeling, intentional activity, and rich scenario contexts. Contact: Helen Remily at

(520) 533-9077. 

Task Tutor Toolkit and Remote Payload Operations Tutor for Procedural Training 

To lower the cost and difficulty of creating scenario-based intelligent tutoring systems for

procedural task training, SHAI developed the Task Tutor Toolkit (T3

), a generic tutoring system

shell and scenario authoring tool. The Task Tutor Toolkit employs a case-based reasoning

approach where the instructor creates a procedure template that specifies the range of student

actions that are "correct" within each scenario. The system enables a non-programmer to specify

task knowledge quickly and easily via graphical user interface, using a "demonstrate, generalize, 

and annotate" paradigm that recognizes the range of possible valid actions and infers general 

principles that are understood (or misunderstood) by the student when those actions are carried 

out. The annotated procedure template also enables the Task Tutor Toolkit to provide hints 

requested by the student during scenarios, such as “What do I do now?” and “Why do I do that?” 

At the end of each scenario, Remote Payload Operations Tutor (RPOT) displays the principles 

A-3 
correctly or incorrectly demonstrated by the student, along with explanations and background 

information. The Task Tutor Toolkit was designed to be modular and general so that it can be 

interfaced with a wide range of training simulators and support a variety of training domains. 

Stottler Henke Associates, Inc. and NASA used the Task Tutor Toolkit to create the

RPOT, a tutoring system application which lets scientists who are new to space mission 

operations learn to monitor and control their experiments aboard the International Space Station 

according to NASA payload regulations, guidelines, and procedures. Contact: Mr. Stephen 

Noneman, (256) 544-2048. Phase II Completed: February 2000. 

Intelligent Tutoring System for Long-Range Acoustic Detection of Submarines 

Stottler Henke Associates, Inc. is developing for the U.S. Navy an Acoustic Analysis 

Intelligent Tutoring System (AAITS) which will enable students to practice the detection and 

classification of sources of underwater acoustic signals such as submarines and whales. Acoustic 

analysis experts will create scenarios using a Scenario Authoring Tool by selecting and viewing

LOFARGRAMs which are frequency-analyzed acoustic datasets displayed as 2D images, 

annotating them with significant features and links among related features, providing reasons for 

requesting each LOFARGRAM, and assigning a final classification. Students will use the 

Tutoring System to carry out this same acoustic analysis. By comparing the details of each

student's analysis with those of the expert, the Tutoring System can identify the acoustic analysis

principles understood and correctly applied by each student, provide specific and individualized 

feedback, suggest relevant training materials, and select appropriate next scenarios. By storing

LOFARGRAMs annotated by experts, AAITS also serves as a knowledge repository which 

disseminates the most current acoustic analysis expertise to sonar technicians on land or at sea. 

A key innovation of AAITS is the use of an application-specific Scenario Authoring Tool that 

enables experts to create scenarios, which encode their expertise and analyses intuitively, by

annotating datasets graphically, using a point-and-click graphical user interface. Phase II project 

start date: February 1999. Contact: Master Chief Joseph Spivey at SPAWAR, (858) 537-0312. 

AI Representations of Military Plans and Tactics at SHAI

Stottler Henke Associates, Inc. has developed several systems which employ artificial 

intelligence representations of military plans and tactics. Selected projects are described below. 

Intelligent Control for Immersive Wargaming 

Stottler Henke Associates, Inc. is developing a wargaming system to fuse entity-level 

control with aggregate-level command to manage a small-scale air campaign. The Phase I

prototype does this by combining an entity-level robotic control architecture called Hap, based 

on research at Carnegie Mellon University, with a strategic command system. The system

allows simulation participants to interact with intelligent allies and enemies. For example, if

there is a goal to destroy an enemy airbase, the aggregate-level command will generate plans to 

form a strike package. Entities controlling the elements (fighters, ECM, and ground attack) 

coordinate together to rendezvous outside enemy airspace, form the strike package, and 

overcome enemy resistance in a team-oriented fashion. This current project was started in May

A-4 
1999 and the contact is David Ross at Air France Research Labs (AFRL) Rome Laboratories - 

(315) 330-7624. 

Multi-User Simulation Environment (MUSE)

Our Phase I prototype dealt with the need to intelligently control all the tactical platforms 

in a scenario. We developed a schema for military warfare tactics representation and execution 

in MUSE using expert system and graphical techniques to capture the human decision processes. 

Agents in the scenario would compute intercept paths to engage enemy aircraft. The networked 

prototype ran with geographically distributed team players, both human and artificial. Phase I

was completed in December of 1998 and the contact is Terry Jackson at AFRL Brooks AFB - 

(210) 536-3908. 

Robustness in Air Campaign Plans 

One important attribute of air campaign plans is robustness. How successful will a plan’s 

execution be in the presence of uncertainty and changes, and how hard is it to change the plan 

once execution has begun? Military planners can gauge the robustness of air campaign plans. 

Furthermore, plans produced by automatic planners tend to be graded poor on this scale. A team 

of cognitive psychologists worked to unravel and understand the thought processes involved. 

Stottler Henke Associates, Inc. was tasked with analyzing this abstract body of work, developing

computable concepts which parallel these processes, implementing them, and interfacing them to 

a planning system in current use. This work involved an understanding of the entire planning

process from the highest level to the most detailed. Client: Klein Associates. Contact: Dr. Tom 

Miller, 513-873-8166. Completion Date: September 1997. 

Related Work by Others 

Mixed Initiative and Socratic Tutoring Systems 

Research in mixed-initiative and Socratic tutoring systems has been carried out for more

than 30 years. In fact, much of the early work in intelligent tutoring systems focused on the

support of dialogs between the software and the student. Early work includes the Scholar system 

that taught South American geography (Carbonell, 1970). It is significant in its early use of 

domain knowledge representations (about geography) that are separate from the instructional 

decision-making knowledge. However, this system taught facts about geography rather than 

reasoning skills and therefore did not pursue many goals typical of Socratic dialog. Collins 

(1976) developed a set of two dozen decision-making rules used by Socratic tutors and embodied 

these rules within the WHY system that taught meteorological reasoning about rainfall 

processes. Clancey (1987) developed GUIDON, a tutoring system for diagnosing blood diseases 

that used as its expert model the Mycin rule-based expert system. A finding of the GUIDON 

project was that expert system rule-bases frequently employed reasoning methods (in this case, 

rules) that were not intuitive when used to generate tutorial dialog utterances. Edelson (1992) 

describes Socratic tutoring in the domain of biology that relies on a case-base for its expert 

model. 

A-5 
This previous work provides useful findings and insights, especially in the identification 

of tutoring strategies, and we expect to build upon this prior work. However, ComMentor will 

require advancements in the areas of using complex plans and rationale structures supplied by

experts as the basis for tutoring dialogs; using multiple expert solutions for assessing student 

solutions; and assessing and enhancing high-level thinking skills and habits as opposed to 

detailed knowledge and lower-level reasoning skills. 

Intelligent Tutoring System Authoring Tools 

The ITS technology has been under development for over three decades by researchers in 

education, psychology, and artificial intelligence at military and academic research labs. 

Research studies carried out so far show that students taught using ITSs generally performed 

better and learned faster, compared to classroom-trained students. However, a major 

impediment to the widespread use of intelligent tutoring systems is that they frequently require

custom software development and are therefore expensive to create. Consequently, the 

development of authoring tools to enable lower-cost development of these tutoring systems is 

also an area of active research. 

Much of the prior research in ITS authoring tools has been in support of procedural task 

training domains where there is a "correct" method for carrying out each procedure. Guralnik 

(1996) describes an authoring tool which applies a content theory of task knowledge which 

enables the tutoring system to generate replies to important questions from the student, such as 

"What do I do next?" and "How do I do that?" (Munro & Pizzini, 1995) describes the RIDES 

system, which enables authors to create graphical training simulations (frequently, of devices) 

integrated with the intelligent tutor. 

The ComMentor will require a new type of ITS authoring tool, one that captures the

reasoning of experts in support of solutions, even when there is no single correct solution. We

expect that ComMentor may draw upon prior research in ITS authoring tools, but ComMentor 

will require significant advances in the capture and application of knowledge structures that 

capture the expert's scenario-specific reasoning. 

Goal-based Scenarios, Problem-solving Architectures, Generic Tasks 

Learning different types of problem-solving tasks will warrant different types of tutoring

system architectures. However, we believe that there are a manageable number of distinct types 

of abstract problem-solving tasks. We are encouraged and inspired by research in Generic Tasks 

(Chandrasekaran, 1987), Components of Expertise (Steels, 1990), and Goal-based Scenarios 

(Schank, Fano, Bell & Jona, 1994), who all argue that the number of types of problem-solving

skills is quite finite. Proponents of Goal-based Scenarios at the Institute for the Learning

Sciences (ILS) at Northwestern University assert that:

Students learn effectively when they are pursuing goals within the simulated scenarios, 

and, 

A-6 
One can categorize goals into a small number of abstract types (e.g., Investigate and 

Decide, Run a System or Organization, Pursuade, etc.) which strongly affect the 

architecture of the appropriate tutoring system. 

Our proposed project team includes Dr. Eric Domeshek, a project manager at SHAI. 

Before coming to SHAI, Dr. Domeshek taught and carried out research at ILS where he 

developed a pair of successfully fielded educational systems for university courses, as well as 

several other AI and multimedia systems for knowledge management. One of these systems, 

named "Invitation to a Revolution," implements a Goal-based Scenario (Pursuade) to help 

students understand the social forces that shaped France just prior to the Revolution. The 

software includes a hypermedia database of over 290 video clips (over 4.5 hours total) of 

experts, as well as 229 clips of 12 different historical characters which support simulated 

conversations between the student and those characters.

A-7 
Appendix B 

“Tanks On the Farm” Sample Session Transcript 

Problem 

The following table represents a transcript from one of our Phase I instructional 

observations. The first column of the table contains timing information and comments inserted

by the session analyst. The second and third columns represent the text of utterances made by

the Instructor and Student respectively. Column four is a first pass at categorizing the type of 

interaction contained in a given row of the table. Column five is a similar attempt to discern 

what triggered a given utterance by the Instructor. Columns six and seven tag each instructor 

utterance according to the instructional theme being addressed (based originally on the TLAC

themes), and what level of scaffolding is being exhibited (based originally on the TLAC-related 

scaffolding levels discussed in the body of the report). 

In all, we held six sessions like this one, three of them devoted to the “Tanks on the 

Farm” problem. This transcript represents a relatively high level of performance by the student. 

It was used, along with the other two “Tanks on the Farm” transcripts, as a central resource in

preparing the Phase I prototype and demonstration scenario. 

 B-1
Vignette Tanks on the Farm (Phase I, 

Session 1)

Date/Time 18 April 2001, 1500 Student DuckInstructor Hannibal (Wolfpack adds at end)Analyst RPD/WolverineComments Based on notes and CD audio recording. Instructors

classified this session as HIGH-LEVELPERFORMANCE. 

Key learning points of Tanks on the Farm: Primary-Timing; Big Picture;Secondary-Clarity of Orders; CdrsIntent not just specific orders;Visualization. Instructors classifiedthis session as HIGH-LEVELstudent performance. 

Time/

Comments Instructor to Student Student to Instructor Type of Interaction 

1321 Analyst comment: Student wasnot directed to take notes orprovided with paper copiesof anything. 

(Not captured on audio recording) I'm going to start the Decision Making Game (DMG). I will not be asking you to do anything. At any point you can tellme to stop when you are ready to decide or makeany other action. Otherwise, I'll continue to developthe situation.

Statement (S):

Instructions on student and instructor roles and behaviors. 

FR (Friendly Forces) and EN (Enemy) organization

given (Not entirely captured on audio recording). Audio starts: Intelligence estimates that 10 Bde is at about 80% strength; that 20 Bde is between 50 and75% strength. So that's the situation on 6th Division. Any questions about those guys? 

Statement (S) of

the problem:information reEN (IF). 

 B-1
Negative. 

Analyst note: Student should have a button orpaperavailable tocheck the EN and FRorganizationsand to lookagain atmission and atCdr's intent asdesired.

Good. OK. So, after having been pulled out of actionabout two weeks ago for refitting, your Bn is nowonce more near full strength. You're making a night administrative move on improved roads to your newAssembly Area (AA) in Viettiville. Viettiville as you can see is about 8km northeast of the map here.That's where your AA is going to be. You're making a night administrative move to Viettiville where you will be part of the Div reserve. 

The front as you can see is about 10-12 km to the 

east of the east edge of the map sheet here. So, inother words, the front is 10-12 km east of where weare. Viettiville is about 8 km northeast of where weare. During the move you are instructed to observe radio listening silence. The EN has been attacking generally from the east with Armor and Mechanized Infantry usually leading with a Bn of tanks preceded by a reconnaissance Co. That reconnaissance Cowill be operating on one or more axes depending onthe terrain in front of the tank Bn. 

Statement (S) of

the problem:information (IF)re FriendlyForces (FR), the situation and Cdr's intent. 

 B-2
But then you are contacted on the radio by a section 

of LAV 25s performing rear area security. They havepassed you only a few minutes ago on the road and are now in Balzerton. The Light Armor section leadermakes the following report. He says, "I count about 25-30 T72s. I say again T72s at Vietti's Farm. Theyare apparently laagering. I see some dismounted crews. I see a pair of T72s, apparently scouts,approaching Balzer Hill from the east. They'll be inposition to see you in less than five Mikes [minutes]. I spot a convoy of trucks, about a half a dozen including two refuelers, approaching the farm on the Viettiville road about 3 klicks [km] from the farm. Iam unobserved at this time." The section leader'svoice is urgent, but not panicked. He seems sure ofwhat he sees. You're continuing to move in yourmarch column toward Balzerton. 

Statement (S) of

information (IF)in the form of a spot report:Probe to stimulate studentbehavior. 

1527/ CD 03:40

Ready to make a decision here. 

Alright, what's it going to be?

 B-3
LAV section, once you see my

attack taking off, I want you to support our engagement byengaging those two T72s moving up in the vicinity of (VIC) of BalzerHill. Alpha Co I want you to comeoff on LAV section's right flank to take out those two T72s and continue your attack at the EN'slaager site, VIC Vietti's farm. Bravo and Charlie Co I want you to move up VIC Balzer Hill and attack the EN's laager site from VIC BalzerHill . You all will kick off this attack. Your firing from Balzer Hill will kickoff the attack. You will be engaging the EN force laagering at Vietti'sFarm. Delta Co follow in trace ofAlpha Co. You will be my reserve. TOW section I want you to move out to the trail that leads up to Viettiville to the north. Defend that trail to defend our company's leftflank. 

Is that it?

That's it. 

1530/ CD 04:56

Now why don't you go ahead and diagram that for usreal quickly here? 

S: Student

instructed toalso make map input. 

CD 05:48 Student concludesmap input. 

How's that, Sir?

 B-4
Fine, so give me your thought process. What's the 

story here. What are you thinking about the situation and why are you doing what you are doing?

Q: Probe overall 

thought process.

1331/ CD 05:58

OK. In five minutes, he should be 

able to see me. 

Who is? Q: Clarification 

of Input. 

These EN T72s that are 

approaching VIC Balzer Hill. [EN]is in a laager site at Vietti's Farmand I have the opportunity to getthe jump on them. I'm sending the TOW section up to the trail to the north as an economy of force measure just in case I'm gettingsome kind of surprise from the north. 

OK. So they're protecting your Bn left flank against 

anything coming down that trail from Viettiville?

Q: Clarification

of Input to probe Assets and EN model. 

Yes, Sir.

 B-5
OK. So why are your worried about that? Q: CIN 

I'm not so much worried about that 

as the fact that I've got an asset that I'm not employing and I can use that to protect my left flank. 

OK.

By the looks of it, I can range the 

laager site from Balzer Hill with the M1s. Looks like it's going to be about max range for the M1s.

Right. So what's the significance of that, if anything? Q: Probe 

understanding ofsub problem.

Well, if there is something past 

Vietti's Farm that we haven't noticed yet, I'm not going to be ableto range that. 

 B-6
OK. But you're not concerned about that at this

point? 

Q: Probe for

understanding offollow-on forces.

I'm not. 

OK. So the idea is that Bravo and Charlie are in a 

position to do significant damage to this EN force inVIC Vietti's Farm? 

Q: PRD Assets. 

Check, Sir.

And, did I understand that Alpha Co is going to be

conducting a maneuver attack against Vietti's Farmfrom there? (refers to map)

Q: PRD Assets

and CIN. 

CD 07:33 No, Sir. They're coming up on line

with the LAV section supportingBravo and Charlie's attack on Vietti's Farm from their positionthere. 

 B-7
So they're also supporting by fire, but they're not 

maneuvering against them? 

Q: PRD Assets. 

Right, Sir.

So what you've got basically is a three Co. attack by

fire against what? 

Q: PRD Assets. 

Against the EN's laager site VIC 

Vietti's Farm. 

What's at the laager site? Your estimation? Q: Look at 

Assets fromview of ENmodel. 

I've got approximately 30 vehicles

probably preparing for some form of offensive maneuver. 

 B-8
Give me your assessment of what you think is going

on here with Red. What's the Red story? What are they trying to do? What do you anticipate is likely tohappen next? 

Q: EN model 

and Visualization. 

Based on what little I know right 

now, seems like he feels relativelysafe right where he is. Two T72sapproaching Balzer Hill. Reconnaissance element isn't all that far out from his laager site where he's presumably in anassembly area. I would imagine heis preparing for some form ofoffensive operation, but it's not going to happen anytime soon, because from what I can gather, he doesn't have too muchreconnaissance out forward. 

 B-9
So your estimation is that he's just getting settled in 

here at Vietti's Farm? 

Q: PRD EN

model. Level 3 prediction of EN behavior. 

Yes, Sir.

CD 09:15 And your rationale for that is....?  B-10 
Well, the fact that he doesn't have 

reconnaissance out there muchfarther. He was pressing forward insay the next 6-12 hours, I would think I would have run into something by now or he would have reconnaissance farther to the south. 

To the south? Why to the south? NA 

Correction, farther to the west. 

So you think this guy's making a east to west move? Q: CIN EN

model. 

Right, Sir.

And he's using Vietti's Farm as AA? Q: Probe EN

model. 

Yes, Sir.

But you're anticipating that the move is not going to 

be immediate? So you think you've actually got the jump on him, based on the fact that he doesn't knowyou're there and you have the chance to destroyhim?

Q: Probe En

model withTiming model to produce inference. 

That's correct. 

 B-11 
What else are you seeing that reinforces this picture 

if anything? And what are you seeing that doesn't? What are you seeing that's maybe inconsistent withthe picture you're painting?

Q: PRD EN

model and Timing. 

What am I seeing that's

inconsistent with that picture? 

If anything.

(Pause.)

Are you pretty happy with it? Are you pretty

comfortable you've got a handle on what's going on here? 

Q: Ask student 

to make selfevaluation. 

I do, Sir. I don't have any evidence 

that there is something else going on right now.

 B-12 
CD 10:35 So there are no other interpretations of what Red 

could be trying to do here in your estimation? 

Q: Probe overall 

EN model. 

I'm sure there are an infinite

number of explanations, but I've gotto choose one. This one matchesas close to the situation as I can ascertain from the information I've got. 

So is this a good situation or bad situation with 

respect to your estimate of what Red is? Are you happy with this? Does this cause you problems? Are you surprised by anything that you're hearing here? 

Q: Ask student 

to make selfevaluation. 

It looks like I'm up against maybe 

one Bde at this point. I still don't know where the others are. I've got the opportunity right now to masson this Bde and surprise him and destroy the EN's armor VIC Vietti'sfarm.

 B-13 
You're estimating there's a Bde at Vietti's Farm? Q: PRD EN 

model. 

Yes, Sir.

CD 11:33 Based on? Q: PRD EN 

model. 

Based on the number of vehicles

there. 

Which was? Q: PRD EN 

model. 

I'm reading 30-50 vehicles. 

It was between 25 and 30 from the report by the 

LAV section leader. Which equates to a Bn. 

S: Reminder of

knowledge level information. 

Roger, Sir. More like a Bn. 

So are you saying you don't believe you've got a

Bde here now. You've got a Bn? 

Q: PRD EN

model. 

Right, Sir.

Does that change anything? Q: Probe EN

model. 

At this point, no. 

So the Bn is what? Q: Probe EN

model. 

 B-14 
Probably a forward element of a 

larger unit. 

What do we know about the way these guys like to 

operate from our experience in the last several weeks?

Q: Probe EN

model. 

(Pause.) You've got me there, Sir.

We understand that they prefer to lead with a Bn of

tanks, right? Is this the Bn of tanks? I'munderstanding that you are reading this as theiradvance element? 

Q: Request CIN. 

I am. If that is the case, I'm still

pretty happy with the decision I've made, because once again, going back to my Cdr's intent--Destroythe EN armor and thus transition to the offense. 

Changesthemes orviewpoint.

So you don't have a problem with ignoring yourinstructions to be in Viettiville by 0630 to be readyfor the Div Cdr's counterattack?

Q: Probes

model of HigherHQ intent. 

I do not. I'm not going to get there 

without taking care of this threat first. 

 B-15 
Is getting to Viettiville still a concern of yours? Q: Probes

model of HigherHQ intent. 

Sure. I think if I can get the jump on 

this guy here at Vietti's Farm, I maybe able to get this over with and still get to Viettiville within a relativelyshort amount of time. 

So your ideal situation would be to handle this

situation and then continue with your instructionswhich are to get to Viettiville? 

Q: PRD 

Yes, Sir.

And, the thinking there being what? Q: PRD 

Thinking there being he doesn't 

know where I am, I do know where he is. I can mass VIC Balzer Hill, hopefully get this forces on line atthe same time--Alpha, Bravo, and Charlie--mass fires on the EN VIC Vietti's Farm and the vehiclesmoving toward Balzer Hill, be done with it and press on through to Viettiville. 

I get the idea of dealing with the problem here at 

Vietti's Farm pretty quickly, I'm asking the thought process now...you destroy this EN tank Bn presumably and then you're going to continue to move on to Viettiville with what in mind? 

Q: Probe

mission. 

CD 14:35 With forming that AA with an

unknown direction to continue that attack. 

So you think the idea of AA in Viettiville is still viable 

at this point?

Q: PRD Big Pic. 

It is.

 B-16 
And your thought process there is? Q:

Rationalizationfor Big Pic.

I can see the direction you're

leading me is that I haven't given any instructions for...

I'm just asking if you still want to get to Viettiville. 

What's the thought process there? 

Q:

Rationalizationfor Big Pic.

I keep giving you the same answer.

I'm not going to get to Viettiville if I don't take care of these vehicles. IfI keep pressing to Viettiville, myright flank is going to gethammered. He could move into myrear or deeper into the Div's area. Neither of those options I like. 

Cd 15:42 Why not? Q:

Rationalizationfor Big Pic.

I'm assuming that behind me are 

forces that are not nearly ascapable of dealing with the forcesat the laager site as I am. 

 B-17 
That's absolutely true. We're in our rear area, so

there's a lot of softer stuff back here. There'sprobably some artillery positions, some logisticspositions...standard kind of rear area activities going on and you're better prepared to deal with this threatthan anyone else. And you're worried that they might be threatening your own rear or your right flank. But it seems to me you're sort of describing the threat interms of threats to you. Tell me a little bit about howyou are reading how this effects the bigger picture--the situation that's starting to develop here withrespect to Red. 

S: Request for

input fromstudent. 

If I were to either press on to AA or 

make an end run around the woods--take the long route--I'mleaving a substantial force in therear area. 

Substantial enough that it is a threat to Div? 

CD 16:48 Yes, Sir. If in fact this is the

advance element of one of these maneuver Bdes, then the rear areaof the Div could be in a lot oftrouble. 

So looking at the bigger picture, you understand that

the Div Cdr's intent is what? 

Is to be in place at this AA for an 

offensive follow-on push. 

So that is the task he gave you--be in the AA--but 

what did he describe as his intent of having you there? 

 B-18 
If I have written this down correctly,

in order to destroy the EN's armor and transition into the offense. 

To destroy the EN's armor, seize the initiative, and 

transition to the offense. And we know that the EN typically likes to lead with a Bn of tanks. The Div Cdr's overall concept is that he wants to launch a counterattack against these guys, blunt theiroffensive, seize the initiative, and transition to the attack. 

Instructor

modelsunderstanding ofintent 

CD 17:46 Right. 

Do you think that's going to happen out of Viettiville? 

(Pause.) It looks like it's going to

happen right here.

Is that what you're thinking? 

Yes, Sir.

OK. Because it wasn't clear to me that was what you 

were thinking, but if it is, I think you're exactly right.In fact, Viettiville is obsolete at this point. The factthat it was AA is not important anymore. That wassimply a control measure. What's really important tothe Cdr is counterattacking, destroying the EN'sarmor and seizing the initiative and maybe it'shappening here. Now, this wasn't our pick, but here'san opportunity, right? 

Exactly.

So is this the Div counterattack?

Yes, Sir. It may as well start here. 

Like you're saying the AA is just a line on the map and at this point ithas become irrelevant. 

 B-19 
CD 18:48 Let's talk a little bit about the timing of your situation.

Why did you decide to make your decision when youdid? What was the key piece of information thatallowed you to make a decision when you made it? 

Q: What cues? 

I had a couple of elements of EN 

information. Hopefully, I knewwhere his lead trace was, and I saw a weakness that he had at the timeand that was about enough that Ineeded to make my decision. 

And the weakness was what? 

The fact that he was in a laager site

which implies he's doing something logistical. He's doing preparationsfor something else.

So he's not necessarily combat ready--all of his

forces. Anything else you see that reinforces that?

Q: Cues. 

The fact that he has so little

reconnaissance forward. And if I don't do something in the next five minutes, we're going to be on parityas far as knowledge.

So you've got presumably about a five minute jump

on this guy in terms of seizing the initiative and the element of surprise. And if you don’t do anything between now and then you lose it. 

Exactly.

 B-20 
Because number 1, he's exposed because he's

doing logistical stuff rather than in a fighting mode. And number 2 he's blind because he doesn't haveanything out there to see where you are so you'vegot the chance to get the jump on him.

What about this truck convoy coming down from

Viettiville? Did that factor into your thinking at all?Why/Why not?

Q: Cues. 

That did not factor into my thinking 

one bit. That left my consciousnessreally early on. 

Not significant?

Could be significant. Right now the 

biggest threat is the armor. The best way for me to carry out myCdr's intent is to focus on the armor. That's a nice lucrative target. It's probably not going tocome my way once this fight startsgoing down, but I may be able to dosome sort of pursuit.

CD 20:51 So you're not really particularly interested in it? It's

not a threat to you; it may be a lucrative target, but you're more interested in destroying the tanks. Doesit tell you anything about the situation as you thinkabout it? 

Q: Cues. 

 B-21 
(Pause.) It may be coming down to 

Vietti's Farm to logistically support this EN that's in the laager site. 

Which tells you what?

Which tells me he's not planning on

moving anywhere for a few hours.

So maybe that reinforces your initial assessment 

that you've got a chance to hit these guys cause they're in need of some kind of resupply. Does it tellyou anything else about geography here?

Q: Cues. 

About the geography?...Obviously

the area to the NE is secure for the EN. 

OK. Why is that significant?

CD 21:49 If I'm in the rear area, there's a gap

in our lines for the EN to get this far in our rear area. 

So does it begin to appear that the situation is

maybe a little bigger than just a tank Bn at Vietti'sFarm?

Summary. 

Yes, Sir. It appears that perhaps

the thrust of this Bde is coming from the NE. 

What does that say about Viettiville as AA? Q: Cues. 

EN is already in that AA. 

Was that a part of your thinking? Did you internalize 

that or miss it altogether? 

 B-22 
To be honest with you, the convoy

dropped from my consciousnessaltogether. 

CD 22:50 Instructorswitches tonew viewpointto summarize earlier info on use of FRassets and link it toVisualization. 

Let's talk about your tactical plan a little bit. You've got three Co. on line: A, B, C. Delta is in reserve VIC Balzerton? They were moving up in trace of Alpha? And then you had your TOW section protecting yourleft flank and A,B,C up engaging the EN. So three tank CO on line engaging a tank Bn. Explain to mehow you see this thing unfolding. 

Bravo and Charlie come up on top 

of the hill. Obviously, the jig is up.They're engaging. Alpha and theLAVs quickly take out the two T72s. We've got the drop on the EN atVietti's Farm and the fact that we've got the weapon with the betterPH/PK is going to carry the day.

So you think this thing is going to be over pretty

quick? 

Yes, Sir.

Alright, then what happens? 

From there, I'm issuing follow-on 

orders to establish a hasty defense in preparation for the rest of thisBde pouring down from Viettiville. 

So you're not going to Viettiville anymore? 

 B-23 
I'm not.

Are you talking to anybody else? Q: Probe T1. 

That was the next thought crossing 

my mind. Obviously I need to let someone know what's going on here. 

1549/ CD 24:19

OK

My report to higher is going to 

sound something like this. Spotted large force EN armor approximately25-30 vehicles VIC Vietti's Farm. Appears to be in AA in prep forfollow-on opns. Follow-on forces in the form of a motorized convoy are approaching from the NE. Amengaging the armor at Vietti's Farm. Will advise. Out. 

What do you think Div's going to say? 

They're probably going to ask for a 

lot more information than I've giventhem. But until I've got my forces up on Balzer Hill and I'm carrying out this attack on the EN laager site, that's less of a concern to me--keeping them informed than fighting this fight. 

CD 25:31 From Div's point of view, are they going to have a 

hard time with this report you're sending them? 

Q: Consider 

situation fromanother's view. 

 B-24 
From here, 

RPD notesonly

They'll probably put together the 

same picture. Viettiville AA is no longer viable and the fight's going to start here at Balzer Hill. 

Hmmm....... You're handling this rather well. Is this

a normal situation? Or is it hard to explain, totallyunexpected? 

This doesn't bother me. It's turning 

me on. I have an enemy weaknessin front of me. 

So you can get into a fight with your advantage 

without having to work for it... 

Exactly

 At the tactical level you like what you are seeing, 

but at the DIV level and what they are seeing and what they are trying to do, is this a good situation?Or is it causing problems? And does that influence the way you behave? Do you need to be looking at the bigger picture, past killing the tanks? And if so, how should that affect your behavior? 

Look from

anotherviewpoint.

What's going to influence my

behavior is where the othermaneuver elements are [Red orBlue?] in relation to me... 

 B-25 
Give me some possibilities here, some

Contingencies?

I may have to plug up this hole in 

the NE that Red is coming through 

So he may be coming down from the NE, that's one.

Did this tank Bn come from the NE? Does that matter? 

No, it doesn't matter 

OK, keep going. Other thoughts?

After the engagement is over? 

Yes, let's assume that you have won. Talk through 

your actions and why.

Move the TOW to the NE, minor 

mods to my positions, but take himon the ground of my choosing. 

So, you're putting your TOW further out to buy you

time, is that right? To learn about these guyssooner? 

Q: PRD 

Yes, sir.

So could they be coming round here? (indicating

track to the N) 

I'd push the TOW up to the NE to

block this avenue [didn't answerquestion] 

 Any other Contingencies?

Friendly or enemy?

Enemy. I'm trying to understand your reading of the 

enemy forces here...

 B-26 
Red is deeper than we imagined. 

Attack is coming from Viettiville, or from the N, that's all the info I havenow.

It could be coming from here [indicates from the 

East]. Does that change anything? 

If it's coming from the E, I'm still not 

in too disadvantageous a position,I'd move my forces a little. 

OK. Let's take a different approach. Let's take a look

at your tactical plan now.

New viewpoint--

PreMortemanalysis.

If your plan were to fail, why, how do you see this

plan failing?

If it were to fail, maybe from an 

overwhelming force coming fromthe N, but I think I have enough forces there to deal with that. 

So, if the enemy comes from the N, that causes you 

problems potentially?

The TOW should be enough, they

will buy me time, and D Co. will buyme time enough to deal with it. 

Any other ways you see this failing? Not necessarily

something that you did wrong... 

 B-27 
If the LAV gave incomplete 

information and there is more out there than just the laager site, and there's a lot of dimounts on BalzerHill. 

So it could be that you have run into a much larger

force than you can see, whether its infantry or mechinf, or tanks, or it’s the rest of the Bde, not just a tankBn. How do you deal with that? Is that plausible, amI being fair with that possibility?

That's fair. What indirect fire assets

do I have? 

You don't have any, you're in an administrative move 

in the rear. You could try to get them... is that an option? 

I could use artillery and air.

Do you think you'll get those without prior

preparation with enough time to bring it into question? 

Yes. I've become the main fight. I'd 

get the assets. More likely airsupport, less likely arty, I'm outsidetheir fan. 

I'd suggest that you need more than that. This is the 

main fight. There's more than just this fight. You'llneed more than that, this is just the beginning. You'll be shaping that for them. You have to be thinking about setting the stage and the inevitable actionsthat will follow this. This guy wouldn't send a tank Bn by itself, he's got something bigger in mind. It's not going to be over in the next half hour. If not you, someone will need the support and whatever you can do to support that, you need to be doing, right? Thoughts about setting the table for the DIV Cdr?

 B-28 
Report to DIV...there's a big fight 

going down in front of me. Need indirect fire and air as soon as you can get it to me. 

You destroyed the tank Bn, are you going into the 

defense to plug this thing? 

I'd call DIV and ask for guidance, 

do you want me to press or hold? 

DIV says you're going to have to wait while they

check out this crazy story about tanks in the rear!

OK, so no support from DIV... 

No, just that they need time to digest the 

information. 

OK, I will go on the defense then. 

OK, why? What's your thought process?

There's more out there, like you 

were saying... 

Why do you think transitioning to the defense is the

way to go versus pressing on? 

I don't know what's further to the

East. 

No terrain suggesting offensive options...? 

Sure.

What do you think about the level of uncertainty

here? Is it a problem? Or is it clear? 

It's somewhat uncertain. I have

three options (AOA for Red). I'mdiscounting one, and I'm set up forthe other two from the NE or East. 

 B-29 
Right, but your assumption is that what you have is

a Bn lead element and you don't know where the rest of the Bde is. You've identified one third of hiscombat power and you're assuming that there'smore. That's a lot of uncertainty, right? How do you cope with that? With your plan, or another way?

Ask for student

input. 

Sure. Circle the wagons, or deal 

with his most likely COA from the NE. 

OK, what are some ways that you can deal with

uncertainty, tactically? 

Shape the terrain, shape the 

situation...

How?

Thrust to the NE, looks like a soft

target right now. 

OK, you could do that. What else? What about use

of your reserve? 

I can hold them back, wait for more

information, or a weakness.

Right, but one way to deal with uncertainty is to hold 

a large reserve right?

Sure.

The bigger the uncertainty, the bigger the reserve. 

Are you comfortable with committing 3/4 of yourcombat power to the plan? Are you comfortable with that? 

Yes.

Why?

ABC are oriented to the NE, TOW

to the N. 

 B-30 
You're not concerned with getting your three tank

Co's decisively engaged in a fight they can't get out of? 

If they do that, I see myself as

accomplishing my mission.

Let's drill back on what we're seeing from the 

enemy. Typically he leads with a Bn of tanks, what'smissing here?

The recon company.

Thoughts?

Maybe they have gone further

south or West. 

So the recon Co has already come through? You've 

been lucky and come up in between them? 

Or they could be at the laager site. 

Yes

I should be prepared for the other

options. 

How?

Well my TOW and D Co. are well 

positioned for the N/NW, and I don't see his recon Co. coming into the flanks of A,B,C as a problem, it wouldn't work out well for them. 

Did you think about this in your plan? Asks student for

self-evaluation.

No, but now we're laying out his 

table of organization, the piece thatis missing was the recon. 

What's one piece of information that you would like 

that would really help you clarify the situation? 

Probe cues. 

 B-31 
Where is Red's main thrust, where 

is the rest of his Bde? 

What would that allow you to do? Probe cues. 

Choose advantageous ground to

defend from or choose a route for a counter attack into Red's flank. 

If you had to choose one or two concepts to 

describe your plan, principles, what would they be, and why?

Seize the initiative, and surfaces(?)

and gaps. 

Explain those a bit? 

Red presented me with a weakness

earlier than expected, it would be foolish not to take advantage. 

So is tempo a big issue here? Reinforce 

learning point.

Right. Five minutes and we would 

have been on parity.

Would you say the time factor was the single biggest 

factor in forcing your decision? 

It almost always is. 

What about surfaces and gaps? 

He's presenting me with this

weakness, I'd be foolish not to takeit. 

What about our surfaces and gaps? From our POV?

The enemy perceived a surface, I 

made it a nasty gap. 

 B-32 
The point is that something happened in DIV 

defense that created a gap for these guys. We're now trying to close it down and maybe turn it around. You need to clarify the situation to DIV andhelp rectify it, set conditions for DIV to deal with the new situation. Looks to me like DIV is dealing with a situation that they weren't aware of, their existingassumptions are out of the window. You're in a position to avert a disaster and you are the guy with your finger in the dyke. You need to be thinking about it as a DIV level problem not just at the tacticallevel.

The friendly situation factored into it 

less than the fight in front of myface.

Why do you think that is? 

Didn't have good SA of DIV forces, 

where they were. 

If I'd given you a better idea of Blue, would that have

changed it? 

Yes, better realization that they had 

penetrated deep into the rear. 

You didn't have a sense of that at the beginning?

Was that a lack of information or just not putting it all together? 

Didn't have enough information at 

the beginning.

Do you have any questions for me? 

 B-33 
I needed more information at the 

front end, an idea of DIV front trace,emphasis that it was an administrative move... 

What did you learn from this? 

Got to think about the bigger picture

and not just the fight in front of me. 

What else?

Look beyond the next half hour, 

what does that Red element mean with respect to the Red big picture... I had a good grasp of the initial situation and coming up with a good solution. 

Yes you did it well, issued orders in about 45

seconds, clearly, except I misinterpreted A Coorders

I didn't use the proper terminology

for orders, attack by fire. 

Hand over to WolfPack...I have two issues. Human factors. When was this

occurring? 

Early morning.

Pre-dawn. After doing what? You'd been on the

road, right? You're a Bn Cdr with 800-1000 people moving down the road, at night... have you everdone that? 

Yes sir, probably everyone but the 

drivers are nodding off.

Nodding off, getting tunnel vision... when you issue 

your orders, you have to take that into account... What are your immediate thoughts on the order in that situation? 

 B-34 
Should probably include some sort

of wake up message, painting moreof the enemy situation. Looking back at my order, I assumed thateveryone had the same enemypicture, which they didn't. 

My order would look like this:Bn alert! Enemy tank

Bn 2-3k to our right! Not even an ordinal direction, to our right! Get them in the ball game. What elseabout going into a fight at that time? 

Got to have simple scheme of

maneuver. From the Redperspective, life's pretty good, he'sdown too. 

Yes, but the point I wanted was, you nailed it. It has

to be simple. Also, you don't want to shoot up yourown guys in a night fight. So take a look at yourforces and where they are...

Reinforce

learning point.

My dispersion of forces is wrong. 

Yes, tight formations. Another factor might be the 

personalities of your commanders, and how theyreact. It might affect who you get to do what. It did me.

My second point is the meta problem. The SA of

your SA. What do you think of the SA that you had? You know what I mean by SA? 

Yes, Sir. Relatively low other than

what's right in front of me. I wasfocused on the weakness.

 B-35 
One of the things I sometimes do is draw on the 

map what I can and can't see... the blue triangle iswhat I can see thru the Sgt in charge of the LAV section and also down the road we just came up. That's the SA of my SA. Me saying to myself what do I know. 

I want to talk about your report to the DIV Cdr. You

said you had a large force, the DIV Cdr is wonderingis that a large force for a Bn Cdr or for a DIV Cdr?

Yes, I think I added numbers after I 

realized what I had said. 

Be specific. 20-25 T-72s 2-3 km from me. I've got to

attack them, I'm in the attack, will advise. 

Tell me about the fight after the fight, your hasty

defense. 

On the screen? 

Sure draw it on the screen.

(Student is drawing...)

Explains what he thinks Red will do... passage of

lines... so, you're going to have a bunch of bad guyscoming. If you're the new (Red) Bn commander on the scene, and you see that there's been a fire fight (at VF), based on the terrain analysis, what would you do? 

I'd dismount outside of gun range 

and probe. 

I agree with the probing, but not the dismounts.

What assets does the Bde Cdr have? I would use a maneuver defense... you want to win the counterrecon fight. Get artillery... (trails off discussion)

End of discussion

 B-36