Adaptive Tutoring for Self-Regulated Learning: 

A Tutorial on Tutoring Systems

by Robert A Sottilare and Anne M Sinatra

ARL-SR-0305 December 2014

Approved for public release; distribution is unlimited.
NOTICES

Disclaimers

The findings in this report are not to be construed as an official Department of the Army position unless 

so designated by other authorized documents.

Citation of manufacturer’s or trade names does not constitute an official endorsement or approval of the 

use thereof.

Destroy this report when it is no longer needed. Do not return it to the originator.
Army Research Laboratory

Aberdeen Proving Ground, MD 21005-5425

ARL-SR-0305 December 2014

Adaptive Tutoring for Self-Regulated Learning: 

A Tutorial on Tutoring Systems

Robert A Sottilare and Anne M Sinatra

Human Research and Engineering Directorate, ARL

Approved for public release; distribution is unlimited.
ii

REPORT DOCUMENTATION PAGE Form Approved

OMB No. 0704-0188

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering 

and maintaining the data needed, and completing and reviewing the collection information. Send comments regarding this burden estimate or any other aspect of this collection of information, 

including suggestions for reducing the burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and Reports (0704-0188), 1215 Jefferson 

Davis Highway, Suite 1204, Arlington, VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to 

comply with a collection of information if it does not display a currently valid OMB control number.

PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS.

1. REPORT DATE (DD-MM-YYYY)

December 2014

2. REPORT TYPE

Final

3. DATES COVERED (From - To)

January 2014–August 2014

4. TITLE AND SUBTITLE

Adaptive Tutoring for Self-Regulated Learning: A Tutorial on Tutoring Systems

5a. CONTRACT NUMBER

5b. GRANT NUMBER

5c. PROGRAM ELEMENT NUMBER

6. AUTHOR(S)

Robert A Sottilare and Anne M Sinatra

5d. PROJECT NUMBER

5e. TASK NUMBER

5f. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

US Army Research Laboratory

ATTN: RDRL-HRT-T

Aberdeen Proving Ground, MD 21005-5425

8. PERFORMING ORGANIZATION

 REPORT NUMBER

ARL-SR-0305

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR’S ACRONYM(S)

11. SPONSOR/MONITOR'S REPORT

 NUMBER(S)

12. DISTRIBUTION/AVAILABILITY STATEMENT

Approved for public release; distribution is unlimited.

13. SUPPLEMENTARY NOTES

14. ABSTRACT

The tutorial described in this report was conducted at the Human-Computer Interaction International Augmented Cognition 

Conference in Creta Maris, Crete, Greece, in June 2014. The purpose of this tutorial was 3-fold: 1) familiarize participants with 

the fundamental principles of adaptive Intelligent Tutoring System (ITS) design; 2) illustrate how ITS design influences/enables 

self-regulated learning; and 3) discuss the need for standards for authoring of ITS, modeling of learners and experts, automated 

instructional strategies, and methods of analysis for ITS technologies. 

15. SUBJECT TERMS

adaptive tutoring, intelligent tutoring systems, self-regulated learning, authoring adaptive systems, adaptive instruction,

educational technology

16. SECURITY CLASSIFICATION OF: 

17. LIMITATION

OF ABSTRACT

UU

18. NUMBER

OF PAGES

80

19a. NAME OF RESPONSIBLE PERSON

Robert A Sottilare

a. REPORT

Unclassified

b. ABSTRACT

Unclassified

c. THIS PAGE

Unclassified

19b. TELEPHONE NUMBER (Include area code)

407-208-3007

Standard Form 298 (Rev. 8/98)

Prescribed by ANSI Std. Z39.18
iii

Contents

1. Tutorial Objectives 1

2. Fundamentals of Tutoring 1

3. Generalized Intelligent Framework for Tutoring 6

4. Learner Modeling 26

5. Instructional Management 42

6. Domain Modeling 48

7. Authoring 50

8. Analysis of Effect 63

Bibliography 67

Distribution List 72
iv

INTENTIONALLY LEFT BLANK.
1

1. Tutorial Objectives

The purpose of this tutorial is 3-fold:

• Familiarize participants with the fundamental principles of adaptive Intelligent Tutoring 

System (ITS) design.

• Illustrate how ITS design influences/enables self-regulated learning (SRL).

• Discuss the need for standards for authoring of ITS, modeling of learners and experts, 

automated instructional strategies, and methods of analysis for ITS technologies.

2. Fundamentals of Tutoring

This section reviews the fundamentals of ITSs. Slide 3 illustrates a typical tutor-user interface 

used by the ITS to deliver content and feedback to the learner, and receive learner input. This 

interface may include a tutor natural language feedback window, which is used by the ITS to 

provide verbal feedback or direction through a virtual human. It usually provides a text feedback 

window, a content presentation window, and a learner response window where the learner 

provides text input in response to ITS directions, questions, or feedback. A running log of the 

conversation or chat window may also be part of the tutor-user interface.

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 3

Typical Tutor-User Interface

NL: Natural Language
2

Slide 4 illustrates the “sweet spot” for ITSs, the area where they may be most useful. This is 

within tutoring for complex skills versus simpler tasks. To support SRL, tutors adapt their 

delivery and the challenge level of scenarios to match those of the learner. If there is no adaptive 

change to match the learner’s needs, then this is simple computer-based training where the 

training is the same for everyone and not adaptive tutoring. 

What is SRL? SRL is learning (acquisition of knowledge or skills) that is guided by 

metacognition (thinking about one’s thinking), strategic action (planning, monitoring, and 

evaluating personal progress against a standard), and motivation to learn. It takes great 

awareness and discipline to guide your own learning. Expert human tutors are much more 

effective and efficient at guiding learning. Just as expert human tutors guide learners, adaptive 

ITSs may also augment SRL by shaping instructional content and scaffolding support to meet the 

learner’s needs. The computer-based ITSs must be situationally aware of the learner’s state and 

the instructional context to be effective guides and support efficient instruction.

As noted in Slide 4, complex skills may include cognitive tasks where challenging decisionmaking and strategic thinking are exercised or affective tasks where interpersonal skills and 

ethical conduct are tested or psychomotor skills where coordination and timing are critical to 

physical tasks (e.g., land navigation) or operating sophisticated systems. ITSs may be used to 

prepare for live training, enhance learning in conjunction with virtual training environments, or 

act as job or decision aids during actual operations.

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 4

Intelligent Tutoring Systems 

Territory

Learning Content

Learning Objectives

Facts

Concepts

Adaptive

Procedures

Abstracted

Concepts

Remember Understand Apply Analyze Evaluate Create

Simple

Procedures

When and Where

- Prior preparation to maximize live training/practice

- Enhance learning within virtual training environments

- Intelligent decision-aiding/mentoring on-the-job

Basic Knowledge & Skills

• Live (classroom, range) 

• Computer-based training

Complex Skills

Intelligent Tutoring Systems

What 

- Cognitive (e.g. complex decision-making, strategic thinking) 

- Affective (e.g., interpersonal skills, ethical conduct)

- Psychomotor (e.g., operating sophisticated weapons/platforms) 

The primary reasons that tutors are limited in their use in education and training (see Slide 5) are

1) they have insufficient ability to adapt to learner needs and 2) they are expensive to author. It 

takes approximately 200 h of interdisciplinary team labor to make 1 h of coursework for tutoring. 

More adaptive equals more expensive.
3

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 5

Limitations of Current ITS

Expensive to author and are insufficiently adaptive* to support highly 

effective and tailored training experiences across a broad spectrum 

of military tasks.

*adaptive systems customize themselves automatically in response to users or changes in 

the environment. 

• Adaptiveness drives the need for additional 

authoring... more authoring, more 

development time, more cost

• Adaptiveness is largely based on knowledge 

of learner performance... other attributes 

influence learning (e.g., individual 

differences)

• ITSs have been primarily applied in limited, 

well-defined domains... with limited 

application to military tasks

Significant evidence exists that computer-based intelligent tutoring systems are just as effective 

as human tutors under certain conditions and generally within more well-defined domains (e.g., 

mathematics, physics) where there is generally one correct answer to a problem. The goal is for 

ITSs to be as effective as expert human tutors under all conditions and domains. The effect sizes 

shown in Slide 6 are baselined against traditional classroom training. The ultimate goal for ITSs 

is to impact learning with effect sizes equivalent to raising average (“C”) students to experts 

(“A” students) through tailored instruction and reinforcement of deep learning principles. In 

other words, the goal is to be as effective as or more effective than expert human tutors.

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 6

Motivation for 

Tutoring Research

Tutoring Methods and Effect Sizes...

2.00 Skilled human tutors (Bloom, 1984) ( ↑ score from 50th to 98th)

1.05 Other tutoring systems ( ↑ median score from 50th to 85th) 

PACT Geometry Tutor (Anderson, Corbett, Koedinger & Pelletier, 1995)

Atlas-Andes (VanLehn, et al., 2005; Rose, et al, 2001)

Diagnoser - physics (Hunt & Minstrell, 1994)

Sherlock (Lesgold, et al., 1988)

0.80 AutoTutor (20 experiments) (Graesser, et al, 2001-present )

0.79 Skilled human tutors (VanLehn, 2011)

( ↑ median score from 50th percentile to 79th)

0.42 Unskilled human tutors (Cohen, Kulik, & Kulik, 1982)

( ↑ median score from 50th percentile to 66th percentile)

0.00 Baseline - traditional classroom training 

 Adapted from information from Dr. Art Graesser, University of Memphis, and Dr. Beverly Woolf, University of Massachusetts - Amherst.

A

B

C
4

Many organizations are feeling the crunch to achieve more with less. While training and 

education remain important to maintaining organizational competency, it may not be practical or 

feasible to achieve and maintain competency in a traditional classroom. One-to-one training 

(Bloom 1984) has been shown (Slide 7) to be more effective than classroom training, but it is not 

practical to have one-to-one tutoring for every person in a large organization.

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 7

Need and S&T Opportunity

Need: A smaller Force requires each Warfighter to have expertise for a greater 

range of skills for complex missions. Need to achieve expertise faster with fewer 

resources. 

Opportunity: Accelerate development of expertise by developing intelligent 

tutoring systems as effective as human tutors – with cost-effective features.

Shift from one-size-fits-all training  affordable, personalized learning.

Enhances mission readiness. Reduces training cost. 

Intelligent 

Tutoring 

System 

So, if we wanted an ideal tutoring system that could adapt to our needs, what would it look like? 

Slide 8 shows a set of salient characteristics for a platinum-level tutor as described by Sottilare 

and Gilbert (2011). 

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 8

Salient Characteristics of an 

Ideal Tutoring System

• self-regulated – support learning of individuals and teams (Army requirement) 

• adaptive – use AI to tailor instruction to the learning needs of individuals and teams of 

Soldiers

• effective & credible – as good or better than an expert human tutor

• relevant – support military training in both ill-defined and well-defined environments

• accurate & valid – use optimal instructional methods based on empirical results

• usable – tailored to different users (trainees, trainers, developers, designers...)

• accessible – service-oriented, available anywhere 24/7/365 

• affordable – easy to author, promotes standards and reuse

• persistent – models the learning needs of Soldiers across their careers

Bronze 

Tutor

Silver 

Tutor

Gold 

Tutor

Platinum 

Tutor

Sottilare, R. and Gilbert, S. (2011). Considerations for tutoring, cognitive modeling, authoring and interaction design in serious 

games. Authoring Simulation and Game-based Intelligent Tutoring workshop at the Artificial Intelligence in Education 

Conference (AIED) 2011, Auckland, New Zealand, June 2011. 
5

Before we continue in our discussion of tutoring systems, we should examine their fundamental 

elements and processes (Slide 9). Nearly every tutoring system has 4 fundamental elements: a 

learner model, a pedagogical (instructional) model, a domain model, and a communication 

model. The green boxes in the slide below show these fundamental elements as modules versus

models because they manage processes in addition to modeling the learner, the instruction, the 

domain, and the communication. 

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 9

Fundamental 

Tutoring Processes

ITSs use the processes in Slide 9 to assess the learner and manage instruction. ITSs are defined 

as intelligent software-based agents that guide instruction by observing and interpreting learner 

data (behaviors, physiology, demographics) to classify learner states (e.g., engagement, 

competency, emotions). The ITS uses these states to adapt/tailor instructions to match the 

learner’s capabilities and needs to optimize learning. Slide 10 shows the interaction between 

tutoring agents, the learner, and the training environment (e.g., simulation, game, presentation).

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 10

Intelligent Tutoring System (ITS) 

Defined

Intelligent computer-guided instruction where softwarebased agents:

• observe and interpret learner data to determine learner states

• adapt and tailor instruction to match the learner’s capabilities and 

needs to optimize effective problem solving and decision-making.
6

Finally, the adaptive tutoring learning effect model shown in Slide 11 illustrates the interactive 

loops of the tutoring process. Learner module data/processes are shown in green boxes. 

Pedagogical module data/process are shown in light blue boxes and domain module processes in 

light orange. 

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 11

Updated Adaptive Tutoring 

Learning Effect Model

blue text = offline processes

red text = key tutor decisions

3. Generalized Intelligent Framework for Tutoring

Traditionally, ITSs are expensive and time consuming to develop. Additionally, they tend to be 

linked to specific content and are not easily changed. Reuse in tutoring systems is virtually

nonexistent. The Generalized Intelligent Framework for Tutoring (GIFT) is an open-source, 

domain-independent intelligent tutoring framework (Slide 13). It is intended to provide 

flexibility and the ability to create full tutors with content of the author’s choice. By being 

domain-independent, it allows for the reuse of materials and a reduction in both the time and 

monetary costs of developing tutoring systems.
7

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 13

Generalized Intelligent 

Framework for Tutoring (GIFT)

An S&T effort to develop an open-source tutoring architecture to: 

– capture best tutoring practices and support rapid authoring, reuse and 

interoperability of ITSs

– lower costs and entry skills needed to author ITSs

– enhance the adaptiveness of ITSs to support self-regulated learning (SRL) 

per the Army Learning Model 

• ontology

• tools

• methods

• standards

• exemplars

Adaptive

Tutoring 

Systems

• Adaptive

• Affordable

• Effective

• Gritty

• Flexible

• Collaborative

• Critical Thinkers

• Automated Authoring

• Automated Instruction

• Accurate Learner Modeling

• Accurate Domain Modeling

• Analysis Tools

The fundamental tutoring processes within GIFT are consistent with the previously described 

components of the tutoring process, which are reviewed in Slide 14.

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 14

Fundamental 

Tutoring Processes

As with most ITSs, GIFT has 4 primary modular components: Domain Module, Learner Module, 

Pedagogical Module, and Gateway Module. Additionally, this diagram represents the process 

that occurs with user input to GIFT when the user is interacting with a training application. 

Additional information about these modules and their processes is provided further along in the 

“GIFT Tutors GIFT” portion of this tutorial. In addition to the 4 primary modules, GIFT also 

provides a sensor module that is used to capture behavioral and physiological data about the 

learner which can then be used to interpret/classify that data into learner states for use by the 

learner module and the pedagogical module as described in the “Adaptive Tutoring Learning 

Effect Chain”. The interaction of the modules is illustrated in Slide 15.
8

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 15

GIFT Modules and Interactions

Sensor

Learner

Domain

(Strategy 

Implementation)

Pedagogical

Gateway

Training 

Application

Domain

(Performance 

Data)

The modules in GIFT can be launched manually through the Module Monitor shown in Slide 16. 

The image on the left is the Module Monitor before the modules have been launched. The image 

on the right is the Module Monitor after the modules have been successfully launched. In GIFT 

3.0 and above, it is also possible to launch the modules in one step, which includes the launching 

of the tutor web page. 

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 16

Slide 17 provides a visual of the Admin Tools in the Module Monitor (left) and the GIFT login 

screen on the tutor web page (right). There is also a simple login page that can be used for 

experiments and does not require a password.
9

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 17

Within the Admin Tools Tab is a series of authoring tools. The GAT (GIFT Authoring Tool) is a 

new element of GIFT 2014-1-X, as is shown in Slide 18. It provides a user-friendly way for 

authors to create their courses in GIFT. The course selection dashboard is pictured in the current 

slide. Each line represents a current course within GIFT, and the green checkmark indicates that 

it is complete and validated. Courses can be opened by highlighting the name and then clicking 

“Edit”.

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 18

GIFT Course Authoring Tool

The functionality of the new GAT is demonstrated in Slide 19. The transitions and elements of 

the course are presented on the left side of the screen. Once one of transitions is selected, the 

associated fields to enter content into are displayed on the right side of the screen. Additional 

transitions can be added using the menu on the top left of the page.
10

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 19

GIFT Course Authoring Tool

Surveys are authored in GIFT through the Survey Authoring System. It provides the ability to 

create questions and surveys. Additionally, it can be programmed such that the surveys are 

automatically graded.

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 20

Survey Authoring Tool

The next set of slides (pp. 11–15) provides an animated sequence that explains GIFT in simple 

terms. GIFT has 3 distinct functions. First, it is an authoring capability to develop new ITS 

components and whole tutoring systems. Second, it’s an instructional manager that integrates 

selected tutoring principles and strategies for use in ITSs. Third, it’s an experimental test bed to 

analyze the effectiveness and impact of computer-based tutoring systems components, tools, and 

methods. GIFT is intended to be a community platform for you to contribute to and to help you

with your research and development. 
11

GIFT is... an Intelligent Tutoring 

System

...an experimental testbed

...an architecture

...a research nexus

...a community platform

... with tools for authoring

... with data extraction/logging

... with interchangeable parts

... as an exit vector for research 

projects

... informed by Advisory Boards

GIFT is based on a learner-centric approach and is therefore designed to be consistent with the 

Adaptive Tutoring Learning Effect Chain. In this chain, the learner data informs learner states, 

which then informs instructional strategy. Selection of the appropriate strategy at the correct time 

is expected to lead to learning gains. 

Each of these processes is captured in individual modules of GIFT. For this tutorial, we will go 

though the functions, inputs, and outputs of each GIFT module. Let’s talk about the first one, the 

Domain Module. Although the Domain Module is one module in reality, it is divided into 2 

modules for you to better understand each of its 2 functions: performance assessment and 

strategy implementation.

The learning effect chain starts by monitoring learner performance and sensor data. Learner 

performance is assessed from data captured in a training application, which is fed into GIFT’s 

Domain Module via the Gateway Module. GIFT processes learner interaction data by comparing 

the Domain Module inputs against designated models of expert performance. In order to keep 

GIFT generic to all training applications, domains are represented as a hierarchical structure of 

“Concepts”.
12

Sensor

Learner

Domain

(Strategy 

Implementation)

Pedagogical

Gateway

Training 

Application

Domain

(Performance 

Data)

• This tutorial will show you GIFT’s Main Modules and Processes.

• The implementation of GIFT is centered around the Adaptive Tutoring Learning 

Effect Chain (ATLEC). Each component is intended to inform strategies to influence 

learning gains.

For all identified Concepts and Subconcepts, a learner can perform “at standard”, “above 

standard”, or “below standard”, as determined by comparsion of performance to the expert 

model. Outputs from the Domain Module are fed into the Learner Module and contain 

performance states associated with specific concepts represented in the domain representation. 

Sensor

Learner

Domain

(Strategy 

Implementation)

Pedagogical

Gateway

Training 

Application

Inputs:

•Student 

Actions 

(as Gateway 

messages)

Outputs:

•Performance Assessments for 

each Concept and SubConcept

•Above-, At-, or BelowExpectation Measurements

•Generic, domain-independent 

message structure

What Does This Module Do?

1. Provides Performance Assessments (*process 

shown*)

2. Implements instructional “Strategies” as “Tactics”

GIFT Supports These Training Applications:

Domain 

(Performance 

Data)

GIFT also uses information collected from sensors to monitor a learner’s reactive states while 

interacting with a system. The Sensor Module takes in raw sensor data streams as inputs and 

applies filters to convert the data into metrics correlated with cognitive and affective states. This 

filtered data is passed to the Learner Module for inferring affective and cognitive state 

determinations. You wouldn’t use the same instruction for a bored student as an energetic one, 

would you? Sensors may be used to collect data about the learner and determine the learner’s 

states. Data from the learner may be filtered or unfiltered.
13

Sensor

Learner

Domain

(Strategy 

Implementation)

Pedagogical

Gateway

Training 

Application

Domain 

(Performance 

Data)

Inputs:

Student Sensor Data

from hardware 

and/or software 

Sensors. 

(It’s your decision!)

Outputs:

Filtered Sensor Information 

based on configuration 

What Does This Module Do?

1. Provides filtered sensor information 

to help assess the learner’s cognitive 

and affective states

GIFT Supports:

So far we’ve learned that performance information from the Domain Module and sensor 

information from the Sensor Module serve as inputs to the next function of the learning effect 

chain, the Learner Module. The Learner Module consumes this information and performs 

processes to designate a “Learner State”, which may be cognitive (e.g., engagement), affective 

(e.g., emotions), or physical (e.g., fatigue or arousal). Once a state is classified, the results are 

then fed to the Pedagogical Module. In addition, the Learner Module maintains information 

about the learner’s stable traits, experiences, and competencies, which are also used to customize 

strategy selections for the individual learner. 

Sensor

Learner

Domain 

(Strategy 

Implementation)

Pedagogical

Gateway

Training 

Application

Domain

(Performance 

Data)

Inputs:

• Filtered Sensor data

• Persistent Learner Data

• Performance State

Outputs: 

“Learner State” as a collection of:

• Affective, Cognitive, and 

Performance Dimensions

• Provides Current, Short-Term, 

and Long-Term views of state

What Does This Module Do?

1. Provides a “Picture” of the Learner 

Independent of training application or sensors

GIFT Supports:

•Current: Changes in performance

•Future: Adaptive Resonance Theory

More research is needed in this area!

The Pedagogical Module uses the “Picture of the Learner” state information and learner trait 

data, and then recommends generic strategies to accommodate individual learners. If the student 

is bored and passing your course, then you probably want to adjust the course. For instance, the 

Pedagogical Module may provide guidance and feedback to aid in performance, adjust course 

difficulty, present more interactive material, or request an assessment of the learner’s ability. 
14

Sensor

Learner

Domain

(Strategy 

Implementation)

Pedagogic

al

Gateway

Training 

Application

Domain

(Performance 

Data)

Inputs:

“Sources of Adaptation”

•Learner Performance State

•Learner Affective/Cognitive State

•Learner Abilities

•Learner Traits

Outputs:

“Generic Instructional Strategy 

Recommendation”

•Provide Guidance

•Adapt Problem/Scenario

•Request Assessment

What Does This Do?

1. Provides generic strategies , such as:

• “Provide Hint” (Microadaptive)

• “Give Practice Content” 

(Macroadaptive)

Training Application Independent GIFT Supports:

An Engine for MacroAdaptive Pedagogy 

(EM2AP)

Once the Pedagogical Module uses learner state information to select a strategy, the strategy

selected is then sent to the Domain Module. The Domain Module then chooses an appropriate 

method to implement the request as a defined tactic or action. The selection of the tactic is based 

on the learner’s ability levels. For example, it may be more appropriate for one learner to receive 

a hint while a metacognitive prompt may be more suitable for another. 

Sensor

Learner Pedagogical

Gateway

Training 

Application

Domain 

(Performance 

Data)

Inputs:

“Instructional Strategy 

Recommendation”

•Provide Guidance

•Adapt Problem/Scenario

•Request Assessment

Outputs:

•Change in user experience

(Dependent on Domain Module 

Triggering the Effect Chain)

•In-game feedback, scenario 

changes, reflective prompts, etc.

What Does This Module Do?

1. Provides Performance Assessments 

2. Implements instructional “Strategies” as 

“Tactics” 

(*process shown*) GIFT Supports:

Virtual Characters (all Domains)

HTML Items (all Domains)

In-game characters (Virtual Medic Sim)

In-game events (Virtual BattleSpace 2)

...

Domain

(Strategy 

Implementation)

The Gateway Module receives the tactic that was output by the Domain Module and presents it 

to the learner. Depending on the type of request, this can involve actions to execute tactics within 

the training application environment (e.g., serious game) or by the Tutor User Interface, a 

window to present information to and receive information from the learner.
15

The Gateway Module translates this interface into GIFT-assessable messages, allowing for the 

whole learning effect to operate. If you already have a training application, the Gateway Module

is where you can build an interface to it and add individualized and research-based instructional 

strategies.

Sensor

Learner Pedagogical

Gateway

Training 

Application

Domain

(Performance 

Data)

Inputs:

•A requested Instructional 

Tactic (i.e. a domain-specific 

application of a generic GIFT 

Instructional Strategy)

Outputs:

•GIFT Tactics for execution

•GIFT messages for 

assessment

What Does This Module Do?

1. Provides an interface to the training application 

for:

• Capture of performance information

• Implementation of requested instructional 

tactics .

GIFT Supports:

INSERT YOUR TRAINING APPLICATION HERE

(Look at the 3 existing Gateways for an example)

Domain

(Strategy 

Implementation)

All of the modules communicate to each other via defined protocols. This means that you can 

easily replace one version of a module with another, which assists in setting up experiments and 

training with a variety of applications. You can make one module and know with confidence that 

the rest of the modules will do their duty. Research done for GIFT can make its way back to the 

community.

You can download GIFT for free (for life) and get support for your development at 

www.gifttutoring.org. 

IMPORTANT THINGS TO REMEMBER ABOUT GIFT

GIFT is Modular

You can change one Domain of instruction for another...

...And still have instructionally valid technique

You can test one pedagogical theory against another...

...And maintain experimental control

You can make an Engagement Detector for a Learner Module

Without knowing about the training domain...

...or how it will affect micro-adaptation

You can put your research to use. 

www.GIFTtutoring.org
16

Slides 30–48 discuss the various releases of GIFT and their affiliated capabilities.

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 30

GIFT 

Releases

GIFT has been continuously developing and growing since its initial release in May 2012. The 

initial release included the fundamental modules and elements that are still part of the current 

releases. An initial version of the Module Monitor is pictured in Slide 31.

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 31

GIFT 1.0

(Baseline Release – May 2012)

• Architecture, Tools & 

Methods

– Domain-Independent Processes

– Service-Oriented Architecture

• Apache Active MQ

– Functional Modules

• Pedagogical, Learner, 

Domain, Sensor, Module 

Monitor, Learning 

Management System 

(persistent Learner Model)

• Tutor-User Interface 

(supports web-enabled 

content)

– Pedagogy - Variable feedback 

based on performance

The initial release also included the ability to record information from sensors, such as EEG

(electroencephalography), a temperature/humidity-sensitive mouse, and a self-assessment sensor. 
17

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 32

GIFT 1.0

(Baseline Release – May 2012)

• Sensors

– EEG – Emotiv Epoch

– Temperature and humidity-sensitive mouse

– Software-based surrogate sensor 

• sensitivity testing

Research has been performed with the various releases of GIFT. An initial experiment was 

performed to assess low-cost sensors and their impact on learning cognition and affect. This 

work helped to inform the future direction of sensors within GIFT. Work is consistently being 

done with GIFT to improve and add to its capabilities. 

The initial release included support for Virtual BattleSpace 2, a serious game, as an external 

training application.

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 33

GIFT 1.0

(Baseline Release – May 2012)

• Tutoring Domains

– Virtual BattleSpace 2

• Training Support Packages

– IED Report

– Surprise Attack

– Room Clearing

• Presence Patrol
18

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 34

GIFT 1.0

(Baseline Release – May 2012)

• Related Experiments Using GIFT

– Low Cost Sensors for Real-time Assessment 

of Learner Cognition and Affect

• Leveraged VBS2 Training Environment

• Emotion Induction Techniques (EITs)

• Classifiers

 Attention

 Distraction

 Drowsiness

 Engagement

 Flow

 Workload

 Anger/

Frustration

 Boredom

 Confidence

 Confusion

 Fear/

Anxiety

 Joy

 Motivation

 Sadness

 Shame

 Surprise

 Wonderment/

Awe

Cognitive States that 

Impact Learning Affective States that Impact Learning

Additional features are being consistently added to GIFT based on both research objectives and 

user input. GIFT 2.0 was upgraded by including a number of different tools that are of use to 

course designers and researchers. 

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 35

GIFT 2.0

(Tools Release – Nov 2012)

• Architecture, Tools & Methods 

Added

– XML-based Authoring Tools to 

support

• Surveys and Tests

• Addition/Removal of Sensors

• Changing Learner Models

• Authoring of 

Content/Assessments

• Creation of Courses and 

Experiments

• Analysis of Experimental 

Data

– Virtual Human Plug-In

• Media Semantics Characters

Additional sensors were added to GIFT 2.0. Once an interface for a commercial sensor is added 

to GIFT, the GIFT community benefits by never having to integrate that sensor again.
19

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 36

GIFT 2.0

(Tools Release – Nov 2012)

• Sensors Added

– Q-Sensor (electro-dermal activity, 

temperature, acceleration)

– Webcam

GIFT 2.0 also included the addition of Microsoft PowerPoint as a training application. This 

addition is extremely useful, as it makes it easier to harness reusable content in the form of 

PowerPoint presentations. Additionally, PowerPoint is a program that individuals of many skill 

levels are familiar with, and it increases the flexibility that GIFT course designers have with their 

included material. 

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 37

GIFT 2.0

(Tools Release – Nov 2012)

• Tutoring Domains Added 

– PowerPoint

• Tactical Combat Casualty Care

– Care under fire principles

– Hemorrhage control

The photo in Slide 38 demonstrates an experiment in action. GIFT is visible on the left side of 

the screen, with the training application on the right. Additionally, the participant is wearing the 

Q-sensor, which measures electrodermal activity. 
20

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 38

GIFT 3.0

(Experiments Release)

GIFT 3.0 included tools such as SIMILE, which made the ability to link real-time interactions to 

learning objectives more straightforward for GIFT course authors. In its present form, SIMILE 

works for TC3Sim/Vmedic; however, future work is being done that will allow it to work with 

additional training applications. Further, the 3.0 release included the engine for Macro and Micro 

Adaptive Pedagogy (eM2AP) for managing instruction and elements of AutoTutor through 

dialogues and tutoring scenarios. 

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 39

GIFT 3.0 (Experiments 

Release – May 2013)

• Architecture, Tools & 

Methods

– Assessment methods 

to link real-time 

interaction to learning 

objectives

• Integrates Student Information Models for Intelligent 

Learning Environments (SIMILE)

– New Pedagogy

• Domain-Independent Macro-adaptive Strategies based 

on learner motivation and expertise

– AutoTutor Dialogues and Tutoring scenarios

GIFT 3.0 included a very useful feature: the ability to export a created tutor. This allows more 

flexibility in the way that an individual student will interact with GIFT. Additionally, a full 

installation of GIFT no longer needs to be completed on each computer that will be running the 

developed tutor.
21

The Human Affect Recording Tool (HART) was included with the release and can be used by 

researchers. HART is an app for the Android platform that implements the Baker-Rodrigo 

Observation Method Protocol (BROMP) 1.0, a method for assessing human affect and 

engagement in field settings, allowing for synchronization between field observations and log 

files of student-software interaction. 

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 40

GIFT 3.0 (Experiments 

Release – May 2013)

• Architecture, Tools & 

Methods (continued)

– Export Tutor Function

– Human Affect 

Recording Tool (HART)

• Android app for assessing 

human affect and engagement 

in field settings per BakerRodrigo Observation Method 

Protocol (BROMP)

HART is described in Baker et al. (2012) and implements the protocol described in Ocumpaugh

et al. (2012).

Additionally, the ability to track facial expressions, posture, and head pose was added with the 

integration of the Microsoft Kinect as a sensor. 

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 41

GIFT 3.0 (Experiments 

Release – May 2013)

• Sensors Added

– Kinect

• Filters

– Kinect

– MultiSense – Perception Markup Language

» real-time facial expressions, body posture

» Schrer, etal (2012)

– Generalized Adaptive View-based 

Appearance Model (GAVAM)

» head pose estimation

» Morency, Whitehill & Movellan (2008)

GIFT 3.0 also included additional example courses that used TC3Sim/Vmedic. Additional 

research was conducted using GIFT as a test bed to examine the self-reference effect in context 
22

of cognitive tasks in a computerized learning environment. The materials that were used in this 

study were released as an additional course titled “Logic Puzzle Tutorial” in GIFT 4.0. 

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 42

GIFT 3.0 (Experiments 

Release – May 2013)

• Related Experiments Using GIFT

– Explicit Feedback with Game-base Training

– Training to Solve Logic Puzzles (cognitive tasks)

– Automated Detection of Engagement and Affect 

Leveraging Interaction Patterns and Sensors

GIFT 4.0 also included an enhanced version of eM2AP, additional sensor filters, and integration 

with new training domains (Marksmanship and Logic Grid Puzzles) based on experimentation 

with GIFT 3.0. 

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 43

GIFT 4.0 (November 2013)

• Architecture, Tools & Methods

– Enhanced Learner Affect Models 

(interaction/sensor)

– Enhanced Performance Assessment Engine

• Pedagogical Authoring functionality added

– New Sensor Filters

• Galvanic Skin Response (GSR) data

• Heart Rate data

– New Training Domains

• Marksmanship

• Logic Grid Puzzles

As mentioned previously, a psychomotor training domain, Marksmanship, was facilitated 

through capabilities in GIFT 4.0. This allows for the development of expert marksmanship 

models. Slide 44 displays the interaction between the different GIFT modules and the marksman 

(learner). 
23

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 44

GIFT 4.0 (November 2013)

• Related Experiments Using GIFT

– Adaptive Marksmanship (psychomotor tasks)

Windows

Messaging

GIFT Gateway

SCATT Plugin

GIFT 

Domain 

Module

SCATT Professional

GIFT 

Learner 

Module

scatt_rt.c

Custom 

Adapter

ActiveMQ

GIFT 

TutorUser 

Interface

GIFT 

Pedagogical 

Module

DIS

protocol

Learner

GIFT 

Sensor 

Module

Physiological

Sensors

Weapon Orientation 

Sensor

Trigger Pressure

Sensor

Further, GIFT 4.0. is being used to assess future capabilities. As it continues to develop, GIFT 

will be adapted to support use by multiple learners in a server-based environment. Further, GIFT 

is being integrated into a mobile learning environment (The US Army Research Laboratory’s 

[ARL’s] Soldier Centered Army Learning Environment [SCALE]) as a learning engine. 

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 45

GIFT 4.0 (November 2013)

• Related Experiments using GIFT (continued)

– Scalability & Performance Testing

• discrete event modeling of network 

performance

• analysis of alternatives

– e.g., nested ActiveMQs

– GIFT as learning engine for SCALE

• Soldier-Centered Army Learning 

Environment

• Validation Experiments

Research efforts are ongoing to mature architecture, authoring tools, and modeling of the learner. 

As they mature, these capabilities are integrated within GIFT.
24

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 46

GIFT 2014-2 (Forecasted 

Capabilities – August 2014)

• Architecture, Tools & 

Methods

– Generalized Microadaptation Strategies

– Automatically 

Generation of Expert 

Models

– Tutor-Game 

Integration tools

– Real-time trainee 

modeling

– Thin/Web Client 

functionality

GIFT 2014-2 and future versions are moving toward separate interfaces and tools for different 

users (learners, power users, and domain experts). In addition, there will be different 

configurations for different situations/environments (e.g., classrooms, experiments, mobile 

learning). 

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 47

GIFT 2014-2 (Forecasted 

Capabilities – August 2014)

• Architecture, Tools & 

Methods (continued)

– Develop/enhance userdependent interfaces

• Learners, Domain 

Experts, Trainers, 

Developers, 

Instructional 

Designers, Researchers

– Configurable deployment 

(classroom, experimental 

station, etc.)

– Virtual Human Toolkit

ARL has the goal for GIFT to be a learning engine for several types of training environments. 

Targets of opportunity include the US Department of Energy’s National Training & Education 

Resource, the US Advanced Distributed Learning Initiative’s Personalized Assistant for 

Learning (PAL), and ARL’s SCALE.
25

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 48

Beyond GIFT 2014-2

• Continue to leverage AutoTutor and AutorTutor

Lite functionality

– Speech Act Analysis, Complex Dialog Management, 

Artificial Intelligence Markup Language, Learner 

Characteristics Curves

• Enhanced cognition and affect detectors

• Extend learning environments to military training 

domains (complex and ill-defined domains)

• GIFT as learning engine for PAL

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 49

Demo of GIFT 

bootup
26

4. Learner Modeling

There are 2 key steps in learner modeling: data acquisition and state classification. Learner data, 

as noted in the adaptive tutoring learning effect model, may be acquired in real-time from 

sensors or learner input (e.g., surveys, interaction with training environment) or in pretutoring 

from a long-term learner model (LTLM) sometimes called a persistent learner model. Slide 51 

examines low-cost sensors for acquiring learner data.

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 51

Learner Modeling

Learner Data Acquisition

WHAT IS IT?

• Tools and methods to unobtrusively acquire learner data (behaviors 

and physiological measures) to support learner state classification 

in real-time using low-cost ($50-200) sensors

WHAT DOES IT OFFER?

• The ability to affordably model learner cognitive, affective and 

psychomotor states in real-time as a basis for instructional 

decisions

TECHNICAL APPROACH:

• Identify a small set of low-cost sensors to acquire learner data to 

support accurate prediction of key learner states:

• attention, engagement, cognitive load

• confusion, boredom, frustration, anxiety, anger

• motivation

EEG ~ $200 Kinect ~ $100 HD Webcam < $50

Slide 52 reviews approaches for using learner data to classify learner states (e.g., emotions, 

engagement, performance, competence). Of particular note are the preliminary findings on this 

slide.

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 52

Learner Modeling

State Classification Methods

Brawner, K. W. (2013). Modeling Learner Mood In Realtime Through 

Biosensors For Intelligent Tutoring Improvements, University of Central 

Florida.

WHAT IS IT?

• Tools and methods to unobtrusively assess learner states (e.g., engagement, performance, 

emotions) in real-time using machine learning techniques

WHAT DOES IT OFFER?

• The ability to accurately model learner cognition, affect and physical states in real-time as 

a basis for tailoring instruction

*Accuracy = (True Positive +True Negative Predictions)/(Total Predictions)

TECHNICAL APPROACH:

• Explore new machine learning methods and develop detectors 

(classifiers) to accurately* predict key learner states

• attention, engagement, cognitive load

• confusion, boredom, frustration, anxiety, anger

• motivation

PRELIMINARY FINDINGS:

• Generalized classifiers are proving to be impractical 

• Offline individual classification models are not reusable

• Real-time classifiers of affect are of good quality (~80% accurate)

• Real-time classifiers of cognition are not as good (<60% accurate) 
27

The learner states of interest during tutoring are primarily centered in 3 areas that moderate 

learning (acquisition of knowledge and skills): cognition, motivation, and affect (see Slide 53). 

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 53

Learner Modeling 

Approaches

• Approach: Investigate & model significant 

influencers of learning

• Cognitive modeling

• cognitive load, engagement (Lepper & Woolverton, 2002)

• attention, distraction, drowsiness, engagement, flow, and 

workload (Carroll, et al, 2011; Kokini, et al, 2012)

• Motivational modeling

• personality, values, goals, interests (Lepper & Woolverton, 

2002)

• Affective modeling

• confusion, boredom, frustration, engagement/flow, curiosity, 

anxiety, delight, and surprise (Graesser & D’Mello, 2012)

• mood modeling - pleasure, arousal, and dominance 

(Mehrabian, 1996; Sottilare & Proctor, 2012; Brawner, 2013)

The results of an experiment that sampled low-cost sensors and their ability to reliably detect 

learner states is shown in Slide 54.

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 54

Learner Modeling 

Approaches (continued)

• Approach: Investigate low cost sensor technologies to inform 

classification of key influencers of learning

• survey of behavioral & physiological sensors (Carroll, et al, 2011)

• state classification using physiological sensors (Brawner and 

Goldberg, 2012; Goldberg & Brawner, 2012; Kokini, et al, 2012)

• EEGs – Advanced Brain Monitoring, Emotiv Epoch, and Neurosky

• GSR sensors – Biopac, Affectiva Q sensor

Brawner, K. and Goldberg, B. (2012). Realtime Monitoring of ECG and GSR Signals during Computer-Based Training. In 

Proceedings of the Intelligent Tutoring Systems (ITS) 2012 Conference, Chania, Crete.

Goldberg, B. & Brawner, K. (2012). Efficacy of Measuring Engagement during Computer-Based Training with Low-Cost 

Electroencephalogram (EEG) Sensor Outputs. In Proceedings of Human Factors and Ergonomics Society Annual Meeting 

(HFES2012). Boston, MA. 

Various machine learning techniques have been evaluated for use in classifying learner states. 

Slide 55 illustrates a clustering technique called “growing neural gas”. Learner data may be 

labeled (supervised), unlabeled (unsupervised) or semi-supervised. If you are interested in 

machine learning techniques, check out WEKA, an open-source software tool for machine 

learning: http://www.cs.waikato.ac.nz/ml/weka/.
28

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 55

Learner Modeling 

Approaches (continued)

Growing Neural Gas Clustering Technique

• Approach: clustering and classification using 

supervised, semi-supervised, and unsupervised

machine learning techniques

• offline modeling

• leading to macroadaptive instructional 

decisions 

• online real-time assessment

• leading to microadaptive instructional 

decisions

In an effort to develop ITS best practices for learner modeling, ARL convened a group of experts 

in 2012 to examine the state of practice, emerging concepts, and future directions. This 

information was captured in the first volume of the Design Recommendations for ITSs, which is 

available for free at: https://gifttutoring.org/documents/42.

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 56

Learner Modeling

Advisory Board

University of Memphis – September 2012

• ARL-HRED (Sottilare, Holden, Goldberg, Brawner, Sinatra)

• University of Memphis (Graesser, Hu, Olney, Pavlik, Rus)

• OSD ADL Initiative (Durlach)

• Carnegie Mellon University (Aleven)

• Worcester Polytechnic Institute (Baker)

• Arizona State University (VanLehn, Burleson)

• Institute for Defense Analyses (Fletcher)

• Office of Naval Research (Perez)

• AFRL (Douglass)

• University of Pittsburgh (Lesgold)

• North Carolina State University (Lester)

• University of Colorado (Nielsen)

• Eduworks, Inc. (Robson)

• University of Massachusetts – Amherst (Woolf)

• University of Canterbury, NZ (Mitrovic)

• University of British Columbia, CA (Conati)

If you were to build a learner model, what kinds of data would you want to capture and what 

states would you want to classify/detect?
29

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 57

What should be in a 

learner model?

Don’t look ahead... 

Make your own list

Learner data, states, traits, and demographics may be useful in the ITS decision processes, but 

data are generally expensive to collect and maintain. So, we want to be selective about which 

data we choose for our model.

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 58

Updated Adaptive Tutoring 

Learning Effect Model

blue text = offline processes

red text = key tutor decisions

LTLMs may span careers or lifetimes of learners and are used to store enduring, variable, and 

transient characteristics of the learner, including traits, states, and demographics.
30

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 59

LTLM Persistent

Long Term Learner Model:

• Traits

• States

• Demographic Data

Competence

Preferences

Trends

Slide 60 shows some student (learner) data that is separated into general information and 

recorded behaviors.

Paneva, D. (2006, September). Use of Ontology-based Student model in Semantic-oriented Access to the Knowledge in Digital 

Libraries. In proc. of HUBUSKA Fourth Open Workshop “Semantic Web and Knowledge Technologies Applications”, Varna, 

Bulgaria (pp. 31-41).

Learner characteristics may be enduring, but even enduring characteristics may change over a 

long period of time. For this reason, the label of long-term learner model may be more 

appropriate than persistent learner model.
31

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 61

Enduring Characteristics... maybe?

•Gender 

•Culture First language

•Physical constraints (e.g., color blind/deaf)

•Qualifications/certifications

•Work History

•Education achievements and history

•Transcripts

•Affiliations

•Values

•Personality 

Slide 62 lists learner characteristics that are variable but fairly persistent. 

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 62

Variable, but fairly persistent long term...

• Second language(s)

• Domain Competencies

• Security Clearance level

• Inductive Reasoning Capacity

• Working memory Capacity

• Divergent Associative Learning Capacity 

(ability to connect new to prior knowledge)

• Learner contact info; identifier

• Learner relations (e.g., teacher/class id)

• Learner security credentials (e.g., password)

• Learner preference information

• Learner portfolio information
32

Slide 63 lists transient characteristics that may be worth tracking/detecting.

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 63

Transient

• Current task

• Current time constraints

• Current affect

• Current goals

• Current location (geographical)

• Current time of day/week

• Current competency model (within learning 

experience)

• Current context (e.g., meeting/car, work, 

home, etc.)

• Physiological state

We just discussed what might be useful to include in an individual learner model. What should 

be in a model of a team of learners?

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 64

What should be in a 

team model?

Don’t look ahead... 

Make your own list
33

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 65

Adaptive Tutoring Learning 

Effect Chain for Teams

Fletcher, J.D. and Sottilare, R. (2013, in press). Shared Mental Models and Intelligent Tutoring for Teams. In Sottilare, R., Hu, X.,

Graesser, A. and Holden, H. (Eds.) Design Recommendations for Intelligent Tutoring Systems: Learner Modeling, Volume I. Army

Research Laboratory.

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 66

Team Model Development 

Process

• Structured Literature Review

• Individual Tutoring

• Team Performance

• Model Constructs

• What models are needed?

• What variables influence each model?

• What variables are observable/unobservable?

• What is the effect size of each variable?

• How do we measure critical variables?

• Structural Equation Models

• Model – Test – Model 
34

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 67

Updated Adaptive Tutoring 

Learning Effect Model

blue text = offline processes

red text = key tutor decisions

In 2013, an extensive review of the literature was undertaken relative to the target team models 

identified by Sottilare et al. (2011) and later described by Sottilare (2013). A sample of the 

outcome for the performance model is shown in Slide 68.

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 68

Filtering Methodology

for Literature Review

20,000 • Found via search terms

6000 • Coded for relevance

300

• Empirical with quantitative information which met our 

meta-analytic criteria (over 10,000 effect sizes)

Team Performance 

Meta- Analysis

332 cases 

examined team 

performance

Contain 

information to 

calculate r

Team Learning 

Meta-Analysis

14 cases 

examined team 

learning outcomes

Contain 

information to 

calculate r

Team Satisfaction 

Meta-Analysis

46 cases 

examined team 

satisfaction

Contain 

information to 

calculate r

Team Viability 

Meta-Analysis

Contain 

information to 

calculate r

21 cases 

examined team 

viability 

The results of the team performance model literature review revealed the following antecedents 

(influencers) of team performance (see Slide 69).
35

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 69

Examining antecedents to effective 

team performance

Outcome K1 N2 Ro Rho % Variance 

accounted for

Overall 183 12910 .18 0.19* 30.0%

Communication 48 3367 .23 0.26* 26.2%

Coordination 25 1798 .21 0.23* 28.8%

Conflict 32 2061 -0.08 -0.09* 23.3%

Coaching/Leader

ship 50 3863 .22 0.24* 33.3%

1. K is the number of effect studies analyzed for this outcome

2. N is the number of individuals evaluated in this outcome

3. Rho is the corrected effect size, Ro is the uncorrected effect size; both weighted for sample size

*Statistically significant; confidence interval excludes zero

The process of identifying antecedents for each of the primary team models continued, and these 

are being used to construct initial team models for GIFT. Moving forward, we will examine the 

relationship between individual and team modeling. Slide 70 shows the interaction between the 

tutoring agents, the training environment, and the learner. Tutoring agents for teams may have 

similar functions but will track progress toward team objectives.

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 70

Individual Interactions Between

Learner, Tutor and Environment
36

You have seen this diagram before—it is the individual tutor loop. 

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 71

Individual Local Tutor Loop

The tutor loop in Slide 72 shows a distributed or server-based interaction between an individual 

learner and the tutor. How will this need to be modified for teams?

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 72

Individual Distributed Tutor Loop

Server

Internet

Going back to the agent-environment-learner interaction for a moment, Slide 73 shows 

interaction between a team of 5 learners and the tutoring agents, a team of 5 learners and training 

environment, and finally, what is different in this diagram, interaction between learners who 

must be accounted for in the team models of the tutor.
37

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 73

Team Interactions Between

Learners, Tutors and Environment

Slide 74 shows individual learners interacting with their own tutor loops (something must 

moderate/manage the team interaction), interaction between the learners (communication, trust, 

performance, shared mental models [cognition, affect]), and how competency influences the 

other models.

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 74

Team Distributed Tutor Loop

Server

Internet

Internet

Internet

Internet

Team modeling has evolved into 6 state models with consideration for 2 additional models for 

context and culture pending the determination of their influence on team performance.
38

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 75

Team State Models

• Performance, Learning

• Viability, Satisfaction

Team Outcome State 

Models

• Surface-Level Composition

• Deep - Level Composition

Team Competency 

State Model

• Team Mental Models

• Transactive Memory Systems

• Situational Awareness

Team Cognitive State 

Models

• Collective Efficacy (Attitudes), Cohesion (Attitudes)

• Justice (Attitudes), Effort (Attitudes)

• Cooperation (Attitudes)

Team Affective State 

Models

• Trust, Psychological Safety Team Trust State 

Models

• Coordination( Mutual Support, Reflexivity, Monitoring, Conflict 

(Task Conflict, Relationship Conflict)/Conflict Management 

• Leadership, Communication, Transition (strategy) and Action 

(tactic) processes, Interpersonal

Team Communication 

Models

• Context

• Culture

Other Elements to 

Consider

Each of the team models is reviewed in Slides 76–82.

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 76

Team Outcome 

State Models

A quantitative or qualitative evaluation of the 

status of team outcomes during any portion 

of the team’s performance period. 

Outcomes include: 

(a) the subjective judgment or objective 

evaluation of team objective 

accomplishment

(b) the degree to which team members 

gained valuable skills or knowledge

(c) the willingness to continue to work with 

the same team in the future

(d) how much the team enjoyed working with 

one another.

Team outcome model updates are event-driven and based on changes to the Team Outcome State Models. As team members 

complete assigned tasks and progress toward team goals/objectives, their performance and learning is registered within the 

individual ITS’ Team Outcome State Models and the ITS then generates an update message to the other ITS so all Team Outcome 

State Models are synchronized with individual contributions to team performance and learning objectives.

• Performance

• Learning

• Viability* 

• Satisfaction*

* Post-exercise
39

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 77

Team Competency 

State Model

Individual team member characteristics that 

have the potential to impact or influence the 

accomplishment of team goals. 

Individual knowledge, skills, abilities, and 

experience in the domain are moderators of 

success.

Interdependence of tasks and matching of 

skills and roles may also influence the 

accomplishment of team goals.

Compatible values (e.g., culture) and 

personality may also be moderators.

This model provides an index of team competency based on a composite of the competence levels of individual team members. 

Successful/unsuccessful performance influences individual competence and may influence team competence. Any significant 

changes in individual performance of team tasks are assessed by the individual competency state model to determine if the threshold 

has been met to change individual competency (e.g., beginner, journeyman or expert). Changes in individual competency may or 

may not be of sufficient significance to affect a change in the team competency state model, but if a change in the team competency 

state model occurs, a message is generated to update the team competency state models of the other team members. 

• Surface Level 

Composition

• Deep Level 

Composition

See Fletcher and Sottilare (2013) for additional insight on Shared Mental Models.

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 78

Team Cognitive 

State Models

An evaluation of the shared cognitive state 

of all team members. This evaluation can 

include the degree to which team 

members:

a) structure knowledge in a similar 

manner

b) understand the roles, expertise, and 

expectations of fellow team members

c) have a shared impression of several 

aspects of the team’s status (e.g., slack 

resources, progression towards team 

objectives)

This state model is a compound model of the Cognitive State of all team members. Cognitive State models already exist as part of

the individual ITS, but synchronization of this information with all the team member’s ITS is critical in assessing the function of the 

team. Depending on the collaborative task and the roles of team members in accomplishing that task, the cognitive model may be 

key in determining instructional strategies. For example, for some tasks the weakest understanding of the task among team members 

may indicate the risk of completing the task successfully. For other tasks, only key team members may need to have higher 

understanding of the task to reach a successful outcome. 

• Team (Shared)

Mental Models

• Transactive Memory 

Systems

• Situational 

Awareness
40

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 79

Team Affective 

State Models

The general of feelings of team members 

towards one another during team 

interaction. This can include, but is not 

limited to feelings regarding 

(a) the team’s ability to accomplish their 

goals 

(b) the team emotional sentiments towards 

one another

This state model is a compound model of the Affective State of all team members. Affective State models already exist as part of the 

individual ITS, but distribution of this information to all the team members ITS is critical in assessing the function of the team. For 

example, if team performance is below expectations and the affective state of one or more team members is negative, knowledge of

their state by other individual ITS provides the opportunity to prompt their associated team members to take action (e.g., 

communicate – support or direct).

• Attitudes (not observable)

• Collective Efficacy

• Cohesion

• Justice

• Effort

• Cooperation

• Trust (next slide)

Values-Attitudes-Behaviors

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 80

Team Trust 

State Models

The shared belief that all team members will fulfill 

their role responsibilities, perform delegated tasks, 

and not attack fellow team members for 

expressing their opinion. 

This state model is a compound model of the Affective State of all team members. Affective State models already exist as part of the 

individual ITS, but distribution of this information to all the team members ITS is critical in assessing the function of the team. For 

example, if team performance is below expectations and the affective state of one or more team members is negative, knowledge of

their state by other individual ITS provides the opportunity to prompt their associated team members to take action (e.g., 

communicate – support or direct). This team state model is a compound model of the trust states existing between team members. 

The trust relationships are bi-direction in that Team Member „A‟ may trust Team Member „B‟ more, the same or less than Team 

Member „B‟ trusts Team Member „A‟. Trust is influenced by several factors including perceived competency, perceived integrity, 

perceived benevolence, knowledge of the other team members (Hung, Dennis and Robert, 2004) and perceived benefits of the 

relationship (Gujral, DeAngelis, Fullam and Barber, 2006). Since teams work toward common goals where roles and responsibilities 

are distributed, perceived competency is an essential element of team performance. The perception that other team members may

be unable to perform their tasks is detrimental to trust and team performance. Personality may also play a part in trust.

Individuals with low openness and/or high neuroticism scores in the Five Factor Model of Personality (McCrae and Costa, 1994) may 

have developed habits unfavorable to the development of trust. Low openness scores might indicate an unwillingness to disclose 

information while high neuroticism scores might result in more frequent perception of events/interactions as negative. Positive or 

negative emotions can also influence the assimilation of information (Linnenbrink and Pintrich, 2002) and thereby communications, 

understanding and trust. 

• Trust

• Credibility

• Psychological Safety 

(Openness)
41

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 81

Team Communications 

State Models

Observable behaviors between group members 

which either directly impact progression towards task 

completion or indirectly facilitate synchronization 

between team members.

This model is composed of interaction data between team members for the purpose of observing team cohesion and task execution. 

Providing accurate information in accordance with operating procedures, providing communications when asked, repeating 

communications to ensure delivery, sharing information and acknowledging receipt of information are all vital actions observed in 

teams with effective communication skills (U.S. Coast Guard, 1998). In team settings communication among members builds holistic

situational awareness and coordinates future actions to be carried out. Based on events and interactions in a scenario, team 

members are responsible for updating one another in real-time

•Coordination

•Mutual Support

•Reflexivity

•Monitoring

•Conflict

•Task Conflict

•Relationship Conflict

•Leadership

•Leadership Styles

•Shared Leadership

•Leadership Behaviors

•Communication (frequency, 

source, receiver(s)

•Transition Processes

•Action Processes

•Interpersonal Processes

•Conflict Management

•Organizational Citizenship 

Behaviors (OCBs)

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 82

Other Influencing Factors

• Context 

• Task or Project 

Characteristics

• Autonomy

• Interdependence

• Familiarity 

• Team Tenure

• Organization Type

• Leader Characteristics 

(Age/Tenure)

• Organizational Resources

• Culture

• Individual Culture

• Team Culture

• Organizational Culture

• National Culture

• Team Climate

• Organizational Climate

• Diversity
42

The team model development process is shown in Slide 83.

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 83

Team Model Development 

Process

• Structured Literature Review

• Individual Tutoring

• Team Performance

• Model Constructs

• What models are needed?

• What variables influence each model?

• What variables are observable/unobservable?

• What is the effect size of each variable?

• How do we measure critical variables?

• Structural Equation Models

• Model – Test – Model 

5. Instructional Management

Good instructional practices are hard to replicate. Modeling expert human tutors is a good place 

to start.

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 85

Instructional Management 

Approaches

• Approach: Model successes of expert 

human tutors

• INSPIRE* model (Lepper, Drake & 

O’Donnell-Johnson, 1997)

• facts about human tutoring (Person & 

Graesser, 2003)

• importance of questioning (Dillon, 1988) 

• relation between deep reasoning 

questions and exam scores (Graesser & 

Person, 1994)

• politeness strategies (Person, et al., 

1995)
43

Dialogue-based tutors take turns interacting with the learner (see Slide 86) to improve the quality 

of their answers without giving them the answer. Dialogue-based tutors guide the learner and 

assess their understanding of concepts. GIFT has incorporated many of the web-based services 

used in AutoTutor and AutoTutor Lite to support dialogue-based tutoring.

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 86

1. Tutor asks a question. 

2. Student answers the question.

3. Tutor gives feedback on the answer. 

4. Tutor and student collaboratively 

improve the quality of (or embellish) 

the answer. 

5. Tutor assesses student’s 

understanding of the answer”

Tutoring Process (Person, et al, 1995, p. 167)

Instructional Management 

Approaches (continued)

SIMILE is a tool to link learner actions in a simulation or game to performance assessments and 

ultimately to tutor decisions about instructional options (e.g., feedback, support, change in 

challenge level). 

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 87

• Approach: leverage objective-task framework in 

existing assessment engines (e.g., SIMILE)

• Student Information Models for 

Intelligent Learning 

Environments (SIMILE)

• standardized, adaptable, and 

generic mechanism for learner 

assessment in simulated training 

environments

• middleware with tools for the 

creation of assessment models 

that are distinct and separate 

from the simulation itself

– Example Shown: Model rules 

for applying a tourniquet in 

the TC3 vMedic Trainer 

Instructional Management 

Approaches (continued)

The relationship between the tutor and the learner is critical to the learner’s engagement and 

motivation. The learner should perceive the tutor as credible and supportive for significant 

learning to occur. While this might not be important for single exposures of the learner to the 

tutor, over the long term, it is critical for the learner to develop a relationship with the 

technology.
44

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 88

Instructional Management 

Approaches (continued)

• Approach: Investigate the influence of the learner’s 

perception of the tutor

• credibility and supportiveness of the tutor (Holden, 2012)

• learner expectations of the tutor (Holden and Goldberg, 

2011)

• social pedagogical agents (Kim, Xu and Sharif, 2008)

• characteristics of learning companions (Kim, 2007; Kim, 

Baylor, Shen, and PALS Group, 2007)

Holden, H. (2012). Understanding the Interaction Behavior of Pedagogical Agents' Emotional Support and Competency on 

Learner Outcomes and Agent Perceptions. In Proceedings of the Intelligent Tutoring Systems (ITS) 2012 Conference, Chania, 

Crete.

Holden, H. and Goldberg, B. (2011). Student Expectations and Tutor Acceptance: The Impact on Student Affect (Mood) and 

Future Usage Intentions. International Defense & Homeland Security Simulation Workshop in Proceedings of the I3M 

Conference. Rome, Italy, September 2011.

Another approach to managing instruction is to categorize instructional differences in different 

learning domains. Research is ongoing to determine generalized instructional management 

methods across and within learning domains.

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 89

Instructional Management 

Approaches (continued)

• Approach: Investigate the influence of learning type in 

selecting effective instructional strategies in computer-based 

tutoring

• cognitive learning (Anderson and Krathwohl, 2000)

• affective learning (Krathwohl, et al., 1964; Goleman, 1995)

• psychomotor learning (Simpson, 1972)

• social learning (Sottilare, et al., 2011; Soller, 2001)

• hybrid learning

Sottilare, R., Holden, H., Brawner, K. and Goldberg, B. (2011). Challenges and Emerging Concepts in the Development of 

Adaptive, Computer-based Tutoring Systems for Team Training. Interservice/Industry Training Systems & Education Conference, 

Orlando, Florida, December 2011.

Each of the learning categories (cognitive = thinking, affective = feeling, and psychomotor

= doing) is reviewed in Slides 90–93.
45

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 90

Cognitive Learning

• Cognitive learning (thinking) – Anderson & 

Krathwohl, 2001

– behaviors indicating increasingly complex and abstract 

mental capabilities 

– Creating (high): ability to put parts together to form a new whole

– Evaluating: ability to judge the value of learned material

– Analyzing: ability to break down material into its component parts

– Applying: ability to use learned material in new situations

– Understanding: ability to grasp the meaning of material

– Remembering (low): ability to recall previously learned material

Anderson, L. W., and Krathwohl, D. R. (Eds.). (2001). A taxonomy for learning, teaching and assessing: A revision of Bloom's Taxonomy of 

Educational Objectives: Complete edition, New York : Longman.

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 91

Affective Learning

 Affective learning (feeling) – Krathwohl, Bloom 

and Masia, 1964

– behaviors indicating emotional growth; the manner in which 

we handle emotions, such as feelings, values, appreciation, 

enthusiasms, motivations, and attitude

– Characterizing (commitment - high): has a value system that controls 

their behavior

– Organizing (responsibility): organizes values into priorities; comparing, 

relating and synthesizing values

– Valuing (appreciation): the worth or value a person attaches to a 

particular object, phenomenon, or behavior

– Responding (interest): active participation on the part of the learner

– Receiving (awareness - low): awareness, willingness to hear, selected 

attention

Krathwohl, D.R., Bloom, B.S., and Masia, B.B. (1964). Taxonomy of Educational Objectives: Handbook II: Affective Domain. New York: David 

McKay Co.
46

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 92

Cognition, Affect and 

Adaptive Tutoring

Pictures courtesy of Graesser & D’Mello (2012)

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 93

Psychomotor Learning

• Psychomotor learning (doing) – Simpson, 1972

– includes physical movement, coordination, and the use of the 

motor-skill areas; development of these skills requires practice 

and is measured in terms of speed, precision, distance, 

procedures, or techniques in execution.

– Origination (high): creating new movement patterns to fit a particular 

situation

– Adaptation: skills well developed and can be modified to fit special 

requirements

– Complex Overt Response: skillful performance of complex movements

– Mechanism: learned responses have become habitual

– Response: early stages in learning complex skill; imitation; trial & error

– Set: readiness to act

– Perception (low): ability to use sensory cues to guide motor activity

Simpson, E. (1972). The classification of educational objectives in the psychomotor domain: The psychomotor domain. Vol. 3. Washington, DC: 

Gryphon House.

Another approach to enhanced adaptive instruction is to implement best practices based on 

learning theories. Two theories are Component Display Theory (Merrill et al. 1992) and the Zone 

of Proximal Development (Vygotsky 1978), which are described in Slides 94 and 95,

respectively.
47

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 94

Component Display Theory 

(Merrill, et al, 1992)

• Strategies that account for the learner’s 

state: 

• Gain attention and motivate

• Adapt to prior knowledge

• Adapt to type of knowledge being presented

• Adapt to learner attributes

• Adapt to the learner’s ability (IQ, EQ, adaptability...)

Merrill, D. , Reiser, B, Ranney, M., and Trafton, J. (1992). Effective Tutoring Techniques: A Comparison of Human Tutors and Intelligent Tutoring 

Systems. The Journal of the Learning Sciences, 2(3), 277-305. 

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 95

Automated Instruction

Zone of Proximal Development

WHAT IS IT?

• Tools and methods to adapt instruction based on 

the learner’s state (e.g., cognitive and affective) 

and level of domain competence 

WHAT DOES IT OFFER?

• The ability to select real-time instructional 

strategies which automatically lower/raise the 

training scenario challenge level and regulate the 

type and frequency of scaffolding (support) 

provided to the learner 

(current state)

(available actions)

(associated reward)

(next state)

TECHNICAL APPROACH:

• Literature Review to identify negative learning 

states (e.g., anxiety, boredom, frustration) and 

associated effective strategies

• Develop classification rules for incorporation 

within GIFT (*eM2AP)

• Develop a Markov Decision Processes to address 

uncertainty associated with learner state 

classification

Level of

Challenge

Level of Competence

What the learner

can achieve with

assistance

*eM2AP = engine for Macro & Micro-Adaptive Pedagogy
48

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 96

Adaptive Tutoring Methods:

• Learner Modeling & Instructional 

Strategy/Tactic Selection

• Data Mining

• Machine learning classifiers & 

clustering techniques

• Markov Decision Processes

• Instructional Management

• Component Display Theory 

(Merrill, et al, 1992)

• Domain-independent strategies

• Macro-adaptation

• Micro-adaptation

• Domain-dependent tactics

Adaptive Tutoring Research 

Technical Approach

Enhanced Adaptive Tutoring Learning Effect Chain for Individuals

Merrill, D. , Reiser, B, Ranney, M., and Trafton, J. (1992). Effective Tutoring Techniques: A Comparison of Human Tutors and Intelligent Tutoring Systems. The Journal of the 

Learning Sciences, 2(3), 277-305. 

Reinforcement Learning in Markov Decision Processes

Component Display Theory implement in GIFT

Generalized Intelligent Framework for Tutoring 

(GIFT)

6. Domain Modeling

While domain modeling is primarily associated with presentation of content to the learner, it also 

includes the following aspects: modeling of experts (also known as an ideal student model),

which is used to assess the progress of the learner and identify errors and misconceptions; 

assessment of performance; and presentation of tactics (actions by the tutor to present content, 

change content, provide feedback, or provide support).

Domain models for typical ITSs today are in well-defined domains (math or physics are popular) 

and are generally procedural in nature (simple). Very few tutors cover psychomotor domains and 

are generally desktop and cognitive. ARL is exploring methods to expand domains to allow them 

to support tutoring in a wider range of domains and a broader range of dimensions, as shown in 

Slide 98.
49

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 98

Domain Modeling

WHAT IS IT?

• Tools and methods to facilitate expansion of 

traditional tutoring domains (well-defined, 

simple, static) to a broader spectrum of 

military tasks ranging from well-defined to illdefined, from simple to complex, and from 

static to dynamic

WHAT DOES IT OFFER?

• The ability to support effective tutoring 

during training in increasingly ill-defined, 

complex, and dynamic training domains

TECHNICAL APPROACH:

• Investigate and identify a range of military 

training tasks along three dimensions: 

definition, complexity and dynamics

• Discover and innovate methods to support 

the adaptive tutoring learning effect chain for 

military training tasks of varying levels of 

definition, complexity, and dynamics

well-defined

ill-defined

complex

simple

static

dynamic

What is unique about elements of the adaptive tutoring 

learning effect chain as a function of definition, 

complexity and dynamics of tutoring domains?

• learner data

• learner states

• instructional strategies

• instructional tactics

• learning outcomes

Dynamic interaction modes and their associated characteristics are explored in Slides 99 and 

100.

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 99

Adaptive Tutoring Research 

Technical Approach
50

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 100

Domain Modeling

Interaction Modes

WHAT IS IT?

• Research, tools and methods to facilitate interaction with learners and delivery of 

instruction during a broad spectrum of military tasks ranging from static to highly dynamic

WHAT DOES IT OFFER?

• The ability to support effective tutoring during Soldier training in increasingly dynamic 

training modes:

• static (desktop training)

• limited dynamic (adaptive marksmanship training)

• enhanced dynamic (multi-learner tasks in instrumented spaces)

• in-the wild (instrumented learners) 

Evaluation of interaction is a necessary precursor to support tutoring in dynamic 

military training domains (e.g., embedded training, mixed and augmented reality 

training, and live training)

TECHNICAL APPROACH:

• Evaluate interaction required to support each training mode and 

discover methods to facilitate:

• capture of learner and environment data

• assessment of learner state

• selection of optimal instructional strategies and tactics

• presentation of tactics (e.g., feedback, direction)

An example of a more dynamic psychomotor domain is presented in Slide 101.

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 101

Domain Modeling

Exemplar: Adaptive Marksmanship 

Automates 

accurate 

performance 

assessment 

assessments learner data

Army Marksmanship Unit

Tailored 

feedback

learning strategies

Expanding tutoring capabilities

to support unique, complex and 

ill-defined Army training domains

Sensor 

feedback

7. Authoring

Authoring may be divided into 2 primary areas: reuse and automation. Improving methods to be 

able to use existing training environments as adaptive tutors is a major goal of our research. 

Automated authoring methods are the most critical need in ITS development today. Tutors are 

expensive and laborious to author, and they require sets of specialized skills to develop them. A 

long-term goal is to have any person with expert domain knowledge be able to author an 

effective tutoring system.
51

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 103

Authoring Goals

Authoring Goals for GIFT 

(adapted from Murray, 1999; Murray, 2003)

• Decrease the effort (time, cost, and/or other resources) for 

authoring and assessing CBTS;

• Decrease the skill threshold by tailoring tools for specific 

disciplines to author, assess and employ CBTS;

• Provide tools to aid the designer/author/trainer /researcher 

organize their knowledge;

• Support (i.e., structure, recommend, or enforce) good design 

principles (in pedagogy, user interface, etc.);

• Enable rapid prototyping of CBTS to allow for rapid 

design/evaluation cycles of prototype capabilities.

• Employ standards to support rapid integration of external 

training/tutoring environments (e.g., games) (Sottilare & Gilbert, 

2011)

Sottilare, R. and Gilbert, S. (2011). Considerations for tutoring, cognitive modeling, authoring and interaction design in serious 

games. Authoring Simulation and Game-based Intelligent Tutoring workshop at the Artificial Intelligence in Education Conference

(AIED) 2011, Auckland, New Zealand, June 2011. 

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 104

Authoring Approaches

• Approach: Investigate methods and develop tools for rapid 

automated development of expert models

• Expert model development is an intensive process

• Exploring automated methods to author via text analysis

• Integrate with tutoring system(s) which make 

instructionally relevant decisions 
52

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 105

• standard structures 

and graphical user 

interfaces for a 

variety of users

• GUIs based on 

function (e.g., 

researcher) and 

functional 

competency 

• learners 

• subject matter experts

• instructional system 

designers

• system developers

• trainers

• researchers

• Approach: functional user modeling

Authoring Approaches

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 106

• Approach: learner affect modeling

• what does the tutor 

need to know about 

the learner to 

classify their affect?

• how does the tutor 

get that 

information?

• which affective 

states are important 

to recognize?

• how does 

classification of 

state influence 

instructional 

decisions?

Graesser and D’Mello (2012)

Authoring Approaches
53

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 107

• Approach: learner configuration authoring tool 

• simple interface for 

authoring learner models 

• tree structure driven by 

XML schema 

• prevents learner model 

authoring errors by 

validating against the 

learner model XML 

schema 

• provides ability to validate 

learner model using GIFT 

source w/o having to 

launch the entire GIFT 

architecture

Authoring Approaches

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 108

• Approach: sensor configuration authoring tool

• behavioral 

sensors

• physiological 

sensors

• state 

classification 

models

• Implemented sensors

• Affectiva QSensor 

• electro-dermal activity (EDA)

• skin temperature and acceleration 

• Emotiv EEG

• temperature and humidity mouse (custom)

• Surrogate sensors for temp, humidity and 

assessment

• Sensors under consideration

• NeuroSky and ABM EEGs

• Webcam (1Hz)

• Zephyr heart rate monitor

• Sonar distance sensor

• Pressure chair (custom)

• Pupil diameter (custom)

• Design Interactive EmoPro

Authoring Approaches
54

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 109

Passive Sensing – Q Sensor

Bob playing in traffic : )

Research question: what is the minimum set of 

sensors needed to assess engagement, workload, 

motivational level and emotional state? 

electro-dermal activity

temperature

X, Y, and Z acceleration

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 110

• Approach: Domain Knowledge File (DKF) authoring 

tool (DAT)

• simple interface for 

authoring DKFs 

• tree structure driven by 

XML schema 

• prevents DKF authoring 

errors by validating 

against DKF XML 

schema 

• provides ability to 

validate DKF content 

using GIFT source w/o 

having to launch the 

entire GIFT architecture

Authoring Approaches
55

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 111

• Approach: survey authoring tool

• author questions

• author surveys

• assign surveys

• present surveys

Authoring Approaches

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 112

• Approach: leverage elements of existing tutoring 

systems (e.g., AutoTutor & AutoTutor Lite)

Authoring Approaches
56

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 113

• Approach: game-based tutoring

• prototype 

integration with 

VBS2

• real-time 

feedback

• learner model 

influences 

challenge level 

within game

Authoring Approaches

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 114

Game-based Tutoring

Game World Learner

learner 

observes 

world 

learner acts 

on world 

Tutoring Agent(s)

agent 

observes 

world 

agent 

observes 

learner 

agent acts 

to change 

world

agent acts 

to provide 

feedback or 

instruction

agent 

observes 

effect

on 

learning 

agent 

observes 

effect

on game 

objectives 
57

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 115

Game-based Tutoring

Sottilare, R. and Gilbert, S. (2011). Considerations for tutoring, cognitive modeling, authoring and interaction design in serious games. Authoring Simulation 

and Game-based Intelligent Tutoring workshop at the Artificial Intelligence in Education Conference (AIED) 2011, Christchurch, New Zealand, June 

2011. 

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 116

Automated Authoring

Standards and Reuse 

Existing Simulators

Engagement Skills Trainer

Virtual

Humans

Serious Games

Existing Tutoring Systems

AutoTutor Web Services

Existing Content

PowerPoint

Content Authoring Tools Standards

Define data structures and interface 

protocols for common tutoring 

components

Promote Reuse

Automate development of interfaces with external training & authoring 

capabilities to support interoperability and reuse
58

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 117

Automated Authoring

Expert Modeling

WHAT IS IT?

• Tools, and methods to automate the development of expert models (modeling desired 

trainee behaviors) for use by adaptive tutors; 

• Expert models, part of the GIFT domain module, are used to assess learner performance 

and the correctness of learner actions during tutoring

WHAT DOES IT OFFER?

• Reduced time, cost, and skill needed to develop expert models for training domains

TECHNICAL APPROACH:

• Investigate and develop methods to automate expert model authoring by extracting rules, 

principles, tasks, standards, conditions and hierarchical relationships from text in field 

manuals and other text-based data sources through data mining techniques

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 118

Authoring Reuse: Interoperability 

with External Environments

DOE’s 

National 

Training & 

Education 

Resource 

(NTER)
59

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 119

Tools for Rapid Development of 

Expert Models (TRADEM)

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 120

Tools for Rapid Development of 

Expert Models (TRADEM)
60

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 121

Tools for Rapid Development of 

Expert Models (TRADEM)

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 122

Tools for Rapid Development of 

Expert Models (TRADEM)
61

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 123

Tools for Rapid Development of 

Expert Models (TRADEM)

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 124

Tools for Rapid Development of 

Expert Models (TRADEM)
62

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 125

Tools for Rapid Development of 

Expert Models (TRADEM)

GIFT-compatible Tutor

or

Chatbot Tutor

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 126

Situated Pedagogical Authoring

• What is it?

– Providing ITS content in an environment that is as

similar to the learner’s environment as possible

– think WYSIWYG

– Hypotheses: Situated authoring will enable authors to... 

– Gain competence with the authoring tool more quickly

– Produce more complete and pedagogically effective 

intelligent tutoring content

(than less situated authoring tools)
63

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 127

SPA Prototype

see & hear the 

virtual human

test

assessment tallies

analyze

identify relevant 

domain knowledge

track 

progress

interact just as 

the learner 

would

8. Analysis of Effect

An important part of ITS research is to determine the best (most effective) methods of learner 

modeling, instructional management (pedagogical strategies/recommendations), and domain 

modeling (e.g., expert modeling, assessment, and content and tactics presentation) to optimize 

outcomes (e.g., learning, performance, retention, competence).

How might this research and analysis be enabled?

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 129

Analysis

Analysis Testbed

WHAT IS IT?

• Investigation and creation of a prototype design tool to support planning, execution, and 

analysis of results for comparative evaluations of adaptive tutoring systems, components, 

tools, models, and methods

WHAT DOES IT OFFER?

• The ability to rapidly assess the effect of a particular tutoring technology compared to 

existing or emerging techniques

TECHNICAL APPROACH:

• Identify and prototype tools to conduct analyses:

• tutor vs. traditional instructional methods

• intervention vs. non-intervention studies

• learner model comparison studies

• instructional strategies/tactics comparison studies

• ablative tutoring studies

+

Analyses

=

most effective

adaptive tutoring 

systems, components, 

tools, models, and 

methods learners
64

The analysis test bed within GIFT allows for scientific evaluation of each element and 

subelements of a tutoring system design. The test-bed methodology shown in Slide 130 is based 

on Hanks et al.’s (1993) test-bed approach. Effect sizes analyzed using this test bed are based on 

Cohen’s d (1992).

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 130

Analysis

Analysis Testbed

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 131

Event Reporting Tool (ERT)
65

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 132

GIFT available at 

www.GIFTtutoring.org

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 133

Demo of GIFT 

Tools
66

SFC Paul Ray Smith Simulation & Training Technology Center

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 134

Thank you for your attention!

Questions?
67

Bibliography

Anderson JR, Corbett AT, Koedinger KR, Pelletier R. Cognitive tutors: lessons learned. The 

Journal of the Learning Sciences. 1995;4:167–207.

Anderson LW, Krathwohl DR. A taxonomy for learning, teaching and assessing: a revision of 

bloom’s taxonomy of educational objectives: complete edition. New York (NY): Longman; 

2001.

Army Training and Doctrine Command (US). The United States Army learning concept for 

2015. Fort Monroe (VA): Army Training and Doctrine Command (US); 2011.

Baker RSJd, Gowda SM, Wixon M, Kalka J, Wagner AZ, Salvi A, Aleven V, Kusbit G, 

Ocumpaugh J, Rossi L. Sensor-free automated detection of affect in a cognitive tutor for 

algebra. In: Proceedings of the 5th International Conference on Educational Data Mining; 

2012 Jun 19–21; Chania, Greece; p.126–133.

Bloom BS. The 2 sigma problem: the search for methods of group instruction as effective as oneto-one tutoring. Educational Researcher. 1984;13(6):6.

Brawner KW. Modeling learner mood in realtime through biosensors for intelligent tutoring 

improvements. [doctoral dissertation]. [Orlando (FL)]: University of Central Florida; 2013.

Brawner K, Goldberg B. Realtime monitoring of ECG and GSR signals during computer-based 

training. In: Proceedings of the Intelligent Tutoring Systems (ITS) Conference; 2012;

Chania, Crete, Greece.

Brawner K, Holden H, Goldberg B, Sottilare R. Recommendations for modern tools to author 

tutoring systems. In: Proceedings of the Interservice/Industry Training Systems & Education 

Conference; 2012 Dec; Orlando, FL.

Carroll M, Kokini C, Champney R, Fuchs S, Sottilare R, Goldberg B. Assessing trainee states in 

intelligent tutoring systems: using low cost sensors for affective and cognitive state 

modeling. Interservice/Industry Training Systems & Education Conference; 2011 Dec; 

Orlando, FL.

Cohen J. Quantitative methods in psychology: a power primer. Psychological Bulletin. American 

Psychological Association, Inc. 1992;112(1):155–159. 

Cohen PA, Kulik JA, Kulik CLC. Educational outcomes of tutoring: a meta-analysis of findings. 

American Educational Research Journal. 1982;19:237–248.

Dillon TJ. Questioning and teaching: a manual of practice. New York (NY): Teachers College 

Press; 1988.
68

Fletcher JD, Sottilare R. Shared mental models and intelligent tutoring for teams. In: Sottilare R, 

Hu X, Graesser A, Holden H, editors. Design recommendations for intelligent tutoring 

systems: learner modeling, volume I. Adelphi (MD): Army Research Laboratory (US); 2013.

Goldberg B. Explicit feedback within game-based training: examining the influence of source 

modality effects on interaction. [doctoral dissertation]. [Orlando (FL)]: University of Central 

Florida; 2013.

Goldberg B, Brawner K. Efficacy of measuring engagement during computer-based training with 

low-cost electroencephalogram (EEG) sensor outputs. Proceedings of Human Factors and 

Ergonomics Society Annual Meeting (HFES2012); 2012; Boston, MA. 

Goldberg B, Brawner KW, Sottilare R, Tarr R, Billings DR, Malone N. Use of evidence-based 

strategies to enhance the extensibility of adaptive tutoring technologies. Paper presented at 

the Interservice/Industry Training, Simulation, and Education Conference; I/ITSEC. 2012;

Orlando, FL.

Goleman D. Emotional intelligence. New York (NY): Bantam Books; 1995. 

Graesser AC, D’Mello S. Emotions during the learning of difficult material. In: Ross BH, editor.

Psychology of learning and motivation. New York (NY): Academic Press; 2012. Volume 57;

Chapter 5; p. 183–225. ISSN 0079-7421, ISBN 978012-3942937, 10.1016/B978-0-12-

394293-7.00005-4.

Graesser AC, McNamara DS. Self-regulated learning in learning environments with pedagogical 

agents that interact in natural language. Educational Psychologist. 2010;45:234–244.

Graesser AC, Person NK. Question asking during tutoring. American Educational Research 

Journal. 1994;31:104–137.

Hanks S, Pollack ME, Cohen PR. Benchmarks, test beds, controlled experimentation, and the 

design of agent architectures. AI Magazine. 1993;14 (4).

Holden H. Understanding the interaction behavior of pedagogical agents’ emotional support and 

competency on learner outcomes and agent perceptions. In: Proceedings of the Intelligent 

Tutoring Systems (ITS) 2012 Conference; 2012; Chania, Crete, Greece.

Holden H, Goldberg B. Student expectations and tutor acceptance: the impact on student affect 

(mood) and future usage intentions. international defense & homeland security simulation 

workshop. In: Proceedings of the I3M Conference. 2011 Sep; Rome, Italy.

Holden H, Sottilare R, Goldberg B, Brawner K. Effective learner modeling for computer-based 

tutoring of cognitive and affective tasks. In: Proceedings of the Interservice/Industry 

Training Systems & Education Conference; 2012 Dec; Orlando, FL.
69

Honey MA, Hilton M, editors. Learning science through computer games and simulations. 

Washington (DC): National Academies Press; 2011. 

Hunt E, Minstrell J. A cognitive approach to the teaching of physics. In: McGilly K, editor.

Classroom Lessons. Cambridge (MA): MIT Press; 1994. p. 51–74.

Kim Y. Desirable characteristics of learning companions. International Journal of Artificial 

Intelligence in Education. 2007;17(04):371–388.

Kim Y, Baylor AL, Shen E. PALS Group. Pedagogical agents as learning companions: the 

impact of agent affect and gender. Journal of Computer-Assisted Learning. 

2007;23(03):220–234.

Kim Y, Xu B, Sharif A. Pedagogical agents rendering social context to an online learning 

environment MathGirls. International Transactions on Systems Science and Applications. 

2008;4(2):99–106.

Kokini C, Carroll M, Ramirez-Padron WX, Hale K, Sottilare R, Goldberg B. Quantification of 

trainee affective and cognitive state in real-time. In: Proceedings of Interservice/Industry 

Training, Simulation, and Education Conference (I/ITSEC2012). 2012; Orlando, FL.

Krathwohl DR, Bloom BS, Masia BB. Taxonomy of educational objectives: handbook II: 

affective domain. New York (NY): David McKay Co.; 1964.

Lepper MR, Drake M, O’Donnell-Johnson TM. Scaffolding techniques of expert human tutors.

In: Hogan K, Pressley M, editors. Scaffolding student learning: instructional approaches and 

issues. New York (NY): Brookline Books; 1997. p. 108–144.

Lepper M, Woolverton M. The wisdom of practice: lessons learned from the study of highly 

effective tutors. In: Aronson J, editor. Improving academic achievement: impact of 

psychological factors on education. New York (NY): Academic Press; 2002. p. 135–158. 

Lesgold AM, Lajoie S, Bunzo M, Eggan G. Sherlock: a coached practice environment for an 

electronics trouble shooting job. Pittsburgh (PA): University of Pittsburgh, Learning 

Research and Development Center; 1988.

Mehrabian A. Pleasure-arousal-dominance: a general framework for describing and measuring 

individual differences in temperament. Current Psychology. 1996;14:261–292.

Merrill D, Reiser B, Ranney M, Trafton J. Effective tutoring techniques: a comparison of human 

tutors and intelligent tutoring systems. The Journal of the Learning Sciences. 1992;2(3):277–

305.
70

Murray T. Authoring intelligent tutoring systems: an analysis of the state of the art. International 

Journal of Artificial Intelligence in Education. 1999;10(1):98–129.

Murray T. An overview of intelligent tutoring system authoring tools: updated analysis of the 

state of the art. In: Murray T, Blessing S, Ainsworth S, editors. Authoring tools for advanced 

technology learning environments. Dordrecht (The Netherlands): Kluwer Academic 

Publishers; 2003. p. 491–545.

Ocumpaugh J, Baker RSJd, Rodrigo MMT. Baker-Rodrigo observation method protocol 

(BROMP) 1.0. Training Manual Ver. 1.0. New York (NY): EdLab, Teachers College, 

Columbia University; 2012. 

Person NK, Graesser AC. The Tutoring Research Group. Fourteen facts about human tutoring: 

food for thought for ITS developers. AI-ED 2003 Workshop Proceedings on Tutorial 

Dialogue Systems: With a View Toward the Classroom; 2003; Sydney, Australia.

p. 335–344.

Person NK, Kreuz RJ, Zwaan RA, Graesser AC. Pragmatics and pedagogy: conversational rules 

and politeness strategies may inhibit effective tutoring. Cognition and Instruction. 

1995;13(2):161–188.

Rose CP, Jordan PW, Ringenberg M, Siler S, Vanlehn K, Weinstein A. Interactive conceptual 

tutoring in Atlas-Andes. In: Moore JD, Redfield C, Johnson WL, editors. Artificial 

intelligence in education: AI-Ed in the wired and wireless future. Washington (DC): IOS;

2001. p. 256–266.

Simpson E. The classification of educational objectives in the psychomotor domain: the 

psychomotor domain. Vol. 3. Washington (DC): Gryphon House; 1972.

Soller A. Supporting social interaction in an intelligent collaborative learning system. 

International Journal of Artificial Intelligence in Education. 2001;12(1):40–62.

Sottilare R. Considerations in the development of an ontology for a generalized intelligent 

framework for tutoring. International defense & homeland security simulation workshop. In:

Proceedings of the I3M Conference; 2012 Sep; Vienna, Austria.

Sottilare R. Characterizing an adaptive tutoring learning effect chain for individual and team 

tutoring. In: Proceedings of the Interservice/Industry Training Systems & Education 

Conference; 2013 Dec; Orlando, FL.

Sottilare R. Future concepts for learner modeling. In: Sottilare R, Hu X, Graesser A, Holden H,

editors. Design recommendations for intelligent tutoring systems: learner modeling, volume 

I. Army Research Laboratory (US); 2013.
71

Sottilare R, Gilbert S. Considerations for tutoring, cognitive modeling, authoring and interaction 

design in serious games. Authoring Simulation and Game-Based Intelligent Tutoring 

workshop at the Artificial Intelligence in Education Conference (AIED); 2011 Jun;

Auckland, New Zealand.

Sottilare R, Goldberg B. Designing adaptive computer-based tutors to accelerate learning and 

facilitate retention. Journal of Cognitive Technology: Contributions of Cognitive 

Technology to Accelerated Learning and Expertise. 2012;17(1):19–34.

Sottilare R, Goldberg S. Research gaps for adaptive computer-based tutoring systems. In: Best C, 

Galanis G, Kerry J, Sottilare R, editors. Fundamental issues in defence simulation & 

training. London (UK): Ashgate Publishing; 2013.

Sottilare R, Goldberg B, Brawner K, Holden H. A modular framework to support the authoring 

and assessment of adaptive computer-based tutoring systems (CBTS). In: Proceedings of the 

Interservice/Industry Training Systems & Education Conference; 2012 Dec; Orlando, FL.

Sottilare R, Holden H, Brawner K, Goldberg B. Challenges and emerging concepts in the 

development of adaptive, computer-based tutoring systems for team training. In: Proceedings 

of the Interservice/Industry Training Simulation & Education Conference; 2011 Dec; 

Orlando, FL.

Sottilare R, Holden H, Goldberg B, Brawner K. The generalized intelligent framework for 

tutoring (GIFT). In: Best C, Galanis G, Kerry J, Sottilare R, editors. Fundamental issues in 

defence simulation & training. London (UK): Ashgate Publishing; 2013.

Sottilare R, Proctor M. Passively classifying student mood and performance within intelligent 

tutoring systems (ITS). Educational Technology Journal & Society. 2012;15(2):101–114.

VanLehn K. The relative effectiveness of human tutoring, intelligent tutoring systems, and other 

tutoring systems. Educational Psychologist. 2011;46(4):197–221.

VanLehn K, Lynch C, Schulze K, Shapiro JA, Shelby R, Taylor L, et al. The Andes physics 

tutoring system: lessons learned. International Journal of Artificial Intelligence and 

Education. 2005;15(3):147–204.

Vygotsky LS. Mind in society: the development of higher psychology processes. Cambridge 

(MA): Harvard University Press; 1978.

Woolf BP. A roadmap for education technology. Arlington (VA): National Science Foundation; 

2010. Report No.: 0637190. 
72

1 DEFENSE TECHNICAL

(PDF) INFORMATION CTR

DTIC OCA

2 DIRECTOR

(PDF) US ARMY RESEARCH LAB

RDRL CIO LL

IMAL HRA MAIL & RECORDS MGMT

1 GOVT PRINTG OFC

(PDF) A MALHOTRA

1 ARMY RSCH LABORATORY – HRED

(PDF) RDRL HRM D

T DAVIS

BLDG 5400 RM C242

REDSTONE ARSENAL AL 35898-7290

1 ARMY RSCH LABORATORY – HRED

(PDF) RDRL HRS EA DR V J RICE

BLDG 4011 RM 217

1750 GREELEY RD

FORT SAM HOUSTON TX 78234-5002

1 ARMY RSCH LABORATORY – HRED

(PDF) RDRL HRM DG J RUBINSTEIN

BLDG 333

PICATINNY ARSENAL NJ 07806-5000

1 ARMY RSCH LABORATORY – HRED

(PDF) ARMC FIELD ELEMENT

RDRL HRM CH C BURNS

THIRD AVE BLDG 1467B RM 336

FORT KNOX KY 40121

1 ARMY RSCH LABORATORY – HRED

(PDF) AWC FIELD ELEMENT

RDRL HRM DJ D DURBIN

BLDG 4506 (DCD) RM 107

FORT RUCKER AL 36362-5000 

1 ARMY RSCH LABORATORY – HRED

(PDF) RDRL HRM CK J REINHART

10125 KINGMAN RD BLDG 317

FORT BELVOIR VA 22060-5828

1 ARMY RSCH LABORATORY – HRED

(PDF) RDRL HRM AY M BARNES

2520 HEALY AVE 

STE 1172 BLDG 51005

FORT HUACHUCA AZ 85613-7069

1 ARMY RSCH LABORATORY – HRED

(PDF) RDRL HRM AP D UNGVARSKY

POPE HALL BLDG 470 

BCBL 806 HARRISON DR

FORT LEAVENWORTH KS 66027-2302

1 ARMY RSCH LABORATORY – HRED

(PDF) RDRL HRM AT J CHEN

12423 RESEARCH PKWY

ORLANDO FL 32826-3276

1 ARMY RSCH LABORATORY – HRED

(PDF) RDRL HRM AT C KORTENHAUS

12350 RESEARCH PKWY

ORLANDO FL 32826-3276 

1 ARMY RSCH LABORATORY – HRED

(PDF) RDRL HRM CU B LUTAS-SPENCER

6501 E 11 MILE RD MS 284

BLDG 200A 2ND FL RM 2104

WARREN MI 48397-5000

1 ARMY RSCH LABORATORY – HRED

(PDF) FIRES CTR OF EXCELLENCE 

FIELD ELEMENT

RDRL HRM AF C HERNANDEZ

3040 NW AUSTIN RD RM 221

FORT SILL OK 73503-9043

1 ARMY RSCH LABORATORY – HRED

(PDF) RDRL HRM AV W CULBERTSON

91012 STATION AVE 

FORT HOOD TX 76544-5073

1 ARMY RSCH LABORATORY – HRED

(PDF) RDRL HRM DE A MARES

1733 PLEASONTON RD BOX 3

FORT BLISS TX 79916-6816

8 ARMY RSCH LABORATORY – HRED

(PDF) SIMULATION & TRAINING

TECHNOLOGY CENTER

RDRL HRT COL G LAASE

RDRL HRT I MARTINEZ

RDRL HRT T R SOTTILARE

RDRL HRT B N FINKELSTEIN

RDRL HRT G A RODRIGUEZ

RDRL HRT I J HART

RDRL HRT M C METEVIER

RDRL HRT S B PETTIT

12423 RESEARCH PARKWAY

ORLANDO FL 32826
73

1 ARMY RSCH LABORATORY – HRED

(PDF) HQ USASOC

RDRL HRM CN R SPENCER

BLDG E2929 DESERT STORM DRIVE

FORT BRAGG NC 28310

1 ARMY G1

(PDF) DAPE MR B KNAPP

300 ARMY PENTAGON RM 2C489

WASHINGTON DC 20310-0300

ABERDEEN PROVING GROUND

12 DIR USARL

(PDF) RDRL HR

L ALLENDER

P FRANASZCZUK

RDRL HRM

P SAVAGE-KNEPSHIELD

RDRL HRM AL

C PAULILLO

RDRL HRM B

J GRYNOVICKI

RDRL HRM C

L GARRETT

RDRL HRS

J LOCKETT

RDRL HRS B

M LAFIANDRA

RDRL HRS C

K MCDOWELL

RDRL HRS D

A SCHARINE

RDRL HRS E

D HEADLEY

RDRL HRT T

R SOTTILARE
74

INTENTIONALLY LEFT BLANK.