Adaptive Intelligent Tutoring System (ITS) Research in 

Support of the Army Learning Model—Research Outline

by Robert A. Sottilare

ARL-SR-0284 December 2013

Approved for public release; distribution unlimited. 
NOTICES

Disclaimers

The findings in this report are not to be construed as an official Department of the Army position 

unless so designated by other authorized documents.

Citation of manufacturer’s or trade names does not constitute an official endorsement or 

approval of the use thereof.

Destroy this report when it is no longer needed. Do not return it to the originator.
Army Research Laboratory

Orlando, FL 32826

ARL-SR-0284 December 2013

Adaptive Intelligent Tutoring System (ITS) Research in 

Support of the Army Learning Model—Research Outline

Robert A. Sottilare

Human Research and Engineering Directorate, ARL

Approved for public release; distribution unlimited. 
ii

REPORT DOCUMENTATION PAGE Form Approved

OMB No. 0704-0188

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the 

data needed, and completing and reviewing the collection information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing the 

burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302. 

Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently 

valid OMB control number.

PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS.

1. REPORT DATE (DD-MM-YYYY)

December 2013

2. REPORT TYPE

Final

3. DATES COVERED (From - To)

NA

4. TITLE AND SUBTITLE

Adaptive Intelligent Tutoring System (ITS) Research in Support of the Army 

Learning Model—Research Outline

5a. CONTRACT NUMBER

5b. GRANT NUMBER

5c. PROGRAM ELEMENT NUMBER

6. AUTHOR(S)

Robert A. Sottilare

5d. PROJECT NUMBER

5e. TASK NUMBER

5f. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

U.S. Army Research Laboratory

ATTN: RDRL-HRT-T

Orlando, FL 32826

8. PERFORMING ORGANIZATION

 REPORT NUMBER

ARL-SR-0284

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR'S ACRONYM(S)

11. SPONSOR/MONITOR'S REPORT

 NUMBER(S)

12. DISTRIBUTION/AVAILABILITY STATEMENT

Approved for public release; distribution unlimited.

13. SUPPLEMENTARY NOTES

14. ABSTRACT

Current Army standards for training and education are group instruction and classroom training also known as one-to-many 

instruction. Recently, the Army has placed significant emphasis on self-regulated learning (SRL) methods to augment 

institutional training. Per the Army Learning Model (ALM), Soldiers will be largely responsible for their own learning. Oneto-one human tutoring has been shown to be significantly more effective than one-to-many instruction, but is not practical. An 

alternative to one-to-one human tutoring is one-to-one computer-based tutoring using Intelligent Tutoring Systems (ITSs),

which have been shown to be effective in promoting individual learning in static, simple, well-defined domains (e.g., 

mathematics). To be practical, high authoring costs and limited adaptiveness barriers must be addressed. This outline 

describes a strategy to address key ITS design challenges and expand the horizons of SRL. Research is needed to: reduce 

cost/skill to author ITSs; enhance the adaptiveness of ITSs; and expand ITSs domains to support more dynamic, complex, and 

ill-defined domains to match the Army’s operational mission. The interdependent nature of Army tasks also requires tutoring 

of squads and other teams. The intent of this report is to inform and educate stakeholders, and focus potential collaborators on 

relevant issues within the adaptive tutoring research space.

15. SUBJECT TERMS

self-regulated learning methods, intelligent tutoring system, army learning model adaptive tutoring

16. SECURITY CLASSIFICATION OF:

17. LIMITATION

 OF 

 ABSTRACT

UU

18. NUMBER

 OF 

 PAGES

26

19a. NAME OF RESPONSIBLE PERSON

Robert A. Sottilare, Ph.D.

a. REPORT

Unclassified

b. ABSTRACT

Unclassified

c. THIS PAGE

Unclassified

19b. TELEPHONE NUMBER (Include area code)

(407) 208-3007

Standard Form 298 (Rev. 8/98)

Prescribed by ANSI Std. Z39.18
iii

Contents

List of Figures iv

1. Background 1

2. Army S&T Requirements for Adaptive Intelligent Tutoring Systems 3

2.1 Integrated Training Environment (WFO T1)..................................................................3

2.2 Accessible Learning Capability (WFO T2).....................................................................3

2.3 Enhanced Gaming Capability (WFO T4)........................................................................3

2.4 Individual Training for Tactical Tasks (WFO T5)..........................................................3

2.5 Adaptive Training and Innovative Learning (WFO T8) .................................................4

3. Research Goals 4

4. Research Vectors 6

4.1 Learner Modeling............................................................................................................6

4.2 Automated Instruction.....................................................................................................9

4.3 Domain Modeling..........................................................................................................10

4.4 Automated Authoring....................................................................................................11

4.5 Analysis.........................................................................................................................12

5. References 14

Bibliography 16

List of Symbols, Abbreviations, and Acronyms 18

Distribution List 19
iv

List of Figures

Figure 1. Effectiveness of various tutoring methods. ......................................................................1

Figure 2. Adaptive tutoring learning effect chain............................................................................2

Figure 3. Adaptive tutoring research vectors...................................................................................6

Figure 4. Domain dimensions........................................................................................................11

Figure 5. GIFT adaptive tutoring testbed.......................................................................................13
1

1. Background

The current Army standards for training and education are group instruction and classroom 

training also known as one-to-many instructional methods. Classroom training has been 

generally focused on acquiring knowledge and applying knowledge in proxies for live training 

environments (e.g., desktop virtual simulations, serious games). Small group instruction in live 

environments has also been used to assess application of knowledge and development of skills. 

A standard feedback mechanism for Army training is the after-action review (AAR) where 

significant decision points and actions are captured for small group discussion conducted after 

the completion of a training event. 

Recently, the Army has placed significant emphasis on self-regulated learning methods to 

augment institutional training. Under the Army Learning Model (ALM) concept (U.S. Army 

Training and Doctrine Command, 2011), Soldiers will be largely responsible for their own 

learning, but will clearly still need personal guidance or tutoring to be effective. One-to-one 

human tutoring has been shown to be significantly more effective than one-to-many instructional 

methods (e.g., classroom instruction: Bloom, 1984; VanLehn, 2011), but it is not practical or 

affordable to have one human tutor for each Soldier in the Army. 

A practical alternative to one-to-one human tutoring is one-to-one computer-based tutoring

methods such as Intelligent Tutoring Systems (ITSs). Figure 1 compares the effect sizes of 

various tutoring methods. ITS methods vary from 0.80 to 1.05 σ as compared to tradition 

classroom training (0.00 σ). These are promising results and equate to an increase of a letter 

grade or more improvement over other methods.

Figure 1. Effectiveness of various tutoring methods.
2

ITSs have been shown to be effective in promoting learning in static, relatively simple, welldefined domains (e.g., mathematics, physics) for individual learners. For our purposes, static 

domains are desktop learning tools where the movement of the Soldier is restricted to activities 

conducted while seated. Static domains can effectively support tasks involving knowledge 

acquisition and decision making but are currently less useful for training tasks involving learner 

motion (e.g., psychomotor learning) and perception.

While ITSs technologies are promising, ITSs are expensive to author and are insufficiently 

adaptive to support tailored, self-regulated training experiences across a broad spectrum of 

military tasks as required by the ALM. In order for ITSs to be practical on a large scale, barriers

to adoption such as expense and limited adaptiveness must be addressed. Adaptive tutors must 

automatically adjust the challenge and level of support to meet each Soldier’s learning needs in 

real time. To accomplish this, adaptive tutors, like their human counter-parts, must be able to:

• acquire learner data (e.g., behaviors),

• use learner data to determine the learner’s state (e.g., confusion), and

• select an optimal instructional strategy (e.g., metacognitive prompt) to address the learner’s

state resulting learning gains (e.g., knowledge acquisition).

This sequence of events is known as the adaptive tutoring learning effect chain (Sottilare, 2012; 

Sottilare et al., 2013) and is shown in figure 2.

Figure 2. Adaptive tutoring learning effect chain.

Research is also needed to support broad accessibility; develop tools to reduce cost and skill 

needed to author ITSs; enhance the perception of ITSs; improve the adaptiveness of instruction 

to meet individual learner needs; and expand ITS domains to support more dynamic, complex, 

and ill-defined tutoring of tasks that match the Army’s operational mission. The interdependent 

nature of Army tasks also requires modeling of and feedback for teams of Soldiers as well as 

individuals. 

This research outline describes a focused strategy to address key adaptive tutoring design 

challenges and expand the horizons of self-regulated learning. The strategy identifies Army 

Science and Technology (S&T) requirements for new capabilities, research goals for adaptive 
3

tutoring, research vectors to discover new methods and innovate existing methods, and transition 

mechanisms.

2. Army S&T Requirements for Adaptive Intelligent Tutoring Systems

The Army S&T community uses Warfighter Outcomes (WFOs) as the authoritative source for 

identifying Warfighter needs. WFOs are used to share research and future technology solutions. 

In the training and education (T&E) domain, the Adaptive Tutoring Research program is 

targeting elements of five WFOs.

2.1 Integrated Training Environment (WFO T1)

A portion of ARL’s adaptive tutoring research is aiming to support interaction between ITSs and 

elements of the existing Army training infrastructure to provide an integrated training 

environment (ITE). Goals include compatibility and interoperability with Army serious games 

and simulations to support engaging and effective learning experiences. 

2.2 Accessible Learning Capability (WFO T2)

Accessible learning is being directly addressed by the research and development of Generalized 

Intelligent Framework for Tutoring (GIFT) (Sottilare et al., 2013), an adaptive tutoring 

architecture that is modular and service-oriented. GIFT will also support access to adaptive 

tutoring resources (domain content, assessments, Web services) via the Internet and allow 

content to be presented in a Web browser.

2.3 Enhanced Gaming Capability (WFO T4)

The integration of adaptive tutoring capabilities with engaging serious game capabilities will 

enable more effective training. Automating the integration games and tutors is a goal of the 

adaptive tutoring research program. This will reduce the cost, time, and skill needed to integrate 

Army training games and ITS.

2.4 Individual Training for Tactical Tasks (WFO T5)

The evolving adaptive tutoring architecture, GIFT, is being evaluated across a broad variety of 

tasks that include individual tactical tasks. Future capabilities will enable individual and small 

unit tasks across cognitive, affective, psychomotor, and social domains. Research to examine 

how tutoring approaches vary based on the dynamics, definition, and complexity of tasks is 

ongoing and described in Section 4. Research Vectors, 4.3 Domain Modeling, later in this report.

This research will enable guided learning across a broad spectrum of Soldier tasks that are not 

possible with today’s ITS or training simulation technology alone. 
4

2.5 Adaptive Training and Innovative Learning (WFO T8)

This last WFO is being directly addressed through adaptive tutoring research and the

development of GIFT, which will support a testbed capability to evaluate various adaptive 

tutoring technologies in a variety of training domains. This research will provide empirical 

evidence to guide ITS design decisions and develop standards of performance captured within 

GIFT to support reuse and may be used to develop effectiveness in the future. 

3. Research Goals

The foundational goal of adaptive tutoring research at the U.S. Army Research Laboratory 

(ARL) is to model the perception and behaviors of expert human tutors to support practical, 

effective, and affordable learning experiences guided by computer-based agents. Ten goals have 

been identified for the adaptive tutoring research program to conduct research to:

1. Significantly lower the cost and skills needed to author effective adaptive tutoring systems 

for the U.S. Army.

2. Make ITS easily accessible and available at the Soldier’s point-of-need.

3. Enable ITS to autonomously perceive the learner’s state and adapt (tailor) instruction to 

meet each Soldier’s individual learning needs.

4. Enhance Soldier engagement to promote more efficient and effective learning and 

retention.

5. Accelerate learning to reduce each Soldier’s time-in-training while retaining the same level 

of learning and retention.

6. Discover adaptive tutoring methods to support team tutoring and collaborative learning to 

match unit-level training needs so teams of Soldiers can train as they fight.

7. Discover, innovate, leverage, and transition adaptive tutoring technologies (tools and

methods) resulting in enhanced learning and retention capabilities and higher levels of 

reuse for training and education.

8. Blur the lines between tutoring and training technologies to support more effective guided 

learning experiences, which reuse the engaging content of simulation with the automated 

tailored pedagogy of ITS (e.g., game-based tutoring) to produce Soldiers who persevere, 

adapt, collaborate, and think critically.

9. Support the unique U.S. Army training scope in domains of higher complexity, higher 

dynamics, and lower definition for both individuals and teams of Soldiers so they can train 

as they fight.
5

10. Conduct Soldier evaluations of adaptive tutoring technologies (tools and methods) to 

determine their learning effect and establish best practices.

A set of longer range objectives are tied directly to the salient characteristics of a vision for 

adaptive tutoring termed platinum tutors. The characteristics of platinum tutors, as opposed to 

the characteristics of ITS of lower capabilities (e.g., bronze, silver and gold tutors), were derived 

from Sottilare and Gilbert’s (2011) initial description of tutoring capability levels. Elements of 

this list align closely with the INSPIRE (intelligent, nurturant, Socratic, progressive, indirect, 

reflective and encouraging) model of tutoring described by Lepper and Woolverton (2002). The 

revised list of platinum characteristics is presented in the following:

• Adaptive—refers to the use of artificial intelligence techniques to perceive and make 

optimal decisions to tailor instruction in real time to meet the self-regulated and blended 

learning needs of individual Soldiers and team of Soldiers. Adaptive systems are able to 

customize themselves in response to changes in the learner and/or the training 

environment.

• Credible—instills trust in the learner through demonstrated expertise in the domain being 

tutored; always provides valid information; more effective than an expert human tutor in 

one-to-one and one-to-many training/tutoring environments.

• Personal and communicative—instills trust in the learner through positive personal 

communication that encourages learner-initiated dialog and question asking leading to 

higher learner performance.

• Relevant—able to support military training across a broad spectrum of operational tasks 

ranging from ill-defined domains to well-defined domains, from simple procedures to 

multidimensional complex activities, and from relatively static interaction to highly 

dynamic interaction.

• Accurate and valid—nearly 100% accurate or at least more accurate than an expert human 

tutor in assessing the state(s) of the learner; uses optimal instructional methods based on

the current/projected state of the learner; instructional methods are derived through 

rigorous experimentation.

• Usable—tailored tools, methods, and interfaces to support different types of users (e.g., 

learners, trainers/instructors, training and tutoring system developers, instructional system 

designers) by structuring their knowledge for easy retrieval.

• Accessible—service-oriented, available anywhere 24/7/365 to support Soldier learning at 

the point of need.

• Affordable—time to author is reduced from years/months to days/hours; noncomputer 

scientists can easily author; promotes standards and reuse to lower development cost.
6

• Persistent—models the learning capabilities and needs of Soldiers across their careers; 

instills trust in the learner by accurately modeling and anticipating the needs of the learner.

4. Research Vectors

To support the goals previously identified, the adaptive tutoring research team has organized 

their research around five vectors as shown in figure 3. The first three vectors (blue boxes) focus 

on the knowledge and methods needed to support the adaptive tutoring real-time process and are 

specifically tied to the adaptive tutoring learning effect chain. The last two vectors (gray boxes) 

focus on offline (non-real-time) processes to support authoring and analysis goals. Together the 

five vectors described in the following subsections will enable the development of GIFT, a 

modular, service-oriented, agent-based tutoring architecture to support standards, reuse, and best 

practices for adaptive tutoring. Additional information about GIFT is available at 

https://www.gifttutoring.org/projects/gift/wiki/Overview.

Figure 3. Adaptive tutoring research vectors.

4.1 Learner Modeling

Learner modeling may be categorized into two processes: learner data acquisition and learner 

state classification. The key research challenges in learner modeling are related to understanding 

and modeling these processes. Learner models are generally composed of learner data, traits,
7

and states. Learner data generally comes from three sources: sensors, historical databases, or 

learner input. Learner data can vary from facts and demographics (e.g., birth dates) to raw 

physiological data to survey data. Traits (e.g., learning style) are defined as habitual patterns of 

behavior, thought, and emotion (Kassin, 2003). Learner states are generally derived from learner 

data and traits; are more transient than traits; and include the following categories (Sottilare et 

al., 2013):

Potential State—Also known as competence or expected success, potential is a long-term 

measure, which is derived from the learner’s previous successful experiences, training, and 

education in fields related to the current training task. Success is commonly testable and 

measures the learner’s knowledge and skill acquisition and retention. 

Performance State—Contrasted with potential state or expected success, performance state 

measures actual learner progress toward goals. Performance is derived from learner behaviors 

including responses to tests/quizzes, decisions, and actions measured by speed and accuracy 

against potential, goals and standards to determine whether the learner is above, at, or below 

expectations for a given lesson, task, or concept.

Cognitive State—A measure of learner thinking capacity, problem-solving capability, and 

focus, the determination of cognitive states uses learner behaviors to indicate increases in 

complex and abstract mental capabilities (Anderson and Krathwohl, 2001). Of significance in 

cognitive learning are attention, engagement, and working memory. A revision of Bloom’s 

taxonomy (Anderson and Krathwohl, 2001) tracks a series of behaviors from low-cognitive 

state to high as follows: remembering, understanding, applying, analyzing, evaluation, and 

creating.

Affective State—A measure of feeling with varying duration and relationship to identifiable 

sources (Gebhard, 2005), affective states include personality (long duration, multiple sources), 

mood (moderate duration, vague sources), and emotions (short duration, specific sources). 

Learner behaviors indicate affective growth and the manner in which the learner handles 

emotions during learning experiences and in particular when presented with significant 

challenges. Reported feelings, values, appreciation, enthusiasms, motivations, and attitudes 

indicate affective states including from low to high: receiving, responding, valuing, organizing, 

and characterizing (Krathwohl et al., 1964).

Motivational State—Broken out separately due to its importance in expert tutoring models 

(e.g., INSPIRE [Lepper et al., 1997]), motivation is influenced by goals, preferences, and 

interests.

Psychomotor State—Associated with physical tasks (e.g., marksmanship), which include 

physical movement, coordination, and the use of the motor-skills. Development of motor-skills 

requires practice and is measured in terms of speed, precision, distance, procedures, or 

techniques during execution (Simpson, 1972). Simpson’s hierarchy of psychomotor learning
8

ranges from low to high: perception–the ability to use sensory cues to guide motor activity; set 

or readiness to act; response–early stages of learning a complex skill through imitation and trial 

and error; mechanism–habitual learned responses; complex overt response–skillful 

performance of complex movements; adaptation–well-developed skills that are modified to 

support special requirements; and origination–the development of new movement patterns to 

fit unique situations.

Since Soldiers often operate in small units, it is a goal to provide an intelligent capability to 

support automated guided learning for teams. As part of this goal, we are researching and 

identifying team models including: team competency, performance, cognition, affect, trust, 

communication, cultural, and psychomotor state models. Other variables of interest for adaptive 

tutoring for teams are context and culture.

Context may be defined as “situational characteristics or events that influence the occurrence and 

meaning of behavior, as well as the manner and degree to which various factors impact team 

outcomes.” Context variables include (but are not limited to) distribution, virtuality, 

organizational policies or procedures that impact team functioning, environmental hazards. 

Context is considered as an influencing condition of teamwork. This means that it is a factor that

shapes the manner or degree to which teams engage in teamwork. Context is specifically 

important because certain situational characteristics (e.g., virtuality or distribution) have the 

ability to impact team outcomes or could even serve to make communication easier or more 

difficult.

Culture may be defined as “assumptions about humans’ relationships with each other and their 

environment that are shared among an identifiable group of people (e.g., team, organization, 

nation) and manifest in individuals’ values, beliefs, norms for social behavior, and artifacts.” 

Culture is considered as an influencing condition of teamwork. Culture (either at organizational 

or individual levels) has the ability to shape the way that individuals view themselves in relation 

to the team. This, in turn, can be directly related to other important variables such as conflict, 

communication, or performance.

Research objectives for learner modeling include:

• expanding learner modeling beyond traditional performance modeling to include learner 

states previously noted in order to support truly tailored tutoring experiences;

• minimizing the number, type, and obtrusiveness of physiological and behavioral sensors 

required to acquire learner data in support of learner state classification;

• discovery and innovation of machine learning techniques to accurately classify learner 

states;

• discovery and innovation of machine learning techniques to accurately classify small team 

states.
9

4.2 Automated Instruction

The goal of automating instruction is to provide a computer-based tutor that is sufficiently 

adaptive to tailor training for each individual Soldier. As part of our research to realize the goal 

of fully automated instruction in a variety of training domains, it is useful to: examine the 

behaviors of successful human tutors; understand the process of instruction; and explore methods 

to manage instruction with the goal of keeping the learner engaged. 

In examining models of expert human tutors, several themes have been identified for effective 

tutoring:

• demonstrate credible knowledge of the domain under training (e.g., tactical combat 

casualty care);

• read cues from the learner and adapt instruction in real time to meet their changing needs;

• encourage question asking;

• provide indirect feedback;

• assess learning often.

A review of the literature identified a set of promising models and best practices to be examined 

and validated through the adaptive tutoring research program. The list of models and best 

practices found in the following are illustrative and not exhaustive: 

• Models of expert human tutors: INSPIRE model (Lepper et al., 1997).

• Best tutoring practices and facts about human tutoring (Person and Graesser, 2003).

• Learner question asking behaviors and the importance of questioning (Dillon, 1988).

• Relationship between deep reasoning questions and exam scores (Graesser and Person, 

1994).

• Politeness strategies of tutors (Person et al., 1995). 

Automated instruction can be classified into two categories of strategies: macro-adaptive and 

micro-adaptive. Macro-adaptive instructional strategies are informed by the learner’s traits 

(values, preferences, interests, and goals) and the learner’s potential state. Macro-adaptive 

strategies are generally implemented prior to the tutoring session to initialize the scenario. 

Macro-adaptive strategies influence the selection of tutoring scenarios based on their level of 

complexity relative to the learner’s potential. For example, a macro-adaptive strategy for a 

learner with low prior knowledge might be to limit the learner’s control of navigation in the 

learning environment. Macro-adaptive strategies do not rely on instructional context (e.g., 

current performance) and are domain-independent. 
10

Micro-adaptive instructional strategies are a real-time adaptation of the initial or planned 

scenario and are informed by a more comprehensive model of learner states and traits. In 

particular, the learner’s performance state is critical in selecting micro-adaptive strategies. For 

example, a micro-adaptive strategy during tutoring might be to assess the level of performance 

and provide additional navigational control as the learner demonstrates higher levels of 

performance. Micro-adaptive strategies are also domain-independent. This is important since 

pedagogical engines using domain-independent strategies may be reused across multiple training 

domains/tasks.

• Error Sensitive Feedback—an intervention triggered when the learner commits errors that 

are either individually or cumulatively significantly divergent from the ideal as defined in 

the expert model of the ITS.

• Mastery Learning—a strategy where the ITS ensures the learner masters (can recall and 

apply) prerequisite lessons or concepts before allowing the learner to move on to the next 

lesson/concept.

• Adaptive Spacing and Repetition—a strategy where the learner more easily recalls 

knowledge items/objects when the knowledge is exposed to the learner repeatedly over a 

long-time span rather than repeatedly studied during a short span of time (Dempster, 1988).

• Metacognitive Prompting—a strategy where the ITS encourages the learner to self-reflect 

and evaluate, self-explain, and self-correct rather than provide the answer directly.

• Fading Worked Examples—“a step-by-step demonstration of how to perform a task or how

to solve a problem.” (Clark et al., 2006, p. 190) from which parts have been deliberately 

removed or faded (Atkinson et al., 2003).

Research objectives for automated instruction are anticipated to result in:

• The creation of models of situated pedagogy to examine optimal challenge and support 

levels for individual learners.

• The creation of constraint-based, intelligent pedagogical agents to select appropriate 

domain independent strategies, select optimal tactics (actions) based on strategy selected 

and situational context, and then present the tactic (take action) to offer feedback, provide 

direction, or change the training environment.

4.3 Domain Modeling

In order to enable the capabilities of adaptive tutors to support the broad range of tasks that 

require training across the U.S. Army, research is needed to expand the underlying capabilities of 

ITS to represent tasks of varying dynamics, definition, and complexity as noted in figure 4. This 

variability in adaptive tutoring domains will allow for greater opportunities for Soldiers to train 

as they fight. 
11

Figure 4. Domain dimensions.

Variable task dynamics refers to the physical interaction of the learner during the tutoring 

experience. This ranges from static (seated position for desktop training) to limited dynamic 

(standing position limited range of mobility in instrumented areas) to enhanced dynamic 

(standing, kneeling, and prone positions with expanded mobility in instrumented areas) to “inthe-wild” (any position with unlimited mobility where sensors and communications move with 

the Soldier). 

Variable task definition refers to how well the domains are understood in terms of standards and 

measures of performance. Well-defined domains (e.g., mathematics) typically have one correct 

path to a successful outcome and a set of specific standards for measuring success. Ill-defined 

domains may have multiple paths to successful outcomes, and they tend to have vague standards 

and less defined measures of success. 

Finally, task complexity refers to the range of difficulty in understanding and performing the 

task. Task complexity can range from simple procedural tasks to more complex 

multidimensional tasks. 

4.4 Automated Authoring

For the purposes of this report, authoring is the creation or integration of ITSs and ITS 

components. The goal of automated authoring is to reduce the time and skill required to generate 

fully adaptive tutoring systems. Authoring tools and standards are two means to meet this goal. 

Authoring tools may be used to automatically produce ITSs or ITS components. Authoring tools 

may also support integration of existing training/tutoring environments (e.g., serious games and 

training simulators) to promote reuse and eliminate the need to produce the domain element of a 

tutoring system. Authoring tools may be used to automatically generate models (e.g., expert 

models) mined from existing data repositories (e.g., Carnegie Mellon’s DataShop), the internet, 

or even field manuals. Finally, standards promoting the modularity and interoperability of ITSs, 
12

their components, tools, databases, and interfaces will go a long way toward reducing the need to 

author.

As part of this research, the author wants to reduce the need for authoring, and create tools and 

methods to make authoring easier and faster. An enhanced set of authoring goals for adaptive 

tutoring systems was derived from goals previously established by Murray (1999, 2003) and 

Sottilare and Gilbert (2011) and is presented here: 

• decrease the effort (time, cost, and/or other resources) for authoring and assessing adaptive 

tutoring systems by establishing processes, tools, and standards for reuse;

• decrease the skill threshold required to author ITSs by tailoring tools for specific 

disciplines to author, assess, and employ adaptive tutoring systems;

• provide tools to aid the designer/author/trainer /researcher in organizing their knowledge;

• support (i.e., structure, recommend, or enforce) good design principles (in pedagogy and 

user interaction);

• enable rapid prototyping of adaptive tutoring systems to allow for rapid design/evaluation 

cycles of prototype capabilities;

• employ standards to support rapid integration of external training/tutoring environments 

(e.g., serious games) to promote learner engagement and reuse of content across learning 

domains (e.g., cognitive, affective, psychomotor, and social).

4.5 Analysis

An essential element of research that involves the development of models is the validation of 

those models to empirically establish their influence on outcomes variables (e.g., learning) and 

their potential for generalization across domains. To this end, a testbed to support a structured

evaluation and comparison of ITS tools and methods is essential. The adaptive tutoring testbed, 

shown in figure 5, was adapted from the Hanks, et al. (1993) testbed and experimentation 

methodology. 
13

Figure 5. GIFT adaptive tutoring testbed.

The adaptive tutoring testbed will be used to assess effect size through: comparative studies, 

ablative studies, formative analyses, and summative analyses. Comparative studies use effect

size (specifically absolute effect size) to differentiate the relative strength of methods in 

influencing outcomes (e.g., learning) as compared to a base case. In the case of adaptive tutoring, 

the traditional base case is one-to-many classroom instruction. The effectiveness of various 

tutoring methods in well-defined, static domains is illustrated in figure 1. The expansion of 

tutoring domains to align with training tasks for military operations (discussed in section 4.3 of 

this report) necessitates additional effectiveness evaluations to support design decisions and best 

practices for military tutoring systems.
14

5. References

Anderson, L. W.; Krathwohl, D. R. (Eds.). A Taxonomy for Learning, Teaching and Assessing: A 

Revision of Bloom’s Taxonomy of Educational Objectives: Complete Edition; New York: 

Longman, 2001.

Atkinson, R. K.; Renkl, A.; Merrill, M. M. Transitioning From Studying Examples to Solving 

Problems: Effects of Self-Explanation Prompts and Fading Worked-Out Steps. Journal of 

Educational Psychology 2003, 95 (4).

Bloom, B. S. The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as 

One-to-One Tutoring. Educational Researcher 1984, 13 (6), p. 6.

Clark, R.C.; Nguyen, F.; Sweller, J. Efficiency in Learning: Evidence-Based Guidelines to 

Manage Cognitive Load; Wiley & Sons. ISBN 0-7879-7728-4, 2006.

Dempster, F.N. The spacing effect. American Psychologist 1988, 43, 627–634.

Dillon, T. J. Questioning and Teaching: A Manual of Practice; New York: Teachers College 

Press, 1988.

Gebhard, P. ALMA - A Layered Model of Affect. In Proceedings of the Fourth International 

Joint Conference on Autonomous Agents & Multi Agent Systems (AAMAS), Utrecht, 

Netherlands, July 2005.

Graesser, A. C.; Person, N. K. Question Asking During Tutoring. American Educational 

Research Journal 1994, 31, 104–137.

Hanks, S.; Pollack, M. E.; Cohen, P. R. Benchmarks, Test Beds, Controlled Experimentation, 

and the Design of Agent Architectures. AI Magazine 1993, 14 (4).

Kassin, S. Psychology; USA: Prentice-Hall, Inc, 2003.

Krathwohl, D. R.; Bloom, B. S.; Masia, B. B. Taxonomy of Educational Objectives: Handbook 

II: Affective Domain; New York: David McKay Co, 1964.

Lepper, M.; Woolverton, M. The Wisdom of Practice: Lessons Learned from the Study of Highly 

Effective Tutors. In J. Aronson (Ed.), Improving Academic Achievement: Impact of 

Psychological Factors on Education (pp. 135–158). New York: Academic Press, 2002.

Lepper, M. R.; Drake, M.; O’Donnell-Johnson, T. M. Scaffolding Techniques of Expert Human 

Tutors; In K. Hogan & M. Pressley (Eds), Scaffolding Student Learning: Instructional 

Approaches and Issues (pp. 108–144). New York: Brookline Books, 1997.
15

Murray, T. Authoring Intelligent Tutoring Systems: An Analysis of the State of the Art. 

International Journal of Artificial Intelligence in Education 1999, 10 (1), 98–129. 

Murray, T. An Overview of Intelligent Tutoring System Authoring Tools: Updated Analysis of 

the State of the Art. Authoring Tools for Advanced Technology Learning Environments 2003, 

491–545.

Person, N. K.; Kreuz, R. J.; Zwaan, R. A.; Graesser, A. C. Pragmatics and Pedagogy: 

Conversational Rules and Politeness Strategies May Inhibit Effective Tutoring. Cognition 

and Instruction 1995, 13 (2), 161–188.

Person, N. K.; Graesser, A. C. The Tutoring Research Group. Fourteen Facts About Human 

Tutoring: Food for Thought for ITS Developers. AI-ED 2003 Workshop Proceedings on 

Tutorial Dialogue Systems: With a View Toward the Classroom (pp. 335–344). Sydney, 

Australia, 2003.

Simpson, E. The Classification of Educational Objectives in the Psychomotor Domain: The

Psychomotor Domain, Vol. 3; Washington, DC: Gryphon House, 1972.

Sottilare, R.; Gilbert, S. Considerations for Tutoring, Cognitive Modeling, Authoring and 

Interaction Design in Serious Games. Authoring Simulation and Game-based Intelligent 

Tutoring workshop at the Artificial Intelligence in Education Conference (AIED) 2011, 

Auckland, New Zealand, June 2011. 

Sottilare, R. Considerations in the Development of an Ontology for a Generalized Intelligent 

Framework for Tutoring. International Defense & Homeland Security Simulation Workshop

in Proceedings of the I3M Conference. Vienna, Austria, September 2012.

Sottilare, R.; Holden, H.; Goldberg, B.; Brawner, K. The Generalized Intelligent Framework for 

Tutoring (GIFT); In C. Best, G. Galanis, J. Kerry and R. Sottilare (Eds.) Fundamental Issues 

in Defence Simulation & Training. Ashgate Publishing, 2013.

Sottilare, R.; Ragusa, C.; Hoffman, M.; Goldberg, B. Characterizing an Adaptive Tutoring 

Learning Effect Chain for Individual and Team Tutoring. In Proceedings of the

Interservice/Industry Training Systems & Education Conference, Orlando, Florida, 

December 2013.

U.S. Army Training and Doctrine Command. The United States Army Learning Concept for 

2015. Fort Monroe, VA, 2011.

VanLehn, K. The Relative Effectiveness of Human Tutoring, Intelligent Tutoring Systems, and 

Other Tutoring Systems. Educational Psychologist 2011, 46 (4), 197–221.
16

Bibliography

Anderson, J. R.; Corbett, A. T.; Koedinger, K. R.; Pelletier, R. Cognitive Tutors: Lessons 

Learned. The Journal of the Learning Sciences 1995, 4,167–207.

Brawner, K.; Holden, H.; Goldberg, B.; Sottilare, R. Recommendations for Modern Tools to 

Author Tutoring Systems. In Proceedings of the Interservice/Industry Training Simulation & 

Education Conference, Orlando, FL, December 2012.

Cohen, P. A.; Kulik, J. A.; Kulik, C.L.C. Educational Outcomes of Tutoring: A Meta-Analysis of 

Findings. American Educational Research Journal 1982, 19, 237–248.

Goldberg, B.; Brawner, K.; Sottilare, R.; Tarr, R.; Billings, D.; Malone, M. Use of Evidencebased Strategies to Expand Extensibility of Adaptive Tutoring Technologies. In Proceedings 

of the Interservice/Industry Training Simulation & Education Conference, Orlando, FL, 

December 2012.

Graesser, A. C.; D’Mello, S. Emotions During the Learning of Difficult Material. In: Brian H. 

Ross, Editor(s), Psychology of Learning and Motivation, Academic Press, Volume 57, 

Chapter 5, Pages 183-225, ISSN 0079-7421, ISBN 978012-3942937, 10.1016/B978-0-12-

394293-7.00005-4, 2012.

Graesser, A. C.; McNamara, D. S. Self-Regulated Learning in Learning Environments With 

Pedagogical Agents that Interact in Natural Language. Educational Psychologist 2010. 

Holden, H.; Sottilare, R.; Goldberg, B.; Brawner, K. Effective Learner Modeling for ComputerBased Tutoring of Cognitive and Affective Tasks. In Proceedings of the Interservice/Industry 

Training Simulation & Education Conference, Orlando, FL, December 2012.

Hunt, E.; Minstrell, J. A Cognitive Approach to the Teaching of Physics; In McGilly, K. (Ed.), 

Classroom Lessons. Cambridge, MA: MIT Press, pp.51–74, 1994.

Kokini, C.; Carroll, M.; Ramirez-Padron, R.; Wang, X.; Hale, K.; Sottilare, R.; Goldberg, B. 

(2012). Quantification of Trainee Affective and Cognitive State in Real-time. In Proceedings 

of the Interservice/Industry Training Simulation & Education Conference, Orlando, FL, 

December 2012.

Lesgold, A.M.; Lajoie, S.; Bunzo, M.; Eggan, G. Sherlock: A Coached Practice Environment for 

an Electronics Trouble Shooting Job; LRDC Report; University of Pittsburgh, Learning 

Research and Development Center: Pittsburgh, PA, 1988.
17

Rose, C. P.; Jordan, P. W.; Ringenberg, M.; Siler, S.; Vanlehn, K.; Weinstein, A. Interactive 

Conceptual Tutoring in Atlas-Andes. In J. D. Moore, C. Redfield, & W. L. Johnson (Eds.), 

Artificial intelligence in education: AI-Ed in the wired and wireless future (pp. 256–266). 

Washington, DC: IOS, 2001. 

Sottilare, R. and Goldberg, B. Designing Adaptive Computer-Based Tutors to Accelerate 

Learning and Facilitate Retention. Journal of Cognitive Technology: Contributions of 

Cognitive Technology to Accelerated Learning and Expertise 2012, 17, 1, 19–34.

Sottilare, R.; Goldberg, B.; Brawner, K.; Holden, H. A modular framework to support the 

authoring and assessment of adaptive computer-based tutoring systems (CBTS). In 

Proceedings of the Interservice/Industry Training Simulation & Education Conference, 

Orlando, FL, December 2012.

VanLehn, K.; Lynch, C.; Schulze, K.; Shapiro, J. A.; Shelby, R.; Taylor, L.; et al. The Andes 

physics tutoring system: Lessons learned. International Journal of Artificial Intelligence and 

Education 2005, 15 (3), 147–204.
18

List of Symbols, Abbreviations, and Acronyms

AAR after-action review

ALM Army Learning Model

ARL U.S. Army Research Laboratory

GIFT Generalized Intelligent Framework for Tutoring

ITE integrated training environment

ITS Intelligent Tutoring System

S&T Science and Technology 

T&E training and education

WFO Warfighter Outcome
19

NO. OF

COPIES ORGANIZATION

1 DEFENSE TECHNICAL

(PDF) INFORMATION CTR

DTIC OCA

2 DIRECTOR

(PDFS) US ARMY RSRCH LAB

RDRL CIO LL

RDRL IMAL HRA

RECORDS MGMT

1 GOVT PRINTG OFC

(PDF) A MALHOTRA

1 ARMY RSCH LABORATORY – HRED

(PDF) RDRL HRM C A DAVISON

320 MANSCEN LOOP STE 115

FORT LEONARD WOOD MO 65473

1 ARMY RSCH LABORATORY – HRED

(PDF) RDRL HRM D

T DAVIS

BLDG 5400 RM C242

REDSTONE ARSENAL AL 35898-7290

1 ARMY RSCH LABORATORY – HRED

(PDF) RDRL HRS EA DR V J RICE

BLDG 4011 RM 217

1750 GREELEY RD

FORT SAM HOUSTON TX 78234-5002

1 ARMY RSCH LABORATORY – HRED

(PDF) RDRL HRM DG J RUBINSTEIN

BLDG 333

PICATINNY ARSENAL NJ 07806-5000

1 ARMY RSCH LABORATORY – HRED

(PDF) ARMC FIELD ELEMENT

RDRL HRM CH C BURNS

THIRD AVE BLDG 1467B RM 336

FORT KNOX KY 40121

1 ARMY RSCH LABORATORY – HRED

(PDF) AWC FIELD ELEMENT

RDRL HRM DJ D DURBIN

BLDG 4506 (DCD) RM 107

FORT RUCKER AL 36362-5000 

1 ARMY RSCH LABORATORY – HRED

(PDF) RDRL HRM CK J REINHART

10125 KINGMAN RD BLDG 317

FORT BELVOIR VA 22060-5828

NO. OF

COPIES ORGANIZATION

1 ARMY RSCH LABORATORY – HRED

(PDF) RDRL HRM AY M BARNES

2520 HEALY AVE 

STE 1172 BLDG 51005

FORT HUACHUCA AZ 85613-7069

1 ARMY RSCH LABORATORY – HRED

(PDF) RDRL HRM AP D UNGVARSKY

POPE HALL BLDG 470 

BCBL 806 HARRISON DR

FORT LEAVENWORTH KS 66027-2302

1 ARMY RSCH LABORATORY – HRED

(PDF) RDRL HRM AT J CHEN

12423 RESEARCH PKWY

ORLANDO FL 32826-3276

1 ARMY RSCH LABORATORY – HRED

(PDF) RDRL HRM AT C KORTENHAUS

12350 RESEARCH PKWY

ORLANDO FL 32826-3276 

1 ARMY RSCH LABORATORY – HRED

(PDF) RDRL HRM CU B LUTAS-SPENCER

6501 E 11 MILE RD MS 284

BLDG 200A 2ND FL RM 2104

WARREN MI 48397-5000

1 ARMY RSCH LABORATORY – HRED

(PDF) FIRES CTR OF EXCELLENCE 

FIELD ELEMENT

RDRL HRM AF C HERNANDEZ

3040 NW AUSTIN RD RM 221

FORT SILL OK 73503-9043

1 ARMY RSCH LABORATORY – HRED

(PDF) RDRL HRM AV W CULBERTSON

91012 STATION AVE 

FORT HOOD TX 76544-5073

1 ARMY RSCH LABORATORY – HRED

(PDF) HUMAN RSRCH AND ENGRNG 

DIRCTRT MCOE FIELD ELEMENT

RDRL HRM DW C CARSTENS

6450 WAY ST

BLDG 2839 RM 310

FORT BENNING GA 31905-5400

1 ARMY RSCH LABORATORY – HRED

(PDF) RDRL HRM DE A MARES

1733 PLEASONTON RD BOX 3

FORT BLISS TX 79916-6816
20

NO. OF

COPIES ORGANIZATION

8 ARMY RSCH LABORATORY – HRED

(PDF) SIMULATION & TRAINING

TECHNOLOGY CENTER

RDRL HRT COL M CLARKE

RDRL HRT I MARTINEZ

RDRL HRT T R SOTTILARE

RDRL HRT B N FINKELSTEIN

RDRL HRT G A RODRIGUEZ

RDRL HRT I J HART

RDRL HRT M C METEVIER

RDRL HRT S B PETTIT

12423 RESEARCH PARKWAY

ORLANDO FL 32826

1 ARMY RSCH LABORATORY – HRED

(PDF) (PDF) HQ USASOC

RDRL HRM CN R SPENCER

BLDG E2929 DESERT STORM DRIVE

FORT BRAGG NC 28310

1 ARMY G1

(PDF) DAPE MR B KNAPP

300 ARMY PENTAGON RM 2C489

WASHINGTON DC 20310-0300

ABERDEEN PROVING GROUND

12 DIR USARL

(PDF) RDRL HR

L ALLENDER

P FRANASZCZUK

C COSENZO

RDRL HRM

P SAVAGE-KNEPSHIELD

RDRL HRM AL

C PAULILLO

RDRL HRM B

C SAMMS

RDRL HRM C

L GARRETT

RDRL HRS

J LOCKETT

RDRL HRS B

M LAFIANDRA

RDRL HRS C

K MCDOWELL

RDRL HRS D

B AMREIN

RDRL HRS E

D HEADLEY