ARL-SR-0325 ● JUNE 2015

US Army Research Laboratory

Domain Modeling for Adaptive Training and 

Education in Support of the US Army Learning 

Model—Research Outline

by Robert Sottilare, Anne Sinatra, Michael Boyce, and 

Arthur Graesser

Approved for public release; distribution is unlimited.
NOTICES

Disclaimers

The findings in this report are not to be construed as an official Department of the 

Army position unless so designated by other authorized documents.

Citation of manufacturer’s or trade names does not constitute an official 

endorsement or approval of the use thereof.

Destroy this report when it is no longer needed. Do not return it to the originator.
ARL-SR-0325 ● JUNE 2015

US Army Research Laboratory

Domain Modeling for Adaptive Training and 

Education in Support of the US Army Learning 

Model—Research Outline

by Robert Sottilare, Anne Sinatra, and Michael Boyce

Human Research and Engineering Directorate, ARL

Arthur Graesser

University of Memphis Institute for Intelligent Systems

Approved for public release; distribution is unlimited.

FOR OFFICIAL USE ONLY (delete if not FOUO)
ii

REPORT DOCUMENTATION PAGE Form Approved

OMB No. 0704-0188

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the 

data needed, and completing and reviewing the collection information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing the 

burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302. 

Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently 

valid OMB control number.

PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS.

1. REPORT DATE (DD-MM-YYYY)

June 2015

2. REPORT TYPE

Final

3. DATES COVERED (From - To)

Oct 2014–June 2015

4. TITLE AND SUBTITLE

Domain Modeling for Adaptive Training and Education in Support of the US 

Army Learning Model—Research Outline

5a. CONTRACT NUMBER

5b. GRANT NUMBER

5c. PROGRAM ELEMENT NUMBER

6. AUTHOR(S)

Robert Sottilare,1 Anne Sinatra,1 Michael Boyce,

1 and Arthur Graesser2

5d. PROJECT NUMBER

5e. TASK NUMBER

5f. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

US Army Research Laboratory

ATTN: RDRL-HRT-T

Aberdeen Proving Ground, MD 21005

8. PERFORMING ORGANIZATION REPORT NUMBER

ARL-SR-0325

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR'S ACRONYM(S)

11. SPONSOR/MONITOR'S REPORT NUMBER(S)

12. DISTRIBUTION/AVAILABILITY STATEMENT

Approved for public release; distribution is unlimited.

13. SUPPLEMENTARY NOTES

1

Human Research and Engineering Directorate, ARL 2

University of Memphis Institute for Intelligent Systems

14. ABSTRACT

While human tutoring and mentoring are common teaching tools, current US Army standards for training and education are 

group instruction and classroom training, also known as “one-to-many” instruction. Recently, the US Army has placed

significant emphasis on self-regulated learning (SRL) methods to augment institutional training where Soldiers will be largely 

responsible for managing their own learning. In support of the US Army Learning Model (ALM) and to provide affordable, 

tailored SRL training and educational capabilities for the US Army, the US Army Research Laboratory (ARL) is investigating 

and developing adaptive tools and methods to largely automate the authoring (creation), delivery of instruction, and evaluation 

of computer-regulated training and education capabilities. A major goal within this research program is to reduce the time and 

skill required to author, deliver, and evaluate adaptive technologies to make them usable by a larger segment of the training

and educational community. This research includes 6 interdependent research vectors: individual learner and unit modeling, 

instructional management principles, domain modeling, authoring tools and methods, and evaluation tools and methods. This 

report (1 of 6 interdependent research outlines) focuses on domain modeling research for adaptive training and education with 

the goal of guiding learning in militarily relevant training and educational domains.

15. SUBJECT TERMS

GIFT, design, adaptive training, domain modeling

16. SECURITY CLASSIFICATION OF:

17. LIMITATION

OF ABSTRACT

UU

18. NUMBER

OF PAGES

46

19a. NAME OF RESPONSIBLE PERSON

Robert Sottilare

a. REPORT

Unclassified

b. ABSTRACT

Unclassified

c. THIS PAGE

Unclassified

19b. TELEPHONE NUMBER (Include area code)

407-208-3007

Standard Form 298 (Rev. 8/98)

Prescribed by ANSI Std. Z39.18
iii

Contents

List of Figures v

List of Tables v

1. Introduction 1

2. Research Goals and Objectives 2

3. Background 2

3.1 Self-Regulated Learning and the US Army Learning Model 3

3.2 Motivation for Research 4

3.3 Adaptive Training and Education Definitions 5

4. US Army Requirements for Adaptive Training Systems and Domain 

Modeling 7

4.1 Adaptive Training and Education Systems and Domain Modeling 8

4.2 Big Data and Domain Modeling 8

4.3 Training at the Point-of-Need and Domain Modeling 9

4.4 Artificial Intelligence (AI) Capabilities and Domain Modeling 9

5. Understanding the Dimensions of Domain Modeling 10

5.1 Cognitive Domain 12

5.2 Affective Domain 13

5.3 Psychomotor Domain 13

5.4 Social Domain 14

6. Domain Modeling Research Goals and Challenges 17

6.1 Representing and Understanding the Influence of Domain Attributes18

6.2 Reducing Time, Cost, and Skill to Author and Deliver Instruction 18

6.3 Improving the Interoperability of Domain Models 19

6.4 Optimizing the Selection of Tactics 20

6.5 Extending Adaptive Training to Militarily Relevant Domains 21
iv

7. Interdependencies with Other Adaptive Training Research 

Vectors 23

7.1 Learner Modeling and Domain Modeling 24

7.2 Automated Instruction and Domain Modeling 25

7.3 Authoring Tools and Methods 25

7.4 Evaluation 25

7.5 Architecture 26

8. Conclusions 26

9. References 27

9. Bibliography 34

List of Symbols, Abbreviations, and Acronyms 36

Distribution List 37
v

List of Figures

Fig. 1 Adaptive training interaction ...............................................................11

Fig. 2 Updated individual learning effect model............................................20

Fig. 3 Updated team learning effect model....................................................21

Fig. 4 Adaptive training research vectors.......................................................24

List of Tables

Table 1 Matrix of collaborative problem solving skills....................................15

Table 2 Team states and behavioral measures..................................................16

Table 3 Team states and measures of cooperation............................................17

Table 4 Team states and team cognition measures...........................................17
vi

INTENTIONALLY LEFT BLANK.
1

1. Introduction

Training and education tools and methods must be of sufficient intelligence to 

understand the needs of individual learners and units of learners, to mitigate 

negative learner states, and to guide and tailor instruction in real-time to optimize 

learning. These tools and methods must also be affordable, effective, and easy to 

access and use. These requirements are enablers of the US Army Learning Model

(ALM), which includes an emphasis on self-regulated learning (SRL) where 

Soldiers are expected to manage their own learning and career development

through the growth of metacognitive (e.g., reflection), self-assessment and 

motivational skills (Butler and Winne 1995). While SRL skills are difficult to 

train and develop, support may be provided to the learner through “adaptive 

training technologies” (tools and methods), which may be focused to guide 

learning and reinforce SRL principles. 

To support ALM, the US Army Research Laboratory (ARL) has developed a 

program of research called “adaptive training”, which includes 6 interdependent 

research areas or vectors: individual learner and unit modeling, instructional 

management principles, domain modeling, authoring tools and methods, 

evaluation tools and methods, and architectural and ontological support for 

adaptive training. The reports documenting these vectors expand the scope of the 

adaptive tutoring research described in ARL-SR-0284 (Sottilare 2013) to support 

ALM requirements in the mid-term and long-term evolution of training and 

educational technology: the Synthetic Training Environment (STE) and the Future 

Holistic Training Environment for Live and Synthetic (FHTE-LS).

This report (1 of 6 interdependent research outlines) focuses on domain modeling

research for adaptive training and education. Today, the majority of intelligent 

tutoring systems (ITS’s)—a form of adaptive training tool to support one-to-one 

computer-based instruction—support well-defined domains in mathematics, 

physics, and software programming. Since Soldiers operate in more complex, 

dynamic, and ill-defined domains, it is necessary to expand the scope of adaptive 

training tools and methods to support training and education in these militarily

relevant domains. Domain modeling is a representation of knowledge for a 

particular task or concept and includes: domain content (a library of scenarios, 

problem sets, or knowledge components); an expert or ideal student model with 

measures of success and a library of common misconceptions; and a library of 

tactics or actions (e.g., questions, assessments, prompts, and pumps) that can be 

taken by the tutor to engage or motivate the learner and optimize learning.
2

2. Research Goals and Objectives

The goal of the research described in this report is to model militarily relevant

training domains to support individually tailored and intelligently guided training 

experiences as prescribed by the US Army Learning Model (US Army Training 

and Doctrine Command 2011). The research provides guidelines, best practices, 

tools, models, and methods in support of this research goal. More specifically, we 

desire to:

• Understand and model the characteristics, similarities, and differences of 

US Army training domains (cognitive, affective, psychomotor, social, and 

hybrid) with respect to their associated knowledge representations to 

support more efficient and effective authoring, instruction, and evaluation 

of adaptive training tools and methods.

• Understand and model the dimensions (definition, complexity, and 

dynamics) of training domain representations to extend the capabilities of 

traditional ITS’s; thereby, support challenging, militarily relevant training 

domains.

This report examines the background and requirements for adaptive training 

capabilities in different domains along with research challenges, dimensions of 

domain modeling, desired end states, and finally, interdependencies with other 

adaptive training research vectors.

3. Background

While human tutoring and mentoring are common teaching tools, current US 

Army standards for training and education are group instruction and classroom 

training— also known as one-to-many instruction. Group instruction and 

classroom training have been generally focused on acquiring and applying 

knowledge in proxies for live training environments (e.g., desktop simulations, 

virtual simulations, constructive simulations, and serious games). 

Classroom training, especially for complex topics, is often taught as a series of 

lists that the instructor goes through in a linear fashion (Schneider et al. 2013). 

This approach puts a heavy burden on the learner to build mental models and 

make conceptual connections. Using this instructional methodology may lead to 

varying degrees of success due to individual differences in skills, traits, and/or

preferences. More complex, ill-defined, or dynamic tasks may be difficult to 

instruct in a classroom environment especially if the cognitive elements of the 

task require spatial interaction to develop/maintain skills (e.g., marksmanship).
3

Small group instruction in live environments has also been used to assess 

application of knowledge and the development of skills. A standard feedback 

mechanism for US Army training is the after-action review (AAR) where 

significant decision points and actions are captured for small group discussion 

that is conducted after the completion of a training event to help capture teachable 

moments and to aid Soldiers in reflecting on their recent training experiences. 

Both classroom training and small group instruction are manpower intensive;

requiring teachers, mentors, and support staff to guide the Soldier’s experience. 

Today, ITSs primarily guide learner training and education for cognitive tasks in 

well-defined domains (e.g., problem solving and decision-making tasks in 

mathematics and physics). Soldiers tend to perform cognitive, affective, 

psychomotor, and social tasks in both well-defined (e.g., building clearing) and

ill-defined domains (e.g., leadership, resource allocation). ITSs generally provide 

static training (e.g., sitting at a desktop computer to train on a serious game) that 

falls short in matching the dynamic nature of many US Army operational tasks

(e.g., psychomotor tasks); and thereby, reducing opportunities to develop and 

transfer skills to the operational environment. 

Research is needed to understand the characteristics, similarities, and differences 

of US Army training domains (i.e., cognitive, affective, psychomotor, social, and 

hybrid) to develop efficient and effective adaptive training and educational tools 

and methods that support self-regulated learning in complex, ill-defined, and 

physically dynamic military domains. 

3.1 Self-Regulated Learning and the US Army Learning Model

In 2011, the US Army placed significant emphasis on the development of SRL 

skills with the expectation that new methods of instruction (e.g., ITS’s) would

augment institutional training (i.e., classroom and small group instruction). Oneto-one human tutoring has been shown to be significantly more effective than 

one-to-many instructional methods (e.g., traditional classroom instruction: Bloom, 

1984; VanLehn, 2011). However, it is not practical nor is it affordable to have 1

expert human tutor to mentor each Soldier in the US Army for every required 

operational task. This alone signals the need for capabilities to support one-toone, tailored training, and educational experiences. 

Additionally, under the ALM, Soldiers are largely responsible for managing their 

own learning, but SRL skills are difficult to train and develop (Butler and Winne

1995; Azevedo et al. 2009; Graesser and McNamara 2010). We anticipate 

adaptive training tools and methods will fill this gap and will provide personalized 

guidance to acquire, apply, retain, and transfer knowledge and skills to the 
4

operational environment. This signals the need for a computer-regulated learning

strategy to augment missing SRL skills; however, adaptive training technologies 

must first become affordable, sufficiently adaptive, and easy to use for this 

strategy to be realized. 

3.2 Motivation for Research

A promising alternative to one-to-one human tutoring is one-to-one adaptive 

training tools that include ITS’s. Meta-analyses and reviews support the claim that 

ITS technologies routinely improve learning over classroom teaching, reading 

texts, and/or other traditional learning methods. These meta-analyses normally 

report effect sizes (sigma [σ]), which refers to the difference between the ITS 

condition and a control condition in standard deviation units. The reported metaanalyses show positive effect sizes that vary from σ = 0.05 (Dynarsky et al. 2007) 

to σ = 1.08 (Dodds and Fletcher 2004), but most hover between σ = 0.40 and σ = 

0.80 (Ma et al. in press; Fletcher 2003; Graesser et al. 2012; Steenbergen-Hu and

Cooper 2013, 2014; VanLehn 2011). Our current best meta-meta estimate from 

all of these meta-analyses is σ = 0.60. This performance is comparable to human 

tutoring, which varies from between σ = 0.20 and σ = 1.00 (Cohen et al. 1982; 

Graesser et al. 2011), depending on the expertise of the tutor. Human tutors have 

not varied greatly from ITS’s in direct comparisons between ITS and trained 

human tutors (Olney et al. 2012; VanLehn 2011; VanLehn et al. 2007). 

Graesser et al. (2015, in press) are convinced that some subject matters will show 

higher effect sizes than others when comparing any intervention (e.g., computer 

trainers, human tutors, group learning) to a control. It is difficult to obtain higheffect sizes for literacy and numeracy because these skills are ubiquitous in 

everyday life and habits are automatized. For example, Ritter et al. (2007) 

reported that the Cognitive Tutor for mathematics has shown an effect size of σ = 

0.30–0.40 in environments with minimal control over instructors. Human 

interventions to improve basic reading skills typically report an effect size of σ = 

0.20. In contrast, when the student starts essentially from ground zero, such as 

many subject matters in science and technology, then effect sizes are expected to 

be more robust. ITS’s show effect sizes of σ = 0.60–2.00 in the subject matters of 

physics (VanLehn 2011; VanLehn et al. 2005), computer literacy (Graesser et al. 

2004; Graesser et al. 2012), biology (Olney et al. 2012), and scientific reasoning 

(Millis et al. 2011; Halpern et al. 2012). As a notable example, the Digital Tutor 

(Fletcher and Morrison 2012) improves information technology by an effect size 

as high as σ = 3.70 for knowledge and σ = 1.10 for skills. The effect size 

attributed to improved instruction and improved domain knowledge have not been 
5

separated in this analysis. Such large effect sizes would never be expected in basic 

literacy and numeracy.

Overall, these are promising results and equate to an increase of about a letter 

grade improvement over traditional classroom instruction. While ITS’s are a 

promising technology to support adaptive training for individuals in well-defined 

domains like mathematics, physics, and computer programming, the US Army 

requires the ability to develop and exercise Soldier skills in more ill-defined 

domains (e.g., leadership) and at the unit level (e.g., collaborative learning and 

team training). Developing and maintaining the ability to make effective decisions 

under stress and in complex environments is also desirable. 

Adaptive systems by their nature require additional content and complexity to 

support tailored learning for each user and as a consequence have a very high 

development cost, a major barrier to adoption by the US Army. Adaptive systems

are also insufficiently adaptive to support tailored, self-regulated training and 

educational experiences across a broad spectrum of military tasks as required by 

the ALM. Today, few ITS authoring tools are generalized across all of the 

domains requiring training and no evaluation criteria or standards have been 

developed to promote reuse and interoperability among ITS’s (Sottilare et al. 

2012b). In other words, current adaptive systems are not yet intelligent enough to 

support the tailored instruction required by the US Army in the breadth of 

domains being trained; but there is a stable foundation of 50 years of science on 

which to grow an adaptive training and education capability for the US Army.

3.3 Adaptive Training and Education Definitions

In support of the ALM and affordable adaptive training and educational 

capabilities for the US Army, ARL is investigating and developing adaptive tools 

and methods. A desired end-state is the automation of authoring (creation)

processes, instruction, and evaluation of computer-regulated training and 

education capabilities to help build SRL skills and support mixed-initiative 

interaction. A major goal within this research program is to reduce the time/cost 

and knowledge/skill required to author, deliver, and evaluate adaptive 

technologies to make them usable by a larger segment of the US Army training 

and educational community.

Adaptive training and education research includes elements of adaptive tutoring, 

distributed learning, virtual humans, and training effectiveness evaluation. For 

additional detail on research specific to ITS’s, refer to ARL-SR-0284 (Sottilare

2013). Definitions are provided for this section to distinguish between adaptive 

training and education elements and also to highlight their relationships:
6

Adaptive Tutoring – also known as intelligent tutoring; tailored instructional 

methods to provide one-to-one and one-to-many computer-guided experiences 

focused on optimizing learning, comprehension, performance, retention, 

reasoning, and transfer of knowledge and acquired skills to the operational 

environment.

Adaptive Tutoring Systems – also known as ITS’s; the mechanism or 

technologies (tools and methods) to provide tailored training and educational 

experiences; adaptive tutoring systems respond to changing states in the learner 

and changing conditions in the training environment to optimize learning; 

adaptive tutoring systems anticipate and recognize teachable moments.

Virtual Humans – artificially-intelligent visual representations of people that 

simulate or emulate cognitive, affective, physical, and social processes.

Distributed Learning – concurrent distribution of training and educational 

content to multiple users at the point-of-need in which content is intelligently 

selected to support learning, increased performance, and long-term competency in 

selected domains.

Training/Learning Effectiveness – evaluation of the impact of training and 

educational tools and methods on usability, learning, comprehension, 

performance, retention, reasoning, and transfer of knowledge and acquired skills 

to the operational environment.

Adaptive Training and Education Systems – a convergence of ITSs and 

external training and education capabilities (e.g., serious games, virtual humans, 

simulations) to support engaging experiences with reduced need for authoring 

(Sottilare 2015).

Generalized Intelligent Framework for Tutoring (GIFT) (Sottilare et al. 2012a; 

Sottilare et al. 2013a) – an open-source, modular architecture whose goals are to 

reduce the cost and skill required for authoring adaptive training and educational 

systems, to automate instructional delivery and management, and to develop and 

standardize tools for the evaluation of adaptive training and educational

technologies.

Adaptive training and education research at ARL is being conducted across 6

interdependent research vectors: individual learner and unit modeling;

instructional management principles; domain modeling, authoring tools and 

methods; evaluation tools and methods; and architectural and ontological support.

This report (1 of 6 interdependent research outlines) focuses on domain modeling 

research for adaptive training systems with the goal of guiding learning in 

militarily relevant training and educational domains. 
7

Soldiers operate in a variety of complex, dynamic, ill-defined domains where 

their ability to persevere in the face of adversity, adapt to their situation, 

collaborate, and think critically are key to the successful completion of their 

assigned missions. In order to develop and exercise these skills, it is paramount 

for Soldiers to train in challenging environments. Presently these few challenging 

training environments have been largely provided through manpower-intensive 

methods or systems with little ability to adapt instruction to support their learning 

needs. To illustrate this point Franke (2011) asserts that through the use of case 

study examples, instruction can provide the pedagogical foundation for decisionmaking under uncertainty. However, this approach is limited in implementation 

by the expanse of potential cases that would need to be consistently updated and 

maintained to support large populations like the US Army. 

As noted previously, adaptive systems like ITS’s have been shown to be effective 

in promoting learning in primarily static (e.g., learners seated at desktop 

computers) instructional settings within relatively simple, well-defined domains 

(e.g., mathematics, physics) for individual learners. For our purposes, static 

instruction includes cognitive, affective, or social training tasks where a desktop 

computer delivers instruction and where the physical movement of the learner is 

limited to activities that can be conducted while seated. For example, static

instruction can effectively support cognitive tasks involving decision-making and 

problem-solving, but are less effective for training tasks involving motion and 

perception (e.g., land navigation and marksmanship). Ideally, we desire portable 

adaptive instructional capabilities to go with Soldiers to support training and 

education at their point-of-need across a wide spectrum of US Army operational 

tasks. Research is needed to develop tools and methods to support broader domain 

modeling, which is representative of the full spectrum of US Army operational 

tasks. Standards, interoperability, and automation (e.g., automated scenario 

generation) (Zook et al. 2012) will likely play a significant role in making 

adaptive training practical. In this way adaptive training technologies will have 

the greatest impact on organizational learning in the US Army.

4. US Army Requirements for Adaptive Training Systems and

Domain Modeling

The Army Science and Technology community uses Warfighter Outcomes 

(WFOs) as the authoritative source for identifying Warfighter needs. WFOs are 

used to share research and future technology solutions. In the training and 

education (T&E) domain, the adaptive T&E research program is targeting 4

specific requirements to support the evolution of US Army training: adaptive 
8

training and education systems; big data; training at the point-of-need; and

artificial intelligence. 

4.1 Adaptive Training and Education Systems and Domain 

Modeling

The primary gap to be addressed under this US Army requirement is the lack of 

adaptive systems (e.g., intelligent tutors) to support individual and collective

(team or unit) training. The US Army needs an adaptive training and education 

capability that is persistent and easy to use/access with minimal startup time. 

There are also requirements to automate an informal AAR (also known as a 

postexercise critique) to reduce the time and skills needed to produce the AAR 

and improve its focus and quality. Another line of thought notes that the artificial 

intelligence in ITS’s could be used to facilitate rapid mission planning and courseof-action analyses as a job aid in operational contexts. 

The major connection between the adaptive training and education requirement 

and the domain modeling research vector is the need to extend adaptive system 

architectures to support instruction in relevant military domains. Affordable 

solutions to support adaptive training in more complex, ill-defined, and dynamic 

domains will more closely align training and operations, thereby resulting in more 

efficient transfer of knowledge and skills. 

4.2 Big Data and Domain Modeling

The primary gap to be addressed under this US Army requirement is that there is 

a lack of capability to handle and process large amounts of structured and 

unstructured data (also referred to as big data). One capability needed is a 

structured data analytics program linking individual data (e.g., achievements) to 

required long-term competencies in military occupational specialties (MOS’s). 

This would allow Soldiers to understand where they rank in terms of experiences 

and achievements among other Soldiers in their MOS. It would also allow the US 

Army to identify specific experiences among successful Soldiers in that MOS and 

provide a model for other Soldiers in that MOS to follow. The data could also be 

used by course managers and instructors to continuously improve instruction and 

the mental models of both human and computer-based instructors. Finally, data 

collected on trainee learning and performance during adaptive training 

experiences could be used to facilitate Unit Training Management (UTM) where 

unit commanders would have access to empirical data to support unit training 

decisions.
9

The major connection between the US Army’s big data requirement and domain 

modeling is ability to collect learner data, learner states, and training environment 

data to make the connection between domain-specific instructional tactics and 

their varying degrees of effectiveness given instructional context (existing 

conditions). This will allow course managers to identify best practices over time 

and to promote agile configuration management of instructional content, and 

effective strategies, tactics, and techniques. 

4.3 Training at the Point-of-Need and Domain Modeling

The primary gap to be addressed under this US Army requirement is the lack of

an easily accessible, persistent, cost-effective, and low-overhead training 

environment. A capability is needed to bring training to Soldiers instead of 

Soldiers going to fixed training locations. This point-of-need training capability 

would be easily distributed, web-based, and built upon open-enterprise 

architecture in the cloud. US Army training and educational opportunities would 

be available on demand anywhere and anytime. However, it should be noted that 

the delivery mechanism (e.g., laptop computer, mobile device, and smart glasses) 

for adaptive training is critical in determining the limitations of the domain model 

scope and complexity. For example, it may be extremely difficult to train all the 

complexities of a psychomotor task in a desktop computer setting.

The major connection between point-of-need training and domain modeling is the 

practicality of extending adaptive training beyond the desktop. Low-cost 

commercial tools (e.g., smart glasses) must be investigated to determine their 

suitability to support the same kinds of tutor-user interaction afforded in static 

desktop applications. The cloud architecture to support adaptive training and

education will be required to operate with and without Internet connectivity 

depending on the location of the learner and their access to the network. For 

example, if a Soldier decides to take a 2-hour (h) training course while traveling 

and knows that Internet connectivity will be intermittent, he might decide to 

download the course to his device and take it offline. The architecture must be 

able to track the Soldier’s progress and upload results when connectivity is again

available. 

4.4 Artificial Intelligence (AI) Capabilities and Domain Modeling

The primary gap to be addressed under this US Army requirement is that the US 

Army lacks an automated capability to replicate the complexity and uncertainty of 

the operational environment. This gap specifically points to the lack of 

adaptiveness in virtual humans, intelligent tutoring systems, and other training 
10

capabilities. This gap leads to Soldiers developing training-response strategies 

that result in less challenging training over time along with lower engagement and 

lower levels of learning and transfer of skills to more challenging operational 

environments. 

The major connection between AI capabilities and domain modeling involve the 

discovery and innovation of techniques to support a concept called, “automated 

scenario evolution”. AI capabilities are needed to support automated scenario 

evolution where AI drives the generation of new “child” scenarios from a singleparent scenario based on dimensions of that scenario and the state of the trainee. 

In this way, the authoring burden for highly complex training and educational 

domains may be reduced.

For example, consider a single scenario where dimensions include variable 

challenge levels based on 3 threats (i.e., low, moderate, high), 3 types of field-ofview (i.e., narrow, moderate, and wide) and clear line-of-sight (i.e., near, 

moderate, and far). AI could spawn 27 new child scenarios based on combinations

of these variables. This requirement is closely linked to adaptive training 

capabilities described in Section 4.1 of this report and the realization of this 

capability will enable the development of affordable self-authoring adaptive 

systems. Through this capability, complex domains may be modeled for adaptive 

training systems without the need for long development cycles or special

authoring skill sets.

AI-based capabilities in adaptive training and education systems may also support 

data acquisition (sensing), natural language, problem-solving strategies, and 

perceptual/interaction mechanisms in the tutor.

5. Understanding the Dimensions of Domain Modeling

There are 4 typical elements that compose ITSs, a prime example of an adaptive 

training and education system: a learner or trainee model, an instructional or 

pedagogical model, a domain model, and some type of user interface. The domain 

model typically includes an expert or ideal student model by which the adaptive 

system measures/compares/contrasts the progress of the learner toward learning 

objectives. The domain model also includes the training environment, the training 

task, and all of the associated instructional actions (e.g., feedback, questions, 

hints, pumps, and prompts), which could possibly be delivered by the adaptive 

system for that particular training domain. Typical interaction between the 

learner, the training environment, and the adaptive system (tutoring agents) is 

shown in Fig. 1.
11

Fig. 1 Adaptive training interaction

Typical training systems examine the interaction between the learner and the 

training environment to measure progress toward learning objectives. The learner 

acts on the environment (e.g., opens a door or makes a choice to move into the 

room or stay outside) and then observes any changes or reactions within the 

environment. Adaptive systems add a layer of software-based tutoring agents that

are designed to guide the learner in much the same way as a human tutor interacts 

with a learner. The tutoring agents observe the behaviors of the learner to assess 

their states (e.g., performance and attitudes) and interact with the learner to 

provide support, direction, and instruction. In addition, they track the effect of 

interactions on learning. Tutoring agents also interact with the training 

environment and may manipulate the environment to present more challenging or 

less challenging scenarios in response to the assessed state of the learner.

One method to describe or model the domain is by the type of task that is being 

trained. A traditional methodology of categorizing tasks is Bloom’s (Bloom and 

Krathwohl 1956) Taxonomy, which describes hierarchically ordered skills 

(competency) in the cognitive task domain. Due to the contributions of others, 

Bloom’s original taxonomy has evolved over time to include an affective 

(Anderson and Krathwohl 2001), psychomotor (Simpson 1972), and a social 

hierarchy (Soller 2001; Sottilare et al. 2011). The following sections describe 

each task domain and its relationship to adaptive training and education.
12

5.1 Cognitive Domain

Sometimes called the “thinking” domain, tasks in this domain stress the learner’s 

thinking capacity (i.e., working memory, executive control, workload 

management, multitasking), problem-solving and planning, decision-making, 

comprehension, reasoning, and attentional focus or engagement. The 

determination of cognitive skills may be based on learner behaviors to indicate 

increases in complex and abstract mental capabilities (Anderson and Krathwohl 

2001). 

A revision of Bloom’s Taxonomy (Anderson and Krathwohl 2001) tracks a series 

of behaviors from low to high cognitive skills: 1) remembering – the learner’s 

ability to recognize and recall information, 2) understanding (also known as 

comprehension) – the learner’s ability to organize, compare, and interpret

information, 3) applying – the learner’s ability to use information to solve 

problems, 4) analyzing – the learner’s ability to examine information and make 

inferences from that information, 5) evaluating – the learner’s ability to use 

information to make optimal judgments, and 6) creating – learner’s ability to 

build new models (e.g., plans) from information. 

Most of the ITS’s in existence today focus on the cognitive-task domain

(Anderson et al. 1995; Ritter et al. 2007; Graesser et al. 2012). Examples include 

model-tracing (also called example tracing) tutors that use a set of steps to walk 

the learner through the process of solving a problem (Koedinger et al. 2012). 

Mathematics, physics, and software programming are the most common types of 

model-tracing tutors. These domains constitute simple procedural tasks and are 

usually rule-based.

Matthews (2014) notes organizations generally do a good job of training 

relatively simple skills. However, a more challenging goal is to teach higher order 

cognitive skills such as decision-making and judgment. The US Army has large 

investments in partial-task and scenario-based training systems that use relatively 

fixed strategies to guide the learner based primarily on individual and team 

performance measures. A concern with these systems is that Soldiers learn how to 

win within the constraints of the system but the effect on retention and transfer is 

not well understood. Research is needed to build adaptivity into these training 

systems and thereby optimize deep learning. A goal of this research is to reduce 

the time to competency to allow time for automaticity through overtraining and 

deeper learning experiences that transfer to the operational environment.
13

5.2 Affective Domain

Sometimes called the “feeling” domain, tasks in this domain are intended to 

develop emotional intelligence or skills in self-awareness and growth in attitudes, 

emotion, and feelings. The goal is to manage emotions in positive ways to relieve 

stress, communicate effectively, empathize with others, overcome challenges, and 

defuse conflict (Goleman 2006). While listed as separate domain, affect has an 

interdependent relationship with cognition and learning. Specifically, confusion, 

frustration, boredom, surprise, delight, flow, and anxiety are considered major 

moderators of learning (D’Mello 2013; Pekrun 2006). For example, cognitive 

readiness—the capability to maintain performance and mental well-being in

complex, dynamic, unpredictable environments that may elicit affective 

responses. Dimensions of cognitive readiness, according to Kluge and Burkolter 

(2013), include concepts such as risk-taking behavior, emotional stability and 

coping, which may be considered part of the affective domain.

A revision of Bloom’s Taxonomy (Anderson and Krathwohl 2001) tracks a series 

of behaviors from low-affective state to high: 1) receiving – the learner takes in 

information, 2) responding – the learner takes in information and responds/reacts, 

3) valuing – the learner attaches value to information, 4) organizing – the learner 

sorts information and builds mental models, and 5) characterizing – the learner 

matches mental models to values and beliefs ultimately influencing (e.g., 

promoting or limiting) the learner’s behavior. 

Very little training (outside of classroom-based training) is currently provided to 

exercise/grow skills in this important task domain and almost no adaptive training 

has been created to support this domain. However, D’Mello and Graesser (2012) 

have produced an affect-sensitive tutor that exercises emotional intelligence using 

the AutoTutor authoring tools. Research is needed to understand measures for the 

affective task domain and any unique characteristics required to author affective 

domain scenarios.

5.3 Psychomotor Domain

Sometimes called the “doing” or “action” domain, tasks in this domain are 

associated with physical tasks (e.g., marksmanship) or manipulation of a tangible 

interface (e.g., remotely piloting a vehicle), which may include physical 

movement, coordination, and the use of the motor-skills. Development of motorskills requires practice and is measured in terms of speed, precision, distance, 

procedures, or techniques during execution (Simpson 1972). Simpson’s hierarchy 

of psychomotor learning ranges from low to high: 1) perception – the ability to 

use sensory cues to guide motor activity; 2) set or readiness to act; 3) response –
14

early stages of learning a complex skill through imitation and trial and error; 4)

mechanism –habitual learned responses; 5) complex overt response – skillful 

performance of complex movements; 6) adaptation – well-developed skills that 

are modified to support special requirements; and 7) origination – the 

development of new movement patterns to fit unique situations.

While this domain is well represented in US Army training, research is needed to 

build adaptiveness into these training systems and thereby optimize deep learning. 

Again, a goal of this research is to reduce the time to competency to allow time 

for automaticity through overtraining and deeper learning experiences that 

transfer to the operational environment.

5.4 Social Domain

Sometimes called the “collaborative” domain, tasks in this domain include a set of 

collaborative characteristics or measures of learning in the social domain as 

defined by Soller (2001): 1) participation, 2) social grounding – team members 

“take turns questioning, clarifying and rewording their peers’ comments to ensure 

their own understanding of the team’s interpretation of the problem and the 

proposed solutions”, 3) active learning conversation skills – quality 

communication, 4) performance analysis and group processing – groups 

discuss their progress, and decide what behaviors to continue or change (Johnson

et al. 1990) and 5) promotive interaction – also known as “win-win” this 

characteristic occurs when members of a group perceive that they can only attain 

their goals if their team members also attain their goals. 

This domain is different from the cognitive, affective, and psychomotor domain in 

that Soller’s collaborative learning skills taxonomy is multilayered: skills, 

subskills, and attributes/behaviors as shown below:

• Conversation Skill

 Task subskill: Coordinate group process, request focus change, 

summarize information, and end participation behavior

 Maintenance subskill: Request attention, suggest action, request 

confirmation, listening behavior, and apologize behaviors 

 Acknowledge subskill: Appreciation, accept/confirm, and reject 

behaviors

• Active Learning Skill

 Request subskill: Information-seeking, elaboration, clarification, 

justification, opinion-seeking, and illustration behaviors
15

 Inform subskill: Rephrase, lead, suggest, elaborate, explain/clarify, 

justify, and assert behaviors

 Motivate subskill: Encourage and reinforce behaviors

• Creative Conflict Skill

 Argue subskill: Conciliate, agree, disagree, offer alternatives, infer, 

suppose, propose exception, and doubt behaviors

 Mediate subskill: Teacher mediation 

The Program for International Student Assessment (PISA) (2015) is a worldwide 

study by the Organization for Economic Cooperation and Development (OECD) 

in both member and nonmember nations. This study focuses on 15-year-old 

students and their scholastic performance in mathematics, science, and reading.

During PISA 2015, OECD defined a matrix of collaborative problem-solving 

skills (Table 1). Some of the skills and associated behaviors in this matrix may 

also apply to situational problem solving in the US Army (e.g., staff-level 

organizations evaluating options to meet objectives during military operations). 

Research is needed to determine if the model in this matrix will generalize beyond 

its original application to 15-year-old students. 

Table 1 Matrix of collaborative problem solving skills

Research is needed to create and apply measures for both collaborative learning

and team training activities. The interdependent nature of US Army tasks also 

requires tutoring of squads and other echelons of teams (collective training).
16

Research is also needed to develop team state models to drive adaptive training 

decisions. An extensive review of the team performance and tutoring literature 

has been conducted (Burke et al. 2015, in press) to determine antecedents of team 

outcomes (i.e., performance, learning, satisfaction, and viability), which include 

behavioral measures (Table 2), along with models of cooperation (Table 3), and 

team cognition (Table 4). 

Table 2 Team states and behavioral measures

Team 

Performance

Team

Learning

Team 

Satisfaction

Team

Viability

Team Behaviors

Communication YES YES YES YES

Coordination YES YES YES YES

Mutual Support YES NOT 

EXAMINED

YES TENTATIVE

Reflexivity YES YES YES NOT 

EXAMINED

Monitoring YES NOT 

EXAMINED

NOT 

EXAMINED

NOT 

EXAMINED

Conflict YES TENTATIVE YES YES

Task YES NOT 

EXAMINED

YES YES

Relationship YES NOT 

EXAMINED

YES YES

Transaction YES TENTATIVE TENTATIVE NOT 

EXAMINED

Action YES TENTATIVE TENTATIVE NOT 

EXAMINED

Interpersonal YES TENTATIVE NOT 

EXAMINED

NOT 

EXAMINED

Leadership YES NOT 

EXAMINED

YES NOT 

EXAMINED

Organizational 

Citizenship 

Behaviors

YES NOT 

EXAMINED

NOT 

EXAMINED

NOT 

EXAMINED

Conflict 

Management

YES TENTATIVE YES TENTATIVE
17

Table 3 Team states and measures of cooperation

Team 

Performance

Team

Learning

Team 

Satisfaction

Team

Viability

Cooperation

Trust YES YES YES NOT 

EXAMINED

Collective 

Efficacy

YES NOT 

EXAMINED

YES YES

Psychological 

Safety

YES TENTATIVE TENTATIVE TENTATIVE

Cohesion YES TENTATIVE YES TENTATIVE

Justice TENTATIVE NOT 

EXAMINED

NOT EXAMINED NOT 

EXAMINED

Table 4 Team states and team cognition measures

Team 

Performance

Team

Learning

Team 

Satisfaction

Team

Viability

Team 

Cognition

Team Mental 

Models

YES NOT 

EXAMINED

YES NOT 

EXAMINED

Transactive 

Memory 

Systems

YES NOT 

EXAMINED

TENTATIVE YES

Situation 

Awareness

YES NOT 

EXAMINED

NOT EXAMINED NOT 

EXAMINED

Blocks in each table indicate either: “YES” to represent a significant effect was 

found between 1 of the antecedents examined in the literature review and 1 of the 

team outcomes; “TENTATIVE” to represent some effect or conflicting evidence 

was found between 1 of the antecedents examined and 1 of the team outcomes; or 

“NOT EXAMINED” to represent that these relationships were not examined as 

part of this literature review and meta-analysis. The “NOT EXAMINED” 

category represents a set of opportunities for future research in team tutoring.

Additional discussion of research in the social domain may be found on team 

cognition (Salas and Fiore 2004), team mental models (Fletcher and Sottilare

2013; Rouse et al. 1992), and situational awareness in team performance (Salas et 

al. 1995).

6. Domain Modeling Research Goals and Challenges

A foundational goal of adaptive training research at ARL is to model the 

perception, judgment, and behaviors of expert human tutors to support practical, 

effective, and affordable learning experiences guided by computer-based agents. 

To this end, 5 primary goals for domain modeling for adaptive training systems 
18

have been identified and are discussed in this section along with the major 

challenges or barriers to success.

6.1 Representing and Understanding the Influence of Domain 

Attributes

Our first goal is to conduct research to determine the influence of task domain 

attributes (e.g., complexity, definition, and dynamics) on cognitive mechanisms 

(e.g., learning, comprehension, performance, retention, reasoning, and transfer of 

knowledge and acquired skills to the operational environment). Research is 

needed to analyze related and unrelated domains to determine commonalities, 

performance measures, and user requirements.

The challenge in meeting this goal is that since each domain contains unique 

domain knowledge, the relationship of these attributes to the cognitive 

mechanisms listed previously are not well understood across various task domains

(cognitive, affective, psychomotor, social, and hybrid tasks). It may be possible to 

generalize relationships in a particular task domain with the goal being to identify 

attributes with high effect on cognitive mechanisms.

Also important to this research goal is the ability to define how these attributes are 

measured, how qualitative inputs are going to be assessed against quantitative 

metrics, and how stakeholder requirements and learner-generated content (e.g., 

social media input on relevance and impact) influences the modeling of each

domain. There needs to be a matching between the desired requirements and what 

is actually authored in the domain. For example, tutors for marksmanship for the 

US Army and Marines may be very similar based on the task domain 

(psychomotor) but may also have different instructional methods or concepts 

based on organizational requirements.

6.2 Reducing Time, Cost, and Skill to Author and Deliver 

Instruction

Our second goal is to discover and improve authoring tools and methods to more 

easily represent domain knowledge including methods to select optimal 

instructional tactics (actions) based on sound instructional strategies (plans), the 

instructional context, and the learner’s states and traits (Murray 1999; Murray

2003; Sottilare and Gilbert 2011; Murray 2015). Specifically, our goal is to 

provide sufficient domain knowledge to effectively adapt the training of the task

for individual learners and units while significantly lowering the cost and skills 

needed to author adaptive training systems for the US Army.
19

Authoring in complex domains is a time intensive process and research is needed 

to determine what attributes of the domain model influence successful outcomes 

(e.g., skill development) and antecedents to those outcomes (e.g., motivation, 

engagement, and grit), which may influence the complexity, definition, and 

dynamics of the domain knowledge (e.g., content, feedback, and assessments) 

presented to the learner and thereby influence the cost.

Adaptive training systems by their nature offer more flexibility and are tailored to 

individual learners. Given the variability of learner attributes across the general 

population, this creates a greater demand for domain authoring. Finding efficient 

methods to create new content and to reuse existing content (e.g., training content 

in existing US Army training simulations) should be a priority for domain 

modeling research. Specifically, we need to examine methods to automate large 

portions of the authoring process including the automated development of expert 

models (sometimes called ideal student models), the automated generation of 

scenario variants from base cases, and the authoring of assessments from which 

the ITS determines progress and corresponding strategies and tactics.

Additionally, our goal is to discover/improve methods to provide adaptive 

instruction that is easily accessible and is available at the Soldier’s point-of-need 

(anyplace) and anytime (24/7/365).

6.3 Improving the Interoperability of Domain Models

Our third goal is to develop standards for interoperability to promote reuse of 

domain knowledge (e.g., content, expert models, question banks, assessments, and 

tactics). Specifically, our goal is to set interoperability standards for US Army 

training resulting in reduced development and maintenance costs and increased 

speed of adoption for adaptive training technologies. The major challenges are

that no standards exist for adaptive training and interoperability—the ability to 

pull and replace one model or domain element with another authored elsewhere—

is extremely low. Effort should be expended to work with the ITS community to 

develop interoperability standards to maximize reuse of domain knowledge. 

Defining standard methods of representing and interfacing with domain 

knowledge will allow a single adaptive training architecture or framework to 

support multiple domains and reduce authoring costs. This is the primary driver 

for the development of GIFT.

The US Army’s focus on scenario-based training along with the significant 

investment in training infrastructure should also be considered in developing 

interoperability. To this end, GIFT includes a standard interface specification, or 

gateway, so that US Army training systems that meet this specification can be 
20

integrated to provide adaptive training capabilities. In addition, the GIFT gateway 

currently has an Institute of Electrical and Electronics Engineers (IEEE) 1278 

Distributed Interactive Simulation (DIS) compliant interface to receive tactical 

data (e.g., entity location) to support instructional decisions and to push 

instructional tactics (e.g., interaction with the learner or changes to the training 

environment) to DIS-compatible training simulations. 

6.4 Optimizing the Selection of Tactics

Our fourth goal is to optimize the selection of tactics—domain-specific actions by 

the tutor—to provide the greatest opportunity for performance, learning, retention, 

and transfer. In GIFT, tactics are the actions taken by the tutor in response to 

learner states and instructional context (e.g., conditions of the scenario or problem 

presented), as shown in Fig. 2 and are constrained by available options provided 

during the authoring process. Improving the usability and efficiency of authoring 

tools will likely result in a greater number of available options for adaptive 

training domains.

Fig. 2 Updated individual learning effect model

Unlike instructional strategies, which are derived from good pedagogical practices 

based on learning theory and influenced by the learner’s states, tactics are 

domain-specific actions by the tutor and may not be generalized across all task 

domains. Research is needed to determine methods to select the best possible 

tactic given the selected instructional strategy, the training domain, and the 

availability of tactics.
21

Modeling the expert behaviors of human tutors may be a starting point, but 

accurate assessment methods are needed for both individual and team-level states. 

These states are critical in selecting appropriate strategies (plans for action) and 

tactics (actions: e.g., assessments, feedback, questions, changes to the training 

environment) per the Learning Effect Model (Sottilare 2012; Fletcher and

Sottilare 2013; Sottilare 2013; Sottilare et al. 2013b) as updated for both 

individuals (Fig. 2) and teams (Fig. 3). Assessment of team states may also be 

useful in determining constraints to be monitored by tutoring agents and 

interactions with the learner and the training environment as shown earlier in 

Fig. 1.

Fig. 3 Updated team learning effect model

6.5 Extending Adaptive Training to Militarily Relevant Domains

Our fifth goal is to be able to extend adaptive training to support militarily

relevant domains. Many military tasks are hybrids of task domains in that they 

include aspects of cognitive (thinking: evaluating, problem-solving, and decisionmaking), affective (feeling: making value judgments), psychomotor (doing:

physical action), and/or social (collaborating: working in teams). US Army 

training differs greatly from traditional ITS’s, which are primarily problem-based 

(e.g., mathematics, physics, computer programming) and generally vary only in 

complexity. Given much of US Army training is scenario-based, the realism of 

the training environment, accessibility of the training, the complexity of the 

scenario, the physical dynamics of the task, and the variable level of definition

are all design considerations for adaptive training systems for military use. It will 

be essential to match the attributes of the environment to the task domain by 

asking the question, “What is necessary to train the task effectively?” This 

variability in adaptive training and educational domains will allow for greater 
22

opportunities for Soldiers to train at the point-of-need and to train more closely to 

how they fight. This is anticipated to result in greater learning, performance, 

retention, and transfer of skills to the operational environment.

As in all training, it will be essential to match the realism of the training 

environment (e.g., serious game, virtual simulation, embedded training) to enable 

learners to progress toward established learning objectives. For embedded training 

where Soldiers bring training with them in their operational platform (e.g., 

ground, air, sea, dismounted), considerations should be made regarding the visual 

resolution of virtual elements of the training environment and what is required to 

train the task (Sottilare et al. 2007). For example, if the resolution of virtual 

targets is insufficient to support either detection or identification of the targets at 

comparable distances to the real world (also known as live environment), then 

negative training may result.

Another consideration in militarily relevant domains is accessibility. Accessible 

learning is being directly addressed by the research and development of GIFT

(Sottilare et al. 2012a, 2012b; Sottilare et al. 2013a), an adaptive tutoring 

architecture that is modular and service-oriented. GIFT will also support access to 

adaptive tutoring resources (e.g., domain content, assessments, Web services) via 

the Internet and allow content to be presented in a Web browser. 

Adaptive training solutions must be able to include the complexities of each to 

provide tailored training across the broad spectrum of Soldier tasks. This includes 

the ability to align more closely with the nature of those tasks to promote transfer 

of skills from training systems to the theaters of operation. Ultimately, this will 

mean moving from desktop training environments to more interactive and 

physical environments. Research will be needed to examine a learning 

progression from the desktop to the wild, a concept where Soldiers can receive 

training anywhere. We examine 4 modes of adaptive training environments to 

support this concept.

Variable task dynamics refer to the physical modes of interaction of the learner 

during the training experience. This ranges from static (seated position for 

desktop training), to limited dynamic (standing position limited range of mobility 

in instrumented areas), to enhanced dynamic (standing, kneeling, and prone 

positions with expanded mobility in instrumented areas), to “in-the-wild” (any 

position with unlimited mobility where sensors and communications move with 

the Soldier). 

Variable task definition refers to how well the domains are understood in terms of 

standards and measures-of-performance. Well-defined domains (e.g., 

mathematics) typically have 1 correct path to a successful outcome and a set of 
23

specific standards for measuring success. Ill-defined domains may have multiple 

paths to successful outcomes, and they tend to have vague standards and less 

defined measures of success. Ill-defined domains may also have unexpected and 

inconsistent confounds that could cause learning to be perceived when there really 

is a mediating underlying factor. Analyzing these can provide greater knowledge 

to answering the “why” behind performance and learning outcomes.

Finally, task complexity refers to the range of difficulty in understanding and 

performing the task. Task complexity can range from simple procedural tasks to 

more complex multidimensional tasks. 

Next, we examine modes of dynamic interaction. Limited dynamic environments 

support hybrid (i.e., cognitive, affective, psychomotor) tasks where a larger 

degree of interaction with the environment and other learners is critical to 

learning, retention, and transfer to the operational environment. Decision-making 

and problem-solving tasks may be taught easily in a limited dynamic mode along 

with tasks requiring physical orientation (e.g., land navigation).

Enhanced dynamic environments support tasks where freedom of movement and a 

high degree of interaction with other learners are critical to learning, retention, and 

transfer to the operational environment. Building clearing and other team-based 

tasks may be taught easily in an enhanced dynamic mode.

“In-the-wild” mode is transferring tutoring to the operational environments and 

could also be called embedded training for Soldiers. In-the-wild mode is critical 

to support tasks where a very high degree freedom of movement and a high 

degree of interaction with other learners are critical to learning, retention, and 

transfer to the operational environment. It is anticipated that psychomotor and 

social tasks may be best taught in-the-wild or an environment more closely 

resembling the operational environment.

7. Interdependencies with Other Adaptive Training Research 

Vectors

This section examines interdependencies between domain modeling and the other 

5 adaptive training research vectors (Fig. 4). This discussion forms the basis for 

the sequencing of research and ultimately bringing adaptive training capabilities

into a state of practice. 
24

Fig. 4 Adaptive training research vectors

Accurate methods to classify individual and team learner states are a necessary 

precursor to selecting optimal instructional strategies as noted in the learning 

effect models for individual learners (Fig. 2) and teams of learners (Fig. 3). In 

turn, instructional strategies along with instructional context are necessary 

precursors to selecting optimal instructional tactics and ultimately significant 

effect on desired outcomes: learning, performance, retention, and transfer.

7.1 Learner Modeling and Domain Modeling

Adaptive training systems are learner-centric systems. Independent of the domain 

under training, accurate modeling of the learner is critical to driving instructional 

decisions in adaptive training systems. However, collection and maintenance of 

this data may be costly; therefore, it is necessary to select measures and states that 

significantly impact our desired outcomes: learning, performance, retention, and 

transfer. Research is needed to determine what this dataset will look like. 

Candidates abound in the literature, but in general these include transient 

data/states, cumulative states (building over time), and enduring data/states 

(Paneva 2006). It should be noted that learning falls into both transient and 

cumulative states. It is necessary to understand progress toward domain 

competency in addition to measures of near-term performance. It is important to 

understand how domain-specific learning (skills) decay over time. 

Transient measures of importance include individual behavioral and physiological

data, and cognitive, affective, physical, and social states to represent learning.
25

Cumulative measures include achievements (e.g., certifications, training, 

education, and experiences), affiliations, work history, and domain competency. 

More enduring information about the learner might include: gender, culture, first 

language, physical constraints (e.g., colorblind/deaf), values, personality 

attributes, or other trait-based information (Sottilare and Brawner 2014). All of 

these measures/states are potential drivers for adaptive training decisions. 

7.2 Automated Instruction and Domain Modeling

In GIFT, instructional management takes place in 2 modules/processes within the 

learning-effect model. One process is instructional strategy selection within the 

pedagogical module. The second is within the domain module where specific 

tactics or actions are selected based on the strategy selection and instructional 

context. Standard representation of tactics and other domain knowledge 

components are needed to support interoperability, modularity, and reuse of 

instructional elements to reduce the cost of authoring and maintaining adaptive 

training systems.

7.3 Authoring Tools and Methods

Authoring tools and methods are needed to search, retrieve, curate, and apply 

domain knowledge to adaptive training systems. Automation of these processes is

needed to reduce the overhead of gathering and organizing domain knowledge 

and authoring adaptive training systems. Two developing automation capabilities 

include tools to rapidly develop expert models through data mining of text-based 

material (e.g., field manuals), and automated generation of child scenarios from 

parent scenarios. Development of new user interfaces for authors is needed to 

make authoring of domain knowledge easier for authors who are domain experts, 

but lack computer science and instructional system design skills. Finally, adaptive 

training systems require capabilities to interpret user-generated content (e.g., 

social media) to support domain-specific changes (e.g., improved accuracy tactics 

selection algorithms, alternate domain content) to ensure the learner population is 

engaged in the training process.

7.4 Evaluation

Research is needed to evaluate and optimize the effect of tactics selected within 

the GIFT domain module upon desired outcomes: learning, performance, 

retention, and transfer. To understand the impact of domain-specific instructional 

decisions, research is needed to provide stealth assessment of learners (Shute et al.
26

2013). Long-term effects (deep learning, retention, and transfer) are also 

important in evaluating the impact of adaptive training methods.

7.5 Architecture

GIFT, as the architecture to facilitate adaptive training, will need to support the 

acquisition and interpretation of data required by the domain module to support 

optimal instructional tactics selection per the learning effect models 

(Figs. 3 and 4).

8. Conclusions

This report outlines the ARL’s plans for conducting research in adaptive training 

and education to support the US Army Learning Model. Specifically, this report 

relates to domain modeling and the answer to the question: What adaptive training 

methods provide the best value (in terms of effectiveness and affordability) for the 

comprehensive modeling of US Army Training and Education Domains?

This report outlined goals to:

• model and understand the efforts required to author domains of varying 

complexity, definition, and physical dynamics;

• support authoring of ill-defined and well-defined task domains;

• support authoring of militarily relevant training and education domains 

across the spectrum of cognitive, affective, psychomotor, and social tasks;

• match the nature of military tasks to training environments and optimize 

transfer to operational environments;

• accurately assess learning and domain task performance in real-time;

• promote optimal learning, performance, retention and transfer (on-the-job 

performance) across domains;

• support individual and team training (e.g., small unit and collective 

training) and education (e.g., collaborative learning and problem-solving)

experiences.
27

9. References

Anderson JR, Corbett AT, Koedinger KR, Pelletier R. Cognitive tutors: Lessons 

learned. The Journal of the Learning Sciences. 1995;4:167–207.

Anderson LW, Krathwohl DR (Eds). A taxonomy for learning, teaching and 

assessing: A revision of Bloom’s Taxonomy of educational objectives: 

Complete Edition. New York: Longman, 2001.

Azevedo R, Witherspoon A, Graesser AC, McNamara DS, Chauncey A, Siler E, 

Cai Z, Lintean M. MetaTutor: Analyzing self-regulated learning in a tutoring 

system for biology. In V Dimitrova, R Mizoguchi, B Du Boulay, AC 

Graesser (Eds). Artificial intelligence in education: Building learning systems 

that care: From knowledge representation to affective modelling (p. 635–

637). Amsterdam: IOS Press, 2009.

Bloom BS. The 2 sigma problem: The search for methods of group instruction as 

effective as one-to-one tutoring. Educational Researcher. 1984;13(6):6.

Bloom BS, Krathwohl DR. Taxonomy of educational objectives: The 

classification of educational goals. Handbook I: Cognitive Domain, 1956.

Burke S, Sottilare R, Salas E, Johnston J, Sinatra A, Holden, H. Towards a 

scientifically-rooted design architecture of team process and performance 

modeling in adaptive, team-based intelligent tutoring systems: Methodology 

for literature review and meta-analysis, and preliminary results. Aberdeen 

Proving Ground (MD): US Army Research Laboratory (US); (2015, in 

press).

Butler DL, Winne PH. Feedback and self-regulated learning: A theoretical 

synthesis. Review of Educational Research. 1995;65:245–281.

Cohen PA, Kulik JA, Kulik CLC. Educational outcomes of tutoring: A metaanalysis of findings. American Educational Research Journal. 1982;19:

237–248.

D’Mello SK. A selective meta-analysis on the relative incidence of discrete 

affective states during learning with technology. Journal of Educational 

Psychology. 2013;105:1082–1099.

D’Mello SK, Graesser AC. AutoTutor and affective AutoTutor: Learning by 

talking with cognitively and emotionally intelligent computers that talk back. 

ACM Transactions on Interactive Intelligent Systems. 2012;2(4):23:1–38.
28

Dodds PVW, Fletcher JD. Opportunities for new “smart” learning environments 

enabled by next generation web capabilities. Journal of Education 

Multimedia and Hypermedia. 2004;13, 391–404.

Dynarsky M, Agodina R, Heaviside S, Novak T, Carey N, Campuzano L, 

Sussex W. Effectiveness of reading and mathematics software products: 

Findings from the first student cohort. Washington, DC: US Department of 

Education, Institute of Education Sciences, Mar 2007.

Fletcher JD. Evidence for learning from technology-assisted instruction. In HF

O’Neil, RS Perez (Eds), Technology applications in education: A learning 

view. p. 79–99 Mahwah, NJ: Erlbaum, 2003.

Fletcher JD, Morrison JE. DARPA Digital Tutor: Assessment data (IDA 

Document D-4686). Alexandria, Virginia: Institute for Defense Analyses, 

2012.

Fletcher JD, Sottilare R. Shared Mental Models and Intelligent Tutoring for 

Teams. In R Sottilare, A Graesser, X Hu, and H Holden (Eds.) Design 

Recommendations for Intelligent Tutoring Systems: Volume I – Learner 

Modeling. Orlando (FL), Army Research Laboratory (US), 2013. ISBN 978-

0-9893923-0-3.

Franke D. Decision-making under uncertainty: using case studies for teaching 

strategy in complex environments. Journal of Military and Strategic Studies.

2011;13(2). 

Goleman D. Emotional intelligence. Bantam, 2006.

Graesser AC, Lu S, Jackson GT, Mitchell H, Ventura M, Olney A, 

Louwerse MM. AutoTutor: A tutor with dialogue in natural language. 

Behavioral Research Methods, Instruments, and Computers, 2004;36;

180–193.

Graesser AC, McNamara DS. Self-regulated learning in learning environments 

with pedagogical agents that interact in natural language. Educational 

Psychologist. 2010;45:234–244. 

Graesser AC, D’Mello SK, Cade W. Instruction based on tutoring. In RE Mayer, 

PA Alexander (Eds), Handbook of Research on Learning and Instruction. 

p 408–426. New York: Routledge Press, 2011.
29

Graesser AC, Conley MW, Olney AM. Intelligent tutoring systems. In S Graham, 

K Harris (Eds), APA Educational Psychology Handbook: Vol. 3. 

Applications to Learning and Teaching (p. 451–473). Washington, DC: 

American Psychological Association, 2012.

Graesser AC, Hu X, Nye B, Sottilare R. Intelligent tutoring systems, serious 

games, and the Generalized Intelligent Framework for Tutoring (GIFT). In 

HF O’Neil, EL Baker, and RS Perez. (Eds.), Using games and simulation for 

teaching and assessment. Routledge: Abingdon, Oxon, UK, 2015; in press.

Halpern DF, Millis K, Graesser AC, Butler H, Forsyth C, Cai Z. Operation ARA: 

A computerized learning game that teaches critical thinking and scientific 

reasoning. Thinking Skills and Creativity. 2012;7:93–100.

Johnson D, Johnson R, Holubec EJ. Circles of learning: Cooperation in the 

classroom (3rd ed.). Edina, MN: Interaction Book Company, 1990.

Kluge A, Burkolter D. Enhancing research on training for cognitive readiness 

research issues and experimental designs. Journal of Cognitive Engineering 

and Decision Making. 2013;7(1):96–118.

Koedinger KR, Corbett AC, Perfetti C. The Knowledge-Learning-Instruction 

(KLI) framework: Bridging the science-practice chasm to enhance robust 

student learning. Cognitive Science. 2012;36(5):757–798. 

Ma W, Adesope OO, Nesbit JC. Intelligent tutoring systems and learning 

outcomes: A meta-analytic survey. Journal of Educational Psychology. In 

press.

Matthews MD. Head strong: How psychology is revolutionizing war (p. 204). 

Oxford University Press, New York, 2014. 

Millis K, Forsyth C, Butler H, Wallace P, Graesser A, Halpern D. Operation 

ARIES! A serious game for teaching scientific inquiry. In M Ma, A 

Oikonomou, J Lakhmi (Eds), Serious games and edutainment applications. 

2011. London, UK: Springer-Verlag; p. 169–196.

Murray T. Authoring intelligent tutoring systems: An analysis of the state of the 

art. International Journal of Artificial Intelligence in Education.

1999;10(1):98–129. 

Murray T. An overview of intelligent tutoring system authoring tools: Updated 

analysis of the state of the art. Authoring Tools for Advanced Technology 

Learning Environments. 2003; 491–545.
30

Murray T. Theory-based Authoring Tool Design: Considering the Complexity of 

Tasks and Mental Models (Chap. 2) In R Sottilare, A Graesser, X Hu, and K

Brawner (Eds). Design Recommendations for Intelligent Tutoring Systems: 

Volume 3 – Authoring Tools and Methods. Orlando (FL): Army Research 

Laboratory (US); 2015. ISBN: 978-0-9893923-7-2.

Olney A, D’Mello SK, Person N, Cade W, Hays P, Williams C, Graesser AC. 

Guru: A computer tutor that models expert human tutors. In S Cerri, W 

Clancey, G Papadourakis, K Panourgia (Eds), Proceedings of Intelligent 

Tutoring Systems (ITS), (p. 256–261). Berlin, Germany: Springer, 2012

Paneva D. Use of ontology-based student model in semantic-oriented access to 

the knowledge in digital libraries. In proc. of HUBUSKA Fourth Open 

Workshop “Semantic Web and Knowledge Technologies Applications”, 

Varna, Bulgaria (p. 31–41), Sept 2006.

Pekrun R. The control-value theory of achievement emotions: Assumptions, 

corollaries, and implications for educational research and practice. 

Educational Psychology Review. 2006; 18:315–341.

Program for International Student Assessment (PISA), 2015. Available at

http://www.cmec.ca/508/Programs-and-Initiatives/Assessment/Programmefor-International-Student-Assessment-(PISA)/PISA-2015/index.html.

Ritter S, Anderson JR, Koedinger KR, Corbett A. Cognitive tutor: Applied 

research in mathematics education. Psychonomic Bulletin and Review.

2007;14:249–255

Rouse WB, Cannon-Bowers JA, Salas E. The role of mental models in team 

performance in complex systems. Systems, Man and Cybernetics, IEEE 

Transactions on. 1992;22(6):1296–1308.

Salas EE, Fiore SM. Team cognition: Understanding the factors that drive process 

and performance. American Psychological Association, 2004.

Salas E, Prince C, Baker DP, Shrestha L. Situation awareness in team 

performance: Implications for measurement and training. Human Factors: 

The Journal of the Human Factors and Ergonomics Society. 1995;37(1):

123–136.

Schneider B, Wallace J, Blikstein P, Pea R. Preparing for future learning with a 

tangible user interface: The case of neuroscience. Learning Technologies, 

IEEE Transactions on. 2013;6(2):117–129.
31

Shute V, Ventura M, Small M, Goldberg B. Modeling student competencies in 

video games using stealth assessment. In R Sottilare, A Graesser, X Hu, and 

H Holden (Eds.). Design Recommendations for Intelligent Tutoring Systems: 

Volume I – Learner Modeling. Orlando (FL), Army Research Laboratory

(US), 2013. ISBN 978-0-9893923-0-3.

Simpson E. The classification of educational objectives in the psychomotor 

domain: The psychomotor domain, Vol. 3; Washington, DC: Gryphon 

House, 1972.

Soller A. Supporting social interaction in an intelligent collaborative learning 

system. International Journal of Artificial Intelligence in Education.

2001;12(1):40–62.

Sottilare R, Marshall L, Martin R, Morgan J. Injecting realistic human models 

into the optical display of a future land warrior system for embedded training 

purposes. Journal for Defense Modeling and Simulation. Apr 2007;4(2).

Sottilare R, Gilbert S. Considerations for tutoring, cognitive modeling, authoring 

and interaction design in serious games. Authoring Simulation and Gamebased Intelligent Tutoring workshop at the Artificial Intelligence in 

Education Conference (AIED) 2011, Auckland, New Zealand, June 2011. 

Sottilare R, Holden H, Brawner K, Goldberg, B. Challenges and emerging 

concepts in the development of adaptive, computer-based tutoring systems 

for team training. In Proceedings of the Interservice/Industry Training 

Simulation and Education Conference, Orlando, FL, Dec 2011. 

Sottilare R. Considerations in the development of an ontology for a generalized 

intelligent framework for tutoring. International Defense and Homeland 

Security Simulation Workshop in Proceedings of the I3M Conference. 

Vienna, Austria, Sept 2012.

Sottilare RA, Brawner KW, Goldberg BS, Holden HK. The generalized intelligent 

framework for tutoring (GIFT). Orlando (FL), Army Research Laboratory 

(US), Human Research and Engineering Directorate (HRED), 2012a.

Sottilare R, Goldberg BS, Brawner KW, Holden HK. A modular framework to 

support the authoring and assessment of adaptive computer-based tutoring 

systems (CBTS). In Proceedings of the Interservice/Industry Training 

Simulation & Education Conference, Orlando, FL, Dec 2012b.
32

Sottilare R. (2013). Special report: Adaptive intelligent tutoring system (ITS) 

research in support of the army learning model – research outline. Aberdeen 

Proving Ground (MD): US Army Research Laboratory (US); Dec 2013. 

Report No.: ARL-SR-0284.

Sottilare R, Holden H, Goldberg B, Brawner K. The generalized intelligent 

framework for tutoring (GIFT); In C Best, G Galanis, J Kerry, and R

Sottilare (Eds) Fundamental Issues in Defence Simulation and Training. 

Ashgate Publishing, 2013a.

Sottilare R, Ragusa C, Hoffman M, Goldberg B. Characterizing an adaptive 

tutoring learning effect chain for individual and team tutoring. In Proceedings 

of the Interservice/Industry Training Systems and Education Conference, 

Orlando, FL, Dec 2013b.

Sottilare R, Brawner K. A long-term learner model to drive optimal macroadaptive decisions by intelligent tutoring systems. Florida Artificial 

Intelligence Research Society, Pensacola, FL, May 2014.

Sottilare R. Challenges in moving adaptive training and education from state-ofart to state-of-practice. In proceedings of “Developing a Generalized 

Intelligent Framework for Tutoring (GIFT): Informing Design through a 

Community of Practice” Workshop at the 17th International Conference on 

Artificial Intelligence in Education (AIED 2015). Madrid, Spain, June 2015.

Steenbergen-Hu S, Cooper H. A meta-analysis of the effectiveness of intelligent 

tutoring systems on K-12 students’ mathematical learning. Journal of 

Educational Psychology, 105. 2013;971–987.

Steenbergen-Hu S, Cooper H. A meta-analysis of the effectiveness of intelligent 

tutoring systems on college students’ academic learning. Journal of 

Educational Psychology, 106. 2014;331–347.

US Army Training and Doctrine Command. The United States Army Learning 

Concept for 2015. Fort Monroe, VA, 2011.

VanLehn K, Lynch C, Schulze K, Shapiro JA, Shelby R, Taylor L. The Andes 

physics tutoring system: Lessons learned. International Journal of Artificial 

Intelligence and Education. 2005;15(3):147–204.

VanLehn K, Graesser AC, Jackson GT, Jordan P, Olney A, Rose CP. When are 

tutorial dialogues more effective than reading? Cognitive Science. 2007;31, 

3–62.
33

VanLehn K. The relative effectiveness of human tutoring, intelligent tutoring 

systems, and other tutoring systems. Educational Psychologist.

2011;46(4):197–221.

Zook A, Lee-Urban S, Riedl M, Holden H, Sottilare R, Brawner K. Automated 

scenario generation: Toward tailored and optimized military training in 

virtual environments. FDG ’12, May 29–June 1, 2012, Raleigh, NC.
34

9. Bibliography

Atkinson RK, Renkl A, Merrill MM. Transitioning from studying examples to 

solving problems: Effects of self-explanation prompts and fading worked-out 

steps. Journal of Educational Psychology. 2003;95(4).

Brawner K, Holden H, Goldberg B, Sottilare R. Recommendations for modern 

tools to author tutoring systems. In Proceedings of the Interservice/Industry 

Training Simulation & Education Conference, Orlando, FL, Dec 2012.

Clark RC, Nguyen F, Sweller J. Efficiency in learning: Evidence-based guidelines 

to manage cognitive load. Wiley & Sons. ISBN 0-7879-7728-4, 2006.

Dempster FN. The spacing effect. American Psychologist. 1988;43:627–634.

Dillon TJ. Questioning and teaching: A manual of practice. New York: Teachers 

College Press, 1988.

Goldberg B, Brawner K, Sottilare R, Tarr R, Billings D, Malone M. Use of 

evidence-based strategies to expand extensibility of adaptive tutoring 

technologies. In Proceedings of the Interservice/Industry Training Simulation 

and Education Conference, Orlando, FL, Dec 2012.

Graesser AC, McNamara DS. Self-Regulated learning in learning environments 

with pedagogical agents that interact in natural language. Educational 

Psychologist. 2010. 

Graesser AC, D’Mello S. Emotions during the learning of difficult material. In: 

Brian H Ross, Editor(s), Psychology of Learning and Motivation, Academic 

Press, Volume 57, Chapter 5, p. 183–225, ISSN 0079-7421, ISBN 978012-

3942937, 10.1016/B978-0-12-394293-7.00005-4, 2012.

Holden H, Sottilare R, Goldberg B, Brawner K. Effective learner modeling for 

computer-based tutoring of cognitive and affective tasks. In Proceedings of 

the Interservice/Industry Training Simulation and Education Conference, 

Orlando, FL, Dec 2012.

Hunt E, Minstrell J. A cognitive approach to the teaching of physics. In McGilly 

K (Ed), Classroom Lessons. Cambridge, MA: MIT Press, p.51–74, 1994.

Kokini C, Carroll M, Ramirez-Padron R, Wang X, Hale K, Sottilare R, Goldberg

B. Quantification of trainee affective and cognitive state in real-time. In 

Proceedings of the Interservice/Industry Training Simulation and Education 

Conference, Orlando, FL, Dec 2012.
35

Krathwohl DR, Bloom BS, Masia BB. Taxonomy of Educational objectives: 

Handbook ii: Affective domain. New York: David McKay Co, 1964.

Lesgold AM, Lajoie S, Bunzo M, Eggan G. Sherlock: A coached practice 

environment for an electronics trouble shooting job; LRDC Report; 

University of Pittsburgh, Learning Research and Development Center: 

Pittsburgh, PA, 1988.

Sottilare R, Goldberg B. Designing adaptive computer-based tutors to accelerate 

learning and facilitate retention. Journal of Cognitive Technology: 

Contributions of Cognitive Technology to Accelerated Learning and 

Expertise. 2012;17(1):19–34.
36

List of Symbols, Abbreviations, and Acronyms

σ sigma

AAR after-action review

AI artificial intelligence

ALM US Army Learning Model

ARL US Army Research Laboratory

DIS Distributed Interactive Simulation

FHTE-LS Future Holistic Training Environment for Live and Synthetic

GIFT Generalized Intelligent Framework for Tutoring

h hour(s)

IEEE Institute of Electrical and Electronics Engineers

ITS intelligent tutoring system

MOS military occupational specialty

OECD Organization for Economic Cooperation and Development

PISA Program for International Student Assessment

SRL self-regulated learning

S&T Science and Technology

STE Synthetic Training Environment

T&E training and education

UTM Unit Training Management

WFO Warfighter Outcome
37

1 DEFENSE TECHNICAL

(PDF) INFORMATION CTR

DTIC OCA

2 DIRECTOR

(PDF) US ARMY RESEARCH LAB

RDRL CIO LL

IMAL HRA MAIL & RECORDS

MGMT

1 GOVT PRINTG OFC

(PDF) A MALHOTRA

1 DIR USARL

 (PDF) RDRL HRT T

R SOTTILARE
38

INTENTIONALLY LEFT BLANK.