ARL-SR-0339 ● OCT 2015

US Army Research Laboratory

Authoring Tools and Methods for Adaptive 

Training and Education in Support of the US 

Army Learning Model: Research Outline

by Scott Ososky, Robert Sottilare, Keith Brawner, 

Rodney Long, and Arthur Graesser

Approved for public release; distribution is unlimited.
NOTICES

Disclaimers

The findings in this report are not to be construed as an official Department of the 

Army position unless so designated by other authorized documents.

Citation of manufacturer’s or trade names does not constitute an official 

endorsement or approval of the use thereof.

Destroy this report when it is no longer needed. Do not return it to the originator.
ARL-SR-0339 ● OCT 2015

US Army Research Laboratory

Authoring Tools and Methods for Adaptive 

Training and Education in Support of the US 

Army Learning Model: Research Outline

by Scott Ososky

Oak Ridge Associated Universities, Oak Ridge, TN

Robert Sottilare, Keith Brawner, and Rodney Long

Human Research and Engineering Directorate, ARL

Arthur Graesser

University of Memphis Institute for Intelligent Systems, Memphis, TN

Approved for public release; distribution is unlimited.

FOR OFFICIAL USE ONLY (delete if not FOUO)
ii

REPORT DOCUMENTATION PAGE Form Approved

OMB No. 0704-0188

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the 

data needed, and completing and reviewing the collection information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing the 

burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302. 

Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently 

valid OMB control number.

PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS.

1. REPORT DATE (DD-MM-YYYY)

October 2015

2. REPORT TYPE

Special Report

3. DATES COVERED (From - To)

October 2014–September 2015

4. TITLE AND SUBTITLE

Authoring Tools and Methods for Adaptive Training and Education in Support 

of the US Army Learning Model: Research Outline

5a. CONTRACT NUMBER

5b. GRANT NUMBER

5c. PROGRAM ELEMENT NUMBER

6. AUTHOR(S)

Scott Ososky, Robert Sottilare, Keith Brawner, Rodney Long, and 

Arthur Graesser

5d. PROJECT NUMBER

5e. TASK NUMBER

5f. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

US Army Research Laboratory

ATTN: RDRL-HRT-T

Aberdeen Proving Ground, MD 21005-5425

8. PERFORMING ORGANIZATION REPORT NUMBER

ARL-SR-0339

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR'S ACRONYM(S)

11. SPONSOR/MONITOR'S REPORT NUMBER(S)

12. DISTRIBUTION/AVAILABILITY STATEMENT

Approved for public release; distribution is unlimited.

13. SUPPLEMENTARY NOTES

14. ABSTRACT

While human tutoring and mentoring are common teaching tools, current US Army standards for training and education are 

group instruction and classroom training, also known as one-to-many instruction. Recently, the US Army has placed 

significant emphasis on self-regulated learning (SRL) methods to augment institutional training where Soldiers will be largely 

responsible for managing their own learning. In support of the US Army Learning Model and to provide affordable, tailored 

SRL training and educational capabilities for the US Army, the US Army Research Laboratory is investigating and developing 

adaptive tools and methods to largely automate the authoring (creation), delivery of instruction, and evaluation of computerregulated training and education capabilities. A major goal within this research program is to reduce the time and skill 

required to author, deliver, and evaluate adaptive technologies to make them usable by a larger segment of the training and 

educational community. This research includes 6 interdependent research vectors: individual learner and unit modeling, 

instructional management principles, domain modeling, authoring tools and methods, architecture, and evaluation tools and 

methods. This report (1 of 6 interdependent research outlines) focuses on research to reduce the time and skills required to 

author adaptive instruction with the goal of guiding learning in militarily relevant training and educational domains.

15. SUBJECT TERMS

GIFT, design, adaptive training, authoring tools, expert models

16. SECURITY CLASSIFICATION OF:

17. LIMITATION

OF ABSTRACT

UU

18. NUMBER

OF PAGES

46

19a. NAME OF RESPONSIBLE PERSON

Scott Ososky

a. REPORT

Unclassified

b. ABSTRACT

Unclassified

c. THIS PAGE

Unclassified

19b. TELEPHONE NUMBER (Include area code)

407-384-3924

Standard Form 298 (Rev. 8/98)

Prescribed by ANSI Std. Z39.18
iii

Contents

List of Figures v

Preface vi

1. Introduction 1

2. Research Goals and Objectives 2

3. Background 2

3.1 Self-Regulated Learning and the US Army Learning Model 3

3.2 Motivation for Research 4

3.3 Adaptive Training and Education Definitions 5

3.4 State of the Art in Authoring Tools and Methods 8

3.5 Overview of GIFT Authoring Tools 9

4. US Army Requirements for Adaptive Training Systems and 

Authoring 10

4.1 Adaptive Training and Education Systems and Authoring 11

4.2 Big Data and Authoring 11

4.3 Training at the Point of Need and Authoring 12

4.4 Artificial Intelligence (AI) Capabilities and Authoring 12

5. Understanding the Dimensions of Authoring 13

6. Authoring Research Goals and Challenges 16

6.1 Describe Human Mental Models and Define Authoring Interaction 

Paradigms 16

6.2 Identify Candidates for Automated Authoring and Develop 

Processes to Reduce the Human Authoring Workload 18

6.3 Extend Authoring Capabilities to Support Integration with Existing 

Training and Educational Technologies 19

6.4 Enable Collaborative Authoring 20

6.4.1 Web Authoring 20
iv

6.4.2 Versioning and Compatibility 21

6.4.3 Sharing and Exporting Tutors 21

7. Interdependencies with Other Adaptive Training Research 

Vectors 22

7.1 Learner Modeling and Authoring 22

7.2 Automated Instruction and Authoring 23

7.3 Domain Modeling and Authoring 23

7.4 Evaluation and Authoring 24

7.5 Architecture and Authoring 24

8. Conclusions 25

9. References 26

Bibliography 31

List of Symbols, Abbreviations, and Acronyms 35

Distribution List 36
v

List of Figures

Fig. 1 Adaptive training interaction ...............................................................14

Fig. 2 Adaptive training research vectors.......................................................22
vi

Preface

This report is 1 of 6 interdependent research outlines in the Adaptive Training 

research program. Portions of this text, which originated in ARL-SR-0325,1

appear in all 6 reports to ensure that readers get the same cross-cutting 

information.

 1 Sottilare R, Sinatra A, Boyce M, Graesser A. Domain modeling for adaptive training and education in 

support of the US army learning model—research outline. Aberdeen Proving Ground (MD): Army Research 

Laboratory (US); June 2015. Report No.: ARL-SR-0325.
1

1. Introduction

Training and education tools and methods must be of sufficient intelligence to 

understand the needs of individual learners and units of learners, to mitigate

negative learner states, and to guide and tailor instruction in real time to optimize 

learning. These tools and methods must also be affordable, effective, and easy to 

access and use. These requirements are enablers of the US Army Learning Model 

(ALM), which includes an emphasis on self-regulated learning (SRL) where 

Soldiers are expected to manage their own learning and career development 

through the growth of metacognitive (e.g., reflection), self-assessment, and 

motivational skills (Butler and Winne 1995). While SRL skills are difficult to 

train and develop, support may be provided to the learner through “adaptive 

training technologies” (tools and methods), which may be focused to guide 

learning and reinforce SRL principles. 

To support ALM, the US Army Research Laboratory (ARL) has developed a 

program of research called “adaptive training”, which includes 6 interdependent 

research areas or vectors: individual learner and unit modeling, instructional 

management principles, domain modeling, authoring tools and methods, 

evaluation tools and methods, and architectural and ontological support for 

adaptive training. The reports documenting these vectors expand the scope of the 

adaptive tutoring research described in ARL-SR-0284 (Sottilare 2013) to support 

ALM requirements in the mid- and long-term evolution of training and 

educational technology: the Synthetic Training Environment and the Future 

Holistic Training Environment for Live and Synthetic.

This report (1 of 6 interdependent research outlines) focuses on research to reduce 

the time and skills required to author adaptive training and education content in a 

variety of militarily relevant domains. Today, most intelligent tutoring systems 

(ITSs), a form of adaptive training tool to support one-to-one computer-based 

instruction, support well-defined domains in mathematics and physics. Since 

Soldiers operate in more complex, dynamic, and ill-defined domains, it is 

necessary to expand the scope of adaptive training tools and methods to support 

training and education in these militarily relevant domains, including team 

training. In turn, the process by which intelligent tutors are currently authored for 

complex domains is time intensive, requiring a degree of skill similar to those in 

computer programming. The authoring tools and methods research vector 

examines the various tools, processes, and technologies that enable adaptive tutor 

creation, including 1) software interfaces for sequencing content, defining 

instructional techniques, strategies and tactics, and integrating disparate systems, 
2

2) the creation and management of instructional content (i.e., content management 

systems and reusable learning objects), and 3) algorithms that provide for semi- or 

fully automated content creation with significantly reduced human input. 

2. Research Goals and Objectives

The goal of the research described in this report is to reduce the time and skill 

required to author adaptive instruction. The authoring tools and methods research

vector is concerned with the creation and management of various resources

required to support adaptive instruction within an ITS or a series of ITSs. This 

report outlines research objectives intended to provide guidelines, methods, tools, 

and/or technologies in service of the stated research goals. The primary objectives 

of this research are intended to provide guidelines, best practices, tools, models, 

and methods in support of this research goal:

• For human authoring: Apply usability heuristics to develop best practices 

to guide authoring of adaptive systems based on the author’s role (e.g., 

domain expert, instructional designer, or course manager).

• For computer-aided authoring: Identify elements of the authoring process 

that are candidates for automated or semi-automated authoring processes 

to remove the human from the authoring process wherever possible.

• For leveraging existing training and educational capabilities: Establish 

standards for the integration of functionally disparate tools and 

technologies (e.g., external training systems and serious games) that may 

be relevant to adaptive systems to reduce the need for authoring.

This report examines the background and requirements for authoring adaptive 

training and educational capabilities in different domains along with research 

challenges, dimensions of authoring, desired end states, and interdependencies 

with other adaptive training research vectors.

3. Background

While human tutoring and mentoring are common teaching tools, current US 

Army standards for training and education are group instruction and classroom 

training—also known as one-to-many instruction. Group instruction and 

classroom training have been generally focused on acquiring and applying 

knowledge in proxies for live training environments (e.g., desktop simulations, 

virtual simulations, constructive simulations, and serious games). 
3

Classroom training, especially for complex topics, is often taught as a series of 

lists that the instructor goes through in a linear fashion (Schneider et al. 2013). 

This approach puts a heavy burden on the learner to build mental models and 

make conceptual connections. Using this instructional methodology may lead to 

varying degrees of success due to individual differences in skills, traits, and/or 

preferences. Alternatively, adaptive training can support those individual 

differences in learners; however, the burden to adapt and tailor instruction shifts 

to the author (who may also be a course manager, course developer, domain 

expert, or instructor) to build mental models of adaptive training components, the

relationships between the components, and the processes (e.g., workflow) by 

which robust tutors are created. For nonautomated authoring tasks, the creation 

and use of these mental models are among the primary research challenges.

Small group instruction in live environments has also been used to assess 

application of knowledge and the development of skills. A standard feedback 

mechanism for US Army training is the after-action review (AAR) where 

significant decision points and actions are captured for small group discussion 

that is conducted after the completion of a training event to help capture teachable 

moments and to aid Soldiers in reflecting on their recent training experiences. 

Both classroom training and small group instruction are manpower intensive,

requiring teachers, mentors, and support staff to guide the Soldier’s experience. 

Today, ITSs primarily guide learner training and education for cognitive tasks in 

well-defined domains (e.g., problem solving and decision-making tasks in 

mathematics and physics). Soldiers tend to perform cognitive, affective, 

psychomotor, and social tasks in both well-defined (e.g., building clearing) and 

ill-defined domains (e.g., leadership, resource allocation). ITSs generally provide 

static training (e.g., sitting at a desktop computer to train on a serious game) that 

falls short in matching the dynamic nature of many US Army operational tasks 

(e.g., psychomotor tasks), thereby reducing opportunities to develop and transfer 

skills to the operational environment. 

Research is needed to understand the characteristics, similarities, and differences 

of US Army training domains (i.e., cognitive, affective, psychomotor, social, and 

hybrid) to develop efficient and effective adaptive training and educational tools 

and methods that support SRL in complex, ill-defined, and physically dynamic 

military domains. 

3.1 Self-Regulated Learning and the US Army Learning Model

In 2011, the US Army placed significant emphasis on the development of SRL 

skills with the expectation that new methods of instruction (e.g., ITSs) would
4

augment institutional training (i.e., classroom and small group instruction). Oneto-one human tutoring has been shown to be significantly more effective than

one-to-many instructional methods (e.g., traditional classroom instruction: Bloom 

1984; VanLehn 2011). However, it is neither practical nor affordable to have one 

expert human tutor to mentor each Soldier in the US Army for every required 

operational task. This alone signals the need for capabilities to support one-to-one 

tailored training and educational experiences. 

Additionally, under the ALM, Soldiers are largely responsible for managing their 

own learning, but SRL skills are difficult to train and develop (Butler and Winne 

1995; Azevedo et al. 2009; Graesser and McNamara 2010). We anticipate 

adaptive training tools and methods will fill this gap and will provide personalized

guidance to acquire, apply, retain, and transfer knowledge and skills to the 

operational environment. This signals the need for a computer-regulated learning

strategy to augment missing SRL skills; however, adaptive training technologies 

must first become affordable, sufficiently adaptive, and easy to use for this 

strategy to be realized. 

3.2 Motivation for Research

A promising alternative to one-to-one human tutoring is one-to-one adaptive 

training tools that include ITSs. Meta-analyses and reviews support the claim that 

ITS technologies routinely improve learning over classroom teaching, reading 

texts, and/or other traditional learning methods. These meta-analyses normally 

report effect sizes (sigma [σ]), which refers to the difference between the ITS 

condition and a control condition in standard deviation units. The reported metaanalyses show positive effect sizes that vary from σ = 0.05 (Dynarsky et al. 2007) 

to σ = 1.08 (Dodds and Fletcher 2004), but most hover between σ = 0.40 and σ 

= 0.80 (Fletcher 2003; VanLehn 2011; Graesser et al. 2012; Steenbergen-Hu and 

Cooper 2013, 2014; Ma et al. in press). Our current best meta-meta estimate from 

all of these meta-analyses is σ = 0.60. This performance is comparable to human 

tutoring, which varies from between σ = 0.20 and σ = 1.00 (Cohen et al. 1982; 

Graesser et al. 2011), depending on the expertise of the tutor. Human tutors have 

not varied greatly from ITSs in direct comparisons between ITS and trained 

human tutors (VanLehn et al. 2007; VanLehn 2011; Olney et al. 2012). 

Graesser et al. (2015, in press) are convinced that some subject matters will show 

higher effect sizes than others when comparing any intervention (e.g., computer 

trainers, human tutors, group learning) to a control. It is difficult to obtain higheffect sizes for literacy and numeracy because these skills are ubiquitous in 

everyday life and habits are automatized. For example, Ritter et al. (2007) 
5

reported that the Cognitive Tutor for mathematics has shown an effect size of σ = 

0.30–0.40 in environments with minimal control over instructors. Human 

interventions to improve basic reading skills typically report an effect size of σ = 

0.20. In contrast, when the student starts essentially from ground zero, such as 

many subject matters in science and technology, then effect sizes are expected to 

be more robust. ITSs show effect sizes of σ = 0.60–2.00 in the subject areas of 

physics (VanLehn et al. 2005; VanLehn 2011), computer literacy (Graesser et al. 

2004; Graesser et al. 2012), biology (Olney et al. 2012), and scientific reasoning 

(Millis et al. 2011; Halpern et al. 2012). As a notable example, the Digital Tutor 

(Fletcher and Morrison 2012) improves information technology by an effect size 

as high as σ = 3.70 for knowledge and σ = 1.10 for skills. The effect size 

attributed to improved instruction and improved domain knowledge has not been 

separated in this analysis. Such large effect sizes would never be expected in basic 

literacy and numeracy.

Overall, these are promising results and equate to an increase of about a letter 

grade improvement over traditional classroom instruction. While ITSs are a 

promising technology to support adaptive training for individuals in well-defined 

domains like mathematics, physics, and computer programming, the US Army 

requires the ability to develop and exercise Soldier skills in more ill-defined 

domains (e.g., leadership) and at the unit level (e.g., collaborative learning and 

team training). Developing and maintaining the ability to make effective decisions 

under stress and in complex environments is also desirable. 

Adaptive systems by their nature require additional content and complexity to 

support tailored learning for each user and, as a consequence, have a very high 

development cost, a major barrier to adoption by the US Army. Adaptive systems

are also insufficiently adaptive to support tailored self-regulated training and 

educational experiences across a broad spectrum of military tasks as required by 

the ALM. Today, few ITS authoring tools are generalized across all of the 

domains requiring training, and no evaluation criteria or standards have been 

developed to promote reuse and interoperability among ITSs (Sottilare et al. 

2012b). In other words, current adaptive systems are not yet intelligent enough to 

support the tailored instruction required by the US Army in the breadth of 

domains being trained; but there is a stable foundation of 50 years of science on 

which to grow an adaptive training and education capability for the US Army.

3.3 Adaptive Training and Education Definitions

In support of the ALM and affordable adaptive training and educational 

capabilities for the US Army, ARL is investigating and developing adaptive tools 
6

and methods. A desired end state is the automation of authoring (creation)

processes, instruction, and evaluation of computer-regulated training and 

education capabilities to help build SRL skills and support mixed-initiative 

interaction. A major goal within this research program is to reduce the time/cost 

and knowledge/skill required to author, deliver, and evaluate adaptive 

technologies to make them usable by a larger segment of the US Army training 

and educational community.

Adaptive training and education research includes elements of adaptive tutoring, 

distributed learning, virtual humans, and training effectiveness evaluation. For 

additional detail on research specific to ITSs, refer to ARL-SR-0284 (Sottilare 

2013). Definitions are provided for this section to distinguish between adaptive 

training and education elements and also to highlight their relationships:

• Adaptive Tutoring: Also known as intelligent tutoring; tailored

instructional methods to provide one-to-one and one-to-many computerguided experiences focused on optimizing learning, comprehension, 

performance, retention, reasoning, and transfer of knowledge and acquired 

skills to the operational environment.

• Adaptive Tutoring Systems: Also known as ITSs; the mechanism or 

technologies (tools and methods) to provide tailored training and 

educational experiences. Adaptive tutoring systems respond to changing 

states in the learner and changing conditions in the training environment to 

optimize learning. Adaptive tutoring systems anticipate and recognize 

teachable moments.

• Virtual Humans: Artificially intelligent visual representations of people 

that simulate or emulate cognitive, affective, physical, and social

processes.

• Distributed Learning: Concurrent distribution of training and 

educational content to multiple users at the point of need in which content 

is intelligently selected to support learning, increased performance, and 

long-term competency in selected domains.

• Training/Learning Effectiveness: Evaluation of the impact of training 

and educational tools and methods on usability, learning, comprehension, 

performance, retention, reasoning, and transfer of knowledge and acquired 

skills to the operational environment.

• Adaptive Training and Education Systems: A convergence of ITSs and 

external training and education capabilities (e.g., serious games, virtual 
7

humans, simulations) to support engaging experiences with reduced need 

for authoring (Sottilare 2015).

• Generalized Intelligent Framework for Tutoring (GIFT) (Sottilare et 

al. 2012a; Sottilare et al. 2013): An open-source, modular architecture 

whose goals are to reduce the cost and skill required for authoring 

adaptive training and educational systems, to automate instructional 

delivery and management, and to develop and standardize tools for the 

evaluation of adaptive training and educational technologies.

Adaptive training and education research at ARL is being conducted across 6

interdependent research vectors: individual learner and unit modeling; 

instructional management principles; domain modeling, authoring tools and 

methods; evaluation tools and methods; and architectural and ontological support.

This report (1 of 6 interdependent research outlines) focuses on authoring tools

research for adaptive training systems with the goal of guiding learning in 

militarily relevant training and educational domains. 

Soldiers operate in a variety of complex, dynamic, ill-defined domains where 

their ability to persevere in the face of adversity, adapt to their situation, 

collaborate, and think critically are key to the successful completion of their 

assigned missions. To develop and exercise these skills, it is paramount for 

Soldiers to train in challenging environments. Presently, these few challenging 

training environments have been largely provided through manpower-intensive 

methods or systems with little ability to adapt instruction to support their learning 

needs. To illustrate this point, Franke (2011) asserts that through the use of case 

study examples, instruction can provide the pedagogical foundation for decision

making under uncertainty. However, this approach is limited in implementation 

by the expanse of potential cases that would need to be consistently updated and 

maintained to support large populations like the US Army. 

As noted previously, adaptive systems like ITSs have been shown to be effective 

in promoting learning in primarily static (e.g., learners seated at desktop 

computers) instructional settings within relatively simple, well-defined domains 

(e.g., mathematics, physics) for individual learners. For our purposes, static 

instruction includes cognitive, affective, or social training tasks where a desktop 

computer delivers instruction and where the physical movement of the learner is 

limited to activities that can be conducted while seated. For example, static

instruction can effectively support cognitive tasks involving decision making and 

problem solving but are less effective for training tasks involving motion and 

perception (e.g., land navigation and marksmanship). Ideally, we desire portable 

adaptive instructional capabilities to go with Soldiers to support training and 
8

education at their point of need across a wide spectrum of US Army operational 

tasks. Research is needed to develop tools and methods to support broader domain 

modeling, which is representative of the full spectrum of US Army operational 

tasks. Standards, interoperability, and automation (e.g., automated scenario 

generation) (Zook et al. 2012) will likely play a significant role in making

adaptive training practical. In this way adaptive training technologies will have 

the greatest impact on organizational learning in the US Army.

3.4 State of the Art in Authoring Tools and Methods

While positive and significant gains have been made to adaptive tutor authoring 

tools and technologies, progress is often measured in point-solutions and limited 

in cross-domain applicability. A number of significant challenges exist within 

tutor authoring.

First, given the current state of the art, it is estimated that 1 h of training requires 

between 10 and 100+ h of development, at an estimated cost of $10,000 per hour 

of training (Sottilare and Brawner 2014). Where authoring systems are concerned, 

tradeoffs are often made between functionality and generalizability. For instance, 

adaptive tutoring content can be authored for well-structured domains (e.g., math, 

physics) using relatively stable authoring tools specifically tailored to said 

content, but these tools fall short in ill-defined domains, including those complex 

domains that are relevant to Army training objectives (e.g., decision making, 

leadership, teamwork). It logically follows that authoring content for Armyrelevant skills and tasks would currently require even more time and money to 

author, given the current state of the art. Therefore, as advances are made in the 

parallel areas of domain modeling, learner modeling, and instructional 

management, so too are tools needed to leverage outputs from those areas to 

author content for ill-defined, complex domains. 

Second, the domain of adaptive tutor authoring is still in its infancy; as such, there 

is a general lack of standardization between tools and methods (workflow) for 

authoring such content. While ITS and traditional computer-based training (CBT) 

bear some superficial similarities in delivery and presentation, the complexity of 

authoring adaptive tutoring is many orders of magnitude greater because of the 

inclusion of learner modeling, pedagogical agents, complex configuration, and 

content creation. Further, the skill of authoring adaptive tutors represents a 

relatively new interaction paradigm. The authoring process might share processes 

associated with other content creation activities, such as assembling a slide deck, 

designing a blog post, or sequencing an interactive narrative (such as a video 

game). Though, none of the aforementioned activities represent a one-to-one 
9

mapping with authoring adaptive tutors. It is a new form of content creation and 

an activity that is yet to be fully defined. Thus, potential authors must form new

mental models regarding the creation of adaptive tutors by integrating new 

knowledge with existing mental models of activities that are perceived to be 

similar. The authoring vector will support users in constructing rich and accurate

mental models of tutor authoring.

Third, the manner in which adaptive tutors will interoperate with external systems 

is currently piecemeal, implemented on a case-by-case basis, which further 

complicates the authoring process and serves as a threat to the generalizability of 

adaptive authoring systems. GIFT, for instance, is a framework; it is intended to 

be interoperable with a variety of external programs and technologies. In practice, 

separate communication pipelines are required for each specific application. 

Further, these pipelines provide inconsistent functionality with respect to the type 

of data that can be exchanged between GIFT and the application; thus, the 

communication and configuration options that are available to tutor creators will 

also vary wildly. Interoperability with existing simulators’ authoring 

environments, content management, and assessment engines is a significant threat 

to a consistent authoring experience, the root cause of which may lie beyond the 

scope of this vector. For instance, the modeling, simulation, and training industry 

is far larger in size and scope than the ITS industry. As a by-product, efforts to 

synchronize the modeling and simulation community with the needs of the ITS 

community have languished (Stottler et al. 2005) because, in part, developmental

order often favors a simulation first and instructional system second approach. 

Those authoring concerns are not limited to training applications. A fully realized, 

generalizable authoring system will also need to be able to interface with content 

management systems, learner databases, social media frameworks, and 

(eventually) intelligent agents. As those capabilities are enabled by the 

architecture, research will be required to define authoring requirements in 

interacting with those supporting components. 

3.5 Overview of GIFT Authoring Tools

GIFT is described as “an empirically-based, service-oriented framework of tools, 

methods and standards to make it easier to author computer-based tutoring 

systems (CBTS), manage instruction and assess the effect of CBTS, components 

and methodologies” (Sottilare et al. 2012a). As GIFT is simultaneously a research 

project and open-source application, it is in continuous development and includes 

a number of technologies, features, and tools targeted toward a variety of users,

including instructional designers, researchers, and students. For instance, the 
10

framework incorporates models of domain content, pedagogical methods, sensor 

data processing, interoperability with external applications, and learners, each 

with its own module and/or configuration. As such, it is important to recognize 

that GIFT is not simply a set of authoring tools; however, research issues related 

to authoring are part of the focus of the current program plan and the explicit 

focus of this report.

Until recently, GIFT applications were created by writing and/or editing 

eXtensible Markup Language (XML). One “tutor” was, in reality, a series of 

separately configured but inter-reliant components. Directly editing the files was a 

complex process that required users to understand how to properly write encoded 

XML. XML editing programs were created to make its editing easier, but this did 

not alleviate users’ need to understand how the various components of GIFT 

interact with one another at the system level. By comparison, the current stable 

versions of GIFT (2014-2 and 2015-1) provide some browser-based interfaces to 

facilitate semi-automated user creation of the XML output; these interfaces take 

the first steps toward unifying a diverse set of authoring tools behind a consistent 

user interaction experience. However, the current implementation of the 

graphical-user interfaces does not change the requirement that authors must 

understand the system-level conceptual model of GIFT. In this report we propose

research that will result in interfaces that help to close the gap between an 

author’s mental model of adaptive tutor creation and the system-level conceptual 

model of GIFT.

Sottilare (2013) outlined goals for GIFT, including the following authoringrelated goals: decrease the effort required for authoring (time, cost, etc.), decrease 

the skill threshold required to author adaptive tutoring, support users in organizing 

knowledge, support pedagogical design, allow for rapid prototyping of adaptive 

tutoring, leverage standards for integration of external media and applications, 

and promote content reuse. The authoring goals for GIFT are concurrent with 

authoring goals for ITS, in general, which endeavor to support training and 

education requirements outlined in the ALM. 

4. US Army Requirements for Adaptive Training Systems and 

Authoring

The Army Science and Technology community uses Warfighter Outcomes 

(WFOs) as the authoritative source for identifying Warfighter needs. WFOs are 

used to share research and future technology solutions. In the training and 

education domain, the adaptive training and education research program is 

targeting 4 specific requirements to support the evolution of US Army training: 
11

adaptive training and education systems; big data; training at the point of need; 

and artificial intelligence. 

4.1 Adaptive Training and Education Systems and Authoring

The primary gap to be addressed under this Army requirement is the lack of 

adaptive systems (e.g., intelligent tutors) to support individual and collective 

(team or unit) training. The Army needs an adaptive training and education 

capability that is persistent and easy to use/access with minimal start-up time. 

There are also requirements to automate an informal AAR (also known as postexercise critique) to reduce the time and skills needed to produce the AAR and 

improve its focus and quality. Another line of thought also notes that the artificial 

intelligence in ITSs could be used to facilitate rapid mission planning and courseof-action analyses as a job aid in operational contexts. 

The major connection between the adaptive training and education requirement 

and the authoring tools and methods research vector is the need to extend 

authoring capabilities in 2 directions. The first of these directions relies upon the 

enabling of the authoring of feedback for AAR in complex domains or in the 

rapid authoring of correct assessment models within simplistic domains. The 

second of these directions relies upon the automation and magnification of the 

authoring process through extensive reuse, content readability by computerized 

process, and the configuration of the system to meet the needs of the learner.

4.2 Big Data and Authoring

The primary gap to be addressed under this US Army requirement is that there is 

a lack of capability to handle and process large amounts of structured and 

unstructured data (also referred to as big data). One capability needed is a 

structured data analytics program linking individual data (e.g., achievements) to 

required long-term competencies in military occupational specialties (MOSs). 

This would allow Soldiers to understand where they rank in terms of experiences 

and achievements among other Soldiers in their MOS. It would also allow the US 

Army to identify specific experiences among successful Soldiers in that MOS and 

provide a model for other Soldiers in that MOS to follow. The data could also be 

used by course managers and instructors to continuously improve instruction and 

the mental models of both human and computer-based instructors. Finally, data 

collected on trainee learning and performance during adaptive training 

experiences could be used to facilitate Unit Training Management where unit 

commanders would have access to empirical data to support unit training 

decisions.
12

The major connection between the Army’s big data requirement and authoring 

tools is ability to collect or generate learner data from content interactions, 

content data for machine traversal, and training environment data to make the 

connection among learner behaviors, instructional tactics, and their varying 

degrees of effectiveness given instructional context (existing conditions). This 

will allow course managers to be aided by machine techniques of content creation 

and to see the effectiveness of their authored material. 

4.3 Training at the Point of Need and Authoring

The primary gap to be addressed under this US Army requirement is the lack of

an easily accessible, persistent, cost-effective, and low-overhead training 

environment. A capability is needed to bring training to Soldiers instead of 

Soldiers going to fixed training locations. This point-of-need training capability 

would be easily distributed, web based, and built upon an open-enterprise 

architecture in the cloud. US Army training and educational opportunities would 

be available on demand anywhere and anytime. However, the delivery mechanism 

(e.g., laptop computer, mobile device, and smart glasses) for adaptive training is 

critical in determining the limitations of the domain model scope and complexity. 

For example, it may be extremely difficult to train all the complexities of a 

psychomotor task in a desktop computer setting.

The major connection between point-of-need training and authoring relies upon 

the practicality of extending adaptive training beyond the desktop. This is 

primarily an architectural content conversion process, avoiding the work of 

making multiple complete instances of content (web, mobile, desktop, smart 

glasses, etc.). Training content (e.g., instructional content, assessments, feedback) 

should be able to be reworked into the appropriate format.

4.4 Artificial Intelligence (AI) Capabilities and Authoring

The primary gap to be addressed under this US Army requirement is that the US 

Army lacks an automated capability to replicate the complexity and uncertainty of 

the operational environment. This gap specifically points to the lack of 

adaptiveness in virtual humans, intelligent tutoring systems, and other training 

capabilities. This gap leads to Soldiers developing training-response strategies 

that result in less challenging training over time along with lower engagement and 

lower levels of learning and transfer of skills to more challenging operational 

environments. 

The major connection between AI capabilities and authoring involves the 

discovery and innovation of techniques to support computer-aided authoring. 
13

Computer-aided authoring comes in a few differing types, from automated 

scenario evolution, to automated question and hint generation, to intelligent 

content search and retrieval. By implementing such concepts with AI support, the 

authoring burden for instructors and course managers developing highly complex 

training and educational domains may be reduced.

AI capabilities, domain modeling, and authoring involve the discovery and 

innovation of techniques to support a concept called “automated scenario 

evolution” being developed by Sottilare (2015). AI capabilities are needed to 

support automated scenario evolution where AI drives the generation of new 

“child” scenarios from a single-parent scenario based on dimensions of that 

scenario and the state of the trainee. In this way, the authoring burden for highly 

complex training and educational domains may be reduced. 

For example, consider a single scenario where dimensions include variable 

challenge levels based on 3 threats (i.e., low, moderate, high), 3 types of field of

view (i.e., narrow, moderate, and wide), and clear line of sight (i.e., near, 

moderate, and far). AI could spawn 27 new child scenarios based on combinations 

of these variables. This requirement is closely linked to adaptive training 

capabilities described in Section 4.1 of this report, and the realization of this 

capability will enable the development of affordable self-authoring adaptive 

systems. Through this capability, complex domains may be modeled for adaptive 

training systems without the need for long development cycles or special 

authoring skill sets.

5. Understanding the Dimensions of Authoring

There are 4 typical elements that compose ITSs, a prime example of an adaptive 

training and education system: a learner or trainee model, an instructional or 

pedagogical model, a domain model, and some type of user interface. The domain 

model typically includes an expert or ideal student model by which the adaptive

system measures/compares/contrasts the progress of the learner toward learning 

objectives. The domain model also includes the training environment, the training 

task, and all of the associated instructional actions (e.g., feedback, questions, 

hints, pumps, and prompts) that could possibly be delivered by the adaptive 

system for that particular training domain. Typical interaction among the learner, 

the training environment, and the adaptive system (tutoring agents) is shown in 

Fig. 1.

Typical training systems examine the interaction between the learner and the 

training environment to measure progress toward learning objectives. The learner 

acts on the environment (e.g., opens a door or makes a choice to move into the 
14

room or stay outside) and then observes any changes or reactions within the 

environment. Adaptive systems add a layer of software-based tutoring agents that 

are designed to guide the learner in much the same way as a human tutor interacts 

with a learner. The tutoring agents observe the behaviors of the learner to assess 

their states (e.g., performance and attitudes) and interact with the learner to 

provide support, direction, and instruction. In addition, they track the effect of 

interactions on learning. Tutoring agents also interact with the training 

environment and may manipulate the environment to present more challenging or 

less challenging scenarios in response to the assessed state of the learner.

Fig. 1 Adaptive training interaction

Authoring is an element of the ITS, which is not overtly visible in Fig. 1, but it 

potentially interacts with and/or influences each of the other elements within the 

tutor. Authoring identifies, for example, what is observed, how an agent interacts, 

and why it selects certain interaction strategies over others. In practice, the 

authoring element requires input from many dimensions: the ability to configure 

each of the other ITS elements (including external training environments), a 

mechanism to incorporate new components and configurations into the ITS, and 

significant human capital (at least for the near future) to help shape the tutor, 

manipulate its many configurations, and harness its output. These dimensions are 

described in greater detail below.

The widespread adoption of adaptive tutoring will require authoring systems that 

are easy to use and contain the functionality needed to support a wide variety of 

training tasks and environments. Murray (2003) described the current state of 
15

adaptive tutor authoring as a series of tradeoffs between usability, depth, and 

flexibility. Given the depth and flexibility required to author for the complex 

domains associated with Army training objectives, the current authoring process 

is complicated, requiring a degree of skill similar to that of computer 

programming.

Mental models will serve as the underlying theory for the user-centered research 

component of this outline (described in Section 6.1). Rouse and Morris (1986)

explained that mental models “are the mechanisms whereby humans are able to 

generate descriptions of purpose and form, explanations of system functioning 

and observed system states, and predictions of future states” (p. 7). Mental models 

influence users’ expectations regarding a system’s functionality and guide user 

interaction behavior (Ososky 2013). An individual’s mental model regarding a 

particular system is influenced by past experiences and perceived similarity of 

other systems to the target system. Further, human mental models do not need to 

be complete or even accurate to be applied to a specific system interaction 

(Norman 1986).

Therefore, to increase the usability of authoring tools, it first is necessary to 

identify and describe authors’ mental models for authoring tools. Potential 

authors, however, may enter into an interaction with authoring tools with a variety 

of backgrounds, including instructional design, training facilitation, and subject 

matter expertise. It is likely that each user group will reveal a mental model for 

interacting with authoring tools, which is significantly different from the others. It 

is important to acknowledge the distinction between user groups because those 

models are likely to be (even more) different from the manner in which the 

authoring system is actually designed. This is because engineers have their own 

mental models about how authoring tools should work, and they use this model to 

develop the realized version of the system (Streitz 1988). The engineer’s or 

designer’s model has been referred to as the conceptual model (Norman 1983).

Stated differently, users may know (in their minds) what type of adaptive content 

they need to author but are unable to leverage the authoring system to achieve 

their goals. Thus, a gap exists between user intent and system usability. Likewise, 

the more that is assumed about how users may interact with authoring tools, the 

larger this gap is likely to be. Norman (1986) described this gap as the gulfs of 

execution and evaluation. The gap between user intent and system representation 

was expressed as 2 gulfs because there are 2 ways to close the gap. Users can 

adapt their plans to the design of the system (e.g., acquire new knowledge of a 

complex system), or the system’s interface can be improved to match the needs of 

the user (e.g., leverage familiar terms and interaction elements). For adaptive tutor 

authoring tools, it will likely be necessary to approach this gap from both sides.
16

6. Authoring Research Goals and Challenges

A foundational goal of adaptive training and education research at ARL is to 

model the perception, judgment, and behaviors of expert human tutors to support 

practical, effective, and affordable learning experiences guided by computerbased agents. To this end, and in support of the authoring goals stated earlier, the 

following 4 primary challenges for authoring adaptive systems have been 

identified and are described in this section. 

• Describe human mental models and define authoring interaction 

paradigms.

• Identify candidates for automated authoring and develop processes to 

reduce the human authoring workload.

• Extend authoring capabilities to support integration with existing training 

and educational technologies (e.g., training simulations, simulators, and 

serious game platforms).

• Enable collaborative authoring.

6.1 Describe Human Mental Models and Define Authoring 

Interaction Paradigms

Stated previously, authoring adaptive tutoring content represents a new interaction 

paradigm. Further, tradeoffs are made in authoring between usability, depth, and 

flexibility. The purpose of this subtask is to attend to the usability component of 

the current state of the art. The end goal of this subtask is to reduce the time and 

skill required to author adaptive tutoring content. 

Designing an interface to support the creation of adaptive tutors is a nontrivial 

task and one that is not quite like many other superficially similar tasks (e.g., 

creating slides, creating a web page, or designing a video game). Many potential 

users may not even fully realize the benefits that adaptive tutoring can provide 

over traditional classroom or CBT. Further, there may be multiple user groups 

that may be interested in creating adaptive tutors, and each of these user groups 

has their own sets of knowledge and interaction expectations. For instance, we 

can theoretically identify potential authors from instructional designers, subject 

matter experts, and researchers, respectively, but do they need to be regarded as 

different types of users for the authoring system?

Therefore, the research approach will examine the design of authoring tools from 

both sides of the system usability gap. We intend to conduct user research to 

illustrate the various mental models of adaptive tutor design, as well as non-
17

system-specific goals of those potential users. We will leverage users’ goals, 

desires, and intentions to drive the requirements for the design of a prototype 

authoring tool interface for GIFT. Outputs from this research will also help to 

identify differences in the needs, if any, between each user group. Research 

activities may include, but are not limited to, interviews, surveys, and 

direct/indirect observation. Authoring tools will be evaluated along dimensions of 

utility (“Does the system support users’ goals?”), usability (“Can the user easily 

accomplish their goals?”), and the time to author an hour of adaptive course 

content. 

It is unlikely, at least in the near term, that it will be possible to design-out all of 

the complexity of authoring an adaptive tutor. We must strive for a reasonable 

balance in the tradeoffs between usability, depth, and flexibility. To that end, 

users may need to learn about concepts that are unique to ITS in order to develop 

new skills and knowledge that will enable users to design effective tutors (for a 

discussion of automated authoring, see Section 3.2). To that end, we propose to 

examine the system side of the usability gap as well. Currently, GIFT is 

composed of a rudimentary set of GUI-based authoring tools defined by systemlevel terminology and theoretical ITS components. We will conduct research that 

will examine conceptual models of tutor authoring with the goal of improving the 

ease of learning of terminology and components associated with ITS. In service of 

this objective, we will survey the waterfront of existing authoring tools and 

technologies to identify themes in authoring components and terminology, as well 

as present suggestions for improvements upon the current workflow. The outputs 

from this research are expected to produce guidelines and best practices that could 

be used to drive the establishments of standards in the field of adaptive tutoring. 

Continuous improvement of prototype authoring interfaces and interaction 

paradigms is the final cornerstone of our proposed research approach. We will 

track the progress of the authoring system after the bridge has been built from 

both sides of the gap. We intend to develop and implement tools that will 

manually and automatically collect data with respect to the use of and experience 

with GIFT authoring tools. Research activities may include, but are not limited to,

website metrics (in the case of cloud-based authoring tools), remote unattended 

usability testing, and collection of qualitative user experience feedback via the 

authoring system or GIFT community forum channels.
18

6.2 Identify Candidates for Automated Authoring and Develop 

Processes to Reduce the Human Authoring Workload

The process of building a tutor is more involved than the creation and 

customization of its various component parts. Many of these component parts will 

not require authoring for the average user. As an example, the pedagogical 

method of instruction from one domain of instruction to another should remain 

mostly the same. The same may be said of the methods of modeling the learner, 

or of the sensors used. Having an authoring system that makes extensive use of 

default settings (or preconfigured sets of settings) allows the individual tutor 

author to focus on the domain of instruction.

In regards to the domain of instruction, the basic process is to 1) gather content, 

2) make the content available, 3) customize it to suit the individual needs of the 

audience, 4) generate tutoring information, 5) perform the delivery of the content 

for informational and practice settings, and 6) take traditional ITS actions (e.g.,

model the learner, select from content, track learner experience) (Olney et al. 

2015).

Currently, the first step of authoring (content gathering) is typically performed as 

a series of searches via a search engine. Modern search engines perform many of 

the functions behind this content gathering automatically (utilizing search history, 

individual profiles, etc.) in an effort to improve search results. If available content 

were already indexed according to keywords and learning objectives, it is 

conceivable that this process may be automated (Jesukiewicz and Rehak 2011; 

GooruLearning 2014).

The actions of indexing this content for availability, such as tagging the content 

with metadata, can be performed in an automated fashion with today’s cuttingedge technology (Veden 2015), which is an architecturally assisted process 

discussed in another research vector. If this content is text related, simple tutoring 

information, such as hint/prompt/pump generation, can be performed in a 

machine-assisted manner (Olney et al. 2012).

Indexed and actionable content, hosted in a repository, can be traversed through a 

machine-assisted process in accordance with the learner model. As an example, 

content for a highly motivated and experienced learner could be queried from a 

content provisioning system with descriptors of the context needed for the 

individual (Ray et al. 2014). Such a provisioning system would be able to match the 

learner to content based on his/her personal traits and the individual content 

attributes (e.g., subject matter, difficulty level, user ratings). A learner can progress 
19

through a learning path planned by the pedagogical system, taking prescribed 

pedagogical actions, such as hint giving, because of their pregenerated nature.

The system above describes a process of automating the process of authoring and 

automatically building an appropriate course, using up-to-date and reusable 

learning objects (RLOs) created and curated in a centralized repository. Portions 

of such a system exist through pedagogical recommendation engines, such as the 

eMAP (Goldberg et al. 2012), content curation systems, such as Nuxeo (Nuxeo 

Platform 2015), exchange languages, such as xAPI (Regan 2013), and 3QL (Ray

et al. 2014), and others. However, research is needed to support the tie-in, 

integration, testing, effectiveness, and accuracy of such an automated system.

6.3 Extend Authoring Capabilities to Support Integration with 

Existing Training and Educational Technologies

The development of an ITS is expensive, labor intensive, and requires highly 

skilled personnel from a variety of disciplines. One way to reduce the time and 

cost of development is to encourage reuse through the use of standards for 

processes, tools, and software components. Templates for domain models and 

learning content could also reduce costs (Sottilare 2014). Content Management 

Systems (CMSs), for example, can be used to reduce costs by supporting learning 

content reuse. However, it is time consuming to upload content, and typically, 

metadata is manually input by administrative personnel, reducing the 

effectiveness of search and retrieval tools. Research continues in automating the 

process of analyzing documents to decompose them into RLOs and generate 

accurate metadata. Data analytics is another method that may be used to provide a 

smart search capability for a CMS. Current research is also exploring “best 

practices” for the use of RLOs to include the use of assessments. Social media 

may also provide a crowd-sourced capability in the generation of learning content, 

as well as vetting current learning content/courses.

In adaptive training environments, virtual humans can be used as virtual 

instructors (e.g., talking heads) to help guide the learner through a training 

session. Nonplayer Characters can also be used to support training scenarios that 

require human interaction. GIFT provides the Media Semantic character set to 

support the use of commercial virtual humans. Further research is required to ease 

the authoring process for the creation and management of virtual humans within 

adaptive tutoring environments.

Another challenge is to improve/standardize authoring support for external 

training applications (e.g., the integration of ITSs with simulations or game-based 

training environments). The Game-based Architecture for Mentor-Enhanced 
20

Training Environments (GAMETE) is one example. GAMETE is focused on 

allowing training developers to add an intelligent tutoring capability into a gamebased simulation environment (Engimann et al. 2014). Automated scenario 

generation (Zook et al. 2012) is another area of interest, as is authoring support 

for tutors intended for units, teams, and teams of teams.

In addition to the areas discussed above, research is needed to determine optimal 

methods for authoring: 

• Techniques, strategies, and tactics within GIFT 

• Performance and competency assessments within GIFT 

• Domain content and expert model development 

• New content and instructional models from existing data sources 

6.4 Enable Collaborative Authoring

It is envisioned that, in many cases, a team of individuals will be developing a 

course; those individuals will have differing responsibilities. As an example, one 

individual may create a pre-/posttest while another may put together lectures, 

another may design practice scenarios, and yet another may review/test the 

completed course for quality assurance. Beyond the user experience concerns for 

individual authoring discussed in the previous section, there are a number of 

follow-on issues that will need to be addressed to enable collaborative authoring. 

As a simple example, content permissions need to be implemented that allow the 

appropriate individuals (or groups) to have access to different components, keep 

items restricted from general users, prevent overwriting of changes from 

unauthorized users, and provide a functional baseline for quality testing. This 

research subtask will examine issues related to collaborative authoring, including 

implementation, compatibility, and exporting. 

6.4.1 Web Authoring

In GIFT’s desktop implementation, authors can create their own questions, 

surveys, and survey contexts using the Survey Authoring System. All of the 

questions that have been created and included in GIFT’s baseline are available to 

all GIFT users. Once an individual creates his/her own surveys and questions, that 

content stays local to the user’s own desktop version unless he/she exports and

then imports the survey to another individual’s version of GIFT. In the 

development of a web-based version of the Survey Authoring System,

consideration will need to be given to where content created by an individual user 

is stored and whether it is included in the survey bank for all to use. This creates a 
21

unique problem, as instructors who may be using GIFT to create and administer 

tests likely will not want their assessments and questions (and answers) readily 

available to other individuals and students. 

To address the storage and availability of content authored by specific individuals, 

research and thought will need to be put into permissions and storage. The survey 

authoring system in particular will likely need to only provide the stored content 

to the individual that created it and anyone working on the research/authoring 

team. Consideration will need to be given to the new challenges that develop from 

moving the authoring tools to a web environment. Development will be needed to 

address the traditional problems of sharing, access, and editing.

6.4.2 Versioning and Compatibility

It is very important that created content is backward compatible with previous 

versions of GIFT. Individuals may invest a large amount of time in developing a 

course in a specific version of GIFT and may resist moving to a newer version if 

their content will not be compatible. In a desktop implementation they can 

continue using their previous version; however, in a web-based implementation, 

this backward compatibility has greater importance. Web-based implementations 

do not necessarily allow for the functionality of “snapshotting” a course in order 

to maintain an existing version.

Furthermore, there is an importance on optional updates of course material. As an 

example, if a resource used within a course has been updated by a third party, the 

course author should have the capability (but not the necessity) to update that 

course to use the new object. Such updates should not be mandatory, as the course 

author should have the opportunity to review the material prior to delivering it to 

students from a shared web-enabled environment.

6.4.3 Sharing and Exporting Tutors

In addition to collaborative authoring, it is logical to assume that authors will 

want to share their created tutors with schoolhouses and/or learners. GIFT 

currently has 2 procedures to share courses: either 1) the export of a course where 

it is presumed that the receiving GIFT user has a full installation GIFT software

and third-party applications or 2) to export a course with an entire install of GIFT, 

resulting in a relatively large software package (500 MB). There are 2 concerns in 

this area that require further attention: 1) examination of the export package for 

more efficient distribution and 2) how to handle configuration errors when the 

receiving user is missing required hardware or software components. These issues 

are applicable even within a cloud-based implementation of GIFT. 
22

7. Interdependencies with Other Adaptive Training Research 

Vectors

This section examines interdependencies between authoring and the other 5 

adaptive training research vectors (Fig. 2). This discussion forms the basis for the 

sequencing of research and ultimately bringing adaptive training capabilities into 

a state of practice.

Fig. 2 Adaptive training research vectors

Authoring tools and technologies enable the selection and configuration of learner 

models, instructional pedagogy, domain content, and evaluation metrics, 

respectively. Conversely, the products and insights generated from each of the 

other vectors provide guidance to the requirements for the authoring experience. 

Finally, reusable content and automated authoring rely upon architecture to fully 

realize those capabilities. 

7.1 Learner Modeling and Authoring

With respect to authoring, tutor creators will require guidance and tools to select 

and configure the appropriate dimensions of a learner model for a given course. 

This might require the content creator to select a subset of data points from a 

larger long-term learner record and/or integrate new attributes into the model. 

Likewise, authors will need to be able to specify where and how these data are 

used within the tutor, as well as how to add the data back into a long-term learner 

record store upon completion of the course. Accurate modeling of the learner is 
23

critical to driving instructional decisions in adaptive training systems, but research 

is needed to determine what this dataset will look like. Candidates abound in the 

literature, but in general, these include transient data/states, cumulative states 

(building over time), and enduring data/states (Paneva 2006). Transient measures 

of importance include individual behavioral and physiological data, and cognitive, 

affective, physical, and social states to represent learning. Cumulative measures 

include achievements (e.g., certifications, training, education, and experiences), 

affiliations, work history, and domain competency. More enduring information 

about the learner might include gender, culture, first language, physical 

constraints (e.g., colorblind/deaf), values, personality attributes, or other traitbased information (Sottilare and Brawner 2014). All of these measures/states are 

potential drivers for adaptive training decisions. Authoring support for learner 

modeling may take the form of useful user interfaces, semi-automated 

configuration support, and connection gateways to external learner database 

systems. Philosophically, an author should have to author items related to the 

assessment of student knowledge and experiences, but the individual differences, 

states, and trait information should be abstracted away from their use in 

nonresearch settings.

7.2 Automated Instruction and Authoring

Authoring will work with the instructional management vector to outline the 

author’s role in instructional management. Specifically, we will work to 

determine what level of control is appropriate with respect to authoring 

instructional strategies. We will receive the outputs of the instructional 

management vector to determine how to design the authoring interaction for the 

management and selection of content appropriate to a chosen instructional 

strategy (e.g., Merrill’s component display theory, Gagne’s 9 events of 

instruction). The instructional management and authoring vectors will also 

collaborate on larger, cross-vector issues, including the design of adaptive 

tutoring for teams. Generally, the goal is not to involve automated instruction in 

the authoring process; philosophically, the content author or subject matter expert 

should only require knowledge of a domain, while the system should know best 

how to instruct it. If any authoring decisions are involved, they should be limited 

to a specific subset of available models, with default values well informed from 

the state of practice.

7.3 Domain Modeling and Authoring

With respect to domain modeling, authoring knowledge representations for 

purposes of intelligent tutoring is a significant challenge within the current 
24

iteration of the GIFT authoring tools, with the authoring and domain modeling 

groups being tightly linked. We will exchange data and information with the 

domain modeling vector to synchronize author requirements for domain 

modeling, determine the level of transparency of the domain modeling process to 

the end user, and develop user interfaces for modeling knowledge, feedback, and 

other instructional interventions. This will be performed with an emphasis toward 

the automation of items that can be automated, or with an emphasis toward mental 

models where the author must be involved.

7.4 Evaluation and Authoring

Training effectiveness is concerned with the impact of training on learning, 

retention, and transfer. Outputs of training effectiveness research will specify 

methods for measuring, analyzing, and instantiating instructional strategies across 

different training domains and team configurations. In practice, these measures 

may take the form of data hooks that will be built into the tutor to facilitate 

automated data collection for either real-time or posttraining analysis. There is 

likely to be considerable overlap between the data needs of training effectiveness 

and those of the tutor itself. Therefore, there exists an opportunity to take 

advantage of these overlaps in the specification of semi-automated and automated 

authoring interfaces, leading to a more robust authoring suite, thus reducing the 

time and resources required to develop a complete tutoring solution for a given 

domain. Data hooks for the automated conduct of authoring studies should be 

developed and used in the same fashion as many of the web-based services now 

commercially available. While not expressly a research task, the task of 

developing new mental models to support new types of system authoring is a 

research task that will be informed through these now-traditional methods.

7.5 Architecture and Authoring

The Architecture vector interacts with each of the other vectors, including 

Authoring. Architecture and Authoring will work together to close the gap 

between the user mental model and the system conceptual model—Authoring will 

provide guidelines for the design of the authoring tool interface and the overall 

user experience. The authoring and architecture group are tightly linked in the 

same manner as the authoring and domain modeling group; one relies upon the 

user mental model, while the other relies upon the computerized content model. 

Architecture will specify the technologies for which user experiences are 

required, and both teams will work together to bring new functionality online,

including social media integration, reusable components, and content 

management systems. Finally, it is expected that Architecture will enable greater 
25

automation of adaptive tutoring content over time, which will require parallel 

development in the authoring experience.

8. Conclusions

This report outlines ARL’s plans for conducting research in adaptive training and 

education to support the ALM (US Army Training and Doctrine Command 2011). 

Specifically, this report relates to research and prototype development of 

authoring tools and methods. We seek to the answer to the question, What 

adaptive training methods provide the best value (in terms of effectiveness and 

affordability) for the authoring of US Army Training and Education tutoring 

domains?

This report outlined the following research goals:

• [For human authoring]: Apply usability heuristics to develop best 

practices to guide authoring of adaptive systems based on the author’s role 

(e.g., domain expert, instructional designer, or course manager).

• [For computer-aided authoring]: Identify elements of the authoring 

process that are candidates for automated or semi-automated authoring 

processes to remove the human from the authoring process wherever 

possible.

• [For leveraging existing training and educational capabilities]: Establish 

standards for the integration of functionally disparate tools and 

technologies (e.g., external training systems and serious games) that may 

be relevant to adaptive systems to reduce the need for authoring.

These research goals, in turn, support the following goals of the authoring 

vector: 

• Decrease the resources (time, cost, etc.) required to author an intelligent 

tutor.

• Decrease the skill threshold required by various user groups associated 

with authoring and managing an intelligent tutor.

• Enable rapid prototyping of intelligent tutors for rapid design and 

evaluation of capabilities.

• Develop standards, including common tools and interfaces, for tutor 

authoring.

• Promote reuse of content, modules, and data structures in tutors.
26

9. References

Azevedo R, Witherspoon A, Graesser AC, McNamara DS, Chauncey A, Siler E, 

Cai Z, Lintean M. MetaTutor: analyzing self-regulated learning in a tutoring 

system for biology. In: Dimitrova V, Mizoguchi R, Du Boulay B, Graesser

AC, editors. Artificial intelligence in education: building learning systems 

that care: from knowledge representation to affective modelling. Amsterdam 

(The Netherlands): IOS Press; 2009. p. 635–637.

Bloom BS. The 2 sigma problem: the search for methods of group instruction as 

effective as one-to-one tutoring. Educational Researcher. 1984;13(6):6.

Butler DL, Winne PH. Feedback and self-regulated learning: a theoretical 

synthesis. Review of Educational Research. 1995;65:245–281.

Cohen PA, Kulik JA, Kulik CLC. Educational outcomes of tutoring: a metaanalysis of findings. American Educational Research Journal. 1982;19:237–

248.

Dodds P, Fletcher JD. Opportunities for new “smart” elearning environments 

enabled by next generation web capabilities. Journal of Educational 

Multimedia and Hypermedia. 2004;13:391–404.

Dynarsky M, Agodini R, Heaviside S, Novak T, Carey N, Camuzano L, Sussex 

W. Effectiveness of reading and mathematics software products: findings 

from the first student cohort; March Report to Congress; 2007 [accessed 

2015 May 15]. http://ies.ed.gov/ncee/pdf/20074005.pdf.

Engimann J, Santarelli T, Zachary W, Hu X, Cai Z, Mall H, Goldberg B. 

Gamebased architecture for mentor-enhanced training environments 

(GAMETE). In: Sottilare R, editor. 2nd Annual GIFT Users Symposium 

(GIFTSym2); 2014 June 12–13; Pittsburgh, PA. Orlando (FL): Army 

Research Laboratory. ISBN: 978-0-9893923-4-1.

Fletcher JD. Evidence for learning from technology-assisted instruction. In:

O’Neil HF, Perez R, editors. Technology applications in education: a

learning view. Mahwah (NJ): Erlbaum; 2003. p. 79–99.

Fletcher JD, Morrison JE. DARPA digital tutor: assessment data Alexandria

(VA): Institute for Defense Analyses; 2012. IDA document D-4686

[accessed 2015 May]. http://www.acuitus.com/web/pdf/D4686-DF.pdf.
27

Franke D. Decision-making under uncertainty: using case studies for teaching 

strategy in complex environments. Journal of Military and Strategic Studies. 

2011;13(2):1–21 [accessed 2015 May]. http://ww.jmss.org/jmss/index.php

/jmss/article/viewFile/385/398.

Goldberg B, Brawner K, Sottilare R, Tarr R, Billings D, Malone M. Use of 

evidence-based strategies to expand extensibility of adaptive tutoring 

technologies. In: Proceedings of the Interservice/Industry Training Simulation 

and Education Conference; 2012 Dec; Orlando, FL.

GooruLearning. Redwood City (CA); n.d. [accessed 2014 Oct 10].

http://www.goorulearning.org.

Graesser AC, Conley M, Olney A. Intelligent tutoring systems. In: Harris KR,

Graham S, Urdan T, editors. APA educational psychology handbook.

Washington (DC): American Psychological Association; 2012. p. 451–473.

Applications to learning and teaching; vol. 3.

Graesser AC, D’Mello SK, Cade W. Instruction based on tutoring. In: Mayer RE, 

Alexander PA, editors. Handbook of research on learning and instruction. 

New York (NY): Routledge Press; 2011. p. 408–426.

Graesser AC, Hu X, Nye B, Sottilare R. Intelligent tutoring systems, serious 

games, and the generalized intelligent framework for tutoring (GIFT). In:

O’Neil HF, Baker EL, Perez RS, editors. Using games and simulation for 

teaching and assessment. Abingdon (UK): Routledge; in press.

Graesser AC, Lu S, Jackson GT, Mitchell H, Ventura M, Olney A, 

Louwerse MM. AutoTutor: a tutor with dialogue in natural language. 

Behavioral Research Methods, Instruments, and Computers. 2004;36:180–

193.

Graesser AC, McNamara DS. Self-regulated learning in learning environments 

with pedagogical agents that interact in natural language. Educational 

Psychologist. 2010;45(4):234–244.

Halpern DF, Millis K, Graesser AC, Butler H, Forsyth C, Cai Z. Operation ARA: 

a computerized learning game that teaches critical thinking and scientific 

reasoning. Thinking Skills and Creativity. 2012;7:93–100.

Jesukiewicz P, Rehak DR. The learning registry: sharing federal learning 

resources. In: Buckingham Shum S, Gasevic D, Ferguson R, editors. LAK 

’12. Proceedings of the 2nd International Conference on Learning Analytics 

and Knowledge; 28 Nov–1 Dec 2011; Orlando, Florida. New York (NY): 

ACM; 2012. p. 208–211.
28

Ma W, Adesope OO, Nesbit JC. Intelligent tutoring systems and learning 

outcomes: A meta-analytic survey. Journal of Educational Psychology. In 

press.

Millis K, Forsyth C, Butler H, Wallace P, Graesser AC, Halpern D. Operation 

ARIES! A serious game for teaching scientific inquiry. In: Ma M, 

Oikonomou A, Lakhmi J, editors. Serious games and edu-tainment 

applications. London, (UK): Springer-Verlag; 2011. p. 169–196.

Murray T. An overview of intelligent tutoring system authoring tools: updated 

analysis of the state of the art. In: Murray T, Blessing S, Ainsworth S, 

editors. Authoring tools for advanced technology learning 

environments. Amsterdam (Netherlands): Springer; 2003. p. 491–544.

Norman, DA. Some observations on mental models. In: Gentner D, Stevens AL,

editors. Mental Models. Hillsdale (NJ): Lawrence Erlbaum Associates, Inc.; 

1983.

Norman, DA. Cognitive engineering. In: Norman DA, Draper SW, editors. User 

centered system design: new perspectives on human-computer interaction.

Hillsdale (NJ): Lawrence Erlbaum Associates; 1986. p. 31–61.

Nuxeo Platform; c2015 [accessed 2015 Jul 7]. http://www.nuxeo.com/.

Olney A, Brawner K, Pavlik P, Keodinger KR. Emerging trends in automated 

authoring. In: Sottilare R, Graesser A, Hu X, Brawner K, editors. Design 

recommendations for intelligent tutoring systems. Orlando (FL): US Army 

Research Laboratory; 2015. (Authoring tools and expert modeling 

techniques; vol. 3). ISBN 978-0-9893923-7-2.

Olney A, D’Mello SK, Person N, Cade W, Hays P, Williams C, Graesser AC. 

Guru: a computer tutor that models expert human tutors. In: Cerri S, Clancey 

W, Papadourakis G, Panourgia K, editors. Proceedings of Intelligent Tutoring 

Systems (ITS). Berlin (Germany): Springer; 2012. p. 256–261.

Olney AM, Graesser AC, Person, NK. Question generation from concept maps. 

Dialogue & Discourse. 2012;3(2):75–99.

Ososky S. Influence of task-role mental models on human interpretation of robot 

motion behavior [dissertation]. [Orlando (FL)]: University of Central Florida; 

2013.
29

Paneva D. Use of ontology-based student model in semantic-oriented access to 

the knowledge in digital libraries. In: Proc. of HUBUSKA Fourth Open 

Workshop, Semantic Web and Knowledge Technologies Applications; 2006 

Sep; Varna, Bulgaria. p. 31–41.

Ray F, Robson R, Brawner K. 3QL: a query language for exchanging third tier 

metadata. Simulation Interoperability Standards Organization Fall 2014 

Workshop. Orlando, FL.

Regan DA. The training and learning architecture: infrastructure for the future of 

learning. Invited Keynote International Symposium on Information 

Technology and Communication in Education (SINTICE); Madrid, Spain;

2013.

Ritter S, Anderson JR, Koedinger KR, Corbett A. Cognitive tutor: applied 

research in mathematics education. Psychonomic Bulletin and Review. 

2007;14:249–255.

Rouse WB, Morris NM. On looking into the black box: prospects and limits in the 

search for mental models. Psychological Bulletin. 1986;100(3):349.

Schneider B, Wallace J, Blikstein P, Pea R. Preparing for future learning with a 

tangible user interface: the case of neuroscience. Learning Technologies, 

IEEE Transactions. 2013;6(2):117–129.

Sottilare R. Adaptive intelligent tutoring system (ITS) research in support of the 

army learning model: research outline. Aberdeen Proving Ground (MD): 

Army Research Laboratory (US); 2013 Dec. Report No.: ARL-SR-0284.

Sottilare R. Challenges in moving adaptive training and education from state-ofart to state-of-practice. In: Proceedings of Developing a Generalized 

Intelligent Framework for Tutoring (GIFT): Informing Design through a 

Community of Practice Workshop at the 17th International Conference on 

Artificial Intelligence in Education (AIED 2015). 2015 June; Madrid, Spain.

Sottilare R, Brawner K. A long-term learner model to drive optimal macroadaptive decisions by intelligent tutoring systems. Pensacola (FL): Florida 

Artificial Intelligence Research Society; 2014 May.

Sottilare RA, Brawner KW, Goldberg BS, Holden HK. The generalized intelligent 

framework for tutoring (GIFT). Orlando (FL): Army Research Laboratory 

(US), Human Research and Engineering Directorate (HRED). 2012a.
30

Sottilare R, Goldberg BS, Brawner KW, Holden HK. A modular framework to 

support the authoring and assessment of adaptive computer-based tutoring 

systems (CBTS). In: Proceedings of the Interservice/Industry Training 

Simulation and Education Conference; Dec 2012b; Orlando, FL.

Steenbergen-Hu S, Cooper H. A meta-analysis of the effectiveness of intelligent 

tutoring systems on K-12 students’ mathematical learning. Journal of 

Educational Psychology. 2013;105(4):970–987.

Steenbergen-Hu S, Cooper H. A meta-analysis of the effectiveness of intelligent 

tutoring systems on college students’ academic learning. Journal of 

Educational Psychology. 2014;106:331–347.

Stottler RH, Richards R, Spaulding B. Use cases, requirements and a prototype 

standard for an Intelligent Tutoring System (ITS)/Simulation Interoperability 

Standard (I/SIS). In: Proceedings of the SISO 2005 Spring Simulation 

Interoperability Workshop, 2005. Orlando, FL. p. 3–8.

Streitz NA. Mental models and metaphors: implications for the design of adaptive 

user-system interfaces. In: Mandl H, Lesgold A, editors. Learning issues for 

intelligent tutoring systems. New York (NY): Springer; 1988. p. 164–186.

US Army Training and Doctrine Command (TRADOC). The United States Army 

Learning Concept for 2015. Fort Monroe, VA; 2011 [accessed 2015 May].

http://www.tradoc.army.mil/tpubs/pams/tp525-8-2.pdf.

VanLehn K. The relative effectiveness of human tutoring, intelligent tutoring

systems, and other tutoring systems. Educational Psychologist. 

2011;46(4):197–221.

VanLehn K, Graesser AC, Jackson GT, Jordan P, Olney A, Rosé CP. When are 

tutorial dialogues more effective than reading? Cognitive Science. 

2007;31(1):3–62.

VanLehn K, Lynch C, Schulze K, Shapiro JA, Shelby R, Taylor L, Wintersgill M. 

The Andes physics tutoring system: lessons learned. International Journal of 

Artificial Intelligence and Education. 2005;15(3):147–204.

Veden, A. Data for enabling content in adaptive learning systems (DECALS); 

c2015 [accessed 2015 Oct 6]. https://github.com/adlnet/DECALS.

Zook A, Lee-Urban S, Riedl M, Holden H, Sottilare R, Brawner K. Automated 

scenario generation: toward tailored and optimized military training in virtual 

environments. Paper presented at the Foundations of Digital Games; 2012;

Raleigh, NC.
31

Bibliography

Anderson JR, Corbett AT, Koedinger KR, Pelletier R. Cognitive tutors: lessons 

learned. The Journal of the Learning Sciences. 1995;4:167–207.

Anderson LW, Krathwohl DR, Bloom BS. A taxonomy for learning, teaching, 

and assessing: a revision of Bloom's taxonomy of educational objectives. 

Boston (MA): Allyn and Bacon; 2001.

Atkinson RK, Renkl A, Merrill MM. Transitioning from studying examples to 

solving problems: effects of self-explanation prompts and fading worked-out 

steps. Journal of Educational Psychology. 2003;95(4):774.

Bloom BS (editor). Taxonomy of educational objectives: the classification of 

educational goals. New York (NY): David McKay Co.; 1956. Handbook I: 

cognitive domain.

Brawner K, Holden H, Goldberg B, Sottilare R. Recommendations for modern 

tools to author tutoring systems. In: Proceedings of the Interservice/Industry 

Training Simulation and Education Conference; 2012 Dec; Orlando, FL.

Burke S, Sottilare R, Salas E, Johnston J, Sinatra A, Holden, H. Towards a 

scientifically-rooted design architecture of team process and performance 

modeling in adaptive, team-based intelligent tutoring systems: methodology 

for literature review and meta-analysis, and preliminary results. Orlando 

(FL): Army Research Laboratory (US); (2015, in press). 

Clark RC, Nguyen F, Sweller, J. Efficiency in learning: evidence-based guidelines 

to manage cognitive load. New York (NY): John Wiley and Sons; 2011.

The Council of Ministers of Education, Canada (CMEC). Program for 

International Student Assessment (PISA). Toronto, Ontario (Canada); 2015. 

[accessed 2015 Sep 9]. http://www.cmec.ca/508/Programs-andInitiatives/Assessment/Programme-for-International-Student-Assessment-

(PISA)/PISA-2015/index.html.

Dempster FN. The spacing effect. American Psychologist. 1988;43:627–634.

Dillon TJ. Questioning and teaching: a manual of practice. New York (NY): 

Teachers College Press; 1988.

D’Mello SK. A selective meta-analysis on the relative incidence of discrete 

affective states during learning with technology. Journal of Educational 

Psychology. 2013;105:1082–1099.
32

D’Mello SK, Graesser AC. AutoTutor and affective AutoTutor: learning by 

talking with cognitively and emotionally intelligent computers that talk back. 

ACM Transactions on Interactive Intelligent Systems. 2012;2(4):23:1–38.

Fletcher JD, Sottilare R. Shared mental models and intelligent tutoring for teams. 

In: Sottilare R, Graesser A, Hu X, Holden H, editors. Design 

recommendations for intelligent tutoring systems: volume I – learner 

modeling. Orlando (FL): Army Research Laboratory (US); 2013. ISBN 

978-0-9893923-0-3.

Goleman D. Emotional intelligence. New York (NY): Bantam; 2006.

Graesser AC, D’Mello S. Emotions during the learning of difficult material. In: 

Ross BH, editor. Psychology of learning and motivation. Waltham (MA): 

Academic Press; 2012. Vol. 57, Ch. 5, p. 183–225. ISSN 0079-7421, ISBN 

978012-3942937, 10.1016/B978-0-12-394293-7.00005-4.

Graesser AC, McNamara DS. Self-regulated learning in learning environments 

with pedagogical agents that interact in natural language. Educational 

Psychologist. 2010;45(4):234–244.

Holden H, Sottilare R, Goldberg B, Brawner K. Effective learner modeling for 

computer-based tutoring of cognitive and affective tasks. In: Proceedings of 

the Interservice/Industry Training Simulation and Education Conference;

2012 Dec; Orlando, FL.

Hunt E, Minstrell J. A cognitive approach to the teaching of physics. In: McGilly 

K, editor. Classroom lessons. Cambridge (MA): MIT Press; 1994. p. 51–74.

Johnson D, Johnson R, Holubec EJ. Circles of learning: cooperation in the 

classroom. 3rd ed. Edina (MN): Interaction Book Company; 1990.

Kluge A, Burkolter D. Enhancing research on training for cognitive readiness 

research issues and experimental designs. Journal of Cognitive Engineering 

and Decision Making. 2013;7(1):96–118.

Koedinger KR, Corbett AC, Perfetti C. The knowledge-learning-instruction (KLI) 

framework: bridging the science-practice chasm to enhance robust student 

learning. Cognitive Science. 2012;36(5):757–798.

Kokini C, Carroll M, Ramirez-Padron R, Wang X, Hale K, Sottilare R, Goldberg

B. Quantification of trainee affective and cognitive state in real-time. In:

Proceedings of the Interservice/Industry Training Simulation and Education 

Conference; 2012 Dec; Orlando, FL.
33

Krathwohl DR, Bloom BS, Masia BB. Taxonomy of educational objectives: New 

York (NY): David McKay Co.; 1964. Handbook II: affective domain.

Lesgold AM, Lajoie S, Bunzo M, Eggan G. Sherlock: a coached practice 

environment for an electronics trouble shooting job. LRDC Report.

Pittsburgh (PA): University of Pittsburgh, Learning Research and 

Development Center; 1988.

Matthews MD. Head strong: how psychology is revolutionizing war. New York 

(NY): Oxford University Press; 2014. p. 204. 

Murray T. Authoring intelligent tutoring systems: an analysis of the state of the 

art. International Journal of Artificial Intelligence in Education. 

1999;10(1):98–129. 

Murray T. Theory-based authoring tool design: considering the complexity of 

tasks and mental models. In: Sottilare R, Graesser A, Hu X, Brawner K,

editors. Design recommendations for intelligent tutoring systems: volume 3 –

authoring tools and methods. Orlando (FL): Army Research Laboratory 

(US); 2015. ISBN: 978-0-9893923-7-2. 

Pekrun R. The control-value theory of achievement emotions: assumptions, 

corollaries, and implications for educational research and practice. 

Educational Psychology Review. 2006;18:315–341.

Rouse WB, Cannon-Bowers JA, Salas E. The role of mental models in team 

performance in complex systems. Systems, Man and Cybernetics, IEEE 

Transactions on. 1992;22(6):1296–1308.

Salas EE, Fiore SM. Team cognition: understanding the factors that drive process 

and performance. Washington, DC: American Psychological Association;

2004. ISBN 978-1-59147-103-5.

Salas E, Prince C, Baker DP, Shrestha L. Situation awareness in team 

performance: implications for measurement and training. Human Factors: 

The Journal of the Human Factors and Ergonomics Society. 1995;37(1):123–

136.

Shute V, Ventura M, Small M, Goldberg B. Modeling student competencies in 

video games using stealth assessment. In: Sottilare R, Graesser A, Hu X, 

Holden H, editors. Design recommendations for intelligent tutoring systems: 

volume I – learner modeling. Orlando (FL): Army Research Laboratory 

(US); 2013. ISBN 978-0-9893923-0-3.
34

Simpson E. The classification of educational objectives in the psychomotor 

domain. The psychomotor domain, vol. 3; Washington (DC): Gryphon 

House; 1972.

Soller A. Supporting social interaction in an intelligent collaborative learning 

system. International Journal of Artificial Intelligence in Education. 

2001;12(1):40–62.

Sottilare R. Considerations in the development of an ontology for a generalized 

intelligent framework for tutoring. International Defense and Homeland 

Security Simulation Workshop in Proceedings of the I3M Conference; 2012

Sep; Vienna, Austria.

Sottilare R, Gilbert S. Considerations for tutoring, cognitive modeling, authoring 

and interaction design in serious games. Authoring Simulation and Gamebased Intelligent Tutoring Workshop at the Artificial Intelligence in 

Education Conference (AIED); 2011 June; Auckland, New Zealand.

Sottilare R, Goldberg B. Designing adaptive computer-based tutors to accelerate 

learning and facilitate retention. Journal of Cognitive Technology: 

Contributions of Cognitive Technology to Accelerated Learning and 

Expertise. 2012;17(1):19–34.

Sottilare R, Holden H, Brawner K, Goldberg, B. Challenges and emerging 

concepts in the development of adaptive, computer-based tutoring systems 

for team training. In: Proceedings of the Interservice/Industry Training 

Simulation and Education Conference; 2011 Dec; Orlando, FL. 

Sottilare R, Marshall L, Martin R, Morgan J. Injecting realistic human models 

into the optical display of a future land warrior system for embedded training 

purposes. Journal for Defense Modeling and Simulation. 2007 Apr;4(2).

Sottilare R, Ragusa C, Hoffman M, Goldberg B. Characterizing an adaptive 

tutoring learning effect chain for individual and team tutoring. In:

Proceedings of the Interservice/Industry Training Systems and Education 

Conference; 2013 Dec; Orlando, FL.
35

List of Symbols, Abbreviations, and Acronyms

AAR after-action review

AI artificial intelligence

ALM US Army Learning Model

ARL US Army Research Laboratory

CBT computer-based training

CBTS computer-based tutoring systems

CMS Content Management System

GAMETE Game-based Architecture for Mentor-Enhanced Training

Environments

GIFT Generalized Intelligent Framework for Tutoring

ITS intelligent tutoring system

MOS military occupational specialty

RLO reusable learning object

SRL self-regulated learning

WFO Warfighter Outcome

XML eXtensible Markup Language
36

1 DEFENSE TECHNICAL

(PDF) INFORMATION CTR

DTIC OCA

2 DIRECTOR

(PDF) US ARMY RESEARCH LAB

RDRL CIO LL

IMAL HRA M&R MGMT

1 GOVT PRINTG OFC

(PDF) A MALHOTRA

1 ARMY RSCH LAB – HRED

(PDF) RDRL HRM D

T DAVIS

BLDG 5400 RM C242

REDSTONE ARSENAL AL 

35898-7290

1 ARMY RSCH LAB – HRED

(PDF) RDRL HRS EA DR V J RICE

BLDG 4011 RM 217

1750 GREELEY RD

FORT SAM HOUSTON TX 

78234-5002

1 ARMY RSCH LAB – HRED

(PDF) RDRL HRM DG K GUNN

BLDG 333

PICATINNY ARSENAL NJ

07806-5000

1 ARMY RSCH LAB – HRED

(PDF) ARMC FIELD ELEMENT

RDRL HRM CH C BURNS

THIRD AVE 

BLDG 1467B RM 336

FORT KNOX KY 40121

1 ARMY RSCH LAB – HRED

(PDF) AWC FIELD ELEMENT

RDRL HRM DJ D DURBIN

BLDG 4506 (DCD) RM 107

FORT RUCKER AL 36362-5000 

1 ARMY RSCH LAB – HRED

(PDF) RDRL HRM CK J REINHART

10125 KINGMAN RD BLDG 317

FORT BELVOIR VA 22060-5828

1 ARMY RSCH LAB – HRED

(PDF) RDRL HRM AY M BARNES

2520 HEALY AVE 

STE 1172 BLDG 51005

FORT HUACHUCA AZ 

85613-7069

1 ARMY RSCH LAB – HRED

(PDF) RDRL HRM AP D UNGVARSKY

POPE HALL BLDG 470 

BCBL 806 HARRISON DR

FORT LEAVENWORTH KS 

66027-2302

1 ARMY RSCH LAB – HRED

(PDF) RDRL HRM AR J CHEN

12423 RESEARCH PKWY

ORLANDO FL 32826-3276

1 ARMY RSCH LAB – HRED

(PDF) HUMAN SYSTEMS

INTEGRATION ENGR

TACOM FIELD ELEMENT

RDRL HRM CU P MUNYA

6501 E 11 MILE RD 

MS 284 BLDG 200A 

WARREN MI 48397-5000

1 ARMY RSCH LAB – HRED

(PDF) FIRES CTR OF EXCELLENCE 

FIELD ELEMENT

RDRL HRM AF C HERNANDEZ

3040 NW AUSTIN RD RM 221

FORT SILL OK 73503-9043

1 ARMY RSCH LAB – HRED

(PDF) RDRL HRM AV W CULBERTSON

91012 STATION AVE 

FORT HOOD TX 76544-5073

1 ARMY RSCH LAB – HRED

(PDF) RDRL HRM DE A MARES

1733 PLEASONTON RD BOX 3

FORT BLISS TX 79916-6816

8 ARMY RSCH LAB – HRED

(PDF) SIMULATION & TRAINING

TECHNOLOGY CENTER

RDRL HRT COL G LAASE

RDRL HRT I MARTINEZ

RDRL HRT T R SOTTILARE

RDRL HRT B N FINKELSTEIN

RDRL HRT G A RODRIGUEZ

RDRL HRT I J HART

RDRL HRT M C METEVIER

RDRL HRT S B PETTIT

12423 RESEARCH PARKWAY

ORLANDO FL 32826
37

1 ARMY RSCH LAB – HRED

(PDF) HQ USASOC

RDRL HRM CN R SPENCER

BLDG E2929 DESERT STORM DR

FORT BRAGG NC 28310

1 ARMY G1

(PDF) DAPE MR B KNAPP

300 ARMY PENTAGON

RM 2C489

WASHINGTON DC 20310-0300

ABERDEEN PROVING GROUND

12 DIR USARL

(PDF) RDRL HR

L ALLENDER

P FRANASZCZUK

K MCDOWELL

RDRL HRM

P SAVAGE-KNEPSHIELD

RDRL HRM AL

C PAULILLO

RDRL HRM B

J GRYNOVICKI

RDRL HRM C

L GARRETT

RDRL HRS

J LOCKETT

RDRL HRS B

M LAFIANDRA

RDRL HRS D

A SCHARINE

RDRL HRS E

D HEADLEY

RDRL HRT

S OSOSKY
38

INTENTIONALLY LEFT BLANK.