ARL-SR-0345 ● NOV 2015

US Army Research Laboratory

Instructional Management for Adaptive 

Training and Education in Support of the 

US Army Learning Model–Research Outline

by Benjamin Goldberg, Anne Sinatra, Robert Sottilare, 

Jason Moss, and Arthur Graesser

Approved for public release; distribution is unlimited.
NOTICES

Disclaimers

The findings in this report are not to be construed as an official Department of the 

Army position unless so designated by other authorized documents.

Citation of manufacturer’s or trade names does not constitute an official 

endorsement or approval of the use thereof.

Destroy this report when it is no longer needed. Do not return it to the originator.
ARL-SR-0345 ● NOV 2015

US Army Research Laboratory

Instructional Management for Adaptive 

Training and Education in Support of the 

US Army Learning Model–Research Outline

by Benjamin Goldberg, Anne Sinatra, Robert Sottilare, and 

Jason Moss

Human Research and Engineering Directorate, ARL

Arthur Graesser

University of Memphis Institute for Intelligent Systems, Memphis, TN

Approved for public release; distribution is unlimited.

FOR OFFICIAL USE ONLY (delete if not FOUO)
ii

REPORT DOCUMENTATION PAGE Form Approved

OMB No. 0704-0188

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the 

data needed, and completing and reviewing the collection information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing the 

burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302. 

Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently valid 

OMB control number.

PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS.

1. REPORT DATE (DD-MM-YYYY)

November 2015

2. REPORT TYPE

Special

3. DATES COVERED (From - To)

1 October 2014–15 July 2015

4. TITLE AND SUBTITLE

Instructional Management for Adaptive Training and Education in Support of the 

US Army Learning Model–Research Outline

5a. CONTRACT NUMBER

5b. GRANT NUMBER

5c. PROGRAM ELEMENT NUMBER

6. AUTHOR(S)

Benjamin Goldberg, Anne Sinatra, Robert Sottilare, Jason Moss, and Arthur 

Graesser

5d. PROJECT NUMBER

5e. TASK NUMBER

5f. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

US Army Research Laboratory

ATTN: RDRL-HRT-T

Aberdeen Proving Ground, MD 21005-5425

8. PERFORMING ORGANIZATION REPORT NUMBER

ARL-SR-0345

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR'S ACRONYM(S)

11. SPONSOR/MONITOR'S REPORT NUMBER(S)

12. DISTRIBUTION/AVAILABILITY STATEMENT

Approved for public release; distribution is unlimited.

13. SUPPLEMENTARY NOTES

14. ABSTRACT

While human tutoring and mentoring are common teaching tools, current US Army standards for training and education are 

group instruction and classroom training, also known as one-to-many instruction. Recently, the US Army has placed 

significant emphasis on self-regulated learning (SRL) methods to augment institutional training where Soldiers will be largely 

responsible for managing their own learning. In support of the US Army Learning Model and to provide affordable, tailored 

SRL training and educational capabilities for the US Army, the US Army Research Laboratory is investigating and developing 

adaptive tools and methods to largely automate the authoring (creation), delivery of instruction, and evaluation of computerregulated training and education capabilities. A major goal within this research program is to reduce the time and skill 

required to author, deliver, and evaluate adaptive technologies to make them usable by a larger segment of the training and 

educational community. This research includes 6 interdependent research vectors: individual learner and unit modeling, 

instructional management principles, domain modeling, authoring tools and methods, evaluation tools and methods, and 

architectural and ontological support. This report (1 of 6 interdependent research outlines) focuses on instructional 

management research for adaptive training and education with the goal of guiding learning in militarily relevant training 

domains.

15. SUBJECT TERMS

GIFT, design, adaptive training, domain modeling, SRL

16. SECURITY CLASSIFICATION OF:

17. LIMITATION

OF ABSTRACT

UU

18. NUMBER

OF PAGES

58

19a. NAME OF RESPONSIBLE PERSON

Robert Sottilare

a. REPORT

Unclassified

b. ABSTRACT

Unclassified

c. THIS PAGE

Unclassified

19b. TELEPHONE NUMBER (Include area code)

407-208-3007

Standard Form 298 (Rev. 8/98)

Prescribed by ANSI Std. Z39.18
iii

Contents

List of Figures v

Preface vi

1. Introduction 1

2. Research Goals and Objectives 2

3. Background 3

3.1 Self-Regulated Learning and the US Army Learning Model 3

3.2 Motivation for Research 4

3.3 Adaptive Training and Education Definitions 5

4. US Army Requirements for Adaptive Training Systems and 

Instructional Management Research 8

4.1 Adaptive Training and Education Systems and Instructional 

Management 8

4.2 Big Data and Instructional Management 9

4.3 Training at the Point of Need and Instructional Management 9

4.4 Artificial Intelligence Capabilities and Instructional Management 10

5. Understanding the Dimensions of Instructional Management 11

5.1 Theoretical Foundations of Instructional Management 12

5.2 Instructional Management in GIFT 14

6. Instructional Management Research Goals and Challenges 17

6.1 Guidance and Scaffolding 17

6.2 Social Dynamics and Virtual Humans 21

6.3 Metacognition and Self-Regulated Learning 23

6.4 Optimizing the Selection of Tactics 26

6.5 Personalization (Occupational and Noncognitive Factors) 27
iv

7. Interdependencies with Other Adaptive Training Research 

Vectors 30

7.1 Learner Modeling and Instructional Management 31

7.2 Domain Modeling and Instructional Management 32

7.3 Authoring Tools and Instructional Management 32

7.4 Evaluation and Instructional Management 33

7.5 Architecture and Instructional Management 34

8. Conclusions 34

9. References 36

Bibliography 44

List of Symbols, Abbreviations, and Acronyms 47

Distribution List 48
v

List of Figures

Fig. 1 Adaptive training interaction ...............................................................12

Fig. 2 Updated individual learning effect model............................................14

Fig. 3 GIFT’s EMAP......................................................................................16

Fig. 4 Team learning effect model .................................................................26

Fig. 5 Adaptive training research vectors.......................................................30
vi

Preface 

This report is 1 of 6 interdependent research outlines in the Adaptive Training 

research program. Portions of this text, which originated in ARL-SR-0325,1 appear 

in all 6 reports to ensure that readers get the same cross-cutting information. 

 1 Sottilare R, Sinatra A, Boyce M, Graesser A. Domain modeling for adaptive training and education in 

support of the US army learning model–research outline. Aberdeen Proving Ground (MD): Army Research 

Laboratory (US); June 2015. Report No.: ARL-SR-0325.
1

1. Introduction

Training and education tools and methods must be of sufficient intelligence to 

understand the needs of individual learners and units of learners to mitigate 

negative learner states and to guide and tailor instruction in real time to optimize 

learning. These tools and methods must also be affordable, effective, and easy to 

access and use. These requirements are enablers of the US Army Learning Model 

(ALM), which includes an emphasis on self-regulated learning (SRL), where 

Soldiers are expected to manage their own learning and career development through 

the growth of metacognitive (e.g., reflection), self-assessment, and motivational 

skills (Butler and Winne 1995). While SRL skills are difficult to train and develop, 

support may be provided to the learner through “adaptive training technologies” 

(tools and methods) that may be focused to guide learning and reinforce SRL 

principles. 

To support ALM, the US Army Research Laboratory (ARL) has developed a 

program of research called “adaptive training” that includes 6 interdependent 

research areas or vectors: individual learner and unit modeling, instructional 

management principles, domain modeling, authoring tools and methods, evaluation 

tools and methods, and architectural and ontological support for adaptive training. 

The reports documenting these vectors expand the scope of the adaptive tutoring 

research described in ARL-SR-0284 (Sottilare 2013) to support ALM requirements 

in the mid- and long-term evolution of training and educational technology: the 

Synthetic Training Environment and the Future Holistic Training Environment for 

Live and Synthetic.

This report (1 of 6 interdependent research outlines) focuses on instructional 

management research for adaptive training and education. An overarching goal of 

instructional management is to leverage prior research in the learning sciences and 

artificial intelligence (AI) communities (both theoretical and empirical) to establish 

a set of best practices on how to author and execute techniques, strategies, and 

tactics across any domain of instruction. This is a very large research space (WangCostello et al. 2013), so it is important to identify the primary themes for applied

instructional management practices.

Today the majority of intelligent tutoring systems (ITSs), a form of adaptive 

training tool to support one-to-one computer-based instruction, supports welldefined domains in mathematics, physics, and software programming. Since 

Soldiers operate in more-complex, more-dynamic, and more-ill-defined domains, 

it is necessary to expand the scope of adaptive training tools and methods to support 

training and education in these militarily relevant task environments. Instructional 
2

management in adaptive training environments represents pedagogical logic and 

policies that dictate what is experienced and what assistance is provided. The goal 

is to align instructional management techniques with empirical research that 

informs what strategies have the greatest influence on performance, retention, and 

transfer. Effective instructional management research will dictate how best to 

instruct a particular task or set of concepts for a given type of learner that promotes 

the development of knowledge and skill across any domain of instruction. 

2. Research Goals and Objectives

The goal of the research described in this report is to model militarily relevant 

training domains to support individually tailored and intelligently guided training 

experiences as prescribed by the ALM (US Army Training and Doctrine Command 

2011). The research provides guidelines, best practices, tools, models, and methods 

in support of this research goal. More specifically, we intend to accomplish the 

following:

• Discover and deliver personalized and adaptive training experiences that are 

informed by experimentation or prior research on tutoring methods and 

techniques that are moderated by individual differences of a learner, or team 

of learners, to promote efficient knowledge and skill acquisition across 

technology-driven training applications.

• Research, design, and develop prototype authoring tools grounded in 

learning and instructional theory and informed by empirical research to 

assist training managers, developers, and subject matter experts (SMEs) in 

building pedagogically sound training experiences without the requirement 

of programming.

• Discover and develop modeling functions that account for uncertainty 

across policies informing pedagogical decisions (e.g., content delivery, 

course navigation, and guidance) with an ability for these functions to refine 

and optimize themselves through reinforcement learning mechanisms over 

time as new interaction and performance data becomes available (e.g., 

Markov Decision Processes).

In this research outline, we present the goals associated with the instructional 

management research vector and review specific subvectors across the program that 

influence the approach and direction of our dedicated efforts. This outline 

emphasizes desired functions and capabilities for ARL’s Generalized Intelligent 

Framework for Tutoring (GIFT), an open-source architecture intended to support 

adaptive instruction.
3

3. Background

While human tutoring and mentoring are common teaching tools, current US Army 

standards for training and education are group instruction and classroom training, 

also known as one-to-many instruction. Group instruction and classroom training 

have been generally focused on acquiring and applying knowledge in proxies for 

live training environments (e.g., desktop simulations, virtual simulations, 

constructive simulations, and serious games). 

Classroom training, especially for complex topics, is often taught as a series of lists 

that the instructor goes through in a linear fashion (Schneider et al. 2013). This 

approach puts a heavy burden on the learner to build mental models and make 

conceptual connections. Using this instructional methodology may lead to varying 

degrees of success due to individual differences in skills, traits, and/or preferences. 

In this fashion, lower-performing students who struggle are provided more handson guidance while higher-performing students who are under-challenged (e.g., 

bored) are allowed to progress through a course at an accelerated rate.

Small group instruction in live environments has also been used to assess 

application of knowledge and the development of skills. A standard feedback 

mechanism for US Army training is the after-action review (AAR), where 

significant decision points and actions are captured for small group discussions

conducted after the completion of a training event to help capture teachable 

moments and to aid Soldiers in reflecting on their recent training experiences. 

Both classroom training and small group instruction are manpower-intensive,

requiring teachers, mentors, and support staff to guide the Soldier’s experience. 

Today ITSs guide learner training and education primarily in physically static 

(e.g., desktop simulations), well-defined domains. Research is needed to expand 

ITS instructional management practices to support militarily relevant domains that 

train across knowledge, skills, and abilities required to be successful Soldiers. The 

focus of this research is to identify pedagogical principles that can be applied to 

manage knowledge and skill acquisition by addressing individual differences, 

managing affective states experienced during learning, supporting metacognitive 

development, enforcing reflection through personalized AARs, and promoting 

retention and transfer to operational environments. 

3.1 Self-Regulated Learning and the US Army Learning Model

In 2011 the US Army placed significant emphasis on the development of SRL skills 

with the expectation that new methods of instruction (e.g., ITSs) would augment 

institutional training (i.e., classroom and small group instruction).
4

One-to-one human tutoring has been shown to be significantly more effective than 

one-to-many instructional methods (e.g., traditional classroom instruction) (Bloom 

1984; VanLehn 2011). However, it is neither practical nor affordable to have one

expert human tutor to mentor each Soldier in the US Army for every required 

operational task. This alone signals the need for capabilities to support one-to-one, 

tailored training and educational experiences. 

Additionally, under the ALM, Soldiers are largely responsible for managing their 

own learning, but SRL skills are difficult to train and develop (Azevedo et al. 2009; 

Butler and Winne 1995; Graesser and McNamara 2010). We anticipate adaptive 

training tools and methods will fill this gap and provide personalized guidance to 

acquire, apply, retain, and transfer knowledge and skills to the operational 

environment. This signals the need for a computer-regulated learning strategy to 

augment missing SRL skills; however, adaptive training technologies must first 

become affordable, sufficiently adaptive, and easy to use for this strategy to be 

realized. 

3.2 Motivation for Research

A promising alternative to one-to-one human tutoring is one-to-one adaptive 

training tools that include ITSs. Meta-analyses and reviews support the claim that 

ITS technologies routinely improve learning over classroom teaching, reading 

texts, and/or other traditional learning methods. These meta-analyses normally 

report effect sizes (sigma [σ]), which refer to the difference between the ITS 

condition and a control condition in standard deviation units. The reported metaanalyses show positive effect sizes that vary from σ = 0.05 (Dynarsky et al. 2007) 

to σ = 1.08 (Dodds and Fletcher 2004) but most hover between σ = 0.40 and 

σ = 0.80 (Fletcher 2003; Graesser et al. 2012; Ma et al. in press; Steenbergen-Hu 

and Cooper 2013, 2014; VanLehn 2011). Our current best meta-meta estimate from 

all of these meta-analyses is σ = 0.60. This performance is comparable to human 

tutoring, which varies from between σ = 0.20 and σ = 1.00 (Cohen et al. 1982; 

Graesser et al. 2011) depending on the expertise of the tutor. Human tutors have 

not varied greatly from ITSs in direct comparisons between ITS and trained human 

tutors (Olney et al. 2012; VanLehn 2011; VanLehn et al. 2007). 

Graesser et al. (2015 in press) are convinced that some subject matters will show 

higher effect sizes than others when comparing any intervention (e.g., computer 

trainers, human tutors, group learning) with a control. It is difficult to obtain high

effect sizes for literacy and numeracy because these skills are ubiquitous in 

everyday life and habits are automatized. For example, Ritter et al. (2007) reported 

that Cognitive Tutor software for mathematics has shown an effect size of σ = 0.30
5

to 0.40 in environments with minimal control over instructors. Human interventions 

to improve basic reading skills typically report an effect size of σ = 0.20. In contrast, 

when the student starts essentially from ground zero, such as many subject matters 

in science and technology, effect sizes are expected to be more robust. ITSs show 

effect sizes of σ = 0.60 to 2.00 in the subject matters of physics (VanLehn 2011; 

VanLehn et al. 2005), computer literacy (Graesser et al. 2012; Graesser et al. 2004), 

biology (Olney et al. 2012), and scientific reasoning (Halpern et al. 2012; Millis et 

al. 2011). As a notable example, Digital Tutor software (Fletcher and Morrison 

2012) improves information technology by an effect size as high as σ = 3.70 for 

knowledge and σ = 1.10 for skills. The effect sizes attributed to improved 

instruction and improved domain knowledge have not been separated in this 

analysis. Such large effect sizes would never be expected in basic literacy and 

numeracy.

Overall, these are promising results and equate to an increase of about a letter grade 

improvement over traditional classroom instruction. While ITSs are a promising 

technology to support adaptive training for individuals in well-defined domains like 

mathematics, physics, and computer programming, the US Army requires the 

ability to develop and exercise Soldier skills in more ill-defined domains (e.g., 

leadership) and at the unit level (e.g., collaborative learning and team training). 

Developing and maintaining the ability to make effective decisions under stress and 

in complex environments is also desirable. 

Adaptive systems by their nature require additional content and complexity to 

support tailored learning for each user and as a consequence have a very high 

development cost, a major barrier to adoption by the US Army. Adaptive systems 

are also insufficiently adaptive to support tailored, self-regulated training and 

educational experiences across a broad spectrum of military tasks as required by 

the ALM. Today, few ITS authoring tools are generalized across all of the domains 

requiring training, and no evaluation criteria or standards have been developed to 

promote reuse and interoperability among ITSs (Sottilare et al. 2012b). In other 

words, current adaptive systems are not yet intelligent enough to support the 

tailored instruction required by the US Army in the breadth of domains being 

trained, but there is a stable foundation of 50 years of science on which to grow an 

adaptive training and education capability for the US Army.

3.3 Adaptive Training and Education Definitions

In support of the ALM and affordable adaptive training and educational capabilities 

for the US Army, ARL is investigating and developing adaptive tools and methods. 

A desired end-state is the automation of authoring (creation) processes, instruction,
6

and evaluation of computer-regulated training and education capabilities to help 

build SRL skills and support mixed-initiative interaction. A major goal within this 

research program is to reduce the time/cost and knowledge/skill required to author, 

deliver, and evaluate adaptive technologies to make them usable by a larger 

segment of the Army training and educational community.

Adaptive training and education research includes elements of adaptive tutoring, 

distributed learning, virtual humans, and training effectiveness evaluation. For 

additional detail on research specific to ITSs, refer to ARL-SR-0284 (Sottilare 

2013). The following definitions are provided for this section to distinguish 

between adaptive training and education elements and also to highlight their 

relationships:

Adaptive Tutoring: Also known as intelligent tutoring, these tailored instructional 

methods provide one-to-one and one-to-many computer-guided experiences 

focused on optimizing learning, comprehension, performance, retention, reasoning, 

and transfer of knowledge and acquired skills to the operational environment.

Adaptive Tutoring Systems: Also known as ITSs, these mechanisms or 

technologies (tools and methods) provide tailored training and educational 

experiences. Adaptive tutoring systems respond to changing states in the learner 

and changing conditions in the training environment to optimize learning and

anticipate and recognize teachable moments.

Virtual Humans: Artificially intelligent visual representations of people that 

simulate or emulate cognitive, affective, physical, and social processes.

Distributed Learning: Concurrent distribution of training and educational content 

to multiple users at the point of need in which content is intelligently selected to 

support learning, increased performance, and long-term competency in selected 

domains.

Training/Learning Effectiveness: Evaluation of the impact of training and 

educational tools and methods on usability, learning, comprehension, performance, 

retention, reasoning, and transfer of knowledge and acquired skills to the 

operational environment.

Adaptive Training and Education Systems: A convergence of ITSs and external 

training and education capabilities (e.g., serious games, virtual humans, and

simulations) to support engaging experiences with reduced need for authoring 

(Sottilare 2015).

Generalized Intelligent Framework for Tutoring (GIFT) (Sottilare et al. 2012a; 

Sottilare et al. 2013a): An open-source, modular architecture whose goals are to
7

reduce the cost and skill required for authoring adaptive training and educational 

systems, automate instructional delivery and management, and develop and 

standardize tools for the evaluation of adaptive training and educational 

technologies.

Adaptive training and education research at ARL are being conducted across 6 

interdependent research vectors: individual learner and unit modeling, instructional 

management principles, domain modeling, authoring tools and methods, evaluation 

tools and methods, and architectural and ontological support. This report (1 of 6 

interdependent research outlines) focuses on domain modeling research for 

adaptive training systems with the goal of guiding learning in militarily relevant 

training and educational domains. 

Soldiers operate in a variety of complex, dynamic, ill-defined domains where their 

ability to persevere in the face of adversity, adapt to their situation, collaborate, and 

think critically are key to the successful completion of their assigned missions. To 

develop and exercise these skills, it is paramount for Soldiers to train in challenging 

environments. Currently these few challenging training environments have been 

largely provided through manpower-intensive methods or systems with little ability 

to adapt instruction to support their learning needs. To illustrate this point, Franke 

(2011) asserts that through the use of case study examples, instruction can provide 

the pedagogical foundation for decision making under uncertainty. However, this 

approach is limited in implementation by the expanse of potential cases that would 

need to be consistently updated and maintained to support large populations like 

the US Army. 

As noted previously, adaptive systems like ITSs have been shown to be effective 

in promoting learning in primarily static (e.g., learners seated at desktop computers) 

instructional settings within relatively simple, well-defined domains (e.g., 

mathematics and physics) for individual learners. For our purposes, static 

instruction includes cognitive, affective, or social training tasks where a desktop 

computer delivers instruction and where the physical movement of the learner is 

limited to activities that can be conducted while seated. For example, static 

instruction can effectively support cognitive tasks involving decision making and 

problem solving but are less effective for training tasks involving motion and 

perception (e.g., land navigation and marksmanship). Ideally, we desire portable 

adaptive instructional capabilities to go with Soldiers to support training and 

education at their point of need across a wide spectrum of Army operational tasks. 

Research is needed to develop tools and methods to support broader domain 

modeling that is representative of the full spectrum of Army operational tasks. 

Standards, interoperability, and automation (e.g., automated scenario generation) 

(Zook et al. 2012) will likely play a significant role in making adaptive training 
8

practical. In this way, adaptive training technologies will have the greatest impact 

on organizational learning in the Army.

4. US Army Requirements for Adaptive Training Systems and 

Instructional Management Research

The Army science and technology community uses Warfighter Outcomes (WFOs) 

as the authoritative source for identifying Warfighter needs. WFOs are used to share 

research and future technology solutions. In the training and education (TandE) 

domain, the adaptive TandE research program is targeting 4 specific requirements 

to support the evolution of US Army training: adaptive training and education 

systems, big data, training at the point of need, and AI. 

4.1 Adaptive Training and Education Systems and Instructional 

Management

The primary gap to be addressed under this Army requirement is the lack of 

adaptive systems (e.g., intelligent tutors) to support individual and collective (team 

or unit) training. The Army needs an adaptive training and education capability that 

is persistent and easy to use/access with minimal start-up time. There are also 

requirements to automate an informal AAR (also known as postexercise critique) 

to reduce the time and skills needed to produce the AAR and improve its focus and 

quality. Another line of thought also notes that the AI in ITSs could be used to 

facilitate rapid mission planning and course-of-action analyses as a job aid in 

operational contexts. 

An effective adaptive training capability depends on sound instructional 

management practices. Instructional management practices consist of techniques, 

strategies, and tactics applied to a domain of instruction to optimize performance 

outcomes (Sottilare et al. 2014). With respect to GIFT, this vector of research is 

concerned with identifying instructional best practices that associate with all facets 

of learning as well as establishing authoring workflows that instantiate those 

practices across multiple environments of instruction. The goal is to provide 

adaptive training solutions that are sufficiently tailored for each individual Soldier 

and for teams of Soldiers.
9

4.2 Big Data and Instructional Management

The primary gap to be addressed under this Army requirement is that there is a lack 

of capability to handle and process large amounts of structured and unstructured 

data (also referred to as big data). One capability needed is a structured data 

analytics program linking individual data (e.g., achievements) to required longterm competencies in military occupational specialties (MOSs). This would allow 

Soldiers to understand where they rank in terms of experiences and achievements 

among other Soldiers in their MOS. It would also allow the Army to identify 

specific experiences among successful Soldiers in that MOS and provide a model 

for other Soldiers in that MOS to follow. This data could also be used by course 

managers and instructors to continuously improve instruction and the mental 

models of both human and computer-based instructors. Finally, data collected on 

trainee learning and performance during adaptive training experiences could be 

used to facilitate unit training management where unit commanders would have 

access to empirical data to support unit training decisions.

A goal of instructional management in GIFT is to optimize performance and 

competency outcomes through personalized training experiences that adapt to an 

individual’s knowledge, skills, and abilities (KSAs). A connection between big data 

and instructional management is applying large datasets from prior course 

interactions to update and improve technique and strategy implementations across 

all available courses. A challenge in defining instructional management logic in a 

domain-independent context is that it requires generalizability across applications. 

An issue with instructional strategy-based research is that an approach taken in one 

domain is hard to translate to a different without performing extensive research to 

validate its application. In addition, it is difficult to define definitive instructional 

management logic based on these uncertainties. As such, big data can be used to 

account for this uncertainty by applying machine learning and data mining 

techniques to assess specific causal relationships between instructional practices 

and outcomes on performance, retention, and transfer. This application of big data 

is used to reinforce instructional management models by optimizing itself over time 

as more and more data is made available.

4.3 Training at the Point of Need and Instructional 

Management

The primary gap to be addressed under this Army requirement is the lack of an 

easily accessible, persistent, cost-effective, and low-overhead training 

environment. A capability is needed to bring training to Soldiers instead of Soldiers 

going to fixed training locations. This point-of-need training capability would be
10

easily distributed, web-based, and built upon open-enterprise architecture in the 

cloud. Army training and educational opportunities would be available on demand 

anywhere and anytime. However, the delivery mechanism (e.g., laptop computer, 

mobile device, and smart glasses) for adaptive training is critical in determining the 

limitations of the domain model scope and complexity. For example, it may be 

extremely difficult to train all the complexities of a psychomotor task in a desktop 

computer setting.

A goal of instructional management in GIFT is to support a model of training by 

convenience and to have mechanisms to support this form of education in the 

absence of live instruction. Effective training applications administered in a pointof-need capacity requires 4 primary components to deliver sound instruction: 1) the 

ability to monitor trainee interaction and behavior to accurately assess performance 

against a granular concept by concept model, 2) the ability to communicate 

guidance and feedback messages that correspond with individual performance, 3) 

the ability to adapt scenario/problem elements to maintain appropriate challenge 

levels for promoting flow, and 4) the ability to manage an automated AAR to 

review scenario performance and promote reflection on the linkage between 

outcomes and overall learning objectives. Each of these components depend on 

each other for the purpose of delivering personalized training experiences through 

an easily distributed, web-based, open-enterprise architecture. In addition, an 

optimal adaptive training capability will leverage existing course materials to better 

serve the development of a competency through point-of-need functions. An 

example is providing a remedial training activity on an identified weakness 

between training events to better prepare that individual for the next period of 

instruction (e.g., recognizing trigger squeeze problems in the Engagement Skills 

Trainer and providing a remedial multimedia training event prior to the next round 

of marksmanship instruction on the live range). 

4.4 Artificial Intelligence Capabilities and Instructional 

Management

The primary gap to be addressed under this Army requirement is that the Army 

lacks an automated capability to replicate the complexity and uncertainty of the 

operational environment. This gap specifically points to the lack of adaptiveness in 

virtual humans, intelligent tutoring systems, and other training capabilities. This 

gap leads to Soldiers developing training response strategies that result in lesschallenging training over time along with lower engagement and lower levels of 

learning and transfer of skills to more challenging operational environments. 
11

A goal of instructional management in GIFT is to facilitate robust AI capabilities 

that enhance the realism and playability of simulation-based training exercises,

specifically those that utilize virtual humans and semi-automated forces. These 

elements must be embedded with AI capabilities that provide an automated reactive 

capacity to trainee inputs and actions that adhere to the complexity and uncertainty 

of an operational environment. In terms of virtual humans, these entities must have 

logic that supports realistic movements and communication exchanges that reflect 

back to a culture or operational environment. This requires AI embedded within 

virtual humans that accounts for cultural norms and customs, along with the ability 

for the entity to adapt its behavior based on cues and actions perceived from the 

trainee (rolling of the eyes, change in vocal intonation, failing to account for 

appropriate cultural greeting, etc.). AI also needs to be embedded in virtual humans 

that enables their use as virtual teammates in a collaborative training scenario. 

This will allow for effective training of team-oriented missions without the 

requirement of using an all-human team. In terms of semi-automated forces, which 

are commonly used in tactical training events within environments such as Virtual 

Battlespace 3 (VBS3), AI techniques must be established that allow a group of 

forces to adapt their movements over time, as it can autonomously learn from 

actions and tactics executed by a trainee or team of trainees. This will allow a set 

of enemy forces to better adapt itself, much like in the real world, for the purpose 

of creating richer training experiences that increase complexity. In addition, AI 

techniques must be investigated to ease the development of training scenarios to 

avoid issues of “replay-ability”. A system such as VBS3 would benefit from 

technologies that enable the generation of multiple scenarios for training purposes 

based solely around a defined set of tasks, conditions, and standards. This promotes 

better training because a trainee needs to adapt his/her application of knowledge 

and skills to an unknown event rather than gaming a scenario by learning the 

various cues and scripts executed by an enemy entity (i.e., knowing an insurgent is 

hiding behind a specific door in a specific hallway). 

5. Understanding the Dimensions of Instructional 

Management

There are 4 typical elements that comprise ITSs, a prime example of an adaptive 

training and education system: a learner or trainee model, an instructional or 

pedagogical model, a domain model, and some type of user interface. The domain 

model typically includes an expert or ideal student model by which the adaptive 

system measures/compares/contrasts the progress of the learner toward learning 

objectives. The domain model also includes the training environment, the training 

task, and all of the associated instructional actions (e.g., feedback, questions, hints,
12

pumps, and prompts) that could possibly be delivered by the adaptive system for 

that particular training domain. Typical interaction between the learner, the training 

environment, and the adaptive system (tutoring agents) is shown in Fig. 1.

Fig. 1 Adaptive training interaction

Typical training systems examine the interaction between the learner and the 

training environment to measure progress toward learning objectives. The learner 

acts on the environment (e.g., opens a door or makes a choice to move into the room 

or stay outside) and then observes any changes or reactions within the environment. 

Adaptive systems add a layer of software-based tutoring agents that are designed 

to guide the learner in much the same way as a human tutor interacts with a learner. 

The tutoring agents observe the behaviors of the learner to assess their states (e.g., 

performance and attitudes) and interact with the learner to provide support, 

direction, and instruction. In addition, they track the effect of interactions on 

learning. Tutoring agents also interact with the training environment and may 

manipulate the environment to present more- or less-challenging scenarios in 

response to the assessed state of the learner.

5.1 Theoretical Foundations of Instructional Management

Instructional strategies have been advocated by researchers and practitioners in 

many different fields, including education, educational psychology, cognitive and 

learning sciences, military training, computer-based training, AI in education, 

computer-supported collaborative learning, educational data mining, etc. (Sottilare 

et. al. 2014). While these fields often vary across intent and scope of research, 
13

common themes have been recognized across multiple reports involving 

interdisciplinary research groups organized by the government and research 

organizations. Examples include (as highlighted in Sottilare et. al. 2014) the 

following:

• A Roadmap to Educational Technology. National Science Foundation, 

Arlington, VA; 2010. http://www.cra.org/ccc/docs/groe/GROE Roadmap 

for Education Technology Final Report.pdf.

• The Army Learning Concept for 2015. US Army, Washington, DC; 2011. 

http://www-tradoc.army.mil/tpubs/pams/tp525-8-2.pdf.

• Committee on Science Learning: Computer Games, Simulations, and 

Education. National Academy of Sciences, Washington, DC; 2011. 

http://www.nap.edu/catalog.php?record_id=13078.

• Assessing 21st Century Skills. National Academy of Sciences, Washington, 

DC; 2011. http://www.nap.edu/catalog.php?record_id=13215#toc.

• Improving Adult Literacy Instruction. National Academy of Sciences,

Washington, DC; 2012. http://www.nap.edu/catalog.php?record_id=1342.

• Organizing Instruction and Study to Improve Student Learning. Institute of 

Education Sciences of the United States Department of Education, 

Washington, DC; 2007. http://ies.ed.gov/ncee/wwc/pdf/practice_guides 

/20072004.pdf.

• Lifelong Learning at Work and at Home. American Psychological 

Association and Association for Psychological Sciences, Washington, DC;

2007. See Graesser AC. Inaugural editorial for Journal of Educational 

Psychology. 2009;101(2):259–261.

These reports emphasize instructional strategies that are supported by empirical 

tests with scientific methodologies; therefore, the strategies are grounded in science 

and are evidence-based. It is important that many of the strategies recommended 

from these groups are geared toward live instruction.

In terms of implementation, instructional management involves both an adaptive 

outer loop (i.e., adjusting problem selection based on performance outcomes) and 

an adaptive inner loop (i.e., providing real-time error-sensitive feedback or 

manipulating the scenario directly) function that accounts for individual differences 

across KSAs within a given domain. An adaptive, intelligent learning environment 

needs to launch the right instructional strategies at the right time in a mechanism 

that attempts to be sensitive to the learner model, maximize learning and 

motivation, and minimize training time and costs.
14

5.2 Instructional Management in GIFT

Before we examine specific goals and research interests associated with 

instructional management in GIFT, it is important to review some high-level 

components of the architecture. This involves an understanding of how information 

and data is represented and how these representations ultimately inform the updated 

Learning Effect Model (LEM) (Sottilare 2015 in press; Sottilare et al. 2015, Fig. 

2). The important takeaway of the LEM is the flow of data with respect to the

selection of an instructional practice. The effect chain is influenced by both 

historical data (e.g., prior experience, prior knowledge, and learner traits) 

maintained over time and real-time data captured during a specific interaction. This 

data is used to adapt instruction on 2 facets: 1) an inner-loop capacity using data to 

influence interaction within a single problem or scenario by providing guidance or 

adjusting difficulty levels similarly to Vygotsky’s (1978) zone of proximal 

development (ZPD) and 2) an outer-loop capacity that configures the next event 

experienced by a learner based on assessments from a prior event or through 

predictions based on historical representations and learner traits (VanLehn 2006). 

This might involve selecting a new problem/scenario, managing a remediation 

event, moving on to a new section of the course, administering an AAR, or ending 

the course.

Fig. 2 Updated individual learning effect model (Sottilare 2015 in press; Sottilare et al.

2015)

In terms of the inner-loop adaptive function, Assistive Technology Learning 

through a Unified Curriculum (ATLEC) primarily interacts on learner assessments 

occurring in real time for both performance and affective related states. Ultimately, 
15

raw interaction data are used to infer a learner state. This involves robust 

assessment techniques that can accurately gauge an individual learner’s 

performance for a given task, the affective responses they are having within that 

task, and their estimated competency for the domain that task is designed to train. 

This inferred learner state is used to inform the selection of an instructional strategy 

to mediate the learning experience. In the current baseline of GIFT (GIFT 2015), 

the instructional strategies supported for the inner loop consists of “provide 

guidance”, “adapt scenario”, “administer assessment”, and “do nothing”. These 

strategies are represented as high-level domain-independent descriptors of an 

action the system can take within a given learner event. These high-level actions 

must then be translated into a specific tactic of execution (see Sottilare et. al. 2014 

for a comprehensive breakdown of strategies versus tactics). 

For outer-loop adaptive functions, components of the ATLEC are administered up

front to configure a lesson’s sequence of interaction. Learner data inform strategy 

selections that associate with Merrill’s (1994) Component Display Theory. This 

interaction is currently encapsulated in a tool used by GIFT called the Engine for 

Management of Adaptive Pedagogy (EMAP) (see Fig. 3). In the latest GIFT 

baseline, the EMAP is used to guide a learner through a set of interactions that 

focuses on 1) presentation of rules for a domain or task, 2) presentation of examples 

where those rules are being applied, 3) administering of a knowledge assessment 

that gauges a learner’s ability to recall facts, and 4) administering of a practice 

assessment that gauges a learner’s skill for performing tasks associated with the 

domain of instruction (Goldberg et al. 2012; Wang-Costello et. al. 2013). 
16

Fig. 3 GIFT’s EMAP (Goldberg and Hoffman 2015)

With a current layout of the functions GIFT supports from an instructional 

management perspective, the remainder of the research outline will target specific 

utilities of interest that a domain-independent architecture needs for wide use across 

military, academic, and industry settings. Specific goals of desired functions will 

be presented, followed by recognized subvectors of interest that require substantial 

research to determine their optimal conditions for use in a learning event. As GIFT 

is intended to be a unified standard for building ITSs of all kinds, it is important to 

recognize the varying instructional management practices that can be implemented 

and what conditions are best for their use.
17

6. Instructional Management Research Goals and Challenges

The foundational goal of adaptive training research at the ARL is to model the 

perception, judgment, and behaviors of expert human tutors to support practical, 

effective, and affordable learning experiences guided by computer-based agents. 

To this end, 5 primary challenges in instructional management for adaptive training 

systems have been identified and are discussed in this section.

In the following subsections, we review the specific overarching themes associated 

with instructional management research in GIFT. Each theme is represented as a 

subvector in this research outline. While reading through these research thrusts, 

make sure to conceptualize their application across various training environments 

and use cases. While the subvectors can be somewhat confined in terms of 

instructional management intent, options for how they can be applied across all 

types of training environments are vast. This includes considering individualized 

versus team-based training events and the technologies being applied to facilitate 

the training itself.

6.1 Guidance and Scaffolding

Scaffolding is a term used to describe the application of instructional supports to 

assist a learner in developing knowledge and skills that the learner would not 

achieve when left to their own devices. Holden and Sinatra (2014) define scaffolds 

as a temporary application of strategy that is gradually removed (“faded”) once a 

learner demonstrates an increase in proficiency or skill. The word ‘strategy’ is used 

loosely here, as it can be a number of interventions that range from guidance, 

feedback, scenario/problem manipulation, and remediation. Prior to the advent of 

computer-based training environments and ITSs, an expert adult or peer (parent, 

teacher, classmate, teammate, etc.) provided scaffolding practices. This individual 

acted as an expert facilitator of KSAs required for learning, motivator, model, and 

means for the learner to reflect (Puntambekar and Hübscher 2005). As such, much 

of the applied guidance and scaffolding practices in adaptive training environments 

are based on what effective instructors and tutors do in real life. In examining 

models of expert human tutors, the following themes have been identified for 

effective interaction: 

• Demonstrate credible knowledge of the domain under training (e.g., tactical 

combat casualty care).

• Read cues from the learner and adapt instruction in real time to meet their 

changing needs.
18

• Encourage question asking.

• Provide indirect feedback.

• Assess learning often.

When designing scaffolds and the logic associated with their execution, 3

dimensions require consideration: what to scaffold, when to scaffold, and how to 

scaffold (Azevedo and Jacobson 2008). With respect to GIFT, each of these 

dimensions must be conceptualized in a domain-agnostic fashion. The first 

consideration that will dictate how to proceed is based on available tools and 

methods for scaffolding. In this vein, the mode of interaction and the specific 

training application itself will determine what approaches can be implemented 

(Goldberg and Cannon-Bowers 2015). For instance, interacting in VBS3 affords 

different scaffolding strategies when compared with interacting with a negotiation 

trainer on a tablet. The type of state (e.g., performance, affective) information being 

monitored in the learner model (Shute et al. 2013), the type of communication 

interfaces available for presenting information (e.g., GIFT’s Tutor User Interface, 

smartphone, smart glasses, haptic device), and the type of adaptations supported by 

the training application (e.g., changing the weather in VBS3 to increase the 

complexity of scenario) all impact the types of scaffolds that can be built to support 

a specific course or lesson. 

Once one recognizes the available tools to trigger and support scaffolding practices, 

determining what to scaffold must be addressed next. What to scaffold can be 

represented in a generalized fashion and is based on defined learning objectives and 

barriers to a learner performing successfully. This could include scaffolding 

specifically on the cognitive level that accounts for the domain or the task 

procedures themselves. Or scaffolds can account for different constructs associated 

with the learning process, such as metacognition (i.e., to regulate goal planning, 

performance monitoring, and help-seeking) and affect (i.e., to regulate motivation, 

boredom, frustration, and confusion) (van de Pol et al. 2010). Recognizing what 

one wants to scaffold along with what tools and methods one has to support those 

types of interventions will lend itself into building the specific scaffolds for 

implementation.

With scaffolds conceptually established, the next step is building logic for 

determining when to apply a scaffold and how to appropriately fade/adapt its use 

to promote efficient transfer of execution. A common perspective to account for

this challenge is based on Vygotsky’s (1978) ZPD. ZPD is an optimal and efficient 

zone of learning that elevates the student from his/her current and actual 

developmental level to one of more potential through balancing challenge with 

ability and through formative guidance to enhance skill (Vygotsky 1978). In its 
19

simplest form, the ZPD creates a formalized state space representation that enables 

a system to contextualize what a learner is experiencing for a given training domain. 

For example, if a training system launches a scenario with a preconfigured 

difficulty setting of expert and a trainee’s skill state is defined as journeyman, the 

system may trigger specific scaffolds that are intended to support the maturation of 

skill levels to meet the challenge level of a selected scenario. In this instance, if 

performance assessments associate execution with increased skill levels, the system 

must be able to recognize this shift and fade scaffolds. From the opposite end, if a 

learner demonstrates skills that fall below his or her predicted ability levels, the 

system must be able to appropriately trigger scaffolds to compensate for that 

inaccurate upfront classification. This must be possible for both dynamic scenariodriven tasks as well as for training applications that use discrete problem sets that 

allow adaptation between each problem. 

Much of the described scaffolding- and guidance-focused end-state objectives are 

composed of a mix of inner-loop and outer-loop pedagogical practices. In short, the 

training application being used will dictate the type of scaffolding that can be 

executed. While GIFT is domain-agnostic, once a training application is applied,

the next step is conceptualizing what adaptive functions are supported and what 

authoring processes are required. In an ideal end-state, a constraint-based, 

intelligent pedagogical agent can be applied to guide the authoring processes and 

to manage run-time decisions. These decisions center on selecting appropriate 

domain independent strategies, translating strategies into optimal tactics (actions) 

based on situational context, and then presenting the tactic (take action) through the 

delivery of feedback, providing direction, or changing the training environment. 

When considering scaffolding for adaptive training environments, several needed 

areas of research emerge. These areas are relevant due to the advancing popularity 

in ITSs and the relative infancy of these environments applied in military contexts. 

Holden and Sinatra (2014) identify several necessary future research areas 

involving scaffolding and GIFT. One involves defining the different focuses to 

scaffolding and distinguishing the conditions under which its implementation is 

best. Several specific types of scaffolding exist on a theoretical level, including 

conceptual, metacognitive, procedural, and strategic (Hannafin et al. 1999). As 

such, the word ‘scaffold’ applied in this context is as a very broad term that 

associates with multiple forms of transitions, interventions, and adaptations 

performed by a system. In terms of pedagogical intent, a range of scaffolds can be 

applied to associate with the following:

• Error-Sensitive Feedback: An intervention triggered when the learner 

commits errors that are either individually or cumulatively significantly 

divergent from the ideal as defined in the expert model of the ITS. 
20

• Mastery Learning: A strategy where the ITS ensures that the learner 

masters (can recall and apply) prerequisite lessons or concepts before 

allowing the learner to move on to the next lesson/concept. 

• Adaptive Spacing and Repetition: A strategy where the learner more 

easily recalls knowledge items/objects when the knowledge is exposed to 

the learner repeatedly over a long time-span rather than repeatedly studied 

during a short span of time (Benjamin and Tullis 2010; Dempster 1989). 

• Metacognitive Prompting: A strategy where the ITS encourages the 

learner to self-reflect and evaluate, self-explain, and self-correct rather than 

provide the answer directly. 

• Fading Worked Examples: “[A] step-by-step demonstration of how to 

perform a task or how to solve a problem” from which parts have been 

deliberately removed or faded (Clark et al. 2006, p. 190). 

While scaffolds are very domain-relevant and application-dependent, tools and 

methods must be established in GIFT to support building and implementing optimal 

scaffolds across all types of supporting training environments. This also extends 

into investigating scaffolds for team-based training events. From this association, 

the following potential research questions arise:

• What is the best approach to the authoring run-time instructional 

management logic in GIFT?

• Are there pedagogical best-practice heuristics that generalize across training 

environments with varying levels of interactivity and user control?

• How do scaffolding best practices translate to collaborative and team-based 

training exercises?

• What are the variations in feedback considerations when comparing 

individualized instruction with team-based training?

• What role do worked examples play in military relevant training 

environments, and how can SMEs use this pedagogical function to create 

better training?

• What probabilistic modeling techniques account for system uncertainties 

and optimize instructional management practices and tutorial planning logic 

over time to avoid solely deterministic execution?

• What architectural modifications are required to support contextualized 

feedback prompts that adhere to a domain-agnostic pedagogical structure 
21

(i.e., communicating scenario relevant feedback by including interaction 

data available in log files)?

• How much control should the learner have on guidance and scaffolding 

support (i.e., help seeking)? 

• When should guidance and scaffolding be on demand (e.g., learner requests 

hint), triggered by a student/system event or state, or blended? 

• What performance factors and individual differences tracked in the learner 

model should moderate scaffold selection and fading logic?

• How do generalized scaffolding techniques vary across training delivery 

modalities (e.g., laptop, smartphone/tablet, simulators, and smart 

glasses/head-mounted displays)?

• How can SMEs embed scaffold supports in virtual world game 

environments commonly used for training purposes (e.g., VBS3)?

• Under what conditions does an adaptive training environment provide 

guidance versus adapting the scenario or problem selection?

• What assessment types trigger variations in scaffolding techniques (e.g., 

conceptual versus affective versus procedural versus metacognitive)?

6.2 Social Dynamics and Virtual Humans

In this research subvector, we review end-state goals associated with the role of 

social dynamics and virtual humans in managing instruction within adaptive 

training environments. This line of research associates with tenets of Social 

Cognitive Theory in that learning is theorized to be an inherently social process 

(Bandura 1986; Vygotsky 1978). As such, techniques should be applied to account 

for high-valued social elements that can potentially improve a piece of educational 

technology. In addition, Army task domains can involve highly socialized 

interactions. To support these task characteristics, adaptive training systems should 

account for the types of interactions a Soldier might face (e.g., negotiating with a 

village elder) and the variables that may influence their course of action (e.g., 

cultural norms for greetings and negotiations). From an adaptive training 

perspective, social dynamics and virtual human research is focused on the 

following: 

• Using technology and AI to replicate interactive discourse common in 

educational and operational settings.
22

• Using technology to embed social elements in the environment, such as 

virtual humans, to create a social grounding function for delivering 

information/guidance.

• Using technology and AI to create realistic and reactive virtual humans as 

training elements in a simulation or scenario (e.g., role player and

teammate).

• Using technology to create a social forum for the purpose of supporting 

peer-to-peer and collaborative learning, both from a real-time perspective 

and a time-agnostic approach allowing interaction at convenience.

Each referenced focus area is based on a specific social element of interest to the 

GIFT community. Of importance is the application of these elements within the 

standardized architecture inherent to GIFT. Specifically, how can the tools and 

methods built to afford these capabilities be translated to support ease of application 

within any type of training event and within any type of training environment 

supported by GIFT? Each identified element associates distinctly different research 

questions and associates distinctly different scientific disciplines. It is a very broad 

subvector of instructional management with many dependencies to other elements 

of the GIFT research program. 

In the case of using ITS technologies to replicate interactive discourse, an end-state 

goal is to establish state-of-the-art natural language processes to create a robust tool 

capable of dynamic question and answer (QandA) exchanges. In this instance, we 

want a training environment to be able to push a question to a learner and a trainee 

to be able to ask questions of the ITS, ideally through natural language and open 

response input methods. As highlighted previously, a characteristic of an effective 

tutor is one who encourages a learner to ask questions. This process in itself 

promotes learning through abstraction and reflection. As such, we need a discourse 

capability that can accurately interpret user responses and intent as they relate to 

the semantic space of instruction, and we want this capability to associate with preexisting training applications (i.e., natural discourse QandA in parallel with 

executing a VBS3 scenario). This capability can be applied in multiple training 

instances and under many conditions, including 1) discourse to support reflective 

QandA sessions to promote higher order cognitive thinking, 2) discourse to support 

training events that involve social exchanges to meet certain negotiation objectives,

and 3) discourse to support realistic communication with virtual entities in an 

environment that associate with both friendly and opposing forces. Much like 

scaffolding capabilities, the application of natural language discourse in an adaptive 

training event associates many dependencies across the various research vectors. 

Specifically, domain modeling, authoring processes, and architecture requirements 
23

are the greatest considerations when it comes to implementing this approach to 

instructional management.

Next, virtual humans are identified as key technology pieces in extending adaptive 

training experiences to account for varying roles in the learning process. From an 

ITS support perspective, virtual human research is focused on the application of 

technology to provide an interactive communication layer that grounds all systemgenerated prompts with a social element. An overarching intent is to facilitate 

interaction and communication with a computer in a way that is natural and 

realistic. The goal is to support highly engaging and interactive experiences through 

socialized sequencing of interaction and enhancing system communication by 

interfacing with a learner through comfortable modalities. Much of the prior 

research in this area focuses on the trust and perception of technology in facilitating 

a role traditionally managed by a person and the impact on motivation and effort 

(Holden and Goldberg 2011; Kim and Baylor 2006; Veletsianos 2010; Veletsianos

et al. 2009). 

Virtual humans can also facilitate critical role players in training events. In these 

instances, AI methods allow virtual entities to realistically react to environmental 

stimuli and user inputs in a nonscripted fashion, making the experience more 

natural and engaging. Social media technologies are believed to offer innovative 

tools for instructional management practices that have yet to be fully taken 

advantage of. As a result, research is required to better understand how best those 

tools and methods can be applied. This involves the use of social media as a means 

to collect valuable user input on training materials to determine objects that need 

modifications, as well as investigating how social media can be used to promote 

collaborative learning, competition, and distributed guidance.

6.3 Metacognition and Self-Regulated Learning 

SRL theory describes the process of taking control of and evaluating one’s own 

learning and behavior (Butler et al. 2011). As a higher-order cognitive function, 

SRL is guided by metacognitive processes (i.e., the knowledge and regulation of 

one’s own cognition), strategic actions and behaviors (i.e., planning, monitoring, 

and assessing one’s own performance), and motivational components (i.e., goal 

setting and self-efficacy) (Flavell et al. 1985; Schraw et al. 2006). These functions 

allow self-regulated learners to set goals, monitor their progress toward defined 

goals, and adapt and regulate their cognition, motivation, and behavior to reach the 

specified goals (Anderman and Corno 2013; Bransford et al. 2000). These 

characteristics also associate with desired competencies within the ALM that 

adaptive training solutions are intended to instill. As such, research in the 
24

instructional management vector is focused on the application of models and 

strategies for enhancing metacognitive awareness and regulation. 

This approach to instructional management varies from traditional guidance and 

scaffolding techniques, as it focuses on behavior and application of strategy, rather 

than on task-dependent performance. One such question is based on GIFT 

supporting SRL and the efficacy of defining persistent metacognitive strategies that 

can be applied across domain applications. Currently, GIFT pedagogy is heavily 

focused on error-sensitive feedback. It works with system authors by translating 

instructional strategy recommendations communicated by GIFT’s pedagogical 

module into tactics as they relate to the specific training context. These tactics are 

used during ITS runtime and selected based on a learner’s individual differences. 

Currently, feedback in GIFT is domain-dependent and requires explicit content 

linked to each concept modeled. When it comes to metacognitive feedback, what 

are the implications to a domain-independent approach? First, modeling techniques 

need to be developed to monitor an individual’s practice of metacognitive strategies 

that can be expressed in a generalized format. An example would be incorporating 

a combined modeling approach, as described in Biswas et al. (2014), or by adapting 

a help-seeking model, as highlighted in Koedinger et al. (2009). 

One such approach is researching and establishing models based around commonly 

available GIFT interactions (e.g., request hint button). How can we use these 

available data inputs to build a representation of how effective students use the 

interface to solve problems and troubleshoot errors? This approach can aid in 

detecting learners not practicing good metacognitive behaviors through machine

learning and data mining practices and can be used to trigger feedback interventions 

to improve their understanding of available strategies. With modeling techniques 

in place, generic strategies and tactics can be identified that are based on effective 

metacognitive behavior. In this instance, the generic strategy of “provide guidance”

can be linked with a generic tactic of “you are ignoring available resources”, thus 

preventing any explicit authoring from a system developer. While tactics can be 

represented in a domain-independent format, their effect is relatively unknown. 

Regardless of the instructional management intent, just like all other facets of 

instruction, metacognitive tutoring depends on sound assessment practices that 

ultimately inform the type of intervention to execute. Due to difficulties in inferring 

upon interaction data alone, another area of interest is the application of alternative 

assessment techniques to better gauge an individual’s understanding of the task and 

the resources available to meet objectives. This involves inference procedures on 

interaction patterns and existing performance outcomes, as well as devising 

approaches to collect new information that is not implicitly available in the training 

environment.
25

Beyond all of the associated types of scaffolds that can serve metacognitive 

development, how does one implement such practices? A challenge that must be 

addressed is establishing an authoring environment and workflow for supporting 

SRL-derived modeling techniques and linking outputs to prescribed strategies that 

influence the regulation of metacognitive behaviors. The ultimate goal is to support 

and influence a learner’s approach to problem solving and learning in general. 

While the research identifies multiple examples of successful strategy 

implementation, what the literature lacks are guidelines for when best to instantiate 

them based on the domain being trained and the environment the interaction is 

taking place within. To enhance GIFT’s authoring functionalities, a generalized 

ontology is required that links specific instructional strategies and techniques with 

high-level domain-relevant content along with the types of tutoring environments 

and the services they can afford. By defining these relative dependencies, an ITS 

developer can embed empirically recommended metacognitive tutoring functions 

based on characteristics associated with the content being produced.

In the context of tutor development within the GIFT architecture, the goal is to 

establish a set of strategies the framework can support and conceptualize their 

application for determining future authoring requirements to support their 

implementation, both from a modeling and pedagogical delivery standpoint. In 

accordance with this subvector of instructional management research in adaptive 

training environments, research questions of interest include the following:

• What modeling implications are associated with supporting instructional 

management to develop metacognitive skills?

• What architectural modifications are required to infer metacognitive 

relevant states from raw interaction data outputs?

• What combination of assessment techniques provides the most accurate 

prediction of metacognitive strategy application?

• What metacognitive strategies can be inherently assessed across all domains 

regardless of the training application?

• How do metacognitive strategies translate across domains and training 

environments?

• What educational data mining techniques can be applied to infer selfregulated learning abilities from common GIFT produced log files?

• How do scaffolding practices of delivery and fading for domain-specific 

guidance functions translate to metacognitive tutoring?
26

• Can metacognitive tutoring models and strategy implementations be 

developed by nontechnical SMEs? 

6.4 Optimizing the Selection of Tactics 

Our fourth goal is to optimize the selection of tactics—domain-specific actions by 

the tutor—to provide the greatest opportunity for performance, learning, retention, 

and transfer. In GIFT, tactics are the actions taken by the tutor in response to learner 

states and instructional context (e.g., conditions of the scenario or problem 

presented), as shown in Fig. 2, and are constrained by available options provided 

during the authoring process. Improving the usability and efficiency of authoring 

tools will likely result in a greater number of available options for adaptive training 

domains.

Unlike instructional strategies, which are derived from good pedagogical practices–

based learning theory and influenced by the learner’s states, tactics are domainspecific actions by the tutor and may not be generalized across all task domains. 

Research is needed to determine methods to select the best possible tactic given the 

selected instructional strategy, the training domain, and the availability of tactics.

Modeling the expert behaviors of human tutors may be a starting point but accurate 

assessment methods are needed for both individual and team level states. These 

states are critical in selecting appropriate strategies (plans for action) and tactics 

(actions; e.g., assessments, feedback, questions, and changes to the training 

environment) per the Learning Effect Model (Sottilare et al. 2013b) as shown for 

both individuals (Fig. 2) and teams (Fig. 4). Assessment of team states may also be 

useful in determining constraints to be monitored by tutoring agents and 

interactions with the learner and the training environment as shown in Fig. 2.

Fig. 4 Team learning effect model (Sottilare in press)
27

6.5 Personalization (Occupational and Noncognitive Factors)

Personalization of content to an individual student’s interests has been shown to 

have a positive impact on learning, retention of information, and motivation 

(Cordova and Lepper 1996). Further, using this strategy has also demonstrated 

positive effects in transfer performance, which is the ability to extend what was 

learned to related tasks (Walkington 2013). This instructional strategy is generally 

referred to as context personalization, and has been shown to be effective in 

adaptive computer environments (Anand and Ross 1987; Cordova and Lepper 

1996; Walkington 2013).

The majority of the context personalization research has been in the area of 

mathematics, with learning and transfer gains being found in areas such as algebra, 

fractions, and word problems (Anand and Ross 1987; Cordova and Lepper 1996; 

Ross et al. 1986; Walkington 2013). The self-reference effect, or the ability for 

reference to the self to improve memory (Symons and Johnson 1997), is often 

studied along with context personalization by including the student’s name or 

personal information in the materials (Anand and Ross 1987; Cordova and Lepper 

1996; Sinatra et al. 2014). In addition to mathematics, the self-reference effect has 

been examined in domain areas such as science and deductive reasoning (Moreno 

and Mayer 2000; Sinatra et al. 2014).

By personalizing the context of the material that is learned, it could have both 

affective and cognitive impacts on the individual. Motivation and positive attitudes 

toward the material has been shown to increase in personalized contexts (Anand 

and Ross 1987; Cordova and Lepper 1996). In an adaptive system with self-entered 

preferences, the student may feel that the system is taking his or her preferences 

into account if interest personalized questions are given, which may result in more 

positive feelings toward the system (Ritter et al. 2007). Additionally, by aligning 

examples and information in the content to be learned with the previous knowledge 

base of the individual, it may both increase interest in the training and lessen the 

mental workload required to learn the material (Anand and Ross 1987; Ku and

Sullivan 2002; Ross 1983; Sinatra et al. 2014). 

Research has suggested that by using examples that are consistent with one’s 

college major and area of expertise it can improve learning outcomes. Specifically, 

it was found that nursing students who received medical examples had more 

positive outcomes than those who were given teaching examples. Conversely, 

education students who were given teaching examples on the same task performed 

better than when they were given medical examples (Ross 1983). Follow-up work 

found that both individually selected interest preferences and general adaptation to 

interests (based on major) showed positive learning outcomes (Ross et al. 1986). 
28

The reason behind these effects may be 2-fold: By using examples that are 

consistent with one’s knowledge base, it may be inherently more interesting to the 

individual; further, it may reduce the amount of cognitive effort that is necessary to 

engage with the material since the student already has a good understanding of the 

context.

It would be advantageous to continue to research into the impact of context 

personalization in adaptive training programs. Current ITS systems, such as GIFT, 

offer more flexibility and features than systems and computer programs that were 

developed in the original context-personalization research conducted in the 1980s 

and 1990s. Additionally, as a domain-independent framework, GIFT can be used 

to examine the impact of context personalization in a variety of other domains, 

whereas in the past the research has primarily focused on math instruction. One 

approach to context-personalization research that can be taken with GIFT is to do 

work similar to Ross (1983), in which the context of the problems and materials are 

specifically matched or mismatched with the individual learner’s specialty area. In 

the context of military training, a Soldier’s MOS and near-term assignments can be 

used to personalize a training experience to better prepare that individual for the 

environment they will be operating within.

Rather than using mathematics as the domain of interest, a military-relevant domain 

can be chosen. Providing materials that are matched to the individual learner’s 

specialty area is expected to have a positive impact on learning and attitudes toward 

the experience. Learning outcomes are expected to be improved, as the individual 

will not already have an understanding of the context of the provided examples but 

also will be able to easily see why it is relevant to their own job. 

To support personalization, additional studies could examine the impact of 

allowing learners to select the context of the questions they will receive based on 

their own preferences or the task that they will be engaging in. In many militaryrelated tasks there are subtle differences in the task that will be performed based on 

the geographic location of their assignment. For instance, if an individual is tasked 

with interacting and negotiating with individuals from a culture other than their 

own, they may engage with a negotiation tutor. However, depending on the culture 

that they are to engage, there may be different phrases or customs that should or 

should not be used. The basic elements of negotiation will be similar but the 

questions and materials can be edited to have geographic and culturally specific 

examples that will be more consistent with the actual experience the individual will 

have. Research can be conducted on the level and types of material and assessment 

personalization that results in positive outcomes and performance. 
29

To support this type of research and personalization, ideally in an ITS, general 

questions can be created, and based on learner entered or selected information they 

can be personalized for the individual student. The following is a math example to 

demonstrate simple question context personalization:

• General question: “You have 4 _____. You are offered 5 _____ in exchange 

for half of your _____. How many ______ do you have after the trade?” 

• If you were using the context of fruit, the problem could read “You have 4 

apples. You are offered 5 oranges in exchange for half of your apples. How 

many pieces of fruit do you have after the trade?” 

• Additionally, if you were using the context of books, the question could 

read “You have 4 science fiction novels. You are offered 5 mystery novels 

in exchange for half of your science fiction novels. How many novels do 

you have after the trade?”

In this example, a general question is authored and then specific items are added in 

to change the context or topic of the question. Research can examine the impact of 

providing questions and examples that are consistent (and inconsistent) with the 

learner’s job or interest areas. To do so in GIFT, edits may be needed to be made 

such that GIFT could store a bank of questions that are similar to each other but 

differ in context. The version of the questions that are consistent with the learnerselected context can then be selected by the system and used during training. 

Additional techniques for personalization of questions and materials may include 

allowing for user-entered information (such as the individual’s name) to be 

integrated into the questions. 

If these or similar studies are shown to be successful, multiple versions of training 

question domains can be generated in ITSs, resulting in the learner engaging with 

the one that is most relevant to themselves based on previous experience and selfentered preferences. As functionality is being developed in ITSs to track student 

learning history and characteristics, it may also be advantageous to store student 

preferences, interests, and the expertise area of the individual. This stored 

information can then be used for context personalization to ultimately increase 

learner performance and retention in ITSs such as GIFT.

While personalization is an instructional strategy, research based on it will 

ultimately impact both the areas of authoring and learner modeling in GIFT. 

Authoring tools and/or support will need to be added into GIFT to allow for 

personalized questions and storing personalized question banks. Storage of learner 

preferences and information will need to be stored by GIFT’s learner module such 

that they can be used in future trainings and engagements with the system. 
30

The following are personalization-related research questions:

• Will matching the context of materials to be consistent with a learner’s job 

or interest area lead to increased learning and performance?

• Will individuals who receive context personalized materials have morepositive attitudes toward the tutoring system?

• What types of personalization (e.g., based on self-selected interest, based 

on job, and based on task) will have the best outcomes?

• Are their individual differences that will moderate the benefits of 

personalization? If so, what types of learner characteristics should result in 

the use of personalized materials?

7. Interdependencies with Other Adaptive Training Research 

Vectors

This section examines interdependencies between instructional management and 

the other 5 adaptive training research vectors (Fig. 5). This discussion forms the 

basis for the sequencing of research and ultimately bringing adaptive training 

capabilities into a state of practice. 

Fig. 5 Adaptive training research vectors
31

Accurate methods to classify individual and team learner states are a necessary 

precursor to selecting optimal instructional strategies, as noted in the learning effect 

models for individual learners (Fig. 2) and teams of learners (Fig. 4). In turn, 

instructional strategies along with instructional context are necessary precursors to 

selecting optimal instructional tactics and ultimately significant effect on desired 

outcomes: learning, performance, retention, and transfer.

As such, it is difficult to conceptualize the application of instructional management 

focused research without involving other vector components of the GIFT program. 

Therefore, the dependency of instructional management research on other 

functional areas of the architecture is significant.

7.1 Learner Modeling and Instructional Management

Adaptive training systems are learner-centric systems. Independent of the domain 

under training, accurate modeling of the learner is critical to driving instructional 

decisions in adaptive training systems. However, collection and maintenance of this 

data may be costly, so it is necessary to select measures and states that significantly 

impact our desired outcomes: learning, performance, retention, and transfer. 

Research is needed to determine what this dataset will look like. Candidates abound 

in the literature but in general these include transient data/states, cumulative states 

(building over time), and enduring data/states (Paneva 2006). Learning falls into 

both transient and cumulative states. It is necessary to understand progress toward 

domain competency in addition to measures of near-term performance, and it is 

important to understand how domain-specific learning (skills) decay over time. 

Transient measures of importance include individual behavioral and physiological 

data, and cognitive, affective, physical, and social states to represent learning. 

Cumulative measures include achievements (e.g., certifications, training, 

education, and experiences), affiliations, work history, and domain competency. 

More-enduring information about the learner might include gender, culture, first 

language, physical constraints (e.g., colorblind/deaf), values, personality attributes, 

or other trait-based information (Sottilare and Brawner 2014). All of these 

measures/states are potential drivers for adaptive training decisions. 

In terms of instructional management, sound pedagogy is reliant on accurate 

representation in the learner model. It is important to identify variables in a learner 

model that moderate learning, account for variance in performance, and can be 

accurately assessed in real time for informing strategy selections. Selecting an 

inappropriate strategy and tactic combination can result in negative learning 

outcomes and can ultimately impact the perceived value of the system. 
32

7.2 Domain Modeling and Instructional Management

In GIFT, instructional management takes place in 2 modules/processes within the 

learning effect model. One process is instructional strategy selection within the 

pedagogical module. The second is within the domain module where specific 

tactics or actions are selected based on the strategy selection and instructional 

context. An important component of instructional management is translating a 

generalized strategy into a tactic that can be executed within a specific training 

environment. This requires understanding what knowledge components make up a 

domain and what tools are available to guide a learner and adapt the training event.

In addition, domain modeling plays a critical role in enabling the use of reusable 

learning objects. When applying instructional management practices in an outerloop capacity through GIFT’s EMAP (Fig. 3), a well-designed domain model can 

be used to identify content that can be presented to a learner along with data that 

supports its application. This supports ease of authoring as well, as a developer can 

leverage existing content if the domain model has overlap with existing course 

representations.

7.3 Authoring Tools and Instructional Management

Authoring tools and methods are needed to configure instructional management 

practices across all adaptive training systems and domains of instruction. There are 

2 types of authors that must be considered. First, training developers and SMEs are 

targeted as end-users. These individuals will use GIFT services to create highly 

personalized adaptive training events by leveraging the pedagogical models 

established within available baseline versions. While a training developer and SME 

are well versed in the knowledge components of a domain, they will lack many of 

the skills required to build adaptive training solutions. The authoring tools must be 

designed to compensate for a developer’s lack of understanding across the various 

disciplines involved in creating such systems. Building tools that guide the various 

authoring processes is critical to the success of GIFT outside of the laboratory 

setting. It will also be important that these tools help individuals maintain a 

contextualized awareness of what they authoring and how that will be translated 

into a run-time instantiation.

The second author to consider is a research scientist interested in instructional 

management research within adaptive training systems. These individuals will 

require the ability to manipulate various components of the GIFT modules to 

support their research. Using the ATLEC as a guiding function to highlight 

instructional management research in ITSs, authoring processes must be 
33

established to configure the inputs, processes, and outputs of the pedagogical

module. This includes defining the variables from the learner model that moderate 

strategy selection (with a large assumption that the modeling techniques applied for 

that variable are accurate), defining the logic and policies that dictate what strategy 

to select based on learner data, and defining how that strategy is translated into a 

specific tactic to be executed in the environment. This is greatly different from the 

training developer/SME example, as the only intended authoring function of this 

group is defining the tactic that would be executed.

Another avenue of authoring that must also be addressed is metadata, which plays 

a large role in instructional management within a domain-independent framework,

as it provides a way to describe your content and guidance functions in a relatively 

generic fashion. In terms of authoring, tools must be developed to facilitate easy 

application of metadata to learning objects. Research is required to determine how 

best to support this function.

7.4 Evaluation and Instructional Management

Training effectiveness research is focused primarily on the impact of instructional 

management practices on learning outcomes of interest: performance, retention, 

and transfer. To optimize the application of adaptive training in the Army, 

automated training effectiveness tools are required to run analyses as data become 

available. Much of these capabilities are researched within communities dedicated 

to educational data mining. The goal is to test pre-established configurations to 

determine their impact on learning and adapt instructional management policies to 

better optimize learning experiences. 

A goal of instructional management in GIFT is to optimize performance and 

competency outcomes through personalized training experiences that adapt to an 

individual’s KSAs. A connection between big data and instructional management 

is applying large datasets from prior course interactions to update and improve 

technique and strategy implementations across all available courses. A challenge 

with defining instructional management logic in a domain-independent context is 

that it requires generalizability across applications. An issue with instructional

strategy-based research is that an approach taken in one domain is hard to translate 

to a different one without performing extensive research to validate its application. 

In addition, it is difficult to define definitive instructional management logic based 

on these uncertainties. As such, big data can be used to account for this uncertainty 

by applying machine learning and data mining techniques to assess specific causal 

relationships between instructional practices and outcomes on performance, 
34

retention, and transfer. This application of big data is used to reinforce instructional 

management models by optimizing itself over time as more and more data are made 

available. 

7.5 Architecture and Instructional Management

Managing instructional strategy selection and tactic delivery depends upon multiple 

components of GIFT. This associates domain modeling to apply context to a 

pedagogical decision, learner modeling to provide trainee relevant information that 

triggers a pedagogical intervention, authoring to provide a means for building these 

linkages and representations, and training effectiveness to determine if a strategy 

or set of strategies had an effect on performance-related outcomes. This highlights 

an important point; while each of the aforementioned components of instructional 

management has separate processes, the architecture is the component that dictates 

implementation design and development.

In terms of architecture, GIFT end-state goals require potential integration with a 

number of technologies that facilitate varying roles of instructional management 

practices. These technologies include tools and methods to support content 

management, natural language processing, text-to-speech processing, virtual 

human authoring and configuration, social media framework connections, and 

training application manipulations (e.g., manipulating the weather in a virtual 

world). In addition, specific architectural modifications will be required to perform 

tasks inherent to the current standards of GIFT, including methods to create 

messaging templates used to auto-populate feedback scripts with context relevant 

information established in log files, the ability to personalize strategy selections on 

an outer- and inner-loop capacity across learners and teams of learners, and the 

application of actionable metadata and API statements to appropriately link learner 

information and prior experience with appropriate training and optimized 

configurations. In dealing with a domain-agnostic intelligent framework such as 

GIFT, the use of machine learning and data mining techniques is required to 

reinforce and optimize pedagogical logic over time. 

8. Conclusions

This report outlines the ARL’s plans for conducting research in adaptive training 

and education to support the ALM. Specifically, this report relates to instructional 

management and the answer to the following question: What adaptive instructional 

methods (strategies, tactics, and techniques) are most effective for individual and 

team-based military training and educational domains?
35

This report outlined the following goals:

• Model the perception, judgment, and behaviors of expert human tutors to 

support practical, effective, and affordable learning experiences guided by 

computer-based agents.

• Leverage prior research in the learning sciences and AI communities (both 

theoretical and empirical) to establish a set of best practices on how to 

author and execute techniques, strategies, and tactics across any domain of 

instruction.

• Discover and develop modeling functions that account for uncertainty 

across policies informing pedagogical decisions (e.g., content delivery, 

course navigation, and guidance).

• Personalize instructional technique and strategy selections based on 

individual differences informed through empirical evaluations and 

reinforcement machine learning methods.

• Develop authoring tools grounded in cognitive and instructional theory and 

informed by empirical research to assist training managers, developers, and 

SMEs in building pedagogically sound training experiences without the 

requirement of programming.

• Support individual and team training (e.g., small unit and collective 

training) and education (e.g., collaborative learning and problem-solving) 

experiences.
36

9. References

Anand PG, Ross SM. Using computer-assisted instruction to personalize 

arithmetic for elementary school children. Journal of Educational Psychology. 

1987;79(1):72–78.

Anderman E, Corno L. Handbook of educational psychology. London 

(England): Routledge; 2013.

Azevedo R, Jacobson M. Advances in scaffolding learning with hypertext and 

hypermedia: a summary and critical analysis. Educational Technology 

Research Development. 2008;56:93–100.

Azevedo R, Witherspoon A, Graesser AC, McNamara DS, Chauncey A, Siler 

E, Cai Z, Lintean M. MetaTutor: analyzing self-regulated learning in a tutoring 

system for biology. In: Dimitrova V, Mizoguchi R, Du Boulay B, Graesser 

AC, editors. Artificial intelligence in education: building learning systems that 

care: from knowledge representation to affective modelling. Amsterdam (The 

Netherlands): IOS Press; 2009. p. 635–637.

Bandura A. Social foundations of thought and action: a social cognitive theory. 

Englewood Cliffs (NJ): Prentice Hall; 1986.

Benjamin AS, Tullis J. What makes distributed practice effective? Cognitive 

Psychology. 2010;61(3):228–247. 

Biswas G, Segedy JR, Kinnebrew JS. A combined theory and data-driven 

approach for interpreting learners’ metacognitive behaviors in open-ended 

tutoring environments. In: Sottilare R, Graesser A, Hu X, Goldberg B, editors.

Design recommendations for intelligent tutoring systems. Volume 2: 

instructional management. Orlando (FL): Army Research Laboratory (US);

2014.

Bloom BS. The 2 sigma problem: the search for methods of group instruction 

as effective as one-to-one tutoring. Educational Researcher. 1984;13(6):6.

Bransford JD, Brown AL, Cocking RR. How people learn: brain, mind, 

experience, and school. Washington (DC): National Academy Press; 2000.

Butler DL, Cartier SC, Schnellert L, Gagnon F, Giammarino M. Secondary 

students’ self-regulated engagement in reading: researching self-regulation as 

situated in context. Psychological Test and Assessment Modeling. 

2011;53(1):73–105. 
37

Butler DL, Winne PH. Feedback and self-regulated learning: a theoretical 

synthesis. Review of Educational Research. 1995;65:245–281.

Clark RC, Nguyen F, Sweller J, Baddeley M. Efficiency in learning: evidence‐

based guidelines to manage cognitive load. Performance Improvement.

2006;45(9):46–47. 

Cohen PA, Kulik JA, Kulik CLC. Educational outcomes of tutoring: a metaanalysis of findings. American Educational Research Journal. 1982;19:

237–248.

Cordova DI, Lepper MR. Intrinsic motivation and the process of learning: 

beneficial effects of contextualization, personalization, and choice. Journal of 

Educational Psychology. 1996;88(4):715–730.

Dempster FN. Spacing effects and their implications for theory and practice. 

Educational Psychology Review. 1989;1(4):309–330. 

Dodds PVW, Fletcher JD. Opportunities for new “smart” learning 

environments enabled by next generation web capabilities. Journal of 

Education Multimedia and Hypermedia. 2004;13:391–404.

Dynarsky M, Agodina R, Heaviside S, Novak T, Carey N, Campuzano L, 

Sussex W. Effectiveness of reading and mathematics software products: 

findings from the first student cohort. Washington (DC): Institute of Education 

Sciences, US Department of Education; 2007 Mar.

Flavell JH, Miller PH, Miller SA. Cognitive development. Englewood Cliffs

(NJ): Prentice-Hall; 1985.

Fletcher JD. Evidence for learning from technology-assisted instruction. In:

O’Neil HF, Perez RS, editors. Technology applications in education: a learning 

view. Mahwah (NJ): Erlbaum; 2003. p. 79–99.

Fletcher JD, Morrison JE. DARPA digital tutor: assessment data. Alexandria

(VA): Institute for Defense Analyses; 2012. IDA Document D-4686.

Franke D. Decision-making under uncertainty: using case studies for teaching 

strategy in complex environments. Journal of Military and Strategic Studies. 

2011;13(2):1–21.

Generalized Intelligent Framework for Tutoring (GIFT) Home Page. Release 

GIFT2015-1. [accessed 2015 Nov 2]. https://www.gifttutoring.org.
38

Goldberg B, Brawner KW, Sottilare R, Tarr R, Billings DR, Malone N. Use of 

evidence-based strategies to enhance the extensibility of adaptive tutoring 

technologies. Presented at the Interservice/Industry Training, Simulation, and 

Education Conference (I/ITSEC); 2012 Dec 3–6; Orlando, FL.

Goldberg B, Cannon-Bowers J. Feedback source modality effects on training 

outcomes in a serious game: pedagogical agents make a difference. Computers 

in Human Behavior. 2015;52:1–11.

Goldberg B, Hoffman M. Authoring instructional management logic in GIFT 

using the engine for management of adaptive pedagogy (EMAP). In: Sottilare 

R, Graesser A, Hu X, and Brawner K, editors. Design recommendations for 

intelligent tutoring systems. Volume 3: authoring tools and expert modeling 

techniques. Orlando (FL): Army Research Laboratory (US); 2015. 

Graesser AC, Conley MW, Olney AM. Intelligent tutoring systems. In:

Graham S, Harris K, editors. APA educational psychology handbook. Volume

3: applications to learning and teaching. Washington (DC): American 

Psychological Association; 2012. p. 451–473.

Graesser AC, D’Mello SK, Cade W. Instruction based on tutoring. In: Mayer 

RE, Alexander PA, editors. Handbook of research on learning and instruction. 

New York (NY): Routledge Press; 2011. p. 408–426.

Graesser AC, Hu X, Nye B, Sottilare R. Intelligent tutoring systems, serious 

games, and the generalized intelligent framework for tutoring (GIFT). In:

O’Neil HF, Baker EL, Perez RS, editors. Using games and simulation for 

teaching and assessment. Abingdon (UK): Routledge; 2015; in press.

29. Graesser AC, Lu S, Jackson GT, Mitchell H, Ventura M, Olney A, 

Louwerse MM. AutoTutor: a tutor with dialogue in natural language. 

Behavioral Research Methods, Instruments, and Computers. 2004;36:

180–193.

Graesser AC, McNamara DS. Self-regulated learning in learning environments 

with pedagogical agents that interact in natural language. Educational 

Psychologist. 2010;45:234–244.

Halpern DF, Millis K, Graesser AC, Butler H, Forsyth C, Cai Z. Operation 

ARA: a computerized learning game that teaches critical thinking and 

scientific reasoning. Thinking Skills and Creativity. 2012;7:93–100.
39

Hannafin M, Land S, Oliver K. Open learning environments: foundations, 

methods, and models. In: Reigeluth CM, editor. Instructional-design theories 

and models: a new paradigm of instructional theory: volume 2. Mahwah (NJ): 

Lawrence Erlbaum Associates; 1999. p. 115–140. 

Holden H, Goldberg B. The impact of student expectations and tutor 

acceptance on computer-based learning environment acceptance and future 

usage intentions. Presented at the International Defense and Homeland 

Security Simulation Workshop; 2011 Sep 12–14; Rome, Italy.

Holden HK, Sinatra AM. A guide to scaffolding and guided instructional 

strategies for ITSs. In: Sottilare R, Graesser A, Hu X, Goldberg B, editors.

Design recommendations for intelligent tutoring systems. Volume 2:

instructional management. Orlando (FL): Army Research Laboratory (US);

2014. p. 265–282.

Kim Y, Baylor A. A social-cognitive framework for pedagogical agents as 

learning companions. Educational Technology Research and Development.

2006;54(6):569–596. 

Koedinger K, Aleven V, Roll I, Baker R. In vivo experiments on whether 

supporting metacognition in intelligent tutoring systems yields robust learning. 

Handbook of metacognition in education. New York (NY): Routledge; 2009. 

p. 897–964. 

Ku HY, Sullivan HJ. Student performance and attitudes using personalized 

mathematics instruction. Educational Technology Research and Development.

2002;50(1):21–34.

Ma W, Adesope OO, Nesbit JC. Intelligent tutoring systems and learning 

outcomes: a meta-analytic survey. Journal of Educational Psychology; in 

press.

Merrill MD. The descriptive component display theory. Englewood Cliffs 

(NJ): Educational Technology Publication; 1994.

Millis K, Forsyth C, Butler H, Wallace P, Graesser A, Halpern D. Operation 

ARIES! A serious game for teaching scientific inquiry. In: Ma M, Oikonomou 

A, Lakhmi J, editors. Serious games and edutainment applications. London

(UK): Springer-Verlag; 2011. p. 169–196.

Moreno R, Mayer RE. Engaging students in active learning: the case for 

personalized multimedia messages. Journal of Educational Psychology. 2000;

92(4):724–733.
40

Olney A, D’Mello SK, Person N, Cade W, Hays P, Williams C, Graesser AC. 

Guru: a computer tutor that models expert human tutors. In: Cerri S, Clancey 

W, Papadourakis G, Panourgia K, editors. Proceedings of the 11th conference 

on intelligent tutoring systems (ITS); 2012 Jun 16–18; Crete. Berlin

(Germany): Springer; c2012. p. 256–261.

Paneva D. Use of ontology-based student model in semantic-oriented access 

to the knowledge in digital libraries. In: Proceedings of the HUBUSKA 4th

Open Workshop: Semantic Web and Knowledge Technologies Applications;

2006 Sep; Varna, Bulgaria. p. 31–41.

Puntambekar S, Hübscher R. Tools for scaffolding students in a complex 

learning environment: what have we gained and what have we missed? 

Educational Psychologist. 2005;40(1):1–12.

Ritter S, Anderson JR, Koedinger KR, Corbett A. Cognitive tutor: applied 

research in mathematics education. Psychonomic Bulletin and Review. 

2007;14:249–255.

Ross SM. Increasing the meaningfulness of quantitative material by adapting 

context to student background. Journal of Educational Psychology.

1983;75(4):519–29. 

Ross SM, McCormick D Krisak N. Adapting the thematic context of 

mathematical problems to student interests: individualized versus group-based 

strategies. Journal of Educational Research. 1986;79:245–252.

Schneider B, Wallace J, Blikstein P, Pea R. Preparing for future learning with 

a tangible user interface: the case of neuroscience. IEEE Transactions on 

Learning Technologies. 2013;6(2):117–129.

Schraw G, Crippen KJ, Hartley K. Promoting self-regulation in science 

education: metacognition as part of a broader perspective on learning. 

Research in Science Education. 2006;36(1–2):111–139. 

Shute V, Ventura M, Small M, Goldberg B. Modeling student competencies 

in video games using stealth assessment. In: Sottilare R, Graesser A, Hu X, 

Holden H, editors. Design recommendations for intelligent tutoring systems.

Volume I: learner modeling. Orlando (FL): Army Research Laboratory (US);

2013. 

Sinatra AM, Sims VK, Sottilare RA. The impact of need for cognition and selfreference on tutoring a deductive reasoning skill. Aberdeen Proving Ground 

(MD): Army Research Laboratory (US); 2014. Report No.: ARL-TR-6961.
41

Sottilare R. Special report: adaptive intelligent tutoring system (ITS) research 

in support of the Army learning model–research outline. Aberdeen Proving 

Ground (MD): US Army Research Laboratory (US); 2013 Dec. Report No.: 

ARL-SR-0284.

Sottilare R. Dimensions and challenges in domain modeling for adaptive 

training. In: Sottilare R, Sinatra A, editors. Proceedings of the 3rd Annual 

GIFT Users Symposium; 2015 June 17–18; Orlando, FL. Orlando (FL): Army 

Research Laboratory (US): 2015, in press. 

Sottilare R. Challenges in moving adaptive training and education from stateof-art to state-of-practice. In: Proceedings of developing a generalized 

intelligent framework for tutoring (GIFT): informing design through a 

community of practice. Workshop at the 17th International Conference on 

Artificial Intelligence in Education (AIED 2015); 2015 June 26; Madrid, 

Spain. p. 1–8. [accessed 2015 Sep 15]. https://giftturtoring.org/attachments 

/.../2015 _AIED_GIFT_WS7_proc.pdf.

Sottilare RA, Brawner KW, Goldberg BS, Holden HK. The generalized 

intelligent framework for tutoring (GIFT). Orlando (FL): Army Research 

Laboratory (US); 2012a.

Sottilare R, Brawner KW. A long-term learner model to drive optimal macroadaptive decisions by intelligent tutoring systems. Pensacola (FL): Florida 

Artificial Intelligence Research Society; 2014 May. 

Sottilare R, Goldberg BS, Brawner KW, Holden HK. A modular framework 

to support the authoring and assessment of adaptive computer-based tutoring 

systems (CBTS). In: Proceedings of the Interservice/Industry Training 

Simulation and Education Conference; 2012b Dec 3–7; Orlando, FL.

Sottilare R, Graesser A, Hu X, Brawner K. Introduction to authoring tools and 

methods for intelligent tutoring systems. In: Sottilare R, Graesser A, Hu X,

Brawner K, editors. Design recommendations for intelligent tutoring systems.

Volume 3: authoring tools and expert modeling techniques. Orlando (FL):

Army Research Laboratory (US): 2015.

Sottilare R, Graesser A, Hu X, Goldberg B. Preface. In: Sottilare R, Graesser 

A, Hu X, Goldberg B, editors. Design recommendations for intelligent tutoring 

systems. Volume 2: instructional management. Orlando (FL): Army Research 

Laboratory (US); 2014.
42

Sottilare R, Holden H, Goldberg B, Brawner K. The generalized intelligent 

framework for tutoring (GIFT). In: Best C, Galanis G, Kerry J, Sottilare R, 

editors. Fundamental issues in defence simulation and training. Farnham (UK): 

Ashgate Publishing; 2013a.

Sottilare R, Ragusa C, Hoffman M, Goldberg B. Characterizing an adaptive 

tutoring learning effect chain for individual and team tutoring. In: Proceedings 

of the Interservice/Industry Training Systems and Education Conference;

2013b Dec; Orlando, FL.

Steenbergen-Hu S, Cooper H. A meta-analysis of the effectiveness of 

intelligent tutoring systems on K-12 students’ mathematical learning. Journal 

of Educational Psychology. 2013;105:971–987.

Steenbergen-Hu S, Cooper H. A meta-analysis of the effectiveness of 

intelligent tutoring systems on college students’ academic learning. Journal of 

Educational Psychology. 2014;106:331–347.

Symons CS, Johnson BT. The self-reference effect in memory: a metaanalysis. Psychological Bulletin. 1997;121(3):371–394.

US Army Training and Doctrine Command. The United States Army learning 

concept for 2015. Fort Monroe (VA): TRADOC; 2011.

van de Pol J, Volman M, Beishuizen J. Scaffolding in teacher student 

interaction: a decade of research. Educational Psychology Review. 2010;22:

271–296. 

VanLehn K. The behavior of tutoring systems. International Journal of 

Artificial Intelligence in Education. 2006;16(3):227–265. 

VanLehn K. The relative effectiveness of human tutoring, intelligent tutoring 

systems, and other tutoring systems. Educational Psychologist. 

2011;46(4):197–221.

VanLehn K, Graesser AC, Jackson GT, Jordan P, Olney A, Rose CP. When 

are tutorial dialogues more effective than reading? Cognitive Science. 

2007;31:3–62.

VanLehn K, Lynch C, Schulze K, Shapiro JA, Shelby R, Taylor L. The Andes 

physics tutoring system: lessons learned. International Journal of Artificial 

Intelligence and Education. 2005;15(3):147–204.

Veletsianos G. Contextually relevant pedagogical agents: visual appearance, 

stereotypes, and first impressions and their impact on learning. Computers and 

Education. 2010;55(2):576–585. 
43

Veletsianos G, Miller C, Doering A. EnALI: a research and design framework 

for virtual characters and pedagogical agents. Journal of Educational 

Computing Research. 2009;41(2):171–194.

Vygotsky L. Zone of proximal development. Mind in society: the development 

of higher psychological processes. Cambridge (MA): Harvard University 

Press; 1978. p. 52–91. 

Walkington CA. Using adaptive learning technologies to personalize 

instruction to student interests: the impact of relevant contexts on performance 

learning outcomes. Journal of Educational Psychology. 2013;105(4):932–945.

Wang-Costello J, Goldberg B, Tarr RW, Cintron LM, Jiang H. Creating an 

advanced pedagogical model to improve intelligent tutoring technologies.

Presented at the Interservice/Industry Training, Simulation and Education 

Conference; 2013 Dec; Orlando, FL.

Zook A, Lee-Urban S, Riedl M, Holden H, Sottilare R, Brawner K. Automated 

scenario generation: toward tailored and optimized military training in virtual 

environments. FDG ’12. In: Proceedings of the International Conference on 

the Foundations of Digital Games; 2012 May 29–June 1; Raleigh, NC. p. 164–

171.
44

Bibliography

Anderson JR, Corbett AT, Koedinger KR, Pelletier R. Cognitive tutors: lessons 

learned. The Journal of the Learning Sciences. 1995;4:167–207.

Anderson LW, Krathwohl DR, editors. A taxonomy for learning, teaching and 

assessing: a revision of Bloom’s taxonomy of educational objectives: complete 

edition. New York (NY): Longman; 2001.

Atkinson RK, Renkl A, Merrill MM. Transitioning from studying examples to 

solving problems: effects of self-explanation prompts and fading worked-out 

steps. Journal of Educational Psychology. 2003;95(4):774–783.

Bloom BS, Engelhart MD, Furst EJ, Hill WH, Krathwohl DR. Taxonomy of 

educational objectives: the classification of educational goals. Handbook I: 

cognitive domain. New York (NY): David McKay Co.; 1956.

Brawner K, Holden H, Goldberg B, Sottilare R. Recommendations for modern 

tools to author tutoring systems. Proceedings of the Interservice/Industry 

Training Simulation and Education Conference; 2012 Dec; Orlando, FL.

Burke S, Sottilare R, Salas E, Johnston J, Sinatra A, Holden H. Toward a 

scientifically rooted design architecture of team process and performance 

modeling in adaptive, team-based intelligent tutoring systems: methodology 

for literature review and meta-analysis, and preliminary results. Aberdeen 

Proving Ground (MD): Army Research Laboratory (US); 2015, in press. 

Clark RC, Nguyen F, Sweller J. Efficiency in learning: evidence-based guidelines 

to manage cognitive load. San Francisco (CA): Pfeiffer; 2006.

Dempster FN. The spacing effect. American Psychologist. 1988;43:627–634.

Dillon TJ. Questioning and teaching: a manual of practice. New York (NY): 

Teachers College Press; 1988.

Goldberg B, Brawner K, Sottilare R, Tarr R, Billings D, Malone M. Use of 

evidence-based strategies to expand extensibility of adaptive tutoring 

technologies. Proceedings of the Interservice/Industry Training Simulation 

and Education Conference; 2012 Dec; Orlando, FL.

Graesser AC, D’Mello S. Emotions during the learning of difficult material. In: 

Ross BH, editor. Psychology of learning and motivation, volume 5. Waltham 

(MA): Academic Press; 2012. p. 183–225.
45

Graesser AC, McNamara DS. Self-regulated learning in learning environments 

with pedagogical agents that interact in natural language. Educational 

Psychologist. 2010;45:234–244. 

Holden H, Sottilare R, Goldberg B, Brawner K. Effective learner modeling for 

computer-based tutoring of cognitive and affective tasks. In: Proceedings of 

the Interservice/Industry Training Simulation and Education Conference; 2012 

Dec; Orlando, FL.

Hunt E, Minstrell J. A cognitive approach to the teaching of physics. In: McGilly 

K, editor. Classroom lessons. Cambridge (MA): MIT Press; 1994. p. 51–74.

Kokini C, Carroll M, Ramirez-Padron R, Wang X, Hale K, Sottilare R, Goldberg 

B. Quantification of trainee affective and cognitive state in real-time. In: 

Proceedings of the Interservice/Industry Training Simulation and Education 

Conference; 2012 Dec; Orlando, FL.

Krathwohl DR, Bloom BS, Masia BB. Taxonomy of educational objectives.

Handbook II: affective domain. New York (NY): David McKay Co.; 1964.

Lesgold AM, Lajoie S, Bunzo M, Eggan G. Sherlock: a coached practice 

environment for an electronics trouble shooting job. Pittsburgh (PA):

University of Pittsburgh Learning Research and Development Center; 1988.

Matthews MD. Head strong: how psychology is revolutionizing war. New York 

(NY): Oxford University Press; 2014. p. 204.

Murray T. Authoring intelligent tutoring systems: an analysis of the state of the art. 

International Journal of Artificial Intelligence in Education. 1999;10(1):98–

129. 

Murray T. An overview of intelligent tutoring system authoring tools: updated 

analysis of the state of the art. In: Murray T, Blessing S, Ainsworth S, editors. 

Authoring tools for advanced technology learning environments. Berlin 

(Germany): Springer; 2003. p. 491–545.

Murray T. Theory-based authoring tool design: considering the complexity of tasks 

and mental models. In: Sottilare R, Graesser A, Hu X, Brawner K, editors.

Design recommendations for intelligent tutoring systems. Volume 3: authoring 

tools and methods. Orlando (FL): Army Research Laboratory (US); 2015. 

Soller A. Supporting social interaction in an intelligent collaborative learning 

system. International Journal of Artificial Intelligence in Education. 

2001;12(1):40–62.
46

Sottilare R, Gilbert S. Considerations for tutoring, cognitive modeling, authoring 

and interaction design in serious games. Presented at Authoring Simulation 

and Game-Based Intelligent Tutoring Workshop at the Artificial Intelligence 

in Education Conference; 2011 June; Auckland, New Zealand. 

Sottilare R, Goldberg B. Designing adaptive computer-based tutors to accelerate 

learning and facilitate retention. Journal of Cognitive Technology: 

Contributions of Cognitive Technology to Accelerated Learning and 

Expertise. 2012;17(1):19–34.

Sottilare R, Holden H, Brawner K, Goldberg, B. Challenges and emerging concepts 

in the development of adaptive, computer-based tutoring systems for team 

training. In: Proceedings of the Interservice/Industry Training Simulation and 

Education Conference; 2011 Dec; Orlando, FL.
47

List of Symbols, Abbreviations, and Acronyms

σ sigma

AAR after-action review

AI artificial intelligence

ALM US Army Learning Model

ARL US Army Research Laboratory

ATLEC Assistive Technology Learning through a Unified Curriculum

DIS distributed interactive simulation

EMAP Engine for Management of Adaptive Pedagogy

GIFT Generalized Intelligent Framework for Tutoring

ITS intelligent tutoring system

KSA knowledge, skill, and ability

LEM Learning Effect Model

MOS military occupational specialty

QandA question and answer

SME subject matter expert

SRL self-regulated learning

TandE training and education

BVS3 Virtual Battlespace 3

WFO Warfighter outcome

ZPD zone of proximal development
48

1 DEFENSE TECHNICAL

(PDF) INFORMATION CTR

DTIC OCA

2 DIRECTOR

(PDF) US ARMY RESEARCH LAB

RDRL CIO LL

IMAL HRA MAIL &

RECORDS MGMT

1 ARMY RSCH LAB – HRED

(PDF) RDRL HRM D

T DAVIS

BLDG 5400 RM C242

REDSTONE ARSENAL AL 

35898-7290

1 ARMY RSCH LAB – HRED

(PDF) RDRL HRS EA DR V J RICE

BLDG 4011 RM 217

1750 GREELEY RD

FORT SAM HOUSTON TX 

78234-5002

1 ARMY RSCH LAB – HRED

(PDF) RDRL HRM DG K GUNN

BLDG 333

PICATINNY ARSENAL NJ

07806-5000

1 ARMY RSCH LAB – HRED

(PDF) ARMC FIELD ELEMENT

RDRL HRM CH C BURNS

THIRD AVE 

BLDG 1467B RM 336

FORT KNOX KY 40121

1 ARMY RSCH LAB – HRED

(PDF) AWC FIELD ELEMENT

RDRL HRM DJ D DURBIN

BLDG 4506 (DCD) RM 107

FORT RUCKER AL 36362-5000 

1 ARMY RSCH LAB – HRED

(PDF) RDRL HRM CK J REINHART

10125 KINGMAN RD BLDG 317

FORT BELVOIR VA 22060-5828

1 ARMY RSCH LAB – HRED

(PDF) RDRL HRM AY M BARNES

2520 HEALY AVE 

STE 1172 BLDG 51005

FORT HUACHUCA AZ 

85613-7069

1 ARMY RSCH LAB – HRED

(PDF) RDRL HRM AP D UNGVARSKY

POPE HALL BLDG 470 

BCBL 806 HARRISON DR

FORT LEAVENWORTH KS 

66027-2302

1 ARMY RSCH LAB – HRED

(PDF) RDRL HRM AR J CHEN

12423 RESEARCH PKWY

ORLANDO FL 32826-3276

1 ARMY RSCH LAB – HRED

(PDF) HUMAN SYSTEMS

INTEGRATION ENGR

TACOM FIELD ELEMENT

RDRL HRM CU P MUNYA

6501 E 11 MILE RD 

MS 284 BLDG 200A 

WARREN MI 48397-5000

1 ARMY RSCH LAB – HRED

(PDF) FIRES CTR OF EXCELLENCE 

FIELD ELEMENT

RDRL HRM AF C HERNANDEZ

3040 NW AUSTIN RD RM 221

FORT SILL OK 73503-9043

1 ARMY RSCH LAB – HRED

(PDF) RDRL HRM AV W CULBERTSON

91012 STATION AVE 

FORT HOOD TX 76544-5073

1 ARMY RSCH LAB – HRED

(PDF) RDRL HRM DE A MARES

1733 PLEASONTON RD BOX 3

FORT BLISS TX 79916-6816

8 ARMY RSCH LAB – HRED

(PDF) SIMULATION & TRAINING

TECHNOLOGY CENTER

RDRL HRT COL G LAASE

RDRL HRT I MARTINEZ

RDRL HRT T R SOTTILARE

RDRL HRT B N FINKELSTEIN

RDRL HRT G A RODRIGUEZ

RDRL HRT I J HART

RDRL HRT M C METEVIER

RDRL HRT S B PETTIT

12423 RESEARCH PARKWAY

ORLANDO FL 32826
49

1 ARMY RSCH LAB – HRED

(PDF) HQ USASOC

RDRL HRM CN R SPENCER

BLDG E2929 DESERT STORM DR

FORT BRAGG NC 28310

1 ARMY G1

(PDF) DAPE MR B KNAPP

300 ARMY PENTAGON 

RM 2C489

WASHINGTON DC 20310-0300

ABERDEEN PROVING GROUND

12 DIR USARL

(PDF) RDRL HR

L ALLENDER

P FRANASZCZUK

R SOTTILARE

RDRL HRM

K MCDOWELL

D HEADLEY

RDRL HRM AL

P SAVAGE-KNEPSHIELD

C PAULILLO

RDRL HRM B

J GRYNOVICKI

RDRL HRM C

L GARRETT

RDRL HRS

J LOCKETT

RDRL HRS B

M LAFIANDRA

RDRL HRS D

A SCHARINE
50

INTENTIONALLY LEFT BLANK.