Production and Distribution Notes

This is a publication by U.S. Government programs with external contributions. 

The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of the Department of Defense, U.S. Government, or other governmental entities. This work is available under the Creative Commons Attribution 4.0 license (CC BY 4.0 

IGO) https://creativecommons.org/licenses/by/4.0. Under this license, you are free to copy, 

distribute, transmit, and adapt this work, including for commercial purposes, under the following conditions:

Attribution—Please cite the work as follows: Walcutt, J.J. & Schatz, S. (Eds.) (2019). 

Modernizing Learning: Building the Future Learning Ecosystem. Washington, DC: 

Government Publishing Office. License: Creative Commons Attribution CC BY 4.0 IGO

Adaptations—If you create an adaptation of this work, please add the following disclaimer 

along with the attribution: This is an adaptation of an original work by the Advanced Distributed Learning (ADL) Initiative, part of the Office of the Deputy Assistant Secretary of 

Defense for Force Education and Training. Views and opinions expressed in the adaptation 

are the sole responsibility of the author or authors of the adaptation and are not endorsed by 

the U.S. Government.

ePub format

• eBook GPO Stock Number: 008-300-00197-2

• eBook ISBN: 978-0-16-095091-9

PDF format

• PDF GPO Stock Number: 008-300-00198-1

• PDF ISBN: 978-0-16-095092-6

Print format

• Print GPO Stock Number: 008-000-01329-2 

• Print ISBN: 978-0-16-095088-9

Acknowledgments

The research for and publication of this book were sponsored by the Advanced Distributed 

Learning (ADL) Initiative, a research and development program reporting to the Office of 

the Deputy Assistant Secretary of Defense for Force Education and Training, part of the U.S. 

Department of Defense.

Visual design: Sae Schatz and Elizabeth A. Bradley
Education is the answer to 

everything. It’s the door opener;

it opens your mind to the possible. 

Alfred Harms, Jr. 

Vice Admiral, U.S. Navy (Ret.)

President, Lake Highland Preparatory School

Special Assistant to the President, University of Central Florida 

VP for Strategy, Marketing, Communications and Admissions, UCF
Senior Editors

J.J. Walcutt, Ph.D., Director of Innovation, ADL Initiative

Sae Schatz, Ph.D., Director, ADL Initiative

Editorial Board

Xiangen Hu, Ph.D., Professor, The University of Memphis

Van Brewer, Ph.D., External R&D Principal (Contractor), ADL Initiative

Jody Cockroft, Research Specialist, The University of Memphis

Katie Flinn, Project Analyst (Contractor), ADL Initiative

Contributors (in addition to the authors listed in the table of contents)

Academia

David Munson, Ph.D., President, Rochester Institute of Technology 

Christopher Guymon, Ph.D., Interim Dean, Graham School, University of Chicago

Christopher Dede, Ed.D., Wirth Professor in Learning Technologies, Harvard University

Susan Singer, Ph.D., VP for Academic Affairs and Provost, Rollins College

Martin Kurzweil, J.D., Director, Educational Transformation Program, Ithaka S+R

Melina Uncapher, Ph.D., Director of Education Program, Neuroscape, UCSF

Benjamin Nye, Ph.D., Director of Learning, ICT, University of Southern California

Kurt VanLehn, Ph.D., Professor, Arizona State University

Education

Ken Wagner, Ph.D., Education Commissioner, Rhode Island Department of Education 

Daniel French, Secretary of Education, Vermont Agency of Education

Nathan Oakley, Ph.D., Chief Academic Officer, Mississippi Department of Education

Keith Osburn, Ed.D., Assoc. Superintendent, Georgia Virtual Learning, Georgia DoEd

Kimberly Eckert, Teacher, Brusly High, Louisiana State Teacher of the Year

Michelle Cottrell-Williams, Teacher, Wakefield High, Virginia State Teacher of the Year 

Sandra Maldonado-Ross, President, Seminole Education Association, Florida

Sue Carson, Former President, Seminole Education Association, Florida

Government

Heidi Schweingruber, Ph.D., Director, Board on Science Ed., National Research Council

Suzanne Logan, Ed.D., Director, Center for Leadership Dev., Federal Executive Institute

Reese Madsen, SES, Senior Advisor to the U.S. Chief Human Capital Officers Council

Edward Metz, Ph.D., Research Scientist, U.S. Dept. of Education; Projects that Work

Erin Higgins, Ph.D., Research Analyst, U.S. Department of Education
Government (Cont.)

Pam Frugoli, Work Analyst, O*NET Competency Model, U.S. Department of Labor

Doug Tharp, Senior Learning Project Manger, Nuclear Regulatory Commission

Andrew Brooks, Ph.D., Chief Data Scientist, National Geospatial-Intelligence Agency

Military

Fred Drummond, SES, Deputy Asst. Secretary of Defense for Force Education & Training

VADM Alfred Harms, Jr., USN (Ret.), Lake Highland Prep School; UCF

Gladys Brignoni, Ph.D., SES, Deputy Commander, FORCECOM, U.S. Coast Guard

Lt. Gen. Thomas Baptiste, USAF (Ret.), President, National Center for Simulation

Maj. Gen. Thomas Deale, USAF (Ret.), Former Vice Director, Joint Force Development

RADM James Robb, USN (Ret.), President, National Training and Simulation Assoc.

Morgan Plummer, Director, MD5, U.S. Department of Defense

Ralucca Gera, Ph.D., Associate Provost and Professor, Naval Postgraduate School

LTC Michelle Isenhour, Ph.D., Assistant Professor, Naval Postgraduate School 

Dennis Mills, Program Analyst, Naval Education and Training Command

Larry Smith, Technical Director, USMC College of Distance Education and Training

Non-Profit Organizations

Bror Saxberg, Ph.D., M.D., VP, Learning Scientist, Chan Zuckerberg Initiative

Russel Shilling, Ph.D., Chief Scientific Officer, American Psychological Association

Jason Tyszko, VP, Center for Education and Workforce, U.S. Chamber of Commerce

Elliott Masie, Founder, The MASIE Center

Amber Garrison Duncan, Ph.D., Strategy Director, Lumina Foundation

Emily Musil Church, Ph.D., Executive Director of the Global Learning XPRIZE

Betty Lou Leaver, Ph.D., Director, The Literary Center

Jeffrey Borden, Ed.D., Executive Director, Inter-Connected Education

Jeanne Kitchens, Credential Engine and Southern Illinois University

Industry

John Landwehr, Vice President and Public Sector Chief Technical Officer, Adobe

Phill Miller, Chief Learning and Innovation Officer, Blackboard

Shantanu Sinha, Director, Product Management, Google

Michelle Barrett, Ph.D., VP of Research Technology, Data Science, and Analytics, ACT

Anne Little, Ph.D., Vice President, Training Solutions Development

Stacey Poll, U.S. Public Sector Business Development Manager, Questionmark

Michael Freeman, Consultant, Training and Learning Technologies 

Michael Smith, Senior Technical Specialist, ICF
CONTENTS

FOUNDATIONS

01 Modernizing Learning.................................................................... 3

J.J. Walcutt, Ph.D., Director of Innovation, ADL Initiative

Sae Schatz, Ph.D., Director, ADL Initiative, Office of the Deputy Assistant 

Secretary of Defense for Force Education and Training

02 History of Distributed Learning .................................................. 17

Arthur Graesser, Ph.D., Professor, The University of Memphis

Xiangen Hu, Ph.D., Professor, The University of Memphis

Steve Ritter, Ph.D., Co-Founder/Chief Product Architect, Carnegie Learning

03 Distributed Learning Instructional Theories ............................. 43

Scotty D. Craig, Ph.D., Associate Professor, Arizona State University

Ian Douglas, Ph.D., Executive Director, Institute for the Science of 

Teaching and Learning, Arizona State University

04 Lifelong Learning.......................................................................... 61

J.J. Walcutt, Ph.D., Director of Innovation, ADL Initiative

Naomi Malone, Ph.D., Research Scientist (Contractor), ADL Initiative 

05 Learning Experience Design........................................................ 83

Sae Schatz, Ph.D., Director, ADL Initiative

TECHNOLOGY

06 Interoperability........................................................................... 107

Brent Smith, R&D Principal (Contractor), ADL Initiative

Prasad Ram, Ph.D., Founder and CEO, Gooru 

07 Data Security .............................................................................. 129

Justin M. Pelletier, Ph.D., Business Director, Eaton Cybersecurity 

SAFE Lab, Rochester Institute of Technology 

08 Learner Privacy ........................................................................... 143

Bart P. Knijnenburg, Ph.D., Assistant Professor, Clemson University

Elaine M. Raybourn, Ph.D., Scientist, Sandia National Laboratories

09 Analytics and Visualization....................................................... 163

Shelly Blake-Plock, President and CEO, Yet Analytics, Inc.

10 Personalization ............................................................................181

Jeremiah Folsom-Kovarik, Ph.D. Lead Scientist, Soar Technology, Inc.

Dar-Wei Chen, Ph.D., Research Scientist, Soar Technology, Inc.

Behrooz Mostafavi, Ph.D., Research Scientist, Soar Technology, Inc.

Michael Freed, Ph.D., Consultant, Reperio
LEARNING SCIENCE

11 Assessment and Feedback........................................................ 203

Debra Abbott, Ph.D., Metis Solutions contractor, Joint Special Operations 

University, U.S. Special Operations Command

12 Instructional Strategies for the Future .................................... 223

Brenda Bannan, Ph.D., Associate Professor, George Mason University

Nada Dabbagh, Ph.D., Associate and Director, George Mason University

J.J. Walcutt, Ph.D. Director of Innovation, ADL Initiative

13 Competency-Based Learning ................................................... 243

Matthew C. Stafford, Ph.D., Chief Learning Officer, Air Training 

and Education Command, U.S. Air Force

14 Social Learning ........................................................................... 269

Julian Stodd, Founder, Sea Salt Learning

Emilie Reitz, Bold Quest Analytical Working Group Lead, Joint Staff J6 

(Joint Fires Division), U.S. Department of Defense

15 Self-Regulated Learning ........................................................... 285

Louise Yarnall, Ph.D., Sr. Research Social Scientist, Center for Technology in Learning

Michael Freed, Ph.D., Consultant, Reperio

Naomi Malone, Ph.D., Research Scientist (Contractor), ADL Initiative

ORGANIZATION

16 Instructional Designers and Learning Engineers .................... 301

Dina Kurzweil, Ph.D., Director, Education and Technology Innovation Support Office, 

Uniformed Services University of the Health Sciences, U.S. Department of Defense

Karen Marcellas, Ph.D., Instructional Design Team Lead, Education and Technology 

Innovation Support Office, Uniformed Services University

17 Governance for Learning Ecosystems.......................................317

Thomas Giattino, Chief, Technology Integration Division, Air Education and Training Command

Matthew Stafford, Ph.D. Chief Learning Officer, Air Education and Training Command

18 Culture Change ........................................................................... 339

Scott Erb, Captain, U.S. Navy (Ret.), Former Commanding Officer, Center for Security Forces

Rizwan Shah, Organizational Culture Advisor, Department of Energy

19 Strategic Planning ..................................................................... 357

William Peratino, Ph.D., Deputy Director, USALearning, U.S. Office of Personnel Management 

Mitchell Bonnett, Ph.D., Distributed Learning Research, Standards & Specifications, 

Directorate of Distributed Learning, Army University, U.S. Army

Dale Carpenter, Superintendent (Acting), U.S. National Park Services

Yasir Saleem, Senior Solutions Consultant, Adobe

Van Brewer, Ph.D., External R&D Principal (Contractor), ADL Initiative

Endnotes ................................................................................................ 388
ADDIE Analyze, Design, Develop, Implement, and Evaluate

ADKAR Awareness, Desire, Knowledge, Ability, Reinforcement 

ADL Advanced Distributed Learning

AI Artificial Intelligence

API Application Programming Interface

AR Augmented Reality

ASVAB Armed Services Vocational Aptitude Battery 

BYOD Bring Your Own Device 

cMOOC Connectivist Massive Open Online Course

CORDRA Content Object Repository Registration/Resolution Architecture

DARPA Defense Advanced Research Projects Agency

DHS Department of Homeland Security

DIS Distributed Interactive Simulation

DoD Department of Defense

EEG Electroencephalogram

EMT Emergency Medical Technician

ESSA Every Student Succeeds Act

FAA Federal Aviation Administration

FATE Fairness, Accountability, Transparency, and Ethics

FERPA Family Educational Rights and Privacy Act

FM Field Manual

fMRI Functional Magnetic Resonance Imaging

FYI For Your Information

GIFT Generalized Intelligent Framework for Tutoring

HLA High-Level Architecture

HR Human Resources

HSI Human–Systems Integration

HTML Hypertext Markup Language

HTTP Hypertext Transfer Protocol

I/ITSEC Interservice/Industry Training, Simulation and Education Conference

ICAP Interactive, Collaborative, Active, and Passive

ICICLE Industry Connections Industry Consortium on Learning Engineering

IDS Intrusion Detection System

IEC International Electrotechnical Commission

IEEE Institute of Electrical and Electronics Engineers

IEEE-SA IEEE Standards Association

InKD Industrial Knowledge Design

IoT Internet of Things

IPS Intrusion Prevention System

ISD Instructional Systems Design

ISO International Standards Organization

IT Information Technology

ACRONYMS
K-12 Kindergarten through 12th Grade

KPI Key Performance Indicator

LMS Learning Management System

LOM Learning Object Metadata

LRMI Learning Resource Metadata Initiative

LRS Learning Record Store

LX Learning Experience

LXD Learning Experience Design

MERLOT Multimedia Education Resource for Learning and Online Teaching

MOOC Massive Open Online Course

MSSP Managed Security Services Provider

NASA National Aeronautics and Space Administration

NGO Non-Governmental Organization

NYCRR New York Codes, Rules and Regulations

OECD Organisation for Economic Co-operation and Development

OER Open Educational Resources

OPM Office of Personnel Management

PERLS PERvasive Learning System

PII Personally Identifiable Information

PLATO Programmed Logic for Automatic Teaching Operations

R&D Research and Development

RFID Radio Frequency Identification

ROI Return on Investment

SaaS Software as a Service

SAKI Self-Adaptive Keyboard Instructor

SAMR Substitution Augmentation Modification Redefinition

SAT Scholastic Assessment Test

SCORM Shareable Content Object Reference Model

SIEM Security Incident and Event Management

SOC Security Operations Centers

STEM Science, Technology, Engineering, and Mathematics

TAPAS Tailored Adaptive Personality Assessment System

TECOM Training and Education Command (part of the U.S. Marine Corps)

TED Technology, Entertainment, and Design

UI/UX User Interface/User Experience

VR Virtual Reality

xAPI Experience Application Programming Interface

XML Extensible Markup Language

xMOOC Extended Massive Open Online Course

Foundations Foundations

Modernizing Learning | 3 

CHAPTER 1

MODERNIZING LEARNING

J.J. Walcutt, Ph.D. and Sae Schatz, Ph.D.

The 21st century is marked by significant technological progress in every field. 

For learning and development, these advancements have helped us realize the 

promise of “anytime, anywhere” learning as well as learning personalized to 

individual needs. More than that, emerging capabilities have thrown open the 

door to transformative possibilities, facilitating learning at scale, optimizing 

learning in response to large and diverse data sets, and developing fully integrated talent management systems for managing and enhancing the future 

workforce.

Emerging technologies are not only changing the formal education and training landscape, they’re also changing our access to—and relationship with—

information and, by extension, affecting the soul of how we think, interact, 

develop, and work. Our expectations for educational institutions, how and 

where learning occurs, and what personal developmental looks like have 

changed—and will continue to evolve into the future. The preK–12 system, 

higher education, federal and state governments, employers, and military 

must similarly adapt to accommodate. 

The landscape of learning has broadened, now encompassing the full spectrum of formal, informal, and experiential training, education, and development. The traditional concept of education is changing. Employers are placing 

less value on formal degrees. Instead, experience matters. Life skills, such as 

grit and teamwork, matter. Performance-based credentials, including competency badges and micro-certificates, are taking the place of transcripts to 

document individuals’ traits, talents, skills, knowledge, preferences, and ex-

 

 
4 | Modernizing Learning

perience. Similarly, age is becoming less of a marker of knowledge, skill, and 

capabilities. These shifts, in turn, are disrupting conventional career trajectories, as age correlates less and less with income and leadership potential, 

and even changing the way we perceive employment and define our value as 

contributors to our society.

We use the phrase “future learning ecosystem” to describe this new tapestry of learning. At the highest level, the future learning ecosystem reflects a 

transformation—away from disconnected, episodic experiences and towards 

a curated continuum of lifelong learning, tailored to individuals, and delivered 

across diverse locations, media, and periods of time. Improved measures and 

analyses help optimize this system-of-systems and drive continuous adaptation and optimization across it. Its technological foundation is an “internet for 

learning” that not only allows ubiquitous access to learning, it also provides 

pathways for optimizing individual and workforce development at an unprecedented pace.

This book focuses on the human and organizational aspects of the future 

learning ecosystem. It provides key terms and models, and it helps identify 

the diverse professional sectors involved in the realization of this vision.

The United States Government has recognized a need for coordination among 

the communities of learning scientists, organizational psychologists, software 

and hardware engineers, teachers, talent managers, administrators, and other 

innovators contributing to this concept. Simply organizing the multiple, interdependent layers of the future learning ecosystem represents an enormous 

undertaking, more so because its many facets must evolve in concert. Improving school classrooms, for instance, means little unless we also transform how 

those experiences translate to collegiate, trade, business, and public-sector 

settings. Similarly, developing systems for earning and communicating credentials creates scant value, unless we also understand how to authentically 

measure the skills and attributes they accredit. And finally, even if we successfully reshape every aspect of our learning and development systems, we 
Modernizing Learning | 5 

must simultaneously consider the larger cultural and societal shifts affected 

by this new approach. How will the reconceptualization of learning affect 

jobs, self-worth, loyalty to businesses, power dynamics, access to education, 

governmental processes, and our nation overall? When the paradigm of learning (something so fundamental to each of our lives) evolves it will have expansive and exciting, but difficult to fully forecast, effects.

WHAT IS LEARNING?

At its most foundational level, learning is any change in long-term memory 

that affects downstream thoughts or behaviors. The process of learning starts 

with awareness of stimuli,1 cognitive encoding of that information,2 and its 

retention in memory. Later, the knowledge must be retrievable (that is, not forgotten) and transferable  to novel situations.3 Throughout our lives, every person learns constantly—all the time, every day. What we each learn, however, 

its veracity, applicability, intelligibility, and whether it aids or limits performance all vary significantly. Each day, we must reconcile among the complex, 

competing information vying for our attention—all vying to “teach” us.

The future learning ecosystem—a 

holistic, lifelong, personalized learning 

paradigm—represents a contrast to the 

Industrial Age model of time-focused, 

one-size-fits-all learning
Contributions from diverse fields—including 

IT, data science, psychology, and learning 

science—form a repository of complementary 

recommendations; together, these define the 

framework of the future learning ecosystem 

These myriad science and 

technology advancements form the 

alloy needed to develop optimized 

learning solutions that maximize 

efficiency while expanding 

effectiveness
Modernizing Learning | 7 

The concept of learning applies across performance domains, not only to cognitive development. It necessarily includes physical and emotional aspects as 

well as inter- and intrapersonal, social, and cultural components. Certainly, 

learning occurs in formal settings, in grade school classrooms or professional 

workshops, but it also happens in self-directed, just-in-time, social, experiential, and other informal ways.4 These varied experiences accumulate in longterm memory and, fused together, affect how we respond to the world.5 In 

other words, formal learning in combination with other life experiences collectively determines someone’s readiness for work, public service, and other 

life challenges. 

Surfacing the Iceberg

To date, our education and training systems have generally focused on the 

delivery and documentation of formal learning. As a result, we’ve fostered a 

society that values the accreditation of formal training and education (think 

college degrees) and proxy measures of aptitude (time-based promotions) 

rather than life experiences and direct measures of competence. Of course, 

this is based largely on our inability to measure, analyze, and share data about 

the latter. With advances in technology, however, we’re surfacing informal 

learning.

In talking about learning, enough with the barriers. We’re 

interested in outcomes. I want effective learning. I want 

measurable learning. I want learning that results in combat 

capability. That’s what we’re looking at, in terms of learning science, 

from our perspective inside the Pentagon. That’s where I’m pushing 

our folks.

Fred Drummond, Deputy Assistant Secretary of Defense for Force 

Education and Training, U.S. Department of Defense
8 | Modernizing Learning

The growing visibility of, and access to, informal learning is reshaping our 

conceptualization of learning: Increasingly away from a separate, fenced-off 

and time-based activity and towards an integrated, diverse lifelong learning 

continuum where all experiences and development add to an interdependent 

set of holistic competencies. This paradigm shift means education is no longer 

viewed as a linear and finite pathway, starting in grade school and culminating with a high school or university degree. Books and teachers, and other 

hierarchical authorities, are no longer the primary gatekeepers of knowledge. 

Vocational schools and formal apprenticeships no longer serve as the primary 

pathways to develop trade skills. Individuals can even cultivate their athletic 

abilities through self-developed and informal learning channels.

Informal learning means more than just self-directed study. Consider, for instance, when a young person travels overseas for the first time. Perhaps without intention, she learns about other cultures, people, history, and food, as 

well as other, more subtle lessons about social dynamics, cosmopolitanism, 

and even self-awareness. Undoubtedly, such experiences are learning, that 

is, they impact long-term memory and change us. But how might society, 

teachers, or employers value such learning? How do we record or account for 

such experiences? How can we define and measure such seemingly intangible 

qualities, such as worldliness, emotional maturity, or empathy?

21st Century Competencies

Elusive personal characteristics, such as good judgment and social awareness, have always mattered. Increasingly, however, pundits are emphasizing 

new capabilities that reflect the changing demands of the world. Automation 

driven by artificial intelligence, ever-increasing computing power, big data, 

advanced robotics, and the proliferation of low-cost advanced technologies 

are the shifting nature of work, along with the organizational dynamics of 

business, government, and society. 
Technology is replacing the physical—

and intellectual—tasks of many professions, from bus drivers and construction 

workers to medics and lawyers. Jobs involving manual labor, memorizing procedures, calculating solutions, and even synthesizing diverse information into novel 

forms are fast becoming the purview of 

computers. Meanwhile human work increasingly focuses on social and cultural 

factors, creativity and creative problem 

solving, digital literacy and technology 

partnership, and rapid adaptability. Modern core competencies tend to emphasize 

higher-order, more nuanced and sophisticated capabilities in lieu of fact-based 

knowledge or procedural skills. Similarly, where in the more recent past, highly 

skilled professionals typically advanced 

by focusing on narrow disciplines, today’s savants are often “expert generalists” able to synthesize across disciplines, 

learn new concepts and contexts rapidly, and adapt to changing conditions.

In contrast to prior decades, there’s a greater expectation for individuals to 

learn continuously and develop new capabilities across their entire careers. 

In large part, this is spurred by the rapidly changing world around us. Pulitzer Prize winning author Thomas Friedman has dubbed this time-frame the 

“Age of Acceleration,” reflecting the exponential growth in technology and 

unbridled transformation across the globe.6 To excel in this age, we must learn 

to thrive in volatility and complexity. We need deep understanding, across a 

range of cognitive, affective, interpersonal, and physical competences, and 

There’s a foundational set 

of cognitive, intrapersonal, 

and interpersonal skills 

that provide the flexibility, 

adaptivity, and capability 

people need to navigate 

through the kind of constant 

change, discontinuous, 

and sometimes irrational 

situations that pervade the 

21st century. 

Education should focus on 

that, much more than it has 

in recent years, because if 

we don’t make that shift, 

we’ll develop a very brittle 

set of people at a time when 

adaptability will be core for 

their survival.

Christopher Dede, Ed.D.

Timothy E. Wirth Professor in Learning 

Technologies in the Technology, Innovation, 

and Education Program, Harvard University
10 | Modernizing Learning

refresh those capabilities as situations evolve. We need to think in terms of 

system dynamics, applying a strategic understanding of complex systems and 

the far-reaching effects of actions taken within them. Organizations, too, must 

learn to shift and grow with evolving needs, rapidly capturing and integrating 

lessons learned and enabling the disseminate of new ideas painlessly across 

their enterprises.

In short, to develop and maintain 21st century competencies, individuals require a greater breadth of interdependent knowledge and skills, at an increased 

depth, that is, more advanced levels of nuanced capabilities, and these competencies must be acquired at a more rapid velocity. To meet such demands, we 

must embrace continuous learning, find more efficient ways to develop and 

maintain relevant knowledge and skills, and develop reliable feedback loops 

that ensure our systems remain relevant in our ever-changing environment. In 

other words, we must profoundly redesign the integrated continuum of formal 

and informal training, education, and experience. 

FUTURE LEARNING 

ECOSYSTEM

The future learning ecosystem is a substantive reimagination of learning and 

development. This concept recognizes the increasing need for cognitive agility, meaning learning is no longer viewed as a single event—nor even a series 

of events—but rather as a lifelong experience of continual growth. Second, 

the pathways through which learners progress must be personalized to their 

unique attributes, skills, interests, and needs in order to achieve necessary 

effectiveness and efficiency in learning. Finally, instruction and information 

presentation methods must more strongly emphasize deep learning and expedite the transfer of learning from practice to real-world settings.

7
Modernizing Learning | 11 

Extensive research, across myriad disciplines, has already examined many 

aspects of the future learning ecosystem. However, to achieve its full implementation and maximal benefits, it’s necessary to harmonize the advancements in learning science, technology, data science, organizational dynamics, 

and public policy. 

Technological Infrastructure

Information technology forms the enabling foundation of the future learning ecosystem. Instructional systems, interoperability standards, cross-platform data integration, and centralized software services form the sinews and 

nerves that transform today’s stovepiped, staccato learning episodes into a 

holistic lifelong experience. Data schemata, technical standards, and governance conventions enable the recording, aggregation, and analysis of diverse 

learning events—opening the possibility for substantial personalization and 

data-driven enterprise adaptations. In other words, an integrated, technological-enabled learning architecture unlocks the anticipated transformation in 

learning. It means that learning can become pervasive—truly accessible anytime, anywhere, in many forms, and for many functions; and accordingly, 

learning can be tailored for optimal effect. 

Design

Where technology will open a new world of learning possibilities, learning 

science and learning engineering—the thoughtful design of learning components and systems—will allow us to capitalize on it. The future learning 

ecosystem opens the aperture of learning and changes its core characteristics. 

The classic instructional systems design model no longer suffices. The design 

of learning, at both the local and enterprise levels, will need new theories and 

practices. Learning designers will need to understand how to differentially 

apply diverse technologies, blend disparate delivery modalities into holistic 
To realize the future learning 

ecosystem vision, six critical 

areas must align.

TECHNOLOGICAL INFRASTRUCTURE

Flexible, interoperable technologies for pervasive learning

DESIGN

Intentional methods applied to optimize learning

COMMITMENT

Contributions to a shared vision across communities

GOVERNANCE

Negotiation of standards, conventions, and ethics

POLICY

Regulations and recommendations for behavior

HUMAN INFRASTRUCTURE

Diversely skilled individuals and organizational structures
Modernizing Learning | 13 

experiences, build-in and apply learning analytics, balance practical logistics against learning outcome criteria, incorporate learning and development 

into personnel and workforce systems, and perform all these actions within a 

heterogeneous system-of-systems, which they only partially control. 

Commitment

The term “ecosystem” refers to complex, interconnected systems. In stark 

contrast to today’s more hierarchical training and education events, where the 

teacher reigns within his classroom or the trainer dictates the design of her 

curriculum, achievement of the future learning ecosystem requires collective 

coordination across diverse communities. The benefits of the future learning ecosystem can only be realized through their gestalt. Learning designers must embed ways to capture learning data, ideally using shared semantic 

vocabularies. Technology vendors must eschew proprietary, closed systems 

and embrace open architectures and interoperability standards. Early childhood educators must plan their curricula with postsecondary, workforce, and 

community intersections in mind. Parents, learners, teachers, administrators, 

human resource planners, and organizational leaders will need to buy-in to 

this concept—and actively contribute to its realization. While interoperable 

technologies may form the foundations of the future learning ecosystem foundations, the social contracts followed by the ecosystem will give it breadth 

and traction. 

Governance

The future learning ecosystem grows from organizational coordination, 

technological interoperability, and the aggregation of learning data across diverse technological and administrative boundaries. Even without (especially 

without) a hierarchical leadership structure, such a complex system requires 

sophisticated governance processes. Cross-sectional governance bodies will 
14 | Modernizing Learning

need to negotiate the conventions for sharing and protecting individuals’ data, 

for designing and updating shared application programing interfaces, and for 

balancing the competing interests of educational, commercial, and governmental organizations. Accreditation bodies will need to evolve to accommodate new types of assessments and credentials. These governance bodies will 

also have a responsibility to consider the social and societal impacts of this 

new learning system. They will need to navigate a spate of new social and 

ethical considerations, envision new legal and regulatory rules, and attempt to 

envision the emergent risks and opportunities as the system matures. While 

government will undoubtedly play a role, we—the stakeholders across highly 

The crisis across the nation is that 

there is so much disparity between 

what each child can access. The 

system has to be pervasive. 

The dream of America is that all 

Americans should have a free 

education through 12th grade.

Alfred Harms, Jr. 

Vice Admiral, U.S. Navy (Ret.); President, Lake Highland 

Preparatory School; Special Assistant to the President and 

VP for Strategy, Marketing, Communications and 

Admissions, University of Central Florida
Modernizing Learning | 15 

diverse communities—have a responsibility to actively participate in these 

governance processes. Unlike a walled garden, where appointed caretakers 

can curate the design, the future learning ecosystem requires the community 

to take an active role in steering its ecology. 

Policy

Governance bodies, along with the actual government and key performers 

within the ecosystem, will inform policies for the future learning ecosystem. 

Policy is the blueprint of recommendations and regulations that define guidelines for behavior within the system. Recommendations might include best 

practices for collecting and personalizing learning in response to data. Regulations, or rules put in place to protect the public, might include guidance on 

the privacy, ownership, and commercialization of learners’ data. Nearly all 

innovation carries a double-edged sword: Creative foresight, social accountability, and ethical principles will need to guide employment of the future 

learning ecosystem for our public sector as well as personal and business-related interests.

Human Infrastructure

Although technological advances make the future learning ecosystem possible, its implementation requires a multitude of differently skilled (human) 

contributors. Hence, as we develop its technology infrastructure, learning theories, and organization processes, we must also cultivate the future learning 

ecosystem’s critical human infrastructure. A new subdomain of technologies 

and learning-focused data scientists are clearly needed. The system will also 

require numerous insightful talent managers, learning engineers, and courseware designers. Teachers, trainers, coaches, and mentors will need to be empowered and trained to take full advantage of this new milieu of learning. 

Even individual learners will play a key role—not only in the “consumption” 
16 | Modernizing Learning

of learning but also in crowdsourced, peer-to-peer, and collaborative learning. 

The future learning ecosystem will affect us all, and in turn, we can each 

shape and contribute to it. 

Blueprint for Implementation

This book examines the future learning ecosystem concept, our collective 

progress towards its realization, and the pivot our systems and society need to 

make away from formal, detached education and training towards experiential, personalized, interconnected learning journeys. The U.S. Government’s 

ADL Initiative has taken the lead in designing this book and is helping to 

coordinate across the broad stakeholder community, both conceptually and 

practically. The following chapters in this publication provide a snapshot of 

the achievements the ADL Initiative and other contributors have made to date, 

what we need to build for tomorrow, and what this near-future system will 

enable our children, workforce, society, and military personnel to achieve.

Learning is a journey, not a destination.

It’s about the dignity of work. How do we create 

in our country a sense of work pride? We have an 

obligation and opportunity to create an environment 

where everyone has skin in the game.

U.S. Congressman Jack Bergman

Lieutenant General, U.S. Marine Corps (Ret.); 

...from a presentation at the 2018 I/ITSEC Conference
History of Distributed Learning | 17 

CHAPTER 2

HISTORY OF 

DISTRIBUTED LEARNING

Art Graesser, Ph.D., Xiangen Hu, Ph.D., 

and Steve Ritter, Ph.D. 

Learning science and associated technologies have advanced dramatically, 

and disruptively, over the last 30 years, and they will no doubt continue to 

evolve through the foreseeable future. To proceed with wisdom, it’s prudent 

to review the past and to examine how we came to our current state, what 

achievements and pitfalls we encountered, and what lessons might translate 

into the future learning ecosystem. 

This chapter specifically examines the evolution of distributed learning. Under this moniker, we’ve included related terms, often used synonymously, 

such as distance learning, distributed or distance education, web-based and 

web-enabled instruction, online learning, and e-learning—just to name a few! 

More recently, “distributed learning” has come to reference an even-wider 

perspective, sometimes incorporating concepts such as distributed simulation, mobile learning, augmented and virtual reality, computer-assisted instruction, and web-based self-directed learning. We touch on those, too. Even 

certain generic terms, such as technology-enhanced learning or educational 

technology, are sometimes used to reference distributed learning, and where 

applicable, we’ve included those concepts as well. 

Although we recognize distinctions among these terms, this isn’t an academic chapter on the nuances of vocabulary. Instead, we attempt to take readers 

on a brief journey, starting with the foundations of distributed learning and 

 

 
18 | Modernizing Learning

considering its evolutionary progress, across many different fields, towards a 

unified, technology-enabled interconnected learning paradigm. 

Certainly, others have written more robust historical accounts, for those interested in more detail. For instance, in a now classic article, Soren Niper 

outlines the three historic generations of distance education, starting with correspondence teaching, followed by multimedia offerings (e.g., cassettes and 

television broadcasts), and finally the third-generation, involving information 

and communication technologies.1 Building upon Niper’s framework, Mary 

Simpson and Bill Anderson wrote a brief and accessible overview of the “History and Heritage in Distance Education.” 2

For truly comprehensive treatments, refer to Michael Grahame Moore and 

William Anderson’s Handbook of Distance Education originally published in 

2003 (or Moore’s update of that classic in 2013).3 Also review Paul Saettler’s 

thorough examination on The Evolution of American Educational Technology 4 and J. Michael Spector and colleagues’ Handbook of Research on Educational Communications and Technology.

5

 In the latter, Michael Molenda’s 

“Historical Foundations” chapter offers a particularly readable treatment of 

the field’s development. 

1980s

In all historical accounts of distributed learning, authors seem compelled 

to highlight its analog foundations—hand-painted slides illuminated by oil 

lamps in the 17th century, correspondence learning by mail in the 18th century, or silent films in the early 20th.

6

 However, for our purposes, the history of 

distributed learning meaningfully begins in the 1980s. This decade witnessed 

the rise of personal computers, with widespread adoption in most schools 

beginning around 1983.7 Their proliferation ushered in Niper’s so-called 
History of Distributed Learning | 19 

third-generation of distance education, shifting away from “boxes of books” 

and towards computer-based learning experiences. 

Computer-based learning generically refers 

to the use of computers to access training 

and education. It can involve synchronous 

and/or asynchronous activities, delivered via 

networked or standalone stations. Early experiments in computer-based learning began 

in the late 1950s and early 1960s, with the 

University of Illinois’s PLATO project often 

cited as the first computer-based system and 

Gordon Pask and Robin McKinnon-Wood’s 

SAKI as the first adaptive trainer. SAKI, 

which stood for Self-Adaptive Keyboard Instructor, used a mechanical device to modify typing exercises in response to 

learners’ performance, typically shortening training time by one-half to twothirds as compared to conventional instructional methods.8

These experiments gave rise to the first-generation of computer adaptive tutors, often called “computer-assisted instruction tutors.” In his meta-analytic 

review of computer-assisted instruction from this time-frame, James Kulik 

found students typically performed better (with an average effect size of .35 

standard deviations), completed learning activities more efficiently (about a 

quarter to a third more quickly), and tended to have more positive outlooks 

on learning with computer-assisted instruction.9 Groundbreaking systems 

emerged around this time-frame, including intelligent tutoring systems, which 

were a substantial advance over computer-assisted instruction tutors with their 

very simple assessment, feedback, and lesson-branching rules. Landmark 

early intelligent tutors included Alan Lesgold’s SHERLOCK, John Anderson and colleagues’ LISP tutor, and John Seely Brown and Richard Burton’s 

SOPHIE.10 These systems used automated computational procedures to guide 

Student using PLATO III, 1970; 

courtesy of the University of Illinois at 

Urbana-Champaign Archives
20 | Modernizing Learning

learners through problem steps, give hints, and provide teacher-like feedback. 

The more advanced intelligent tutoring systems showed even higher learning gains, an effect size of .76 standard deviations, according to more recent 

meta-analyses conducted by James Kulik, Phil Dodds, and Dexter Fletcher.11

Many of the early instructional technologies weren’t yet distributed, but that 

was changing. Throughout the 1980s, U.S. federal agencies, including the Department of Defense, National Science Foundation, and Department of Education sponsored significant research on computer-based instruction, including distributed learning.12 In 1989, the U.S. Office of Technology Assessment 

delivered a Congressional report, called Linking for Learning, summarizing 

the progress such investments had made over the decade: 

Distance learning is expanding. ...a national survey of representative 

school districts indicated that an estimated 22 percent of school districts 

now use distance learning, some 33 percent expect to be using these 

resources by 1990. The second trend is more subtle. Distance learning is 

changing educational boundaries—boundaries traditionally defined by 

location and by institution. In the pooling of students and teachers, distance learning efforts reconfigure the ‘classroom.’ No longer bound by 

the physical space, classrooms extend to other students in the same district, to other districts, to other States, or even across national borders.13

The report also called for increased research on distributed learning, particularly regarding its effectiveness, methodology, and design. “The quality and 

effectiveness of distance learning are determined,” it explained, “by instructional design and technique, the selection of appropriate technologies, and the 

quality of interaction afforded to learners.” This was a job for instructional 

designers.

The origins of Instructional Systems Design (ISD) trace back to the 1960s, 

but the 1980s saw a proliferation of ISD models appear in the literature. 

Roughly around this time, the ADDIE concept also materialized, apparently spontaneously,14 as a generic framework underpinning the various mod-
History of Distributed Learning | 21 

els. Traditional ISD approaches grew out of the behaviorist paradigm, and 

similarly, most early computer-based learning used drill-and-practice tactics grounded 

in behaviorism.15 As Kulik observed at the 

time, “Most programs of computer tutoring 

derive their basic form from Skinner’s work 

in programmed instruction. Skinner’s model 

emphasized (a) division of instructional materials into a sequence of small steps, or instructional frames; (b) learner responses at each step; and (c) immediate feedback after each response.” 16

Some educators in this decade also advanced an industrialized model for distributed learning, as best expressed by Otto Peters. He positively compared 

distance education to industrial production, citing the division of labor, mass 

production, realization of economies of scale, and reduced unit costs. His 

model wasn’t intended as an instructional theory, but rather as an organizational concept that, in his own words, described the industrial “objectification 

of the teaching process.” 17

Nonetheless, the state of learning science in educational technology was progressing. The 1980s saw a growing influence from the cognitivist school, for 

instance, with the development of concepts such as cognitive-load theory. Although this theory’s antecedents began in the 1950s, it wasn’t until the 1980s 

that John Sweller connected those earlier cognitive principles to practical educational tactics. Based on observations of students studying, Sweller proposed 

that inherent bottlenecks in our cognitive processes create barriers to learning that teachers can mitigate through careful instructional design. In other 

words, Sweller’s theory posits that certain factors can increase our cognitive 

load and distract us from learning the relevant information; more importantly, 

his theory offered actionable recommendations to teachers and designers for 

mitigating those distractions, including implications for educational technology designers.18

ADDIE

Analyze, Design, Develop, 

Implement, and Evaluate

...an evergreen model, general enough 

to suit pretty much any process
22 | Modernizing Learning

Benjamin Bloom was also exploring the impacts of cognitive science on education. His influential research on the “two-sigma problem” attracted the 

attention of many learning researchers. Bloom found that students who receive instruction via one-on-one (human) tutoring using mastery learning 

techniques outperform those who receive group-based instruction in classrooms.19 This foundational study has become a rallying point for proponents 

of computer-based adaptive learning. 

Although Bloom’s classic study, as well as most of the computer-based learning so far, emphasized individual instruction, by the mid-1980s learning scientists had begun exploring more constructivist and collaborative techniques, 

building upon the constructivist educational theories of Jean Piaget, for example, and of collaborative constructivist Lev Vigotsky.20 The most radical 

constructivist educational theories begin with the premise that objective “reality” is unknowable, and, instead, individuals construct a subjective, contextualized reality within their own minds. Less radical constructivists still 

emphasize the active construction of knowledge that tends to settle into the 

constraints of the objective physical and social world. For educational environments, this implies that students learn best by engaging with instructional 

material, actively generating learning experiences rather than passively interpreting information. Constructivism catalyzed a change in educational theory, moving it away from instructor- and content-centric views and towards a 

learner-centric one.21 Social constructivism takes this premise a step further, 

emphasizing collaboration and the impacts of social interactions on learning 

and the construction of knowledge by groups.22

Social constructivist educational theories spurred the development of computer-supported collaborative learning, software designed to support interactive learning and computer-mediated communications. Businesses and universities began to develop communicative and educational technologies, such 

as Xerox’s NoteCards and Carnegie-Mellon University’s Andrew.23 Marlene 

Scardamalia and her colleagues from the University of Toronto also made 
History of Distributed Learning | 23 

significant impacts on this field. For instance, they experimented with computer-supported intentional learning environments that enabled collaborative 

meaning-making by helping students share ideas, pictures, and notes via network computers.24 Projects like this influenced the wider field of educational 

technology, encouraging a fundamental shift towards social learning. 

Such interest helped foster the idea of a “virtual classroom,” a multi-person 

anytime, anywhere learning environment facilitated by networked computer-mediated communications. “Suddenly it came to me,” Starr Roxanne Hiltz, 

from the New Jersey Institute of Technology, explained. “A teaching and 

learning environment did not have to be built of bricks and boards. It could 

be constructed in software. It could be Virtual! In an era when many teachers 

and students have their own microcomputers, it was no longer necessary for 

them to travel to a classroom...the classroom could come to them, over their 

telephone lines and through their computer.” 25

The digital collaborations spawned in the 1980s led to contextually rich environments in the ensuing decades. While Hiltz and her colleagues developed 

virtual classrooms, other built entire worlds. Virtual worlds, or “synchronous, persistent network[s] of people, represented as avatars, facilitated by 

networked computers” 26 and synthetic environments, or realistic simulated 

environments, similarly emerged during this era. One example of this is Michael Naimark’s concept of “surrogate travel,” virtual recreations of real environments navigable via a LaserDisc.27 Another instance is the NASA Ames 

Laboratory’s virtual reality system, which used stereoscopic head-mounted 

displays and a fiber-optic data glove. Finally, Habitat, developed by Lucasfilm 

Games in association with Quantum Computer Services, Inc., is often-cited 

as one of the first attempts to develop a large-scale, multiplayer, commercial 

virtual world.28 Such systems would require several intervening decades to 

reach fruition, but the contributions of these forerunners can’t be understated. 

While the education community developed virtual worlds and collaborative 

virtual classrooms, the training industry similarly explored collective-learn-
24 | Modernizing Learning

ing capabilities, in their case, for multi-person 

training simulations. Promoted by organizations such as NASA and the U.S. military, 

computer-supported trainers first emerged 

in the 1940s. Initially, these instructional 

simulations were used as substitutions for 

live training that was too costly, unsafe, or 

otherwise inconvenient. However, during 

the 1970s, the training community began 

to value instructional simulation beyond 

mere substitution, seeing it as a unique 

instructional tool and a potential platform 

for team-based practice. Encouraged in part 

by the demand for collective and improved 

training, researchers started developing collective, distributed simulation-based training 

technology. The Defense Advanced Research 

Projects Agency’s (DARPA) Simulation Network (SIMNET), fielded in 1987, serves as a 

notable example.29 However, distributed simulation wouldn’t become a truly 

viable learning modality until the 1990s and the rise of the global internet. 

1990s

Computer-based learning continued to expand throughout the 1990s, in conjunction with the increasing prevalence of personal computers, improvements 

in their multimedia capabilities, and advances in computer networking. Most 

notably, the 1990s were profoundly marked by the growth of the world wide 

web (invented in 1989), and with it, broad access to networked communications. 

By the end of this the 1980s, “virtual” 

exploration was demonstrated routinely 

at NASA Ames and elsewhere. The 

picture above, taken in 1990, shows 

an operator using NASA’s Virtual 

Interface Environment Workstation, 

developed by NASA and VPL Research, 

Inc.; photo courtesy of NASA.
History of Distributed Learning | 25 

The first operational web-based courses appeared in the mid-1990s, and by 

the end of the decade, around 60% of all U.S.-based universities had webbased offerings.30 Simultaneously, the e-learning industry emerged. Throughout the ‘90s, vendors developed tools to help teachers and institutions manage 

their e-learning resources. The associated software was released under a diversity of titles, including course management systems, virtual learning environments, learning platforms, and managed learning environments, as well 

as learning management systems and learning content management systems, 

which remain popular today. 

In addition to traditional e-learning, some researchers began to promote adaptive hypermedia. In contrast to typical websites, which provide the same text, 

links, and multimedia to all viewers, adaptive hypermedia systems create user 

models of each visitor and then adapt the information and links presented. 

Peter Brusilovsky and colleagues developed and tested adaptive hypermedia 

systems that integrated web communication and intelligent tutoring concepts.31

Along with adaptive hypermedia, the so-called “second-generation” of adaptive tutors—formally called intelligent tutoring systems—also matured. As 

one notable example, the cognitive tutors developed by Ken Koedinger and 

his colleagues trained middle school students in mathematics at thousands of 

schools throughout the United States and showed impressive learning gains in 

rigorous evaluations.32 In their meta-analysis on the topic, Kulik and Fletcher 

show that intelligent tutors in the ‘90s reportedly average effect sizes of nearly 

one standard deviation—gains nearly twice as high as the first-generation of 

computer-assisted instruction tutors.33 The learning gains of these intelligent 

tutors are approximately equivalent to human tutors.34

Affective computing originated as a branch of computer science around the 

middle of this decade, notably by Rosalind Picard.35 Those researchers examined how to simulate emotions in AI, and they developed ways for machines 

detect emotions in humans. Both goals would prove relevant for education. 

The former helped inform research on pedagogical agents, or animated char-
26 | Modernizing Learning

acters that serve as tutors or peers in instructional technologies.36 The latter 

would help inform the adaptive responses of personalized learning systems, 

such as by responding to students exhibiting boredom or frustration.37 Later, 

as this discipline matured through the 21st century, researchers such as Rafael 

Calvo and Sidney D’Mello would develop ways to more reliably, less invasively sense these states, using tools such as eye-trackers, facial and gesture 

recognition, mouse movements, and posture sensors.38

With all of these emerging technologies, it was becoming increasingly clear 

that new evidence-based principles of learning were needed. One such advancement came from Richard Mayer and his multimedia learning theory. 

Building on Sweller’s cognitive-load theory as well as other cognitive principles, Mayer carefully described learners’ mental processes when interacting 

with multimedia instruction and then offered guidance on optimizing it, such 

as: Present an explanation in words and pictures rather than solely in words, 

and present corresponding words and pictures contiguously rather than separately.39 Mayer’s work had significant impacts on the field; it made cognitive 

science more accessible to educators and gave instructional designers clear 

advice they could implement. 

Instructional theories related to computer-mediated communication also 

gained traction.40 Although these concepts emerged in the 1980s, it wasn’t until this decade, with its ready access to web-based communication, that they 

blossomed. Randy Garrison, a prolific scholar in this area, wrote of the time 

“...we are entering a postindustrial era of distance education characterized 

by the ability to personalize and share control of the educational transaction 

through frequent two-way communication in the context of a community of 

learners.” 41 Where the previous decade tended to emphasize the industrial 

value of distributed learning tools, in the 1990s, theorists such as Garrison 

began to place greater emphasis on the facilitation of teaching and learning at 

a distance. Even Otto Peters, who first proposed the industrial model of distance education, asked in the 1990s whether there were “early signs of a ‘new 

51
History of Distributed Learning | 27 

era’ which might be called ‘postindustrial’?” 42

While instructional theorists cheered the pedagogical opportunities offered 

by the world wide web, some universities had even grander designs. In his 

book, Mega-universities and Knowledge Media, John Daniel examined the 

transformative power of large-scale, open distance learning in postsecondary 

education, highlighting its promise to decrease costs, create flexibility, and 

provide greater access to higher education (particularly in underprivileged areas). Daniel specifically examined the solutions offered by mega-universities, 

such as the British Open University. By definition, these institutions remove 

barriers to enrollment and serve a minimum of 100,000 students. “Providing 

education and training for the burgeoning population of the developing world 

is not only a challenge for the countries concerned,” Daniel wrote. “The security of humankind may well depend on it.” 43

The power of the web to change society via education could not be ignored. 

Marking its impact, the U.S. Congress established the Bipartisan Web-based 

Education Commission in 1998, part of the reauthorization of the Higher Education Act. In the Commission’s subsequent—and evidence-rich—capstone 

report, titled The Power of the Internet for Learning, it urged Congress to 

make e-learning a center-piece of the nation’s education policy, saying “The 

Internet is perhaps the most transformative technology in history, reshaping 

business, media, entertainment, and society in astonishing ways. But for all 

its power, it is just now being tapped to transform education. ...It is now time 

to move from promise to practice.” 44

The six promising trends cited by the Commission’s report included greater 

broadband access; pervasive computing, “in which computing, connectivity and communications technologies connect small, multipurpose devices, 

linking them by wireless technologies;” 45 digital convergence, or the merging 

of telecommunications, radio, television and other interactive devices into a 

ubiquitous infrastructure; education technology standards; emerging adaptive 

technologies that combine speech and gesture recognition, text-to-speech, 
28 | Modernizing Learning

language translation, and sensory immersion; and finally, the dramatically 

decreasing cost of internet bandwidth. 

With the benefit of hindsight, we can 

append several additional trends to this 

list. One example is mixed reality, a continuum including virtual reality (VR) 

and augment reality (AR). Although pioneered throughout the 1950s through 

1980s, their first practical applications 

for education and training came in the 

mid-1990s. VR offerings at that time 

typically used either head-mounted 

displays or cave-like projections rooms 

to create immersive experiences.46 In 

contrast to VR, which attempt to wholly replace reality with virtual sights 

and sounds, AR systems inject virtual stimuli into actual situations, such 

as overlaying graphics onto a real-time, real-world video. However, in both 

cases, the technology was still expensive and generally cumbersome—but it 

has been advancing rapidly. Still, empirical evaluation of the effectiveness of 

these technologies for improving learning or motivation remains surprisingly 

minimal, even to this day.

Distributed simulation also saw marked progress during this decade. The 

developments of SIMNET, the decade prior, had given birth to the era of 

networked real-time simulations. Now, those same proponents that drove the 

creation of SIMNET sought to develop synthetic environments capable of 

seamlessly integrating live, virtual, and constructive simulations within a 

common environment.47 Towards that end, engineers were developing new 

interoperability standards to support synchronous instructional scenarios, including the Distributed Interactive Simulation (DIS) and the High-Level ArVirtual Fixtures, considered the first 

immersive augmented reality system, was 

built by Louis Rosenberg while at the U.S. Air 

Force Research Laboratory. Pictured above, 

Rosenberg using the system in 1992; photo 

courtesy of AR Trends.
History of Distributed Learning | 29 

chitecture (HLA) protocols,48 and researchers were examining the viability of 

using the world wide web for distributed simulation.49

The U.S. Government was also looking at better ways to leverage web-based 

learning, particularly for military and workforce development. These requirements led to the creation of the Advanced Distributed Learning (ADL) Initiative. The ADL Initiative traces its antecedents to the early 1990s, when Congress authorized the National Guard to build prototype electronic classrooms 

and learning networks for their personnel. By the mid-1990s, DoD realized 

the need for a more coordinated approach, and the 1996 Quadrennial Defense 

Review formalized this by directing development of a Department-wide strategy for modernizing technology-based education and training. This strategy 

became the original ADL Initiative. In 1998, the Deputy Secretary of Defense 

directed the Undersecretary of Defense for Personnel and Readiness, in collaboration with the Services, Joint Staff, Undersecretary for Acquisition and 

Technology and the Comptroller, to lead the burgeoning program. He also 

directed the creation of a department-wide policy for distributed learning, 

development of a corresponding “master plan” to carry out the policy, and 

resources for the associated implementation. Shortly thereafter, aspects of the 

ADL Initiative grew into a federal-wide program, with a mandate to help unify e-learning systems through coordination, shared technology standards, and 

the application of modern learning theory. 

The advanced distributed learning strategy requires re-engineering the 

learning paradigm from a “classroom-centric” model to an increasingly 

“learner-centric” model, and re-engineering the learning business 

process from a “factory model” (involving mainly large education 

and training institutions) to a more network-centric “information-age 

model” which incorporates anytime-anywhere learning.50

Part of the ADL Initiative’s mission involves technology standards for distributed learning. In the 1990s, standards such as Hypertext Transfer Protocol 

(HTTP) and Hypertext Markup Language (HTML) were just appearing. Sim-
30 | Modernizing Learning

ilarly, Extensible Markup Language (XML) was released in the mid-1990s, 

helping to turn the web from a presentation medium to a data-rich platform 

and, notably, opening the door to the semantic web. 

Whole books could (and most certainly have been) written about the technological advancements seen in the last decade of the 20th century. For our purposes, a few other notable ones included the growing prominence of AI and 

data mining, availability of natural language interfaces, commercialization of 

personal digital assistants and associated cellular communications, and creation of DVDs. Unprecedented demand for computational models also developed, encouraging researchers to craft extensive model sets for all manners of 

industries including airport facilities, call centers, businesses, health centers, 

and even fast-food restaurants.51 Cognitive modeling approaches, initially explored in earlier decades, started to be realized in applied systems. DARPA’s 

Pilot’s Associate, for instance, incorporated artificial intelligence and cognitive modeling to infer an aircraft pilot’s intentions and support her decision 

making. These sorts of cognitive and neuroscience advances also marked this 

era, and later lead president George H. W. Bush to designate it “the Decade 

of the Brain.”

2000s

The 2000s continued to see acceleration in learning technologies, aided by expanding broadband access, consumer smartphones, streaming video services, 

e-book readers, and the rise of social media. As mobile phones permeated 

across the globe, practitioners embraced mobile learning (or m-learning). In 

developing nations, m-learning became a lifeline, delivering education to millions of otherwise disconnected or underserved people.52 Even in industrialized countries, m-learning opened new doors, offering an innovative platform 

for context-aware, pervasive learning.53
History of Distributed Learning | 31 

Content designed for m-learning often took the form of bite-sized, 

microlearning chunks. Although microlearning and mobile learning are distinct concepts, the two overlap and intersect considerably, with both emphasizing flexible self-pace content, and contextualization of learning. Smartphone-based microlearning helped realize the original promise of anytime, 

anywhere—truly ubiquitous learning, delivered at the point of need. 

While m-learning developed, conventional online learning continued to grow. 

By the end of the decade, 80% of U.S. school districts offered online courses.54 Nearly all universities included some form of e-learning, and many corporations, such as Cisco and AT&T, had migrated substantial portions of their 

corporate training online.55 Commercial learning management systems, such 

as Blackboard and WebCT, held prevalent market share, and open-source 

competitors, such as Moodle and Sakai, were gaining popularity. 

The growing demand for e-learning software reinforced the need for associated technology standards, such as the Learning Object Metadata (LOM) 

and Dublin Core for defining content metadata, and the Sharable Content Object Reference Model (more commonly known as SCORM) specifications for 

making e-learning content interoperable across systems.56 Dovetailing with 

these specifications, researchers promoted the concept of “instructional objects,” or encapsulated learning materials that could be remixed and reused. 

As Fletcher predicted in 2005: 

...the emphasis in preparing materials for technology-based instruction 

(or performance aiding) will shift from the current concern with developing instructional objects themselves to one of integrating already 

available objects into meaningful, relevant, and effective interactions. 57

With such goals in mind, proponents began creating learning registries and 

content repositories—federated systems intended to support seamless discovery and access to content, such as the Content Object Repository Discovery 

and Registration/Resolution. Architecture (CORDRA)58 and the Multimedia 

Education Resource for Learning and Online Teaching (MERLOT) project. 
32 | Modernizing Learning

Although the idea of object registries has floundered somewhat in the intervening years, 59 the promise of ready access to learning continues to gain 

ground.

Interest in making education broadly accessible spurred the open educational 

resources movement, committed to making learning resources free and widely available to teachers, trainers, and learners.60 Creative Commons, and its 

open licensing model, formed around this time, and Wikipedia launched in 

the same year. Wired magazine also coined the term “crowdsourcing” in the 

mid-2000s, defining it as “...taking a function once performed by employees 

and outsourcing it to an undefined (and generally large) network of people in 

the form of an open call”—a concept the open educational community quickly embraced.61

The campaign for open education also drove development of massively open 

online courses or MOOCs. Although MOOCs wouldn’t become widely popular until 2012, they first appeared in 2008. Platforms, such as Udemy and 

Peer 2 Peer University, were founded soon after, offering free online courses to thousands of students. MOOCs also introduced a new learning paradigm. The first MOOCs grew out of connectivist learning theory, developed 

by George Siemens and Stephen Downes. Dubbed “a learning theory for the 

digital age,” 62 connectivism suggests that knowledge is distributed across 

networks of connections—particularly in our complex modern world. Consequently, it emphasizes continuously learning, the ability to see connections 

among information sources and across different fields, and the importance 

of current, diverse knowledge. The original, connectivist MOOCs are sometimes called cMOOCs, to accentuate their emphasis on social learning, cooperation, and the use of collaborative learning tools.

In addition to connectivism, several other learning theories developed 

throughout the 2000s. For example, the National Research Council published 

How People Learn, 63 an influential book encapsulating far-reaching insights 

on classroom teaching and learning. Lorin Anderson and David Krathwohl 
History of Distributed Learning | 33 

released their two-dimensional revision of Bloom’s famous taxonomy.64 David Merrill published his First Principles of Instruction,

65 which helped to 

integrate competing behaviorist, cognitivist, and constructivist learning theories. Steve Fiore and Eduardo Salas published a compendium dedicated to 

applying collaboration dimensions of learning science to online learning,66

and the Institute of Educational Sciences released its seven cognitive principles of learning, backed by solid empirical data and readily applicable in the 

classroom.67

The research and practice of personalized learning environments matured, 

growing out of the fields of constructivism and adaptive hypermedia 68 as 

well as intelligent tutoring systems and artificial intelligence in education.69

The flipped classroom concept, originally developed in the 1990s,70 gained 

widespread popularity. This instructional technique reverses the classic 

schoolhouse model by delivering didactic instructional content outside of the 

classroom and using face-to-face time for 

interactive learning, notably those activities 

traditionally reserved for homework. The 

growth of online learning tools and streaming technologies made flipped classrooms 

more accessible to teachers. Salman Khan, 

who founded the Khan Academy in 2004, 

also significantly contributed to their popularity, helping to broadly familiarize teachers and the public to the concept.71

Likewise, the application of spaced learning tactics gained widespread acceptance 

during this decade (one of the seven cognitive principles of learning by the Institute of 

Educational Sciences72), although its roots 

date back to the 19th century. Also called 

The influential How People 

Learn, and its sequel How 

People Learn II, are openly 

available from the National 

Academies at www.nap.edu
34 | Modernizing Learning

distributed practice, this principle highlights that learning occurs best (that 

is, is best encoded in and made retrievable from long-term memory) when its 

presentation happens over time rather than massed into shorter, less frequent 

intervals. Paul Kelley, headteacher at a British high school, helped popularize 

spaced learning in his 2008 book Making Minds, which drew notably from 

neuroscience principles. In it, he wrote, “As of this moment, scientific analysis 

of learning has hardly made any impact on education. In contrast, knowledge 

in areas of technology and science generally is growing rapidly. As we will 

see, this knowledge is often quite at odds with the conventional wisdom of education. The scientific understanding of the human brain, and how it works, is 

beginning to show that learning is not an abstract transmission of knowledge 

to an infinitely plastic intelligence but a biochemical process with physical 

limitations.” 73

Conversation-based learning environments with pedagogical agents and 

avatars on the web flourished during this decade—into and the future. Students could learn by holding conversations in natural language, such as in the 

AutoTutor system developed by Art Graesser and colleagues 74 and in virtual reality environments, such as Crystal Islands developed by James Lester 

and colleagues 75 and the Tactical Language and Culture System developed by 

Lewis Johnson.76 These systems promoted constructivism and collaboration, 

with engaging social and emotion sensitive interaction. 

Desire for increased, evidence-based rigor was also seen among assessments 

of learning.77 Although not a new concept, learning scientists strongly promoted the use of tests for learning,78 and urged teachers to move away from 

multiple-choice items in favor of more active techniques, such as writing essays, which most teachers didn’t know could also be automatically graded 

with high reliability.79 Relatedly, by the end of this decade, increasing computing power and the expanding amounts of learning data encouraged the development of learning analytics, led by George Siemens and his colleagues,80

and educational data mining, led by Ryan Baker and his colleagues.81 These 
History of Distributed Learning | 35 

closely related fields, each evolved to have 

professional societies and journals of their 

own, apply principles of data science to 

learning data, often collected from interaction logs or assessments built into educational technologies. Although researchers continue to debate the finer points of 

these definitions, both fields emphasize 

the use of measurement, collection, and 

analysis of data relevant to learning and 

development, along with the application 

of those analyses for enhancing some aspects of the learning system.82

2010-PRESENT

From a learning science and technology lens, the 2010s blend into the prior 

decade, but there are technological advances that have changed the landscape 

dramatically. This decade ushered into our world accurate spoken language 

understanding, smartphones at all spectrums of societies, ubiquitous gaming and social media, tracking of performance in log files at fine grain sizes, 

sensing algorithms that detect emotions and identity of people, MOOCs on 

thousands of topics, hyper-realistic animated agents, collaborative problem 

solving, and disruptive AI that will replace many jobs. It is impossible to forecast the most impactful inventions of our current era. However, a few trends 

already stand out for our current decade, but whether they will stand the test 

of time remains to be seen. 

MOOCs have continued to develop, although not without their critics and concerns. More commonly, today, MOOCs follow the so-called Extended MOOC 

model. These xMOOCs share some features with cMOOC, including open 

An early AutoTutor interface from the 

1990s, courtesy of Graesser et al.
We just finished up a manuscript for the Journal of 

Cognition and Development describing where we’ve 

come from in the learning sciences and where we’re going. 

We traced the funding investments from the 1970s until now 

and noted that the funding is coming from different places, 

including multiple federal agencies and private foundations. 

For example, the Office of Naval Research has a long track 

record of funding in this space, as does the Department of 

Education in many capacities—not just through the Institute 

of Education Sciences but also through predecessors, like 

the National Institute of Education.

Federal agencies take different approaches to funding this 

research, in part due to the differences in agencies’ missions, but the goal of understanding how people learn is 

shared. We observed that these investments either took 

a content-agnostic approach—studying learning principles 

typically studied in the laboratory that may have wide ranging 

benefits for learning, such as retrieval practice for example—

or, they took a content-dependent approach. For instance, 

investments in reading were a focus in the ‘70s and ‘80s and 

then again in 2010 with the Institute of Education Sciences’ 

Reading for Understanding Initiative... This content-dependent approach is a very different approach than the content-agnostic one; it’s about identifying nuances and challenges within a content area from a cognitive science angle. 

Both the content-agnostic and content-dependent approaches have been funded in parallel over the years, and 

both have made important contributions to our understanding of how people learn. You need the content agnostic approach to identify promising learning principles but the content dependent is also necessary because each content area 

has unique needs. Ultimately, we need to combine these 

two approaches; however, they’re taken by different types 

of cognitive scientists. It would be beneficial if those groups 

started working together. 

Erin Higgins, Ph.D. 

Program officer within the Institute of Education 

Sciences, U.S. Department of Education

Look for Higgins, Dettmer, and Albro, currently in press
History of Distributed Learning | 37 

access and large scale. However, where cMOOCs stress connectivist learning, xMOOCs generally use more traditional, instructivist methods, focusing 

instead on scalability. Spanning both industry and academia, the most popular xMOOCs launched in 2012 including Coursera, edX, and Udacity. These 

platforms, which attempt to provide learning at scale, have been significantly 

aided by the development of cloud computing in the 2000s and by the consumer release of Amazon Web Services and Microsoft Azure. Cloud systems 

made the “service” model of computing viable, freeing software applications 

to become device and location independent, allowing for more frequent application updates, and creating a near-infinite capacity to scale on-demand.

Cloud computing also helped realize the Internet of Things (IoT), the network 

of smart devices that can connect to networks and share data. Cisco’s Chief 

Futurist, Dave Evans, estimates the IoT was “born” around 2008 or 2009, but 

researchers have only begun exploring its applications for learning.83 In the 

context of education and training, IoT helps bridge real and virtual contexts, 

allowing learners to interact with networked physical objects that also have 

digital footprints.84 These objects might include embedded RFID sensors, 

spatial beacons, or wearable technologies, such as FitBits or Google Glass.85

Some wearable technologies also incorporate neurophysiological sensors, 

such as heart-rate monitors or eye-trackers. The commercial versions of these 

still usually suffer from noisy data, and are only starting to be meaningfully 

integrated into applied learning systems. Applications of psychophysiological 

tools (e.g., eye-tracking, skin conductance), brain imaging tools (e.g., fMRI, 

EEG), and affective computing are rapidly advancing in laboratory contexts, 

and researchers are already having success detecting students’ emotions from 

low-cost video feeds, pulled from the stock cameras on phones and laptops.86

Further, several new DARPA programs are teasing science fiction–like results 

as they explore neural interfaces; these have already shown to enhance human 

cognition and learning in clinical experiments, and they could one day enable 

complex human-machine teaming.87
38 | Modernizing Learning

Each of these applications produces an overwhelming amount of digital 

by-products—a smog of data. The explosion of learning data, and corresponding growth and diversity of learning platforms, has once again created a 

need for new technology standards. The ADL Initiative began developing the 

Experience API (xAPI) in 2011, with its first public release in 2013. xAPI lets 

software applications share (potentially big) data about human performance, 

along with associated instructional or performance context information. xAPI 

helps analysts aggregate and collectively analyze learner data from different 

systems—from traditional LMSs to mobile devices, simulations, wearables, 

and physical beacons. xAPI also represents one piece of the developing Total 

Learning Architecture, a set of specifications that promises to connect the 

many dissimilar and stovepiped learning technologies into a more cohesive 

system-of-systems.

The sophistication of the 21st century learning environments and complexity 

of data within them have the unfortunate consequence of driving up costs. 

An expensive system, say costing $50 million, is economically plausible if it 

delivers training to 10 million learners—but not if only 100 people benefit. 

There have been a number of efforts to reduce costs in addition to improving 

learning and motivation. For example, intelligent tutoring systems have been 

expensive to develop in the past, so the Army Research Laboratory, led by 

Bob Sottilare, organized a community of over 200 researchers and developers 

to articulate adaptive instructional system guidelines in a 7-volume book series that covers learner modeling, instructional management, authoring tools, 

domain models, assessment, team tutoring, and self-improving systems.88

This Generalized Intelligent Framework for Tutoring (GIFT) initiative also 

includes a functional computational architecture that can be used to develop 

and test systems. 

Another emerging approach to reducing costs is to use crowdsourcing in content creation and modification, with machine learning to automatically tune 

quantitative parameters in self-improving systems.89 Unfortunately, the field 
History of Distributed Learning | 39 

still lacks a systematic, widely accepted approach to estimating costs and development time for building and testing these complex learning environments. 

With the increasing automation in education and training, there’s been a corresponding push to create semantically rich data, that is, to give the meaning to the underlying data elements—in ways computers (and other humans) 

can understand. The developers of xAPI, for instance, are attempting to build 

semantically rich usage profiles as well as published, shared vocabularies. 

Proponents of competency-based learning are attempting a similar feat, but 

in their case, to define the data elements that make up a human competency. 

Volunteers supporting the IEEE established a working group in 2018 to revise 

the decade-old Reusable Competency Definition (1484.20.1), expanding its 

utility and harmonizing it with other standards for competencies and competency frameworks.90

The working group’s efforts are timely, as more formal education programs 

are embracing competency-based degrees, i.e., postsecondary programs 

where students earn diplomas by demonstrating mastery through real-world 

projects—rather than through time-based credit hours. In competency-based 

T3 INNOVATION NETWORK

In early 2018, the U.S. Chamber of Commerce and Lumina Foundations launched 

the T3 Innovation Network to bring businesses, postsecondary institutions, technical standards organizations, human resource professionals, and technology vendors 

together to explore Web 3.0 technologies for an increasingly open and decentralized 

public–private data ecosystem. Since its kickoff, the Network has grown into a thriving network of over 128 organizations who are addressing three key challenges: (1) 

The need for harmonization among technical data standards groups to ensure data is 

interoperable and shareable across systems and stakeholders; (2) The need to apply AI 

solutions to improve how learning objectives, competencies, and skills are authored, 

translated, and distributed; and (3) The need to empower learners and the American 

worker with data to improve their agency and ability to manage and connect to opportunities in the talent marketplace.
40 | Modernizing Learning

programs, students are typically assigned learning coaches, rather than didactic instructors, and they have access to an array of open-source resources, 

including videos, textbooks, and online communities.91 As of 2014, there were 

already an estimated 200+ competency-based learning postsecondary degree 

programs in the U.S., but policy regulations are lagging.92 It’s not clear how 

this trend will resolve, but we fully expect the core concept expand in the 

coming years.

Like competency-based degrees, micro-credentials, and the associated technology standards for digital badges, have garnered growing attention. Training and education credentials, such as licenses and diplomas, have existed 

for centuries as a way to verify someone’s educational pedigree. Like their 

more robust cousins, micro-credentials assert that a person has demonstrated a particular competency. Unlike more formal credentials, however, learners can receive micro-credentials for smaller learning segments, and (at least 

hypothetically) micro-credentials reflect the performance-based approach of 

competency-based learning. Whether micro-credentials catch on remains to 

be seen. Practical and policy challenges still face the field; although, organizations such as the Lumina Foundation, Digital Promise, and BloomBoard 

are working to overcoming them. Meanwhile some commercial organizations 

are charging ahead with their tiny certs, including Udacity’s nanodegrees and 

edX’s MicroMasters.93

Given these many technological inventions, the rise of learning analytics, 

surge in neuroscience research, and developing maturity of learning science, 

educators and instructional designers are forced to rethink their discipline as 

well as their own capabilities. If done correctly, the future of learning will 

look noticeably different from its Industrial Age ancestor. Correspondingly, some have embraced the concept of learning engineers—a new (and still 

forming) paradigm that describes the “instructional designer” of the future. 

In 2017, the IEEE created a working group, named the Industry Connections 

Industry Consortium on Learning Engineering, to help mature the idea, led 
History of Distributed Learning | 41 

by Bob Sottilare, Avron Barr, Robby Robson, Shelly Blake-Plock, and others. 

In 2018, Chris Dede, John Richards, and Bror Saxberg released their guide to 

Learning Engineering for Online Education.

94 Saxberg, who also serves as 

a Consortium advisor and as vice president of learning science at the Chan 

Zuckerberg Initiative, described the emerging discipline:

A Learning Engineer is someone who draws from evidence-based information about human development—including learning—and seeks 

to apply these results at scale, within contexts, to create affordable, reliable, data-rich learning environments.95

To add, from another of his quotations: 

There will come a time when we look back at how we “used to do learning,” and, just as we now look at medicine in the 19th century, wonder 

how we ever made progress without using the science and evidence that 

we can now generate. We’re not there yet—but we may be on our way.96

Saxberg’s words ring true, not just for learning engineers but for the wider 

learning and development sector. Much has changed as technology advanced 

and learning science evolved. The concept of “distributed learning” has progressed, from its simple roots as a pragmatic tool to bridge the transactional 

distance, to today’s cacophony of ubiquitous, adaptive, on-demand instruction. A central goal of the ADL Initiative and its larger community has always 

been to bring clarity and coordination to this discipline. Today, more than 

ever, the distributed learning community needs organizational, theoretical, 

technological, and policy structures to bring unity. We are, perhaps, in the 

middling ugly-duckling years of the field’s maturation. The promise of responsive and evidence-driven ubiquitous learning is there, crafted by contributors for over 40 years. It’s now our challenge to resolve the complexity, 

to bridge across its numerous facets as our connectivist peers have taught us, 

infuse deliberate learning theory into our work as learning science scholars 

advise, and, as the learning engineers promote, to embrace a comprehensive 

approach to enhancing the full continuum of learning. 
The first hurdle is to move past the “recorded slides and 

talking head” form of online learning. The instructors need 

to be trained on advances in digital learning technology and 

methodology. The second hurdle is ensuring that the organization 

has a modern experience driven learning environment that supports 

these more interactive and personalized experiences. The third is to 

communicate expectations between the instructor and learners that 

this isn’t a lecture, rather, it’s a facilitated dialogue, not limited to a 

particular place and time—but available for continuous reference and 

enhancement. 

John Landwehr

Vice President and Public Sector Chief Technical Officer, Adobe
Distributed Learning Instructional Theories | 43 

CHAPTER 3

DISTRIBUTED LEARNING 

INSTRUCTIONAL THEORIES

Scotty D. Craig, Ph.D. and Ian Douglas, Ph.D.

Learning has moved beyond the classroom. It’s happening everywhere, all the 

time, formally and informally, incidentally and intentionally—and increasingly supported by digital technologies. For more than a decade, online education has consistently expanded.1

 The U.S. Department of Education estimates 

that 5.8 million students enrolled in distance 

education courses in 2015, the most recent 

year for which statistics exist, accounting 

for 28% of the total student population.2 The 

Association for Talent Development reported that 88% of corporations offered e-learning as part of their workforce development in 2017, and 27% of high-performance organizations used e-learning for a majority of their training.3

 MOOC 

clearinghouse Class Central reported that MOOCs also grew, serving over 80 

million students in 2017.4

No doubt the impact of distributed learning will continue to grow; hence, 

educational decision makers, instructional designers and learning engineers, 

teachers, and trainers should understand the best practices for technology-enabled learning—and implement these to their best abilities and resources. 

This isn’t just our opinion. For instance, the Every Student Succeeds Act, 

signed into law by President Barack Obama in 2015, requires that students in 

America be taught to highest academic standards and asks schools to employ 

evidence-based approaches to learning, supported by a scientific process that 

 

 

Distributed learning 

should employ evidencebased practice, built on 

the science of learning
44 | Modernizing Learning

provides evidence of effectiveness. Similarly, the World Bank, cited “acting 

on evidence to make schools work for learning” among their three priorities 

for 2018, writing “Act on evidence—to make schools work for all learners. 

Use evidence to guide innovation and practice.” 5

Building evidence and properly validating a theory within a scientific discipline, however, can take decades. Then more years to communicate its premise to the wider community—not withstanding those pockets who will inevitably resist the idea of evolution. Meanwhile, as this process plods forward, 

practitioners are anxious for improvement. So, they embrace theories that, on 

their face, seem to make sense, even if there’s little proof to accompany them. 

Commercial interests further complicate matters, as companies are often 

quick to adopt popular theories, promote their unique value propositions, and 

build technology around them—all before adequate research has concluded. 

But the world isn’t so grim. The scholarly pursuit of learning science is increasing. The National Academies recently released a sequel to their excellent 

compendium, How People Learn. This new volume, How People Learn II, 

published near the end of 2018,6 included new research on educational technologies, including findings on neurological processes, lifelong learning, and 

the impact of social and cultural factors. There’s also growing awareness from 

policymakers and administrators of the importance of learning science and 

greater numbers of research programs at institutions such as the aforementioned Department of Education and World Bank. 

In this chapter, we mix optimism with some healthy caution. In the next sections, we overview research that provides some guidance on designing for 

technology-supported learning and practical best practices for establishing 

associated design teams. We’ve omitted many quality theories, for the sake 

of brevity, but will summarize a few of the most relevant to the design of 

distributed learning. Our main goal is for readers to take away the ideas that 

distributed learning theories exist, authors have taken steps to make them 

accessible to practitioners, and new distributed learning systems—whether 
Distributed Learning Instructional Theories | 45 

concerned with the content development level or the enterprise infrastructure 

level—should be informed by this work. 

INSTRUCTIONAL THEORIES

As outlined in the preceding chapter by Art Graesser and colleagues 

(Chapter 2), learning science theories have generally evolved with the zeitgeist of cognitive science. Early educational theories followed the behaviorist 

model, emphasizing drill-and-practice tactics, reward and punishment, feedback, and repetition. Cognitivist theories came next. In contrast to the behaviorists, cognitivists sought to understand the mind and apply principles of 

cognitive processing to the design of learning content. A third prominent paradigm, constructivism, followed. Constructivists argued that humans create 

rather than acquire information; it’s therefore impossible for some “correct” 

understanding of the world to be transferred from one person’s memories to 

another. Individuals must learn through engagement.7

As one might expect, each of these paradigms encouraged the development 

of various instructional theories. Seeing the proliferation of competing theories, Dave Merrill set out to evaluate and eventually harmonize the field. 

His resulting work, First Principles of Instruction, had wide impact.8 For the 

first time, a framework incorporated the breadth of theories—and all within a 

concise set of principles. The inset below summarizes them, but we encourage 

readers to read Merrill’s original article where he includes crisp guidance for 

instructional designers on each.

In How Learning Works, Susan Ambrose and colleagues followed in Merrill’s 

footsteps.9 They built on his First Principles and added to them new synthesized research on teaching. Their subsequent framework includes seven categories, each with several underlying recommendations written specifically for 

teachers and instructional designers.
46 | Modernizing Learning

Both Merrill and Ambrose et al.’s work recommends that practitioners create 

active learning environments. However, in practice, this suggestion is often 

watered down, distilled to superficial criteria like measures of classroom attendance or homework completion, or it’s otherwise simplified to proxy indicators, such as attitude or interest. None of these truly meet the mark. As 

Michelene Chi and her collaborators have observed: 

In short, although “active learning” is a great idea for overcoming “passive learning,” we have identified three concrete practical challenges 

that teachers may face when developing lessons that promote “active 

learning.” First, broad recommendations such as engage students cognitively, encourage meaningful learning, and get students to think about 

it do not tell teachers how to create activities that overcome “passive 

learning.” Second, teachers have few criteria to use in deciding which 

are the best “active learning” activities to design and implement. Third, 

there are no guidelines for teachers regarding how to best modify their 

favorite existing assignments in order to optimize “active learning.” 10

Chi and colleagues developed the Interactive, Collaborative, Active, and Passive (ICAP) framework to provide guidelines for fostering active learning 

environments. The ICAP categories describe hierarchical levels of cognitive 

engagement, with “passive” learning typically producing the weakest learnProblem Centered – Engage learners in solving real-world problems

Activation – Active learners’ relevant previous experience

Demonstration – Demonstrate what’s to be learned (don’t merely talk about it)

Application – Have learners use their new knowledge or skill to solve problems

Integration – Encourage learners to transfer new learning into their everyday lives

1 PRINCIPLES OF INSTRUCTION (DAVE MERRILL)

st
Seven principles for smart teaching from Ambrose et al.

1 Learners’ prior knowledge can help or hinder learning 

Teachers should talk with other instructors and use diagnostic tests of prior knowledge 

to learn about their students. Be explicit to students about the connection between new 

material and their prior knowledge; this aids long-term retention.

2 How individuals organize knowledge influences how they learn

It also affects how they apply what they know. So, make use of techniques that make 

knowledge organization schemes explicit, such as concept maps. Look for patterns of 

mistakes and misconceptions in learners’ conceptions.

3 Learners’ motivation determines, directs, and sustains learning

Help learners see the value in what’s being taught and how it helps their future 

development. Provide authentic tasks with an appropriate level of challenge (simulations 

and games are useful). Get learners to understand the reasons for success and failure.

4 Learners must acquire and integrate component skills

To develop mastery, learners need to practice integrating component skills and know when 

to apply what they’ve learned. Be aware of expert “blind spots”—steps they perform 

unconsciously and are, therefore, not well-articulated in instruction. Provide isolated 

practice of component skills in diverse contexts and then facilitate the integration of 

component skills in more challenging tasks. 

5 Goal-directed practice with targeted feedback enhances learning

Phrase instructional goals in terms of capabilities rather than knowledge (refer to 

Chapter 13, in this volume, on competency-based learning). Provide time for deliberate 

practice, and pair this with feedback that focuses on specific items that need improvement.

6 The social, emotional, and intellectual context impacts learning 

Learner current development is influenced by the context. A positive and constructive tone 

of communications within the learning community often improves learners’ motivation and 

behavior.

7 Students must learn to monitor and adjust their own learning

Help learners develop metacognitive skills, such as self-monitoring. A malleable, rather 

than fixed, perspective of intelligence can also be promoted and has been found to 

influence performance.
48 | Modernizing Learning

ing outcomes and “interactive” learning often promoting the strongest. Interactive learning encourages learners to actively integrate new and prior knowledge, draw inferences to fill knowledge gaps and confusions, and otherwise 

enact strategies that build rather than merely rehearse knowledge, ultimately 

supporting deeper learning and increased transfer to new domains. Notably, 

this research highlights that it’s the way learners engage in different activities 

that makes them more or less passive; learners’ engagement levels aren’t necessarily “cooked in” to the instructional interventions, themselves.

Example of “watching a video” at various levels of engagement:

PASSIVE

Receiving

ACTIVE

Manipulating

CONSTRUCTIVE

Generating

INTERACTIVE

Dialoguing 

Watching the video, 

without doing 

anything else

Actively engaging 

with the playback, 

such as rewinding 

and pausing; taking 

verbatim notes

Explaining concepts 

from the video; 

taking paraphrased 

notes; contrasting 

the video to other 

materials

Debating with a peer 

about the message 

in the video; actively 

analyzing the position 

of the video in a small 

group discussion

Example application of Chi and colleagues’ ICAP framework

Much of our preceding discussion has emphasized the science of teaching or 

the practice of instructional design. However, as Ambrose and her coauthors 

highlighted in their own work, recommendations on designing and delivering 

instruction miss more than half the equation. Though tightly bound, learning and development are wholly different phenomena from education and 

training. With this understanding, Ambrose et al. highlighted three critical 

components of learning:

1. Learning is a process, not a product.

2. Learning involves changes in knowledge, beliefs, behaviors, or 

attitudes, which must unfold over time. 

3. Learning is not something done to others, but rather something 

learners must do themselves.

Teachers, trainers, and instructional designers can’t directly manipulate 
Distributed Learning Instructional Theories | 49 

what’s happening in learners’ minds, but some theories give guidance on how 

to encourage better learner processes.

Self-regulated learning theory, for example, describes learning processes 

guided by the learner him- or herself and which is, at least partially, intrinsically motivated. At its most basic, self-regulated learning involves planning, 

executing, and then reflecting on some activity. Hence, it involves the application of metacognitive knowledge and monitoring skills, such as understanding different cognitive tactics and correctly identifying the difficulty of 

different tasks. 

Louise Yarnall and her colleagues describe self-regulated learning in more detail later in this book (Chapter 15). In short, one way to envision it is as a cycle, 

involving different phases that someone undertakes to strategically and intentionally improve performance.11 These phases start with task definition, where 

someone works to understand the problem at hand along with any available 

resources. This is followed by a goal setting and planning phase, where learners establish objectives and select tools and strategies to meet them. Next, an 

enactment or engagement phase occurs, where learners implement their chosen strategies and attempt to perform the task. Finally, there’s an evaluation 

or adaptation phase, where learners assess their actions and outcomes, and 

revise their goals, plans, and strategies, accordingly. Although these actions 

are, by definition, learner driven, individuals without strong metacognitive 

skills can be taught. For instance, teachers and trainers can provide scaffolds 

to help guide learners through these self-directed learning processes.

In closing, this section has offered the barest summary of instructional 

theories. Some other sources serve as useful supplements. Harold Pashler and 

colleagues published seven principles for instructional strategies, including 

recommendations for spaced learning, using worked examples in combination 

with problem solving, combining graphic and verbal descriptions, integrating 

abstract and concrete concepts, using quizzing and questions to eliminate 

misconceptions, and supporting self-regulated learning by helping learners 
50 | Modernizing Learning

allocate study time.12 Art Graesser built on prior work to define 25 principles 

of learning (clearly an overachiever in learning frameworks!).13 These roughly 

group into recommendations for reducing processing load, facilitating learning 

by implementing strategies within (e.g., feedback and deep questions) and 

around the learning content (e.g., testing effects and spaced learning), and 

suggestions for helping learners understand the process of learning (e.g. selfregulated learning and desirable difficulties). Finally, for a truly comprehensive 

historic treatment, Peter Jarvis authored a three-volume set, beginning with 

the book, Towards a Comprehensive Theory of Human Learning.

14

INSTRUCTIONAL 

TECHNOLOGY 

THEORIES

Classic instructional theories emphasize learner-content, learner-teacher, or 

learner-learner interactions. Starting in the around the 1960s, researchers began to also examine learner-interface dynamics, leading to unique pedagogies 

for educational technology. Early work on instructional media involved comparison studies, often looking at technology-mediated versus traditional settings. These found “no significant differences,” but this was the behaviorist era 

and (as described below) instructors tended to employ the instructional media 

in the same way they might deliver traditional teaching. In the 1980s, with 

growing interest in the cognitive perspective, researchers began to look more 

closely at media attributes and their interactions with individual differences.15

Coming out of this growing appreciation of instructional technologies, Richard Mayer published his highly influential Cognitive Theory of Multimedia 

Learning. Multimedia learning is a combination of more than one modes of 

information presentation, such as visual images with a narration, within a 
COHERENCE

eliminate extraneous 

information

SIGNALING

highlight essential 

information

REDUNDANCY

use graphics and narration 

(not on-screen text)

PRE-TRAINING

start lessons with a quick

refresher and an overview

MODALITY

use graphics and narration

versus animations and text

MULTIMEDIA

words + pictures are

better than words alone

PERSONALIZATION

use a conversational

style, not a formal one

VOICE

narrate in a friendly human

(not machine) voice

IMAGE

the narrator’s image isn’t 

needed on-screen

SPATIAL CONTIGUITY

put words and related 

pictures near each other

TEMPORAL CONTIGUITY

show words and related 

pictures simultaneously

SEGMENTING

present lessons in

under-paced segments

MAYER’S 12 PRINCIPLES OF MULTIMEDIA LEARNING 
52 | Modernizing Learning

learning environment. Mayer’s theory builds on core 

cognitive mechanisms. For 

instance, it acknowledges the limited capacity of 

working memory, assumes 

that learners have two cognitive processors that handle 

new information differently 

(an auditory and a visual processor), and that learners must be cognitively 

engaged to produce new knowledge structures.16

Recommendations for technology-enabled instruction naturally followed from 

these tenets. For example, given the limits of working memory, multimedia 

learning materials need to moderate the amount of essential processing required by the learners depending upon their prior knowledge, experience, and 

competencies. Furthermore, given our brains’ two processing channels, complementary information should be delivered simultaneously to both to more 

efficiently support learning. Many other design principles can also be derived; 

these cluster under 12 principles, as summarized in the adjacent graphic.

Another uniquely technology-centric theory is described by the Substitution 

Augmentation Modification Redefinition (SAMR) model, popularized by 

Ruben Puentedura.17 It emphasizes a unique challenge with learning technologies; that is, often people use them in similar ways and assume similar context to traditional, formal education settings—with classrooms, instructors, a 

fixed body of content to be learned and a fixed amount of time. This model 

helps explain why, for instance, the first web-course designers attempted to 

recreate printed texts online or why the original virtual classrooms took so 

many cues from physical ones. 

The SAMR model defines levels of technology use in teaching and learning. 

The most basic, and most often implemented level is substitution, where the 

...education doesn’t educate you 

unless it changes you.

Betty Lou Leaver, Ph.D.

Director, The Literacy Center; Manager, MSI 

Press; Former Provost, Defense Language 

Institute Foreign Language Center
Distributed Learning Instructional Theories | 53 

technology is used to perform the same task as was done before. For example, 

an instructor uses PowerPoint to replace acetate slides or students use laptops 

to replace paper notebooks. Alternatively, the highest level is redefinition, 

where technology supports new learning tasks that were previously inconceivable. This level represents the future of learning and is a foundational 

reason for reimagining instructional design.

Technology is changing the way we live, and future instructional technology theories should reflect new approaches to learning, including in individual, social, and lifelong learning contexts. However, many of our current 

best practices were developed before this digital explosion, leading us to ask, 

“How will we transform our current models for learning and not just consider 

how we make incremental improvements to the traditional approach?”

SUBSTITUTION

Technology acts as direct substitute, with no functional improvement

AUGMENTATION

Technology acts as direct substitute, with functional improvement

MODIFICATION

Technology enables significant task redesign

REDEFINITION

Technology enables new tasks, previously inconceivable

The SAMR model highlights our tendency to 

use new technologies in old-school ways. SAMR
54 | Modernizing Learning

VISION FOR THE FUTURE 

OF LEARNING THEORY

One challenge with learning theories is that they’re prone to focus just on the 

design, delivery, and evaluation of instruction. Even with additional consideration for technologies used for learning, we’re still omitting part of the puzzle. 

Earlier in this book, Walcutt and Schatz outlined six elements that need to 

be considered for the future learning ecosystem: technology infrastructure, 

design, commitment, governance, policy, and human infrastructure. The construction of learning elements—including those theories covered so far in 

this chapter—fall into their “design” category. Undoubtedly, the careful design of learning content, associated delivery and evaluation techniques, and 

learner-support methods are critical. However, the other elements in this also 

framework warrant consideration. 

Certainly, Walcutt and Schatz aren’t the first to suggest a wider aperture. 

Badrul Khan,18 for instance, proposed an eight-dimension framework for 

e-learning, comprised of institutional, management, technological, pedagogical, ethical, interface design, resource support, and evaluation factors. 

Shahid Farid and colleagues built on Khan’s work.19 They use empirical data 

from stakeholders about roadblocks to e-learning in postsecondary environments. Farid et al.’s model includes software, technical, institutional, personal, and cultural dimensions. Beatrice Aguti and her colleagues also developed a broader model for higher-education contexts, but this time for blended 

learning. Their framework has four dimensions, including e-learning course 

delivery strategies, e-learning readiness, quality e-learning systems, and effective blended e-learning.20 For our purposes, we’re less concerned with the 

potential similarities and differences of these various frameworks. Our point 

is simply that learning—and particularly technology-enabled learning—happens within a broader context. 
Distributed Learning Instructional Theories | 55 

From this broader perspective, it’s 

clear successful distributed learning enterprises will rely up-on interdisciplinary, effective teams of 

practitioners. Where teachers once 

presided over their classes or principals over their schools, the emerging learning ecosystem has less defined boundaries, and it also relies upon a greater diversity of expertise (as 

described in more detail in Chapter 19, which discusses learning engineers).

Successful, future (distributed) learning will be developed by organizations 

able to build and support multidisciplinary teams. For instance, in lieu of an 

isolated instructor, we could imagine a team of three to five members working 

together to develop learning experiences. This team might involve an instructor or content expert, an instructional design or learning science practitioner, 

a technology expert, and perhaps even a data scientist. Additional members, 

such as usability experts and psychometricians, might also be required.21

Finally, to be truly successful, there needs to be a larger learning organization 

(administration) in place to facilitate interactions and coordination.22

This new team structure will also require strong leadership.23 Leaders responsible for learning will need awareness of the expertise available to them and 

know how to integrate different kinds of expertise into learning development 

processes. They’ll need to understand evaluation, at multiple levels (such as 

within the content, to assess learners, but also at an institutional level to evaluate the learning experience, itself), and they’ll need to consider broader implications, such as privacy, ethics, and social factors. During learning design and 

development phases, leaders will need to look for efficiencies. For instance, 

they’ll need embrace the reuse of learning materials, looking for ways to reduce the cost of development efforts by reusing already developed content 

elements, technologies, or tools. 

Take care to avoid the 

Everest Syndrome—the 

urge to embrace new 

instructional technology 

just because it’s there
56 | Modernizing Learning

Thus, learning leaders should continually ask themselves questions, such as:

• Do we have all the specific expertise on our team to meet our goals?

• Is the team working effectively as a community with shared purpose?

• Are we making good use of existing reusable resources and tech?

• Are our evaluation processes (at all levels) the best we can achieve?

• Are we aware of the evidence in support of each instructional resource, 

method or technology we use?

• Do we have someone capable of interacting with the output of the 

learning science community to identify relevant knowledge that can be 

adapted into our process?

The development of instructional materials has sometimes been compared to 

software development.24 The development of software in the early days of the 

personal computer involved one or a few individuals crafting an application 

with a primary focus on function; however, modern software development 

involves large teams of different specialists (e.g., software architects, software engineers, user-experience designers, cybersecurity specialists) working 

together and collectively considering a broad range of design attributes (e.g., 

functionality, security, aesthetics, usability). Modern software developers 

are also comfortable with the idea of reuse and “mashups” (combining data 

or functionality from different sources). Numerous repositories of reusable 

code are readily available on the internet. Also, connections called Application Programming Interfaces (APIs) allow different operational software platforms to share data across them, enabling sophisticated functionality, such as 

Google maps, or up-to-the-minute data, such as from the U.S. Government’s 

data.gov, to be embedded in any other application. However, the same ethos 

isn’t always found in modern instructional development—both the organizational dynamics of multidisciplinary instructional teams and the infrastructure needed to share and integrate learning materials need to be cultivated.

However, promoting successful interdisciplinary teams is challenging, not because adequately skilled individuals are unavailable but because they often 
Distributed Learning Instructional Theories | 57 

lack teamwork and collaboration skills—skill which learning professionals 

are rarely taught explicitly.25 Thus, a key step towards achieving the future 

learning ecosystem will involve the maturation of organizational processes, 

teamwork-focused professional development for various contributors, and a 

culture shift—similar to one that happened within software engineering.26

On the content-sharing side of the equation, we’ve already seen significant 

efforts to encourage reuse in instructional development, but so far, these have 

met with limited success, particularly compared the level of reuse and data-sharing that happens in software development. Roughly two decades ago, 

SCORM was developed to help facilitate learning content reuse, and there 

Projects That Work is an ongoing research study with the goal to provide teachers 

data-driven information to make decisions to use service learning flexibly, efficiently, and effectively. The premise is that if schools and teachers have continuously 

updated lists of projects that were highly rated by 20 or 25 previous classes 

around the country, these projects would (a) be known to teacher and (b) could 

be replicated, providing all students the opportunity to realize the potential of 

what service learning has to offer....Preliminary findings revealed that about 90% 

of students were highly engaged by service learning and produced positive results 

from many types of service learning projects. Many of the findings to date echo prior research demonstrating the role of well-designed programs that include specific 

activities to prepare students with a clear and compelling rationale for the project 

and with specific roles and responsibilities. The key to replication in schools with 

less expertise in service learning may focus on teachers having information on key 

components of projects. It’s important to ensure that the projects are feasible for 

teachers and students to do, and that they lead to students’ belief that they’re 

making a difference and perceive that they’re learning. 

Edward Metz, Ph.D.

Projects That Work

90% ...of students were highly engaged, when

taught via well-designed, service-learning methods
58 | Modernizing Learning

have been multiple attempts to build repositories of reusable educational resources, such as the MERLOT.27 Another, more recent repository, the Open 

Educational Resources Commons,28 offers curated content with open licenses; it also encourages co-creation and participation by users. 

Newer repositories are now integrating evidence in support of the assets provided to the community. The What Works Clearinghouse from the Institute 

We used AutoTutor for the Office of Naval Research and put it into ALEKS, 

a commercial adaptive learning system. It went ok, but then we tried to do 

a scale-up in a school district. We were able to get a big teacher preparation 

session. They were reasonably optimistic. The strategy was to let them use 

ALEKS on their own before getting AutoTutor. We found that initially a lot 

of people liked it, but then they had school vacation and then after that they 

had a huge snow storm and were out for about 8 days of schools. Then they 

had a very short time for standardized testing for the state (about 5 weeks) 

that resulted in universal attrition. In talking with the teachers, they had to 

teach to the test, but ALEKS is based on mastery learning. It won’t allow you 

to do topics you’re not ready for....From a learning perspective, it makes 

sense, long-term, but teachers have many logistical needs that aren’t directly 

represented in adaptive systems. They have to have the kids know information/

knowledge at a certain time whether or not the student is technically 

ready for it—even if they aren’t going to remember it. Their knowledge 

repository might collapse later because they didn’t get the foundational 

information when they needed it but it’s what they needed for the test. 

Benjamin Nye, Ph.D.

Director of Learning, Institute for Creative Technologies, 

University of Southern California
Distributed Learning Instructional Theories | 59 

of Educational Sciences at the U.S. Department of Education is one example of a research-evidence repository.29 This clearinghouse identifies studies with credible and reliable evidence of effectiveness, and it disseminates 

free reports and summaries on its website. The What Works Clearinghouse 

currently has over 700 summaries on effective educational innovations and 

over 10,000 reviewed studies available in its repository. A number of similar government-sponsored research communities can also be found, such as 

CLEERhub 30 for National Science Foundation research on Engineering Education; the National Academies Press, with open-access e-books on hundreds 

of topics, including Behavioral and Social Sciences and Education;31 and the 

Defense Technical Information Center for military-funded research.32

CONCLUSION

In summary, extensive research has been conducted to inform instructional 

theory, but there continues to be a gap between scholarly findings and their 

practical application. However, there are many excellent resources for teachers, trainers, instructional designers, policymakers, and administrators. Unfortunately, many of these resources still assume that learning will occur under traditional (Industrial Age) conditions; so, consider them with caution. 

Some theories have been developed specifically with instructional technologies in mind. Seek these out, but also remember it takes years to properly 

validate a theory; so, watch out for hype, particularly when commercial profits 

or someone’s reputation is on the line. Also, when designing for technology-supported learning use a measure of creativity, to avoid succumbing to the 

“just substitution” mindset. Similarly, also be willing to rethink the design, 

delivery, and coordination of learning processes. Emerging technologies are 

radically changing the ways we train, educate, learn, and develop, and they’re 

similarly changing the ways learning professionals operate—embrace teams, 

seek out shared materials, and embrace a culture of reuse.
Education in the future will be more of an iterative process. 

Currently, people pursue their education at the beginning of 

their lives and then they go to work. Education beyond that initial 

period typically only happens because of some disruption in their 

lives—they lose their job or other changes in circumstances. It’s 

difficult to access at that stage in life, but in the future, while you’ll still 

have early-life education, it might look a bit different—with more of 

an emphasis on work-ready skills and learning to continuously learn. 

There will also be many more opportunities for dipping in and getting 

back out of the workforce throughout someone’s life. Education will be 

more just-in-time and based on the needs of the moment. Technology 

will support that, but it requires significant change in the way 

education institutions operate and the way employers do things.

Martin Kurzweil, J.D.

Director of the Education Transformation Program, Ithaka S+R
Lifelong Learning | 61 

CHAPTER 4

LIFELONG LEARNING

J.J. Walcutt, Ph.D. and Naomi Malone, Ph.D.

The world has progressed in so many ways over the past 100 years, yet our educational structures have stayed relatively unchanged. Incremental progress 

has certainly occurred, to include improvements in classroom organization 

and information delivery, but the developmental models, progression of formal educational offerings, and recognition of learning via grades and degrees 

have proven resistant to change. As a society, we still focus on controlled 

settings for learning and group-based information delivery. The sequence is 

linear, the instruction is split into finite end-points, and the whole process is 

assessment-oriented.

We rely on outmoded developmental models (such as Jean Piaget’s stages of 

cognitive development) and use a failure-focused mindset when measuring 

learning, that is, students’ developmental speed and depth of knowledge are 

judged against expected averages, largely defined by age-based phases. In a 

K–16 setting, those who fail to conform to expectations are “behind in their 

development,” and in workforce or military settings, those who lag are judged 

as incapable, unmotivated, or possessed of other character flaws. We assign 

grades based on achievement and determine progression through the system 

based on time factors, such as credit hours or classroom attendance, along 

with single-point, high-stakes testing. Similarly, we make strategic-level curriculum decisions based on these goals, such as how to achieve increased 

seat-time or time-on-task, assuming that more time spent learning will result 

in improved outcomes (even though data suggest that students need non-instructional assimilation time and varied experiences to aid comprehension, 

and that learning needs to be context-based).1

 

 
62 | Modernizing Learning

In the modern world, we’re 

inundated with unlimited, 

unfiltered, and unmanaged data; 

individuals can learn anything they 

wish, but we run the very real risk 

of increasing low-level information 

acquisition to the detriment of 

higher-order comprehension.
Lifelong Learning | 63 

We largely place students in controlled settings (classrooms), where information is filtered by a teacher or curriculum designer to ensure its accuracy 

and intelligibility, where goals are clearly defined, the level of information 

provided is appropriate for learners, the pace is controlled, and someone is 

available to help monitor the informational content and its delivery. In many 

ways, this is where we’ve seen improvement in learning over the last century. 

Many of the advancements in instructional theory have focused on formal 

learning experiences, and teachers and administrators made efforts to bring 

those findings into the classroom.2

However, learning isn’t confined to the classroom. The world outside the 

schoolhouse is filled with limitless sources of potential learning. We’re increasingly exposed to torrents of data, questionable “facts,” and diverse unconnected information. It’s incumbent upon the individual—the learner—to 

determine the value of that information and how it connects to other data or 

experiences. The speed and diversion of information in our modern world impacts our abilities to synthesize useful knowledge, effectively retrieve it, and 

translate or apply it in practice. 

Information overload is a significant and growing issue; volumes of data are 

bombarding people at faster and never-ending rates. When exposed to too 

much data, the human brain will tend to focus on the clearest, easiest to understand, most familiar elements—and discard the rest.3

 It’s the body’s natural way of functioning in a focused and emotionally stable state. However, 

in today’s data-rich climate, this sometimes means retention of false or misleading information, which can lead to poor decisions at both individual and 

collective levels. Thus, as the world continues to become increasingly volatile, uncertain, complex, and ambiguous, we need educational practices that 

ensure people are prepared, not only for today’s classroom but for tomorrow’s 

global landscape. 

That preparation doesn’t end at 18 or 25 (or even 100!) years of age. With increasing average lifespans 4

 and worldwide pace of change, continuous lifelong 
64 | Modernizing Learning

learning has become a necessity. New inventions create or destroy whole industries each year, and AI is altering the nature of work in fundamental ways; 

add to that increasing lifespans and the evolving view of employee-company 

permanency. All this means that many people will change careers—not just 

jobs—multiple times within their lives.5 Thus, we need to expand the timeframe of learning beyond K–12 and even beyond traditional higher education 

and vocational schools. While these forms of formal, developmental education are likely to persist for some time, we can expect more learning to occur 

later in life—in the 30 to 65 age range.

It’s time to change course by moving away from incremental 

improvements to our existing education system and instead, 

reimagining how foundational scientific principles can inform 

a new model of learning—one that spans the lifetime. 

LIFELONG LEARNING VISION

Our vision for lifelong learning takes a more naturalistic perspective, acknowledging that learning is pervasive. It happens all the time and everywhere, in 

the classroom, online, at home, and through lived experience. Learning is personal, changing in form based on the unique personality, interests, skills, attributes, circumstances, and beliefs of each individual. It’s fluid and nonlinear. 

Various subjects don’t exist in distinct and disconnected packages; instead, 

diverse concepts that can be learned together. It’s flexible. People can achieve 

success in countless ways via individualized learning trajectories that maximize their unique potential, rather than boxing them into a finite set of “accepted” developmental boxes. It’s holistic. Future learning experiences will 

reach beyond the cognitive domain to emphasize the whole person, including 
INDUSTRIAL AGE PAST

FUTURE

FOCUS

Mastery of 

knowledge and skills

(mostly cognitive and psychomotor)

Holistic development across

facets, merging cognitive, physical,

social, emotional, and so on

Facilitator, mentor, and coach,

within a larger, connected network

EDUCATOR

Expert authority figure; 

learning designer and director

EXPERIENCE

Mostly structured, often passive

and linear, with summative assessments

More personalized and active,

with a greater formative focus

Discrete, episodic, largely age-based

(K-12, higher education, career training)

TIMING

Continuous lifelong learning, 

integrated across experiences 

ACCESS

Limited access choices, usually either

in a face-to-face setting or online

More diverse and blended choices,

truly enabling “anytime, anywhere”

TECHNOLOGY

Dedicated systems in silos,

often focused on formal learning

Distributed systems-of-systems,

an interconnected ecosystem
66 | Modernizing Learning

their social, emotional, and physical development. Education will be designed 

to help cultivate people who can thrive in a complex and chaotic future, rather 

than simply ushering them through the linear, K–12 milestones we have today.

4 KEY TENETS: LIFELONG, 

HOLISTIC, UBIQUITOUS, AND 

ASSET-FOCUSED

Our lifelong learning model includes four main principles. First, as its name 

implies, it considers learning a continuous, lifelong experience. Today, we 

tend to view learning in discrete developmental phases—early childhood, 

then K–12, and, finally, higher education or workforce training. In the future, 

we’ll view learning as an ongoing process, where information is constantly 

synthesized, all the time and from copious sources. The second tenet of this 

model is that learning isn’t constrained to cognitive development. Rather, we 

must recognize learning as an interplay among cognitive, social, emotional, 

and physical skills, attributes, and capabilities. Third, learning involves a mix 

of formal, nonformal, and informal activities. Today, we primarily measure 

and accredit knowledge and skills acquired in formal settings and assessed 

within similar structures. However, in the future, life experience and independent, informal learning will also be measured and recognized as much as—

or, in some cases, more than—formal learning. As our capacity to measure 

learning and experience improves, we’ll also be able to examine individuals’ 

experiences more systematically, to better understand what they know, comprehend, and are capable of achieving. Finally, this is an asset model, not a 

failure model. This means learners of all ages are viewed through a lens that 

considers where they are today and where they’ll grow to tomorrow. 

Each of these tenets is described in more detail below.
Lifelong Learning | 67 

1. Learning is lifelong

Although 90% of brain volume is attained by age 6, learning occurs across 

the lifetime and continues to affect the brain’s capabilities. Certainly, early 

childhood experiences impact individuals’ ability to compensate effectively as they age.6

 However, research on neuroplasticity demonstrates that the 

brain can reroute information and make up for trauma due to brain injury. 

Essentially, people can gain or regain skills otherwise lost during the trauma.7

There’s also significant evidence that neural development continues throughout the lifespan.8

 Although cortical thickness, mass, and connectivity seem to 

decrease with age, adults can compensate by activating interdependent neural 

mechanisms gained from life experience. In other words, although the brain 

develops most rapidly in childhood, learning can effectively occur throughout 

OECD LEARNING FRAMEWORK 2030

The Learning Framework 2030, from the Organisation for 

Economic Co-operation and Development, defines a 

vision and underlying principles for future of educational 

systems. Still a work-in-progress, the framework is being 

developed by a community of experts, school networks, 

teachers, students, youth groups, parents, universities, 

local organizations, and social partners. Its vision is to help 

every learner develop as a whole person, able to fulfill his 

or her potential and contribute to worldwide wellbeing. 

The current version of the framework emphasizes:

• New solutions for a rapidly changing world with diverse global challenges

• New transformative competencies for innovation, responsibility, and awareness

• Learner agency—the responsibility for one’s own education throughout life

• A new, broad set of desired knowledge, skills, attitudes, and values 

• Individual and collective educational goals for wellbeing

• Design principles for eco-systemic change

www.oecd.org
68 | Modernizing Learning

life and is shaped by individuals’ behaviors.9 What and how much individuals 

learn depend on a variety of micro- and macro-level factors. Micro-level factors include individual choices, motivations, and the ability to self-regulate, 

particularly outside of formal education settings. Macro-level factors include 

learners’ neighborhoods, societies, and cultures. 

Some of these factors make adults particularly well-suited for learning. Clarity of interests and goals, and greater self-awareness make this time-frame 

conducive to personal growth and often encourage a greater motivation to 

learn. Adults also have a greater wealth of experiences to draw upon, which 

can help them synthesize new information more deeply and efficiently.10 However, placing the control of learning into adults’ own hands may encourage 

them to focus too narrowly on limited, task-specific forms of learning. We’ll 

need structures that protect and support a comprehensive view of learning. 

Otherwise, we risk having deep experts embedded within stovepiped knowledge communities who lack a general understanding of how the pieces fit 

together to work within a holistic, efficient system.

2. Lifelong learning must encompass 

whole-person development

The ability to effectively participate in life is not exclusively determined by 

one’s cognitive abilities or educational attainment. Rather, resilience, motivation, circumstance, exposure, metacognition, self-regulation, and other personal attributes contribute to a person’s ability to navigate life. This position 

is strengthened by the finding that “brain development and cognition (and the 

connectivity between cortical areas) are influenced and organized by cultural, 

social, emotional, and variability in learning.” 11

In other words, whole-person development necessarily incorporates cognitive, 

social, emotional, and physical capabilities and these are, in turn, influenced 

by cultural systems. 
Lifelong Learning | 69 

Harvard’s 60-Year Curriculum initiative encourages a new 

paradigm of thinking about learning and the education 

process. It recognizes that people learn throughout their 

work lives—and often beyond, into retirement. 

It’s just a subset of the larger territory that we’re looking at; it’s an underappreciated subset but important for our economy and civic health. We 

need to recognize that the world is changing and that we don’t leave people 

out to dry because their first career fizzled out and dried, and we didn’t have 

a mechanism to help them out. Under the spotlight, we have K–12, higher 

education, and retirement, but when you have a career change and the world isn’t 

helping you, it’s murky. We held a conference recently focused on the concept 

of education ages 15–75. We asked, “How do we make that a different span of 

life during which people feel supported? Do we need unemployment insurance?” 

We’re interested in figuring this out. For example, what if I’m really struggling and 

I don’t know if I want to be a researcher or a designer? The real question now is 

what do you want to be first? We didn’t have those dialogs in the past; it’s totally 

different now. 

Christopher Dede, Ed.D.

Wirth Professor in Learning Technologies

Harvard Graduate School of Education

We’re training people for jobs that 

aren’t going to exist anymore.

James Robb

Rear Admiral, U.S. Navy (Ret.) 

President, National Training and Simulation Association
70 | Modernizing Learning

We will need new models of learning and theories of development to effectively address the “whole-person” learning paradigm. To date, much of the 

human-development has focused on the early stages of life (prior to adulthood). As we move away from a front-loaded notion of education and towards 

a lifelong learning concept, we’ll need to expand this body of research to 

incorporate adult learning, changing societal conditions, and the goal of developing more holistic capabilities across time and space.

COGNITIVE DEVELOPMENT

Although mature theories of cognition and learning already exist, these will 

need to be expanded and potentially reevaluated within the future lifelong 

learning model. Discussions of cognitive development usually point back to 

the foundations built by Jean Piaget (1936) and Lev Vygotsky (1978).12 Piaget’s theory of cognitive development defined four critical periods in which a 

young child develops sensorimotor intelligence, preoperational thought, concrete operations, and, finally, formal operations. Interestingly, the final stage 

spans ages 11 to adulthood. People who reach this final stage (and not all do, 

according to Piaget) are able to think abstractly. Since we now know that 

learning occurs throughout an entire lifetime, what happens after reaching 

this stage? Vygotsky’s sociocultural theory of cognitive development offers 

some answers; it focuses on a person’s journey to individualized thinking 

through a co-constructed process of social and cultural interaction. Therefore, 

the individual learns either by using self-regulatory tools (e.g., self-speech) or 

by observing and/or taking direction from others. Though both Piaget’s and 

Vygotsky’s theories recognize the interplay between self-development and directed learning, they take some opposing views; neither accounts for development across the lifetime, and neither consider how a person can achieve a set of 

meta-skills across disciplines, experiences, and formal and nonformal learning. 

Further, technology is changing the nature of human cognition. We can now 

offload data storage and “lower-order” cognitive tasks to computers, aggre-
Lifelong Learning | 71 

gate and analyze large sums of information as never before possible, 

and access content ubiquitously. 

These affordances create the opportunity to exponentially accelerate 

human cognitive development, both 

in time and scale. For example, if 

human brains have finite working 

memory capacities,13 then computers can expand this—to not only 

enable humans to work with more 

information (without task shedding) 

but to also better digest and comprehend wider amounts of information simultaneously. As another 

example, since humans are highly 

influenced by life experiences and a 

computer can provide opportunities 

to experience simulated situations, 

we can expand our store of experiences in significantly shorter amounts of 

time, benefiting from what might be called “unlived experiences.” 

Computers are augmenting human cognitive development, not merely by increasing access to information but by also affecting our brains structurally 

and neurologically. Across its lifetime, the brain will continue to develop and 

learn but also, as new generations are born, they will increasingly have the 

benefit of access to accumulated knowledge and experiences of those who 

came before. In a twist of irony, while the stage theories of Piaget and Vygotsky have been overcome, the basic belief that cognitive development as 

a mixture of human natural capacity building and social-historical influence 

remains correct. What they didn’t foresee was the expansion of capabilities 

that human-computer interactions could achieve.

Key hurdles in developing 

today’s students to be ready 

for life include a lack of early 

childhood experiences and 

foundational language that 

can serve as a springboard 

for later learning opportunities. Expectations are not 

always where they need to 

be, from teachers or leaders; 

consistently higher expectations are needed. 

Nathan Oakley

Chief Academic Officer, Mississippi 

Department of Education
SOCIAL DEVELOPMENT

Like cognitive developmental models, the lifelong learning paradigm obliges 

that social developmental theories be expanded. Social development researchers have primarily studied younger ages 14 or special needs populations.15 No 

doubt, developing social skills in young people is a worthy goal; however, researchers have focused on these populations while sparing much less attention 

for older populations and lifelong social learning. A body of research does 

WE AREN’T BUILDING RESILIENCE IN AMERICANS ANYMORE. 

It starts with little kids, as little as 6 months old. We used to give them a spoon 

and a pot, and they were creative with what they had. Now they’re given little 

kid toys—each toy has one function. These toys have pre-thought goals, and by 

providing them, we’re taking away kids’ creativity. Resilience is not fortitude; it’s the 

creativity to find your way out of a hard situation. It also isn’t singular; it’s social and 

emotional. We can hide feelings but that’s uncomfortable. Instead, we have to learn 

that emotions should be managed; there are times we should be mad and times we 

shouldn’t, and we should know the difference. 

Betty Lou Leaver, Ph.D.

Director, The Literacy Center; Manager, MSI Press; Former Provost, 

Defense Language Institute Foreign Language Center
Lifelong Learning | 73 

exist around interpersonal employment skills, but social tendencies, changes, 

growth, and goals across the lifetime require more attention. 

A recent 20-year retrospective study in the American Journal of Public 

Health found participants with higher “social competence” characteristics, 

such as sharing and cooperating, were more likely to have higher education 

attainment and better-paying jobs. 16

In the lifelong learning model, there’s an expectation that formal education 

will evolve to encompass these skills, in both the formative and later years of 

life. Additionally, we expect that resumés will acknowledge these skills in the 

future. If we’re going to reframe our focus from creating workers to developing whole persons—individuals who can be successful across life experiences—social skills figure prominently into the holistic model. That means not 

only understanding lifelong social skills and finding ways to cultivate them 

but also rewarding individuals who possess them. 

EMOTIONAL DEVELOPMENT 

A prominent model of emotional development, developed by Carolyn Saarni, 

measures emotional competence as a set of affect-oriented behavioral, cognitive, and regulatory skills people develop over time within their social environment.17 These skills include individuals’ awareness of their own emotions, ability to discern and understand others’ emotions based on situational 

and expressive cues, and capacity to cope with distressing emotions using 

self-regulatory strategies. Similar to Piaget’s and Vygotsky’s models, Saarni’s 

model uses phases to categorize the developmental process, and it only addresses development from early childhood to adolescence. Moving toward a 

lifelong learning model of education requires more research on adult emotional development as well as the impact of individuals’ emotional wellbeing (e.g., 

mental health, and ability to deal with stress) across all ages. 
74 | Modernizing Learning

The Collaborative for Academic, Social, and Emotional Learning,18 a not-forprofit dedicated to enhancing social and emotional learning, recommends a 

more robust model that integrates intrapersonal, interpersonal, and cognitive competence. It includes five key areas that encompass various behaviors, 

mindsets, strategies, and skills: 

• Self-awareness, such as accurate self-perception and self-efficacy

• Self-management, for instance, impulse control 

• Social awareness, including empathy and respect

• Relationship skills, such as teamwork and communication

• Responsible decision-making, including reflection and ethics

Research suggests that early emotional regulation skills have a significant impact on development and outcomes in later life.19 For example, emotional regulation is part of the spectrum of skills needed to be successful in the classroom. The emotional regulation and interpersonal strategies children develop 

in early years allow them to navigate the school system, and more than that, 

these skills become key tools for success in life—arguably more than the academic knowledge itself. But can these skills be taught? Substantial evidence 

exists 20 that suggests: yes. Explicit teaching of social and emotional skills 

There’s a significant emotional impact of constant change and 

intellectual learning required for a multi-career expectation. We’re 

living in constant acceleration, and we’re trying to keep up with 

it. So, the question is: What’s the foundation we need to provide 

people so they can thrive on chaos? Some of the answer lies in 

really raising what we think about teamwork—something the 

military studies very deeply. The team becomes the buffer on which 

the group defends. 

Christopher Dede, Ed.D.

Timothy E. Wirth Professor in Learning Technologies in the Technology, 

Innovation, and Education Program, Harvard University
Lifelong Learning | 75 

leads to better interpersonal skills and decreased anti-social behaviors, and 

it also improves students’ academic achievement. The interactions between 

social and emotional development and outcome performance make sense. For 

instance, consider that distractions of any type during learning, including 

internal anxiety, stress, or personal or professional challenges, can detract 

from one’s ability to acquire and encode new information. However, emotional regulation, resilience, and persistence can improve both learning as well 

as decision making under stress.21 Accordingly, emotional regulation skills 

developed early can improve long-term functioning and can also be improved 

with time, experience, and formal education. Nonetheless, more inquiry is 

needed to examine how such capabilities directly impact adult performance 

and lifelong learning, and importantly, improved developmental metrics and 

instructional approaches are needed for honing these skills in life. 

PHYSICAL DEVELOPMENT 

Formal investigation into motor and physical development traces its foundations to the 1920s, when doctors began weighing infants to determine if they 

met appropriate growth benchmarks.22 More significant research began in 

earnest the 1970s and 1980s, spurring significant advancements in the understanding of average motor development, constraints both within and external 

to a person, and the benefits of aiding, enhancing, and improving motor skills. 

However, like other developmental domains, much of the research in physical 

development has been limited to early childhood and disorders, with some 

unique focus areas for special populations such as sports and military personnel. Yet, beyond the scope of these specific groups, general physical maturation and the impacts of motor skills and practice have been less studied, 

although that is changing. 

Body development, awareness, health, and wellbeing have large impacts on 

long-term functioning. Increasingly, improved methodologies and new technologies are creating ways to better understand how a body develops into and 
76 | Modernizing Learning

As a society or culture, if 

we think about reimagining 

learning as a lifelong endeavor, 

we’ll help so many kids. 

We need to get out of the 

structure of grades and look 

at learning as “I’ve mastered 

it now, or I haven’t yet.” We 

need to tailor education. 

Michelle Cottrell-Williams

Teacher, Wakefield High School, 2018 

Virginia State Teacher of the Year

across adulthood, how physical capabilities can be honed, and how these connect 

to other developmental domains such as 

emotional stability, social capabilities, and 

cognitive development.23

Simultaneously, wearable devices and the 

so-called quantified self 24 have created 

enthusiasm about improving physical activity and the nuances of each individual 

body. They are providing individuals with 

the ability to have access to personalized 

data that was previously unavailable and 

empowering people to make improved 

decisions about their health and physical 

activities as a result.25 The medical benefits of these technologies have not yet 

been fully understood at a societal level, 

nor have they been fully utilized for optimizing human motor capabilities outside 

of specific, controlled settings, such as 

Olympic athletic training. However, as the 

research continues, it’s not unreasonable 

to believe a new theory of physical and motor development that encompasses 

average, lifelong populations will be forthcoming—one that actively incorporates considerations for human-technology interaction, the processes and 

impacts of physical development across society, psychophysical literacy, and 

the interplay of motor development with social, emotional, and cognitive development. 

Understanding, philosophically, the holistic connectivity of human capabilities, and how behaviors are enacted across contexts, will be important within 
Lifelong Learning | 77 

a whole-person development model.26 A better understanding of the self, to 

include the physical self, is needed to achieve more holistic, personalized, 

developmental trajectories.

3. Learning is ubiquitous

Lifelong learning comprises all phases of learning and stages of life, and it 

occurs across diverse contexts, from school to the workplace, at home and 

within the community.27 Lifelong learning activities can happen in formal 

settings (e.g., courses offered by a university), nonformal contexts outside of 

fully structured institutions (e.g., meet-up workshops), and in informal and 

spontaneous ways (e.g., while chatting with a co-worker or reading a post on 

social media).28

Learning already occurs in all of these ways, all the time, and everywhere. To 

date, however, we’ve largely documented (and, subsequently, largely valued) 

only formal learning experiences. Informal and experiential learning can have 

as much, or even more, impact on individuals’ abilities to acquire, assimilate, 

and apply knowledge. With the development of data science, machine learning, and interoperable data standards that allow us to measure and classify 

experiences, we’re unlocking the ability to better capture and communicate 

a person’s true skill level as well as his or her ability to perform in a variety 

of settings and across communities. It’s irrelevant where a person “learned” 

something—the transfer of that learning into practice is what matters. 

The idea that learning happens everywhere and all the time isn’t new. Rather, 

it’s our ability to measure it and communicate about it (e.g., through competency badging and credentialing) that’s novel. This also ties to the whole-person principle described in the preceding subsection. That is, various skills 

contribute to someone’s success in the world. In military contexts, for example, there’s much talk of grit and resilience, and in higher education, we often 

reference executive functioning and well-roundedness; however, such capaci-
78 | Modernizing Learning

ties are rarely measured or reported in transcripts and personnel records. Assessing their applications in real-world contexts and giving “credit” for other 

lived experiences will also enable our ability to create personalized learning 

trajectories, improve talent management into the future, and create equitable 

opportunities for more people.

4. Lifelong learning must employ an asset model

In developmental psychology, an “asset model” refers to an approach that recognizes individuals’ unique assets and focuses on adding capabilities to them. 

This concept is compared to the “deficit model,” which focuses on areas of 

weakness and involves comparing individuals to group averages. The benefits 

of using an asset model are several-fold. First, there’s a psychological benefit 

in the form of increased energy and the improved outcomes that result when 

a positive focus is used for learners. This can be seen in sports psychology 

with relation to performance on the field 29 and is directly translatable to the 

classroom or boardroom. Building people up to an optimal capability is far 

more encouraging than forever attempting to “fix” them. 

STACKED CREDENTIALS: For example, in business, a student takes 

three courses and completes them satisfactorily, and they earn a 

certificate in the area of finance. After navigating that successfully, 

they take three courses in marketing and receive another certificate. 

Then those groups of certificates are stacked into a personalized 

master’s degree. This approach allows the student to acquire 

credentials in bite-sized chunks and offers more flexibility.

David Munson, Ph.D.

President, Rochester Institute of Technology 
Lifelong Learning | 79 

Second, asset models help support whole-person development. Asset models 

better allow for the inclusion of skills and attributes outside of those measured 

on averaged, norm-referenced assessments. By looking at these other factors of 

success, we can better recognize, help develop, and otherwise enable such skills. 

Finally, an asset model can better support a focus on continual, lifelong learning. The structure of this type of model naturally defines success at every 

level, with every addition, and yet has an infinite number of notes, skills, and 

competencies that one can attain. The reframing of both the learner and the 

educational system can aid in the reimagination and refocus on how we can 

improve the system and work toward optimization of each individual, rather 

than focusing on creating able-workers ready for an industrialized nation.

IMPLEMENTATION

The previous section outlined a vision for lifelong learning in the future. This 

section outlines specific steps we can take towards that vision.

USE MULTIPLE THEORIES TO INFORM EDUCATIONAL DESIGN. Lifelong learning means learning across time, space, purpose, media, and formality. We’ll need to transition this strategic-level concept into tactical-level interventions for classrooms, workshops, training exercises, experiential 

learning, and other formal and nonformal activities—implementing and integrating theoretical approaches from multiple disciplines, including instructional design, information management, educational psychology.30

STACK CONTENT-SPECIFIC, CONTENT-AGNOSTIC, AND SOCIAL AND 

EMOTIONAL LEARNING. Refocusing education to incorporate a holistic 

view of human development will necessarily require a shift in educational 

requirements. However, the ability to add more requirements to an already 

packed schedule isn’t reasonably feasible. Rather, we’ll need to change the 
80 | Modernizing Learning

organizational structure of formal education and training pipelines as well as 

take advantage of project-based learning options where multiple skills across 

the cognitive, emotional, social, and physical domains can be simultaneously developed. Content-agnostic learning strategies, for meta-skills such as 

self-regulation and executive functioning, will also need to be learned at the 

same time. A stacking of skills, content, and connectivity across topics should 

become the norm, rather than the exception, particularly for formal and nonformal education.

MAKE TECHNOLOGY INTEROPERABLE TO MEASURE AND CONNECT. 

This vision of lifelong learning depends, in part, on the collection and analysis 

of learner data. To enable that, we’ll need to first define measures appropriate 

for formal, nonformal, and experiential learning. We’ll also need to develop 

the associated technology, including interoperable systems that can safely and 

ethically aggregate data across time, space, and communities. This “internet 

for learning” will need to securely store a person’s data and make it accessible, across a lifetime, by approved entities who can use those data to personalize learning episodes and developmental trajectories.

USE THE SCIENCE OF LEARNING TO OPTIMIZE THE LIFELONG 

LEARNING SYSTEM. Learning can be enhanced by employing a set of instructional principles, such as specific teaching and assessment principles. 

As described in the preceding chapter (Chapter 3), many existing instructional theories already articulate well-documented best practices for supporting 

evidenced-based teaching and testing. However, we need to widen our perspectives—to consider whole systems, the range of interacting micro- and 

macro-factors, and their interplay across space, time, and purpose. To accommodate individualized pathways through education programs and other 

developmental experiences, we’ll also need to change how information flows 

and how people progress through the system. This will impact secondary and 

postsecondary education, trade training, workforce development, and life 

experiences. While it’s possible to allow technology advancements to drive 
Lifelong Learning | 81 

these changes, it would be wiser to help cultivate the ecosystem more holistically. We need to collect evidence and recommend best practices about the 

elements within it and their collective impact as well as incentivize those elements that bring out its best features—for individuals and society, writ large. 

Learning science, both its extant research and its inquiry principles, can aid 

this endeavor, but we must commit to using it for this larger vision.

The Department of Defense focus falls short by focusing on eighteen- to 

nineteen-year-olds and not thinking about how we can support kids at the 

younger ages. So, by the time we get them in DoD, we’re dealing with 

resilience issues and putting band-aids on problems. We spend 20 years 

building a new weapon system but our kids in second grade are going to 

be in DoD in 10 years. The first thing DoD needs to do is consider learning 

as a continuum to include civilian education. Social emotional learning and 

executive functioning need to be a focus. There’s a whole bunch of things 

that need to be mitigated before we get them in DoD. 

Russ Shilling, Ph.D.

Chief Scientific Officer, American Psychological Association

Former Senior Innovation Fellow, Chan Zuckerberg Initiative; Former Executive 

Director of STEM, U.S. Department of Education; Former Program Manager, 

Defense Advanced Research Projects Agency; U.S. Navy Captain (Ret.)
82 | Modernizing Learning

Too often, we define the symptoms, not the underlying issue. 

We tend to problem-solve instead of problem-find. As you 

try to create these innovative things, you’ve got to do a really good 

job separating problem and symptom as you build the ecosystem. 

The idea of an ecosystem is based on interdependencies, so a 

technological ecosystem has to work like a biological ecosystem: 

At the same time, it must take into account all the components—to 

include the people. These solutions are often developed without the 

human and user in mind. People don’t always think about the end-user 

when they develop technology. 

Jeffrey Borden, Ed.D.

Executive Director, Inter-Connected Education; Chief Academic Officer, 

Ucroo Digital Campus; Former Chief Innovation Officer, St. Leo College
Learning Experience Design | 83 

CHAPTER 5

LEARNING 

EXPERIENCE DESIGN

Sae Schatz, Ph.D.

The phrase “fog of war” is generally attributed to Prussian military theorist 

Carl von Clausewitz, who wrote his quintessential treatise, On War, in the early 19th century. In it, he describes war as the realm of uncertainty; this gives 

rise to our classical understanding of the “fog” as a state where information 

is scant, unreliable, and hidden from view.1 However, in the modern world of 

smartphones, broadband, and social media, this concept is taking on a different cast. Today’s “fog” isn’t caused by a dearth of information but rather by 

the overwhelming glut of it. The quantity of resources represents only one 

part of the challenge. So much of the available information is inaccurate, contradictory, inapplicable, or disconnected. There’s a signal-to-noise problem. 

Added to all of this, we’re expected to monitor multiple information feeds, 

carryout parallel multitasking, and pay attention to alerts and interruptions.2

Sometimes humorous phrases—infobesity, infoxication, data smog, or info 

pollution—describe the phenomenon, but its effects are no laughing matter. 

One result of the pace and abundance of resources is, paradoxically, a drop in 

productivity. For example, workers need an average of ≈25-minutes to “reset” 

after being interrupted by a work email, and such distractions account for 

around one-third of the time a typical knowledge worker spends on the job.3

In addition to issues with efficiency, information overload can profoundly 

impact effectiveness. Notably, it dangerously affects attention, encoding, and 

decision-making processes. For instance, when overloaded, individuals are 

 

 
84 | Modernizing Learning

more likely to monitor the most superficial data and defer to familiar concepts while ignoring conflicting evidence. Attention-deficit disorder specialist 

Thomas E. Brown has even found that most people, i.e. those without the 

syndrome, report symptoms similar to it multiple times a day, including the 

inability to concentrate and to pay attention to what needs to be done.4 In 

decision-making contexts, overload depletes mental resources, driving individuals to expedient (rather than optimal) choices, encouraging them to avoid 

decisions or defer to negative or default options, and allowing unrelated emotions to play an undue role. 

 We’re data rich but 

 increasingly knowledge poor. 

Unfortunately, as discussed in the preceding chapter (Chapter 4), creating 

“more” education and training won’t solve this problem. In fact, as we look towards the future learning ecosystem, with its vision of diverse and pervasive 

lifelong learning, we run the risk that—rather than optimizing our learning 

and development—we instead add to this destructive cacophony. The learning ecosystem has other potential pitfalls, too; for instance, like today’s world 

wide web learners might be faced with the daunting task of independently 

curating and synthesizing their own instructional resources. Further, with its 

reliance on technology, poor usability and breakdowns with other nonfunctional requirements (so called “-ilities”) could become insurmountable barriers to its effective and efficient use. In other words, without care, there are 

an excess of ways that the learning ecosystem could add to the “noise” rather 

than strengthening and clarifying the “signal.” 

Solving this problem will require several concomitant solutions. Notably, 

applying holistic instructional strategies (Chapter 12), developing learners’ 

self-regulation abilities (Chapter 15), and thoughtfully applying automated 

personalization (Chapter 10) are all essential. In addition, the intentional inte-
Learning Experience Design | 85 

gration of these practices, along with the strategic design of learning systems 

and careful attention to their practical interaction details must be considered. 

Hence, this chapter focuses on the design of learning experiences as a necessary complement to the other critical elements informing the future learning 

ecosystem. 

LXD: DESIGNING HOLISTIC, 

LEARNER-CENTERED 

EXPERIENCES

Broadly defined, design refers to a series of interrelated actions, purposefully taken to achieve specific outcomes or goals. People often associate the 

word with artistic activities, such as painting or fashion; while it fully applies 

One of the key hurdles in developing students for life is that 

we’re still trying to assess them on information from the 

past—the way we used to teach. Forty percent of students 

will work on jobs that don’t exist yet. We need to teach them 

the skills to collaborate and innovate....If we can google it, 

then we shouldn’t spend our time teaching it! I need to be 

able to facilitate their learning.

Michelle Cottrell-Williams

Teacher, Wakefield High School

2018 Virginia State Teacher of the Year
86 | Modernizing Learning

to these fields, “design” also pertains to any problem-solving discipline that 

uses a combination of grounded knowledge, skill, and creativity. For instance, 

teachers may design a curriculum for optimal transfer-of-training, and software developers may design a new app for security and reliability. Even military leaders discuss operational design as a core element of their planning 

processes.

Learning experience design, abbreviated as LX or LXD, is a relatively new 

concept, originating around a decade ago.5 It largely grew out of user experience design: 

The term “user experience” or “UX” wasn’t always an overused Silicon Valley buzzword. Coined in the mid ‘90s by Don Norman, while 

he was vice president of advanced technology at Apple, it refers to an 

abstract way to describe the relationship between a product and a human. Back then, Norman argued that technology must evolve to put user 

needs first—the opposite of how things were done at the time. It wasn’t 

until 2005 that UX gained mainstream relevance: 42 million iPods were 

sold that year and the mass market experienced great design at scale. ...

Instructional design is now approaching a similar transition.6

With roots in UX, it’s unsurprising that educational technologists were among 

the first to embrace LXD, nor that much of the discussion around it has concentrated on design thinking, usability, and interaction design methods for 

technology-aided learning. LXD practitioners also frequently emphasize the 

application of user-centered design, sometimes drawing a distinction with conventional instructional design by contrasting LXD’s learner-centered methods.7

 Increasingly, though, LXD proponents are widening its scope beyond 

(learning) product design, focusing more on broad learning outcomes with 

an extensive toolkit to apply towards this end. For instance, Margaret Weigel 

and her colleagues with Six Red Marbles have begun emphasizing LXD’s holistic approach to design and its synthesis of instructional design, educational 

pedagogy, neuroscience, social sciences and UI/UX principles.8 There’s also 
Learning Experience Design | 87 

growing consideration for informal and social learning, game-based learning 

methods, neuroscience-informed principles, and the shifting role of teachers 

from learning providers to learning facilitators. The field, however, still has 

some maturing to do, and several related disciplines can help inform this.

Industrial Knowledge Design, or InKD (pronounced like “inked”) developed around the same time as LXD, and it shares a similar focus: 9

InKD...describes an approach involving interrelated techniques drawn 

from diverse evidence-based scientific disciplines, aesthetic principles, 

and professional best practices which together help practitioners more 

effectively and efficiency achieve purposeful knowledge transfer goals 

and objectives. 

Like LXD, InKD considers interaction design and usability principles, and in 

many practical ways the two concepts overlap. InKD, however, grew out of 

different foundations and, as such, contributes some unique perspectives. It 

adds to LXD by identifying a set of (1) foundational scholarly fields to draw 

upon for theories and concepts as well as (2) practical applied fields from 

which to derive actionable tools and processes. Specifically, InKD draws 

from information science fields concerned with the analysis, collection, classification, manipulation, storage, retrieval, movement, dissemination, and 

protection of information. These include, for instance, instructional design, 

knowledge management, informatics, semiotics, and media design. It synthesizes these with neurocognitive fields concerned with how individuals interact 

with data, process information, and form knowledge; these include, for example, learning science, cognitive science, human factors psychology, cognitive 

ergonomics, and marketing. 

The stated goal of InKD practitioners is to use evidence-based techniques to 

increase individuals’ motivation to receive information, its effective conveyance, recipients’ encoding and later retrieval of that information, its actionability, and the overall impact of communications. In contrast to LXD, InKD 
88 | Modernizing Learning

has taken a more academic route, which 

contributes definitions and conceptual 

linkages to the burgeoning discipline. 

This helps ground LXD in established 

theory and evidence-based practice, and 

it gives LXD designers a full “rolodex” 

of disciplines with methodologies and 

tools ripe for use.

For example, marketing and related disciplines, such as consumer behavior, 

public relations, and advertising, offers 

ample guidance applicable for learning. 

While that may sound surprising, in 

practice, marketing and learning professionals share many similar goals: Both 

try to understand their audiences, generate motivation, capture attention, make 

their messages memorable, and affect 

their audiences’ downstream behaviors. 

Of course, marketers generally want to 

sell products or services, while learning 

professionals may seek to foster an accurate and robust understanding. Still, 

the techniques are often the same.

One distinctly applicable approach from marketing is experience design. It’s 

a practice usually used in business and entertainment contexts to elevate routine customer “interactions” into more compelling and memorable customer 

“experiences.” Experiential designers are successful when they encourage 

people to create meaningful emotional and social connections and to construct 

personal narratives that involve episodic memories and positive associations 

with the artifacts of that experience (such as a product, in marketing terms).10

But there’s this whole 

other world, conceptually, 

in different sectors who 

aren’t having conversations 

with each other. It’s 

shocked me that people 

really are doing it in silos.

Emily Musil Church, Ph.D.

Executive Director of Global Learning,

Prize Development and Execution, XPRIZE
Learning Experience Design | 89 

Experiential design practitioners assert that well-designed experiences convey a more salient “sense” of a product or brand, enhance customer emotions 

towards it, build loyalty, and ultimately enhance revenue.11 Applications of it 

have supported these claims; for instance, a major hospital faced with increasing competition and declining customer-satisfaction used experience design 

to create a 13% increase in perceived quality of care and a decrease of 33% 

in customer complaints with no other facility management changes.12 Other 

successful use cases, from car rentals to circus entertainment, have also been 

reported,13 and we’ve likely all experienced the effects of well-designed consumer experiences firsthand at theme parks or popular coffee shops.

Philosophically, experiential design isn’t too different from classical experiential learning. Popularized by David Kolb, experiential learning is “the 

process whereby knowledge is created through...the combination of grasping and transforming experience.” 14 Experiential learning recognizes that not 

all experiences enrich learning. Instead, meaningful learning occurs when a 

learner “‘touches all the bases’—experiencing, reflecting, thinking, and acting—in a recursive process...” 15

Experiential learning theory offers a useful model for conceptualizing the 

processes, and proponents of it have published extensive theories, techniques, 

and studies about it—some quite useful for LXD.16 However, like much of 

traditional instructional design, experiential learning theory generally takes a 

straightforward approach, focused on cognitive processes with less attention 

for emotional and social mechanisms, and it tends to treat learners as motivated, self-regulated, and logical actors. This is a place where marketing can usefully augment educational theory. Experiential designers take more holistic 

approaches, beyond rational cognition or even the immediate experience, and 

they focus more on practical outcomes. For example, experience design offers 

a set of tools for selectively manipulating contextual variables to influence 

experiences and for creating these outcomes at scale. One popular framework 

involves five categories that designers need to affect, and when all five are 
90 | Modernizing Learning

successfully integrated, they form a “holistic experience”: 17

• Sense – Reactions to sensory stimuli within or around an experience

• Feel – Emotions and their intensity in response to an experience

• Think – Mental engagement, e.g., problem-solving or creative thinking

• Act – Personal identity and behaviors; a desire to engage or act

• Relate – Experiences that provoke a social identity; co-experiences

Experiential designers, and marketers more broadly, tend to more willingly 

accept the reality that humans aren’t rational actors. This gives them more “levers” for affecting outcomes, and it frees them from unfeasible expectations 

about the logic of consumers’ (or learners’) thoughts and actions. The study 

of why and how people make seemingly illogical decisions has grown in popularity over the last 20 years. Today, under the name behavioral economics, 

practitioners have defined a litany of routine decision-making biases, mental 

heuristics, and cognitive filters that, largely, everyone uses. 

Behavioral economics grew out of work by Nobel Prize recipients Herbert A. 

Simon and Daniel Kahneman (among others), and it’s been popularized by 

Dan Ariely18 and Freakonomics authors, Steve Levitt and Stephen J. Dubner. 

It also has roots in the psychology of influence and persuasion, notably from 

work by Robert Cialdini.19

Behavioral economists Cass Sunstein and Richard Thaler (who also received 

a Nobel Prize for his work) have expanded the field, widening it to explore 

ways to “nudge” decisions at large scales. Their canonical book, Nudge,

20 outlines principles for subtly coaxing people towards better choices. Proponents 

have used these to great effect. For instance, Collin Payne and colleagues used 

small cues at a grocery store to increase shoppers’ likelihood to buy fresh 

fruits and vegetables (e.g., designated sections for produce in shopping carts 

and big green arrows on the floor). These yielded a 102% increase in purchasing for fruits and veggies, with 9 out of 10 shoppers following the green 

arrows to the produce section when first arriving at the store.21
Learning Experience Design | 91 

UX design, experience design, behavioral economics, and nudge all highlight 

ways in which subtle features and thoughtful design can influence outcomes. 

But when designing a new system—whether for learning or performance—

how do you think through all of the factors potentially affecting behavior? 

How do you ensure the various elements are designed in harmony and with 

common ends in mind? Human–Systems Integration (HSI) offers some answers here.

HSI is a philosophy and set of processes that focus on systems-level human 

performance design and development activities. It grew out of the U.S. Department of Defense after a 1981 General Accounting Office report revealed 

that 50% of all military equipment failures were caused by human error and a 

corresponding U.S. Army report that found that many military human errors 

could be traced back to poor development processes that failed to sufficiently 

consider human performance concerns.22 Basically, HSI combines systems 

96%

11% According to polls conducted jointly by Gallup and the 

Lumina Foundation, 96% of chief academic officers at 

higher education institutions felt their programs were 

“very” or “somewhat” effective at preparing students 

for the world of work—but only 11% of business leaders 

strongly agreed. Business leaders said graduates lack the 

skills and competences their companies actually need. 

Source: Preety Sidhu and Valerie J. Calderon (2016). https://news.gallup.com
92 | Modernizing Learning

engineering methods, human factors principles, and human-centered design 

practices—yielding a practical toolkit for designers of any system that includes people, technology, and desired organizational outcomes. 

HSI has four core tenets:

► Emphasize Humans – Emphasize human performance early and often 

in the system design process; give humans equal treatment to hardware 

and software 

► Optimize the Total System – Optimize overall system performance at 

the comprehensive (big picture) level and not simply at the individual 

component levels

► Consider the Full Lifecycle – Take a long view; maximize a system’s 

benefits—while controlling its costs and mitigating risks—across the 

entire system lifecycle

► Facilitate Design – Facilitate multidisciplinary design; help “translate” 

among specialists in different disciplines as well as between designers 

and other stakeholders

EMPHASIZE

HUMANS 

OPTIMIZE THE TOTAL 

SYSTEM

CONSIDER THE FULL 

LIFECYCLE

FACILITATE

DESIGN

Human–Systems Integration is a philosophy and set of 

processes that focus on systems-level human performance 

concerns throughout a system’s lifecycle. Its purpose is to 

mitigate the risk of downstream system failure. 
Learning Experience Design | 93 

Under each tenet, HSI practitioners have developed systematic processes, design tools, and documentation methods. While many of these are designed 

for projects involving highly complex sociotechnical systems (e.g., building 

a new aircraft carrier), they can provide LXD designers, at any level, with 

inspiration and an extensive toolkit to draw from, and HSI’s core tenets serve 

as valuable touchstones for LXD, as well. 

Summary

Each of the disciplines discussed in this section can contribute to a maturing 

understanding of LXD. The foundations of LXD create its underlying 

philosophy and conceptual paradigm, and its underpinnings in UX offer 

readymade design thinking principles and user-centered design processes 

applicable for learning contexts. InKD widens this aperture to more fully 

integrate information science and neurocognitive science, along with their 

subfields. In so doing, InKD brings an array of grounded theories and applied 

tools to LXD. 

Commercial fields also offer useful methods. For instance, experience design has concepts, methods, and use-cases for constructing memorable and 

motivating holistic experiences, often at scale through mass customization 

techniques. Similarly, behavioral economics helps us understand more about 

individuals’ real-world (“predictably irrational”) decisions, and it teaches us 

ways to “nudge” behaviors, whether to persuade individuals or shift whole 

communities. 

Finally, LXD designers can leverage the four HSI principles as well as its 

robust collection of established processes and developer tools. Notably, HSI 

uniquely contributes methods for integrating human-centered design principles with systems engineering, balancing local outcomes against global considerations, and facilitating these designs at scales within production teams 

and formal organizations.
94 | Modernizing Learning

RECOMMENDATIONS

Each of the fields of study discussed so far offers a wealth of insights for 

learning design. Below is a list of recommendations drawn from across them, 

although it surely only scratches the surface. 

1. Identify and focus on the actual goal 

Across all application areas, a prerequisite of effective design is its conceptualization as a goal-directed process. While this may sound evident, too often 

people fail to identify the actual goal, and instead focus narrowly on immediate actions or process outcomes, without thinking through the larger “why.” 

Consider, for instance, compliance training—something many of us have endured. Originally, the true goal of a compliance course may have been to 

address some actual risk, say, to train employees to avoid cyber-scams. The 

program manager assigned to the job, however, may inadvertently change 

the goal from reducing cybersecurity incidents to mitigating organizational 

risk—a seemingly small change. As the job progresses, the goal drifts further, 

from designing training that mitigates organizational risk to creating an intervention that shifts risk. This, in turn, may influence programmatic decisions; 

for instance, the program manager might begin to view the mere exposure to 

training information (rather than effective transfer-of-training) as sufficient 

for shifting the risk. 

Logically, then, the program manager may select the most economical approaches for creating that exposure. Meanwhile, the instructional designer is 

likely given a stack of materials and told to “train” employees on them—albeit with limited resources. Now, his apparent goal becomes communicating 

as much information as possible under challenging constraints. Subsequently, 

supervisors’ goals become checking off each employee from a completion 

list, and employees’ goals become completing the training as quickly as pos-
sible....and so on until, ultimately, 

everyone’s best intentions yield limited actual utility. 

UX and user-centered design have 

proven processes for uncovering 

strategic goals and designing solutions for them; so, LXD already 

excels in this area. Jesse James Garrett’s Elements of User Experience 23

is an oft-cited resource for learning 

designers, even though his work 

focuses on digital product design, 

more generally. His five-layer model 

starts with Strategy (defining goals 

and user needs), and then progresses through Scope (requirements 

and specifications), Structure (interaction models and architectural 

design), Skeleton (interface, navigation, and information designs), and 

Surface (sensory elements and aesthetics) elements. 

Application of Garrett’s methods, or 

similar goal-focused design processes, can profoundly and positively affect 

learning design. Using these sorts of approaches means focusing on outcomes 

rather than processes. They also require that designers (at all levels throughout the processes) challenge assumptions, strive to understand and work towards strategic (rather than just local) goals, and consider creative approaches 

that fall outside of traditional practices, such as using informal interventions, 

holistic experience design, or nudge techniques. 

We did a fairly broad study with 47 

large, well-known companies from 

around the world, and we synthesized 

the attributes of their learning 

organizations. In all cases, what we 

found was that they are missionfocused. They created an architecture 

clarifying how data-driven decisions 

about training connect to the mission. 

Their organizational structures focused 

on growing internal people and were 

really helpful to outcomes and buy-in.

Michael Smith

Senior Technical Specialist, ICF
96 | Modernizing Learning

2. Apply holistic user-centered design methods

Results published by the National Academies Press show that only 34% of 

technology development projects in the U.S. are successful, and projects most 

frequently fail because “(1) an inadequate understanding of the intended users 

and the context of use, and (2) vague usability requirements, such as ‘the system must be intuitive to use.’” 24 As education and training increasingly rely 

upon technology, it’s important to incorporate UX, interaction design, human 

factors, ergonomics, and other closely related human-centered disciplines into 

learning design processes. 

User-centered design is more than just usability. It needs to consider people holistically. Experience design offers some insights here. For instance, 

rather than focusing largely on cognition, also consider other internal processes such as emotion, confidence, and motivation. Recently, the Interaction 

Design Foundation published an article highlighting how LXD, like all other human-centered design applications, is really attempting to solve one (or 

more) of these five common problems—only one of which directly addresses 

cognition: 25

• Lack of knowledge – Doesn’t understand the material or instructions

• Lack of skill – Lacks skill, practice, or ability to apply knowledge

• Lack of confidence – Lacks positive, yet realistic, self-perceptions 

• Lack of motivation – Disinterest in applying cognitive effort or action

• Lack of resources or tools – Problems that prevent otherwise 

knowledgeable, skills, confident, and motivated persons from acting

Again, it’s important to consider these dimensions creatively and holistically. 

As Bror Saxberg, VP for Learning Science at the Chan Zuckerberg Initiative, 

has pointed out: “Even physical and mental health matter—a very hungry 

student is unlikely to start, persist, or put in mental effort no matter how 

gloriously designed a learning experience he’s put in. Getting students access 

to a healthy breakfast is potentially a great personalization of the learning 
Learning Experience Design | 97 

environment!” 26 In other words, in learning, sometimes more effort needs to 

be invested in providing resources (beyond those with apparent education or 

training utility), refining the learning context, or instilling confidence. Think 

beyond pure information conveyance! 

3. Design for real—messy, irrational—humans

Cognitive science and behavioral economics teach that humans are predictably irrational. We’re prone to making expedient (rather than optimal) decisions, substantially more motivated to avoid loss than seek gain, and vulnerable to a slew of other biases. Recognize that learners have these “flaws.” That 

doesn’t imply you should deceive or condescend—none of us is a rational 

actor! Rather, acknowledge and design for the messiness of humanity. This 

may mean, for instance, designing for emotional effect or carefully avoiding 

information overload during a learning experience. 

As part of a creative, holistic user-centered design approach, also consider 

nudge techniques to augment the more obvious learning interventions. Nudges can help individuals overcome inherent biases and might be useful, for 

example, in encouraging self-regulated learning practices, such as studying or 

reflection. Also, reach beyond the straightforward cognitive domain, and consider nudges related to other behaviors that may impact learning, like wellbeing and self-care. Behavioral economics and nudge theory offer excellent 

examples to inspire these interventions. Related fields, including industrial 

design, graphic design, and communication, also offer tools for designing interfaces, spaces, contexts, and content elements to achieve persuasive effects.

4. Design holistic experiences

Nothing exists as a simple point in time. As experience design and instructional theory both teach, a given experience is preceded by a preparatory or 

anticipatory phase, and it’s followed by a reflective one. Design for these pre- 
98 | Modernizing Learning

and post-learning phases to the extent possible. Further, an experience has 

different components. Drawing from experience design, it’s useful to intentionally design across the full range of sensory stimuli (sense), emotional factors (feel), cognitive elements (think), personal connections and engagements 

(act), and social identity/co-created elements (relate). Also consider the collective effect of integrating these five facets, and think about how to address 

them before, during, and after a learning experience. 

Similarly, don’t forget about the power of aesthetics when designing for humans. Psychological research actually shows that “pretty things work better”—that is, individuals’ perception of aesthetics directly impacts their performance outcomes.27 Such aesthetic principles have been well codified for 

most media by applied creative types; however, practitioners of more “serious” disciplines are often more hesitant to invest in them. In fact, some 

subcultures, such as certain academic disciplines or military sectors, wholly 

reject the application of aesthetics (under the assumption, presumably, that too 

much polish will detract from the “seriousness” of the message—even though 

scholarly research supports the positive impact of quality aesthetic design). 

Some experiences are all about people-to-people interaction. There 

are lots of things that can’t be learned online, such as hands-on design 

projects or working on a start-up company. Sometimes you have to sit 

with your buddies (fairly intensively, for a few years) or travel overseas. 

Sometimes you have to be there and you have to immerse yourself.

David Munson, Ph.D.

President, Rochester Institute of Technology 
Learning Experience Design | 99 

We’re only beginning to understand the psychology of emotional design even 

though it’s been around for decades. Yet, it has formidable promise for LXD.

5. Use systematic processes to design effectively 

within larger organizations

Increasingly, learning professionals are working in diverse production teams 

and deploying interventions at larger scales. This marks a shift in the way education and training interventions are developed: Where once they were largely 

artisan creations crafted independently by experts, they’re progressively more 

likely to be designed and implemented by teams and situated within larger 

organizations. HSI offers useful tools for navigating the practical challenges 

that come with these changes. 

A first lesson from HSI is to consider a long view of a system attempting to 

maximize its benefits, while controlling costs and mitigating risks, across its 

entire lifecycle; that is, through its initial design and development phases, 

along with its implementation, operation, and eventual retirement stages. The 

point is, when designing a new process, system, or learning experience, consider it within the context of the organization across time: How will it be 

designed and eventually built? How will it be rolled-out to stakeholders? How 

will it be maintained and continuously improved over time? When should it 

be retired?

HSI similarly offers methods for conceptualizing organizational components. 

Typically, HSI practitioners recognize the manpower, personnel, training, 

safety and occupational health, human factors engineering, habitability, and 

survivability domains. They try to take these factors into account when created integrated designs. For instance, if enough operators (manpower) aren’t 

available, then they might increase the experience requirements of operators 

(personnel) so that each can perform more efficiently. While these classical 

domains have some applicability for learning systems design, LXD practi-
100 | Modernizing Learning

tioners will likely need to modify this model. What’s more important than 

its specifics, however, is the broad, system-wide perspective it encourages. 

When designing a learning experience, it’s useful to not only consider its 

delivery but also, for instance, how many learning professionals are needed 

to implement it (manpower), what skills those professionals need (personnel), 

how they’ll be preparation for their roles (training), and the context in which 

they’ll deliver the intervention (habitability). 

Finally, HSI practitioners often facilitate a multidisciplinary design process, 

helping to document and “translate” between specialists in different disciplines (e.g., between sociologists and computer scientists) and negotiate requirements among interested parties (e.g., brokering compromises between 

training specialists and manpower analysts). In practice, this means that HSI 

practitioners spend considerable time eliciting inputs from various stakeholders, documenting assumptions, clarifying friction points, and developing 

“shared representations” that transform these requirements and analyses into 

meaningful, unambiguous formats such as storyboards, concept maps, process diagrams, storyboards, and wireframes. These HSI processes and tools 

are useful for LXD designers, as well. 

6. Maximize global outcomes vs. local processes 

Explicit in the “learning ecosystem” concept are the notions of diversity and 

interconnectivity—across an entire lifetime (or, at least, career). This connectivity creates new opportunities for us to consider learning experiences 

in concert rather than as isolated events. Other chapters in this book discuss 

instructional strategies for connecting learning events (Chapters 4 and 12, in 

particular). This chapter, however, adds practical considerations that LXD is 

uniquely positioned to address.

First, consider the impact of lower-level decisions in aggregate. What’s the 

gestalt, or combined impression, they collectively produce? Do certain implicit 
Learning Experience Design | 101 

messages, such as emotional or motivational suggestions, carryover from one 

event to the next? For instance, imagine a multipart workshop taught by four 

different instructors. If each asks trainees to complete some kind of pretest, 

engage in initial icebreaker activities, and respond to post-training questions, 

the trainees are likely to grow bored, lose motivation, and might even become 

cognitively overloaded. A clever designer might find ways to tie the different 

segments together, introduce novelty throughout the four segments, build-in 

time for cognitive reset, and find ways to simplify the overall UX. Similar 

considerations apply as we scale-up our reference frames and begin integrating more diverse learning experiences across time, subject, and media. 

Second, when designing learning interventions, it’s tempting to try to optimize each individual event, without considering their collective, long-term 

result. For example, consider a company that’s decided to shift from traditional, weeks-long vocational courses to on-the-job, just-in-time training. On the 

one hand, this method helps avoid inefficient massed learning where individuals often wastefully forget much of what they learned. On the other, it risks 

creating disjoint learning that individuals struggle to meaningfully integrate 

and comprehend beyond a superficial level. It may also create unforeseen burdens on more experienced operators in the job environment. There is nothing 

The challenge is getting teachers to share the imperfect; they 

like to reach perfection, and convincing them to share an 

imperfect product is difficult. The historical Vermont paradigm 

has been teachers are artisans working in relative isolation, but 

that system is breaking down. It’s antiquated and not working 

relative to personalizing student learning. We need to scale and 

showcase the very good work that’s going on in some of our 

districts—but not all. 

Daniel French

Secretary of Education, Vermont Agency of Education
102 | Modernizing Learning

inherently wrong with just-in-time learning; rather, the point is to consider 

system-wide learning strategies that balance holistic efficiency and longitudinal performance against local optimizations. If each module, course, or individual designer develops local optimums in isolation, we risk creating overall 

inefficiencies and ineffectiveness. Strategy—informed by learning science—

must be applied to, and integrated across, all levels.

7. Continue to synthesize theories and practices 

from diverse disciplines 

LXD, like the future learning ecosystem concept writ large, represents a synthesis of varied and emerging disciplines. Learning design teams in the future will likely involve instructional designers, learning scientists, learning 

engineers, technologists, data scientists, and other professionals. LXD fills a 

unique void, helping to integrate the diverse perspectives across these team 

members, giving voice to learners’ (and other stakeholders’) needs, and encouraging the use of disciplined human-centered design practices. 

It’s impossible for any one person to thoroughly know all of the disciplines 

that inform LXD, but it’s important for LXD designers to avoid “reinventing 

the wheel” with their work. As this chapter has shown, many existing domains 

offer useful theories, processes, use-cases, and tools. Seek out these prior 

solutions; curate and remix them for your own purposes. Look in creative 

places, such as the advertising literature or systems engineering manuals, and 

look to conventional principles of instructional design, learning science, and 

cognitive psychology, too. This discussion on LXD isn’t meant to supplant 

those important fields but rather to supplement them by integrating design 

principles that consider human-system interactions, applied cognition, organizational dynamics, and user experiences. Together, in synthesis, these various methods can help learning designers to not only create quality instruction 

but to better achieve learning outcomes for real people, in real-world contexts. 
Everyone comes through the same 

education system, and we get locked 

into believing that’s the way we 

learn—when we really don’t.

Doug Tharp

Senior Learning Project Manger

Nuclear Regulatory Commission

Technology
Interoperability allows data to easily flow, even among applications 

developed for different purposes, using a standardized vocabulary, 

structure, and cadence. Interoperability implies common standards 

that promote system-to-system communications, potentially across 

organizational boundaries and institutional firewalls, using specified 

data formats and communication protocols. These standards form 

the fundamental building blocks for technology-enabled lifelong 

learning by establishing consistent protocols that can be universally 

understood and adopted by related systems to enable data exchange 

about learners, activities, and experiences.
Interoperability | 107 

CHAPTER 6

INTEROPERABILITY

Brent Smith and Prasad Ram, Ph.D.

Interoperability, when applied to learning, transcends the full spectrum of 

learning environments, systems, data, and organizational entities that individuals encounter throughout their lives. The highly mobile nature of our population requires that information about learning be shared in an efficient manner 

across this ecosystem of learning. When individuals advance in their careers, 

or transfer from one career to another, or simply progress through the continuum of learning from one organization to another, high quality data about their 

learning experiences needs to be shared. However, today’s learning ecology 

consists of stovepiped organizations using highly customized management 

systems, accessing disparate sources of data in any number of nonuniform 

architectures. To achieve the future learning ecosystem concept, we’ll need 

to exchange data across a full range of products, made by different vendors, 

and encountered throughout the entire continuum of lifelong learning. Key 

to managing all of this data is interoperability—afforded through the use of 

internationally accepted technical specifications and standards. 

In today’s digital world, information is readily accessible anywhere and everywhere. Large-scale social networks, interactive content, and ubiquitous 

mobile access are emerging as driving technologies in education and training. 

At the same time, data science presents new opportunities for assessing the 

effectiveness of learning content for different learners, understanding organizational trends across large volumes of learning data, and using amassed data 

to continually improve training and education. Yet, there are interoperability 

challenges. Today’s digital learning ecosystem is fragmented. Data from one 

 

 
108 | Modernizing Learning

system can’t always integrate with data from another, which means learning 

records aren’t easily transferable between institutional systems and across organizations. Training and education institutions don’t even record the same 

learner activities or capture learner achievement information in the same formats, which further complicates our ability to aggregate data.

FORMAL LEARNING 

PROGRESSION

Beginning with K–12 education, most state educational systems use products 

from multiple vendors, and each district deploys their systems independently. Historically, these applications have used limited (or no) underlying data 

standards. Instead, most employ their own internal data models, and integration across systems requires a patchwork of connections at the state and/

or local levels.1 Consequently, there are gaps in the integration among disparate applications, and many systems are simply not interoperable. Ideally, 

data from multiple products, such as learning management systems, student 

information systems, and learning object repositories, would be aligned to the 

same common data standards, enabling seamless coordination across these 

applications.2

The existing higher education system is also its own stovepipe. With its focus on credit hours, semester-long courses, and formal credentialing, these 

institutions often fail to account for new practices available in a digital, and 

globally connected, world—such as emerging global online learning environments that increasingly blur formal and informal practices. Students are 

now much more interested in interactive and self-guided approaches, and with 

so much information online (and often available for free), universities are no 

longer the only places to find higher-level learning. Consequently, the value 
Interoperability | 109 

of a degree is gradually decreasing as employers place greater weight on a 

candidate’s capabilities developed outside formal education.

Within military education and training there are many different schools and 

training programs designed to foster technical, professional, and leadership 

skills in service members. Many of these programs, their instructional technologies and personnel information systems, exist in stovepipes. Further, historically there’s been a separation between the education and training communities across the U.S. Defense Department. Education traditionally occurs 

incrementally and involves grappling with ambiguity while thinking and reflecting about the concepts being learned.3

 Training is linked to readiness and 

offers opportunities to apply knowledge, skills, and abilities in a manner that 

provides immediate feedback and progress measurement.4 Within the current 

context, training and education have different reporting structures, motivations, and logistical requirements such as fuel, personnel, and the access to the 

appropriate environments or equipment. Combined, this leads to data being 

acquired from many different sources but with little-to-none of it standardized or connected.

The biggest problem we have is the 

lack of connected infrastructure across 

postsecondary learning systems.

Amber Garrison Duncan, Ph.D.

Strategy Director, Lumina Foundation 
110 | Modernizing Learning

Types of Interoperability

Rapid technological change has become the norm in the modern landscape of 

training and education. Within learning contexts, the pressure of such changes is felt acutely by educators, trainers, administrators, and learners alike. 

Table 6-1 shows a different view of the various learning technologies, environments, organizations, and outcomes a given learner might encounter 

throughout his or her career. This matrix highlights the numerous types of 

interoperability required to facilitate a future learning economy. This is largely due to the organizational design of the current learning landscape as well 

as the different reporting structures and responsibilities for when and where 

training and education occur. 

Some Representative Examples of Where and How Learning Occurs

Learning 

Technologies

Learning

Environments

Learning 

Organizations

Learning

Outputs

Electronic 

Classrooms

Interactive e-Books

Learning Mgmt. 

Systems

MOOCs 

Mobile Devices

AR and VR Systems 

Live, Virtual, and 

Constructive 

Simulations

Embedded Training 

and Performance 

Support

IoT systems

Wearables 

Performance 

Qualification Systems

Instructor-Led 

Classrooms

Live Training Ranges

At Home and Cafés

In Transit on the

Train or Bus

On-the-Job 

Experiences and 

Mentorship

Field Trips and 

Military Staff Rides

Workshops and 

Conference

Libraries

Navy Ships Afloat

Austere Job Sites and 

Military Stations

Simulation Centers

K–12 School Districts

Trade Schools 

Colleges and 

Universities

Postsecondary 

Accreditation 

Agencies

Licensing and 

Credentialing Bodies

Corporate Human 

Resources Programs

Military Manpower, 

Personnel, Training, 

and Education 

Systems

Industry Associations

International 

Organizations and 

NGOs

Transcripts

Diplomas and 

Degrees

Standardized Test 

Scores (SAT, ACT, 

ASVAB)

Licenses and 

Certifications

Digital Badges and 

Micro-Credentials

Formal Performance 

Evaluations

Resume Listings

Continuing 

Education Units 

and Professional 

Development Units

School and 

Workplace Credits

Table 6-1: Learning Activity Matrix 
Interoperability | 111 

Many types of interoperability are required:

► Systems Interoperability – Digital systems need to work together. The 

existing systems we use to collect, manage, analyze, and report on data 

are often disconnected and don’t always work well together. Some of the 

technology challenges center around data standards, including inconsistency of standards and the inability to access data in a usable format. 

Progress is being made by numerous ongoing efforts across government, industry, and academia.

► Application Interoperability – Systems are comprised of numerous 

disconnected applications that, theoretically, must all be capable of communicating about how they’re impacting learning for each individual. 

Currently, different applications track performance differently, and the 

ability to infer information about each activity within an application is 

not always well-defined. 

► Data Interoperability – The seamless, secure, and controlled exchange 

of data between applications is critical to maximizing our ability to understand individuals’ learning. Not only are data are often stored in isolated data within applications, but these datasets often use custom or 

proprietary data models. Common data standards, along with supporting data governance and metadata information, are needed to maximize 

return on investment in interoperable applications, perform workforce 

planning, and support other derived benefits from data analytics.

► Human-Machine Interoperability – The different environments where 

learning takes place impact the types of learning technologies used. As 

new tools and technologies come into play, individuals must become 

more technically savvy and industry must find ways to better support 

the seamless transition of learning across a multitude of computing platforms, devices, and learning modalities. 
112 | Modernizing Learning

► Organizational Interoperability – Data ownership is a critical obstacle that impedes true interoperability. In the knowledge economy, data 

is often monetized and leveraged for purposes other than learning. Organizations as well as savvy individuals are reluctant to share their data. 

Creating cross-platform and interorganizational interoperability will 

require a change in culture, and, arguably, that poses an even more difficult challenge than technical interoperability.

Already, educational institutions, training organizations, and instructional 

technologies collect some learner data, such as, demographics, assessment 

results, teacher observations, learner-created content, attendance, and course 

grades. However, these data points don’t provide a complete picture of a learner unless connected with data collected throughout the continuum of learning. 

Additionally, we’re touched by learning broadly throughout our daily lives 

by numerous informal interactions, both with other people and through our 

own self-directed efforts, but none of those data are captured in a manner that 

allows aggregation, comparison, or analyses. 

Resolving these interoperability challenges is key to setting the foundations of 

a global learning economy that enables learners to constantly update, retool, 

rethink, and relearn.

VISION

Common standards and shared technical specifications create the underpinnings needed for the future learning ecosystem, from a technology interoperability perspective. These standards consist of published documents that establish key interface specifications, communication protocols, and data structures 

designed to facilitate interoperability among connected components. In this 

context, interoperability specifications form the fundamental building blocks 
Interoperability | 113 

for lifelong learning by establishing consistent protocols that 

can be universally understood 

and adopted by any component 

of the learning ecosystem to enable data exchange about learners, activities, and experiences.

In the future, such interoperability will unlock rich data 

about learners and learning 

activities, empowering organizations to build comprehensive 

solutions that meet the needs 

of their specific populations. 

Standardized, documented interfaces will also enable “plug-and-play” replacement of new 

or upgraded capabilities on existing platforms. In other words, 

interoperability will allow organizations to add, modify, replace, 

remove, and support different learning technologies (from different 

vendors) throughout their lifecycles. 

Interoperability will facilitate data aggregation across the continuum of learning. Analyses of these data, in turn, will enable learners to optimize their 

learning journeys across their many diverse learning activities, throughout 

their careers, and, ultimately, across their lives. These data could also help 

address institutional questions, such as determining which academic courses produce the best learning outcomes or predicting workforce skill gaps. 

Combined with the science of human capital management, enterprise learning 

analytics could also help organizations address their strategic talent management goals, including succession planning, career assessments and growth, 

development, retention, and knowledge sharing. 

The more people 

understand Google and 

the benefits of crossdomain work, the more 

they want it—and the 

more the silo boxes are a 

problem. 

Jeanne Kitchens

Chair of the Technical Advisory Committee 

for Credential Engine; Associate 

Director of the Center for Workforce 

Development, Southern Illinois University
114 | Modernizing Learning

Several types of technical interoperability are needed to achieve this vision. 

These include standard ways of defining competencies (for use in both learning and performance contexts), for encoding data about individuals’ performance and behaviors, for aggregating and visualizing these performance data 

in meaningful ways, and for describing and locating various learning activities. The following subsections describe each of these in more detail.

Competencies

Interoperable frameworks that form the 

“common currency” of the future learning ecosystem

Competencies form the interoperability crossroads of the future learning ecosystem, serving as the Rosetta Stone between different learning systems and 

workforce applications. A competency describes a set of skills, knowledge, 

abilities, attributes, experiences, personality traits, and motivators needed to 

perform a particular task. Competencies might include technical, business, 

leadership, social, ethical, or emotional capacities, or any number of other 

personal traits and capabilities. Additionally, competencies may be highly 

dependent on their usage 

context; differences in environmental factors, task 

complexity, and related processes or policies can all impact their applications. 

A competency model (also 

called a competency framework) combines multiple competencies, and their underlying factors, into a 

framework related to particular domain, career, or job area. Some competency 

models further separate this information into levels of mastery, such as information about the level of competence required at different occupational levels, 

Organizational competencies need to 

be encapsulated within a competency 

framework to map all learning activities 

a learner might encounter 

within an organization.
Interoperability | 115 

and these various elements within a competency framework can have many 

nonexclusive relationships with one another.

Education and training organizations may use these frameworks to inform 

learning outcomes, and they’re also widely used in business for defining and 

assessing requirements for both hard and soft skills associated with job performance. The use of common competency frameworks will allow data from different sources to be meaningfully interpreted in and translated to other contexts.

One challenge is that there’s no standard for competencies. Different industries, 

accreditation authorities, and trade associations use a variety of different existing frameworks. Some follow any number of specifications and others do not. 

Many competency frameworks include rubrics, performance levels, or other 

data that can be used to evaluate proficiency while others rely on supplementary, 

external components to house assessment and evaluation criteria. Some competencies are linked to the environment in which the competency is expressed, 

and others are motivated by training or education objectives (e.g., knowledge, 

skills, abilities). To enable the future learning ecosystem vision, shared vocabularies, classifications, and frameworks of competencies will be needed, and 

these will need to allow for commonality and reuse of competency objects and 

their descriptors across diverse organizations. Shared metadata vocabularies 

might also be required, to include descriptors such as the type of skills included (e.g., psychomotor or cognitive skills), skill decay estimates, or relevant 

environmental factors that impact or inform the description of a competency.

Activity Tracking

Data about learners’ performance and behaviors

Activity streams are a nearly ubiquitous feature on many of the applications 

we use on a daily basis. For example, newsfeeds on social media use activity streams to record users’ interactions. Activity streams use serialized data 

that consist of statements about behaviors. Such statements typically involve 
a subject (the person doing the activity), a verb (what the person is doing), and 

a direct object (what the activity is being done to or with); optionally, other 

elements that describe the performance context can also be incorporated. The 

resulting dataset tells the story of a person performing an activity. Examples 

include “Mike posted a photo to his album” or “Emily shared a video.” In 

most cases, these components will be explicit, but they may also be implied. 

Within the future learning ecosystem, activity streams need to capture what 

individuals do, which learning activities they perform, and how they perform. 

Each entry in the stream should be timestamped, meaning that a learner can 

have progress measured as a function of time, not simply a function of state. 

The goal of activity streams is to provide data (and metadata) about activities 

in rich, human-friendly formats that are also machine-processable and extensible. This interaction data will need to be published by any activity a learner 

engages with. In some instances, data might be generated by a learner’s performance, and, in other cases, a system might generate data based on system 

events or key milestones achieved by a learner. Alternatively, data may be 

The American Council for Education has developed a digital competency-based 

credential that will enable an individual to transfer learning from work to a degree 

path. The T3 Innovation Network* is testing the use of competency translation 

algorithms to review curricula and competencies. The algorithms are reviewed 

by faculty to confirm and are at an 80% accuracy rate right now—and that will 

continue to improve. The ability to use advanced technology will help us start to 

harmonize towards a more common competency language since we, as humans, 

cannot connect the 1000+ frameworks that exist without technology’s help. 

Amber Garrison Duncan, Ph.D., Strategy Director, Lumina Foundation

The T3 Innovation Network is an initiative of the U.S. Chamber of Commerce 

Foundation for exploring emerging technologies and standards in the talent 

marketplace to better align student, workforce, and credentialing data.

116 | Modernizing Learning
Interoperability | 117 

generated to establish the context of the learner, the application, or other components within the learning ecosystem. 

The subject of an activity is nearly always the learner but could, foreseeably, 

be an instructor, cohort, or other human or machine agent. The direct object of 

an activity depends on its context, as do the verbs (although to a lesser extent). 

Universal terms, particularly verbs, will need to use a common vocabulary 

across systems, otherwise the data will lack semantic interoperability and lose 

much of its utility. By formalizing a common vocabulary, activities can reference an established set of attributes along with rules for how the dataset is 

stored and retrieved by components in the learning ecosystem.

Universal Learner Profiles

A common place to aggregate and visualize learners’ data

The current way learner records are managed is insufficient for the evolving 

needs of instructors, learners, and organizations. Today, a transcript is typically 

used to record learners’ permanent academic records. Transcripts usually list 

the courses taken, grades received, honors achieved, and degrees conferred 

from a formal academic institution. Only this most basic of information 

follows individuals across their different learning episodes. Teachers and 

trainers have little visibility into individuals’ past performance, such as what 

other instructors have noted about them, the informal or nonformal learning 

they’ve experienced, or their strengths, weaknesses, and individual needs. 

In the future, transcripts—or “learner profiles”—will still need to expand to 

incorporate a broader range of credentials, micro-credentials, and other learning activity information along today’s formal learning information. They will 

also need to become more dynamic, shifting away from being static records 

and instead acting as dynamic tools that learners and organizations can use to 

determine learners’ unique paths to achieving proficiency in all their desired 

competencies. 
Society expects us to be innovative.

It’s imperative that we evolve because changes are happening 

whether we lead them or not. The demands society places on 

innovation mean that we’ve got to stop looking through the 

lens of today and start looking through the lens of tomorrow 

with a vision for K-12. Our kids today will be the workers, 

leaders, academics, or soldiers of tomorrow.

So, the questions to ask are: How can we use technology to 

help us pedagogically? Can we conduct formative rather than 

just summative assessments of individual aspects of learning 

that ultimately enable us to give learners a better education 

than they’ve ever had? 

Our academic standards are now in a machine-readable format 

and we can do true gap analyses to make inferences to inform 

teachers’ decisions and also save untold billions of dollars. 

Information rich micro-credentials, such as badges, support 

measurable progression, process, and evidence of learning. 

Using xAPI that records these steps builds a documentation 

of learning that lives beyond the institutional level. It supports 

lifelong talent-management and allows our systems to be 

seamlessly aligned across time and communities. We need 

to ensure that the same measurement we use is both useful 

today and understandable to the next community. 

Learning is truly measurable.

Keith Osburn, Ed.D.

Associate Superintendent

Georgia Virtual Learning 

Georgia Department of Education
Interoperability | 119 

Learner profiles have the potential 

to empower personalized learning 

within the future learning ecosystem through better data that can inform learning in new and meaningful ways. As envisioned, a learner 

profile is analogous to a mashup of 

information about a learner, populated from various sources and consisting of both explicit and derived 

data. A future learner profile might 

include a broad range of information, such as demographic data, data about a 

person’s interests and preferences, and existing competencies and those that 

need to be developed (in the personal, academic, and career arenas). It also 

might include information someone’s learning strengths, needs, and the types 

of learning interventions that have been successful in the past. We use the 

term “universal” when describing the learner profile, because we envision 

data from multiple systems flowing into a shared representation. Further, as 

a learner’s interests change, or as he or she becomes competent in new areas, 

the profile would continually update to reflect the latest “state” across time. 

Safeguarding learner data to preserve privacy is an important legal and ethical consideration. We could also imagine that individuals would need to control their own data; so, we anticipated that individuals would have access to 

obtain, share, and interact with these artifacts as well as to control the other 

people, organizations, or applications that can access them. 

Activity Registry

Arrays of diverse learning activities

An activity registry is an approach to capturing, connecting, and sharing data 

about available learning resources. Unlike federated repositories, search en-

...it’s more about 

the person, not the 

technology. 

Emily Musil Church, Ph.D.

Executive Director of Global 

Learning, Prize Development and 

Execution, XPRIZE
gines, or portals, activity registries 

are a Resource Distribution Network with open APIs that anyone 

can use to register, expose, or consume learning resources and information about how those resources 

are used. Key features include the 

ability to generate and manage content metadata (data about the publisher, location, content area, standards alignment, ratings, reviews, 

and more), manage taxonomies and 

ontologies, manage the alignment of 

content with competencies, generate 

and manage paradata (data about the 

metadata, such as resource usage, 

comments, rankings, and ratings), 

perform semantic search services, 

and create machine-actionable metadata for AI-based recommenders. 

An activity registry houses metadata, paradata, assertions, analytical 

data, identities, and reputations that 

flow through the distribution network. An activity registry will also 

contain access information and permissions for different learners. The activity registry requires a trusted relationship with different learning-related 

activities as well as other essential services, such as launch and discovery. We 

imagine that any of the communities or organizations that consume a learning 

resources will also capture information about how those resources are used, 

such as their context, user feedback, user ranking, rating, and annotations, and 

We have 11 missions in the 

Coast Guard and every one of 

them is diverse, with different 

stakeholders for each. Our 

borders, to include our waterways, 

are very important, and if we were 

to go to war, we’d be supporting 

the Navy. So it’s important that 

we’re connecting and maintaining 

readiness; yet, we’re bounded 

by many different DHS and DoD 

policies. It’s critical that we can be 

disruptive thinkers about training, 

and it’s even more critical that we 

can interoperate.

Gladys Brignoni, Ph.D.

Deputy Commander, Force Readiness 

Command and the U.S. Coast 

Guard’s Chief Learning Office
Interoperability | 121 

these paradata might be incorporated into the activity registries. We imagine 

that such usage data and third-party analytical data could become valuable for 

resource discovery and for understanding what learning resources are most 

effective. 

Learning Content Metadata

Data that describe learning resources

To effectively enable activity registries, the resources they point to will need 

to be described in some manner. Such descriptions are encoded as metadata. 

In training and education, many different metadata formats have already been 

explored, including Learning Object Metadata (LOM; IEEE 1484.1.1), which 

is commonly used with SCORM managed content, the Dublin Core Metadata 

Initiative, and the Learning Resource Metadata Initiative (LRMI).5

LMRI is a particularly common metadata framework, used for describing learning resources in web-based instruction. LRMI was adopted by 

Schema.org6 in April 2013, which allows anyone who publishes or curates 

educational content to use LRMI markup to provide rich, education-specific 

metadata about their resources with the confidence that this metadata will be 

recognized by major search engines. Founded by Google, Microsoft, Yahoo, 

and Yandex, Schema.org’s vocabularies are developed by an open community 

process with a mission to create, maintain, and promote schemas for structured data on the internet, including on web pages, in email messages, and 

beyond. LRMI’s adoption into Schema.org provides many benefits. In theory, 

nearly any Schema.org “thing” could be defined as a learning resource. Therefore, LRMI addresses those metadata properties that distinguish content when 

it’s deliberately used for learning. This was done by adding learning-resource 

properties to key root types. For example, LMRI incorporates the CreativeWork property, which includes descriptors such as Educational Use, Educational Alignment, and Course,

7

 the latter of which is defined as a sequence of 
122 | Modernizing Learning

one or more educational events and/or other types of CreativeWork that aims 

to build knowledge, competence or ability of learners.

Talent Management

Bridging education, training, and workforce silos

The preceding subsections highlighted the interoperability afforded by technical standards. We’ve primarily discussed these standards within the context 

of training and education, but they also apply to workforce activities. The 

worlds of human resources, training, and education have never been more 

closely linked. Organizations, employees, departments, data, customers, and 

partners can no longer function successfully in their own silos. As mentioned 

above, today’s training and education systems are often disconnected from 

one another; further, they’re rarely interoperable with internal or external HR 

systems. This results in incomplete or duplicative data, inefficient or inaccurate reporting, complex and costly vendor management, and inefficient and 

manual HR transactional processing.8 Standards and specifications that allow 

these disparate systems to communicate have the potential to assist organizations of all sizes to improve performance and workforce satisfaction.

The systems around talent management need to work seamlessly. Within the 

future learning ecosystem, an employees’ digital records will include data 

from various stages of their careers related to recruitment, training and development, and performance management. Many new standards are actively 

being developed through the International Standards Organization (ISO) relevant to business-crucial areas such as compliance and ethics, workforce costs, 

diversity, leadership, occupational health and safety, organizational culture, 

productivity, recruitment, mobility and turnover, skills and capabilities, succession planning, and workforce availability. All these areas contain specific 

metrics and reporting recommendations. Creating systems that combine these 

workforce data with other training and education information will enable the 
Interoperability | 123 

advancement of evidence-based human capital management policies and provide access to lifecycle data for transaction processing. It will also provide the 

data needed for workforce planning and strategic decision making.

Talent, too often, is treated as an afterthought. With increasing retirements 

and a fluid workforce, organizations are finding it more difficult to manage the 

end-to-end employee data lifecycle due to duplicative HR IT systems across 

agencies that are unable to interface and exchange data. The different systems 

in use today are like different countries—with distinct languages, customs, 

and religions. They use diverse data formats and moving data between them 

is difficult and, when done, is often accomplished in nonstandard ways. To 

improve the interoperability of HR systems, the different applications need 

a common record that covers all aspects of the employee lifecycle, from hire 

to retire, for each person. Additionally, to achieve greater synergy within an 

organization and to drive human capital performance across the breadth and 

width of organizational competencies, organizations must shift from ad-hoc 

to strategic talent management programs. 

These enhancements to workforce HR systems will benefit learning institutions, as well. Experts commonly agree that most learning takes place on the 

job.9 Hands-on experience allows individuals to refine their job skills, make 

decisions, address challenges, and interact with others in the organization. 

They also learn from their mistakes and receive feedback on their performance, and they may engage in coaching, mentoring, collaborative learning, 

and other forms of social learning. Rarely (if ever) are these informal learning 

experiences tracked. By understanding how and when these types of learning 

take place, we can construct more robust profiles of individuals, whether to 

inform their learning journeys or to increases the collective intelligence of 

their organizations.

The capacity of an organization to innovate, change, and become more effective depends on employees’ capabilities, thus highlighting the importance 

of developing those individuals.10 However, just as we need better indicators 
124 | Modernizing Learning

for undergraduate performance, we need better measures of performance in 

the workplace. The competitive nature of the global economy and the world 

stage increase the need to focus on the human capital supply chain that organizations employ. While this concept is attractive to organizations, there 

are ongoing challenges for its implementation. Workforce planning requires 

authoritative data for proper modeling and predictive analytics. Recruitment 

requires integration with onboarding and performance data to improve hiring 

strategies. By enabling a common language that the different systems can 

read from and write to, we’re able to identify hidden dependencies and relationships within an organization and provide other analytics that help them 

make better and faster data-driven decisions. 

In America, companies are struggling to close a skills gap, which is 

negatively impacting their ability to compete and grow in a global 

economy. The U.S. Chamber of Commerce Foundation’s Talent 

Pipeline Management initiative is exploring how employers can close the 

skills gap by improving how they communicate or “signal” their hiring 

requirements. Through this work, they’re creating the Job Data Exchange 

that enables a set of open data resources, algorithms, and reference 

applications for employers and HR technology partners to use to improve 

how competency-based hiring requirements are defined, validated, 

and communicated. This provides a critical linkage between the job 

performance data, credentialing systems, and learning record systems.11

IMPLEMENTATION STRATEGIES

The future learning ecosystem promotes an increasingly complex world of 

interconnected information systems and devices. The promise of these new 

applications stems from their ability to create, collect, transmit, process, and 

archive information on a massive scale. However, the vast increase in the 

quantity of personal information being collected and retained, combined with 
Interoperability | 125 

our increased ability to analyze it and combine it with other information, create valid concerns about managing these volumes of data responsibly. There 

is an urgent need to strengthen the underlying systems, component products, 

and services that make learning data meaningful. The following subsections 

outline a foundation for an enterprise-wide learning ecosystem that can adapt 

and grow with the needs of the organization.

1. Identify and describe organizational competencies

Organizations need to inventory the skills required to successfully perform 

all business functions within their institutions. These include technical, professional, and leadership capabilities across numerous departments, divisions, 

or lines of business. Each role within the organization will typically include 

a career trajectory with an accompanying learning trajectory for the knowledge, skills, attitudes, and other contributing factors an employee needs to do 

their job effectively. Different roles may share the same competency but have 

different contexts for how that competency is performed or a weighting for 

how much it impacts a particular job. A competency framework provides the 

common reference model across HR, training, and education systems, and 

the critical indicators associated with competencies within it help quantity individuals’ performance. As new tools, technologies, and processes transition 

into the work environment, competency models will need to be continually 

updated effectively operate in the future. 

2. Formulate a data strategy

The current landscape of disparate learning and personnel systems will continue to evolve for the foreseeable future. A cohesive data strategy needs to be 

implemented to help identify all of the relevant data types required to support 

the human capital supply chain, to define the relevance of different data types 

over time, to identify an approach for capturing the decay of importance be-
126 | Modernizing Learning

tween different data types, and to identify authoritative sources for generating 

each data type. An effective data labeling strategy will enable automation, 

increased analytics, and an associated lifecycle for how long the different 

data elements remain relevant. Data labeling attaches meaning to the different types of data, correlated to the different systems that generate the data 

across the lifelong learning continuum. This allows all systems in the learning 

ecosystem to use the data as needed, such as to adaptively tailor learning to 

individuals. Patterns of data should also be explored to derive additional insights at institutional levels. Consider both structured and unstructured data 

that may be generated in different areas, and develop clustering strategies for 

how to organize the different data types so that all components have access to 

the data they need.

3. Define standards, specifications, and vocabularies

The more we can formalize the requirements for standards, specifications, 

and shared vocabularies used across the nation, the easier it will be to integrate components into the ecosystem. While there are many automated technologies that can semantically align different terms, there are benefits to designing systems that use shared vocabularies to describe learning activities, 

digital content, learners, and competencies. Activity tracking across learning 

activities also works best when each activity uses a common library of terms 

for different instructional modalities, media types, or as a roll-up to other 

systems in the organization (e.g., talent management). 

4. Define the governance strategy

Organizations need to be responsive and proactive in recruiting, educating, 

and preparing their existing workforce for the future. The knowledge and 

skills required to be successful today will change and new tools, technologies, 

and methodologies will migrate their way into the organization. Emphasis 
Interoperability | 127 

should be placed on the protection of personally identifiable information, protected intellectual property, and other proprietary organizational data. As new 

systems come online, all aspects of the data strategy, competency framework, 

and human capital supply chain will need to be revised. Workforce planning 

strategies should tie into the lifecycle management of these critical components. Governance should also be addressed in the data strategy so that specific indicators and outcomes can be tracked, measured, and analyzed. 

These four steps provide a strategic framework from which a learning ecosystem can be built. These aren’t trivial tasks and will be implemented differently in each organization, depending on its size, complexity, and goals. 

Collectively, these steps allow organizations to embrace the future learning 

ecosystem concept and to benefit from the rich data it will produce, allowing 

businesses to maximize their workforces and learning-delivery organizations 

to optimize and manage the quality of training and education experiences 

they offer. 

As we think about creating an interoperable system across DoD, my initial 

thoughts are just how big the problem is. With four different Services, 

how is it that you generate buy-in from each—who for a long time have 

been doing their own thing? I think the first question is: What are the 

digestible parts that would have commonality across the Services? If you can 

determine that, then how do you get buy-in to the digestible parts? Because 

in the end we all want to know how we’re going to do it better, faster, 

cheaper; this is a problem of all the Services.

Thomas Baptiste

Lieutenant General, U.S. Air Force (Ret.)

President and CEO, the National Center for Simulation 
In thinking about risks associated with the learner data needed 

to power personalized adaptive learning, privacy and security 

are clearly at the top of my mind. But we need to expand the set of 

values beyond those two in determining if the use of student data is 

responsible/ethical. There’s value in advancing knowledge, ensuring 

students are successful, and promoting the development of practices 

that have the potential to affect a lot of people. This multi-faceted 

approach isn’t new: A lot of these values are considered in the context 

of human-subjects research reviews. It’s important for the academic 

community to have a similar process for considering such a range of 

values in evaluating our practice, in addition to our research. 

Martin Kurzweil, J.D.

Director, Educational Transformation Program, Ithaka S+R
Data Security | 129 

CHAPTER 7

DATA SECURITY

J.M. Pelletier, Ph.D. 

Data breaches, like traffic accidents, are inevitable. Yet, it’s also a requirement 

that we progress as a nation to a digitized learner ecosystem. Accordingly, this 

chapter describes the ways we can be proactive in simultaneously managing 

the likelihood of occurrence, damages of impact, and potential for contagion 

of breaches across learner data systems. An effective learning architecture requires security to preserve privacy, to prevent cheating by the individual, and 

to prevent intrusion by external threat actors. Accordingly, balanced effort 

is required across the three pillars of security: confidentiality, integrity, and 

accessibility. While most security investigations focus on confidentiality and 

integrity, the access to that data enables timely and well-informed decisions. 

Further, users are highly likely to invalidate security controls if accessibility 

is inadequate. All of these concerns can be addressed by hardening devices 

and networks in a way that places users at the center of each improvement. 

To do this efficiently, data security design should enable individuals and organizations to limit the spread of breaches within current and future learning 

architectures. Thus, this chapter describes principles and strategies that will 

allow distributed learning environments to keep pace with developments in 

cybersecurity.

Data Security Threats and Challenges

Several issues must be addressed as we progress to a nation that embraces 

the accumulation of data. We must recognize both those elements that protect the human as well as the need to protect the system and integrity of the 

 

 
130 | Modernizing Learning

data. In America, an individual’s privacy is a fundamental right, and security 

preserves the dignity that 

privacy allows. Yet a subset of learners will struggle with integrity; learners 

cheat by bypassing access 

controls to steal answers or 

alter grades. Also, foreign 

adversaries constantly use 

cyber means in their attempt to assess national capabilities and influence organizational priorities. Finally, regardless of resource investment, we’re seeing consistent increases in both the impact and probability of breaches.

AMATEUR THREATS

The most immediate concern is that malicious software is becoming increasingly automated. Learners need little technical ability to steal answers and 

change grades. There are many thousands of free, step-by-step tutorials that 

walk would-be attackers through the process of conducting the most-known 

technical penetration.1

FOREIGN THREATS

A broader challenge exists in relation to foreign adversaries. As advanced 

persistent threats become increasingly capable, the former cornerstones of 

security are quickly becoming obsolete. This is especially true as our nation races to keep pace with adversarial advances in quantum and classical 

supercomputing capabilities.2 Any widespread use of data management systems should be resilient to known attack methods and provably secure against 

cryptographic brute force, side channel, and intercept attacks.

You can’t hold firewalls and intrusion 

detection systems accountable. You 

can only hold people accountable.

Daryl White

Chief Information Officer, Department of Interior

...as quoted in the Information Security 

Management Handbook, Sixth Edition, V7
Data Security | 131 

SOCIAL ENGINEERING

The value of access to an organization’s learning architecture shouldn’t be 

underestimated. Expert-level social engineering results in manipulation of the 

behavior of entire societies. There’s a broad and deep body of knowledge in 

the use of deception and propaganda to control populations. This happens on 

an individual level through confidence artistry and is scalable to any number 

of persons using similar techniques. Centuries of Russian military thought 

and government experimentation in social engineering provide us with the 

domain of маскировка (mask-ee-rove-ka), which roughly translates to “masquerade,” “disguise,” or “deception,” and includes the concept of reflective 

control. Reflective control is the art of strategic injections of (usually truthful) 

information to cause a person or society to choose freely actions that are most 

beneficial to the other party. The select injection of truth can manipulate perception. Further, a single, well-designed falsehood within a trusted environment has disproportionate network effects. Just as a person can be manipulated to act on behalf of another person’s interest, so too can organizations. At a 

societal level, this shapes political will and, ultimately, public policy. 

INVESTMENT MODELS

Recent research in information security economics has attempted to build 

models that help evaluate optimal levels of information security investment. 

These generally apply risk management strategies to calculate an optimization function associated with expected monetary loss, assessed vulnerability, 

and likelihood of a breach. Some of these models consider breach contagion 

effects, but they aren’t prescriptive in suggesting how the economically optimal funding amounts should be invested.3

There seems to be consensus among economists and cybersecurity experts 

that the only solution is to spend more money on any sort of solution that 

can lock down the data. That way of thinking about security is analogous 

to allocating enormous resources to make every car into a tank!
132 | Modernizing Learning

SUMMARY OF CURRENT 

BEST PRACTICES

Subsequently, cyberspace has become an asymmetric battlefield, upon which 

attackers operate at a disproportionate cost advantage and seek to win through 

attrition. While these problems may seem intractable, there are specific best 

practices that can preserve the confidentiality, integrity, and accessibility of 

distributed learning architectures without exorbitant expense. The single most 

critical practice for security requires regular examination of standards, requirements, protocols, and implementations. Effective cybersecurity requires 

extensive review of technology specifics, which is far beyond the scope of 

this document. Instead of an exhaustive review, we consider here a few extant 

vulnerabilities within the current distributed learning protocols. The goal is 

two-fold: first, to support immediate improvement and, second, to support 

ongoing sustainment in security that will result in cost-efficient reliability 

across distributed learning architectures. The implementation plan at the conclusion of this chapter recommends a process for further review and hands-on 

validation, which will allow a rank-ordered task list after a structured risk 

management process.

Future Learning Ecosystem 

Implementation Layers

Data security is a mature, albeit continuously evolving, field relative to the 

general IT field. Those best practices translate to the distributed learning architectures, which build upon common operating systems, servers, and network technologies. However, the future learning ecosystem will also need 

unique interoperability data formats, transport layers, interfaces, and storage 

solutions. Two examples of these are xAPI and Kafka. 
Data Security | 133 

EXPERIENCE APPLICATION PROGRAMMING INTERFACE 

xAPI is an example of an interoperability specification that enables data collection about a wide range of online and offline learning experiences. It provides a standardized data structure and vocabulary for data captured across 

a variety of learning technologies. xAPI is designed for simplicity and flexibility, and it provides a basis for communicating and evaluating learning 

throughout the future learning ecosystem. A non-exhaustive list of application areas include: real-world activities, experiential learning, social learning, 

simulations, mobile learning, virtual worlds, and serious games. 

Systems conformant to the xAPI specification record interaction data, such as 

between people and learning content. These interactions can occur anywhere 

and often signal the potential for learning. The recording process involves the 

transmission of statements to a Learning Record Store (LRS), which is part of 

the xAPI technical specification. Each LRS can then share the recorded xAPI 

statements with other LRSs and across a range of other learning technologies 

(as access controls permit).

The xAPI specification provides this interoperability through a series of implementation layers:4

xAPI Interoperability Implementation

Layer Four

Correlates training data with broader 

job performance metrics

Layer Three

Designs data to flow seamlessly across 

applications regardless of semantics

Layer Two

Records any learning experience, 

including informal learning

Layer One

Improves upon previous (SCORM) 

tracking by adding new capabilities with 

current best practices
134 | Modernizing Learning

Security within and across each of these layers must allow for consistently reliable application without exposing organizations to unnecessary information 

risk. This is especially important as data become increasingly standardized 

across the wide range of learning interactions tracked by xAPI. Any security 

evaluation starts with an assessment of each of the controls that are currently 

in place. A preliminary analysis reveals that there are several vulnerabilities 

that require immediate consideration.

KAFKA 

Apache Kafka is an example of a message-oriented middleware system that 

can process learning record changes at a massive scale. It was developed as 

the message collection and analysis mechanism at LinkedIn and is probably 

best-known for allowing data processing with very high volumes of variable 

message rates in real-time.5 Its features include partitioning, replication, and 

fault-tolerance, which make it ideal for distributed messaging of big data. 

Generally, it is a unified platform that allows for reliable and asynchronous 

message exchange. 

ACCESS IS KEY. Our biggest issue right now in readiness: How do 

we get training to the point of need? Access is really the thing we 

have to focus on. Our networks are very secure, but they’re very slow and 

performance is lackluster. And then we have BYOD (bring your own device), 

but not all Marines have tablets and computers though they all have 

phones. So the real question is: “What is the balance between access and 

security?” I feel like I’m always fighting against the network guys—how do 

we get both yeses? Is the mission goal security or learning? 

Larry Smith

Technical Director 

U.S. Marine Corps College of Distance Education and Training
Data Security | 135 

Other examples of message-based middleware that can work for learning processing are based on the Advanced Message Queuing Protocol 1.0, which is 

an international standard (ISO/IEC 19464) with several implementation options that are optimized for smaller systems. Some of these options include 

ActiveMQ, Apache Qpid, and RabbitMQ.6

VISION FOR DATA SECURITY 

IN THE FUTURE LEARNING 

ECOSYSTEM

As standards and best practices change, so should the implementations within 

the distributed learning architecture. This suggests the need for a theoretical 

orientation that allows for a practical, continuous evaluation of learner data 

security.

Nothing humans make is impregnable, yet data sharing is also unavoidable. In 

the realm of distributed learning, this means we should acknowledge that we 

can neither completely eliminate the risk of a student hacking a data stream 

to get examination answers, nor entirely prevent sophisticated attacks from 

taking or changing information. However, we must find ways to (a) reduce 

the likelihood of successful attacks and (b) develop barriers to reduce their 

impact if penetration occurs. The core tenet of normal accident theory is that 

technological failures are inevitable when a system is complex, tightly coupled, and has catastrophic potential.7

Thus, we must consider the potential issues that can result from failures within interdependencies among complex systems involving identification, access 

control, authorization, auditing, network segmentation and boundary enforce-
136 | Modernizing Learning

ment, endpoint protections, encryption, and transaction security. The assumptions of normal accident include:

• Humans cause errors;

• Small accidents tend to escalate into big ones; and

• The organization of technology—not technology itself—usually 

causes problems.8

Current research in normal accident theory 

and organizational reliability suggests we 

should design strategies that handle breaches 

as inevitable and aim to prevent their spread.9

Despite our best efforts, systems within every 

distributed learning architecture have been, 

or eventually will become, compromised; so, 

our goal is to minimize that impact. In practical terms for distributed learning, it’s recommended to maintain segregation of data 

lakes through network and organizational 

segmentation. Each department or agency 

should maintain separate content networks 

and, within them, build compartmented sections for each learner type. This will simultaneously control the spread of breaches and 

preserve data integrity by creating a content 

blockchain. Further, centrally-managed syndication and subscription to content will help 

preserve confidentiality to avoid aggregation 

that can reveal organizational priorities and 

strategic aims. A few specific security concerns are addressed in the remainder of this section to ensure network and endpoint protections, in the near 

term, and security sustainability, in the long run.

The most common application 

of accident theory is on 

roadways, where we assume 

there is no set of systems that 

will entirely prevent all traffic 

accidents. Subsequently, safety 

mechanisms like seat belts and 

bumpers limit the impact of 

each accident, and medians 

and shoulders on high-speed 

corridors provide spacing 

that prevent the tragedy of 

fatal collisions from becoming 

multi-car catastrophes.
Data Security | 137 

Hardening Networks

The highly technical nature of a firm’s information storage and retrieval system makes the Intrusion Detection System (IDS) and Intrusion Prevention 

System (IPS) useful components for breach identification. While most intrusion detection and intrusion prevention systems monitor network traffic, 

host-based anomaly detection can reveal and report unauthorized attempts 

to access examination answers or to manipulate grades. There are also several commercially available Security Incident and Event Management tools 

(SIEM), which explicitly monitor network logs and data flows for indicators 

of compromise. The inclusion of these tools is likely to significantly increase 

awareness of security compromises, reduce detection timelines, and inform 

organizational needs for response. For distributed learning, data streams 

should be designed as one-way valves. Data lakes should be tightly patrolled 

with a SIEM and organizational Security Operations Centers (SOC), which 

monitor the SIEM data and conduct live response around the clock. Several 

Managed Security Services Providers (MSSPs) provide SOC capabilities for 

organizations that are too small to maintain their own defenses.

CROSS-LEVELING STATE-OF-THE-ART SECURITY

A more extensive review of the xAPI and Kafka standards, in light of the 

Kerberos protocol, is likely to yield an elegant alternative to the current security schema. Furthermore, the integration of a robust security layer within the 

API can provide abstraction that simplifies the instantiation of authentication 

mechanisms across content providers and distributed learning hosts.

Kerberos was developed as the network authentication protocol for on-campus 

communications at Massachusetts Institute of Technology. Its main strength 

is that it’s designed to be secure even when performed over an insecure network. More specifically, passwords never transit the network during the session authentication process. Each transmission is encrypted using a secret key 
and attackers can’t gain unauthorized access to a service without compromising an encryption key or breaking the underlying encryption algorithm. It’s 

designed to protect against replay attacks, where an attacker eavesdrops and 

retransmits legitimate communications. Further, the protocol uses symmetric 

key encryption, which makes it computationally efficient at the device-level, and thereby suitable for use on resource-constrained devices. The use of 

symmetric key encryption also provides resilience to potential compromises 

of the Certificate Authority within a Public Key Infrastructure. Finally, Kerberos has a widely-available open-source implementation, which facilitates 

non-proprietary integration into government-owned systems.10

HARDENING DEVICES

The best way to harden a device is to take an offensive mindset to determine 

the ways an attacker might attempt to compromise that system. Through attack/defend exercises, defenders can learn which vulnerabilities result in the 

most critical exploits and investigate ways to remedy security deficiencies. 

Regular penetration tests will reveal vulnerabilities on each device and across 

EXAMPLE – PRELIMINARY VULNERABILITY ANALYSIS 

The most significant residual risks associated with Kerberos occurs when 

endpoints are compromised. If the Authentication Server is compromised, 

attackers can generate a validly encrypted Ticket Granting Ticket. If the 

Ticket Granting Server is compromised, attackers can configure it to ignore 

the initial authentication to the domain controller, as well as obviate the 

service prescription. That allows the attacker to generate tickets for any 

service, not just those that would be normally defined by the Authentication 

Server, but it cannot authenticate new users to the domain or allow offline 

password cracking. If the Service Server is compromised, there is no 

fraudulent ticket generation, but it can bypass the need for client to have a 

ticket at all.

Note: The Golden Ticket Attack grants tickets and persistent access to any service for 10 

years, but can be prevented with a relatively simple network security setting.
Data Security | 139 

networks. Across any distributed learning architecture, this is most important 

for those devices related to the access control for a personalized learning data 

store network (public key repositories, domain controllers, certificate authorities, and authentication servers) and for evaluation materials (content repositories that include answer keys).

SOCIAL HARDENING: DEVELOPING RESILIENCY

Learning management is integral to the concept of social hardening. To economize network and device hardening efforts, an explicit evaluation of human 

behaviors within the organization along with associated training interventions 

are needed. From a security perspective, social hardening is an opportunity 

to develop organizational resiliency because humans start to learn the “why” 

behind the design of technical controls and how they can prevent and contain 

EXAMPLE – SOCIAL HARDENING ACTIVITY 

Purple team exercises are one mechanism for social hardening. They teach 

network defenders what attacks look like on their own network. These 

testing and training events involve live attack/defend scenarios for IT staff and 

a cross section of their managerial hierarchy. A team of penetration testers 

(red team) openly engage in attacks while the threat-hunting defenders 

(blue team) try to spot and deny those attacks in real-time. These scenarios 

can take place on the organization’s environment, in a virtual replica of that 

environment, or within a simulated disaster scenario on a near-neighbor environment. Purple team scenarios can also involve sets of novice and expert 

end-users, who are valuable for considering and evaluating impacts, based on 

their experience. The incorporation of end-users can provide insight to how, 

when, and why users may attempt to bypass security controls. In practice, 

these exercises reduce the time it takes to detect attacks, test organizational 

response procedures, discover previously hidden vulnerabilities, and ultimately result in a superior organizational security posture. Purple teaming 

exercises often work best when performed by internal staff and facilitated by 

an independent third-party. When necessary, external penetration testing can 

be a strong substitute, if internal red teams are unavailable.
140 | Modernizing Learning

data breaches. The most important component of social hardening, though, is 

institutional retention that creates a culture of best practices.

Implementation Recommendations

Broadly speaking, the plan for implementing security within the future learning ecosystem should include four phases. Some form of this plan is likely the 

most rapid and cost-effective way to improve cybersecurity capabilities in an 

extensible and forward-leaning manner. 

PHASE 1: DEFINE SECURITY REQUIREMENTS. The need for security is 

evident, but which standards to mandate is less clear. Security requirements 

engineering is a first step in this process because it offers a disciplined look at 

the interoperability needs across the system-of-systems. This step will identify and stress-test the various security procedures already in place, validate 

which portions of those protections are inadequate, and conduct collaborative 

attack/defend exercises with current content providers to validate findings. 

EXAMPLE – IMPLEMENTATION ACTIVITIES 

Phase 1 is likely to include a series of purple team exercises. These involve 

a team of penetration testers (red team) who openly engage in attacks, while 

the threat-hunting defenders (blue team) try to spot and deny those attacks 

in real-time. These scenarios create local, contextualized learning because 

they generally take place on the organization’s actual environment or in a 

virtual replica of that environment. 

During Phase 2, the process of local learning that occurred during Phase 1’s 

purple team exercises should be transformed into multimodal instructional 

content. This should involve building the notes and findings from Phase 1 into 

case studies that educate the wider community. For example, these might 

include lectures, online labs, and evaluation materials specifically designed to 

teach learning technologists about the threats and cybersecurity protocols of 

their own organizations. 
Data Security | 141 

This will culminate in an economized application of available resources focused on prioritizing efforts to those most likely, critical, and impactful security improvements.

PHASE 2: DESIGN, IMPLEMENT, AND EVALUATE SECURITY LEARNING ACTIVITIES. Learning new processes is an optimal way to project continuous improvement into the future. During the execution of Phase 1 there’s 

an ability to monitor and evaluate the practices of security requirements engineering. These evaluations can yield individual and organizational learning 

activities that are derived from use cases within actual distributed learning 

architectures. Further, integrating Phase 1 and 2 schedules and teams is highly likely to generate cost-efficiency. These learning activities will build understanding regarding the specific processes for security engineering within distributed learning environments, which is highly likely to yield future security 

improvement across multiple generations of the technology itself.

PHASE 3: DRAFT SECURITY POLICIES AND STANDARDS. In addition 

to mandating specific security protocols and acceptable technologies in the 

short-term, there’s an opportunity to inculcate a long-term process focus 

within security policies and standards. For example, requiring a third party to conduct an annual security vulnerability assessment is common to the 

military (e.g., Army FM 3-19.30.2) and has been adopted within the financial 

industry (e.g., 23 NYCRR 500). An organization’s draft security policies and 

standards should help integrate both product- and process-focused needs into 

an attempt at establishing lasting security across its learning ecosystem.

► Networks – Network hardening should be the first step in securing 

learner data. This can take several forms, though it could involve an 

initial round of vulnerability testing, development, and deployment 

of a Kerberos-inspired alternative relevant for the learning ecosystem 

data formats, repositories, and transportation layers (such as defined by 

the xAPI and Kafka standards). This is especially promising given the 
142 | Modernizing Learning

potential for applying normal-accident theory to achieve a high-reliability learner data schema that hardens both network and, later, devices. 

► Devices – Device hardening is likely to present challenges due to the 

disparate nature of machines seeking to read, write, and execute the files 

associated with learner data. Subsequently, this step consists of systematic review of individual agency standards, and will recommend a viable 

minimum standard for device connectivity. 

► Humans – Social hardening is a difficult challenge, especially for technology-focused personnel. Evaluating and improving the human component in data security requires an understanding of both human behavior 

and technology to define policies and standards that shape behaviors that 

deny human-enabled cyber-attack vectors. Careful review of existing 

personnel security standards, such as the Army’s Threat Awareness and 

Reporting Program (AR 381-12), are likely to yield a series of best practices for securing the human element in distributed learning architectures. 

PHASE 4: PREPARE EXPECTATIONS AND MANAGE RISK. No security 

plan can completely eliminate risk. The accelerated pace of technological 

change makes this especially true for systems that aggregate, store, and process data. The final phase of this plan explicitly examines the risks, controls, 

and residual risks associated with current security findings in light of expected future technologies. The result of Phase 4 should include an assessment of 

when the policies and standards drafted in Phase 3 may require update. Major deliverables should include a list of assumptions, findings, and indicators/

warning of disruptive impact on the analyses conducted in this plan.
Learner Privacy | 143 

CHAPTER 8

LEARNER PRIVACY

Bart P. Knijnenburg, Ph.D. and Elaine M. Raybourn, Ph.D.1

Privacy is particularly important for distributed learning systems because 

managing learners’ trust among disparate sources is like managing the privacy of apps on one’s phone—a difficult task that arguably becomes even 

more pertinent when it deals with sensitive learning data. Particularly, some 

systems explicitly consider every activity of their users as a potential learning 

activity, thereby playing into people’s tendencies to learn and train not just in 

the classroom but also in natural settings. 

Modern digital learning systems employ ubiquitous data collection to enable 

highly personalized and pervasive learning recommendations. Going beyond 

a fixed, one-size-fits-all curriculum of activities, these systems track learners’ 

progress in minute detail and tailor subsequent learning activities to their performance. While this helps tremendously in achieving highly efficient learning practices, the data collection and user modeling practices employed by 

such systems may cause privacy threats that act as a barrier to their adoption. 

As users’ trust in personalization providers is starting to fail, it’s crucial to 

investigate the privacy implications of such data collection and learner modeling practices.

Social networking capabilities, often featured in learning systems, may also 

introduce privacy considerations that may inhibit their adoption. Users have 

expressed severe privacy concerns with social networks, yet users of these 

applications tend to struggle with managing their privacy on these networks. 

Hence, it’s important to provide thoughtfully designed privacy management 

mechanisms in learning applications.

 

 
144 | Modernizing Learning

PRIVACY IN THE FUTURE 

LEARNING ECOSYSTEM

In many existing learning systems, privacy controls are an afterthought—a 

series of privacy settings accompanied by a complicated privacy policy. In 

contrast, the future learning ecosystem should employ the philosophy of privacy by design2

 to allow developers and researchers of such systems to select 

the characteristics that best alleviate users’ concerns. Moreover, the implementation of user-tailored privacy will allow systems to model learners’ privacy concerns and provide them with adaptive privacy decision support.3

While this may nominally lengthen the development cycle, it prevents a situation where the system has numerous complex privacy settings and a complicated privacy policy that learners are unable to navigate—or worse yet: no 

privacy protections at all.

Data Collection

Many types of data might be available through a digital learning system, including learner runtime activity, competencies, and context. Such data can 

be collected anonymously or identifiably connected to a learner’s profile. The 

data collection practices of a digital learning application can have unique privacy implications depending on the type of data collected, its source, and its 

potential identifiability. This section discusses how to consider those aspects 

when defining and developing the data collection practices of a digital learning application.
Learner Privacy | 145 

PRIVACY DECISION-MAKING

Arguably the most important advice for developers of distributed learning 

systems is to study the privacy concerns and practices of the (potential) users 

of those systems. One of the most consistent findings in privacy research is 

that people vary extensively in their information disclosure practices.4 Generally, users of digital systems acknowledge the benefit of data collection for 

personalization, but when taken too far, the same data collection can deter 

users from using the system extensively or even dissuade them from using the 

system at all.5 The point where this happens differs per user. Understanding 

how different learners make privacy-related decisions can inform strategies 

that help alleviate these issues.

An often-used conceptualization of people’s conscious information disclosure 

decisions is the “privacy calculus,” which suggests people make privacy decisions by balancing the perceived risks and perceived benefits of the available 

choice options. Therefore, it’s important that digital learning systems highlight 

the relevance of a requested disclosure behavior, and that they refrain from 

asking for information in situations where relevance is not readily apparent.

Research has also demonstrated that user trust has a significant influence on 

disclosure behavior in digital systems.6 Therefore, building trust is an important strategy for increasing acceptance of the data collection and tracking 

practices employed by modern digital learning systems. Trust can be built 

by ensuring learning applications originate from trustworthy sources, and by 

employing sensible, transparent data collection practices from the outset.

However, people aren’t always rational in their privacy decision-making: 

when they make “heuristic” privacy decisions, they don’t carefully weigh 

risks and benefits; instead, they rely on superficial but easily accessible cues, 

such as website reputation, ostensible privacy guarantees, and design quality. 

Digital learning systems should survey their users to learn more about the 

heuristic decision processes that may negatively affect disclosure. Moreover, 
146 | Modernizing Learning

they should tailor to learners’ heuristic privacy decision-marking processes 

by giving them sensible default settings and providing both rational (e.g., privacy policy) and heuristic (e.g., privacy certifications or seals) sources of trust. 

Learners with low levels of motivation (privacy concerns) and/or low self-efficacy (privacy literacy) are more likely to make heuristic privacy decisions. 

If rational privacy decision-making is required, digital learning systems can 

attempt to instill motivation and ability in their users by providing contextualized privacy controls and easy-to-understand privacy information, such as 

instructions designed as cartoons or comic strips.7

COMMUNICATION STYLE

Privacy in digital learning systems extends beyond personalization; it’s also 

relevant to the interpersonal (“social networking”) aspects of these systems. 

Social networks typically provide a plethora of mechanisms to manage one’s 

privacy beyond disclosure, and research finds that users tend to employ a wide 

variety of strategies to limit their disclosure, such as the six privacy management strategies uncovered by Pamela Wisniewski and her colleagues 8

 (see 

Figure 8-1). These archetypes arguably extend to other social network-based 

systems, including social learning platforms and other applications or features 

that leverage social networks in learning systems. 

Internet users also choose their social network based on their preferred communication style. Research9

 suggests services that broadcast implicit social 

signals (e.g., location-sharing social networks) are predominantly used by 

“FYI (For Your Information) Communicators,” who prefer to keep in touch 

with others through posting and reading status updates. They tend to benefit 

from the implicit social interaction mechanisms provided by broadcast-based 

social network systems. People who are not FYI Communicators, on the other 

hand, would rather call others, or otherwise interact with them in a more direct manner. They tend to benefit more from systems that promote more direct 

interaction. In order to tailor to both types of communicators, digital learning 
Learner Privacy | 147 

systems should employ both automatic social-network style sharing (for FYI 

Communicators) and direct, chat-style interaction (for non-FYI Communicators). Further, since the communication styles of FYI and non-FYI Communicators are at odds, developers should also pay attention to effects of integrating different communication styles within a single application.

 Digital learning systems that employ or 

 implement social network components should 

 tailor their privacy functionality to different 

 privacy management styles 

LEVELS OF IDENTIFIABILITY

The use and sharing of learners’ personally identifiable information (PII) deserves special attention, because it presents the risk of revealing the identity 

of learners to other parties. PII can be defined as any information that could 

be used on its own or with a combination of other details to identify, contact 

or locate a person, or to identify a person in context. The privacy concerns 

associated with PII can be mitigated by allowing users of a digital learning 

system to remain fully anonymous.

Fully anonymous interaction means that there are no persistent identifiers 

associated with the user. This is difficult to accomplish in digital learning 

systems, though, since most learning activities follow a trajectory over multiple interactions, which means that the system must be able to recognize 

the learner across these interactions. More realistically, users can be allowed 

to interact with the digital learning system under a pseudonym. The effectiveness of pseudonyms and other means of de-identifying personal data has 

been called into question, however, since such data may still be at risk of being re-identified, especially in digital learning systems that collect data with 

high dimensionality and sparsity.10 Regardless, researchers have argued that 
Privacy Management Archetypes 

PRIVACY

MAXIMIZER

Highest level of 

privacy behavior 

across the majority 

of privacy features

SELECTIVE

SHARER

Leverages 

more advanced 

privacy settings

PRIVACY 

BALANCER

Moderate levels

of privacy 

management

TIME SAVER

Use strategies to be 

passive consumers, not 

bothered by others

SELF-CENSOR

Censors by withholding 

basic and contact 

information

PRIVACY 

MINIMALIST

Lowest level of privacy 

modification behavior

Block Apps

and Events

Selective

Sharing

Block People

 Reputation

Management

Timeline and 

Wall Moderation

Withholding 

Basic Info

Friend List 

Management

Altering 

News Feeds

Restricting Chat

Limiting Access Control

People tend to use various privacy management strategies to greater or lesser extents

Withholding 

Contact Info

Figure 8-1 (derived from work by Wisniewski and colleagues) 
Learner Privacy | 149 

de-identification of server data is still a good security practice, as it would 

take considerable effort to re-identify all users if the server is compromised.

COLLECTING LEARNER DATA

Digital learning systems have an opportunity to collect a wide array of data 

about their users. Long-term persistent data tracking allows learning systems 

to personalize learning and uncover useful insights about the learner base. 

However, each type of data also has unique privacy implications that must be 

considered. At the most granular level, digital learning systems can collect 

“learner runtime activity”—users’ step-by-step actions that can be used to 

track users’ progress and to adapt the learning experience to their specific 

abilities, knowledge, and pace. 

Continuous tracking may create a digital panopticon that restricts user freedom. Therefore, users should be given easy-to-use notice and control mechanisms to manage the boundary between leisure and learning. Additionally, 

users’ runtime activity should be carefully protected through a combination 

of strict access control, de-identification, obfuscation, encryption, and/or client-side personalization (see later sections).

INFERENCES 

Learners’ privacy concerns can also be impacted by the inferences made 

about them by the digital learning system. Users of personalized systems are 

negatively impacted when these systems make incorrect inferences about 

them. Even when inferences are correct, they may not always be wanted by 

the learner or be in their best interest. For example, research has shown that 

people are intuitively uncomfortable with the idea that sites track their data,11

which may reduce their trust and negatively affect their disclosure behavior. 

Theories and recommendations for self-regulated learning practices should 

be incorporated into the trust-building requirements during the development 

phase.
150 | Modernizing Learning

OUTPUT MODALITIES 

AND DEVICES

Future digital learning systems are envisioned to be pervasive, multi-device experiences that might include smartphones, 

smart TVs, e-books, smart watches, and 

a multitude of other devices. These devices each present unique privacy considerations. Personal devices, such as 

smartphones and wearables, are ideal for 

real-time learning but can also create distractions. Therefore, learning experiences 

on such devices should be structured such 

that they don’t disturb learners or reveal 

information about them in uncontrolled 

ways (such as a push reminder displayed 

as a popup—while projecting to a group from a smartphone). Strategies for 

achieving this include planning notifications carefully, avoiding interruption 

of a learner’s current task, and adapting notification timing to the learner’s 

context. Devices that are shared by multiple people (e.g., smart TVs) should 

also avoid leaking personal information in social settings. To do so, notifications on such devices should provide generic recommendations that mask 

details unless they are requested by the learner.

DATA LOCATION AND OWNERSHIP

A typical reason for integrating learning experiences in a distributed platform 

is to provide recommendation and adaptation capabilities across these learning experiences. This requires the implementation of data collection and storage facilities, communication channels, and adaptation capabilities. In most 

systems, these components will be centralized, so building trust between the 

learner and these components is extremely important. This can be done by 

panopticon / pan·op·ti·con / 

noun – a circular prison design, 

built for surveillance; so that 

all (pan-) inmates could be 

observed (-opticon) by a single 

watchman at all times—and 

so inmates knew they were 

always being watched.
Learner Privacy | 151 

putting these components under control of a trusted, local entity, such as a 

single department or organization. However, this may also shield the components from important insights that can be gained from data collected across 

instances, and it may make the mobility of learner data more cumbersome. 

Instead, one could build a learning platform where all users, departments, and 

organizations share the same centralized components. However, a single entity that collects the data of all users creates an attractive target for hackers.12

A good trade-off is therefore to put these components at a level that is “low” 

enough for learners to trust but high enough to allow efficient mobility and 

user-modeling synergies. In other words, data/insight mobility problems can 

be reduced through portability requirements and standardized APIs.

Another question is how each learning application on the platform can access 

learners’ data. Since users are likely to trust different applications to different extents, an access control mechanism is needed to allow applications to 

optimally utilize the learners’ data while at the same time respecting each 

learner’s privacy preferences. A recent development in adaptive systems is to 

perform the calculations required to compute adaptations “client-side” rather 

than on a centralized server. Research shows that such client-side methods alWe think there’s a big opportunity to open that data up to an ecosystem 

concept. For example, predictive analytics can help identify who will do 

poorly or who will do well in courses...but should we show that to students? 

Will we create a self-fulfilling prophecy? It’s important to consider the possible 

unethical deployment of this. The way to avoid it is to use a governance 

system to manage the data systems and be thoughtful about this. 

Phill Miller, Chief Learning 

and Innovation Officer, Blackboard
152 | Modernizing Learning

leviate privacy concerns.13 However, client-side adaptation methods can only 

use limited inference methods (e.g., if-then rules, simple classification), and 

research has shown that users are concerned that their data can be hacked if 

their device is stolen, and that their user model is lost forever in case they lose 

or break their device.

Given these considerations and limitations, we suggest a three-tier data management and personalization approach: On the first tier, learner competency 

data is used by the platform to decide what learning applications to recommend to the user (meta-adaptation). On the second tier, individual applications 

can use similar data—albeit with regulated access control—to make app-level 

adaptations (macro-adaptation). Finally, on the third tier, client-side mechanisms can use fine-grained learner runtime data and behavioral tracking to 

make subtle adjustments to the learning experience (micro-adaptation).

DATA OWNERSHIP AND STEWARDSHIP 

The end-user license agreement of most modern online services claims full 

ownership over the personal information they collect about their users. The 

legality of this claim is questionable though: The legal concept of “owning 

information” is still new, and laws are still being written about this topic. 

Moreover, preliminary investigations among users show that there are merits 

in granting end-users ownership of their personal information, and it may expedite the movement of data among different digital learning systems. However, data ownership is not exclusive, and it may be desirable to give other 

entities (e.g., applications, employers, researchers) partial co-ownership over 

an individual’s data. These co-owners should request minimal amounts of 

data, avoid duplicate storage, and de-identify data where feasible. 

Data ownership puts an important responsibility on the shoulders of the learners. It allows them to play an active role in making sharing decisions about 

their data, but not all users may be motivated and capable of taking on this 

responsibility. 
Learner Privacy | 153 

In the 401(k) model, learners formally own the data, but they can partially 

delegate the responsibility of making decisions regarding their data to a fiduciary, such as a teacher or administrator. As a “data steward,” this fiduciary 

would then be allowed to make decisions on the learner’s behalf; although, 

there should be a strict policy that outlines the 

limits of these powers. This policy can outline 

several practices that are always allowed, never allowed, or require the explicit consent of the 

user. In the latter case, such consent should not 

just be a notice with an option to “opt out.” Rather, it should ask the user to formally opt-in to the proposed practice—this 

practice makes it more likely that learners will make an informed consent 

decision. 

Finally, when more than one party has a say over the disclosure and use of certain data, Private Equality Testing can be used to create a Two-Person Concept solution (a concept proposed by U.S. Air Force Instruction 91-104 [16]) 

that prevents any single person from intentionally or unintentionally leaking 

data or becoming victimized by extortion or social engineering attacks.

Data Sharing 

Data collected in digital learning systems can be used for purposes outside the 

system. One such purpose is to make the data available to the learner themselves, which allows quantified self–like innovations. Beyond this, learning 

systems can allow learning materials, activities, and outcomes to be shared 

with fellow learners (enabling social learning experiences), researchers (catalyzing learning innovation), and employers (informing organizational decision-making). This section covers the privacy-related consequences of the 

social, academic, and organizational use of data collected and generated by 

digital learning systems.

Structure data 

ownership like 

a 401(K)
154 | Modernizing Learning

QUANTIFIED SELF

By sharing learner data with the learners, themselves, digital learning systems can create a “quantified self” experience that allows them to gain insights into their own data. For example, carefully constructed personalized 

infographics can allow individuals to explore the common and unique sides 

of their identities.14 Such insights are an important reason for many people to 

accept the potential privacy intrusions that come with wearable technologies 

and constant tracking. As such, the quantified self can be a motivating factor 

behind the data collection efforts of a digital learning system. Also, the quantified self can be a catalyst for learning. Translating self-tracked parameters 

into a game-like structure can create new motivational and heutagogical support structures that encourage and enable users to push themselves further.

SOCIAL LEARNING EXPERIENCES

Sharing learner data across learning environments can, in some cases, be 

considered a violation of regulations, such as the Family Educational Rights 

and Privacy Act or General Data Protection Regulation. Hence, care should 

be taken that the learner (not the system) makes the decision to disclose such 

information. Even learners willing to share might not want to share with all of 

their contacts because they could be bothered by an overload of social activity.15

As such, users should be allowed to select a subset of their contacts for sharing purposes, and the learning system can actively help them in this process. 

RESEARCH AND ORGANIZATIONAL DECISION-MAKING

Learning data can also be used for research and organizational decision-making. Privacy experts argue that secondary use of information should be explicitly communicated to users, otherwise they may be surprised to find out 

about it and feel that their privacy is violated.16 Moreover, there are laws and 

regulations surrounding research and employment-related practices that need 

to be adhered to. For example, whereas employment discrimination is ille-
Learner Privacy | 155 

gal, algorithmic decisions have been shown to incorporate unwanted biases. 

Therefore, ethical considerations need to be made before using machine judgment for, e.g., promotion decisions.

Privacy Support Mechanisms 

Several techniques for privacy support can be implemented in digital learning 

systems. This final section discusses their benefits and shortcomings. 

PRIVACY NOTICES 

Online privacy policies are often written in a legalistic, confusing manner 

and require a collegiate reading level to understand them. Indeed, while many 

people claim to read online privacy policies, many don’t actually review them 

or don’t read closely enough to understand them.17 A lot of work has therefore 

gone into summarizing privacy statements, but summarized privacy notices 

are often too simplistic to accurately represent the policies they reflect.18 One 

way is to add textured agreements, which add layers of emphasis to make 

the text more readable,19 but these have been shown to increase (rather than 

decrease) the amount of time people spend reading the agreements. Although 

the consensus is that people should be informed about the privacy decisions 

they are asked to make, the reality is that doing so often makes them more 

fearful or unwilling to come to a decision. The conclusion, then: It’s better 

not to rely on any privacy notices, but to instead make the privacy decisions 

themselves simpler. 

CONTROL MECHANISMS 

Simple privacy controls can help users take control over their privacy settings. 

For example, in social sharing settings, recipients can be grouped to simplify 

the decision landscape and graphical representations of the control matrix 

can help users understand and manage their sharing patterns. Selective infor-
156 | Modernizing Learning

mation sharing is just one of many strategies users may employ to alleviate 

privacy tensions. Likewise, privacy control can be provided in more diverse 

and intuitive ways than a traditional “sharing matrix” in which users specify 

who gets to see what. Research has found that it’s important to give users the 

privacy features they want, lest they experience reduced connectedness and 

miss out on social capital.20

 Unfortunately, while users claim to want full 

 control over their data, they often avoid the 

 hassle of actually exploiting this control 21

In combination with overly permissive defaults, users’ avoidance of control 

mechanisms leads to a predominance of over-sharing. In order to facilitate 

control, digital learning systems should use smart default settings and make 

the available controls as simple as possible.

PRIVACY NUDGING

Nudges are subtle yet persuasive cues that make people more likely to decide 

in one direction or the other. An example of a privacy nudge is a justification 

that makes it easier to rationalize a privacy decision. Justifications include 

providing reason for requesting the information, highlighting the benefits of 

disclosure, appealing to the social norm, or providing a symbolic character 

to represent the trustworthiness of a recipient (e.g. a “privacy seal”). Another 

approach to nudging users’ privacy decisions is to provide sensible default 

settings, which tend to nudge users in the direction of that default.

The privacy nudges evaluated to date usually only work for some users, however, and they leave others unaffected or even dissatisfied. Some researchers 

argue that this is because nudges take a “one-size-fits-all” approach to privacy.22 Since such nudges are rarely good for everyone, they may actually 

threaten consumer autonomy. It’s therefore best to only use nudges if there’s 
RECOMMENDATIONS

DEFAULTS

JUSTIFICATIONS

MEASURE MODEL ADAPT

ORGANIZATIONAL

 CONSTRAINTS 

 AND PRACTICES

DATA USER

OTHER

FACTORS

RECIPIENT

USER PRIVACY MODEL

USER

BEHAVIORS

USER

CHARACTERISTICS

Figure 8-2: A schematic overview of user-tailored privacy

TWO EXAMPLES TO HELP ILLUSTRATE THE USER-TAILORED PRIVACY CONCEPT

A digital learning system normally tracks users’ location (Data) in order to give contextrelevant training exercises (Organizational practice). However, user-tailored privacy 

knows that like many young mothers (User characteristic), Mary (User) does not want her 

location (Data) tracked outside work hours (Other factor). It therefore turns the location 

tracker off by default when Mary is not on the clock (Default).

David needs to decide how to share his recent milestones—two certificates he’s just 

earned (Data)—within his organization (Recipient). Due to the rules of his employer 

(Organizational constraint), user-tailored privacy requires him to share these milestones 

with his direct supervisor (Recipient). Moreover, from his previous interactions (User 

behaviors), the user-tailored privacy knows that David keeps close ties to several other 

divisions. User-tailored privacy therefore suggests (Recommendation) that he should share 

his new certifications with the heads of these divisions (Recipient) as well, explaining that 

they’re likely to be interested in exploiting his newly gained skills (Justification).

1

2

User-tailored privacy aims to strike the balance between giving learners 

no control over or information about their privacy at all versus giving 

them full control and an overload of information about it.
158 | Modernizing Learning

consensus among learners about privacy. In these situations, nudges apply 

privacy by default but give learners a choice in the event that they want a different setting after all.

USER-TAILORED PRIVACY 

User-tailored privacy is a novel means to support users’ privacy decision-making practices.23 A user-tailored privacy–based system first measures users’ 

privacy-related characteristics and behaviors, then uses this as input to model 

their privacy preferences, and finally adapts the system’s privacy settings to 

these preferences (see Figure 8-2). 

The first step to user-tailored privacy is to measure learners’ privacy-related 

characteristics and behaviors. To accomplish this step, learning system developers should acknowledge the plurality and multi-dimensionality of users’ 

decision-making practices. They should also note the variability of learners’ 

privacy practices; although, these can often be captured by a concise set of 

“privacy profiles,” and, similarly, the potential data recipients can often be 

organized into a number of groups or “circles.”

The next step is to model privacy. This can be done in a way that matches the 

learners’ current privacy practices; however, in some cases, it may be better 

to suggest privacy practices that are complementary to their current practices, 

and in still other cases, it may be best to completely move beyond learners’ 

current practices. The model can also take the practices and constraints of 

users’ organizations into account. Finally, using this user model, user-tailored 

privacy can personalize the privacy settings of a digital learning application 

as well as the justifications it gives for requesting certain information, its privacy-setting interface, and its learning recommendation practices. 

Arguably, user-tailored privacy relieves some of the burden of the privacy 

decision from a learner by providing the right privacy-related information and 

right amount of privacy control, without being overwhelming or misleading.24
Learner Privacy | 159 

Implementation Recommendations

We recommend several steps in the development process that will both build 

intuitive privacy controls into the design of the learning ecosystem as well as 

create privacy-sensitive recommender agents to guide learners.

1. DECISION-MAKING

Build trust: Ensure that the learning applications originate from trustworthy 

sources. Employ sensible data collection practices and a privacy-by-design 

philosophy from the outset. Finally, provide contextualized privacy control 

mechanisms and easy-to-understand privacy information

2. COMMUNICATION STYLE

Tailor to different privacy management strategies: Give Selective Sharers the 

ability to selectively expose data to specific apps and groups of people. Allow Self-Censors to use non-personalized mechanisms for selecting learning 

material and to restrict their forms of sharing. Allow Time Savers to opt out 

of active notifications and social features. Give Privacy Maximizers all of 

the functionality; Privacy Balancers mechanisms for curation, blocking, and 

avoiding direct interaction; and Privacy Minimalists adaptive and social functionalities within the ecosystem.

“What we found when we studied the FAA is that the lines between training and 

operations are blurring. ...Aircraft have sensors with analytics; so, they can make 

profiles and tell if pilots do something unsafe. It allows the FAA to look into a program to provide information back to pilots. But the pilots, being union-driven and 

structured, said “No, you can’t watch us!” So, they made a union the go-between 

guardian for that data. This way, if there is an issue, there are a series of approvals 

and guardians of the data, so that the pilot can’t be punitively damaged but can be 

informed.” – Michael Smith, Senior Technical Specialist, ICF

U.S. FEDERAL AVIATION ADMINISTRATION
160 | Modernizing Learning

3. LEVELS OF IDENTIFIABILITY

Devise appropriate levels of identifiability: Use, but do not rely on, de-identification for privacy purposes while allowing creative and (self-)evaluative 

environments to use pseudonymity. Formal and diplomatic settings should 

enforce a real-name policy.

4. COLLECTION OF DATA TYPES

Protect learner runtime activity: Reduce unfettered context tracking to prevent the creation of a digital panopticon, and provide easy-to-use notice and 

control mechanisms to control the boundary between leisure and learning. 

Protect learner runtime activity using access control, encryption, de-identification, and obfuscation and, where possible, process and use learner runtime 

activity data locally.

5. OUTPUT MODALITIES AND DEVICES

Don’t disturb the user: Plan notifications carefully and provide easy controls 

for notification urgency. Adapt notification timing to the learner’s context.

Prevent leaking personal information in social settings: Provide generic notifications that do not reveal (potentially sensitive) details and change the 

amount of information provided in each notification depending on the number 

of people who are near the learner.

6. MANAGE ADAPTATIONS

Implement the centralized components of learning platforms at the appropriate level: Put centralized learning components under the auspices of a trusted 

entity and support the portability of learning models. Allow for interoperability of learning applications through standardized APIs.
Learner Privacy | 161 

Regulate access of individual learning applications to the centrally collected 

data: Allow learning applications to do their own adaptations and put access 

control mechanisms in place to regulate the use of centrally collected data.

Use client-side micro-adaptation: Collect and analyze learner runtime data 

in client-side applications. Prevent the unnecessary storage of this data, and 

handle it in an ephemeral manner to prevent data loss or theft.

7. DATA OWNERSHIP AND STEWARDSHIP

Give learners ownership over their data: Allow learners to peruse their raw 

data and user models, and enable them to take their data with them across 

different learning institutions or employment organizations.

Give employers and learning applications limited co-ownership: Allow employers and learning applications to co-own appropriate data while requesting 

minimal amounts of data. Avoid duplicate storage and de-identify data.

Allow learners to designate a “data steward”: Allow learners to delegate responsibilities to a “data steward” to manage their data under a fiduciary policy, and implement the Two-Person Concept using Private Equality Testing.

8. SOCIAL LEARNING EXPERIENCES

Give users control over what to share: Refrain from sharing any learning outcomes with others by default. Instead, require an explicit decision from learners before sharing learning outcomes with others. Allow learners to limit their 

connections to those they deem relevant for each application and implement a 

“learning buddy” recommender.

9. RESEARCH AND ORGANIZATIONAL DECISION-MAKING

Let learners know about secondary data use: Communicate secondary data 

use practices to learners and indicate exactly the data used and its purpose.
162 | Modernizing Learning

Act responsibly regarding research and organizational decisions: Anonymize 

research data, and make sure that promotion decisions are made in a non-discriminatory manner.

10. PRIVACY NOTICES

Increase the chance that learners read privacy notices: Use privacy nutrition 

labels to give learners a quick overview, and make notices textured to emphasize the details. Make the privacy notices attractive and approachable, such 

as by using use comics, and similarly, make the decisions simpler—ideally, to 

the point that notices are no longer required.

11. CONTROL MECHANISMS

Use accessible, graphical privacy controls: Make controls obvious and easily 

accessible. Use graphical methods to give users easy-to-understand controls, 

beyond just information access. Use a privacy setting interface that works for 

everyone (where possible) and keep it simple.

12. PRIVACY NUDGING

Use nudges if there is a consensus: Use justifications and defaults when virtually all learners agree on the optimal privacy setting, and incorporate nudges 

to provide learners choice in case they want different settings.

13. USER-TAILORED PRIVACY

Employ user-tailored privacy to support learners’ privacy decision-making 

practices: Measure learners’ privacy preferences in context, exploiting their 

multi-dimensional nature. Carefully balance recommending current, complementary, or novel privacy practices as well as proactive and conservative adaptation strategies.
Analytics and Visualization | 163 

CHAPTER 9

ANALYTICS AND 

VISUALIZATION

Shelly Blake-Plock

Analytics and data visualization are now mainstream. The maturation of 

cloud services and the adoption of new web technologies have accelerated 

both fields. Among the most important innovations has been the development of new streaming data systems. These technologies can handle the exponentially increasing scales of data produced—not only by traditional web 

and social media technologies but also by machines and sensors deployed in 

cyber-physical systems such as consumer wearables, smart city implementations, and connected industrial devices.

This chapter summarizes the state-of-the-art in streaming data, learning analytics, and data visualization for non-technical readers. It provides context, 

lays out a vision, and provides high-level guidance on implementation approaches. The goal is to provide practical knowledge to teachers and trainers, 

business users, and programmatic decision-makers, helping them to envision 

how learning analytics and visualization can increase the capacity of learning 

organizations as well as the general approach to such implementing systems.

What are we talking about?

There’s so much data in the world. Each of us produces a cloud of “data exhaust,” with every mouse click or upvote. Learners, too, generate masses of 

data—information that could inform education and training if we could access, 

analyze, and meaningfully visualize it. Two closely related fields—educational 

 

 
164 | Modernizing Learning

data mining and learning analytics—are providing tools to meet those goals.

Both fields differ slightly, for instance, based on their origins, primary application areas, and preferred AI algorithms.1

 Learning analytics grew out of 

efforts in the semantic web, and it’s practitioners tend to emphasizes big-picture analyses and decision-support for teachers and learners. Educational data 

mining developed out of the adaptive instructional technologies tradition, and 

it tends to focus on automated adaption and reductionist modeling.2 For our 

purposes, in this chapter, we’re less concerned with the finer details distinguishing the two disciplines. Instead, we’re focused on their shared purpose: 

Understanding and applying data-intensive approaches to education and training, particularly for large-scale learning data—so called big learning data.

3

As the phrase “big data” implies, training and education analytics often (but 

not exclusively) employ machine learning techniques. Machine learning is a 

subset of AI that uses algorithms to automatically uncover patterns in data to, 

for instance, assign classifications, estimate the influence of different variables 

on downstream outcomes, or make predictions based upon historical data. In 

the training and education domain, these applications have notably matured 

over the last 20 years, coalescing into the two communities mentioned above. 

But what can you do with these tools? People have applied analytics to a 

variety of learning systems. For instance, some applications use analytics to 

predict engagement and then recommend personalized resources to encourage students’ participation.4

 Others can analyze students’ interactions and 

proactively alert instructors as to which may need help.5 One well-known 

example, Purdue University’s Course Signals, used current data from an LMS 

combined with historical data (such as course attendance and prior grades) 

to forecast which students would fall behind in a course and then alert both 

learners and their teachers about their risk levels.6 Other tools apply similar 

retention management approaches across an entire student body, identifying 

those at highest risk of dropping out—in time for the administration to intervene.7

 Basically, any of the analytics applications we’ve come to expect of 
Analytics and Visualization | 165 

e-commerce systems, from time-sensitive personalized recommendations to 

system-wide trend analyses, can translate into analytics for learning.8

DIP YOUR TOE 

INTO THE STREAM 

Streaming-data analytics is a uniquely exciting and freshly emerging subfield 

within analytics. When we say streaming data, we’re generally talking about 

a range of data types that are event-based and track some variety of activities, 

whether human or machine in origin. The invention of streaming data has impacted the way we think about what data, itself, represents as well as how it’s 

leveraged to guide human insights or automated machine processes. 

For instance, in the domain of sales and marketing, event-based data has increased our capacity to understand the market and prospective customers. It 

provides a window (for example, via analysis of social media streams) into 

the story of the prospect’s journey, both as it relates directly and indirectly to 

a product or service offering. In the entertainment industry, streaming data 

informs recommendations of content, such as movies and television shows on 

Netflix. In politics, streaming data helps analysts identify and capitalize on 

public sentiment and social trends.

Data-stream architectures are contrasted against traditional 

batch-processing systems. Data streams are characterized by 

data moving at a high velocity. They also have strict constraints 

for processing the incoming data online, within limited amounts 

of memory and time, and they must always be ready to provide 

analytical predictions when queried.
166 | Modernizing Learning

Just as these technologies and data architectures have transformed business, 

entertainment, and politics, so too are they able to transform learning. In the 

learning space, the availability of activity-based data streams offers an opportunity to trace and understand learners’ journeys. Analytics in the service of 

these streams of data can provide accessible, automated, and near real-time 

data visualizations, as well as trigger alerts and interventions based on key 

performance indicators. These journeys—which comprise learners’ activity 

and behavior profiles—can be considered highly formative, quantifiable micro-assessments.

Digitizing the Analog World

We often see a desire to digitize the analog world. We wear digital watches 

that resemble their windable cousins. We create “offices” in our computers, 

mirroring the components of the physical workplace. In education, we digitize 

chalkboards, loose leaf, and books. But the inclination to recreate the analog 

world within the digital domain eventually confronts both the limits of analog practice as well as the more esoteric surprises of what, when it works in 

our favor, we call innovation. When we move from tangible “things,” such 

as chalkboards and books, to conceptual practices and processes, such as assessment, the situation gets particularly dicey. Esoteric and nuanced concepts 

become oversimplified to the point of caricature. This leads to notions being 

thrown around such as, AI will replace educators! or Automation could never 

substitute for teachers!—arguments that tend to betray a misunderstanding of 

both AI and teachers. However, in the world where access to learning is distributed across the internet, expansive in breadth and always available, there 

are practical limits to the analog approach of teaching. While there’s little 

danger that AI will “replace” human teachers, their role—and the way we implement training and education, writ large—needs to evolve in collaboration 

with evolving technologies.
Analytics and Visualization | 167 

 In a world that needs learning at scale, the real  

 conversation should be, how can AI serve the 

 needs of teachers—and vice-versa?  

Data at Scale

Contrast the analog “data set” with the contemporary “data assets” created 

by social media newsfeeds. These data assets support the creation of time 

series–based behavioral profiles that hold the activity records, built up over 

time, from users’ behaviors on social media platforms, including likes, comments, shares, photo posts, video watches—all user actions. These become 

part of the user’s behavioral profile, and, in turn, become nodes on a vast 

social graph. Each node owns a narrative. That data asset is key to the social 

media industry’s business model. It’s the aggregate of these profiles that creates the opportunity for more targeted advertising, and, at scale, it’s a most 

impressive record of formative experiences—of individuals, yes, but more so 

of vast aggregate populations.

For social medial data assets, value isn’t encapsulated in a single pinpoint 

score. It’s not even found in the ability to estimate a single user’s likelihood of 

accepting a given advertisement (although this certainly brings some benefit). 

Rather, or (at least) more importantly, value derives from the cumulative amalgamation of all these behavioral profiles. The power is in the aggregate. Only 

the scale of the aggregate provides the rich raw data necessary to uncover the 

array of patterns, categories of human interest, and shared narratives of human experience. It’s a matter of scale. Similarly, the challenge streaming data 

poses to the traditional view of assessment comes down to a matter of scale. 

A gradebook at scale will never offer the insights into learning experiences 

that an activity feed at scale can provide. This isn’t to denigrate gradebooks; 

rather it’s a reminder to recognize their functions and where their value lies. 
Consider a typical gradebook full of letter grades and percentages. In 

one sense, this table of letters and numbers offers a substantial bit of 

information about how one student may have progressed over time or 

how she compares to her peer group’s scores. But in another sense—in 

the sense informed by a world of streaming data, where data convey a 

narrative about students’ digital experiences—the gradebook tells us little 

about what actually happened, how it was done, and what it suggests 

about the learner. The gradebook, and the modes of assessments 

that inform it, are analog technologies. They’re no worse than digital 

technologies merely because they’re not computerized, but they are 

technologies reflecting an earlier paradigm—a paradigm ill-equipped to 

support learning at scale in a digitized, interconnect world.
Analytics and Visualization | 169 

Supporting Decision Making

Learning practitioners have long sought to increase their insights into formative development. For instance, teachers may subconsciously wonder, How 

far along is each student in his or her learning journey? Unfortunately, difficulty in gathering the data points needed to make confident and continuous 

formative appraisals makes the alternative—a big summative assessment—

seem like the only option. This can be understood as a scale problem. Yet, by 

leveraging activity and event-based data in a manner similar to what social 

media employs, we can create formative profiles of learners. These, in turn, 

can empower (human) educators and trainers to make better decisions about 

instruction and help them tailor guidance in ways that would otherwise be impossible. We can similarly empower learners, administrators, systems teams, 

content and experience providers, and a whole host of constituents across the 

learning ecosystem with information relevant to improving, and making more 

meaningful, their own pieces of the puzzle.

The result of this merging of activity and event-based streaming data, along 

with the subsequent human applications of the knowledge derived from it, 

could offer a path towards something of a Golden Age for formative assessment—but this Golden Age doesn’t stand a chance if either the technologies 

or instructional strategies employed fail to attend to the matter of scale.

A challenge, therefore, is to reconceptualize assessment from the point of view 

of learning at scale, as opposed to its traditional analogs found in “un-scaled” 

contexts. Computational learning analytics are core to this conversation. Any 

notion of assessment in the digital world must consider the impacts of scalable, continuous, multifactor data. The future of assessment is analytics.

The time is ripe to investigate new models of assessment that take advantage of advancements in cloud services, streaming data architectures, APIs, 

and a new generation of web-based applications. By applying these tools to 
learning, we can surface meaningful patterns previously too obscure, if not overly 

complex, to act upon. 

This prompts us to consider a wholly 

new human-machine model of assessment for the digital age, not simply a 

digitized version of analog assessment 

at scale. For example, it’s routinely noted 

that automation can maximize the efficiency and timeliness of tactical learning 

interventions (e.g., micro- and macro-adaptations). However, automation can also 

help identify those interventions best 

addressed by a human—who, in a webscale context, needn’t be a single preassigned instructor. Rather, learners could 

be served by a distributed network of potential teachers and mentors, and based upon various automated analyses, the 

system could recommend the optimum (human) learning facilitators for different situations (including, potentially, the individual learners, themselves). 

In this way, we enable widespread distribution, not just of individual instruction, but of the entire ecosystem—including its human capital. 

This suggests a new paradigm for learning and assessment, one where 

machines and humans complement one another—a symbiotic system. 

In addition to automating the collection and analysis of data, it’s possible to 

automate its visualization via learning analytics dashboards.9 The idea proposed here is to fully leverage activity and event-based data to provide 360° 

views of learners in real-time. 

One key topic of focus for future 

learning is data analytics. We 

currently use very fanaticized or 

ritualized measures, like time on 

task or changes in knowledge 

in a single area. How do we get 

that mind reset to the galactic 

view of learning?

Elliot Masie

Founder, The MASIE Center
Analytics and Visualization | 171 

These dashboards could readily visualize concepts, such as:

• Frequency, time, and duration of individual, cohort, global activities

• Frequency, time, and duration of engagement with specific content

• Outliers among actors or content, in terms of level or type of activity

• Relations between actors, such as shown by a directed network graph

• Individual or cohort performance aligned to KPIs or business goals

• Recommended interventions to support learners’ progress 

• Trends among content engagement activities and learning pathways 

• Outliers among actors, in terms of similarity or dissimilarity of content 

usage, types of engagement, or times and durations compared to a 

cohort or global group

Further, in future iterations—once enough relevant data points have accumulated—machine-learning algorithms could help uncover common learning 

trajectories or the factors that make different pathways more or less effective 

for different categories of learners. These sorts of activity patterns could be 

visualized, for example, by using heatmaps to depict which instructional content successful learners spend the most time with or by using polar graphs to 

indicate the behavioral trends exhibited by learners of different aptitudes as 

they interact with a given learning object (e.g., fast-forwarding through parts 

of a video or abandoning a simulation at certain times). For learners, dashboards can help individuals visualize their own gaps and proficiencies, and 

help them take steps towards managing their own learning.10 For administrators, these algorithms could help forecast enterprise-level planning issues, 

inform education and workforce strategic-level decisions, or suggest incremental improvements for the system itself. Ultimately, a “mission control” 

dashboard comprised of modular data cards—each representing different insights and each providing ways to query the data—could be available to each 

“persona” with in the learning ecosystem, including for learners, instructors, 

content developers, administrators, and policymakers. 
172 | Modernizing Learning

IMPLEMENTATION 

RECOMMENDATIONS

Because the field of streaming data and the capabilities it supports are still 

emerging, we expect future innovations to eclipse the suggestions made in 

this chapter. But in terms of a starting point, the section below outlines practical implementation steps to consider when looking to bring this new wave of 

digital transformation to bear.

1. Needs Analysis and Data Assessment

As in most processes, the first step involves problem framing. Determine 

what outcome data are needed and what types, quality, and amounts of data 

are already available. Ask questions to identify factors, such as the state of 

current and historical data assets and data-producing sources, both within 

and external to the current system as well as the status of currently accessible 

It’s almost cliché to say, “learning is a journey.” But when most people use 

this platitude, it’s possible they really mean, “Sure, you’re going to find 

out new things in the future, but this class ends in three weeks and you’d 

better finish this learning by that time.” An assumption of the learning 

ecosystem concept, and the closely related philosophy of personalized 

lifelong learning, is a shift away from output-focused, time-based 

learning—characterized by high-stakes summative tests—and instead 

towards more a process-focused outlook on learning—supported by a 

steady stream of formative assessments. This represents a fundamental 

shift for learning and assessment—away from discrete mathematics and 

towards continuous equations.
Analytics and Visualization | 173 

data, including the shape of the data model and where, when, and how it was 

delivered and stored. Also document the status of the current data architecture and system design, and information about its previous incarnations (if 

any), including its historical levels of use and expectations for the scale to be 

served by the new system. Finally, as appropriate for any project, catalog the 

known risks and protocols (such as privacy, data governance, and security); 

the objectives and goals of digital transformation, so as to provide guidance 

on what new data sources will need to be integrated into the system to provide 

desired metrics and insights; and the timeline, scope, and budget, in order to 

best enable (what will most often be) a phased approach to implementation of 

the complete system.

2. Data and Visualization Designs

Practitioners often make mistakes during the data design phase that only surface later in the process. To limit exposure to errors, poor design, and the accumulation of technical debt, it’s useful to work backwards. Begin by laying 

out key questions; simultaneously, it’s helpful to draw prospective visualizations for these questions, particularly in collaboration with their respective 

end-users. Next, identify performance indicators that provide insights to those 

questions, and determine what data sources may best inform these performance indicators (whether or not those data sources currently exist). Then 

design the “ideal” data model, incorporating the hypothetical data sources 

previously identified; take care to deliberately consider how different data 

sources may react to one another and how data from multiple sources may be 

needed to inform recommended actions—possibly including actions taken by 

other providers within the larger ecosystem. Once this optimum data model is 

developed, look for available data sources to fill, or at least partially address, 

its proposed components; also, consider potential limitations or access issues 

with these data. Finally, revisit and tailor the visualization mock-ups to the 

final data model. 
174 | Modernizing Learning

There are a variety of ways to visualize data. Key factors to consider include 

the velocity of data streaming through the system, the shape of the data, semantic features including both human- and machine-readable attributes, potential correlations or potential false flags among the data, and the metrics 

necessary to demonstrate progress towards key performance indicators. Additionally, strive to design visualizations to be as transparent as possible, to 

help end-users build appropriate levels of trust in the algorithms and make 

informed decisions based upon the analyses they depict.

Related concerns, such as privacy or access to data streams, should also be 

considered during the design phase. Adhering to industry or organizational 

policies, such as learner privacy rules, may limit the ability to create a robust 

profiles. Sparse data may impede the ability to generate analytics using many 

established big-data methods. It’s important to realistically scope the data 

model and visualizations to a realistic volume and robustness of data, and to 

determine the minimum amounts needed to produce useful insights around 

the identified key indicators.

3. Architecture Development

Once the conceptual data model is designed, the next step is to develop it. 

To achieve the “future learning ecosystem” vision, learning applications need 

to capture and structure (or at least semi-structure) learner activity data, to 

support its aggregation and utility at scale. xAPI is among the most capable and flexible learning-data specifications for this purpose, and it can be 

leveraged alongside other data formats (either non-activity-based or from 

non-learning-domains) to provide a fuller view of learner experience.

When applying the xAPI specification to capture and store data, an xAPI 

Profile should be used, either an off-the-shelf Profile or, if none suffice, then 

a new one created for this system. xAPI Profiles define the accepted terms (or 
Analytics and Visualization | 175 

variables) within a given implementation as well as their uses and semantic 

values. xAPI Profiles create clear, domain-based modeling structures that help 

define the scope of a project, making it easier to deliver human-readable data 

and provide navigable machine-readable data across the ecosystem. Profiles 

can also serve as a useful tool to ensure a clear alignment of business processes and learning objectives to the proposed data model before its implemented. 

Next, choices will have to be made regarding the integration of other data 

sources. Some learning data sources already may be delivered natively in 

xAPI formats. These data will usually be validated and made available by a 

learning record store, a particular kind of datastore defined by the xAPI specification. Standardized data and APIs, such as those offered by xAPI, make 

data aggregation relatively easy. However, there may be other learning data or 

non-learning activity (such as on-the-job workflows across web services) that 

aren’t natively structured as xAPI statements. One option is to instrument the 

external source to deliver xAPI data, but this can be difficult when working 

with proprietary third-party software. An alternative is to coerce the data into 

an xAPI format using API methods. However, it won’t make sense to force 

all data into an xAPI-based data model. There’s no reason to transform data 

into xAPI formats if it’s not a good fit. Instead, this heterogeneous data either 

may be modeled to another specification or just passed directly through the 

Kafka Streams processor (described below), where it can be subscribed-to by 

different applications and joined with disparate data in downstream analyses.

Once the native data format and external data streams have been defined, 

they’ll need to be implemented within a streaming data architecture. These 

can follow several models, but we would usually recommend the Kappa 

Architecture 11 as the software architecture pattern for a real-time learning 

ecosystem. This paradigm treats everything as though it were streaming 

data and processes these data into a stream that may be leveraged by various 

microservices. This approach generally makes it easier and more efficient to 

deal with various forms of data, as opposed to creating polyglot solutions and 
176 | Modernizing Learning

maintaining a separate code base for batched and non-streaming data or—in 

the case of xAPI—each non-conformant data source or data type that may pass 

through the system (e.g., from student information systems, HR technologies, 

and legacy databases). In this architectural paradigm, regardless of the nature 

of the source, the data comes into the stream as logged events. This is a huge 

benefit to real-time analytics because from an operational perspective, the 

subscriber to the data stream never has to request that the data producer batch 

the data. Instead, the subscriber always has access to the log and can replay 

the events in the log as necessary to perform operations.

When considering the integration of data from different sources, it’s important to carefully consider how users’ identities will be handled. Identity management should be organized so that everything is kept orthogonal. When 

designing a streaming data architecture, it’s also best to keep identity management and administrative provisioning matters close to the point of ingress; 

so that no data elements slip through unaccounted for.

As mentioned above, streaming architecture may be served by implementing 

an open-source stream processor, such as Apache Kafka.12 Identity management and security applications will need to work in concert with the Kafka implementation. Once set up, data from all sources will flow into Kafka 

to be processed and sent down into a data stream. Data in that stream may 

be subscribed to by any application, such as business intelligence tools or a 

learning record store. The application listens to the stream and pulls out a 

copy of a piece of data when it recognizes it. Microservices provide these 

capabilities and help to automate data flow. Ideally, data will automatically 

go where they’re supposed to; so that they can be analyzed, visualized, aggregated, verified, or so on, by various subscribed applications. Meanwhile, 

all of the original data passing through the stream eventually ends in a data 

lake, where it may be accessed and queried manually or via machine means 

later, as necessary. And, as mentioned above, all of the data is now available 

as logged events—which provides considerable operational efficiencies. This 
Analytics and Visualization | 177 

stream processor model is in contrast to point-to-point architectures, where 

all applications within a learning ecosystem attempt to connect with one another, to exchange data bilaterally. Point-to-point architectures scale poorly.

Finally, a word of caution: Generally speaking, especially in enterprise scale 

implementations, we would refrain from using third-party SaaS integration 

solutions. They add cost and licensing complications, may affect throughput, 

and can be a burden in the event things break or the third-party ceases to provide the services. Third-party services can also create unanticipated security 

challenges. In our personal experience, it’s almost always better to build natively or to provide data translation services of your own design.

The system for military promotion is well known, albeit difficult 

to use. Whenever you get to a point of appraising someone’s 

abilities, people become serious about what the metric is and 

how it is being collected. They want to know, “How do I achieve 

the metric?” They will focus their minds on the details of metric 

collection and if they don’t get promoted, they will expect a debrief 

that provides clarity on why they missed the mark. They want to 

know with credibility; it can’t just be a machine saying you just didn’t 

get promoted/recommended. This is all part of treating them right. 

We will always need humans in the loop when dealing with human 

performance assessment.

James Robb

Rear Admiral, U.S. Navy (Ret.)

President, the National Training and Simulation Association
178 | Modernizing Learning

4. Deployment

The fourth implementation step is to choose the deployment environment. 

There are a variety of commercial and specialized cloud architectures that can 

support streaming data. Depending on your needs, you’ll likely be choosing 

between enterprise SaaS and Virtual Private Cloud instances and creating the 

templates to size them appropriately. On-premise deployment is an option, 

though it may greatly increase complexity and cost both during deployment 

and in ongoing maintenance.

Most implementations will follow a general pattern of alpha to beta to production deployment. As part of your alpha deployment, you should identify and 

address issues around privacy and security protocols, identity management 

and administrative provisioning, quality assurance, and continuous integration regimes. You’ll also need to conduct systems testing. During the beta 

implementation and testing period, you’ll stress test the system with real users; take this opportunity to identify bugs as well as ways to improve the user 

experience both for end-users and for those maintaining the system.

5. Production Implementation

Production implementation marks the beginning of a new phase. Depending 

on the volume and consistency of data, machine learning techniques (to potentially include deep learning approaches) can be applied to these real-world data 

flowing through the system. Deep learning processes could unlock a host of 

innovations in this space, including ways to link cognitive machine processes 

with biometric, decision-making, and event-based human learning activities. 

Be warned, however, by their very nature, streaming architectures can be 

fragile. New product development by a vendor may break an endpoint. This 

will have to be fixed in order for the data from that vendor to be able to flow as 
Analytics and Visualization | 179 

it is supposed to. Because other services may be depending on data from that 

vendor in order to process jobs, breaks such as this can cause bottlenecks that 

affect the larger system. For that reason, it’s crucial that stream-processing 

systems be attended to by services teams, 

either locally or via managed services. 

Luckily, making fixes is usually a relatively painless process so long as you’ve done 

your due diligence into the quality of the 

data sources feeding into your system. Further, because most breaks will be caused by 

things like changes to endpoints or reconfigurations of APIs, they’re usually 

well-documented and part of the product plan shared with the team—meaning most breaking changes will be telegraphed well in advance and can be 

planned for.

Just as important to the success of the analytics and data visualizations services within the future learning ecosystem will be scalability and extensibility. Advances in learning tools, web technologies, and AI are likely to alter 

future learning analytics and data visualizations. Likewise social changes in 

behavior, expectations, methods of instruction, access to learning, and preferences among both formal and informal learners will influence the nature 

of the events captured in activity data streams. The technologies deployed to 

serve learning analytics and data visualization objectives, therefore, should 

be as flexible, extensible, and open, as possible. The systems must be built to 

withstand whatever is thrown at them. Dedication to open source standards 

and specifications will aid in meeting this need.

Conclusion

In the end, the quality of insights gleaned from analytics and visualizations 

will be tied to the quality of their data models, the velocity and variety of the 

Some practitioners use 

the acronym FATE when 

discussing Fairness, 

Accountability, Transparency, 

and Ethics in AI
data they employ, and the accuracy of the data’s representations. As the truism 

goes, there are lies, damned lies, and statistics.

13 Statistics, and even more so 

infographics and visualizations, when misapplied can obfuscate the “truth” of 

data. It’s far too easy to make bogus claims, given any data set—particularly 

one as complex, personal, and socially and culturally situated as learning. 

Consequently, the design of the data, application of algorithms, and layout of 

visualizations are of great consequence. Small decisions during these design 

and development phases can lead to significant downstream effects—hopefully positive ones—for learners and other learning stakeholders.

In a learning management system, you can get a gradebook, much 

like analog systems today but available online. But with the advances 

in assessment analytics, you can delve much deeper to gain insight 

into how reliably your questions and tests are measuring what 

they’re supposed to measure. You can determine if your question 

bank is fair, valid, and reliable. You can see in multiple views in a 

dashboard, and you can even see it within, and eventually across, 

education, defense, commercial, and healthcare. 

Stacy Poll

U.S. Public Sector Business Development Manager

Senior Account Manager, Questionmark
Personalization | 181 

CHAPTER 10

PERSONALIZATION

Jeremiah Folsom-Kovarik, Ph.D., Dar-Wei Chen, Ph.D., 

Behrooz Mostafavi, Ph.D., and Michael Freed, Ph.D.

Scientific studies show that personalized learning produces better outcomes 

than static, one-size-fits-all instructional experiences.1 When instruction is 

personalized, learners show improved recall and better near- and far-transfer. Personalized learning can engender deeper understanding as well as hone 

higher-order cognitive skills, such as leadership and adaptive thinking.2

Customized experiences, like those a skillful tutor might craft, are the gold 

standard for learning, but these don’t scale well, given the costs and limited availability of expert teachers and trainers. Computer-assisted instruction 

can mitigate scalability issues, and personalized learning technologies can (at 

least partially) unlock the benefits of one-on-one learning, similar to working 

with a personal mentor.3

Generally speaking, personalized learning technologies attempt to create different experiences for different learners (or for the same learner at different 

points in time). At the simplest level, this might involve customized settings 

based on individuals’ preferences or differentiated instruction, where predetermined categories of learners receive different instructional packages (e.g., 

a system that offers unique pathways for novice and intermediate students). 

More notably, personalized learning can incorporate adaptive mechanisms—

adjusting the learning experience based upon a stream of incoming data. This 

sort of adaptive learning is usually what’s meant when people tout the benefits of personalization. (And, on the whole, this chapter focuses on adaptive 

learning, as well.) 

 

 
Modern technologies increasingly employ 

a spectrum of personalized learning methods to tailor instructional elements, such 

as task selection and tutorial examples,4 to 

better suit individuals’ goals and characteristics, prior experiences, demonstrated 

knowledge and performance, environmental conditions, and/or social contexts. 

For example, as someone gains proficiency, a system may alter the order and frequency of problems, progression through 

the curriculum, and types of feedback 

given. Adaptive learning systems can help 

ensure learners have truly mastered each 

required objective, guiding them through 

activities that exercise and verify each of 

the enabling objectives and progressively 

scaffolding learners to reach mastery. Additionally, as evidence accumulates from 

multiple learners, some systems can use 

data-driven methods to identify trends, 

such as portions of the instructional sequence that are problematic or unintuitive. Other systems can use learners’ behaviors to recommend peer-to-peer and 

team matchmaking, or to identify when a 

student needs human (versus automated) 

feedback. 

Adaptive learning technologies, on average, produce substantially better outcomes than conventional, group-based or 

There are numerous ways 

that consumers are already 

experiencing personalization: 

Coupons printed at grocery 

store cash registers, dynamic 

home pages of e-commerce 

sites based on previous 

purchases and store browsing, 

personal-assistant capabilities, 

recommend restaurants, and 

driving directions to get there. 

Consumers now expect the 

benefits of those experiences 

in other online experiences—

like learning. 

The personalization capabilities 

become a virtual concierge for 

learning experiences, making 

recommendations based on 

a combination of needs and 

interests of the learner.

John Landwehr

Vice President and Public Sector 

Chief Technical Officer, Adobe
Personalization | 183 

non-adaptive learning.5 Adaptive technologies can also make learning more 

efficient, delivering training and education in less time or at lower run-time 

costs. For instance, learners can spend less time reviewing material already 

familiar to them, and they can receive remediation as soon as it’s needed. 

Adaptive systems can also use fewer, or at least shorter, assessments because 

questions can be carefully chosen to maximize their utility in estimating each 

learner’s capabilities. 

LIMITS OF CURRENT PRACTICE

While personalized learning has already been used in various settings, its full 

potential hasn’t been achieved. Part of the problem is that these systems are 

typically designed to meet specific, narrowly focused instructional needs, and 

as such, their benefits tend to be localized. Widespread implementation of idiosyncratic solutions also means that methods of development, evaluation, and 

reporting are nonstandard. This makes the transfer of data between insular systems difficult, which limits the available adaptations and means that instructional episodes are likely to appear disconnected and inconsistent to learners. 

Another challenge is their development costs, which, historically, have averaged around 100–300 hours of time—from highly skilled researchers, software engineers, and subject-matter experts—for each hour of learner interaction.6

 A significant portion of this time is spent building the learning and 

behavioral models that make automated adaptation possible. Considering the 

hundreds of hours of instruction needed for a single domain, along with the 

personnel and time required for its development and testing, the cost of personalization can be high. 

When considering its many benefits as compared to current one-size-fits-all 

practices, however, even expensive adaptive learning offers an overall ad-
184 | Modernizing Learning

vantage. More than that, with the advancement of model-building techniques 

using machine-learning methods and the increasing availability of authoring tools,7

 development is becoming more efficient. Today, a modern system 

could be built with as few as 20–30 expert hours for one hour of instruction.

Overall, this field is fast growing, and new technologies are improving the 

sensitivity, impact, efficiency, and cost-effectiveness of personalized systems 

every day. The following sections outline a general approach to designing and 

deploying personalized learning, with a particular focus on how new adaptive 

learning capabilities will inform the future learning ecosystem. 

DESIGNING PERSONALIZED 

LEARNING

When preparing to implement a personalized learning approach, it’s useful 

to consider which aspects of a learning experience are most impacted by personal differences as well as how instructional elements might be varied in 

response to those differences. The availability of historical, real-time, and 

external data sources will also influence the adaptive system. The next three 

subsections step through high-level considerations for data collection, data 

analysis, and what and how to personalize learning.

Data Sources

Adaptation requires something to adapt to; this could include demographic 

and background information as well as real-time performance, sensor, and 

behavioral (event-based) data from learners. There may also be important 

contributions from information sources outside of the learner directly, such as 

contextual information and instructor inputs.
Personalization | 185 

Relatively static data, such as learner traits and prior experiences, can inform simpler forms of customization, such as role-based differentiation, or 

help seed a new learner profile within a system. Some personal traits shown 

to meaningfully inform personalization include goal orientation, general 

self-efficacy, computer attitudes, and metacognitive abilities. Constitutional 

attributes, such as job title or military rank, can also be useful, in particular, 

because they’re often easy to obtain and can somewhat substitute for past-performance information (if those data aren’t available). Prior knowledge and 

skills, unsurprisingly, are among the most useful historical data for informing 

personalization.8

Learner performance data can include both static data, such as from historical 

test results and portfolio scores, as well as more timely data from quizzes, 

exercises, simulations, and other activities within the given instructional experience. Learner performance can be used to inform complex inferences, 

through methods such as item-response theory or Bayesian knowledge-tracing; simpler approaches, such as comparisons to threshold metrics and population norms, also provide some utility. However, even basic learner-performance data isn’t always easy to collect; sometimes, for instance, individuals 

or organizations may feel threatened by the measurement and recording of 

their scores. Despite this, learner performance data makes a big difference to 

personalization; it’s worth the effort to devise quality measures, collect the 

data, and analyze them carefully.

A new source of data available in some settings comes from sensors, i.e., devices that can measure physical or physiological information about learners 

objectively, removing some ambiguity surrounding the mediators and moderators of their performance. Some specialized sensors, such as galvanic skin 

response and heart-rate variability monitors, can detect learners’ mental and 

emotional states (to an extent). What’s more, specialty hardware isn’t always 

required; low-cost sensors are already built into many devices, such as laptops 

and cell phones, and these can track location, context, gaze direction, pupil 
dilation, and various other 

inputs from voice, gesture, 

and posture cues. Data from 

these low-cost sensors has 

already been used to infer 

states such as stress, boredom, and confusion. Instrumentation within software can even use keyboard 

and mouse inputs, such as 

slower typing or repetitive 

mouse movements, to infer 

learners’ attentiveness, engagement, or irritation, as well as help confirm a 

learner’s identity or uncover signs of cheating.9

Related to both learner performance and sensor data, learner experience data 

refers to event-based data that describe what learners see and do. Compared to 

learner performance data, learner experience captures not just the outcomes 

but all the steps that explain each outcome—the fine-grained, step-by-step 

activities a learner (or other relevant human or machine agents in the setting) 

perform. These could include pausing a video, selecting (and then changing) 

a quiz answer before submitting, or requesting help from an automated tutor. 

Important insights may also come from external sources, outside of the immediate delivery technology or instructional activities. For example, other social 

interactions, such as casual discussion boards in online courses, can be mined 

via natural language processing to learn more about learners’ interests and 

attitudes or to inform social network analyses. Contextual information about 

the learning environment can also be used. For example, time and location 

data can be collected by learners’ sensors and then integrated with external 

weather and map databases to inform real-time context-relevant learning examples. Similarly, logistical considerations may affect learning delivery conDifferent people have different 

strengths so how can we structure the 

training based on those differences? 

How do we deliver the required 

training in less time and have our 

military personnel better prepared 

when they come out the back end?

Thomas Baptiste

Lieutenant General, U.S. Air Force (Ret.)

President and CEO, the National Center for Simulation 
Personalization | 187 

siderations; these could include the digital devices available to that learner 

(e.g., smartphone versus laptop), the number of seats available in a particular 

course, or cost and time constraints. Organizational factors may also inform 

personalization in various ways. As one example, consider how the design 

and delivery of learning might change depending on whether someone is 

completing a training course for workforce compliance reasons, because of 

professional development goals, or out of personal curiosity. 

Another form of external data comes from human observations and inputs, 

including from learners themselves, their peers, instructors, and supervisors. 

For instance, an instructor might input a critique about a student’s persuasive 

writing, or an observer/trainer might score exercise trainees against a performance rubric. A student may even self-report data, or it might come from peer 

evaluations or 360° surveys. (The point is, it’s not necessary for all aspects of 

the future learning ecosystem to be digitized and automated! In fact, this is 

an important area for ongoing research, i.e., how to best integrate technology 

with learning facilitators in a symbiotic—rather than substitutional—way.)

Finally, it’s important to note that learner data is often more useful when it’s 

more robust, more personal, and more contextualized—but these same characteristics also increase privacy concerns. A balance must be carefully struck. 

(Refer to Chapter 8 for a more detailed discussion.)

Data Analyses

Collected data need to be analyzed in some meaningful way, and then the system should use those analyses to make diagnoses, predictions, and adaptation 

decisions. What kinds of decisions can personalized-learning technologies 

make? The most obvious answer is they can estimate learners’ content mastery and then take actions to fill capability gaps and remedy misconceptions. 

People learn at different rates, and some of the most impactful interventions 

a system can make are simply to ensure each learner progresses at his or her 
188 | Modernizing Learning

optimal pace so that all learners reach mastery, without skipping over important subcomponents or suffering through already-known materials. 

Definitionally, mastery describes an estimate of a learner’s competence, the 

true value of which is hidden from observation. Mastery results in observable 

performance, such as correctness and speed of responses.10 Mastery estimates 

can be informed by static data from a learner’s profile or demographic inputs, 

particularly initially. During a learning episode, mastery estimates are best 

informed by newly generated, contextually relevant data. Take care, however, to acknowledge the limitations of mastery estimations. Lucky guesses, 

accidental inputs, trial-and-error, and any number of accidental or intentional errant behaviors can create inaccuracies. Adaptive systems should always be designed with a healthy skepticism around learner mastery data and 

should incorporate ways to verify and mitigate bad estimates. Some ways to 

guard against inaccurate mastery models include via instructor inputs, learner-choice recommendations that override system behaviors, and open learner 

models that let learners view (and sometimes directly or indirectly change) 

their mastery estimates.

In addition to mastery, many individual states and traits impact learning and, 

thus, can be useful targets of analysis. Learner states are malleable features 

that change from moment to moment, while learner traits are more fixed and 

change only over longer periods of time, if at all. Affective states, such as 

frustration or boredom, can reduce individuals’ motivation to learn; physiological states, such as hunger or lack of sleep can also affect learning, both 

by impacting emotions and by moderating cognitive functions. As mentioned 

earlier, personality traits (e.g., goal orientation and general self-efficacy) can 

also provide some insights; additionally, personal characteristics, such as social identity traits or learning goals, may be useful. 

Finally, aggregations of data from many learners over time can inform trend 

analyses or, at sufficient scale, be used to train machine-learning algorithms 

that uncover hidden patterns. At a minimum, collective data can provide some 
Personalization | 189 

general benchmarks, such as average completion time requirements. In more 

sophisticated systems, these data can also improve automated diagnoses and 

adaptation recommendations as well as inform system-wide improvements, 

such as identifying problematic sections in the instruction, optimum learning 

trajectories for different types of learners, and ways to incrementally improve 

the learning interface, content, or delivery.

Adaptations

The next important consideration concerns the kinds of adaptation the system will make. This could involve modifications to many factors, including 

display elements, what and when content is presented, the task sequence, the 

contents of instructional materials, embedded content features (e.g., selection 

of relevant examples), extrinsic content features (e.g., feedback and hints), instructional strategies and tactics, delivery methods, delivery devices, performance standards, learner goals, and various other interactions. These forms 

of adaptation can be expressed, to a greater or lesser extent, at the micro-, 

macro-, and meta-levels. 

I want to be in a position where there’s truly 

personalized learning based on a student’s 

individual needs while at the same time balancing 

it with content-standard expectations. I’d love to 

see opportunities for students to dig in deeper, to 

have responsive educational opportunities.

Nathan Oakley, Ph.D.

Chief Academic Officer

Mississippi Department of Education 
190 | Modernizing Learning

First, the micro-level focuses on task-specific adaptation in response to learner actions within a learning session, problem-solving opportunity, or single 

task. This could be, for example, in the context of one algebra problem or 

within a simulation scenario. Intelligent tutoring systems produce this sort of 

adaptation, albeit usually for fairly constrained purposes and subject areas. 

Intelligent tutoring technologies are becoming commodities, and it’s easy to 

find commercial and open-source options with an internet search. However, 

many of these off-the-shelf tools work best in well-defined subject domains; 

so, while there are several available mathematics tutors, there are fewer for 

writing and fewer still for social and emotional skills. For ill-defined domains 

or specialized material, developing task-specific personalization can be a 

time-consuming effort, requiring extensive inputs from learning, engineering, and subject-matter experts. In those cases, the need for human expertise 

creates a bottleneck to their development 11 and is part of the greater cost of 

personalized learning. 

Second, the macro-level focuses on content-wide adaptation. This could involve choosing the next instructional topic, sequencing instructional blocks 

within a curriculum, asking learners to repeat unmastered concepts, or allowing them to skip previously learned areas. The granularity of a given “topic” 

or “block” can vary widely, but they’re meant to refer to learning episodes 

(rather than their component tasks or larger aggregates). Macro- and micro-level adaptation typically occur within a bounded system, that is, within 

a single application. 

A third type of personalization is emerging at the meta-level. Meta-adaptation is applied across disparate curricula, learning systems, and/or organizational functions. In contrast to the micro- and macro-levels, adaptation at the 

meta-level occurs in system-of-systems environments. Meta-adaptation may 

involve, for instance, choosing which application to use to meet a particular 

learning goal—e.g., whether to train a medic on a new procedure via an online 

simulation, in a blended-learning workshop, or with an on-site mobile training 
In education, we typically focus on 

supply rather than demand, compliance 

rather than growth, academic facts 

rather than context and experience. 

We’re trying to shift this. And we’re 

doing it in such a way to align our 

workforce, K-12, and post-secondary 

sectors. We need different pathways 

because kids are different. So, their 

journeys through our system should 

also be different. 

Ken Wagner, Ph.D.

Education Commissioner 

Rhode Island Department of Education 
192 | Modernizing Learning

team. As this example highlights, different learning systems use distinct, and 

often complementary, approaches.12 Intuitively, each experience might work 

better (or worse) for each learner. Consider, for instance, how professional 

development goals, workshop scheduling logistics, available technologies, urgency of earning the licensure, and risk tolerance of the organization might 

affect the way the hypothetical medic is trained.

Meta-adaptation could also augment learning activities within a given 

system. Imagine that the imaginary medic is learning that new procedure 

through a simulation, and the system diagnoses a gap in an interrelated area, 

say, pharmacology, that’s not explicitly addressed by the current system. In 

this case, the meta-adaptive solution may be able to recommend external 

remediation resources, such as a book chapter, micro-learning refresher, or 

online course. 

Meta-adaptation is a property of modern learning ecosystems, which combine multiple learning systems to let them share data and work together. This 

highlights one reason why it’s important to use standardized protocols, machine-readable data, and well-defined metadata in learning systems. When 

data are shared across systems in standardized ways, it enables the personalization of unified and optimal learning paths 13—at the broader, lifelong learning scale.

Technological Considerations

The design, deployment, and impact of personalization are heavily influenced 

by the technical environment where the system is deployed. This section 

highlights a sample of those considerations.

HARDWARE AND SOFTWARE

Computer-based personalization clearly requires hardware and software. Less 

obviously, these systems may require specialized components, for example, 
Personalization | 193 

extensive and highly secured digital storage for massive amounts of learner data, flexible servers capable of processing online AI algorithms at scale, 

or federated systems that share data across APIs. Similarly, depending upon 

the selected data sources, unique hardware devices may be required, such as 

wearable sensors, environmental beacons, or instructor input tablets. 

BANDWIDTH 

Although personalized-learning technologies can function natively on a client 

application, we imagine most systems will use networked components (and, 

likely, software as a service or SaaS solutions). However, bandwidth limitations may affect some deployments. For example, K–12 schools may need 

to share a limited internet connection across many users, or military units 

afloat or in austere conditions might have long periods without connectivity. 

In such cases, personalized-learning applications should be designed to reduce network usage, function despite slow response times, or operate without 

a connection. Methods for implementing this include batch processing, local 

replication, and caching of expected next steps when possible.

DATA 

Personalized learning requires data. 

Data models can be informed by extant data, whether collected through largescale validation and norming studies, from other applications in a learning 

ecosystem, or from centralized data repositories. A word of caution, however: 

More isn’t always better. It’s important to judge the extent to which previously 

collected data accurately reflects the current population. In precision settings, 

for example, bias has been detected from differences as subtle as the order of 

questions within a test.14 As this highlights, data quality is a key concern—

whether data come from external sources or from system-collected inputs. 

Resilience to error, completeness, objectivity, fairness, timeliness, and consistency (to name a few) are all critical factors for personalization.15
194 | Modernizing Learning

Another key consideration involves storage and processing requirements. 

Some algorithms require data from hundreds or thousands of learners to calibrate a system before it becomes useful. Furthermore, depending on the algorithms, the amount of data generated could dramatically increase memory 

and computational requirements.

MACHINE LEARNING 

Data, at large scales, can be used to train machine-learning algorithms. 

These can, for example, predict which learning paths will work best for 

different types of learners or create a self-improving system that detects 

obsolete content based on usage patterns.16 Furthermore, machine learning 

can automate how personalization works for different populations or uncover 

changing interaction patterns over time. However, machine learning is not a 

silver bullet. It also requires a significant amount of data, which means many 

learners will need to use a system before a machine-learning algorithm is 

ready to fully deploy. Furthermore, many organizations will require ongoing 

validation of personalized-learning interventions, which may involve human 

oversight of an algorithm’s functionality, leading to increased complexity and 

costs. Machine-learning can also suffer from transparency and explainability 

limitations.

TRANSPARENCY AND EXPLAINABILITY

Personalized learning systems should function transparently, that is, in a 

way that allows stakeholders to the see the data, analyses, and reasons for 

actions. Transparency is defined in contrast to black-box technical systems, 

which might perform the same actions but without a way for users to trace the 

system’s decision processes. Ideally, outputs from personalization should be 

available at individual and aggregated levels, and they should allow users to 

drill-down (or drill-through) to reach explanatory detailed views. Data visualizations and dashboards designed for learners, instructors, administrators, 

supervisors, and/or commanders may prove useful here.
Personalization | 195 

Ideally, personalized learning systems should be explainable as well as transparent; this helps stakeholders understand the system’s actions in order to properly evaluate and accept them.17 Consider this distinction: A technical system 

that lacks transparency might contain proprietary functions and black-box 

machine learning; however, opening a window onto these algorithms won’t 

necessarily make their underlying logic or emergent behaviors understandable. Transparency without consideration for end-user explanations can still 

create confusion; hence, personalized-learning systems also provide explanations of the reasons for their estimates and adaptations. As one example, a 

personalized-learning system may use probabilistic math to update estimates 

and combine them into decisions. Studies show that merely displaying probabilities isn’t useful, because even well-educated users may struggle to intuI think it’s an interesting and exciting future, if there are 

multiple paths of developing competencies and ultimately 

getting the job you want. For way too long, we’ve had this 

single path to get to success. It’s often served more as a filter 

than as a capability-building mechanism. 

Shantanu Sinha

Director, Product Management, Google; Former Founding 

President and Chief Operations Officer, Khan Academy
196 | Modernizing Learning

itively understand them. Instead, explainable systems might provide natural 

language descriptions and evidence for their decisions in familiar terminology. Recent research is also investigating how to construct explanations after 

the fact for those complex systems that don’t normally explain themselves.18

The output of transparent and explainable systems should be actionable for 

the end-users. Systems shouldn’t simply output data; they should help make 

data meaningful to the stakeholders who use it—e.g., as open learner models, 

instructor dashboards, or visualizations designed for administrators and organizational decision-makers.19 And when these systems incorporate good explainability, users are more likely to trust them, understand their limitations, 

take actions in response to system recommendations, and continue using the 

systems over time.

CONTROL 

Transparent and explainable systems let users see why and how an application 

works, but what if those stakeholders want to control some of its functions? 

Systems can allow learners, instructors, and other human stakeholders to influence their estimations and/or actions. This sort of human-machine teaming 

is an ongoing area of research.20 Ideally, learning stakeholders should be able 

to retain the kinds of control they want while they offload tasks to complementary technology that augments them with faster processing of large or 

detailed data.21

USABILITY

Finally, to effectively implement personalized learning, usability and user acceptance are critical performance metrics. Usability stakeholders include not 

only learners, instructors, and administrators, but also the instructional designers who plan and implement personalized learning, system engineers who 

need to monitor data models and adaptation algorithms, and even developers 

of other applications within a learning ecosystem. 
Personalization | 197 

BUILDING EFFECTIVE 

PERSONALIZED LEARNING

Ultimately, the purpose of personalization is to help individuals achieve learning objectives more effectively and efficiently. But how do we determine how 

well a particular system—its data, analyses, and adaptive interventions—performs? The first question to ask is whether a system is functional, i.e., does 

it give different learners experiences that fit their needs? Can we verify that 

it performs as designed and expected? It’s useful to break these evaluation 

factors down into several categories. For instance, how does the system—as 

a software application—perform? Consider elements such as: the amount of 

work done by a user without help from the system, time-related information 

about the work processes, information related to the accuracy of underlying 

models, and the behavior of users in interacting with the system. It’s also 

useful to evaluate the content within the application, for instance the extent to 

which a system produces recommendations for every possible target learning 

outcome, quality of the instructional “catalog” the system draws from, and 

quality of instructional interventions made.

The quality of instructional interventions can be measured in many dimensions, including the breadth, sensitivity, and completeness of different learning interventions, the number of unique recommendations the system makes 

in proportion to the entire catalog, or how often the system recommends the 

same few popular results to different users. Relatedly, questions to ask include: 

What were the differences in support and feedback between learners? What 

was the difference in the order of progression from one topic to the next? Did 

students get stuck at any point during task- and content-specific operations 

and, if so, where? How often did trainees drop out of training or pause it? 

Were there indicators of off-task behaviors or attempts to game the system?
198 | Modernizing Learning

The next question to ask is whether the system is effective, i.e., does it make 

adaptations that enhance the outcomes of the learning experience? Can we 

validate that it achieves the broader outcomes we’re seeking? Most obviously, 

these may include training effectiveness and efficiency measures, i.e., did the 

system produce better topic mastery or faster speeds of completion versus 

other methods? More than that, other outcomes may be equally desirable, 

such as increasing retention rates, improving motivation, fostering certain attitudes, or encouraging social interactions.

Finally, there are practical considerations for evaluating a personalized learning system: What does it cost? How much time and how many resources were 

needed to develop it, and what are the costs of its operation and maintenance? 

Are the components of the system modular, scalable, extensible, and reusable? 

How much data does it collect, and how are those data handled? And, ultimately, is the system providing good return on investment.

CONCLUSION 

Personalization is among the most important ways to achieve effective learning outcomes, and computer-assisted personalization can bring this benefit to 

more learners. The field of learning science has advanced our understanding 

of what and how to adapt learning (through decades of research in educational 

theory and cognitive science), and innovations in technology are improving 

our ability to implement these methods, efficiently and effectively at scale. 

The promise of personalizing learning will be realized when individual components and learning systems work together, as a system-of-systems, sharing 

data and optimizing learner trajectories across longitudinal and diverse experiences. The potential for learning personalization is immense, and researchers and educators are just beginning to explore the possibilities.
One of the most important points we 

heard...is the need for a list of principles—

and that technology shouldn’t lead. It should 

be about technology enabling our systems 

to attain equitable and ethical outcomes. 

Amber Garrison Duncan, Ph.D.

Strategy Director, Lumina Foundation

Learning 

Science
If we can assess both formal and informal learning experiences 

in our students, what might you use that data to inform? 

Teamwork skills? Classroom management? Extracurricular skills? As 

a teacher, I have little control over the design of statewide summative 

assessments, but I could put those skills and others, like thinking and 

problem-solving, at the forefront. Developing these skills is also an 

objective. Students are always aware that they’re honing a number of 

skills that will help them be successful outside of our class.

Kimberly Eckert

Teacher, Brusly High School; Louisiana State Teacher of the Year 2018
Assessment and Feedback | 203 

CHAPTER 11

ASSESSMENT AND 

FEEDBACK

Debra Abbott, Ph.D.1

The future learning ecosystem will change the management and processing 

of learners’ data across systems, communities, and time. As new analytics 

capabilities evolve, they will catalyze change in several ways: by increasing 

the level of insight into how learners develop over longer periods of time, by 

enhancing the ability of instructors to make teaching more responsive and 

adaptive, and by recommending experiences and learning pathways designed 

to meet the needs of individuals. However, new technologies won’t enhance 

learning if they’re applied without purpose. The current system too often elicits an abundance of learner performance data without making effective use 

of it. And, too often, other factors essential to learning—such as motivation 

and long-term goals—are ignored, or learners receive feedback that’s neither 

useful or actionable and, hence, quickly forgotten. This chapter lays out an updated framework for assessment and simultaneously emphasizes the importance of analyzing the intent behind assessment activities, reforms available 

through improvement of formative feedback, and affordances required in a 

technology-enabled system of assessment.

Background and the Limits of Current Practice

As technology rapidly transforms training and education, the choices regarding learning assessment have become more confusing for instructors and 

riskier for education and training program managers, who must navigate a 

 

 
204 | Modernizing Learning

bewildering forest of accountability-oriented data on programs, classrooms, 

and outcomes. Unfortunately, such recordkeeping often takes on a life of 

its own as data, originally connected to specific learning goals, becomes an 

enterprise asset to be gathered, maintained, and reported for its own sake. 

Additionally, developments in research, paradigm shifts in assessment, and 

changes in the landscape of learning have essentially rewritten the rules of the 

game. The professional development of education and training stakeholders, 

however, has not kept pace with these changes, and this has frequently led 

teachers, instructional designers, and others to operate under outdated models of assessment—where assessments are primarily summative, quantitative, 

and focused on decontextualized snapshots of learner performance. 

Valerie Shute and Matthew Ventura sum up the consequences of this state of 

affairs: 

Many of today’s classroom assessments don’t support deep learning or 

the acquisition of complex competencies. Current classroom assessments (referred to as “assessments of learning”) are typically designed 

to judge a student (or group of students) at a single point in time, without 

providing diagnostic support to students or diagnostic information to 

teachers.2

We need formative rather than just summative assessments; 

we need to push and melt these technology tools to do a better 

job and use the analytics in linear or nodal fashion. The goal is to 

understand individual aspects for education that ultimately enable 

us to give them a better education than they’ve ever had. 

Keith Osburn, Ed.D.

Associate Superintendent, Georgia Virtual Learning

Georgia Department of Education
Assessment and Feedback | 205 

Often, instead of providing a clear path towards a solution, the advance of 

technologies—including algorithms that personalize learning, new delivery 

platforms, and a host of other rapidly expanding choices—muddy the waters. 

There’s a risk that novelty effects or the complexity of some learning technologies mask flaws in design. Learning science informed by research-based 

principles can help. Whether learning takes place in virtual reality or a classroom seminar, the history, principles, and processes of learning science constitute a valuable toolkit for learning ecosystem designers and developers.

PRECONDITIONS FOR 

ASSESSMENT: THE ESSENTIALS

In Visible Learning, John Hattie names two elements as “essential to learning”: 

(1) a challenge for the learner and (2) feedback.3 Similarly, both factors serve 

as a foundation, or as the minimum requirements for, assessment. If challenge 

is insufficient, neural connections are neither strengthened nor altered in a 

learner’s brain, and if useful feedback isn’t present, the learner is acting blindly, unable to relate her performance to either current or future learning goals.

New-age learning analytics have moved the needle considerably as they allow 

for continuous, real-time monitoring of performance and can present upto-date dashboards to stakeholders. This is a far cry from assessment in the 

age of our grandparents. For most of the 20th century, a “factory model” of 

training and education prevailed and, with it, an assumption that teaching is a 

transmission process, with learners on the receiving end. The goal was to fill 

everyone’s head with knowledge and deliver a uniform product, the graduated 

student, to society. Instructors were told that a period of teaching needed to be 

followed by an assessment, followed by another period of teaching and another 

assessment, ad infinitum until a program of instruction ended. Assessment 
206 | Modernizing Learning

was thought to be an on-again-off-again occurrence situated in this linear 

process. 

Many decades ago, the design of assessments wasn’t considered particularly important, since they were like accessory events to the primary focus of 

teaching and learning. Paper-based activities such as tests and essays prevailed—except in special settings, such as art, speech, or physical education 

where performance mattered. And in this environment, it was assumed that 

students could receive feedback in the same manner as they did any other 

sort of information: Many instructors never thought twice about red-inking 

students’ papers or telling them harshly that they lacked writing or thinking 

aptitudes—a practice that would lead some learners to a state of learned-helplessness. Conversely, it was acceptable to praise high-performing students’ 

abilities and intellect, often undermining their growth mindsets and instilling 

a false sense of the level of effort required to learn.

Nowadays, most classrooms, whether they exist within in a company, at a 

military base, or on a computer screen, experience at least some differences 

in assessment practice and the attitudes toward it. Assessment state-of-the-art 

This was once the forefront of assessment!
Assessment and Feedback | 207 

in many places can be described (albeit cautiously) as more learner-centered 

than in the past. These changes may be ascribed to the impact of constructivist learning theories and methods such as active-learning and learner-centered 

design. Improved practices and attitudes have also resulted from numerous 

assessment movements that have achieved notice, if not popularity, over the 

last few decades: authentic assessment, performance assessment, alternative 

assessment, formative assessment, portfolio assessment, embedded formative 

assessment, longitudinal assessment, and assessment for learning (which is 

distinct from assessment of learning).

So, in this new age must we always be assessing? What’s best for learners? 

For now as well as in the foreseeable future, some forms of student work 

and performance will be prioritized above others as the significance of any 

given assessment is socially constructed. For example, in adult education, assessments that mirror authentic types of workplace tasks may be more greatly valued and better serve to articulate learning objectives. It’s important to 

recognize that not all actions or learning artifacts individuals produce will 

have equal value relative to learning goals, program objectives, or learning 

outcomes. Part of the challenge, therefore, lies not only in designing and delivering effective assessments but also in prioritizing their applications and in 

considering their broader roles within the learning ecosystem.

Building upon the progress made to date, assessment in the future must continue to empower education and training stakeholders. Understanding assessment is no small feat, but to start, it’s useful to clarify the true purpose for 

systems of assessment, including for singular high-stakes assessments, and to 

encourage a mindset shift away from 20th century preconceptions that couple valid measurement almost exclusively with summative measures such as 

tests, papers, quizzes, and the like. It’s also useful to become versed in developments arising from research in formative assessment, as well as its close 

cousin, feedback—which has a symbiotic relationship with learning. Finally, 

as we embrace a more technology-centric approach to learning, it’s useful 
208 | Modernizing Learning

to consider the affordances that learners will require in environments where 

assessment may occur in real-time and continuously. 

Purpose of Assessment

The ostensible reason for assessing learning is to aid decision-making. However, assessments are quite often used to hold an entity or a person accountable for meeting predefined criteria or achieving certain outcomes. As such, 

student learning outcomes are almost always written to reflect some level of 

desired change, such as the desire for increased performance on a standardized test; advancement in subject-area ability; or achievement of a curriculum 

objective defined by a certification entity, a state-level department of education, or an employer. In a classroom, quizzes may be used to hold students 

accountable for studying; at an organizational-level, standardized tests may 

hold school districts accountable for collective performance, and in workforce 

contexts, assessments might be used to assign accountability for adhering to 

regulations by verifying employees have completed compliance training. 

However, despite their practical utility, these sorts of accountability assessments of learning often have less utility for learning. Susan Hatfield, in Kansas State University’s long-running practitioner paper-series on improving 

learning in higher education, highlighted the distinction: 

The best way to determine the reason for doing the assessment is by 

examining the focus of the plan. Is the focus simply on collecting data? 

Or is the focus on using data to improve student learning? Assessment 

plans designed to appease others generally involve a lot of data collection but are rarely put to meaningful use. Plans that focus on student 

learning connect collected data to potential courses of action.4

The potential “courses of action” Hatfield mentions might occur at various 

conceptual levels, from more immediate task- or course-focused perspectives 

to organizational and lifelong learning considerations. In other words, whether 
Assessment and Feedback | 209 

Recently I took a test for Google for 

Education Certification. I thought it would 

be a typical test so I crammed... how I 

always tested. That’s usually how you have 

to prepare for nearly every standardized 

test I’ve ever taken. However, when 

I started the test I realized it’s not a 

crammable test! It’s all practice based so I 

actually learned while I took it. I had all the 

tools, it felt like fun, and most of all it was 

meaningful. I hold this experience dearly! 

When I took the Level 2 test in the series, 

I didn’t prepare the same way! I looked at 

problems and thought through scenarios. 

I didn’t even realize the hours passing in 

the test. I wasn’t bogged down. From 

then, I pushed myself to start assessing my 

students in the same way. 

Authenticity is the key. We’re stuck in 

a century that’s long gone. We need to let 

go of that and start encouraging the sort 

of growth mindset that allows students to 

perform and grow and struggle with dignity. 

This is how they’ll feel prepared for life. 

The school of life... it’s all competency 

based. 

Kimberly Eckert

Teacher, Brusly High School

Louisiana State Teacher of the Year 2018
210 | Modernizing Learning

used for accountability or formative learning, assessments that inform 

macro-level decisions should (and generally do) differ from those at microlevels. Macro-level decisions rarely, if ever, are based on a single source of 

the evidence. In an education system, for example, the higher one goes up 

the decision-making tree from classroom, to school, to district, to state, the 

more important it becomes to aggregate the results from multiple, varied 

assessments and make a thoughtful human judgment called an evaluation. 

Evaluation is a complex art, dependent upon accurate data and a capacity 

for judgment derived from knowledgeable instructional practice. Experience 

with effective assessment and instruction is, in fact, the crucible that enables 

individuals to make good evaluative judgments.

As evaluation enters the picture, it widens the aperture about the purpose and 

utility of assessments. Evaluations and other macro-level assessments should 

emphasize measures of effectiveness, that is, meaningful outcomes in terms 

of the impact of learning, such as college admittance rates or improvement in 

job performance. Measures of effectiveness are contrasted against measures 

of performance, or process-focused measures such as a student’s grade-point 

average or how many people completed a training workshop.

This distinction gets to the heart of training and education. Whether individuals are enrolled in a high school composition course, corporate training program, or professional military education seminar, the aim of most formal and 

informal learning is to engender practical competence—competence that’s 

necessarily instantiated in a particular context or environment. For example, if you tell students to achieve a set of general communication outcomes, 

they’re likely to shrug and disengage. However, if you focus those students on 

writing their college entrance essays, corporate work plans, or five-paragraph 

field orders, they’re not only likely to show greater motivation but assessments 

of their abilities are apt to be more authentic, meaningful, and reliable. 

One of the most persistent problems in (adult) training and education stems 

from inadequate understanding of how applied performance—real people 
Assessment and Feedback | 211 

performing real jobs—relates to learning outcomes. Part of the challenge 

lies in understanding the distinctions among competence, competencies, and 

learning outcomes. Competence is a hidden property, inherent to a person, 

team, or organization. It can’t be directly assessed. Competencies, on the other hand, are the clusters of knowledge, skills, attitudes, attributes, and other 

characteristics that attempt to itemize competence. In turn, these competency 

descriptions can be used to articulate job requirements or to inform learning 

outcomes for training and education. (See Chapter 13 for more details on competency-based learning.)

Unfortunately, the more a certain activity requires higher-order cognitive 

and social-emotional competence, such as intrapersonal communication or 

leadership skills, the more difficult its components are to identify, define, 

and assess. Similarly, practical competence requires the interplay of different competencies (such as empathy and communication skills combined with 

subject-matter expertise), which also creates difficulty. This is the classic “iceberg problem.” For example, capabilities your boss thinks are important for 

your job are anchored to its most visible aspects, while you know that your job 

also involves another set of less visible, less well-defined facets. The same is 

true outside of an employment context; those capabilities that prepare someone for life, or to be a good member of our society, are problematic to characterize, delineate, and measure. 

In summary, having a clear view of the purpose of an assessment is the first 

step towards increasing its productive utility. The true purpose should be analyzed: Is the desire to measure the most meaningful, or merely the most convenient, things? Has the system of assessment sufficiently addressed real-world 

competencies, and are the assessments of sufficient breadth and depth to realistically measure them? Finally, what evidence is there that assessment results 

are being used to improve instruction? To the latter question, the results from 

assessments can inform instructional adaptations or organizational decisions, 
212 | Modernizing Learning

and in particular, they can be used to generate valuable feedback to learners, 

teachers, trainers, and organizations. 

WHAT LEARNERS NEED 

FROM ASSESSMENTS 

By their very existence, assessments affect learning. Individuals will change 

their behaviors if they know they’ll be tested, and completing an assessment 

encourages learners to recall and exercise their knowledge and skills. However, substantially more value comes from actually using the evidence collected 

from an assessment. Unfortunately, all too often reams of data are produced 

without any practical application of them.

1. Serviceable Feedback 

The importance of feedback to assessment is vastly underrated, and what constitutes high-quality feedback is often misunderstood. At the most foundational level, quality feedback should enable an instructional system to close 

the loop—to come full-circle—while simultaneously affording learners and 

organizations data that improves their development processes. Royce Sadler 

observed in his widely cited article on formative assessment: 

If the information is simply recorded, passed to a third party who lacks 

either the knowledge or the power to change the outcome, or is too deeply coded (for example, as a summary grade given by the teacher) to lead 

to appropriate action, the control loop cannot be closed and “dangling 

data” substitute for effective feedback.5

The “control loop” in Sadler’s quotation concerns the system-control function, which conceptualizes learning as a loop and feedback as an intervention 
Assessment and Feedback | 213 

used to iteratively close the gap between the actual level and desired level of a 

particular capacity. Assessment results that don’t meaningfully inform some 

aspect of teaching and learning, or that fail to help this progression, are consider “dangling data.”

The term “feedback” is not only vague but itself a misnomer. Assessment 

expert Dylan Wiliam is fond of saying that it more aptly refers to the view 

from the front windshield rather than the rearview. It can refer to performance 

observations or advice, reflective prompts and questions, or other information 

relevant to an individual or group; and it may refer to past, present, or future 

performance.

So, as long as teachers and trainers deliver accurate and relevant feedback, 

what’s the difficulty? It was Sadler 6 who again uncovered the key: There are 

several reasons a learner may have trouble implementing feedback—even if 

it’s of exemplary quality and delivered early enough in a period of instruction 

to be useful. First, the line may be blurry for the learner between the work 

as realized and what was intended; individuals may see potential where instructors may see flawed work. Second, terminology or criteria related to the 

instructional task may not be understood. Third, students may fail to grasp 

tacit knowledge. For example, statements such as “this doesn’t follow logically from what goes before” makes no sense to students who don’t recognize 

the hallmarks of subpar writing structure: It looks fine to them. Last, learners 

often cannot consolidate or apply advice fast enough for learning to stick. To 

be effective providers of feedback, then, teachers and trainers need to better 

understand learners’ own visions of their work, their challenges, and any gaps 

in their learning. Also, learning facilitators would be wise to implement learner self-assessments and peer assessment, since both can go a long way toward 

meeting these needs.

Another model for the creation of more comprehensive and appropriate feedback comes from the work of John Hattie and Helen Timperley.7

 They believe 

that learners need three questions to be answered concerning their perfor-
214 | Modernizing Learning

mance. First, they need information about the performance goal, which answers the question, “Where am I going?” This includes specific and comprehensible success criteria and is referred to as the “feed up” stage. It’s followed 

by the “feedback” stage, which answers the question, “How am I going?” 

Lastly, the question is, “Where to next?” This final stage is called “feed forward,” and it’s probably the most critical juncture for applied learning and 

development. Hattie and Timperley also identify four targets for feedback: 

feedback about the task, about the processing of the task, about self-regulation, and about the self as a person. Their three questions apply to each of 

these categories, and together these twelve targets become a useful, heuristic 

catalog for learner feedback.

2. Evidence-Based Systems

As the characteristics of training and education evolve, enabled by the affordances of the future learning ecosystem concept, new models of assessment 

and feedback can be more readily supported. For example, the proliferation 

of new media devices, wearable sensors, and IoT appliances has correspondingly created an abundance of data. Even without these new hardware tools, 

someone’s activities (say, in a social-media app or on an e-commerce site) can 

be tracked with uncanny precision. By analyzing an individual’s behaviors, as 

revealed by these data, we can start to better understand their attitudes and 

capabilities in ways unimaginable with legacy assessments.

Valerie Shute and colleagues popularized the concept of “stealth assessment,” 

which involves interweaving assessments, informed by evidence-centered 

design principles, directly and invisibly into the fabric of an application environment. For instance, they integrated stealth assessment into a popular 

video game (Plants vs. Zombies 2), and from player interactions could infer 

measures of their problem-solving skills. Shute et al. have recommended this 

approach for applied, competency-based assessments, particularly for certain 
Assessment and Feedback | 215 

ill-defined capabilities otherwise difficult to evaluate, such as persistence, 

creativity, self-efficacy, openness, and teamwork.8

Shute and her colleagues advise against hiding assessments or evaluating individuals without their awareness; rather the term “stealth” refers to the frictionless integration of the measurement, where it’s inherently situated within 

a task rather than an exogenous activity to it. Two other characteristics of 

stealth assessment are that it’s continuous (in contrast to single-point summative testing) and probabilistic (in contrast to the predefined criteria frequently 

used by standardized exams with well-defined correct and incorrect answers).

Stealth assessment can be supported by, or otherwise inform, various data-driven analysis methods. As discussed in Chapter 9 of this volume, learning analytics and educational data mining are two such approaches. Stanford 

University Professor Candace Thille has drawn parallels to the way similar 

technologies have transformed e-commerce: Companies can predict buying 

patterns, use targeted advertising, and employ frequent A/B testing to continuously improve their businesses. Analogous capabilities are being applied to 

Feed Up Feedback Feed Forward

the task

the processing of the task

about self-regulation

about self as a person

FOUR TARGETS FOR FEEDBACK
216 | Modernizing Learning

learning to uncover learner needs by group or type, help personalize learning 

based on individual needs and characteristics, or help predict which individuals are likely to succeed in a given course.9

“The big power of this technology is that we can construct these interactions, 

collect this data on students’ interactions, and use it to drive very powerful 

feedback loops in the learning system.” – Candace Thille 10

However, stealth assessment, learning analytics, and educational data-mining 

can suffer from the “dangling data” problem that Sadler mentioned. In other 

words, it’s possible to estimate someone’s problem-solving ability, let’s say, 

without taking steps to support its improvement or even communicating the 

evaluation results to the learner. Ideally, such data shouldn’t merely be used to 

pass external judgment—the results should be put to work, helping individuals and organizations better meet their goals. Further, this doesn’t just mean 

using the data to inform automated personalization or AI-based adaptation. 

With the growing use of automation, we run the risk of disempowering learners, teachers, and trainers. Despite their enormous potential, automated systems are only as strong as their weakest link—which is very often the user 

interface and user experience. Even today, in arguably simpler times, computer-assisted instruction is fraught with UI/UX design challenges, delivery tool 

mismatches, and assessments that learners perceive as irrelevant. While new 

technologies can enable more frequent and better attuned assessments, these 

may be relatively meaningless if they fail to offer learners and instructors 

sufficient interaction affordances, such as for understanding and making use 

of the assessments, feedback, and subsequent intervention recommendations.

3. Learner Autonomy

Professor Jon Dron, from Athabasca University, posited a theory of trans-
Assessment and Feedback | 217 

actional control, which may be relevant, here. It builds on Michael Moore’s 

well-known theory of transactional distance, which essentially shows that the 

relative “distance” someone feels in an e-learning context is based on the 

amount of interaction and structure in it, rather than the physical separation 

between learners and instructors. 

Dron extended the transactional distance theory to highlight the impact that 

control, or the extent to which choices are made by teachers and learners, is 

the fundamental dynamic of it. The central idea is that flexibility, negotiation 

of control (or “dialogue”), and autonomy all matter a great deal in learning 

contexts.11 The solution isn’t as simple as giving learners (or instructors) full 

autonomy; rather, a thoughtful approach, considerate of control, is needed. As 

Dron explains: 

Most learning transactions tend towards control by either the learner or, 

more often, the teacher. From a learner perspective, being given control 

without the power to utilize it effectively is bad: learners are by definition not sufficiently knowledgeable to be able to make effective decisions about at least some aspects of their learning trajectory. On the other hand, too much teacher control will lead to poorly tailored learning 

experiences and the learner may experience boredom, demotivation, or 

confusion. Dialogue is usually the best solution to the problem, enabling 

a constant negotiation of control so that a learner’s needs are satisfied...

The ideal would be to allow the learner to choose whether and when to 

delegate control at any point in a learning transaction.12

A key takeaway is that learners must be afforded enough autonomy to remain 

engaged, construct their own knowledge and skills, and develop their selfregulation abilities. Striking the right balance between teacher-controlled—or 

AI-controlled—learning versus learner-regulated anarchy is key. As Dron’s 

quotation highlights, systems that favor negotiated control, as much as 

possible, are preferred. In the future learning ecosystem, this prompts us to 

consider how control is distributed across individual and collective learners, 

teachers, and automated systems. 
218 | Modernizing Learning

RECOMMENDATIONS

Given the principles of assessment and feedback, as well as the opportunities 

(and challenges) afforded by new technologies, there are several precepts to 

consider regarding assessment and feedback for the future.

1. FIRST AND FOREMOST, CULTIVATE LEARNER MOTIVATION. As long 

as designers of instruction strive to cultivate learners’ interest and motivation 

with regard to assessment activities, then they are excellent change agents.13

When designed and implemented well, assessments afford rich opportunities 

to develop learners’ concepts, communication skills, subject-area expertise, 

judgments, and abilities. 

2. MAKE ASSESSMENT AND FEEDBACK LEARNER-CENTERED. Learners aren’t merely passive vessels but active participants who seek out useful 

feedback when motivated to do so.14 Educators and trainers must try to view 

assessment through their eyes. Success in assessment is tied to learner engagement (like everything else in education and training). Even in an imaginary 

future, where AI systems have the ability to determine learning priorities, 

content, and sequence, learners will still need to be actively engaged, given 

explicit feedback, and afforded agency over their own learning. 

3. INTERWEAVE ASSESSMENTS THROUGH INSTRUCTION. Instruction and assessment have a truly symbiotic relationship; they’re inextricably 

linked and interactive.15 A variety of types of assessment activities should be 

threaded throughout lessons, modules, and courses of instruction. Even so, 

assessments will always vary in terms of their relative importance, and this 

is as it should be: The extent to which an assessment fulfills the overarching 

objective of instruction represents the degree to which it possesses socially 

constructed value.
Assessment and Feedback | 219 

1. Cultivate learner 

motivation

4. Vary the types of 

data collected

5. Mitigate the 

fluency illusion

6. Plan for curricular 

alignment early on

7. Integrate feedback 

into learning design

8. Plan for systemic 

change

2. Make assessment 

and feedback 

learner-centered

3. Interweave 

assessments through 

instruction
4. VARY THE TYPES OF DATA COLLECTED. A functional system of assessment for learning should be eclectic and incorporate a variety of measures 

such as quantitative, qualitative, estimated, and predictive data types. This 

approach suits the social-science aspect of the measurement objective. Looking ahead, as the vision of an interconnected learning ecosystem comes to 

fruition, assessment evidence from highly varied sources can be collected, 

COMPUTERS AND HUMANS WORKING IN CONCERT: At Arizona State University, 

we have some huge introductory courses, for example, College Algebra with 3000 students. 

About 5 years ago, we created an adaptive general education structure. There are approximately 13 modules for College Algebra, but if students finish early, they can enroll in the 

stretch-version of the course—it doesn’t cost and it gives them credit for the second semester. We use a program called ALEX for instruction, adaptive testing, and adaptive placement 

to determine which courses each student is ready to take (Algebra, Precalculus, or Calculus). 

Sometimes ALEX isn’t perfect; so, perhaps someone ends up in College Algebra and they 

get through the course in the first month—that’s fine! There’s one more aspect of this, but 

it doesn’t scale well: Students are also required to attend class, where they’re coached by 

teaching assistants and take exams. Every week or so, they work in small groups with more 

difficult problems, and they’re scored as a group because collaborative problem-solving is 

an important skill. During those days, it’s active—and the students are loud!—but it helps 

keep the them engaged in the course. Note that these are undergraduates; so, the in-person 

courses also act as a bit of a clinically counseling apparatus. The assistants help mentor, and 

if they find struggling students, they can refer them to a counselor. We have a web-based 

system for the counseling staff, too. We’re really into helping the first-time freshman! 

Courtesy of Kurt VanLehn, Ph.D., Professor, Computing, Informatics, 

and Decision Systems Engineering, Arizona State University
Assessment and Feedback | 221 

stored in persistent learner profiles, and examined in aggregate. This will start 

to shed more light on competencies in situ as well as the interplay among diverse knowledge, skills, attitudes, and other characteristics.

5. MITIGATE THE FLUENCY ILLUSION. Today, our most highly valued 

assessments are usually summative performances (e.g., final exams, formal 

presentations, final projects, professional portfolios) that differ in significant 

ways from practice and study contexts. This discrepancy can create a “fluency 

illusion,” where individuals misjudge their capabilities by thinking that their 

fluency—or ability to remember and apply skills—in practice settings will 

translate to performance scenarios. To mitigate this, learners require opportunities for practice assessments such as pre-tests or trial performances that are 

spaced out in time, occur in a mix of locations or under varying conditions, 

and are sequenced in a special way that mixes problems or content elements 

(referred to by educators and psychologists as “interleaved” practice).16

6. PLAN FOR CURRICULAR ALIGNMENT EARLY ON. Good assessment 

is planned for very early in the instructional design process, and it begins 

by imagining what post-instructional success looks like. Outcomes and 

assessments are like the “bones” of instruction and should be constructed first, 

so that lessons may be structured around them.17 This process is referred to as 

the backwards design of assessment.18 Relegating assessment to an ancillary 

concern typically puts validity at risk by increasing the likelihood of measuring 

achievements that are unrelated to the specific learning objective of interest.

7. INTEGRATE FEEDBACK INTO LEARNING DESIGN. As with assessment, 

feedback approaches should be incorporated early into the instructional design process. While feedback as a dialogue between instructors and learners 

is highly productive, learners can (and often do) obtain feedback from multiple sources. How these multidirectional and distributed feedback loops fit into 

the design of instruction requires planning.19 Explicit and thoughtful efforts 

are needed, particularly as automation becomes more profuse, threatening to 

reduce individuals’ control and transparency of learning. Good feedback de-
222 | Modernizing Learning

sign ensures that learners receive useful information that’s timely, actionable, 

and customized to their needs. 

8. PLAN FOR SYSTEMIC CHANGE. The most challenging aspect of assessment is often the sleuthing necessary to figure out how all the parts fit together: How do the instructional design, delivery, assessments, and measurement 

data collectively tell the story of what a learning experience was like for a 

group or individual, and how can we improve such experiences systematically? Organizationally, there should be a forcing function or mechanism that 

causes the results of assessments to be utilized. However, teachers and trainers, or automated systems, shouldn’t make those decisions alone. Taking action in response to assessment is important, but equally critical is considering 

how to bring learners into that equation. 

Conclusion

It’s strange that we don’t hear more frequent comparisons made between 

the practice of teaching and the practice of medicine. Both require intense 

amounts of skill, professional development, and consistent practice. As assessment expert Dylan Wiliam says: Teachers need professional development 

because the job of teaching is so difficult, so complex, that one lifetime is not 

enough to master it.20 Mastering assessment in teaching is a bit like mastering 

triage skills in the emergency room, in that successful intervention depends 

on successful evaluation of the unique situation of each individual. And, yes, 

because so much of our survival and future success depends on acquiring 

effective training and education, one’s learning needs often are (at least in a 

theoretical sense) as urgent as many health needs. Perhaps because nearly all 

of us have been coaches, trainers to workplace apprentices, or teachers to our 

own children, the instructional process may have lost its mystique somewhere 

along the way. Hopefully, a clearer vision may help us appreciate the mystery, 

regain some enthusiasm, and redefine as well as reimagine assessments to 

work more effectively and purposefully to uplift and motivate our students.
Instructional Strategies for the Future | 223 

CHAPTER 12

INSTRUCTIONAL STRATEGIES 

FOR THE FUTURE

Brenda Bannan, Ph.D., Nada Dabbagh, 

Ph.D., and J.J. Walcutt, Ph.D.

As education and training opportunities become ever more available—on demand, anywhere, anytime, and across our lifespans—individuals increasingly 

experience bursts and waves of disconnected, transitory, and episodic learning. Hence, it’s our challenge, as learning science practitioners, to help learners filter data noise, focus on relevant information, and meaningfully connect 

new learning to past experiences. Towards that end, this chapter provides a 

framework that illustrates a shift in thinking about instructional strategies, refocusing these principles to better support the future learning ecosystem and 

foster connections across learners’ lived experiences. Building on traditional 

instructional strategies shown to be effective in formal learning contexts, we 

propose new approaches that cut across individuals’ learning episodes, potential careers, and lifespans.

Background

For decades, the design of instructional strategies (and learning systems, in 

general) has been largely treated as a micro-level, reductionistic, and linear 

activity—focused on analyzing particular learning outcomes, aligning them 

with suggested instructional strategies, and then delivering instruction in 

straightforward ways to elicit desired responses. However, today, learning occurs in a multidimensional frame, blending formal, nonformal, and informal 

experiences that transcend time, space, medium, and format. The complexity 

 

 
224 | Modernizing Learning

of our lives and diversity of available technologies warrant a shift in learning 

theory, away from standalone learning episodes that push information in a 

singular manner and towards a multipoint, multimodal view where learning 

crosses the boundaries of time, context, delivery methods, and devices. 

Although networked technologies have already made it possible to support 

ubiquitous lifelong learning, our teaching methods and instructional strategies haven’t caught up with these new learning affordances. We’re still designing at the module, course, or program-level, ignoring broader learning 

pathways, and discounting the additive peripheral events learners encounter 

throughout their lives. We need to modernize our conceptualization of “instructional strategies,” and expand these principles to support a more open, 

flexible, and personalized learning ecosystem. We need to create continuous 

and meaningful lifelong learning and find ways to incorporate elements from 

diverse and informal contexts into it. 

Fostering more cohesive, coherent learning will likely involve designing some 

manner of “macro-level instructional arcs” that span a mosaic of individual 

and collaborative learning experiences—meaningfully intersecting different 

events across a lifetime. It will also require us to make better use of multimodal communication tools to help individuals curate information and generate 

knowledge across experiences. This position reflects the connectivist view of 

learning, which perceives knowledge as a network, influenced and aided by 

socialization and technology.1 From this standpoint, knowledge isn’t only contained within an individual or information artifact; it’s also distributed externally through networks of internet technologies and communities, accessible 

via social-communication tools. Learning takes place in these autonomous, diverse, open, interactive, collaborative, and global knowledge systems. Hence, 

recognizing relevant information patterns, constructing new connections, 

and nurturing and maintaining connections become critical skills for achievement. Individual learning opportunities can be (and have been) designed 

with this paradigm in mind; 2 the full solution, however, requires even more. 
Instructional Strategies for the Future | 225 

At IES [the Institute of Education Sciences within the U.S. Department of 

Education], we funded two R&D centers to bridge cognitive science and 

education....This important work was especially useful in demonstrating what the 

research to-date has not addressed. When you take something that has been extensively researched in the lab setting—like self-explanations, making comparisons, or 

studying worked examples—and then implement those principles in the curriculum, 

there are a lot of design decisions need to be made: What kinds of comparisons need 

to be made? And how do you present these ideas on a textbook page? What information do you highlight and how do you highlight it in a textbook? In the lab, these 

types of questions don’t come up. Another issue is, how do you combine learning 

principles like retrieval practice, worked examples, etc.? Historically, we’ve studied 

these principles in isolation, but when you combine them into a year-long learning 

experience, there are many questions about how to do that effectively. 

Erin Higgins, Ph.D.

Program officer within the Institute of Education Sciences

U.S. Department of Education

Limits of Conventional Instructional Design

Traditionally, an instructional designer begins with some given set of criteria 

such as the lesson’s purpose and subject matter, learners’ general characteristics, and likely some logistical constraints. From these, designers extrapolate 

the type (e.g., psychomotor, cognitive, affective) and level of learning outcomes (e.g., remembering and understanding, applying and understanding), 

objectives of the associated assessments (e.g., formative, summative), and other delivery factors (e.g., course schedule, perhaps). They break the goals into 

objectives, the objectives into tasks, and then select some set of instructional 

interventions to help learners master each component. They continue working in this linear fashion—breaking down the plans into smaller and smaller 

parts, and carefully considering the content, delivery, and learner activities for 

each. This is known as “backwards design.” 3
226 | Modernizing Learning

The traditional approach to designing instruction generally assumes a given 

target—a particular individual or cohort—as well as a specific setting and 

general set of conditions. It focuses on determining the appropriate configuration of instructional interventions in insular and finite curricular units, such 

as a course or training program. However, as we envision learning across 

lifetimes, this model no longer suffices. In the future, we need instructional 

design that encompasses diverse learning experiences, media, populations, 

and contexts—many of which will fall outside the instructional designer’s 

purview. In other words, we need an updated approach that: 

• Facilitates learning as a gestalt, derived from the collective 

sum of all learning events and experiences; 

• Recognizes learning outcomes are increasingly self-directed and 

stitched across different contexts, networks, and communities; and 

• Actively incorporates technology to enable learning—not 

only as an instructional delivery mechanism but also as 

the “glue” to connect learning events to one another.

Consequently, we need a multidimensional model of instructional design that 

integrates traditional micro-level interventions as well as macro-level principles, that considers not only instructor interventions but also learners’ own 

agency, and that actively connects experiences across the crisscrossing landscape of learning.

Strategies and Tactics; Instruction and Learning

Instructional design terminology is used in a hodgepodge of ways.4 We won’t 

attempt to unkink it, but it’s useful to highlight several terms. First, consider 

“instructional strategies” (also frequently called “teaching strategies”). This 

is the most common way to refer to the instructional interventions used by 

teachers, trainers, and instructional designers. In more careful discussions, 

this concept is typically divided into “instructional organizers,” at a more 
Instructional Strategies for the Future | 227 

global level, and “instructional tactics” at a more granular one.5 Exactly where 

the lines are drawn between these levels is a bit fuzzy—and largely irrelevant 

to our discussion. What’s more applicable is the general idea that there are 

instructional design distinctions at different conceptual and granular levels. 

The second important distinction comes in comparing instructional strategies 

to learning strategies. Where instructional strategies are devised and applied 

by learning experts to some planned block of instruction, learning strategies 

are personal methods used to improve one’s own knowledge, skills, and experiences across the range of formal and informal learning. In theory, learning 

strategies and instructional strategies mirror each other. For example, an instructor might design a lecture, provide some illustrative examples, and give 

feedback. Meanwhile, a learner may work to memorize terms, mentally compare-and-contrast new ideas to prior knowledge, and reflect on performance.

In many ways, the distinction between instructional strategies and learning 

strategies is a question of control. As discussed in the previous chapter, transactional control (or the extent to which the learner makes decisions versus 

some external authority, such as the instructor or software) is an important 

factor. As one might expect, control of learning can be handled in different 

ways: Internally by the learner, externally by some structure or authority, or 

insufficiently, without effective support from either internal or external sources. Also, as Jon Dron’s transitional control theory emphasizes, some form of 

negotiated control, in the middle of internal–external control continuum, is 

best.6 Hence, the notable concept here is not only the contrast of instructional 

strategies to learning strategies, but also the potential for their integration—

that is, blending learner-directed and authority-directed strategies together.

One final distinction for the future learning ecosystem is belied by its name. 

Why is it an ecosystem; why not just a regular, old system? An ecosystem, by 

definition, is comprised of interconnected parts, with the behaviors of many 

individual agents affecting one another as well as the environment’s overall holistic pattern. It’s a dynamic system, in the engineering sense, involv-
228 | Modernizing Learning

ing many dispersed, interdependent, interacting elements, and, notably, it’s 

not guided by some top-down, centralized control. Some portions may be 

structured and designed, while others act or interact with their own agency. 

Consequently, for our learning ecosystem, how we understand instructional 

structure and learning is an essential consideration. 

THE EXPANDING 

CONTEXT OF 

FUTURE LEARNING

To advance instructional theory, it’s necessary to expand its design towards a 

modern, longitudinal view of learning, one that facilitates connectivist principles and seeks to amplify outcomes throughout an array of teaching and 

learning situations, across multiple contexts, diverse learning objectives, and 

disparate learning modalities. This section outlines eight principles likely to 

shape the purpose and application of instructional strategies in this complex 

future context.

1. Connect diverse learning experiences 

Explicit in the “ecosystem” concept are the notions of diversity and interconnectivity. Most relevant, here, are the diversity of learning experiences and 

their complex interconnectivity with one other. As humans, all of our experiences naturally affect one another. The question is not simply “how to ensure 

learning episodes are somehow additive,” but rather how to intentionally build 

meaningful and effective connections among learning episodes that advance 

overall learning goals. Even within a relatively constrained setting, like a single course, instructors and instructional designers need to broadly consider 
multiple and varied learning modes and, 

importantly, how to help connect learners’ 

experiences across them. As a simple example, consider a semester-long class that 

incorporates face-to-face seminars, online 

courseware, an additional smartphone app 

used to remediate some students, and informal resources, such as videos or blogs, 

that students find online. Courses that 

blended these sorts of resources are already common. Part of the challenge, however, is gracefully navigating the available 

set of learning-resource options and intentionally integrating them so that they not 

only coexist but also correlate.

This mosaic of learning components, of 

course, is often more complex than this 

example describes. In reality, learning 

experiences span multiple formal and informal events, timespans, and contexts, 

contributing to an ever-evolving trajectory 

of reconfigured and connected experiences, through the lifespan, across multiple 

contexts, and intersecting with varying 

developmental dimensions (such as psychomotor, social, emotional, and cognitive 

learning). An ongoing challenge for learning professionals, then, will be to help 

learners integrate these myriad experiences in thoughtful ways.

The transitions for 

learners from K–12 to 

postsecondary education 

are significant, and if we 

really want to learn about 

accumulated learning, we 

have to have data systems 

that talk to each other. In 

the science standards, 

we’re thinking about the 

progression of learning 

over time. Learners need 

time to digest what they’re 

learning in a deep way.

Heidi Schweingruber, Ph.D.

Director, Board on Science Education, 

National Research Council, U.S. 

National Academies of Sciences, 

Engineering, and Medicine
230 | Modernizing Learning

2. Connect to, and enable outside 

connections from, learning opportunities 

beyond the planned instruction

The preceding example described the integration of learning resources around 

a central unifying core (a single course). This is good, but we need to think 

even broader. In addition to the planned activities designed in or around a 

particular formal learning event, learning professionals need to consider the 

impact of learning activities that take place outside of their direct control or 

even full awareness, such as independent self-directed learning, informal experiences, and other external formal activities (such as courses taught by other 

teachers on different subjects). Too often, teachers and trainers focus solely 

on the activities taking place within their purview, that is, within their formal 

learning episode. This may cause those learning professionals to inadvertently overlook individuals’ prior experiences, concurrent learning activities, or 

the future learning events they might encounter. Linking to prior or external 

learning isn’t new guidance, but the growing availability of well-designed 

informal learning resources combined with interconnected technologies and 

interoperable data make these linkages more achievable and more necessary. 

For the future, it’s important to consider instructional strategies that tie-in to 

these other learning activities and also to create “hooks” in the formal learning materials we create, so that learners or other learning professionals can 

better link our work into their own learning environments.

3. Connect learning across levels of abstraction

When a child learns to read, we first start by teaching sounds and letters; once 

these are learned, we teach words, sentences, punctuation, grammar rules, 

comprehension, and eventually one day maybe professional investigative 

journalism or creative screenwriting. The point is that different capabilities 

emerge from the integration of competencies at a given level of analysis. The 
Instructional Strategies for the Future | 231 

“levels of analysis” concept describes the level of abstraction at which something is affected or evaluated, with the implication that the elements at each 

level relate to one another. Computational neuroscience David Marr has gone 

so far as to say: 

Almost never can a complex system of any kind be understood as a simple extrapolation from the properties of its elementary components...If 

one hopes to achieve a full understanding of a system...then one must be 

prepared to contemplate different levels of description that are linked, 

at least in principle, into a cohesive whole, even if linking the levels in 

complete detail is impractical.7

In the learning domain, considering learning at different abstraction levels 

helps us plan the immediate activities (micro-level interventions), broader but 

still bounded experiences (macro-level interventions), and expansive lifelong 

learning arcs (meta-level interventions). As indicated in the earlier “Strategies and Tactics; Instruction and Learning” section, precisely distinguishing 

where one level ends and another begins is less important than the general 

concept. That concept is that we need to consider is how to better combine 

the micro- and macro-level approaches to designing instruction (the typical 

instructional tactics and strategies experienced designers already use) along 

with new macro-level strategies to create a multidimensional, multilayered 

model that helps learners aggregate and make sense of learning experiences 

across devices, modalities, episodes, and learning dimensions. The idea is to 

support learners beyond the context of a given course or training event, to 

help them integrate these into a more holistic course of study. For instance, a 

university mentor might help a graduate student understand how the differ1. Link learning experiences to each other

2. Link to other outside formal and informal learning 
232 | Modernizing Learning

ent courses, job-study projects, and internships coalesce—creating integrated 

meaning beyond their individual parts. How do we provide similar support, 

but more broadly and outside of a narrow academic context? How do we help 

people extrapolate meaning across otherwise unconnected activities and integrate experiences in ways that expand those activities’ individual values? And 

how do we do this across longitudinal periods—not only during a semester or 

academic program, but at a lifelong learning scale? 

4. Consider the “in between” learning spaces

This multilayered model of learning might appear to simply connect pinpoints 

of learning across time, space, and modality—like a pointillist painting that 

reveals an image from separate daubs of paint. But the concept goes beyond 

that. Unlike paint blotches, which are individually contained and otherwise 

inert, each learning experience is dynamic and complex. Further, the “space” 

between learning experiences—that is, the new value derived from merging or reconceptualizing learning “frames” in response to their integration or 

comparison—differs from the largely additive emergent qualities of a Georges Seurat masterpiece. In other words, the challenge for learning professionals 

is this: How do we capitalize on the abundance and diversity of learning experiences in creative and deeply meaningful ways? Can we do more, for instance, than simply reminding students of prior knowledge or asking working 

3. Connect across

 different levels 

 of abstraction

MICRO MACRO META
Instructional Strategies for the Future | 233 

professionals to consider how new concepts fit into their jobs? Can we build 

something more than the sum of the learning parts? 

Some “levels of analysis” hierarchies include a middle or meso level to refer to 

the connections between the other levels. We’re modifying this concept slightly and using the term meso-level to refer specifically to those interventions 

aimed not merely at linking across experiences but also producing unique 

added value from the correlations. This involves more than just linking across 

time horizons or subject matters, although those are both relevant. It also involves aggregating concepts at a given level so that new and integrated capabilities emerge.

5. Help learners filter overload

As discussed in Chapter 4, cognitive overload poses a serious problem for 

individuals, who can readily become overwhelmed by the sheer amount and 

velocity of information. Learners need new supports that help them filter out 

“noise” and meaningfully integrate the relevant “signals.” If not addressed, 

we run the risk of increasing information acquisition to the detriment of deep 

comprehension and robust knowledge construction. The multilayer, interconnected model we’ve discussed in this section emphasizes this complexity. The 

challenge for learning professionals is to help learners navigate through information overload and to develop the internal cognitive, social, and emotional capabilities needed to self-regulate against it. Some strategies to support 

this have been discussed in prior chapters, including social and emotional 

4. Use the interplay of experiences to create something new

MESO
234 | Modernizing Learning

competencies (Chapter 4), self-regulated learning skills (Chapter 15), and social learning supports (Chapter 14). Mentoring learners in these areas can 

help, as can specifically teaching techniques for managing overload including 

connectivist skills, curation, and metacognition.

6. Help learners use connectivist learning strategies

Connectivism emphasizes the importance of distributed knowledge and capability. For example, rather than knowing how to bake banana bread, one simply needs to know where to find recipes online, how to select the best video 

tutorials, and which friend to phone when a little extra assistance is needed. 

Navigating through these technical and social networks is a primary skill—a 

critical learning strategy—associated with connectivism. Although the multilayered, interconnected model discussed so far has emphasized instructional

strategies (i.e., those things learning professionals do to help support learning), it’s also important to consider learning strategies. By definition, these 

must come from the learners, themselves; however, learning professionals can 

enhance and support learners’ abilities. Instructors and good instructional design can help learners develop their connectivist learning skills and associated 

self-regulation strategies to help them navigate complex social, cultural, and 

informational networks. 

7. Help learners curate resources and knowledge

Information and communication technologies offer new ways of discovering, 

organizing, and later retrieving information. Often learning instances and 

other information can be digitally captured, processed, aggregated, and stored 

for retrieval across time, contexts, and devices. This notion relates to connectivism, and it highlights the importance of developing related learning strategies (e.g., how to organize and retrieve curated information). Over the last 

decade, personal learning environments have become popular; these online 
Instructional Strategies for the Future | 235 

systems help learners and their teachers manage learning resources. Looking 

ahead, learning professionals will need additional tools and mentorship strategies to continue to support such curation activities across increasingly “noisy” 

and diverse settings.

8. Blend instructor- and learner-controlled strategies

This section has outlined guidance for instructional strategies as well as possible interventions to help develop and activate learners’ own internal learning strategies. This final item highlights that both internal expert-directed 

learning controls as well as learner-directed self-regulatory interventions are 

critical. Over time, individuals should develop the desire and ability to exert 

more independent control. However, many learners need help cultivating their 

self-directed learning abilities, hence a negotiated mix of instructor-controlled 

and learning-controlled approaches is needed. The role of the instructor in 

these new multidimensional contexts, therefore, needs to expand and grow 

in flexibility, shifting to encompass the roles of activator, facilitator, coach, 

mentor, and advisor.8

STRATEGIES FOR 

MEANINGFUL 

FUTURE LEARNING

The prior section outlined eight principles for the application of instructional strategies in the future learning ecosystem context; however, it didn’t describe the strategies, themselves. Hundreds of instructional strategies and, 

likely, thousands of corresponding tactics have been tried and tested. Rather 

than provide a litany of these, we’ve identified five generalizable principles 

of meaningful learning well-suited for instructional strategies in this context. 
236 | Modernizing Learning

These methods will help create active, constructive, cooperative, authentic, 

and intentional learning interventions. 

Meaningful learning is grounded in and driven by epistemological orientations and theoretical foundations that are primarily constructivist, social constructivist, and connectivist in nature. In constructivism, learning is characterized as “constructing” or creating meaning from experience such that 

knowledge comes from our interpretations of our experiences in an environment and emerges in contexts where it’s relevant.9

 In other words, the mind 

filters inputs from an environment or experience to produce its own unique 

reality or understanding. Therein lies the intentional (goal-directed, regulatory), active (manipulative, observant), constructive (articulative, reflective), 

and authentic (complex, contextualized) principles of meaningful learning. In 

social constructivism and connectivism, learning becomes a process of collection, reflection, connection, and publication.10 Therein lies the cooperative 

(collaborative, conversational) principles of meaningful learning. 

Strategies in Application: An EMT Example

Consider an example of a young woman who, upon high school graduation, 

enrolls in an Emergency Medical Technician (EMT) training program. The 

program incorporates multiple courses delivered via didactic instruction and 

labs, followed by integrative in-the-field clinical experiences. Throughout 

the program, her learning is supplemented by various digital tools including 

e-books, practice simulations, and a micro-learning study app. 

At a micro-level, the instructional strategy of scaffolding can be used to create a supportive and responsive environment to help the novice EMT progress 

towards becoming a paramedic. Scaffolding involves assessing what learners 

can do, helping them reflect on what they know, identifying needs and goals, 

providing individualized assistance towards these goals, and offering opportunities for learners to internalize and generalize their learning. In this ex-
Instructional Strategies for the Future | 237 

ample, the instructors might engage the EMT trainee in intentional, goal-directed, and regulatory behaviors to prompt a connection between what she 

learned in the EMT training course and how she can extend the physical and 

cognitive dimensions of EMT training into future paramedic training. 

The instructional strategies of modeling and explaining can also be used 

to help transition learners in their learning trajectories. In modeling and explaining, instructors demonstrate a process while also sharing insights beyond the obvious, such as telling learners about why a task is performed in a 

certain way. In the case of the EMT trainee, her instructors—whether human 

or AI coaches—can model and explain what, how, and why paramedics perform certain procedures while also demonstrating the social and emotional 

aspects involved in these tasks. Modeling and explaining can take place in 

authentic contexts, which helps present the concepts at the appropriate level 

of complexity and portray the interplay of dimensions associated with them. 

For instance, for the EMT example, this could be done in a simulated or real 

ambulatory run. The EMT trainee, in this case, might be asked to articulate, 

Manipulative and Observant

ACTIVE

Articulative 

and Reflective

CONSTRUCTIVE

Collaborative and Conversational

COOPERATIVE

Complex and Contextualized

AUTHENTIC

Goal-directed

and Regulatory

INTENTIONAL

The characteristics of meaningful 

learning, adapted from Howland, 

Jonassen, and Marra (2012)
238 | Modernizing Learning

reflect, and engage in constructive thinking through observation of expert 

performance. She might also be challenged to extend her knowledge beyond 

her comfort zone, such as to consider the next phase of her professional and 

personal development as a future paramedic.

In addressing more macro-level instructional interventions, we can expand 

traditional strategies to incorporate organizational, elaborative, exploratory, 

metacognitive, collaborative, and problem-solving elements across the various dimensions of learning. These macro-level strategies can be connected 

or “threaded” to incorporate higher-level objectives, such as encompassing 

a defined career path or advancing a current professional situation. Each individual’s journey through a lifetime of formal and informal experiences is 

somewhat unique and may incorporate multiple contexts and educational 

events. Hence mapping and organizing a learner’s cohesive transition, with 

the important consideration of “the spaces in-between” (the meso-level of design), as well as the integration of instructional experiences and major life 

events, become important areas of focus for future learning design.

Upon completion of paramedic training, coaching and mentoring can be 

used as crossover instructional strategies to further scaffold learners towards 

the next phase or experience in their lifelong learning trajectory. Coaching 

and mentoring are related. They involve observing learner performance and 

offering assistance to bring it closer to expert performance (coaching), as 

well as acting as role model, advising, and supporting learners in attaining 

goals and in overcoming barriers and challenges (mentoring). As learners set 

goals for real-life situations, coaches and mentors provide support through 

dialogue, with social negotiation, and by engaging learners in actively seeking 

information, researching the issues, and finding solutions to meaningful and 

authentic problems.11

In the EMT example, this means engaging the EMT trainee, who (let’s say) 

is now a paramedic, in authentic (complex, contextualized) and cooperative

(collaborative, conversational) activities to help her think about how to extend 
STRATEGIES FOR MEANINGFUL LEARNING

Instructional strategies such as scaffolding, modeling and explaining, and coaching and 

mentoring can support meaningful learning within and across different levels: 12

COOPERATIVE (collaborative, conversational)

• Enable collaborative and conversational interactions between learners and instructors, 

mentors, tutors, or instructional systems

• Encourage learners to engage in collaborative and conversational activities through 

sharing ideas, listening to each other’s perspectives, and co-constructing knowledge

• Help learners work together in communities to accomplish the task at hand

AUTHENTIC (complex, contextualized)

• Use authentic processes and contextualized examples to present concepts and domain 

knowledge at appropriate levels of complexity 

• Engage learners in authentic activities that are complex and contextualized 

• Encourage learners to actively seek information, research issues, and find solutions to 

meaningful and authentic problems

CONSTRUCTIVE (articulative, reflective)

• Enable active and constructive learning by challenging learners to perform beyond their 

comfort zones 

• Engage learners in active and constructive thinking, for instance, by representing their 

understanding in different ways, using different thought processes, and challenging them 

to develop and defend their own mental models 

• Create opportunities for learners to think constructively while considering experts’ 

performance, articulation, and reflective practice

INTENTIONAL (goal-directed, regulatory)

• Encourage goal-directed and regulatory behavior by keeping learners’ intentions at the 

forefront of the learning task 

• Engage learners in reflective and intentional behavior, encouraging them to analyze their 

actions, compare them to others, and, ultimately, to form expert knowledge and skills 

• Help learners set achievable goals and manage the pursuit of these goals through a 

process of exploration and inquiry

ACTIVE (manipulative, observant)

• Engage learners in active learning through observing the consequences and results of 

their actions and by assessing and evaluating their knowledge 

• Enable learners to consciously think about their observations and actions thereby 

constructing new knowledge and restructuring their understandings accordingly
240 | Modernizing Learning

her physical, cognitive, emotional, and social knowledge of being a paramedic 

further, maybe encouraging her to consider the perspectives of a physician’s 

assistant. This might involve shadowing a physician’s assistant at a hospital, 

observing what they do, and actively considering how her current and emerging medical knowledge and skills as well as her social and emotional competencies (such as bedside manner) might apply. This type of experience allows 

learners to work in authentic settings, and it engages them in collaborative and 

conversational interactions with their coach or mentor as well as with their 

peers. All this enables them to share ideas, listen to each other’s perspectives, 

and co-construct knowledge. As illustrated in this example, the instructional 

strategies of scaffolding, modeling and explaining, and coaching and mentoring can be used as crossover instructional strategies to create meaningful connections that help learners transition across experiences, set lifelong learning 

goals, and achieve those goals across the lifespan. 

Macro-level instructional strategies can inform larger and larger units of instructional and professional development, and adding meta-level structures 

also helps support a lifetime of growth across multiple careers, experiences, and interests. This supports continual expansion of knowledge, multiple 

learning itineraries based on learners’ competencies and interests, and multiple tools for manipulating resources. This includes not only formal learning 

experiences but also informal and life experiences, all intimately connected. 

Viewing learning across the lifespan as a networked and connected ecosystem of experiences opens new opportunities for instructional strategies. Each 

individual may have a different learning trajectory and mosaic of experiences 

threaded together across education and training, major career events, multiple 

careers, and other lifetime activities. Like a puzzle that’s never quite finished, 

learners progressively add to their learning landscapes while also benefiting 

from the integration of the elements within them. The technological advances 

described throughout this volume have created the capacity to provide learners with connected and cohesive learning across their lifespans. 
Instructional Strategies for the Future | 241 

SUMMARY

Instructional strategies can incorporate interventions, such as scaffolding, 

modeling and explaining, and coaching and mentoring, to provide the glue 

that meaningfully supports connected and cohesive experiences across a 

learner’s lifetime. Thinking about the continuum of future learning, we need 

to consider these strategies at multiple levels—not only within a particular 

instructional event or course of study, but across learners’ longitudinal trajectories. Accordingly, a significant challenge for the future is the differentiated 

application of instructional interventions across conceptual areas, learners’ 

developmental phases, content modalities, and levels of abstraction—while 

also considering the impact of composite learning experiences.

Such learning experiences can be implemented using experiential, collaborative, and personalized instructional models that target cognitive, psychomotor, emotional, and social skills across distributed contexts including individual and collaborative activities; these, of course, will also be facilitated 

by a variety of delivery formats, modalities, and technologies. Thus, we must 

consider a new model for how to organize and recommend instructional strategies within this non-linear, lifelong, personalized learning continuum. How 

do we ensure such strategies are coherent to learners and that they improve 

upon (rather than add noise to) the potentially overloaded learning environment? How do we help teachers, trainers, mentors, and automated systems, as 

well as learners themselves, use appropriate strategies in this crowded future 

learning environment? Many other learning science questions persist. However, it’s clear that to realize the full promise of the future learning ecosystem, 

we need to apply considered strategies across it—strategies that combine micro- and macro-level instructional activities with macro-level considerations, 

that identify and support “the spaces in-between” learning episodes at the 

meso-level, and that help learners develop and apply their own learning strategies to navigate the complexity of the world around us.
We need a better system to federate and integrate multiple 

learning experiences throughout a career, across organizational 

units. Transcripts have been used for years, for child to young-adult 

education...but there isn’t a good portable transcript system for 

professionals to securely identify what learning experiences they’ve 

completed and their interests to learn related content areas through 

personalization. As workers move through their organizations and 

careers, the learning record really should follow them more closely 

and accurately.

John Landwehr

Vice President and Public Sector Chief Technical Officer, Adobe
Competency-Based Learning | 243 

CHAPTER 13

COMPETENCY-BASED 

LEARNING 

Matthew Stafford, Ph.D.

Competency-based learning isn’t new. It evolved from the following four innovations: The parsing of learning into specific chunks of skills and knowledge; the creation of learning outcomes to clearly establish levels of mastery; 

assessments that allow learners to demonstrate their mastery; and most recently, a focus on the learner and the learning (outputs) versus a focus on the 

teacher, the curriculum, and the time invested (inputs). 

The first of these advances traces back centuries to the age of guilds and 

apprenticeships. Master craftsmen parsed their specialties into a variety of 

discrete tasks and then trained their apprentices to perform those activities to 

appropriate levels of mastery. Another remnant of the age of guilds is the concept of varying levels of mastery. Aspiring craftsmen started as apprentices 

and advanced through the assorted levels. Only after demonstrating mastery 

of every aspect of the craft, would the tradesman graduate the apprenticeship 

at the full craftsman status.

This parsed-learning approach still exists across widespread training programs today. The military employs this approach with its enlisted personnel, 

training and certifying members on specific tasks. One can also see it in industry and, not surprisingly, in the wide variety of vocational-education programs that prepare students for jobs in industry. These modern settings also 

borrow the performance levels from classic trades-training to indicate progress from novice to master. Ironically, the Air Force—the youngest of the U.S. 

 

 
military branches—even employs old “guild 

language” to label its Airmen’s skill levels: 1 

for helper, 3 for apprentice, 5 for journeyman, and 7 

for craftsman.1

Although born in training, the application of this “levels 

of mastery” approach eventually found its way into education, largely due to 

research into learning theory. In 1956, for instance, Benjamin Bloom posited 

specific levels of mastery within the cognitive domain of learning.

2

 Equipped 

with these descriptions, teachers and instructional designers had consistent 

levels of capability they could target. Well-defined cognitive outcomes mark 

the second of the four innovations that led to competency-based learning. 

What educators needed next were authentic assessments to validate that 

learners had reached the desired levels of mastery. Authentic assessments are 

Historically, a cooper (“barrel-maker”) would train an apprentice on selecting 

trees and forming the individual staves. Equipped with these skills, the apprentice 

would progress to assembling the staves into the barrel form, installing the 

retaining hoops (forged by a fellow craftsman, a blacksmith) and “rounding” 

the barrel’s interior. Next, the apprentice would master the art of finishing the 

barrel, so it would seal. Then there was the complex task of cutting the croze—

the groove into which the head and foot rest, 

installing the head... It was a complex series of 

tasks requiring a variety of specialized tools! 

Even after mastering barrel-making, however, 

an apprentice had more to learn. In addition 

to barrels, coopers also made casts, vats, 

buckets, tubs...a sundry of wooden vessels 

made from individual wooden staves. Only 

after mastering all of the knowledge, tools, 

processes, and specific tasks associated with 

all of the vessels, would the master craftsman 

honor an apprentice with the title “cooper.”

...the unlikely forerunner 

to competency-based 

learning
Competency-Based Learning | 245 

those in which students have to demonstrate meaningful applications of their 

knowledge and skills. A classroom assessment that matches real-world workplace activities, for instance, would be “authentic.”

Authentically assessing performance in the cognitive domain, however, is difficult. The mastery of these less tangible concepts—the ability to formulate an 

effective argument, for instance—is complicated. Demonstrating conceptual 

mastery is even more so. Educators are forced to “sample” desired behaviors 

and then, equipped with these samples, make informed judgments on the levels of mastery students have achieved. Over time, educators have progressed 

in this art, creating performance-based assessments that actually measure levels of mastery, even in “soft skills.” Effective, authentic assessments were the 

third innovation contributing to competency-based learning; however, assessments play a far more important role than simply measuring mastery—they 

actually drive learning.

Contemporary learning theory, based on evidence-informed research and 

neuroscience principles, makes it clear that the best results occur when individuals take responsibility for their learning. Terry Doyle, an accomplished 

learning science author and professor emeritus, is fond of reminding his 

readers, “The one who does the work does the learning.” 3 Assessments can 

empower learning by making learners do the work. For instance, instead of 

devising detailed courses of study, teachers can instead focus on designing 

effective assessments, describing them to students, and then helping learners 

find their own paths to success.

It may sound shocking to some; however, this is how most informal learning 

occurs. Someone buys a lawnmower and turns to YouTube to figure out its 

assembly and how to get it running. Someone else goes online to figure out 

how to change the oil filter in an antique automobile. Gamers have special 

websites to share tips on how to win in their favorite video games. Even those 

who sit and practice the lost art of “reading the manual” are benefiting from 

informal, self-directed learning. In each case, there are no formal classes. 
246 | Modernizing Learning

Learners can spend as much or as little 

time, as necessary, to achieve their learning 

goals. The focus is on reaching the desired 

level of mastery. This learner-centric approach, where the source of learning is less 

important than the mastery of it, catalyzed 

the final innovation contributing to competency-based learning.

This innovation is, arguably, the most revolutionary for contemporary learning professionals: In competency-based learning, 

performance becomes the constant and 

time becomes a variable. This is in direct 

contrast to the traditional approach to training and education, where time is constant. 

In this classical model, learners attend 

classes that run so many days, in programs 

that span so many months... The Carnegie credit-hour system, underpinning 

many U.S. educational programs, exemplifies this time-based approach. Similarly, traditional learning professionals talk of “seat time” or “contact hours.” 

In all cases, time is the constant and performance varies. Some learners sit 

through an entire course of instruction and master all of the objectives, earning an ‘A.’ Others, sitting alongside these top performers the entire time, don’t 

do as well. Performance varies.

In competency-based learning, however, all of the learners work to achieve 

the desired level of mastery. Some will do it the first day. Others will take 

longer. Further, in these outcomes-focused settings, some learners may show 

proficiency even before exposure to the prescribed curriculum. Perhaps they 

already mastered the skills and knowledge in previous experiences. Regardless of the source, if they demonstrate mastery, they earn the credential and 

The classical model of 

education posited learning as 

a somewhat passive pursuit. 

Learners sat and listened to 

lectures or read from books in 

order to memorize facts.
Competency-Based Learning | 247 

advance in their learning. Others will require a complete program of instruction. Again, performance is the constant and time is the variable.

Another aspect of competency-based learning that causes confusion is the 

concept of competencies. There are many different interpretations of this term. 

For some, it refers specifically to a performance and encompasses knowledge, 

skills, abilities, aptitude, and self-concept. Others define competencies far 

more narrowly, describing them in terms of specific skills or specific areas 

of knowledge. Looking at the definitions below, it is easy to see why there’s 

confusion over the term.

A few of the competing definitions of a competency include:

• “...a clearly defined and measurable statement of the knowledge, 

skill, and ability a student has acquired in a designated program,” per 

the Southern Association of Colleges and Schools Commission on 

Colleges.4

• ...a measurable pattern of knowledge, skills, abilities, behaviors, and 

other characteristics that an individual needs to perform work roles 

or occupational functions successfully. Competencies specify the 

In competency-based 

learning, performance 

is key; performance 

standards are held constant 

while time may vary.
Suppose a coach goes into the team assembly room and explains: 

Next Friday, I’m going to put this 48” stick into the ground 

vertically, like this. I’ll expect each of you to jump over it without 

touching it. Those who do so will accompany me to the trackand-field competition the next day.

What would happen? The traditional approach would be to 

build a course that teaches athletes how to jump higher. In this 

instance, however, the coach has turned the learning task over to 

the learners: Those athletes who want to attend the competition 

are going to put a 48” stick into the ground and start practicing 

ways of jumping over it. Some will try a standing broad-jump 

approach (a vertical leap from a stationary position); others will 

try a running jump. Still others might try the famous “Fosbury 

Flop,” the popular high-jumping technique where athletes pass 

over obstacles and land on their backs. Each athlete will approach 

the task in their own way, leveraging their individual strengths so 

they can demonstrate mastery of the assigned task. 

Harry S. Truman once noted, “It is amazing what you can 

accomplish if you do not care who gets the credit.” In essence, 

competency-based learning applies a similar level of humility 

to learning. It’s amazing what learners can 

master if we cease caring how or where 

they learned it and instead focus just on 

the mastery. 
Competency-Based Learning | 249 

‘how’ of performing job tasks, or what the person needs to do the job 

successfully,” per the U.S. Office of Personnel Management.5

• “...observable, measurable pattern of knowledge, skills, abilities, 

behaviors, and other characteristics needed to perform institutional or 

occupational functions successfully,” per the U.S. Air Force.6

• “...a student’s ability to transfer content and skill in and/or across 

content areas,” as defined in the book, Off the Clock, which outlines a 

roadmap to competency-based education.7

Some commonalities exist among these definitions. Like most competency 

definitions, these focus on capabilities that are transferable across a range of 

performance requirements, inherent in which are the notions of functional 

utility and portability. These definitions also highlight knowledge and skills; 

the more holistic definitions, however, look beyond these two facets to also include other capabilities that may impact competence. In their 1993 touchstone 

work on competencies, Competence at Work, Lyle and Signe Spencer listed 

five components of competencies: 8

• Motives – Motives drive, direct, and select behavior towards certain 

actions or goals and away from others

• Traits – A person’s habitual or enduring characteristics

• Self-Concept – A person’s attitudes, values, or self-image

• Knowledge – Information a person has in specific content areas

• Skill – The ability to perform a certain physical or mental task

In their 1999 work, The Art and Science of Competency Models, Anntoinette 

Lucia and Richard Lepsinger offered a slightly different conceptualization.9

Readers can see in the above figure how Lucia and Lepsinger’s approach correlates with Spencer and Spencer’s; however, the pyramid provides better insight into the ways in which some characteristics support others and how, 

when combined, they all manifest in behaviors—i.e., in performance.
250 | Modernizing Learning

Lucia and Lepsinger argued that aptitude and personal characteristics are 

foundational, and while such characteristics may be innate, they can be influenced. Skills and knowledge, of course, are more easily affected; they can be 

imparted through training and education—through development. At the top of 

the pyramid, all of the characteristics manifest in behaviors—in performance. 

There are two categories of competencies within most institutional models, 

core and occupational. Core, or “institutional,” competencies are applicable 

to everyone in the organization. Occupational, or “specialty,” competencies 

are applicable only to certain vocational specialties, positions, or jobs. For 

instance, every employee of a city would need at least some level of proficiency in “teamwork and cooperation” or “initiative,” but only firefighters would 

need to master a firefighting competency.

The applicability of competencies to skills development (like firefighting) is, 

for most, more easily understood than the relationship between competencies 

and cognitive development. This partially explains why competency-based 

learning has been adopted more slowly in education than in training. Within 

TRAITS: Person’s

habitual or enduring

characteristics

SELF-CONCEPT:

Person’s attitudes, 

values or self-image

MOTIVES drive, 

direct, and select 

behavior towards 

certain actions/goals

KNOWLEDGE: Information 

a person has in specific

content areas

SKILL: Ability to

perform a certain

physical or mental task

Behaviors

Skills Knowledge

Aptitude Personal 

Characteristics

The Competency Pyramid per Lucia & Lepsinger, with definitions from Spencer & Spencer
Competency-Based Learning | 251 

the scholarly literature, however, there are many examples of purely cognitive 

competencies, such as analytical thinking, critical thinking, conceptual thinking, diagnostic skill, and commitment to learning, to name a few. Like their 

vocational counterparts, these cognitive competencies are transferable—applicable to a wide variety of educational pursuits. 

Using Competencies to Guide Learning

Competencies serve as broad targets for learning. Readily available to both 

learners and teachers, they serve as a “contract” for learning and describe 

the “finish line” for the accompanying learning experience. When learners 

achieve desired levels of mastery in all assigned competencies, they progress 

to subsequent learning events or complete their programs. 

A well-crafted competency model will typically list competencies, provide 

definitions, and be accompanied by descriptions of proficiency-levels. As positions are created, as workers are hired, or as students move through educational programs, competencies and desired proficiency levels are selected. 

Supervisors, trainers, and faculty members then devise learning experiences 

and assessments to ensure their people can reach and demonstrate the desired levels of learning. Once the desired competency is demonstrated to the 

required level of mastery, the performance is credentialed—captured in a 

certificate, badge, or other record, so there’s a lasting record of this capability. 

Tracking competency development facilitates learning portability. For instance, by credentialing competency completions, learners can prove they 

possess given capabilities, which is useful for meeting the entry criteria of 

future learning experiences or for verifying personal qualification should they 

move to other jobs. Similarly, tracking competencies gives parent organizations more opportunity to effectively employ workers’ skills and knowledge, 

that is, organizations can move workers to those areas where their competence 

is most needed.
252 | Modernizing Learning

Because competency-based learning facilitates precision in tracking and employing developmental investments, it’s popular within industry. It’s particularly valuable to employers hiring new workers. Prior to competency-based 

learning, employers had to assume that prospective employees possessed the 

required attributes, attitudes, skills, and knowledge simply based on their formal learning credentials and the limited time spent in interviews. It’s an unreliable approach. Just because prospective employees have high school diplomas, for instance, offers no guarantee they can perform the arithmetic needed 

to make change at a cash register or even to read its operating instructions! In 

contrast, since competencies aren’t awarded until mastery performance has 

been demonstrated, employers see exactly what their prospective employees 

know and can do. They’ve demonstrated and received credentials for these 

capabilities prior to applying for the job.

Competency-based learning is not yet universally accepted within education, 

but acceptance is growing. One of the more interesting experiments, in this 

vein, is described in Fred Bramante and Rose Colby’s book, Off the Clock: 

Moving Education from Time to Competency.

10 Bramante served as the Chairman of the New Hampshire Board of Education where he faced a high-school 

dropout rate of 20%. To address this, he led the school system to embrace 

competency-based learning, implementing his approach in 2009. By 2011, the 

cumulative dropout rate was 4.68% and still falling. Students were mastering 

the competencies necessary to earn their high school diplomas but doing so in 

nontraditional ways. The key was focusing on the learners and the learning—

the outcomes. This is at the heart of competency-based learning. 

Post-secondary institutions are also gradually embracing competency-based 

learning. Educators have found students enjoy the flexibility and the fact that 

they can progress as quickly through the programs as their efforts and capabilities allow. Western Governors University was an early adopter of competency-based learning; however, the benefits of the approach quickly attracted 

others. The University of Michigan, the University of Wisconsin system, Pur-
Competency-Based Learning | 253 

due University, Northern Arizona University, and Southern New Hampshire 

University, among many others, are offering competency-based programs.

CONCERNS

Minimizing Learning

Perhaps foremost among the competency-based learning detractor arguments 

is the concern that in the rush to impart marketable skills for students, the 

competency-based learning institutions are pushing students into “knowledge-less” versions of the traditional liberal learning. In other words, those 

seeking to discredit competency-based learning claim it’s too utilitarian and 

specific, at the expense of broad-based learning and critical thinking. While 

such programs lead to a skilled and potentially employable workforce, critics 

argue the upward mobility of those workers is limited in terms of perspective 

and their potential to step outside the initial knowledge specializations. This 

argument also implies (or sometimes openly alleges) that the true aims of 

competency-based programs are to expedite program completion and ensure 

high graduate-to-employment statistics, which help sell these programs to future students. Detractors argue competency-based learning institutions are 

Focus on the 

outcomes (learning) 

versus inputs 

(teacher, time)

LEARNERFOCUSED

Have learners

demonstrate their 

mastery in realistic 

applications

AUTHENTIC 

ASSESSMENTS

Ensure learning 

outcomes clearly 

define associated 

levels of mastery

LEVELS OF 

MASTERY

Parse learning 

into specific, 

output-focused 

chunks 

CHUNK 

LEARNING

Considerations for Competency-Based Learning
K–12 SCIENCE STANDARDS: The development of the Next Generation Science 

Standards is an innovative example of bringing research-based learning to scale. The 

National Academy of Science, Engineering, and Medicine developed the Framework 

for K-12 Science Education informed by research on learning that is developmental 

and interweaves science and engineering practices with core ideas and crosscutting 

concepts. Moving from the Framework to standards with clear performance 

expectations came with a hand-off to Achieve, an education nonprofit established in 

1996 by governors and business leaders that works with states to prepare students 

for college and career readiness. 

Achieve reached out to states inviting them to be lead state partners in developing 

the standards to an overwhelmingly positive response, resulting in 26 state partners. 

This was the start of the tag line, “For States, By States.” The collaborative approach 

continues to today with the launch of Achieve’s Science Peer Review Panel to 

enhance the implementation and spread of high-quality lessons aligned with the Next 

Generation Science Standards. Creating a sense of ownership and the providing 

tools to implement. To date, 19 states and the District of Columbia have adopted 

the Standards, and 21 additional states have developed their own standards based 

on the Framework. 

Susan Singer, Ph.D.

Vice President for Academic Affairs and Provost, Rollins College 

www.nextgenscience.org/framework-k-12-science-education
Competency-Based Learning | 255 

creating a new hierarchy within the educated populace: A distinction between 

those who receive a “cheap, fast food-style or ‘good enough’ education from 

those who receive a quality one.” 11 Said another way, the concern is that competency-based learning graduates receive a lower quality of education more 

pointedly focused on vocational development than on habits of the mind, and 

that habits of mind (supposedly in contrast to the attained competencies) are 

more transferable and, ultimately, more valuable beyond entry-level positions. 

It’s a position worth noting.

Quality

Competency-based learning certainly has the potential to be of lower quality. 

The concern is not so much about competency-based learning, in general, but 

in how competency-based learning is operationalized within individual institutions. A vocationally focused program that grants credit for demonstrating 

acceptable levels of supporting, transferable skills, such as speaking, writing, 

critical thinking, and active listening, might indeed produce graduates who 

aren’t on par with their peers from traditional higher-education institutions, 

who’ve had to delve more deeply into these areas as part of their academic 

experiences. Again, however, it depends. It depends on the assessments used 

within the vocationally focused programs and the degree to which the transferable skills were tapped and reinforced during the program. If an institution 

sets its requirements for performance very high, it can force all but those who 

have that level of mastery into its more traditional learning opportunities. 

Employing Competencies Effectively

Perhaps the most important concern raised over competency-based learning 

isn’t actually a rejection of the concept but, rather, concern over how it and 

the resulting competencies are employed. In 2003, George Hollenbeck and 

Morgan McCall questioned why the competency-based approach to executive 
256 | Modernizing Learning

development hadn’t produced better executives. They wrote:

As we begin the 21st century, evidence abounds that executive and leadership development has failed to meet expectations. Unless we change 

our assumptions and think differently about executives and the development process, we will continue to find too few executives to carry out 

corporate strategies, and the competence of those executives available 

will be too often open to question. The “competency model” of the executive, proposing as it does a single set of competencies that account 

for success, must be supplemented with a development model based on 

leadership challenges rather than executive traits and competencies. Executive performance must focus on “what gets done” rather than on one 

way of doing it or on what competencies executives have.12

Hollenbeck and McCall weren’t calling for the rejection of competency-based 

learning but were simply arguing that it’s not sufficient to develop or possess 

individual competencies; instead, it’s how they’re collectively employed that’s 

truly important in terms of occupational success. By way of a metaphor, one 

can produce the perfect brick (the competency), and with a stack of these 

bricks, one can build a cathedral that soars into the sky or a brick outhouse. 

It’s how one employs competencies that matters. This is a valid concern. Application is all important.

U.S. AIR FORCE EXAMPLE

The U.S. Air Force is attempting to integrate competency assessment and 

the credentialing of mastery into its workplace. The Air Force can do this 

because, unlike most learning institutions, it has a continuing relationship with 

the graduates of its education and training programs, which affords unique 

opportunities to ascertain the impact of learning within the work environment. 

The effort is already attracting interest even though it’s yet to be executed. 

Air Force administrators predict the assessment and tracking mechanisms 

will be online by 2022. This use-in-the-workplace example segues to the final 

competency-based learning concern, the attachment to talent management.
Competency-Based Learning | 257 

VISION

A national competency-based system will enable a great deal of flexibility. 

Learners will learn at their own pace. A common characteristic of competency-based learning is that it enables learners to advance as they reach mastery 

because the focus in on outcomes (i.e., mastery of the given competencies) 

and not on the amount of time spent completing a set curriculum. Said another 

way, if a learner can prove mastery of a “communication – speaking” competency, developed earlier in life, she won’t have to sit through a class rehashing 

the material. More than that, articulating competency models helps clarify the 

instructional domains and give structure to learner models—both of which 

aid personalization and automated adaptation of learning. This, in turn, allows learning to be tailored to individuals in multiple ways, not only targeting 

their individual strengths and weaknesses, but also helping to optimize the 

availability of instructional opportunities, plan personal schedules, and so on.

Competency-based learning will also increase resource efficiency. Allowing learners to bypass education and training requirements for competencies 

they’ve already mastered can accelerate individuals through programs. Perhaps they can use this time to pursue other competencies or, instead, they 

might need to employ the competencies they’ve mastered on the job. Either 

way, learners and host institutions only expend resources on competence evaluation and on aiding those learners working towards mastery.

Competency-based learning can help individuals better tailor their planning 

and learning priorities. If, for instance, a learner is working while attempting 

to master a list of specific competencies, he might choose to “front-load” those 

competencies most vital to short-term success on the job. Learners might also 

leverage insight into the competency requirements to choose a learning methodology that they find more effective for themselves or to help inform future career planning. Consider this example: The figure on page 260 shows 
258 | Modernizing Learning

an excerpt from the Department of Energy’s 268-page catalog, Leadership 

Development Seminars July 2013–2014 Edition. It links learning opportunities both within and beyond the government to aid employees seeking to 

master the Executive Core Qualifications (i.e., the competencies specific to 

executive-level leadership for the Federal Senior Executive Service).13 The 

Department of Energy’s catalog includes government-offered courses, courses offered by various universities and private industry organizations, and even 

informal learning opportunities—all mapped to the same set of Executive 

Core Qualifications. Such correlations provide an invaluable tool for motivated learners to build competence in areas specific to their employers’ needs.

Equity and Diversity

A less obvious benefit of competency-based learning is the manner in 

which it may help address inequities within the U.S. populace. The Lumina 

Foundation has researched this, noting that competency-based learning offers 

So much of our education system is based on where you live and how 

much money you have. We’re lacking national equity. But if you learned 

it, it should count. I don’t care where you learned it. Lots of people 

aren’t being served by the current system, but they should be. By 2025, 

60% of Americans will need a postsecondary credential. We currently 

don’t have a system that can produce those results unless we leverage 

every postsecondary learning opportunity and everyone together. 

Amber Garrison Duncan, Ph.D.

Strategy Director, Lumina Foundation
Competency-Based Learning | 259 

a mechanism to get education into the hands—and minds—of disadvantaged 

Americans.14 This includes under- or unemployed adults, adults with some 

college exposure but with no credential, and historically underserved communities. Education has long been credited as a bridge from poverty to prosperity. Competency-based learning expands access to that bridge.

Translation

Competency-based learning is growing as a “currency” for learning. The 

transition from the Carnegie, credit-hour based approach to transcripting education is underway. The Lumina Foundation, an independent, private foundation in Indianapolis, has set for itself Goal 2025, a goal to have 60% of U.S. 

working-age adults possess meaningful and marketable learning credentials 

beyond a high school diploma by 2025.15 To achieve this, the Foundation is 

pressing for, “A new, national system of transparent quality credentials” and 

“a national expansion of competency-based learning...that recognizes measuring academic progress based on demonstrations of what students know 

and can do.” A leader in competency-based learning, Lumina is working 

with learning institutions and governmental organizations across the U.S.—

and they’re not alone. The U.S. Department of Labor, U.S. Department of 

Education, U.S. Office of Personnel Management, and elements of the U.S. 

Department of Defense are also pursuing competencies.

There is talk of a “Rosetta Stone” to translate competencies, so associated 

credentials can move more easily across organizational lines, expediting individuals’ progress towards their learning goals. Given the rate at which new 

competency models are entering the marketplace, however, this might not 

be the best approach. Leveraging the “currency” metaphor—which is enormously popular among competency-based learning proponents—is helpful. A 

“Rosetta Stone” would serve as a sort of “currency calculator” to compute exchange rates among credentials. That would be a complicated process. Part of 

the challenge, however, is that one may not be able to track the exchange rates 
260 | Modernizing Learning

The U.S. Department of Energy’s Leadership Development Seminars July 2013–2014 

Edition links learning opportunities both within and beyond the government to aid 

employees seeking to master the Executive Core Qualifications. This catalog lists 

over 550 courses offered by the U.S. Office of Personnel Management as well as 75 

universities, colleges, and private industry organizations throughout the continental 

U.S. and more than 700 leadership readings, mapped to various Executive Core 

Qualifications. Each listing is cross-referenced and includes a brief description of the 

course as well as its date, location, cost, and contact information.

Note: The listing of these courses does not constitute endorsement of their content by 

the U.S. Department of Energy or any agency of the U.S. Federal Government.

Department of Energy Learning Opportunities Correlated to Executive Core Qualifications
Competency-Based Learning | 261 

for every currency interaction (pesos to dollars, dollars to rubles, and fenings 

to pesos, for instance), particularly as their values fluctuate. There would be 

so many different currencies to track! Why not leverage the same approach 

used for different currencies? Evaluate the relative value of the currency in 

relation to commodities. If one knows how many of a given currency it takes 

to purchase a commodity, such as a loaf of bread or barrel of oil, currency 

conversion is easy.

Within a competency-based system, the commodity is performance—what 

individuals know and can do. Hence, to exchange competency information 

among institutions, these organizations needn’t learn one another’s competency models; they need only focus on what those competencies can “buy,” that 

is, the credentialed performance. 

However, credentials are only as effective as the reliability of their measurements, and some challenges still remain in this area. For instance, the same 

competency may manifest differently in different contexts. For example, 

seemingly universal competencies, such as leadership, may vary widely between professions. As leaders, surgical doctors need more procedural knowledge than business leaders who, in turn, may need more skill in motivating 

their staff to increase sales. Therefore, while some competencies have similar 

“bundles” of required knowledge, skills, and other attributes, others require 

different components sets to determine applied competency. Credentialing 

bodies must take these issues into consideration when determining how to 

assess and manage credentials for competencies. 

Another challenge involves determining the evaluation criteria for performance standards; this can be particularly difficult. Some questions to ask are: 

What methods will be used to assess performance (tests, portfolios, writing)? 

Who’s responsible for the assessment? How will these assessments be used?16

Clearly, there are numerous questions to address before an integrated, overarching competency-based system can be realized. However, the subsection 

below outlines some recommended ways forward.
262 | Modernizing Learning

The currency of the future labor market will be skills or 

competencies, which will demand competency-based 

education in both early-life and lifelong learning.

Martin Kurzweil, J.D., Director, Educational 

Transformation Program, Ithaka S+R

IMPLEMENTATION 

RECOMMENDATIONS

1. Decide if competency-based learning 

is right for the organization 

The first step in embracing competency-based learning is to conduct a comparative analysis of the change versus the status quo. What’s the demand for 

competency-based learning? Can it enhance learning effectiveness and efficiency within the institution? Are the leaders supportive? Are the faculty and 

staff supportive? Is there sufficient talent to create the competencies, the resulting competency model, the levels of mastery, and the assessments so vital 

to competency-based learning’s success? Albert Einstein is reported to have 

said, “If I had an hour to solve a problem, I’d spend 55 minutes thinking about 

the problem and five minutes thinking about solutions.” Before embarking 

on a change to competency-based learning, consider carefully the potential 

benefits and challenges. Make sure that the investment will pay sufficient dividends. Lastly, make sure that the organization is willing to make the journey, too. To understand the relative value of the competency-based learning 

approach, consider other organizations that have already embraced it. Find 
Competency-Based Learning | 263 

organizations similar to your own with similar missions and challenges. Look 

at what they did to embrace competency-based learning and how they’ve employed it. ...and as much as possible, learn from others’ mistakes!

2. Build a competency model 

Next, construct and validate a competency model. There are several approaches one can take. Many institutions simply select from existing models where 

the competencies, performance levels, and other accoutrements seem to fit 

their needs, modifying their model as needed during validation. A second 

method is job analysis. With this approach, researchers dissect the various 

jobs performed within an organization figuring what core and/or occupational competencies are required and at what proficiency levels. Typically, the 

researchers will interact with workers to ensure the analysis is thorough and 

that all competencies have been properly identified. Another method involves 

leveraging panels of experts, surveys, and interviews to create a competency 

model. This is a fairly common approach and benefits from the fact that most 

organizations fail to capture the full breadth of tasks and knowledge within 

the human capital management documentation. A final method, and one rated 

most effective by experts, is a criterion sampling method. With this approach, 

researchers work with organizational members to establish criteria to identify the most outstanding performers. Applying this criteria, the researchers 

then interview these performers to determine “what makes them tick” and 

what competencies make them so successful in their jobs. The resulting model helps drive workforce development by focusing on the competencies most 

closely aligned with success—outstanding performance—thus benefiting 

both the employer and employees.

Validation can occur simultaneously as the model is being created. In essence, 

validation is a means to ensure the predictability of the competency model. 

If an employee who reaches the prescribed level of mastery in each of the 

listed competencies is judged to be an outstanding employee, then the model 
264 | Modernizing Learning

has a high degree of predictability and validity. If, however, those employees 

who reach all of the desired levels of mastery are still found wanting, then the 

model probably needs more work. With predictability—the “gold standard” 

for competency models—it’s easy to see why criterion sampling is a preferred 

method for creation and validation. Perhaps not surprisingly, starting with top 

personnel offers a shortcut to creating a model capable of predicting outstanding performers!

3. Develop authentic assessments for competencies

Once a model has been successfully created and validated, the next step is to 

develop authentic assessments through which learners can demonstrate levels 

of mastery. For industrial and vocational organizations, assessments can be 

based on actual job performance. For most technical skills, workers need only 

demonstrate their ability to perform their work-specific tasks correctly to earn 

certification for a given level of mastery. For “soft skills” and cognitive competencies, the assessments are usually more difficult. As noted previously, educational programs usually rely on samples of behavior and faculty judgment 

to assess competency mastery. A student required to demonstrate mastery in 

multiplication, for instance, with levels of mastery determined by the number 

of digits in the numbers being multiplied, would never be asked to multiply 

every possible combination of appropriate-length numbers. That would be 

ridiculous. Similarly, a student required to construct and deliver persuasive 

arguments would only have to perform this task a limited number of times before a faculty member felt confident in certifying a level of mastery in the task.

There are standardized tests for soft skills, for example the California 

Critical Thinking Skills test and a number of leadership and communication 

assessments. The key to building or selecting assessments is to ensure they’re 

valid (i.e., assess what they are supposed to assess), reliable (i.e., consistently 

produce similar results), and authentic (i.e., match similar challenges learners 

will encounter outside of the classroom—in the workplace, for instance).
Competency-Based Learning | 265 

4. Develop learning paths to reach desired mastery

This is where creativity and ingenuity can pay big dividends. If program 

leaders pursued a criterion-sampling approach to creating and validating a 

competency model, they may be able to ask those outstanding performers, 

“How did you learn that?” The same is true of others able to demonstrate mastery of competencies without taking any institutional classes or courses. The 

answers can be fascinating. It may turn out, for instance, that an employee 

O*NET – OCCUPATIONAL INFORMATION NETWORK

Sponsored by the U.S. Department of Labor, O*NET provides a database of 

general occupational descriptions, including typical job and employee attributes, necessary skills and knowledge, and workplace characteristics. These 

are provided as free, open-access resources for broad use across businesses, 

educators, job seekers, and HR professionals. To date, O*NET contains standardized descriptors for nearly 1000 occupations across the U.S. economy; 

these form a common foundation for codifying occupational competencies. 

Looking ahead, O*NET developers are exploring ways to create an overarching architecture across competency frameworks, and they’re starting to use 

GUIDs (Global Unique Identifiers) to connect credentials to O*NET competencies. Ultimately, O*NET developers imagine this work will remake the resumé, perhaps turning it into a clickable or drill-down document that contains 

someone’s entire “competency portfolio” but at levels of detail usable by employers. Further, the capability to relate bundles of competencies to specific 

education and training modules, classes, or sequences of courses could enable 

help individuals determine what competencies they need to achieve their career goals, and how to, or where to go, to acquire those capabilities.

www.onetonline.org
266 | Modernizing Learning

There’s no such thing 

as “nontraditional”

education anymore.

Fred Drummond

U.S. Deputy Assistant Secretary of 

Defense for Force Education and Training

who demonstrates mastery of “leadership,” gained the associated capabilities 

through the process of earning a Gold 

Award (Girl Scouts) or Eagle Scout (Boy 

Scout) designation as a child. 

Of course, many students and workers 

will need help in mastering core and 

occupational competencies. It’s tempting to offer a single course that covers 

a wide variety of topics, competencies, and proficiency levels; however, focused learning, addressing the specific desired competencies and proficiency 

levels, coupled with sufficient time for reflection and practice, is key. Further, 

it’s more efficient: Institutions invest only what’s needed to achieve success, 

and learners don’t waste time or effort picking up unnecessary or duplicative skills and knowledge. Obviously, this applies more specifically to workplace-learning than to educational applications. Development of the cognitive 

competencies so foundational to education requires a depth and breadth of 

learning far broader than a specific vocational focus.

As noted earlier in the “athlete example,” one needn’t create a program or 

course for every learning need. Immersive learning experiences, such as 

special work assignments, often allow learners to reach their goals more 

effectively and efficiently than formal classes. Another option to consider is 

the guild approach, as addressed at the start of the chapter. Maybe assigning 

an “apprentice” to a “craftsman-mentor” is the key. Also, one shouldn’t 

exclude off-duty, nontraditional learning. For instance, an employee or student 

struggling with public speaking may not need a speech class; perhaps joining 

a local Toastmasters club will foster her skills. Hence, it’s useful to document 

the various ways other people have developed their own capabilities; these 

can serve as models and potential pathways for those seeking to earn their 

own credentials. Consider accumulating this information into a catalog, 
Competency-Based Learning | 267 

where learning experiences are cross-referenced to specific competencies and 

proficiency levels.

5. Lastly, organizational leaders need to ensure 

there’s a mechanism for tracking and reporting 

competency mastery

This isn’t a simple task. Those responsible for this will have to consider the 

broad array of users who need access to the information. Certainly, learners 

need to know how they’re progressing—where they’re strong, where they’re 

weak, and what they need to do to achieve their learning goals. For educational institutions, faculty and staff will need access to the information. There’s 

also a need for transcripting learning progress for sharing with learners and 

other institutions. Industrial entities will have a variety of data-users as well. 

Like students, workers will want to know where they stand. Supervisors will 

TECH TOOLS EXAMPLE: JDX

The Job Data Exchange (JDX) is a new set of open data resources, algorithms, 

and reference applications for employers and their HR technology partners to 

use in improving how employers communicate competency and credentialing 

requirements for in-demand jobs. Today, 50% of open, available positions in 

the U.S. country go unfilled because employers can’t find the right talent for 

their critical positions. At the same time, education, training, and credentialing 

providers are in need of better, faster, clearer signaling from employers on what 

skills are most in demand in a changing economy. The JDX isn’t a “job board,” 

rather, it will be a resource for employers and their HR technology partners to 

more clearly define competency and credential requirements for jobs distributed 

to talent sourcing partners such as job boards and preferred education, training, 

and credentialing partners. The U.S. Chamber of Commerce Foundation and 

their parters are pilot testing the JDX throughout 2019 across six states and the 

District of Columbia. 

See: www.uschamberfoundation.org/workforce-development/JDX
268 | Modernizing Learning

want to know how their individual workers are progressing in their development, and also where their teams are strong or weak in terms of needed competencies. Similarly, progressive levels of supervision will want insight into 

this aspect of workforce development.

Within the military, the term force readiness describes how ready a military 

force is to execute its warfighting mission. Competency-based learning provides a granular look into force readiness, providing senior leaders insight 

into where they need to invest their developmental resources. Prior to World 

War II, the U.S. Marine Corps correctly anticipated the nation would face a 

war in the Pacific. The Corps purchased equipment to effect beach landings; 

however, there was also a corresponding need to teach Marines to fight in this 

extraordinarily challenging, sea-to-shore environment. In essence, the Corps 

determined a new competency was required, assessed the developmental need 

this new competency created (gap analysis), then began training Marines to 

execute the new mission. 

 A holistic look at workforce, student, or 

 military-unit competencies can help leaders 

 make learning investments more wisely 

Summary

As noted at the beginning of this chapter, competency-based learning isn’t 

new. It is, however, an exciting way to approach learning. The power it gives 

to learners—the control they have over their own learning journeys—creates 

an excitement both for the learners and those guiding them to their eventual goals. Competency-based learning also fosters creativity as both learners 

and leaders seek new ways to attain and demonstrate mastery. Lastly, competency-based learning offers that “common currency” that permits learners, 

workers, and their institutions to both understand developmental needs and to 

share achievements across institutional barriers.
Social Learning | 269 

CHAPTER 14

SOCIAL LEARNING

Julian Stodd and Emilie Reitz

Formal learning is a story written by an organization and addressed to its 

people. Social learning, in contrast, is a story largely written by the learners, 

themselves. It’s about tacit, tribal, and lived wisdom that exists within distributed communities. It’s often untidy, diverse, and deeply personal, as people 

bring their own perspectives and experiences into the learning space. Modern 

organizations are increasingly interested in how to unlock the power of social 

learning. This chapter explores that question; it describes what social learning 

is and elucidates a design methodology of Scaffolded Social Learning.

1

 This 

is considered against the backdrop of the Social Age, the evolved reality within which we live, and an understanding of the impacts this has on learning 

through its forms of power, knowledge, and control. 

Living and Learning in the Social Age

Technology is the most visible manifestation of change we see around us: the 

rise of social collaborative technologies, leading to the proliferation of connectivity, and the democratization of organization at scale. Put simply, we’re 

now connected in many different ways, almost all of which are outside of the 

oversight or control of any formal organization or entity.2 In network terms, 

there’s high resilience and great redundancy in our connections—which is 

significant. Historically, mechanisms of connection were local and tribal, or 

large-scale and formal. We connected within formal hierarchies and formal 

organizations, and within those spaces, we were expected to conform, to wear 

the “uniform,” use the appropriate “language,” and accept the imposition of 

 

 
270 | Modernizing Learning

“control.” Today, our global connections—our connections at scale—are 

broadly social, distributed, and with the imminent proliferation of synchronous machine translation, often culturally diverse. We’re substantially liberated from language, time, and place. And with these changes comes a shift in 

individual expectations, feelings of entitlement, and perceptions of fairness. 

In turn, this leads to a shift in power across individual and collective and formal and informal dynamics. There’s a broad rebalancing taking place around 

the world, slowly draining power away from formal systems (hierarchy) and 

into social ones (community). An important part of shifting power dynamics 

is the fracturing of the social contract between individuals and organizations. 

The notion of “career” is evolving; it no longer emphasizes lifelong loyalty 

between an employee and a company. Instead, our public reputations, our 

personal networks, and the broader communities that surround us become our 

“job security.” 3 This has broad implications for learning and development.

 In the Social Age, learning is increasingly 

 dynamic, co-created, and adaptive, and we 

 must invest in that co-creation 

As our commitment to formal organizations becomes increasingly transient 

and transactional, we’re seeing new entities emerge, or adapt, to fill gaps in 

adult education, vocational training, credentialing, and other talent management functions. Many of these entities are socially moderated and utilize social learning approaches. We already see early stages of this: Into this void 

step the MOOCs (democratized teaching), the tech entities such as LinkedIn 

and Udemy (democratized, beyond formal control), and portable credentials 

such as the Open Badges initiative. Looking ahead, we’re also seeing new 

“guilds” emerging.4 These guilds hold emergent political powers across institutions, and rather than being constrained by traditional structural organizational boundaries, they’re instead defined by the bounds of knowledge and 

capability, such as cybersecurity or anesthesiology.5
The type of learning these new entities offer is different. No longer hindered 

by decades of organizational stagnation and “known knowledge,” it’s typically more dynamic, co-created, contextual, adaptive, and free. This speaks 

to the challenge of how organizations need to adapt to the new ecosystem: 

Clinging to old models of organizational design (nested power structures), 

formal learning (learning as a form of control), formal hierarchies of power 

(systems of consequence), and known knowledge (unchallenged, static organizational dogma), is a sure fire way to be disrupted, from the level of organizations up to the scale of nations, themselves.6

 And hence, the old structures 

of formal power are ceding some of their relevance—unless they can adapt.7

We’re used to seeing training and education as discrete parts of a stable system, but today, in the context of the Social Age, learning and development 

are dynamic parts of a dynamic system—and we must adapt them to fit the 

changing times, not just the new modes of delivery available. In other words, 

our adaptations must fundamentally readdress the design, facilitation, assessment, and support of learning. We must develop new methodologies for learning, and invest heavily in the communities and social leaders who will deliver 

these new capabilities so that we don’t simply survive—but thrive, and avoid 

disruption and failure, in the Social Age.

The New Nature of Knowledge

Delving into semantics may kill us, but let’s briefly consider the nature of 

knowledge, not at the deepest philosophical level but at the rather mundane 

Social learning is a type of informal learning; 

it’s frequently experiential and often facilitated 

by distributed communities. It’s generally untidy, 

diverse, and deeply personal, as people bring their 

own perspectives and experiences to the learning.
272 | Modernizing Learning

and practical one: Our ways of knowing are changing. We’ve moved from 

“concentration” to “distribution.” Where once we memorized and codified 

knowledge, and held it in libraries, books, vaults, and experts (in concentrated “centers of learning”), today it’s dispersed, distributed, and free—yet, not 

without its problems (validity, bias). 

Clearly, we still need “formal” knowledge with its mechanisms of validation, replicability, and rigor. But in many cases, we seek just enough and just 

“good-enough” knowledge to get us to the next step of the journey, like the information we access from our smartphones while racing through the airport, 

let’s say, trying to make a swift decision about our connecting flight. Another 

key difference between formal learning and social learning is that “formal” is 

often abstract and frequently decontextualized while “social” is inherently applied, because it’s done in the everyday reality. Where formal learning often 

takes place in special spaces (classroom, laboratories), social learning more 

often occurs in performance settings (around the water cooler) or at the point 

of need (a YouTube “how to” video or Reddit answer). 

Is this type of distributed, community-moderated knowledge always correct? 

Absolutely not, but to be fair, neither was all of our “old” knowledge. And crucially we’re still creating the mechanisms of validation for social knowledge 

that may make it ever better. This is a feature of the Social Age that’s often 

misunderstood: What we see around us today isn’t the end state. It’s often the 

first early prototype. In contrast, the old system is relatively evolved and static. The new one is still in constant motion; it’s always improving.

If we worry about validity to the point where we take no action, then we can’t 

benefit from social learning. Conversely, if we liberate social learning with no 

account of the risks, we’ll be overtaken by it. We must learn to balance both, 

in a persistent dynamic tension.
1 Learning is changing

Against the backdrop of the Social Age, the type of knowledge we engage with everyday has 

changed, often co-created, geolocated, adaptive, and hidden within our social communities.

2 Scaffolded social learning can support social learning 

Scaffolded social learning is a design methodology, and modality of learning, which creates a 

loose structure, a scaffolding, within which learning communities carry out “sense making” 

activities, all the while engaging with both formal and informal social knowledge.

3Learning isn’t confined to formal or controlled structures

A significant amount of learning takes place outside of formal structures and within 

communities that are trust-bonded, complex, and powerful. Our challenge is to create the 

conditions for these communities to thrive.

4 Stories fuel social learning—and can benefit those willing to listen

Within these communities, learners create stories, narratives produced both individually 

and collectively; these stories can inform the wider organization, if it has the humility and 

willingness, to learn from them.

5 Social learning is just one part of a larger, Social Age strategy

Adopting social learning is just one part of a wider cultural transformation, and that 

transformation could break every other part of an organization.

HIERARCHICAL HYBRID SOCIAL
274 | Modernizing Learning

Formal and Social Systems: Dynamic Tension

The formal system is everything an organization can see, own, and control. 

Formal systems are where we create formal learning, and they’re extremely good at certain things: collectivism, consistency, and achieving effects 

at scale. Flowing around and through the formal system are social systems. 

These aren’t held in contractual relationships but in trust-based ones. The 

social system is multilayered, contextual, often internally conflicted, and ever 

changing. Social systems are also good at certain things that formal ones aren’t: They’re good at creative dissent, gentle subversion of outdated processes, questioning of systems, radical creativity, social amplification, movement, 

momentum, curiosity, and innovation.

Healthy, modern organizations exist in a “dynamic tension” between the two, 

and social learning takes place at this intersection, incorporating parts of the 

formal and parts of the social. Our challenge is to maintain, not deny or destroy, 

this tension.8 If the formal system triumphs, we get greater consistency and 

hear the story that the formal organization agrees with, but we may not achieve 

true learning. If the social system wins, and subverts formal structures entirely, we lose our ability to validate quality, have consistency, and achieve effectiveness at scale. But if we can master both, we can thrive: formal structure 

and social creativity held in a dynamic tension. To do so requires a scaffolding, 

an evolution of mindset, and a willingness, on both sides, to listen and learn. 

In a recent research project from a healthcare setting, we (this 

chapter’s authors) asked learners which technologies they use to 

collaborate. They identified 17 different platforms, only one of which 

was sanctioned for official use by their organization. Knowledge has 

already flown the coop; denying the change won’t prevent it. Instead, 

we must engage to help better the rapidly evolving social system.
Social Learning | 275 

FACILITATING A SOCIAL 

LEARNING CULTURE

1. Create the conditions for effective social learning 

Authority within formal systems is represented by rank, title, and formal 

qualification. In social systems, authority is granted by the collective based 

upon reputation, trust, fairness, and the investments made over time. It’s this 

social authority that we draw upon within social learning communities; it’s 

reputation that counts. In the context of social learning, our ability to learn 

and collaborate socially depends partly on our social authority as well as our 

levels of social capital. Much like we need political skills to thrive in formal 

spaces, so too do we need social skills to thrive in informal spaces. Self-regulated learning abilities, as described in Chapter 15, are also critical. Hence, 

as we think about ways to enable social learning, it’s important to consider 

how to foster productive communities as well as how to support the social and 

learning processes of their various members.

2. Scaffold formal, social, and individual learning

Consider an approach for social learning called Scaffolded Social Learning.

9

It’s a methodology for the design, delivery, facilitation, and support of this 

type of co-creative learning. It defines principles related to co-creative spaces, 

formal learning assets, and learning community support structures that help 

formal organizations integrate social learning into their contexts.

First, consider that in social learning, individuals will engage with formal 

assets (stories written by the organization, codified and accepted knowledge), 

social assets (tribal, tacit knowledge, held within the community), and individual knowledge (worldview, preconceptions, biases, and existing knowledge). 
276 | Modernizing Learning

We can create spaces and

provide support for this to

happen, using scaffolded social

learning approaches

Communities take the formal

story and add local and

individual context

They carry out 

sensemaking activities

GREAT FOR BUILDING DIVERSIFIED 

STRENGTH, RADICAL CREATIVITY, 

AND INDIVIDUAL CAPACITY

GREAT FOR DRIVING CONSISTENCY, 

VALIDITY, AND STANDARDIZED 

STRENGTH AT SCALE

Organizations capture

their codified strength

in formal stories

They share these stories

through formal learning

They use technology for

distribution, assessment,

and compliance
Social Learning | 277 

From a design perspective, one can, for example, vary the amounts of formal knowledge provided, create conditions for sharing tribal knowledge, and 

schedule reflective opportunities for individuals to explore their own experiences. The “scaffold” in Scaffolded Social Learning represents these structures. In other words, this scaffolding supports specific activities designed 

to facilitate and integrate formal, social, and individual learning, and to help 

people “make sense” of it all, both individually and collectively as a group.

Second, at a technical level, consider the implementation of Scaffolded Social 

Learning. It involves choreographing experiences across these formal, social, 

and individual constructs. Like a good play, learning can be sequenced into 

a “running order,” so that formal learning assets are released at certain times 

that coincide with community activities, such as group storytelling. To extend 

the theater metaphor, scaffolded learning also involves a range of supporting 

roles, both front of stage and back of house, such as community managers, 

storytellers, coaches, and social leaders. These learning facilitators help define the learning spaces, encourage activities that provoke and support the 

manipulation (the processing) of new knowledge, and create opportunities for 

people to bring in and demonstrate their own specific expertise. These actions 

help manage the learning tempo, maintain its momentum, and drive up engagement.

3. Use gentle learning interventions to 

nurture social learning communities

Specific co-creative behaviors can enrich the activities of a social learning 

community. For instance, putting loose structure into conversations and creating common patterns of activity can help to draw out coherent narrative 

threads across concepts. As an example, consider the social learning tactic 

of curation. In Scaffolded Social Learning, the learning facilitator might not 

bring a formal example of, let’s say, good teamwork or effective problem-solving, but rather would encourage learners to bring their own. Now, one person 
278 | Modernizing Learning

may bring an example that seems terrible to others, and another person might 

offer one that seems off-track. Hence, another step is to encourage the co-creative behavior of interpretation. This is where someone writes a narrative, 

shares a story of precisely why he sees the case study as relevant or how it 

relates to her personal journey. In other words, this involves interpreting the 

thing they curated and exchanging stories across the community.

Will we agree? Well, that doesn’t matter: Social learning isn’t about conformity and agreement; it’s about broadened understanding, context, and perspective. We don’t get to deny the validity of others’ examples, but we’re absolutely 

allowed to challenge and engage in debate about them. Indeed, challenge can 

be another co-creative behavior: I tell a story, you respond, I try to paraphrase 

your story, you respond, we both collaborate and respond to a third story, and 

we come together to co-create an overall narrative.

4. Assessment is feasible, but don’t apply it blindly

Our effectiveness as social learning designers is largely tied to our ability 

to define and master the usage, combination, and creativity of co-creative 

learning approaches, and to use them to craft engaging and effective learning 

spaces together. However, it’s worth saying that organizations can measure 

the effectiveness of social learning equally as well as they can measure the effectiveness of their formal training and education programs (although with the 

caveat that that’s not saying much!). Like with formal learning, it’s generally 

worthwhile to triangulate assessment approaches:10 Do learners feel they’ve 

learned? Does the community believe they’re learning? Do learners score 

more highly on formal knowledge tests or in simulation-based exercises? Are 

there any noticeable changes to the processes or products developed outside 

of the learning context? 

While you can technically measure anything, the pertinent question may be, 

How will that information be used? The collaborative technologies often used 
Social Learning | 279 

to support social learning have many convenient built-in measures; various 

systems can report metrics about “engagement” (used as a byword for “clicking”) or “interaction.” They can also produce social network graphs or output 

all manner of frequency statistics (e.g., log-on averages, average number of 

posts). Technology certainly allows us to measure, but hard thinking should 

be done on what to measure, how to best measure it, and what to do as a result. 

Unless we can answer these three questions clearly, it’s best not to measure 

at all. Measurement is enticing and important, but when misapplied, it can 

lack value, waste resources, and even impede learning. The best advice is to 

consider measurement carefully. Focus on outcomes, and where applicable, 

triangulate among (1) self-assessed, (2) observational, and (3) formally moderated measures.

5. Build social learning spaces 

and foster communities 

At the heart of social learning are the learning spaces—the places people 

come together to carry out collective sensemaking activities. To be very clear, 

space means something very different than community. Consider the analogy 

of building a new town: You can build houses, landscape gardens, construct 

a mall, and pave a town square. You can even move people into those houses. 

But none of this creates the community. It’s only begins to emerge when two 

of those people come together, on a street corner, let’s say, and have a conversation about what a terrible job you’ve done on the brickwork. The buildings 

form the space; the conversation forms the foundations of the community. 

Spaces for social learning might be a classroom, a chatroom, or some kind of 

learning management system—however, none of those are the community. 

In social learning, as in our allegorical town, individuals interact across multiple spaces, on the street corner, at the marketplace, or in someone’s home. 

In a learning context, multiple spaces—multiple technologies—may support 

a community, and their conversations may span across them, starting in one 
280 | Modernizing Learning

and graduating to another. It’s useful for the 

design of social learning spaces to takes this 

into account and to explicitly design for different types of social interactions, such as conversational spaces, collaborative spaces, infrastructure spaces (for formal components), 

subversive spaces (to complain about the 

“brickwork”), assessment spaces, and so forth. 

Each learning space is differentiated by 

notions, such as its permanence and consequence. For example, a conversational 

space needs high impermanence, while a 

formal assessment one may carry great permanence. Collaboration spaces should be 

low consequence, and performance spaces may carry high consequence. Social learning takes place across these 

diverse constructs and associated technologies—it’s not bounded by a single system or conceptual frame. Hence, the ability to construct such spaces as a coherent ecosystem is a core skill for socially dynamic organizations, i.e., organizations adapted to benefit from social learning approaches.

To encourage social learning communities, we need to create the conditions 

for them to emerge. Start by dedicating time to growing the community prior 

to moving into any formal learning activities. Before you can be purposeful, 

you need to be coherent; that is, before meaningful learning can begin, you 

first need to establish a high functioning community.

SENSEMAKING ENTITIES

Coherent communities are sensemaking entities; they help figure out information, identify misinformation, determine value, and recommend responses. 

Our social communities help us to filter the signal from the noise, and then to 

A well-designed Scaffolded 

Social Learning experience 

will contain differentiated 

learning, rehearsal, and 

performance spaces
Social Learning | 281 

understand those signals. In the context of social learning, where much of the 

sensemaking is done in the community, this helps provide a diversified view, 

and the more diverse in worldview, experience, cultural profile, and capability 

the community is, the more effective its sensemaking can become.11

MECHANISMS OF ENGAGEMENT

Within formal systems, we’re assigned roles by the organization, but in social 

systems, our roles are more nebulous and change more often. Sometimes we 

bring specific expertise, resources, or capability; sometimes we bring challenge, sometimes support, and other times we’re cross-connectors, linking 

different communities. Sometimes we simply come to learn. When considering social learning communities, it’s worth remembering that we don’t need 

everyone to engage in a certain way; we just need broad engagement. It’s fine 

for people to take diversified roles.

RITUALS AND CHOREOGRAPHY

There’s a role for ritual; in our own research, people described the “rituals of 

welcome and engagement” as the single most important factor for their future 

success within a community. Such rituals are something within our control; 

when designing the scaffolding for social learning, we can actively design rituals or consciously adopt existing ones. We can work with community members on their rituals of engagement for new members, for example, and can 

work with their formal managers on the rituals they’ll use to share stories of 

their learning back to the rest of their teams.12 It’s all part of the choreography 

of learning. This means we pay equal attention to every part of the learning 

experience, from the email that invites someone to join to the registration 

instructions they receive, the way we thank them for sharing stories, and the 

ways we graduate them at the end. It’s important to script and craft each part 

as an element in the overall running order. Pay attention to them all. Together, 

rituals and choreography form a powerful tool of community-building and, 

ultimately, of learner engagement.
282 | Modernizing Learning

HIDDEN COMMUNITIES

We’ll never find all the communities within an organization. Some (like our learning 

communities) are visible and formally sanctioned, others exist outside our networks and 

experience. Some even exist in active opposition, deliberately hidden from us. When we 

ask people what their most valuable communities are, for learning, they often speak 

of these hidden communities, formed on 

WhatsApp or as Facebook groups—places 

beyond formal oversight and consequence. 

It’s worth remembering that hidden communities aren’t new; we’ve always existed within a web of communities, but in the context 

of the Social Age, the boundaries between 

formal and social communities have blurred. Although formal communities 

haven’t substantially encroached beyond their organizations, social ones have 

invaded that previously sacrosanct space. The difference today is that these 

hidden communities can form and operate, at scale, and do so right under our 

noses. This is the consequence of the democratization of communication and 

connectivity.

SANCTIONED SUBVERSION

Moving ourselves beyond a binary understanding of which answers are “right” 

or “wrong” is valuable. Sometimes the answer lies in breaking the question. 

Subversion itself can be of great benefit to formal systems, if they’re willing to 

listen, because established organizations are typically very bad at subverting 

(or evolving) themselves. Consider this: How many organizations put as much 

time and effort into deconstructing redundant process and un-writing outdated rules, as they do into forming new ones? Very few! What happens around 

We belong to many different 

communities. Some communities 

are visible to both us and the 

organizations we work for, while 

others are hidden deep in our social 

networks, out of sight from formal 

institutional authorities but still 

very relevant and connected to us 

individually in our day-to-day.
Social Learning | 283 

this organizational detritus? Typically, it’s subverted; people work around redundant systems and suboptimal process. And they do so not only individually, but collectively too; indeed, when people join a new organization, much 

of what they learn, at the local or tribal level in the early days of a new job, 

comes exactly from this type of crowdsourced subversion, usually under the 

generic banner of “this is how we get things done around here.”

Conclusion

Stories, communities, learning—these are all expressions of power, and in the 

context of the Social Age, power itself is evolving. As we engage more broadly and more intentionally in social learning, we’ll discover that our formal 

power doesn’t carry through into social spaces: within these learning communities, you can shout all you like, but it’s social authority, reputation-based 

influence, and social capital that count the most. In the course of adopting 

social learning, we inadvertently (but necessarily) erode the power of the formal organization. 

As we cultivate the social community, this newly empowered collective 

will demand ever greater freedom and power. If our aim is learning 

transformation, then this power is what will drive the change. 

It’s a champagne bottle to uncork with care. The balance between formal systems of control and socially moderated ones creates an important dynamic tension. When managed effectively, a socially dynamic organization can 

emerge, one that integrates the very best of the formal (system, process, hierarchy, and control) with the very best of the social (creativity, subversion, 

innovation, amplification). That’s our challenge: to craft more collaborative 

models of learning, and to learn how to build an organizational culture in 

which learning can thrive both for today and through our emerging future 

learning ecosystem.
Americans should have self-sovereign management of their 

lives. Right now, medical records are yours but not so much 

your educational records; you don’t really control any of that info right 

now. We’re working on envisioning what the future looks like following 

these guiding principles: to give each person their own destiny, 

balance on the supply and demand side...and put it into the hands of 

the ones who want to earn the competencies and credentials. It gives 

them the power to drive the marketplace. Currently, the providers 

squarely have the advantage, but we need to make it a new space 

where learners are empowered.

Jeanne Kitchens

Chair of the Technical Advisory Committee for Credential Engine; Associate 

Director of the Center for Workforce Development, Southern Illinois University
Self-Regulated Learning | 285 

CHAPTER 15

SELF-REGULATED 

LEARNING 

Louise Yarnall, Ph.D., Michael Freed, Ph.D., 

and Naomi Malone, Ph.D.

There’s a growing need for continuous 

modes of lifelong learning to cope with 

the acceleration of knowledge production and flow aided by new technologies. 

In response, both schools and workplaces are progressing towards more 

independent, learner-centered forms 

of education and development. Potential support for lifelong learning comes 

from improvements in AI technologies 

that permit more personalized learning, 

and greater access to mobile and search 

technologies that provide ubiquitous access to information. In the workplace, 

trainers are increasingly using cloud-based software, augmented reality, and 

virtual reality to prepare workers, support their lifelong learning needs, and 

enable diverse collaboration methods.1 In higher education, institutions are 

increasingly offering online education options and providing students with 

information resources and communication tools to aid their independent research and collaboration. However, despite these trends, both educators and 

employers report challenges with this shift towards greater learner-control. 

For instance, some learners have difficulty taking responsibility for their own 

 

 

Self-regulated learning refers 

to the thoughts, feelings, and 

actions some learners use 

to independently attain their 

learning goals. Self-regulated 

learners are metacognitively, 

motivationally, and behaviorally 

active in their own learning.
286 | Modernizing Learning

learning,2 and others may struggle to assimilate their diverse experiences—

leading to a situation where they have increased exposure to information but 

reduced overall comprehension. 

Learners need to become skillful at regulating their learning over time and 

across different settings, especially to acquire thinking, writing, and analysis 

skills.3 However, individuals often struggle to manage their learning without 

effective and perceptive external support, such as what a teacher, mentor, or 

well-structured piece of courseware might provide.4 Consequently, developing 

effective self-regulated learning skills requires educators and trainers to help 

learners notice knowledge gaps, try new strategies, and adopt more proactive 

mindsets. Incorporating support for this approach into new technologies can 

also help learners acquire the meta-level skills needed to manage their own 

learning across their lifetimes.

Empirical research is beginning to identify effective tools and strategies for 

aiding self-regulated learning; however, the paradigm originally emerged 

during the 1980s when education researchers studied why some K–12 students 

succeeded in traditional classrooms better than others. They found the most 

SELFREGULATED 

LEARNING

SELF-REFLECTION

SELF-JUDGMENT

• Self-evaluation

• Causal attribution

SELF-OBSERVATION

• Self-satisfaction/affect

• Adaptive/defensive

FORETHOUGHT

TASK ANALYSIS

Goal setting •

Strategic planning •

SELF MOTIVATION BELIEFS

Self-efficacy •

Outcome expectations •

Intrinsic interest/value •

Goal orientation •

PERFORMANCE

SELF-CONTROL

• Imagery

• Self-instruction

• Attention focusing

• Task strategies

SELF-OBSERVATION

• Self-recording

• Self-experimentation

Figure 15-1: The three phases and 

subprocesses of self-regulated learning, 

derived from Barry Zimmerman’s work
Self-Regulated Learning | 287 

effective students demonstrated a set of learning strategies and mindsets including metacognitive strategies (e.g., goal-setting, self-monitoring, self-evaluation), cognitive strategies (e.g., rehearsal, organization, elaboration), environmental management strategies (e.g., time management, study area 

management), and self-beliefs (e.g., self-efficacy, intrinsic and extrinsic goal 

orientation, effort regulation).5 Since these behaviors stemmed from learners’ 

personal choices, researchers categorized them as “self-regulated” learning.

By the 1990s, researchers agreed that learners self-regulate during three iterative phases: the forethought phase, where a learner plans and initiates action; 

the performance phase, during which learning actions occur; and the self-reflection phase, in which a learner reflects on and evaluates performance, adjusting as necessary. Barry Zimmerman, one of the preeminent scholars in 

the self-regulated learning field, developed a model of these three phases, 

grounded in social cognitive theory (see Figure 15-1).6

More recent evidence has demonstrated that some self-regulation strategies—

time management, effort regulation, and critical thinking—have positive impacts on academic outcomes, but that other strategies—rehearsal, elaboration, 

and organization—have less empirically convincing effects. Further, in both 

school and workplace settings, a small number of these strategies have the 

largest impacts, accounting for 17% of the overall variation in learning outcomes.

7

 These include:

1. CONFIDENCE, SELF-EFFICACY, INTERNAL LOCUS-OF-CONTROL –

Effective learners believe they can learn because they’re in control and tend to 

take a more “active” approach to learning. By contrast, less effective learners 

doubt they can learn (because they think they’re not smart enough or not in 

control) and, consequently, take a more “passive” approach to learning.8

2. GOAL SETTING AND PLANNING – Effective learners set appropriate 

learning goals, anticipate the resources required, and set benchmarks for their 

progress. By contrast, less effective learners may not set goals or may simply 
288 | Modernizing Learning

plunge in, then run out of time or lack access 

to appropriate learning resources.9

3. PRIOR KNOWLEDGE AND STRATEGY 

USE – With stronger prior knowledge, effective learners engage in greater instances of 

planning and monitoring, both independently 

and in collaboration. With lower prior knowledge, less effective learners use just a few 

strategies.10

4. METACOGNITIVE MONITORING – Effective learners note and address gaps and 

misunderstandings while they learn. Less effective learners fail to notice or address such 

difficulties in their learning.11

5. POST-LEARNING REFLECTION – Effective learners consider what they’ve learned, 

taking stock of what remains to be learned. 

Less effective learners fail to reflect sufficiently after learning and may rush 

to the next task.12

RECOMMENDATIONS

Helping learners develop better self-regulated learning skills will require new 

supports, added into the many contexts where people engage in learning. To 

cultivate awareness of Zimmerman’s three phases of self-regulated learning 

and to develop effective habits at the cognitive, metacognitive, emotional, and 

behavioral levels, we propose three conceptual levels of self-regulated learning support: micro-, macro-, and meta-interventions. The micro-level focuses 

...it’s not going to replace 

teachers, it shifts the role 

and nature of a teacher 

to a master facilitator.

Thomas Deale 

Major General, U.S. Air Force (Ret.) 

Former Vice Director for Joint Force 

Development on the Joint Staff
Self-Regulated Learning | 289 

on individuals and the tools they use to better navigate a personalized trajectory. The macro-level focuses on how to navigate the selection and progression across learning experiences. At the meta-level, there’s a recognition that 

building appropriate learning habits requires focused practice in the cognitive, social, emotional, and physical capabilities that contribute to resilience, 

effective decision-making, and lifelong personal growth. We describe applications of these three levels in the suggested interventions below. 

1. Use formative assessments to personalize 

support for self-regulation skills and mindsets 

Although research shows the benefits of supporting learners’ self-regulation, 

these interventions often rely upon the discretion and knowledge of their educators. Hence, better supporting self-regulated learning depends, in part, on 

enhancing the skills of teachers, workforce trainers, and managers, in addition to learners, themselves. To start, it’s useful to help stakeholders identify 

the specific self-regulation skills and/or mindsets needed in a given learning 

situation; a first step towards that is to translate self-regulated learning assessment methods from research into practice. For instance, several diagnostic tools can help identify the signs and symptoms of a learner with weak 

self-regulation mindsets or strategies. These diagnostic tools could be embedded into online courseware or used by teachers, trainers, and learners in both 

classroom and workplace settings. 

Drawing on the three-level support approach to self-regulated learning: Tools 

can be devised to support individual educators in learning specific diagnostic 

techniques (micro-level), to help them anticipate where self-regulated learning challenges may occur before any extended learning activity (macro-level), 

and to serve as a regular formative assessment to encourage the maintenance 

of effective mindsets and habits of self-regulated learning (meta-level). Below 

are some self-regulated learning assessments that could be put to use:
290 | Modernizing Learning

SELF-REPORT INSTRUMENTS

Technology can deliver self-report, self-regulated learning assessments; the 

results from these may be shared with teachers and trainers or fed into adaptive learning algorithms to provide more personalized support to learners. 

Such assessments may target key elements known to support self-regulated 

learning, including: level of motivation (e.g., The Motivated Strategies for 

Learning Questionnaire 13 ) and the skills of goal-setting, time-management, help-seeking, preparing the study environment for focused work, and 

self-evaluation (e.g., The Online Self-Regulated Learning Questionnaire 14 ).

In Marcus Buckingham’s work, StandOut, he designed an assessment...

One of the things that he applied there—that’s extremely successful—is a weekly 

check-in with a supervisor. Once a week, through technology, it sends a request: 

These were your goals last week. Were you able to reach these goals? What are your 

new goals? Did you use your strengths? What did you like? What did you detest? 

Responses help the supervisor know things like, John keeps disliking this, and I need 

to get this off his plate and make it less painful for him. This is what he’s liking, where 

he’s using his strengths. I need him to do more of this. It allows for side questions, 

too, like, how motivated are you in what you do? How are you as an employee in 

working with this environment? 

Then there were 5 critical questions provided quarterly asking if the team is 

growing and learning. It’s a huge help for a leader, and it also prompts me to go in 

and say, “This is what John is working on. This one’s important, so can you put it 

at the top of your list? Thanks for the great idea; I’m glad you’re working on it.” 

These learning interventions cause us to have a conversation in a less threatening 

format and talk back-and-forth. There are lots of benefits and these are the kinds 

of interventions we intend to apply during the Leadership for a Democratic Society. 

Suzanne Logan, Ed.D., SES

Director of the Center for Leadership Development and Federal 

Executive Institute, U.S. Office of Personnel Management

ASSESSMENTS IN ACTION: GOVERNMENT WORKFORCE EXAMPLE
Self-Regulated Learning | 291 

STRUCTURED INTERVIEW PROTOCOLS

Drawing from questions in existing research interview protocols, technology can be adapted to deliver helpful queries to teachers and trainers. These 

may, for example, help them consider and investigate the potential factors 

contributing to weak outcomes, either observed among students in school or 

personnel in a workplace setting. Factors useful for reflection include assessing learners’ skills for organizing and transforming information, setting goals 

and planning to learn, seeking information, keeping records and monitoring 

learning progress, preparing their study environment for learning activities, 

engaging in self-evaluation, meting out self-consequences, reviewing texts 

and notes, help-seeking, and rehearsing and memorizing. (See, for example, 

the Self-Regulated Learning Interview Schedule.15)

MEASURING SELF-REGULATION PROCESSES AS EVENTS

Education technology researchers working primarily in learning management 

systems are already moving towards designing more complex, process-oriented measures that can determine individuals’ deployment of self-regulated learning strategies over time. Measurement methods include think-aloud 

protocols and technologies that detect errors in tasks or employ online trace 

methodologies (e.g., of mood and task steps) that measure individuals as they 

go about their learning activities.16 To better support self-regulated learning, 

researchers will need to study how to adjust these types of detection methods 

for delivery and use across different learning technology platforms, such as 

mobile, augmented reality, and virtual reality. 

2. Build confidence, self-efficacy, and internal 

“locus of control” about learning

To realize a vision of self-regulated learning across a lifetime, more needs 

to be understood about the preconditions for developing habits of lifelong 
292 | Modernizing Learning

learning. International studies indicate wide variation in how well both early 

childhood education and family upbringing sets the stage for lifelong learning;17 however, it generally begins with establishing confidence and independence as learners. Over the past 35 years, K–12 education researchers have 

found evidence that open-ended instructional practices, such as guided inquiry activities, foster confidence and independence in learning more than other 

practices, such as traditional close-ended question-and-answer routines.18 Introducing open-ended practices in childhood can help set the conditions for 

lifelong learning, but continued support for self-regulation is needed even in 

adulthood. For example, some research indicates that those countries with the 

highest levels of lifelong learning among adults have robust adult education 

systems.19

Based on the three-level support approach to self-regulated learning, described at the beginning of this section, individual educators can be tutored 

in confidence-building techniques (micro-level); in methods for identifying 

likely areas of low confidence in an upcoming lesson (macro-level); and in 

noting, reflecting on, and accepting their own challenges with maintaining 

confidence during learning (meta-level). 

The one point I hope every single person can internalize—as 

the neuroscience evidence shows us—the brain is learning 

every single second of every single day. So, the way every 

individual learns is the same, but what they’re learning differs and 

that depends on context—internal and external. Our job is to align 

our learning goals to what the brain is actually learning. That’s a big 

paradigm shift for leadership. 

Melina Uncapher, Ph.D.

Director of Education Program, Neuroscape; Assistant Professor of 

Neurology, Weill Institute for Neurosciences and Kavli Institute for 

Fundamental Neuroscience, University of California San Francisco
Self-Regulated Learning | 293 

3. Develop goal-setting and planning skills

To improve self-regulated learning, goal-setting and planning, strategies 

should be translated into user-friendly tips to guide individuals while they 

learn. Such self-regulated learning support should be made available across 

a range of learning contexts, from face-to-face to online environments. The 

three-level support approach to self-regulated learning is useful here, too. Individual learners and learning facilitators can be linked to templates and tools 

to support goal setting and planning (micro-level). They can be encouraged 

to reflect on the pacing and time management required in multiple stages and 

phases of upcoming lessons and projects (macro-level), and they can be encouraged to confront resistance to goal setting and planning by seeing the 

success stories of those who employ these techniques regularly (meta-level).

4. Activate prior knowledge to enrich selfregulated learning strategy use 

Past education and experience represent both a potentially rich learning resource and a possible threat, since old habits and misunderstandings can block 

the grasp of new ideas and procedures. For this reason, educators, trainers, 

and instructional designers should incorporate activities and tools to elicit 

learners’ prior knowledge and help them reflect on which elements of it are 

potential building blocks and which are possible barriers.

Based on the three-level support approach to self-regulated learning, ways 

for activating prior knowledge might include: Linking individual learners 

and learning facilitators to lessons about how to elicit and document prior 

knowledge relevant to a particular lesson (micro-level). Identifying the useful 

prerequisite knowledge as well as the naïve concepts that might pose learning 

hurdles in upcoming lessons or projects (macro-level), and supporting individuals’ capacity to activate useful prior knowledge and to counter or encapsulate less useful prior knowledge (meta-level).
294 | Modernizing Learning

More research is needed in this area, however, to uncover new methods for 

estimating learners’ prior content knowledge and self-assessed self-regulation skill levels. Since traditional testing can negatively impact learners’ motivation, finding new assessment methods is a critical step to enhancing personalization models beyond their current level. Currently, traditional testing 

approaches and curriculum sequences favor comprehensiveness and certification. Work is needed to understand how adjusting the frequency and forms 

of assessment can inspire rather than hinder self-regulated learning. Methods 

worth exploring include integration of self-reflective assessments of content 

knowledge and self-regulated learning skills with validated measures of traditional content knowledge and skills.

5. Support metacognitive monitoring

As learning platforms and media proliferate, the community will need a wider 

range of ways to gather trace data on how and under what conditions learners 

use self-regulated learning supports. This line of research is likely to innovate around new approaches to using xAPI to collect student data, usefully 

aggregate datasets across experiences, and apply learning analytic models to 

analyze them. Such work need not focus only on individual learners’ patterns, 

but should also consider patterns within content pathways from multiple users. Such data traces can support more personalized and optimal recommendations of what content to review next and can strengthen systems to covert-

...it’s not just what you learned but 

rather how much it changed you.

 Betty Lou Leaver, Ph.D.

Director, The Literacy Center; Manager, MSI Press; Former Provost, 

Defense Language Institute Foreign Language Center
Self-Regulated Learning | 295 

U.S. DEFENSE TECHNOLOGY EXAMPLE

By providing fast access to short-form learning materials (“microcontent”), mobile applications can make it easy to use brief windows of 

available time for learning. Such applications can use AI to identify highinterest topics, select learning activities most likely to benefit the learner, 

and then recommend micro-content on selected topics and activities. For 

example, PERLS, a mobile app developed with DoD support, presents 

recommendations in the form of electronic cards that users flip through 

to find preferred content, and underlying these recommendation is a 

dynamic model of self-regulated learning. The app has been evaluated 

with several DoD organizations, including U.S. Northern Command and 

Joint Knowledge Online to augment training in areas such as Defense 

Support of Civil Authorities. Early results show that learners using PERLS 

reported heightened enjoyment and motivation to learn, and they 

performed as well as others required to take a full, formal courses.20

Next

Discover

Dabble Bridge

Familiarize

Practice Get Help

Assess

Refresh Use

Extend

EXPLORE

STUDY

SHARPEN
296 | Modernizing Learning

If we believe that the 

exploration of knowledge 

must continue, then 

we can’t only teach the 

knowledge we currently 

have. Truth and facts are 

constantly unfolding. If we 

just decide that by 2018 

we have all the knowledge 

we’ll ever need then we’re 

making a serious mistake.

Christopher Guymon, Ph.D.

Interim Dean of the Graham 

School, University of Chicago, 

Office of the President

ly strengthen or fade self-regulated learning 

support in a continuous fashion.

One aspect of self-regulated learning support that has not been adequately studied 

concerns understanding both the optimal frequency of self-regulated learning support and 

the optimal tools for providing this support. 

These factors are likely to vary by the content 

to be learned as well as the learning platform 

(e.g., LMS, mobile smartphone). R&D developers should be prepared to make the case 

for which self-regulated learning skills they 

plan to target, highlighting those skills most 

important for learners of their content and 

most amenable to support with their particular learning experience. Such design specifications can improve the field’s understanding 

of how different technologies can support 

specific self-regulated learning skills. 

Supporting metacognitive monitoring across 

the three levels of abstraction might include: 

Connecting individual learners and learning 

facilitators to tips and guidelines for noticing and remedying points of confusion, poor 

procedure or technique, and weak understanding (micro-level); identifying 

points for checking on understanding and procedures in upcoming lessons 

and projects (macro-level). Additionally, new methods may be able to track 

progress over time, measuring the effectiveness of techniques in reducing 

misunderstandings and, in turn, providing systematic feedback that sharpens 

procedures over time (meta-level).
Self-Regulated Learning | 297 

6. Foster habits of post-learning reflection 

Educators, trainers, and instructional designers need to provide extended 

post-training self-regulated learning support for learners, helping them to reflect, learn how to reinforce, and know when to refresh past learning. Such 

post-training support could be delivered by mentors and coaches, aided by the 

parent organization, or take the form of persistent technology-based tools for 

self-coaching and reference. 

To return once again to the three levels of abstraction, ways to foster post-learning reflection might include: Providing lessons to individual learners and 

learning facilitators about the kinds of useful questions to pose (micro-level); 

scheduling and building-on reflection activities across an extended lesson or 

project (macro-level), and rewarding learners for engaging in reflection activities, such as offering them the chance to unlock a range of new learning 

opportunities based on their reflective participation (meta-level). 

Summary

Successful self-learners do more than just study and memorize. They stay alert 

and are curious to discover new, valuable learning. They skim a lot of content 

to find the important points. They search informally to nurture motivation for 

intensive study and periodically review afterwards to fight forgetfulness. And 

they find the time to do it all. 

Though more than 70% of work-related learning is self-learning, few 

technologies help self-learners deal with these challenges. Ideally, technology 

will reduce the difficulty and friction of all self-learning activities, while 

making it easier to learn in small slots of available time, whenever and 

wherever these occur. Targeting and supporting self-regulation skills 

throughout personalized learning trajectories will aid learners of all ages and 

promote enhanced learning efficiency across lifetimes.

Organization
In an era where gaining access to information is no longer 

difficult, a continuing culture of high-stakes testing (focused on 

testing knowledge recall) runs counter to what we need. We should 

instead value the ability to sift through information and to connect, 

assimilate, aggregate, interpret, and apply data. If our teachers could 

help students view information from societal, cultural, economic, and 

other perspectives, and if they could help students practice writing 

and data-validation skills and creativity, we could accomplish so much 

more than just teaching them how to answer multiple-choices tests.

Anne Little, Ph.D. 

Vice President, Training Solutions Development, SAIC
Instructional Designers and Learning Engineers | 301 

CHAPTER 16

INSTRUCTIONAL DESIGNERS 

AND LEARNING ENGINEERS

Dina Kurzweil, Ph.D. and Karen Marcellas, Ph.D.

For over 60 years, instructional designers have supported teaching and learning, primarily by identifying effective ways to present material in formal educational and training environments. Given advances in technology, increasing 

access to data, and the explosion of formats and venues for learning, designers 

in the future will have to gain more knowledge and expertise than ever before as they develop their professional craft. Consequently, a new concept is 

entering this complex field: the learning engineer. Who are these individuals? 

What are their areas of expertise? How do their knowledge and skills relate 

to, expand upon, or differ from those of instructional designers? This chapter 

describes the history of instructional design and explores how the field of 

learning engineering will need to develop and expand upon instructional design methodologies to support teaching and learning in the future. 

Background: Design of Instruction

Traditionally, a number of specialists have collaborated in developing learning 

experiences and tools. Their titles and roles may differ somewhat depending 

on the project or the available personnel, but one commonly used team structure includes a technologist, a learning science expert, and an instructional 

designer. Technologists generally have technology backgrounds and use either personal experience with education or some learning science knowledge 

to help develop instructional technology tools. Some certainly have robust 

 

 
302 | Modernizing Learning

educational knowledge, but usually this isn’t the norm. In contrast, learning 

scientists are educational researchers who are deeply knowledgeable about 

how humans develop and learn, particularly from a cognitive perspective. 

Both of these roles can act in support of instructional designers, who apply a 

systematic methodology based on theory, research, and/or data to plan ways 

to teach content effectively. Instructional designers work in both educational and training environments. They’re problem solvers who use different instructional models to promote learning. In other words, they’re responsible 

for “the theory and practice of design, development, utilization, management, 

and evaluation of processes and resources for learning.” 1

A BRIEF HISTORY OF INSTRUCTIONAL DESIGN

The field of instructional design is historically and traditionally rooted in cognitive and behavioral psychology. It first emerged during a period when the 

behaviorist paradigm dominated American psychology. Its practice can be 

traced back to the late 1950s and early 1960s, but in those early days, one 

wasn’t referred to an “instructional designer.” Rather, those who worked in 

this field were typically called educational psychologists, media specialists, or 

training specialists.2

Through the 1960s and 1970s, the growth of digital computers influenced 

learning theories, and many new instructional models adopted an “information-processing” approach. The 1970s also heralded the systems approach to 

instructional design, including one of its best-known models, the Systems Approach Model, published by Walter Dick and Lou Carey.3

 The Dick and Carey 

approach offered a practical methodology for instructional designers, and it 

emphasized how each component of the model works together. Dick and Carey also highlighted how technology, media, and research were all impacting 

the field at that time and, consequently, how “modern” instructional designers 

differed greatly from their counterparts in the 1960s in terms of academic 

background, training, research, and tools.4
Instructional Designers and Learning Engineers | 303 

Throughout the 1970s and 1980s, the instructional design field continued to 

evolve; a later survey of instructional design models found they had differentiated into having a classroom orientation (focused on development of instructional materials for a single lesson or set of lessons by teachers), a product 

orientation (focused on development of specific products by teams), or a system orientation (focused on development of curricula by teams).5

 Present-day 

instructional design continues to have different application specialties, and it 

continues to be influenced by technology. However, rather than model instructional design theories on technology, as in the 1960s and 1970s, contemporary 

instructional designers explore ways to incorporate technology into their work. 

Experienced instructional designers recognize that technology has numerEXAMPLE: The proliferation of video cameras makes it possible for any 

instructor to record videos for use in courses; the role of the instructional 

designer is not simply to facilitate the incorporation of video but rather 

to examine instructional goals and identify areas where it can be used 

most effectively to support student learning, while also possibly identifying 

appropriate use of lower-technology and lower-bandwidth solutions in 

other areas. They also work with faculty to define content that would 

best be suited for video. Continuing with the video example, instructional 

designers also look at the video’s effect on learning and develop ways to 

improve the both the product and the learning outcomes.
304 | Modernizing Learning

ous uses for learning—but it’s still just a tool. While technology can provide 

many benefits, its effective use in training and education requires carefully 

defining its role and ensuring it remains subordinate to the learning goals. In 

recent history, we’ve seen a push for instructional designers to focus more on 

technology, shifting emphasis away from instructional theory. However, the 

systematic design, development, implementation, and assessment of teaching 

and learning requires that instructional designers keep instructional methods 

central to their work and examine all technology with an eye towards promoting more effective learning. 

“Technology is not an end in itself; any successful use of training technology 

must begin with clearly defined educational objectives.”6

INSTRUCTIONAL DESIGN ACTIVITIES

Instructional designers’ primary role is to support good instructional practice. 

As many professionals in the teaching and learning fields have known for decades:7

 Teaching is a complex activity that, when done effectively, is closely 

tied to the success of learners.8

Many times, instructional designers work with subject-matter experts, such 

as training facilitators, teachers, and/or other faculty members, to help them 

translate their content knowledge into effective learning experiences, usually 

for formal learning contexts. Often, these content experts have less familiarity 

with effective instructional practice; hence, instructional designers introduce 

them to key principles and help them incorporate more effective methods. 

Instructional designers help their clients think more critically about a range of 

issues related to instruction, including the needs of learners, curricula, learning environments, and associated policies.9

Instructional designers generally use systematic models and methods, such 

as the systems approach, backwards design, successive approximation model, 
Instructional Designers and Learning Engineers | 305 

and the Kemp instructional design process. Their approach usually involves 

identifying desired outcomes and determining the skills, knowledge, and attitude gaps of a targeted audience. They apply theory and best practices to plan, 

create, assess, evaluate, select, and suggest learning experiences to close those 

gaps.10 Instructional designers may be involved with the entire instructional 

process or with portions of it. For example, early in a project, they’re often involved with the systematic review and critical appraisal of existing materials. 

Using research and theory, instructional designers may also conduct analyses 

before the actual instructional design and development occur. Later in the 

process, instructional designers may emphasize the importance of assessment 

and evaluation, to ensure learning experiences have met their intended goals. 

A common theoretical and practical understanding of innovation also contributes to instructional designers’ work, and the best instructional designers 

ensure their clients, fellow educators and trainers, and leadership recognize 

how the different tools, processes, materials, and innovations that make up 

learning systems can enhance their learning offerings. Hence, instructional designers need to additionally have a creative spirit of design,11 including 

an imaginative, creation-oriented, and interdisciplinary character as well as 

the creative spirit to remain flexible and perceptive in their practice. That is, 

despite the proliferation of formal processes, such as instructional systems 

design, instructional design remains an art—albeit one firmly grounded in 

science and theory.

A learning engineer is someone who draws from evidence-based 

information about human development—including learning—and seeks to 

apply these results at scale, within contexts, to create affordable, reliable, 

data-rich learning environments. 

Bror Saxberg, Ph.D., M.D.

Vice President of Learning Science, Chan Zuckerberg Initiative
306 | Modernizing Learning

“Designing is a process of pattern synthesis, rather than pattern recognition. 

The solution is not simply lying there among the data... it has to be actively 

constructed by the designer’s own efforts.”12

CHANGING CHARACTER 

OF LEARNING

The growth of technology and access to learner data has led to advances in 

learning science and made the learning environment more complex. This, in 

turn, affects the roles of instructional designers, who must now interact with 

a variety of formal and informal modes of learning, social and experiential 

learning theories, as well as new tools, processes, and people. This complex 

infrastructure has been called the “learning ecosystem.” It encompasses the 

physical and mechanical elements of educational and training environments; 

the theories, processes, and procedures that drive their use; and learners’ 

(complex) relations to and interactions within that environment. This includes 

all elements that make up learning, from the formal classroom and those traditional instructional activities, to the technologies used to support informal 

learning. The complexity of the future learning ecosystem is turning instructional design into an even more dynamic activity, where designers must be 

aware of how all these elements come together, how each works, and how to 

best orchestrate learning across time, space, and media. 

These advancements have similarly transformed the expectations of leaders, 

educators, trainers, and learners, and at the same time, they’ve created an 

abundance of choice for anytime/anywhere learning. The strategic challenge 

is that, unlike when learning occurred primarily in a classroom with limited technology options, today there are many resources available in personal 

learning ecosystems, classrooms, training programs, and beyond. Given that 
Instructional Designers and Learning Engineers | 307 

most of these new resources rely on technology, the challenge is no longer 

about mastering a few platforms in a constrained environment—it’s about understanding the benefits of multiple resources, maintaining awareness of the 

wide variety of capabilities, choosing the best ones for learning, and balancing the entire ecosystem of multiple resources in a way that provides greater 

support overall. Such rapid advancements have made it ever more challenging 

for conventional training, education, and instructional practitioners to build 

effective strategies, tools, policies, and designs; hence, there’s need for a new 

player: the learning engineer. 

Learning Engineers

In December 2017, the Institute of Electrical and Electronics Engineers 

(IEEE) Standards Association Standards Board recommended creation of a 

new 24-month working group, called the Industry Connections Industry Consortium on Learning Engineering or ICICLE, to provide definition to and support for the burgeoning field of learning engineering. Creation of this group 

marks a groundswell of attention on the learning engineering field, although 

its original concept dates back to the 1960s, from Nobel Laureate Herbert A. 

Simon, who wrote at the time:

The learning engineers would have several responsibilities. The most 

important is that, working in collaboration with members of the faculty whose interest they can excite, they design and redesign learning 

experiences in particular disciplines. [...] In particular, concrete demonstrations of increased learning effectiveness, on however small a scale 

initially, will be the most powerful means of persuading a faculty that a 

professional approach to their students’ learning can be an exciting and 

challenging part of their lives.13

Learning engineering, as conceived today, is an interdisciplinary approach 

based on an in-depth foundation and education in proven theoretical models 

and methods, educational paradigms and instructional approaches, and sci-
308 | Modernizing Learning

entific and analytical methods. Learning engineers use data and knowledge 

of enterprise structures to help promote good decision-making in the use of 

learning ecosystem components. With its focus on data, and in using validated methods that put learning data to work in the service of improved learning outcomes and institutional effectiveness, this emerging field takes a step 

beyond traditional instructional design. Learning engineers do this, in part, 

by combining big data with design-based research to improve the design of 

learning experiences.14 Additionally, learning engineers use theoretical and 

practical understanding to scale innovations across the learning ecosystem.

Learning engineers can help with the complexities of integrating various technologies, workflows, interactions, and data-driven processes to enable learning. They may engage with widely varying technologies, including learning 

management and learning content management systems, mobile learning 

AI: In many ways it’s solving similar problems as before but doing it more 

effectively with data. For example, we can search and find content with a 

much deeper understanding of its meaning. We can get better at questions 

such as: “What’s the student really trying to learn? Can we find the part of 

a video that would be most helpful? How else can we make this experience 

easier for students?” 

Shantanu Sinha

Director, Product Management, Google

Former Founding President and Chief Operations Officer, Khan Academy
applications, course authoring tools, MOOCs, digital simulations and game 

environments, virtual/augmented reality, micro-credentials, learning applications and tool developments, learning records and analytics dashboards, 

video and other streaming content, and new applications involving wearable 

and IoT technologies. Though learning engineers may not necessarily write 

software code or serve as systems administrators, they can influence the design, development, integration, implementation, and use of a wide variety of 

technologies. They might, for instance, recommend AI algorithms, such as 

deep learning, to analyze data gathered in rich learning experiences to create 

a clearer picture of learners. This information can be used to inform how 

learning is supported, for instance, by deepening student engagement in their 

courses, improving the efficiency of teachers’ instructional methods, or providing learning tailored to individual needs.15

“Bringing together teams of collaborators with different kinds of expertise—

teaching, subject matter knowledge, instructional design, and data analysis—

is a prerequisite for realizing the full potential of learning system data.”16

The growing and dynamic learning ecosystem means learning engineers are 

likely to play much larger roles in the planning, design, development, and 

analysis of diverse and complex instruction. Learning engineers, like instructional designers, will be expected to anticipate changes or new developments 

in applicable technologies or in the instructional fields affecting their specialty areas and programs. They’ll also need to continually improve their 

instructional strategies to reliably identify best practices and opportunities 

for change. Accordingly, learning engineers need to possess a wide scope of 

competencies, including a foundation in learning science as well as the use of 

data to improve learning practice. They need to know good learning design 

principles, be conversant in learning analytics and enterprise learning technologies, and have some unique areas of relevant expertise, such as cognitive 

science, computer science, or human-computer interaction.

Instructional Designers and Learning Engineers | 309 
310 | Modernizing Learning

In general, learning engineers tend to focus more on technology and data-driven decision-making than do instructional designers. At the highest levels of 

expertise, learning engineers typically act as partners to provide leadership, 

advice, and guidance throughout an organization and to serve in key staff 

positions, such as a specialist at agency or major military command headquarters, or in a generalist capacity as an educational specialist at a school or 

university. Learning engineers’ focus on data could give them an inroad for 

working with learning professionals who need grounding in assessment or 

in how learning works, such as training facilitators, teachers, and faculty at 

educational institutions. Those drawn to evidence-based practices might be 

especially interested in working with a learning engineer. In a higher education environment, for example, learning engineers could provide a valuable 

I don’t think that from a military perspective that we’ve completely taken 

advantage of large data management. Here’s a great analogy: We have 

hundreds, if not thousands, of hours of full motion video, but how much do 

we actually analyze based on the current tools...? Eighty-plus percent isn’t 

reviewed in detail. Until recently, we were working on automating that and 

that’s one element I look at data management for—turning those mountains 

of data into decision-quality information. 

Thomas Deale 

Major General, U.S. Air Force (Ret.) 

Former Vice Director for Joint Force Development on the Joint Staff
Instructional Designers and Learning Engineers | 311 

service by helping to link research and teaching, both promoting current research into effective teaching and encouraging faculty members to conduct 

such research. Learning engineers can also work in many different industries, 

perform many different tasks at various organizational levels, and, indeed, 

work side-by-side with instructional designers and other learning professionals—but with a different focus.

Instructional designers and learning engineers should collaborate and partner 

to assess learning needs, develop strategies, and implement plans based on all 

the component parts and connections within the ecosystem in which learning occurs. Both instructional designers and learning engineers have valuable 

knowledge and competencies that can help make the most effective use of 

learning resources, and together, they can contribute to transforming how we 

think about teaching, learning, education, and training. 

IMPLEMENTATION 

Define the Roles

While they can work together and have some overlapping skill sets, there are 

important distinctions between learning engineers and instructional designers. Notably, while learning engineers’ skills are grounded in applied learning sciences, they additionally emphasize data science, analytics, user experience, and applied research. Learning engineers also have a greater depth 

and range of experience, including some expertise in the implementation and 

improvement of learning ecosystems—that is, in working with diverse, technology-enabled, data-driven learning systems. 

Before becoming a learning engineer, someone must acquire the highest levels of knowledge in learning theories, models of learning, data about learning, 

research into learning, and the management of learning. They’re also likely 

to need higher levels of technical experience than instructional designers. As 
312 | Modernizing Learning

such, unlike an instructional designer who can start at the entry level and 

develop skills over time, learning engineers must have more extensive educational backgrounds and prior experience. The mix of knowledge and experience, or, more specifically, the ability to filter expert knowledge through 

the lens of practical experience, helps characterize the learning engineering 

approach to instructional solutions. 

To be clear, education alone won’t give learning engineers the practical 

knowledge nor integrated experience they need to be successful. A typical 

learning engineer wouldn’t come out of an undergraduate program; rather, 

we’d expect a learning engineering protégé to build upon undergraduate work 

in education or a relevant technical field with applied experiences and subsequent advanced preparation. For instance, someone might first train and 

work as an instructional designer and then later seek additional education in 

the research, learning sciences, and data-based problem-solving elements of 

learning engineering. 

Education and Professional Growth

What would the education of a learning engineer look like? As discussed 

previously, they must have a solid grounding in learning science as well as 

experience with instructional design, curriculum development, evaluation, 

and other educational areas. They should understand statistical modeling approaches for education and training, analysis of large datasets, and the use of 

evidence to improve learning. Befitting the word “engineer,” they also need 

some background in math or science, to help them identify and solve complex, 

sociotechnical problems in logical ways.

We must be cautious in thinking about learning engineering as simply a university degree. Learning engineering should be a cross-disciplinary program, 

likely at the master’s or doctoral level. These programs should also be competitive. Universities should evaluate applicants for sufficient prior knowledge 
Instructional Designers and Learning Engineers | 313 

and experience. Entrants into a program could have various areas of relevant 

expertise, and the purpose of the program would be to engage them in developing a common vocabulary, breadth of awareness, and solid ability to examine data to identify learning evidence. 

A learning engineering graduate program could have various areas of focus to 

complement the vocabulary and data elements. For instance, a technology-focused concentration in a learning engineering program could incorporate artificial intelligence, simulation, augmented/virtual reality, intelligent tutoring 

systems, or UI/UX for learning. But at the heart of any program must be 

learning science and design. Using science and theories as guardrails is valuable for all types of learning professionals in creating engagement, establishing context, and promoting application. Though technology may be helpful in 

many cases, implementing technology is not the goal—good instruction and 

learning are the focus. 

In the end, the graduate from such a program should be able to design and implement innovative and effective learning solutions in complex systems, potentially at scale, and aided by advanced technologies when appropriate. They 

should be able to use data and a solid, theory-based evaluation framework to 

improve learning and assessment in practice. Whether applied to industry, 

government, military, or academic settings, these graduates should bring value above-and-beyond that provided by traditional instructional designers. 

Job Series, Titles, and Competencies

The path to the job of the instructional designer or learning engineer may begin 

with teaching in K-12 or higher education; working in technology in corporate, 

government, or military environments; holding an academic research position; 

or filling some other responsibility related to educating or training people. 

Because the U.S. Federal Government has a strict classification system for 

employment, and because it employs so many education professionals, it 
314 | Modernizing Learning

serves as a useful lens through which to view the learning engineer role. The 

Office of Personnel Management classifies jobs in the Federal Government, 

and its General Schedule outlines the occupational groups, series codes, and 

classifications of positions including their duties and responsibilities, descriptions, and standards.17 Each occupational group (such as the 1700 “Education 

Group”) is indicated by the first two numbers of a four-digit sequence, and the 

subspecialties in that group fall within the specified range, for instance between 0000 to 0099. The 1700–1799 occupational series covers education and 

training–related professions, such as “training instruction” (1712) and “public 

health educator” (1725). The requirements and description for learning engineering should be included within this general series. 

Currently, instructional design falls within the 1750 sub-series (i.e., the “instructional systems series”). It seems like a clear solution to expand this sub-series to incorporate the competencies necessary for learning engineers and 

related future learning professionals. For instance, the title could change from 

“instructional systems series” to “teaching/learning support and instructional 

systems series.” This would follow a trend in the industry acknowledging the 

importance of supporting teaching and learning, broadly. Also, more detailed 

language about the work performed by learning engineers, their education 

qualifications, and experience requirements could be added to the description. 

Correspondingly, the upper end of this job series should be reviewed to ensure 

that pay and benefits are appropriately aligned with the necessary experience 

and education. If we don’t reframe this series (or take similar actions), it’s 

more likely key learning engineering components will become lost within an 

organization or devalued in career planning or performance appraisals; we 

also risk learning engineers being conflated with instructional designers. 

The success of the instructional designer or learning engineer of the future will 

ultimately rest on how institutions and their leaders connect, communicate, 

support, and value those specialties. Learning engineers shouldn’t be seen 

as “one-time stops” or clearinghouse consultants for educational products. 
60YC: THE 60 YEAR CURRICULUM

The dean of DCE [Harvard’s Division of Continuing Education], Hunt Lambert, is leading this 

effort to transform lifelong learning, which is now a necessity in our dynamic, chaotic world. 

The 60YC initiative is focused on developing new educational models that enable each person 

to re-skill as their occupational and personal context shifts. The average lifespan of the next 

generation is projected to be 80-90 years, and most people will need to work past age 65 to 

have enough savings for retirement. Teenagers need to prepare for a future of multiple careers 

spanning six decades, plus retirement. Educators are faced with the challenge of preparing 

young people for unceasing reinvention to take on many roles in the workplace, as well as for 

careers that do not yet exist. 

On-the-job learning is familiar to most adults; many of us take on tasks that fall outside of our 

academic training....but our children and students face a future of multiple careers, not just 

evolving jobs. I tell my students to prepare for their first two careers, thinking about which is a 

better foundation as an initial job—but also building skills for adopting future roles neither they 

nor I can imagine now...Given this rate of change, education’s role must be long-term capacity 

building—enhancing students’ interpersonal and intrapersonal skills for a lifetime of flexible 

adaptation and creative innovation—as well as short-term preparation so that they are collegeor career-ready. Education must also advance two other goals beyond preparation for work: 

to prepare students to think deeply in an informed way and to prepare them to be thoughtful 

citizens and decent human beings... 

The 60YC initiative centers on the least understood aspect of this challenge: What are the 

organizational and societal mechanisms by which people can re-skill later in their lives, when 

they do not have the time or resources for a full-time academic experience that results in a 

degree or certificate? Thus far, attempts to address this issue have centered on what individual 

institutions might do. For example, in 2015 Stanford developed an aspirational vision called 

Open Loop University. Georgia Tech followed in 2018 with its model for Lifetime Education. 

The hallmarks of these and similar models center on providing a lifelong commitment to alumni 

that includes periodic opportunities to re-skill through services offered by the institution; 

microcredentials, minimester classes, and credit for accomplishments in life; personalized 

advising and coaching as new challenges and opportunities emerge; and blended learning 

experiences with distributed worldwide availability. I believe a possible third approach is to 

reinvent unemployment insurance as “employability insurance,” funding and delivering this 

through mechanisms parallel to health insurance...

Much remains to be understood about how 60YC might become the future of higher education. 

In my opinion, the biggest barrier we face in this process of reinventing our models for higher 

education is unlearning. We have to let go of deeply held, emotionally valued identities in 

service of transformational change to a different, more effective set of behaviors. I hope higher 

education will increase its focus on the aspirational vision of 60YC as an important step towards 

providing a pathway to a secure and satisfying future for our students. 

Excerpt from the The EvoLLLution online newspaper, 19 October 2018

by Christopher Dede, Ed.D., Wirth Professor in Learning Technologies, Harvard University 18
316 | Modernizing Learning

Instead, they should be leading the way to optimize experiences and systems 

of learning (which may or may not involve technology), and helping organizations meet their missions through the growth and evolution of their training 

and education programs. This will require learning engineers to work both 

together with other experts and on their own to navigate client expectations, 

integrate emerging capabilities, choreograph complex interactions, and help 

learners achieve more efficient and effective results. 

Conclusions and Recommendations 

As the learning ecosystem becomes more complex, those who teach others, 

whether they are facilitators, faculty members, or other professionals may well 

find it difficult to keep up with the changes. Instructional designers and learning engineers are specialists in education and training; they can help teachers, 

trainers, and organizations transform teaching and learning environments for 

the modern age, and they can also help fellow learning professionals expand 

their own knowledge and skills in the use of best practices for education and 

training.

Instructional designers and learning engineers have complementary skills and 

knowledge. They both have a thorough grounding in the learning sciences and 

an ability to identify appropriate instructional interventions, but learning engineers will provide more data-driven solutions and focus more on advanced 

technologies and enterprise-wide elements. 

As these positions evolve, we need to ensure instructional designers and learning engineers have defined responsibilities and roles, so that both they and 

their organizations know whom to approach for different needs, understand 

how they work together, and can uniquely value each skill set. Overall, we 

need to recognize the benefits that both instructional designers and learning 

engineers bring and, thus, ensure they continue to play an active, valued role 

in project teams, organizations, and the larger learning community.
Governance for Learning Ecosystems | 317 

CHAPTER 17

GOVERNANCE FOR 

LEARNING ECOSYSTEMS

Thomas Giattino and Matthew Stafford, Ph.D.

The transition from independent systems to a learner-centered ecosystem is 

attractive to learning professionals who have previously undergone a similar 

evolutionary process in their field. Readers may recall the first forays into 

online learning, which largely emerged within individual programs, departments, or colleges. As enrollment and interest grew, other organizations went 

online as well, resulting in duplication and increased costs. In most instances, 

an overarching entity—an agency, industry, school district, university, or university system—stepped in to harmonize the e-learning systems, standardize 

their technology and approaches, and ensure e-learning results were captured 

and reported similarly. Today, as the learning ecosystem reaches maturity, the 

question emerges: who will run it and how? 

In dealing with competing and constantly changing demands, limited resources, 

a vast array of products and capabilities, and a need for integration across 

their systems, learning professionals recognized the need for an overarching 

governance structure.

Heraclitus of Ephesus noted, “Life is flux; the only thing that is constant is 

change.” Learning professionals will certainly agree; their field has changed—

and continues to change so rapidly that it’s difficult to keep abreast of developments. The proliferation of content, the myriad delivery modalities, and even 

the collective understanding of how the human mind actually learns have 

 

 
318 | Modernizing Learning

driven these professionals into near-constant reconsiderations of their field 

and all it encompasses. Learning professionals have reacted to this flood of 

capabilities by stitching together patchwork systems of systems. As teachers 

approach them with new requirements, usually coupled with a request for 

a new technological capability, learning professionals have expanded their 

patchworks accordingly. The result is a workable set of individual tools, but 

only just. Often, learners and teachers have to switch between capabilities—a 

tool for audio/video, another tool for asynchronous chat, still another for synchronous collaboration. It is a “time of plenty,” but it is also plenty confusing!

Learning professionals are starting to describe these product-and-service 

composites as “ecosystems,” with the term “ecosystem” adopted from biology. Scientists describe ecosystems as groups of living organisms interacting 

with one another and with their environment, with a high level of interdependence. Some ecosystems, such as an ecological biome, are ungoverned, but 

others have some centralized mechanisms. A good example of this scientific 

understanding is the human body. The various organs each perform specific 

functions, but they work together, within an environment that provides oxygen and nutrients, to ensure the overarching system (the body) functions 

successfully. It’s a complex system of systems that’s also managed centrally, 

as all of these functions are governed by the human brain. 

 Without centralized governance, the various 

 components of an ecosystem cannot maximize 

 effectiveness and efficiency. 

For our learning ecosystem to function optimally, it needs centralized coordination, but where should it come from? An initial, obvious answer is to 

look towards technology vendors. For instance, Apple was an early leader in 

the system-of-systems technology movement. Apple realized it could increase 

its market share by making all of its devices work with one another and by 

simultaneously allowing users to personalize their networks, build content, 
Governance for Learning Ecosystems | 319 

and control their cross-platform experiences. Microsoft and Google followed 

suit. In each case, the connection between customers, their hardware, online 

capabilities, and content increased the effectiveness of each component and, 

in turn, its value to customers. 

Looking to large-scale technology or media companies to orchestrate the 

interoperable systems, implementation and operation processes, ethics and 

norms, and organizational policies for a learning ecosystem, however, is a 

risky prospect. The learning ecosystem concept necessarily involves many 

diverse components, likely derived from different vendors, across organizational boundaries, and for different phases and aspects of learning. Seeking 

centralized oversight from a single corporation risks “vendor lock” or confinement to potentially expensive and proprietary solutions. Further, many key aspects of governance extend beyond technology, media, data, or delivery. Each 

organization will want to answer these sorts of questions for itself, away from 

the commercial interests of even the best-intentioned industry organizations. 

For instance, how an organization chooses to use learners’ data, how tightly 

coupled talent development systems are with human resources functions, and 

how best to negotiate between stakeholders’ competing requirements are all 

key governance considerations.

Most of our challenges have been cultural and political, not 

technical nor operational. If people can see the big picture, and 

they can see where they are and why it makes sense, it can be very 

beneficial. If I can get them to see it, then they can understand it, 

and more importantly, they can carry the message to the next office 

because it makes sense. 

Reese Madsen

Senior Advisor for Talent Development, U.S. Office of Personnel Management;

Chief Learning Officer, Office of the Secretary of Defense (Intelligence and Security)
For the most part, too, education and training vendors have been less concerned about governance and more concerned about sales. Governance is a 

customer concern. So then, the question for customers—for those organizations who design and deliver learning—is: How do we create a governance 

structure that both centralizes general oversight of the ecosystem while simultaneously maintaining necessary flexibility that allows for content ownership 

by communities, data ownership by users, and tool creation by developers?

We’re such a small state that we can’t build our own systems. This 

means we need to be the best “masher-uppers.” We’ve worked with 

other new England states, but now we’re focused more broadly. 

It’s not so much urban versus rural, it’s that we’re an outlier, a 

progressive state that’s always focused on the individual learner. 

We’re in a slightly different place than other states because we’re 

not a top-down, centralized education system. Rather, we put a lot of 

emphasis on local control. 

Daniel French

Secretary of Education, Vermont Agency of Education
Governance for Learning Ecosystems | 321 

E PLURIBUS UNUM 

(OUT OF MANY, ONE)

A look back through American history provides an instructive example of 

how one might develop a governing structure for an ecosystem. Like the independent systems of the first educational forays into online learning, early 

American settlements existed in relative isolation from one another. The settlements were responsive to their inhabitants’ needs but looking holistically, 

there was a great deal of overlap and duplication in governmental functions. 

Each settlement handled its security, infrastructure, communications, and 

transportation needs often without even considering other settlements. As 

these settlements grew, interdependencies developed to create colonies. Each 

colony had its individual identity, its own governance structure and, as with 

settlements, only limited concern for the wants and needs of neighboring colonies. This changed, however, with the arrival of a common threat.

The move toward independence from England, which precipitated the arrival 

of what was then the world’s most capable military force, drove a loose alliance among colonies. At first, the colonies attempted to keep their independent identities, with primarily decentralized control; however, this first governance structure, the 1777 Articles of Confederation, proved a failure. The 

Articles failed to create a sufficiently strong, centralized government capable 

of guiding the fledgling nation. This resulted in infighting and made the central government unable to overcome challenges or capitalize on opportunities 

collectively. 

As the weaknesses of this confederated approach became obvious, representatives from across the colonies—the men who became the Framers of the 

Constitution—gathered to reconsider their centralized form of governance. 

Some argued emphatically for simply modifying the Articles, retaining the 

balance of power at the colony level. Others took an enterprise approach, ar-
322 | Modernizing Learning

guing that only a strong, centralized government would be able to quell the 

bickering that had made the Articles-based government so ineffective.

In 1788, the Framers’ U.S. Constitution was ratified, implementing a unique 

“ federalized approach”—a state within a state in which the former colonies 

(now “states”) were provided the authority for tactical issues, while the 

centralized government retained supreme power and oversight to deal with 

those issues affecting the enterprise (the entire nation). Such a “ federalized 

approach” to governance is an ideal structure for learning ecosystems!

The evolution from a loose affiliation of learning-focused entities, each with 

its own needs, systems, and rule sets, to an overarching centralized governance solution parallels the Air Force’s experience in designing and deploying 

its “Learning Services Ecosystem.” From the authors’ interactions with other 

agencies, the evolutionary track is remarkably similar for a wide variety of organizations, whether from the industry, academic, or government sectors. In 

each instance, success was predicated on the organization’s understanding of, 

and commitment to, an enterprise solution coupled with the ability to receive, 

evaluate, and act on the varying needs of all organizational constituencies 

within the ecosystem. In other words, where governance has proven most successful, there’s been an intentional balance between individual constituents’ 

needs and the centralized needs of the community. 

Since humankind first saw the need to join together to satisfy common needs, 

there has been some form of governance. Learning ecosystem governance is 

no different. An effective governance structure is born out of a small group of 

professionals who decide to combine their individual needs, capabilities, and 

resources to provide better support for, and service to, their organizational 

constituencies. These professionals come together to discover the breadth of 

the organization’s stakeholders and the key issues to be addressed. They then 

work across the organization to select representatives—the framers—who 
Governance for Learning Ecosystems | 323 

discuss the issues, create an ecosystem charter, and manage its governance 

over time. It’s a labor-intensive and emotional process, but when successful, 

it’s an extraordinarily fulfilling undertaking. 

IMPLEMENTATION

The process through which ecosystem administrators can design and implement a governance structure, necessarily includes the following steps:

Step 1: Identify Stakeholders and Select Framers

The first step in establishing governance necessarily involves identifying the 

breadth of inclusion: Which entities (colonies) will be included and which 

will be left to fend for themselves? Next, there has to be an opportunity for 

the entities to come together to share their wants, needs, expectations, and 

resources. These stakeholders will become the initial architects of the ecosystem governance structure. To make this opportunity successful, organizers 

must ensure the appropriate representatives are selected to participate. These 

representatives will become the “framers” of the new ecosystem charter. Organizers can consult with stakeholders for nominations but may also ask to 

have certain personnel appointed for their special skills or knowledge. 

Because of the technological focus of ecosystems, organizations are likely 

to send representatives from their most technologically advanced programs: 

technology experts who understand systems, data, and the capabilities available in the marketplace. This is expected and desired; however, representatives from all stakeholder groups need to be included as well. Collectively, 

the framers will need to understand the entire organization’s needs, products, 

processes, and capabilities. Without a holistic understanding of the organization, the framers are likely to ignore key constituencies or issues.
324 | Modernizing Learning

The framers should also include members who can think locally, addressing 

individual requirements and concerns, while also thinking globally to understand an enterprise perspective. It’s not always possible to find people who 

can do both; so, organizers should try to find a balance among the selected 

members to ensure all the constituencies are heard. The result shouldn’t be 

a patchwork of individual interests, rather the collective perspectives should 

inform an overarching strategy for addressing the broadest array of requirements and desires. 

Step 2: Select Issues

Once the constituencies are determined and framers selected, organizers will 

need to consider the breadth of topics to discuss. The selected framers will 

undoubtedly expand the discussion when they meet, but it’s necessary to have 

“an entering argument”—a list of key questions to answer. These will vary 

with each organization’s unique situation; however, the following brief list 

might prove helpful in building a governance conference, as they are somewhat common to most organizations: 

MEMBERS

1. Who determines who “ joins” the ecosystem? One centralized administrative function involves determining who may “join” both in terms of people and 

organizations who want to belong, and also in terms of systems and capabilities that constituencies might want to integrate into the ecosystem. The governance structure must provide avenues for entry and, simultaneously, ensure 

new people and new capabilities aren’t injurious to others within the enterprise. 

2. How will constituencies be replaced? Representation is foundational to the 

success of a governance structure, as it ensures constituencies have a voice 

in the design, development, and direction of the ecosystem across its life. 

There’s a risk in representation, however. Constituencies need to be heard, 
Governance for Learning Ecosystems | 325 

but the governance structure must ensure no single constituency takes control 

of the ecosystem to the detriment of others. In addition to rules for expected 

behavior, mechanisms are needed to censure misbehaving representatives or 

shed inactive ones.

3. How will the governance structure be organized? There are multiple approaches; however, an approach needs to be selected, coordinated, approved, 

and promulgated so all constituents understand where their representation 

lies, where decision-making authority lies, and where they can go to request 

reconsideration of their proposals should they be denied. The model adopted 

by the Framers of the U.S. Constitution (the federalized approach) is worthy 

of consideration: The centralized (“national”) government oversees enterprise-level concerns while subordinate organizations (“states”) have the capability to make certain changes to keep their operations moving. 

Members

Who determines who joins the ecosystem?

How will constituencies be represented?

How will governance structure be organized?

Policy

Who is responsible for establishing centralized policy?

Who will enforce policy?

How can enterprise-level functions be supported?

Resources

Who will provide the resources and how?

Who will provide support and how?

Processes

How can the ecosystem address change?

How can the ecosystem remain relevant and responsive?

How does the ecosystem interact with partner/other organizations?

How will users experiment and adapt?
326 | Modernizing Learning

POLICY

1. Who’s responsible for establishing centralized policy? Like the federalized 

approach to U.S. governance, some functions and decisions will affect all 

constituents, while others are best handled locally. It’s necessary to determine 

the functions that affect multiple constituencies, as well as the constituencies’ 

needs and processes for managing these centralized functions. How will aggregate requirements be identified, needs agreed upon, decisions made, and 

results promulgated across the ecosystem? 

2. How will the ecosystem address change? Change is difficult. Framers will 

need to consider a variety of potential scenarios to devise a system responsive 

to change. The following scenarios present examples framers might consider:

U.S. AIR FORCE LEARNING ECOSYSTEM GOVERNANCE

The U.S. Air Force is deploying their Air Force Learning Services 

Ecosystem. Air Education and Training Command built the ecosystem 

and also established its charter, a managing body that oversees its 

operation, and its policy and support structures. 

The ecosystem’s governance structures were adopting from the model 

prescribed in the IT Infrastructure Library, the British Government’s guide 

to IT service management. It’s a hierarchical model, much in line with the 

approach prescribed in the U.S. Constitution. For the U.S. Air Force, at 

the enterprise level, there’s Force Development Governance, overseeing 

how the Service will develop Airmen, how many will be developed, and 

in which areas. Below that, there’s an operational level of execution—Air 

Education and Training Command—overseeing the specific programs 

supporting Force Development and IT/Educational Technology. 

Air Education and Training Command manages ecosystem operations 

with a level of decentralized execution, so stakeholders can address their 

own concerns, but where users’ concerns have the potential to affect 

the entire ecosystem, they’re elevated for an enterprise-level solution.
Governance for Learning Ecosystems | 327 

• A new training program is created per a senior leader’s directive. Its 

administrators wish to claim to high levels of synchronous bandwidth. 

Ecosystem administrators need to know how this will this be funded. 

• Multiple games and simulations run within local systems. Ecosystem 

administrators will have to determine which will migrate to the 

ecosystem and what opportunities might exist for sharing technological 

advances inherent in the best of these with other ecosystem users. 

• Senior leaders have opted to increase the workforce. Ecosystem 

administrators will have to determine how the enterprise will support 

this increase in throughput. If external education/training is required, 

they will also have to ascertain how the ecosystem will track learning 

occurring outside of the organization.

3. Who will enforce policy? This is an important consideration, as constituencies will often bring special talent to bear to change or incorporate new 

capabilities, software, or hardware into the ecosystem. How will unauthorized 

variations be detected and how will they be handled?

RESOURCES

1. Who will provide support and how? Support is a complex topic and one 

often overlooked in the rush to bring aboard new capabilities. Systems tend 

to come with “a maintenance tail” to keep them functioning effectively and 

current to industry and security standards. More importantly, users—be they 

teachers, learners, data analysts, or records keepers—need support too. The 

governance framers, in their desire to balance enterprise-level and individual constituency-level concerns, may opt for the federalized approach, where 

some level of support is provided locally and other support nodes are centralized for the entire ecosystem. Support is often a major hurdle for framers as new ecosystems come on line: Users will want to retain their existing 

support capabilities while ecosystem administrators tend to favor centralized 

approaches. This is a critical resource consideration.
328 | Modernizing Learning

2. Who will provide resources and how? This question should drive framers 

to discuss the sources, types, and quantities of resources required, and who 

can provide them. It’s a broad category, encompassing money, manpower, 

machines, infrastructure (facilities, electricity, internet capability, etc.), and 

much more. Some resource considerations follow:

► Funding – Centralized funding is attractive for constituencies but, without their investments, they may find it easier to strike out on their own 

when decisions don’t go their way. Framers shouldn’t underestimate the 

power of constituencies “having skin in the game!” For government entities (and some non-government organizations as well), we often find 

financial resourcing extraordinarily complex, as funds are split across 

organizations and labeled for very specific expenditures. “Pooling resources” becomes surprisingly difficult, which creates a risk of promoting individual actions and encouraging redundancies. Framers should 

ensure their resourcing strategy doesn’t create “insurgencies” within 

their organization.

► Manpower – The pooling of manpower is often recommended as an approach to enhance efficiency; however, it’s often predicated on a notion 

that dispersed manpower possesses some level of excess capacity that 

will be employed most effectively if amassed. That’s not always the case. 

If five people, working in five organizations, are overwhelmed with their 

existing workloads, having them bring their workloads to a common 

location will simply increase the difficulty they experience serving their 

former constituents–making them even more overwhelmed. So, while 

there’s often value in centralizing some functions, care must be taken 

to be realistic in the level of effort required and to find the best balance 

between local and centralized labor resources. 

► System integration – For good reasons, constituencies will often argue 

to retain their systems. Training and conversion costs, and the trauma 

of switching systems, are very real concerns. Yet the governance struc-
Governance for Learning Ecosystems | 329 

OVERALL, TOP DOWN CONTROL OF SCHOOLS IS PROBLEMATIC. We 

need more of a resource model that focuses on the question, “How 

can the top help you do your job?” to encourage more autonomy, 

diversification, and innovation. When you have a really bloated 

bureaucracy, it doesn’t help people. 

Benjamin Nye, Ph.D.

Director of Learning, Institute for Creative Technologies, USC

ture will have to find efficiencies and ensure that systems work together. 

Recognizing potential duplications and overlaps, and dealing with these 

fairly, is an important part of technological governance. Those constituencies forced to adapt must receive sufficient assistance to ensure their 

operations are not adversely affected.

PROCESSES

1. How will users experiment and adapt? The educational technology market 

is changing constantly. Users will want to explore new capabilities to meet 

their organizational needs. Restraining creativity will frustrate users and 

drive them out of a centralized-governance approach. The best way to counter 

this is to provide space for experimentation—an “innovation sandbox.” This 

approach supports the insatiable appetite of some users for tinkering; however, it also drives these users to follow system protocols that govern the entire 

ecosystem. This approach benefits all: The ecosystem is not corrupted by experimentation, and those experiments that prove worthy of pursuing have already demonstrated an ability to function within the ecosystem successfully. 

An additional benefit is the way this approach aids in “policing.” The innovators who leverage the sandbox are much less likely to try to sneak capabilities 
330 | Modernizing Learning

onto the ecosystem (with potentially dire consequences for the enterprise) if 

they have an approved place and method for experimentation as well as a way 

to advance their successful innovations to the central governance structure for 

adoption into the ecosystem.

2. How can enterprise-level functions be supported? There are organizational-level decisions that must be addressed within the ecosystem, such as the 

workforce end-strength, qualifications the workforce must meet (learning 

needs), and the transcripting or certification of learning. All of these questions, and many more, must be considered by the framers establishing ecosystem governance. For the Air Force, negotiating the enterprise-level functions 

required participation from several overarching working groups, including the 

Service’s Force Development Council, which addresses strategic-level considerations affecting training, education, and experiential learning; an Air Force 

Learning Council, which determines content requirements for specific programs; and an Air Force Educational Requirements Board, which determines 

advanced academic-degree and professional military education requirements 

for Airmen. Each of these strategic-level bodies has data requirements, and 

each produces decisions that drive ecosystem functions.

3. How does the ecosystem interact with external or partner organizational 

systems? If the learning ecosystem is set up to provide certificates, badges, 

or some other credential, are talent-management systems capable of leveraging those credentials for decision-making? Will supervisors have the means 

to verify employees are properly trained to perform specific tasks? The integration of the learning ecosystem into the overarching organizational IT 

structure is foundational to its value to the organization. This is complicated, 

requiring a strategic mindset to establish and maintain integration. Two examples will help explain an administrator’s concerns:

• A partnering organization wants a reciprocal arrangement through 

which their employees can learn within your ecosystem and receive 

credit electronically, delivered to their personnel records system. Your 
Governance for Learning Ecosystems | 331 

leaders want the same for employees who train in their programs. 

Ecosystem administrators will have to craft these reciprocal 

agreements and develop the data-transfer capabilities to make these 

arrangements successful.

• A local community college would like to partner with the 

organizational training unit to offer associate degrees. The college 

wants access to the employees’ training records as well as the ability 

to report courseware completions back to the ecosystem. Senior 

leadership agrees; they want this too.

4. How can the ecosystem remain relevant and responsive? There must be 

mechanisms in place to ensure stakeholders have awareness of what’s transpiring within the ecosystem and have a voice in its evolution. There also has 

to be some level of senior-leader oversight to adjudicate disagreements that 

arise between stakeholders and ecosystem administrators. Lastly, like the 

U.S. Constitution, there must be a mechanism for updating the governance 

structures and policies. How will the organization drive change in the ecosystem to ensure it continues to meet the needs of the future?

Step 3: Build a Charter

Once all issues have been debated and preliminary decisions made, the framers should produce a charter. This, in effect, is the ecosystem’s constitution, 

describing the manner in which it will function and prescribing the processes 

by which it will remain responsive to organizational and user needs. A published charter ensures common understanding of authorities, decision-making, 

and resource-allocation processes, and it outlines steps stakeholders can take 

to resolve disagreements or seek change. The charter should be coordinated 

through stakeholders’ organizations and concerns adjudicated by the framers 

before final, senior-level approval and implementation. Once approved, ecosystem administrators must adhere to the charter precisely. Doing so ensures 
332 | Modernizing Learning

transparency in ecosystem administration but also serves to reduce the number of complaints or offer a credible defense should complaints surface.

Returning to the Heraclitus quote that began this chapter, “Life is flux; the 

only thing that is constant is change.” Ecosystem administrators will face 

change. Charters are created for specific needs at specific moments in time. 

Those needs can change. The U.S. Constitution, for instance, was ratified 

in 1788. In the course of its existence, 33 amendments have been proposed 

by Congress and sent to the states for ratification. Of these, only 27 have 

been ratified and have become part of the Constitution. Arguably, each of 

these proposed amendments represented a disagreement between contemporary Americans and the Framers; disagreements that must be addressed and 

resolved. Through the ratification process, the nation keeps its governance 

aligned with the nation’s evolving needs. Ecosystem charters need to be similarly responsive. Change should be possible, but the change process should be 

sufficiently difficult, so the charter isn’t in constant flux. Should that happen, 

the charter will lose its power and meaning. All stakeholders should have a 

voice in changes to the charter, so they can weigh the advantages and disadvantage and respond appropriately.

Step 4: Coordinate (Ratify!) the Charter

There is a tendency, within modern organizations, to employ a hierarchical 

“coordination process” for the approval of organizational positions or initiatives. This seems logical; however, in returning to the example of the ratification of the U.S. Constitution, one can discover even more wisdom in the Framers’ approach: Although the Constitution established a representative form of 

government, where elected and appointed officials would bring the needs of 

their people forward for debate, it’s interesting to note this isn’t the system 

the Framers established for ratification of their governance structure, their 

Constitution. Instead of handing this task to legislative bodies— the established hierarchy of governance—the Framers authorized “conventions.” They 
Governance for Learning Ecosystems | 333 

understood the existing colonies’ governance structures might not be inclusive 

enough, so they authorized this approach. 

Conventions were held across the nation. 

Most had lax participation requirements, 

much more permissive than the requirements for a governmental position. As a 

result, a vast array of constituencies could 

step forward, air concerns, and identify 

strengths and weaknesses in the proposed 

constitutional-governance structure. 

Ecosystem administrators should be 

equally inclusive in establishing their 

“governance conventions,” to maximize 

inclusion. They should provide the charter to the various stakeholder groups to 

let them discuss it and provide input. Certainly technology experts must be consulted—but so must personnelists, organizational planners, and, of course, trainers 

and educators. They will guide organizers in building an ecosystem that delivers 

learning effectively. Since the ecosystem will produce data, it’s also important 

to include those people who will have to access and employ ecosystem data. 

Consider providing the charter to the registrar or human-capital record-keeping departments. Lastly, don’t forget the learners! To maximize effectiveness, 

the learning ecosystem will have to be designed, developed, and deployed 

with learners in mind. How will the system meet learners’ wants and needs 

if they’re excluded from the governance discussion? Consider “conventions” 

carefully; those excluded from consideration are likely to become the most 

resistant to the resulting governance structure.

We need a common 

space where key actors 

in postsecondary learning 

can coordinate without 

hampering innovation. It’s an 

important piece of bringing 

the systems together. We 

need touchpoints without 

over-programming. 

Amber Garrison Duncan, Ph.D.

Strategy Director, Lumina Foundation
334 | Modernizing Learning

Step 5: Build Responsiveness into Administration

Much in the same way that organizers cast a wide net in establishing their 

conventions, ecosystem administrators should ensure all stakeholder communities remain aware of and involved in the evolution of the system. This 

requires managers to identify and continually refine the requirements of the 

supported population. “Responsiveness” is the watchword, requiring managers receive and respond to needs quickly and accurately. 

In addition to responding, ecosystem administrators must be proactive in providing feedback to stakeholders on system operations. Metrics, for instance 

addressing support, system functionality and availability, and costs, are invaluable for ensuring stakeholders are attuned to the enterprise-level requirements of the ecosystem. These must be provided to stakeholders regularly and 

can also guide senior leaders’ resourcing decisions for ecosystem investments.

Although the governance structure has already been addressed, with the need 

for each stakeholder to have a voice in system administration, ecosystem administrators must ensure there’s transparency in this process. “Frequently 

School districts are a traditional, sole-service delivery model, and districts have 

exclusive rights over the learning needs of the students assigned based on residence. 

They have to be all things, to all kids, all of the time. This is impossible, and it’s arbitrary 

because it’s based on just where you live. If a kid wants something but the school 

doesn’t have it, we assume the kid is wrong but the system is right. For example, if a 

kid in a rural community loves art but the district doesn’t offer much art, we ask the kid 

to put the passion on hold and instead get excited about history or some other offering 

the district is good at. We say that the district is right and the student is wrong—in a 

deep profound way. But the kid and family are right and the system needs to adjust 

and adapt to provide those pathway options. Of course, districts can’t do it alone; they 

have to form partnerships.

Ken Wagner, Ph.D.

Education Commissioner, Rhode Island Department of Education
Governance for Learning Ecosystems | 335 

asked questions,” chat rooms for stakeholder feedback, meeting minutes for 

governance meetings, and regular, open communications between administrators and stakeholders are critical to building trust across the organization. 

Administrators should also alert users to current and upcoming problems, 

maintenance schedules, and actions taken to resolve problems. Too much information is better than too little in building trust. Resources are limited, and 

administrators invariably have to deny stakeholders’ requests. Trust and transparency aid considerably in how a negative response is received and perceived. 

Administrators must also engage stakeholders in bringing aboard new components. Cooperative efforts that maximize participation can expand interest in and support for new capabilities. In addition, administrators may find 

stakeholders who can benefit from such initiatives willing to share resources 

for their implementation.

Step 6: Address Grievances

Although cooperation is the aim, there will be disagreements. There must be 

an organizational “court of appeals” for those situations where administrators 

and stakeholders disagree. There must also be a level of strategic oversight to 

ensure the ecosystem and all its stakeholders are moving together to satisfy 

organizational needs. Most organizations have some level of “corporate structure” that facilitates strategy-making, execution, and decision-making. Ecosystem administrators must ensure their operations are included within that 

corporate structure. To ensure their senior leaders comprehend the value of 

ecosystem operations and the challenges faced, administrators should report 

regularly to senior leaders. Within the Air Force, for example, ecosystem administrators periodically send written reports to Major Command commanders and Air Force senior leaders. In addition, there’s an oral report—the State 

of the Command—that specifically addresses the ecosystem and is delivered 

by the Force Development Commander to the assembled body of Air Force 

senior leaders at an annual conference. 
336 | Modernizing Learning

SUMMARY 

This chapter described the evolution large organizations must make as they 

move from functionally isolated information-technology schemes toward 

enterprise solutions. The example of the American colonies’ transition from 

relatively independent polities, to loosely affiliated states, and later to interdependent states governed by a constitutionally ordained centralized government, provided a foundational metaphor to help readers orient to this 

evolutionary process. In our case, today’s functional, formerly independent 

learning institutions will need to come together. Although, they’ll still require 

some level of autonomy to address local events and requirements, events with 

We need a new approach that enables our people to 

innovate effectively across the DoD training and education domain. 

There’s good strategic guidance from our government leadership, but 

it’s not carried forward in a meaningful way because it’s negated by 

excessive and often misinterpreted policies. The problem is further 

compounded by competing interests, stovepipes, and lack of resources. 

A new approach should actively pursue and remove irrelevant 

administration, process, and governance that kill modernization and 

rapid-development initiatives. A new approach should get people from 

out behind their desks and away from “business by email.” Finally, a 

new approach should encourage face-to-face exchange of ideas, better 

cross-organization coordination, and dedicated investment in discovery 

activities outside of the traditional R&D mechanisms. Only by doing 

these things will we truly achieve the transformational goals expected 

of us working in the training and education space.

Dennis Mills

Program Analyst, Naval Education and Training Command, U.S. Navy
Governance for Learning Ecosystems | 337 

a broader impact must be handled at the enterprise level, across the learning 

ecosystem, to capitalize on opportunities, reduce costs, and avoid unintended 

consequences that can occur within this complex system of systems. 

Organizations can take steps to ensure their governance structures remain 

attuned to stakeholders’ needs, are stable to ensure dependability, and are 

simultaneously flexible to support growth and innovation. Establishing these 

internal governance structures is a critical first step in deploying an effective 

learning ecosystem supports, and remain concurrent with, evolving learning 

needs and opportunities.

As with nations, initial governance starts at home, by establishing processes, 

policies, rules, and norms for managing an ecosystem within a given organization. Over time, different organizations will encounter more opportunities 

for interdependency, and new external governance structures, like the United 

Nations or World Trade Organization in our metaphor, will be needed. As we 

ponder the enormity of governance required for lifelong learning systems, 

however, it’s useful to be reminded of Gall’s law, written by systems theory 

critic John Gall:

A complex system that works is invariably found to have evolved from 

a simple system that worked. A complex system designed from scratch 

never works and cannot be patched up to make it work. You have to start 

over with a working simple system.1

The most advanced learning ecosystem efforts will ensure all system components work together, that learning is captured and reported across organizational and temporal boundaries, and that the entire construct is learner-focused, giving users control over their learning and, to the extent possible, their 

learning environment. Yet, for such an overarching system to be successful, 

it must start locally, with well-developed processes and mature governance 

methods within individual enterprises. Over time, then, we can extend those 

approaches out, building the complex, lifelong learning ecosystem across our 

societies—albeit, one step at a time.
Everything people want is obtainable, but there’s a great deal 

of myopic thinking, especially within the government. We hear 

too often, “We’ve never done it that way, so why should we change 

now?” The focus is too often less about the mission and more about 

the change. Having a good change agent is key. We need the executive 

branch pushing from the back and Congress pulling from the front. It 

needs to be a comprehensive system to be effective. 

Reese Madsen

Senior Advisor for Talent Development, U.S. Office of Personnel Management;

Chief Learning Officer, Office of the Secretary of Defense (Intelligence and Security)
Culture Change | 339 

CHAPTER 18

CULTURE CHANGE

Scott Erb and Rizwan Shah

Our current learning systems were developed in response to the industrial 

revolution and the accompanying shift from an agricultural civilization to an 

urban, manufacturing society. The focus of much of education and training 

was to produce people ready to enter the workforce with predictable, wellknown, replicable skills that matched the needs of the industrial economy. In 

order to produce these workers, the system required teachers who also had 

predictable, well-known, replicable skills in teaching. Thus, a system of “normal schools” was built to train teachers.1

 But as we shift to an information 

economy, a new set of skills not easily taught within the existing education 

and training framework will be required, which will drive a shift in the way 

we imagine, approach, and develop learning experiences. 

Rethinking learning from the industrial model to the information model 

will necessarily be disruptive to existing organizations. The adoption of new 

learning science methods and technologies will require change to their cultures, shifting away from incremental compliance cultures with established 

delivery and assessment methods to more fluid multiplatform and multimodal 

methods, coupled with pervasive data capture and advanced analytics. Organizations able to successfully navigate this cultural change will thrive; those 

unable to do so will be left behind. 

This chapter discusses some considerations for the change in culture that will 

be needed to modernize learning, remove barriers, and restructure incentives 

to inspire the organizational shifts needed in order to achieve the future learning ecosystem.

 

 
340 | Modernizing Learning

FEAR OF CHANGE

Undertaking significant organizational changes can create feelings of uncertainty, anxiety, and being threatened.2

 When the area of change is something 

as fundamental as learning, such fears can multiply.3 If not adequately addressed, these feelings can manifest as either passive or active opposition, 

resulting in immediate failures and resistance to future attempts.4

There are various human factors that make change difficult,5 some of which 

are particularly relevant for the future learning ecosystem. For instance, consider the fear of automation. The potential for AI to replace workers in the 

economy, including teachers, doctors, and lawyers, has been widely publicized in the popular press over the last few years.6

 This has had the effect of 

amplifying the natural fear of one’s skills becoming obsolete in a changing 

economy. 

Another, related example, involves the fear of losing of control. The imposition of change may make individuals feel their self-determination is being 

undermined, particularly when that change involves increased automation, 

complexity, and difficult-to-understand data analytics. Individuals might feel 

uncertain about their roles, the direction of the organization, or their abilities 

to contribute and maintain relevance.7 Team members who were instrumental 

in creating the current way of doing business may worry about the perception 

The culture shift—that’s probably the 

most difficult thing to navigate. 

Kurt VanLehn, Ph.D.

Professor, Computing, Informatics, and Decision Systems 

Engineering, Arizona State University
Culture Change | 341 

that the need for change means their way had failed. Similarly, those who help 

administer the current system (such as the present-day teachers, trainers, and 

instructional designers) may wonder whether they’ll be able to translate their 

current skills into the new environment—will they still be competent and be 

viewed as competent by others? 

“Nothing so undermines organizational change as the failure to think 

through the losses people face.” – William Bridges

Add to these underlying insecurities the fears of increased scrutiny. Data analytics, increasingly important in all aspects of learning and absolutely critical 

to measuring the effectiveness of changes in a learning environment, may 

cause concerns that instructors or program managers will be held adversely 

accountable if the data don’t demonstrate high levels of perfection. Learners 

may feel exposed and uncomfortable, as well, as the data we can collect and 

analyze gets richer and more actively informs a growing range of actions—

not just within a given learning episode but potentially affecting jobs, careers, 

and lives.

Another reason some resist change is because it looks like additional work. 

In general, production must often continue in the existing system while new 

systems are established;8 this is certainly the case for the future learning ecosystem. Add to that the new processes and requirements of the future system, 

the looming prospect of ongoing lifelong learning, and the complexity of it all. 

It seems like a daunting task.

Combined, these form a landscape that leaders of organizations seeking to 

innovate must understand and successfully navigate. Discomfort with change 

may manifest in different ways as an organization attempts to implement 

it. Within sufficiently established bureaucratic organizations, resistance can 

be accomplished by citing page and paragraph of existing policy, or by constructing unnecessarily onerous approval processes. People with a long his-
342 | Modernizing Learning

tory within an organization, and who may have seen several generations of 

leadership, may become passive resistors—intent on waiting out the latest fad 

while continuing to execute the status-quo processes they’re responsible for. 

However, for organizations to remain viable they must embrace—appropriate—change. If not, organizations that were once industry leaders can fall behind, or worse, close their doors altogether. In which case, those who resisted 

the change, and the leaders who failed to overcome that resistance, will have 

helped bring about the downfall that they believed they were preventing. A 

tough situation, but there are proven techniques to facilitate culture change 

and maximize the probability that the change leads to better outcomes. 

CHANGE MODELS

There are several change management models that are useful across a variety 

of settings, and which can inform options for creating acceptance to advance 

learning (see adjacent figure).9 The models vary in complexity. Some have 

only a few steps, but these fail to target all the necessary areas; others have 

more detail but risk draining resources and time. As such, no pre-packaged 

model is sufficient. Rather, a composite of these models, combined with lessons learned from working within government, military, and current education structures, must be utilized.

General Principles for Encouraging Change

CREATE AND COMMUNICATE THE UNIFYING VISION

To begin, each organization requires a unifying vision of why it exists and 

why it’s changing. Organizational consultant and motivational speaker, Simon 

Sinek, has written extensively on ways to develop this vision; he emphasizes 

that the first step is understanding the organization’s guiding purpose, and 
KOTTER’S THEORY

1. Increase urgency

2. Build guiding team

3. Develop the vision

4. Communicate for buy-in

5. Empower action

6. Create short term wins

7. Don’t let up

8. Make change stick

KÜBLER-ROSS’S

MODEL

5 Stages of Grief

1. Denial

2. Anger

3. Bargaining

4. Depression

5. Acceptance

BRIDGE’S 

TRANSITION MODEL

NUDGE THEORY

No set process. 

Help people change by 

nudging rather than using 

traditional methods.

SATIR’S MODEL

1. Late status quo

2. Resistance

3. Chaos

4. Integration

5. New status quo

ADKAR

Awareness of the need

Desire to support it

Knowledge of how

Ability to do it

Reinforcement to 

make the change stick

LEWIN’S MODEL

a set of 

goals to 

reach

Unfreeze

Change

Refreeze

MCKINSEY’S 7-S

New 

Ending Beginning

Transition

Staff

Strategy 

Structure

Systems

Shared

Values

Skills Style

3 Hard

Elements

4 Soft

Elements
344 | Modernizing Learning

that it’s most powerful when framed as a statement of belief: 10 We believe... 

The leader must own this statement deeply and personally, yet also develop it 

collaboratively with her core leadership team. The leadership team must also 

ensure that every member of the organization understands the vision—the 

WHY of the organization. 

Similarly, it’s important to incorporate a vision for the future, one that stimulates unity of effort and inspires individuals to take initiative to move forward. 

A compelling vision of what the organization looks like in the future helps 

generate the buy-in and initiative needed to implement change. Sinek stresses 

the importance of communicating why change is needed and reinforcing the 

message frequently. Communicating the vision once and expecting it to take 

hold throughout the organization is a recipe for failure.

Individuals exist within organizations, which exist within communities, which 

exist within the larger ecosystem. Accordingly, when communicating across 

this magnitude of diverse constituencies, shaping the narrative for each requires intentional consideration. While managers and administrators may be 

focused on inputs and efficiency, instructors tend to be more focused on outputs (e.g., how well are the students performing?). Communicating the WHY 

for change must acknowledge each team member’s role and remain grounded 

in the organization’s overarching purpose. 

Finally, helping the entire organization (not just the leadership!) contribute to 

this vision creates ownership, builds a common compelling story, and inspires 

initiative. It’s also likely to generate ideas that leadership didn’t consider and 

to reveal easy early wins to help build momentum. Open-ended questions 

can help drive creativity here, for instance: What does the new normal look 

like, feel like, and sound like? How do our students or employees say that 

they imagine the future? What feedback might instructors give to leadership, 

if the new system is working? What feedback would indicate that an experiment isn’t working? What new problems does success create? Are we ready 

to recognize and take on the new challenges? What are the characteristics of 
Culture Change | 345 

a learning organization? How might we change the way we communicate to 

improve organizational learning?

It’s also useful to identify key influencers at this phase in the change management process; they can help carry messaging throughout an organization and 

across different stakeholder groups. The influencers may not necessarily be 

the most senior people (those with the formal authority); rather, they should 

be those with the social leadership to influence the rest of the organization. 

Once adequate levels of initial awareness and buy-in have been achieved, the 

organization can begin experimenting with process or technology changes.

ENABLE EFFECTIVE COLLECTIVE ACTION

Innovative organizations rarely fail from lack of vision. Often, ideas are plentiful, while implementation is lackluster. Innovating, especially within large, 

established, and successful bureaucratic organizations depends not only on 

having a sound vision but also on the ability to manage the organizational 

disruption that change entails. However, it’s not the leader’s responsibility to 

design and manage the implementation plan; rather, it’s critical that the entire 

organization participate. The leader’s job then becomes to do only a few difficult things: (1) inspiring the team to pursue the “why” by doing things that 

generally move the organization in the right direction, at generally the right 

speed, (2) ensuring the team has the resources to make progress, frequently 

by removing resistance, and (3) creating safety for the team by putting the 

innovation and culture change risk on her own shoulders. 

The leader must resist, at all costs, the temptation to answer specific questions in any form of “just tell us what you want us to do.” Providing detailed 

instructions for HOW to achieve the WHY is all but guaranteed to derail the 

innovation and accompanying culture change efforts. The leader must give 

ownership to each team member (at the appropriate levels) to decide what 

to build and HOW to build it. There are a variety of ways for the leader to 

communicate this transfer of ownership, perhaps the most simple is to ask the 
346 | Modernizing Learning

questioner for his intent, followed by asking whether that intent enhances the 

organization’s WHY.

The team should then craft a process for how they will implement the innovation and the change-ideas of the team. While the team should craft the process 

themselves to ensure that the right domain expertise is incorporated and to 

provide ownership of the outcomes, some general principles should be followed to address common sources of resistance and their underlying causes. 

ANTICIPATE AND HANDLE RESISTANCE

Leadership’s approach to implementing change, and ultimately to creating a 

culture that thrives in rapidly-changing environments, must acknowledge the 

fears that change can cause, recognize how those fears manifest in the organization, reframe them into aspirations with a strong explanation of “why,” create safety for those who implement the change, demonstrate (rather than just 

state) that failed experiments are just as (if not more) important as those that 

succeed, ensure incentives are aligned to the new culture, and be persistent. 

The roll-out for new processes or technology implementations should take 

into account the sources and manifestations of resistance to the greatest degree possible. Although attempts at perfection here will undoubtedly result 

in unacceptable delays, failing to have a deliberate process that accounts for 

resistance will invariably corrupt the results of experiments. If resistance to 

a new experience is too high, the data collected will reflect the level of resistance rather than the effectiveness of the new process or technology itself. 

Notably, the design of the system itself also matters. Too often, early prototypes are designed for minimum functionality but lack corresponding reliability, usability, and user experience considerations—which distracts from 

the experiment and can turn stakeholders against the entire change process. 

For instance, the user interface is important. If the new tool takes more than 

cursory training to begin using, the experiment is not yet ready for the audience. A new technology tool should be easy to understand, easy to use, and 
Culture Change | 347 

make the end-users feel like they’re more effective with it than without it, all 

within a few minutes. A good rule might be “as easy to use as an iPad for 

a 10-year-old.” Failure to fully appreciate this will strengthen fears of competence, skill obsolescence, or more work. Human-centered design and user 

interface programming are complex and time-consuming, but users are so 

accustomed to technology that’s well-designed that failing to do so early on 

may have severe consequences.

The author of the book, The Checklist Manifesto, Atul Gawande, notes 

that he’s never seen the “Big Bang” approach to change succeed.11 That 

is, dictating a change from the top of the leadership structure to happen 

at a specific place and time hasn’t been seen to work. Clearly an approach 

that respects the intent of leadership while also preserving ownership of 

outcomes and processes, and inspiring innovation at the point of contact 

between provider and customer (in the old model, between student 

and teacher), is needed. This approach should be common enough to be 

replicable, but flexible enough to be rapidly tailored to specific cases, and 

to grow as an organization’s experience grows. Furthermore, it should 

be maintained deliberately to ensure that lessons learned in the change 

process are collected, understood, and disseminated. If critics see mistakes 

repeated, they’ll become more effective critics! We suggest creating a guide 

for introducing new projects within the organization. This guide should be 

owned by the innovation leader (who may also be the organizational leader 

or a senior member who reports to the leader), and used and updated by the 

project managers.

Another unique area of concern involves the use of learning data. Clarify 

upfront what data will be collected and how it will be used. Understanding 

learning outcomes and modernizing learning will require handling big data 

and advanced analytics. In learning environments, it’s tempting to focus most 

of our attention on students. Teachers, staff, and program managers will also 

want to understand that they and their data are safe. 
348 | Modernizing Learning

IMPLEMENT INCENTIVES AND REWARDS FOR EXPERIMENTS

Developing a culture that thrives on change depends on the ability to experiment—to innovate, to rapidly try out new ideas, and to learn from these 

attempts. That necessarily means change, innovation, and innovative organizations are dependent on failing early and at low cost. Consequently, leaders 

must not only create the time and resources for experiments, but also publicly 

reward experimentation—especially when it “fails.” Business, military, and 

government leaders are familiar with the value of publicly acknowledging 

team members for exceptional performance, but the norms against failure often make celebrating falsified hypotheses an unfamiliar event. 

Astro Teller, director of Alphabet’s moonshot factory, Google [x], has a method for doing so that may serve as a best practice for innovators in the learning 

domain. Teller explains that giving lip service to the idea of “failing fast” isn’t 

enough. Employees need to be free of the fear of punishment—and in fact 

truly believe they’ll be rewarded—for failing fast, that is, for learning and for 

rapidly sifting through possible avenues for innovation and change. As Teller 

recently explained in a podcast: 12

When one of our projects that actually has, like, a nontrivial number of 

people, at least a few people full time on it, ends their project...we bring 

them up on stage, and we say, “This team is ending their project today; 

they have done more in ending their project in this quarter than any of 

you did to further innovation at [x] in the quarter.” ...then I say, “And 

we’re giving them bonuses...You know what guys? Take a vacation, and 

when you come back the world’s your oyster. You’ll find some new project to start or you can pick which project to jump into, depending on 

which one’s going best....The word failure, and trying to get people to 

fail is a bit of a misnomer....Failure when it’s actually just “you got a 

negative result for no reason and it’s meaningless” is a bad thing. I’m not 

pro-failure; I’m pro-learning.
Culture Change | 349 

IMPLEMENTATION PLAN 

Connecting the general theories, methods, and models of change management, we recommend a hybrid approach that capitalizes on the key points of 

each. Six areas of focus are recommended when starting the culture change 

process for modernizing learning systems. 

Educate 

The first step towards preparing an organization to embrace the future learning ecosystem concept involves communication and foundational (re)education. Resetting the WHY of the organization is critical to not having to repeat the culture change process with ever-increasing frequency. The future 

learning ecosystem idea isn’t a defined end-state but, rather, a commitment 

to ever-evolving, learner-focused support via interoperable technology and 

other emerging capabilities. Our goal, therefore, is to foster an organizational 

culture that embraces change as a way of life rather than an organization that 

has successfully navigated from one static state to another.

There’s nearly always a fear of change. The goal is to reduce this fear by 

increasing education about the change. Extra time needs to be spent helping 

people understand what they need to achieve and, of course, why. It’s not just 

about garnering their buy-in, it’s about reducing their fear. Step one, then, is 

to ensure everyone is educated on the goals for the future learning ecosystem. 

For example, explaining the value of interoperability at the technological level 

and envisioning the new methods learners and teachers will use to operate in 

a human-computer shared space will be important. However, the next step is 

to listen: To carefully consider stakeholders’ fears and give them an opportunity to work through their concerns, contribute to the larger vision, and 

become ambassadors to the idea in their own ways. 
350 | Modernizing Learning

Support

Everyone needs to know where and how to get support, not just philosophically, but also from a management perspective. Within the U.S. Government, 

the Office of Personnel Management’s USALearning program provides an 

immediate go-to for the development of this system, and the ADL Initiative 

offers support for research associated with new aspects of it. Within higher 

education and K-12, other support systems are being developed; for example, 

the Lumina Foundation and U.S. Chamber of Commerce are working together 

to support employers and employees making this transition. Further, other 

standards organizations and professional societies, such as the IEEE, can also 

offer guidance and recommendations to government, academic, and industry 

constituents. 

Providing resources is another important aspect of support, whether those 

involve time, labor, or financial investments. We often see situations where 

people are given a new mission (e.g., “we expect you to increase employee 

engagement”) but aren’t given any ideas, resources, or support to aid the process. If people are expected to make changes, they’ll need resources to do 

so—not only to support the change, itself, but also to facilitate the overhead 

required by the change process. Commitment to change requires resources, 

You have to unleash people 

and empower them using 

climate and culture.

Ken Wagner, Ph.D.

Education Commissioner

Rhode Island Department of Education
Culture Change | 351 

and more than that, it requires a demonstration of “skin in the game” via the 

allocation of resources.

Buy-In 

What’s the return on investment? That’s the question at the highest level, 

but at personal levels, individuals need motivation and will ask themselves, 

what’s in it for me? (or WIIFM, usually pronounced as “wiff-um,” a common acronym in military). So, both quantitative, logical messages and more 

personal, evocative messages need to be crafted. That is, we need to consider 

both the ROI and WIIFM for teachers, instructors, managers, leaders, senior 

leaders, and learners, as well as for businesses, schools, universities, and government agencies. They need to understand why these changes need to occur 

and the pathway through the transition. They need to understand why it will 

help them personally and how it will be implemented and/or integrated with 

existing systems. 

Making transition feel easy is one of the most important challenges to tackle 

and among the most important one to get right. This book is meant to serve as 

an initial step in that process. It’s intended to help paint a picture of the “art 

of the possible” and take the first steps towards clarifying why these changes 

will improve the system; however, the specific buy-in rationale will be unique 

to each organization and stakeholder group. 

Multi-Messaging

It’s one thing to make changes within a small system or even within a department where like-minded or similarly oriented individuals reside. However, 

once change is nationwide and includes systems of systems as well as multiple 

communities, it requires multiple but complementary messages to be cultivated and disseminated. In this case, it’s necessary to achieve two primary 
The U.S. Marine Corps Training and Education Command 

(TECOM) Future Learning Group demonstrates how an organization can implement evidence-informed organizational 

learning processes to support innovation. Established in 

2017, the TECOM Future Learning Group is a special staff 

unit that advises the Commanding General of TECOM. Its mission is to seek and assess 

innovative methods and technologies to improve Marine Corps training and education. The 

figure above shows their process. 

Beginning with “new observations and exploration,” the group contributes to organizational learning by identifying current and future Marine Corps learning needs, competencies, 

gaps, and goals—and how they relate to the individual, group, training and education units, 

and the overall Marine Corps. Next, the group scans the horizon for emerging science and 

technology, such as augmented and virtual reality–based training simulations, adaptive mobile 

learning applications, and new methodologies for enhancing instructor development. They 

ask questions to explore the prototypes, test new methods and technologies, gather 

data and analyze it to form conclusions, and ultimately provide recommendations to 

TECOM leadership. These results and recommendations go on to inform organizational 

learning methods, policies, procedures, systems, and processes.

The TECOM Future Learning Group also shares the knowledge and practical applications 

they uncover with stakeholders both within and (as appropriate) beyond their Command. 

Findings are also integrated into current and future Marine Corps programs at the individual, 

group, and organization levels, and the results feed-back into their organizational learning 

process, driving the ongoing improvement cycle to enhance Marine Corps learning. The 

TECOM Future Learning Group’s work helps overcome the research–practice gap and 

more rapidly integrate new capabilities into Marine Corps programs. It also facilitates organizational culture change, encouraging more innovation in Marine Corps training and education—helping the Service move from an Industrial Age model of learning to an Information 

Age paradigm. 

New observations

and exploration

Ask questions

Assess, evaluate, test, or,

experiment to gather data

Analyze data and

form conclusions

Communicate results

and recommendations

Inform organizational 

learning methods, 

policies, procedures, 

systems, and processes

Integrate and 

implement new 

learning insights 

where relevant at the 

individual, group, and 

organization levels

Distribute new knowledge, 

insights, and practical 

applications to stakeholders

ORGANIZATIONAL 

INNOVATION 

USE-CASE

From Kendy Vierling, Ph.D., 

Director, Future Learning 

Group, USMC TECOM
Culture Change | 353 

goals: (1) ensure that the messages to the individual communities (e.g., K-12, 

higher education, employers, military, and government) are in line with their 

singular goals and (2) that there’s a meaningful message that transcends and 

unites these communities. In particular, we need to be clear that the benefit to 

both human development as well as to our national development lies in the coordination across these communities, that is, in collectively optimizing learning and development. The future learning ecosystem requires that we have a 

shared, single goal but with an unlimited set of pathways for attaining it.

Compliance and Policy 

Individuals in compliance and policy roles need motivation for accepting the 

future learning ecosystem concept. The stated goal for compliance and policy is often to ensure no problems occur—that is, to mitigate risk. This is 

especially true in the context of information technology and associated cybersecurity and data handling. But to evolve and optimize, risk must be taken. 

Consequently, we need to work with compliance and policy stakeholders to 

find the acceptable amount of risk. Who decides that? Who’s responsible if 

a breach occurs? These individuals have experience and knowledge, but are 

often engaged later in a change process, which creates obstacles to obtaining 

their buy-in or integrating their ideas into the fledgling system. We need them 

to give their direct input, be part of the conversations for planning, and help 

us move smartly towards this new vision of learning. 

Implement

Average projects (those not tied to cultural change) usually involves linear 

planning and straightforward management, with efficiency among their 

performance goals. However, in an innovation context, where culture change 

is a necessary criterion, different metrics need to apply. There’s a temptation 

to revert to traditional managerial methods, to emphasize speed, to reward 
354 | Modernizing Learning

only successful trials, and to backslide into comfortable processes. That will 

spell disaster for the future learning ecosystem—it cannot function without 

the genuine buy-in of stakeholders or the radical change of participating 

organizations. 

So, a slower but more deeply rooted approach is needed. Consensus-building 

working groups, community standardization efforts, and extensive communication will need to support collective implementation planning. This isn’t 

likely to be a speedy process. Leaders will need to balance a reasonable sense 

of urgency with a considered appreciation of the culture-change process. 

In terms of transforming the education enterprise, we need strict or 

serious policy—but not only policy, we also need resourcing, direction, 

and enforcement. The devil is in the details here because if you were 

to transform the training and education system into something that is 

really capabilities-based then the whole flow diagram is going to change. 

It wouldn’t be a block, like the “Class of 2028.” Instead, it would be a 

continuous flow, and you’d have a completely different process. Some may 

finish slower and present a gap in the pipeline. Others may get completed 

early and be ready to move on to the next phase. But if the entire system 

isn’t reformed, the next phase won’t be ready for them.

James Robb

Rear Admiral, U.S. Navy (Ret.) 

President, National Training and Simulation Association
Culture Change | 355 

Each organization will need its own experiments, incentives, and implementation plans, and these must be devised through collective participation. Similarly, the larger community—possibly at a nation-wide level—needs to coordinate. This may require extensive cross-cutting communities of practice 

and will certainly mean negotiation of experiments and incentives across domains. Just how this implementation plan is designed and what it will contain 

isn’t yet clear; however, it’s apparent that it must serve multiple levels—for the 

individual stakeholders, their local organizations, and the collective multi-organizational community. And it’s also clear that each organization will need 

to devise its own messages, measures of commitment, and ways of contributing to the larger vision. We are just beginning down this pathway. We have the 

opportunity to do so “the right way,” in concert and with thoughtful coordination; it’s important that we resist the urge to speed ahead with shortsighted 

implementation plans that sacrifice longevity for temporary achievements. “If 

you want to go fast, go alone; but if you want to go far, go together.” 13

Summary

It’s easy to avoid change, to play the cynic, wait out new ideas until the organization returns to the status quo, or find excuses to avoid uncomfortable actions (e.g., remaining in the “analysis paralysis” process). Individuals and bureaucratic organizations, in particular, are often remarkably clever at finding 

ways to avoid change. It’s also tempting to view the future learning ecosystem 

as simply another technology—as a thing that can be installed and activated, 

and then fueled with educational materials that instructional designers merrily create using more-or-less conventional methods. But this won’t suffice. If 

effective, the future learning ecosystem concept will extensively affect how 

we each live, work, and learn. It will affect organizational dynamics, societal 

systems, and maybe even the overall zeitgeist of our time. Such impacts can’t 

be achieved through technology alone. They require coordination, a shared 

vision, and commitment to it. They require a culture change. 
...you’ve got to be opportunistic in fixing problems so that 

you’re not just fixing one but rather fixing multiples. At the 

same time, you have to try to build and control the narrative; use it as 

a barometer and take some of the danger out of change. You realize 

you’ve reached where you need to when people start to give the story 

back to you. It helps to know that you’ve got a narrative from the 

beginning that does something that will keep people engaged but will 

also allow you to implement it later.

Jeffrey Borden, Ed.D.

Executive Director, Inter-Connected Education; Chief Academic Officer, 

Ucroo Digital Campus; Former Chief Innovation Officer, St. Leo College
Strategic Planning | 357 

CHAPTER 19

STRATEGIC PLANNING 

William Peratino, Ph.D., Mitchell Bonnett, Ph.D., 

Dale Carpenter, Yasir Saleem, and Van Brewer, Ph.D.

In this chapter, we explore some of the most immediate steps required to realize the future learning ecosystem across educational, academic, business, 

government, and military sectors. We discuss the larger system, including 

people, processes, and technologies, and recommend considerations related to 

its design, development, and implementation. 

Today’s Learning Journey

Currently, in the U.S., most children begin formal learning in the conventional education system. Primary and secondary programs follow a fairly linear, time-based model that creates a conservative, general trajectory where 

children progress through academic milestones more or less as an age-based 

cohort. Students are largely taught as groups in classrooms and provided with 

similar lessons and homework. Usually, these curricula focus on key areas 

of knowledge acquisition to include mathematics, reading and writing, science, and history, often with a few additional areas included such as art, music, physical education, and health. Frequently, development of self-regulated 

learning capacities as well as social, emotional, and physical competencies 

aren’t formally included, although some students may encounter outstanding 

teachers or participate in extracurricular activities that foster these abilities. 

As students approach postsecondary schooling, more differentiation occurs. 

They can choose elective classes (although often limited by local availability), 

 

 
358 | Modernizing Learning

and in some districts, school choice programs offer more diverse options such 

as magnet, charter, virtual, home, and private schools. Increasingly, students 

can even opt for fully online high schools, including relatively low-cost national and international programs.1

 Enterprising students, as well as their 

teachers and mentors, also have access to an increasing wealth of educational resources, which they’re exposed to at younger and younger ages, from 

sources including the National Academies, Khan Academy, TED, and various 

MOOCs, as well as associated resource repositories such as MERLOT, OER 

Commons, and Connexions. There’s also an unprecedented amount of informal (and sometimes questionable) online resources from YouTube, Wikipedia, 

and Reddit to countless other blogs, web sites, and apps. 

Once students graduate from high school, they can enter the public or private-sector workforce, seek additional vocational training, or matriculate to 

higher-education institutions. Postsecondary education traditionally involves 

two- and four-year degree options as well as trade and certificate programs. 

Colleges and universities also frequently offer advanced degrees in the form 

of graduate certificates, master’s degrees, and doctorates. While many schools 

still follow traditional methods, the higher-education sector is rapidly evolving with various new choices including competency-based degrees, fully online options, and hybrid programs. 

Increasingly, individuals can also acquire credentials outside of a formal higher-education institution; for instance, intensive “bootcamps” have become 

popular in fields such as software coding, project management, and cybersecurity. We expect this trend to continue and, in the future, we’ll see more and 

more varied credentials—including experience-based credentials earned outside of structured programs. In other words, we anticipate that more programs 

will be available to accredit individuals for their capabilities and knowledge, 

regardless of whether they acquired those competencies in formal or informal 

settings. This will substantially shift the way we view formal learning as well 

as many related human resources processes (e.g., recruitment and promotion). 
Strategic Planning | 359 

It’ll change the resumé, too, putting less emphasis on the jobs someone has 

held or the degrees earned, and more on his or her demonstrated capabilities.

After individuals enter the workforce, their learning journeys continue. They 

can seek vocational training and additional credentials, attend workshops and 

seminars, or pursue any number of informal and self-directed learning opportunities. Some companies also offer continuing education or professional development programs for their employees. In the U.S. alone, businesses spend 

roughly $90 billion annually on corporate training (as of 2018).2 These offerings range in formality. On the more formal side, there are programs such as 

McDonald’s Hamburger University, the “Harvard of the fast food industry,” 3

which trains more than 7,500 students a year,4 and Starbucks helps its employees earn first-time bachelor’s degrees online through their partnership 

with Arizona State University.5 Less formal programs come in many shapes 

and sizes, including corporate coaching and mentorship, developmental seminars, official and informal feedback, corporate e-learning and webinars, and 

numerous informal learning approaches. There are abundant resources available, and individuals and organizations have a whole slew of learning and 

development opportunities to choose from.

A complementary phenomenon to consider is the increased workforce “churn” 

(the word economists use to refer to people switching jobs). A large, longitudinal study by the Bureau of Labor Statistics found that baby boomers held 

an average of 11.9 jobs between the ages 18 to 50,6 and in another report, the 

Bureau found the median tenure at a given employer, across all ages of wage 

and salary workers, was only 4.2 years as of January 2018.7 Many expect to 

see continued workforce churn in future years, and, increasingly, we also anticipate individuals will have more careers across their lifetimes. As the pace 

of global and technological change continues, jobs will increasingly morph 

or become obsolete, and individuals at all levels of work will need to engage 

in additional learning as they progress through their careers. In other words, 

as discussed in Chapter 4, we’ll see increasing, and increasingly necessary, 
360 | Modernizing Learning

continuous, lifelong learning—including ongoing up-skilling and re-skilling 

for workers. 

Like the private sector, the public sector and military workforce face similar 

opportunities and challenges. In general, the same informal learning opportunities exist for these special populations. Agencies throughout the U.S. Government offer wide-ranging learning and development programs, covering 

the full gamut of formality. For example, the Office of Personnel Management 

hosts the Federal Executive Institute that provides training in strategic development for senior executives. The National Park Service provides access to a 

wide range of personal learning opportunities through its internal Common 

Learning Portal, and the Department of State uses its Virtual Student Federal Service program to provide on-the-job experiential learning opportunities 

to students around the country. But the U.S. Department of Defense is most 

notable among these agencies. It’s been considered the “greatest training organization of all time” 8

 and invests more funds in innovating education and 

training for its workforce than any organization in history, with the bulk of 

these efforts focused on programs for its military personnel. 

The DoD conducts formal individual, collective, and staff programs, and it 

actively encourages mentorship, peer-to-peer learning, and self-development. 

It employs the spectrum of learning modalities including in-resident and computer-aided instruction, simulation-based and embedded training, m-learning, 

augmented and virtual reality, and hands-on experiential learning. The DoD 

also has strict education and training requirements tied to assignment and 

promotion, and particularly for key accession points, it employs several standardized test, such as the Armed Services Vocational Aptitude Battery and the 

Tailored Adaptive Personality Assessment System. 

Unlike the private sector, service members generally have fairly constrained 

entry- and exit-points into the military workforce, and almost always, individuals separate from active duty military service before they fully retire from all 

work. Once service members separate from the military, they may return to 
Strategic Planning | 361 

the DoD or federal government as a civilian or contractor or seek employment 

in another sector. The latter often requires some retraining as well as careful 

translation of military-centric capabilities into private-sector roles.9

BUILDING TOMORROW’S 

LEARNING JOURNEY

Never before have so many high-quality opportunities for learning existed. 

Yet, tomorrow’s learning environment will be even more advanced as information and communication technologies, automation, and innovation continue to change how we interact, behave, and learn. We have great momentum, 

but how do we optimize this future system? Towards that end, we’ve integrated a set of 10 near-term strategic recommendations for the wider future 

learning ecosystem—drawn from across this book. 

1. Bridge existing silos

Public and private school enrollments in the U.S. have steadily risen over 

the preceding decades.10 The education, training, and talent development industries are similarly expanding along with corresponding increases in both 

not-for-profit and open-access resources. However, many of these expansions 

are happening in isolation. For example, learner records are typically housed 

in stovepiped data silos. Someone might spend 13 years in school as a child 

and then graduate with a high school diploma and a transcript with letter 

grades. Any additional specialties, sub-competencies, extracurricular activities, or other insights are usually absent from this documentation. The same is 

true of the university or vocational school outputs, and, typically, for previous 

work experiences, which may be documented (say, on a resumé) but are rarely 

assimilated as meaningful data. A similar story happens throughout service 
362 | Modernizing Learning

members’ and civil servants’ careers—there’s lack of robust data as well as 

a lack of permeability of data between formal and informal learning, and 

among academic, business, and government institutions. 

The future learning ecosystem will enable an environment where the different 

tools, technologies, and systems a person encounters can communicate data 

about his or her performance and the contributions of different activities to it. 

Key to this vision, the various systems will need to interoperate, collect and 

share meaningful data, and use that compiled information to promote tailored 

instruction. In other words, we’ll need greater interoperability across learning systems and, correspondingly, greater portability of learning-related data. 

Part of the change will also likely involve creating systems of learner-owned 

and managed data that use metadata to ensure authenticity, respect learners’ 

privacy needs, and broker across different systems. This will require a unique 

set of capabilities to accommodate 

security, privacy, architecture, and 

content, and it will place demands on 

development, deployment, employment, and assessment of learning systems. This technological architecture 

for learning forms the essential backbone of the future learning ecosystem—connectivity across time and space make the entire vision possible—

hence why interoperability, data specifications, and learner-centric universal 

profiles top our recommendations list. 

2. Foster full-spectrum competencies

Increasingly, schools and employers are recognizing the impact of social, 

emotional, metacognitive (self-regulatory), and physical development. While 

these competencies have always been important, there’s greater recognition of 

A universal learning profile will 

act as an external repository where 

individuals can hold their data and 

share it as desired to drive educational 

choices, personalization, employment 

eligibility, and personal growth.
1 Bridge existing silos 

• Enable system interoperability and data sharing

• Develop learner-owned universal learner profiles

• Research questions of security, privacy, architecture, and content-sharing 

2 Foster full-spectrum competencies

• Integrate social, emotional, metacognitive, and physical development

• Apply asset models (versus norm-referenced developmental models)

• Use personalized interventions across the developmental dimensions

3 Reveal and enable informal learning

• Acknowledge and integrate informal learning 

• Foster individuals’ self-regulated learning capabilities

• Make it easier for groups to engage in social learning 

4 Improve assessment

• Limit high-stakes summative assessments, particularly in K–12

• Integrate more formative, portfolio-based, and experiential assessments

• Make assessment data and feedback visible to learners 

5 Up-skill and empower learning professionals

• Help learning professionals develop the new capabilities they need

• Reevaluate the organization of learning professionals; focus on teams 

• Define and support the development of learning engineers

6 Plan for integration across learning and personnel functions

• More tightly integrate training and education with talent management

• Update organizational systems to better accommodate informal learning 

• Consider up-skilling and re-skilling programs

7 Facilitate a mindset shift

• From cognitive and teacher-centric to holistic and learner-centric systems

• From linear and time-based to personalized and nonlinear

• From isolated to more interconnected learning systems

8 Enable learning at scale, technologically and methodologically

• Build extensible, open-architecture components

• Research methods that support interconnected, lifelong learning

• Consider changes across the social and organizational structures

9 Design for convenience and equity of access

• Make UI/UX considerations paramount

• Ensure all learners have sufficient connectivity and access to technology 

• Carefully consider the social implications of the learning ecosystem

10 Ensure laws, policies, and governance keep pace

• Evaluate (and update) formal laws and policies

• Encourage participation in cross-cutting professional organizations 

• Develop cross-cutting working groups and governance processes
364 | Modernizing Learning

their impact on life-long functioning and as a result, motivation to more actively and intentionally support their development. Developing these “full-spectrum” competencies must begin at the earliest opportunities and should also 

continue throughout our lives, because as individuals grow they’ll encounter 

new challenges that will continue to test their holistic capabilities and require 

personal strategies to navigate effectively. 

It’s necessary to begin with the foundational years (kindergarten through 8th

grade), to widen the curriculum focus to include social, emotional, metacognitive, and physical development as part of formal education. Creating objectives for teachers in these areas provides the policy justification they need 

to spend time in the classroom explicitly focused on developing the whole 

student. Inclusion of these competencies, however, necessitates a shift to an 

asset model of growth, which places more emphasis on what students can do 

currently and what they need to learn next—as opposed to focusing on areas 

where improvement is needed to meet norm-referenced or “typical development” milestones. This shift from achievement-orientation to growth-orientation can also improve motivation to learn and promote lifelong interest in 

self-driven learning.

In the secondary and post-secondary institutions, we should continue to integrate these competencies into the more traditional curricula, while still recognizing that individuals grow and mature at different rates. In other words, the 

asset-model of development will need to continue into early-adult and adult 

education, which will create a greater need for personalized education as the 

range of individuals’ potential capabilities widens. 

In workplace settings, too, we anticipate employers will increasingly value 

and seek to hire for “full-spectrum” competencies, hence developing and 

measuring them throughout adulthood will grow ever more critical. However, 

developing these competencies and assessing their current levels within each 

person is challenging, particularly in the less-controlled post-secondary and 

workforce settings. Consequently, it’ll be necessary to better leverage and 
Strategic Planning | 365 

be able to measure the impacts of informal and nonformal learning on these 

outcomes. Data that meaningfully capture these experiences, as well as the interests and noncognitive abilities (social, emotional, and physical) individuals 

demonstrate in these settings, can help inform these assessments as well as 

drive future learning and development opportunities, encourage learners’ motivation for self-regulatory learning, and help stitch together learning across 

separate episodes. Among other things, this will also require “lifting the veil” 

between workplace and learning-place settings, allowing closer integration 

between learning and performance (or operational) venues. 

3. Reveal and enable informal learning

The future learning ecosystem moves us toward a holistic approach 

to learning that connects the various “above ground” and 

“below ground” learning structures, processes, and systems. 

In learning and development circles, there’s a popular notion called the 

70:20:10 model.11 It estimates that around 70% of learning is informal or onthe-job, about 20% involves peer and social learning, and only about 10% is 

formal training and education. While this model is merely a general concept, 

not a firm quantitative rule, it helps underscore the importance of surfacing 

informal learning—i.e., the 90% of learning that occurs outside of formal 

settings. Informal learning is pervasive and interwoven into the fabric of our 

professional, academic and personal lives, and we must be able to reveal and 

understand this complex set of behaviors to achieve the goal of holistic, lifelong learning.

As we progress to a more chaotic and data-saturated world, self-regulated 

learning skills, or the ability monitor and motivate oneself in learning, will 

become even more important. Accordingly, learning in the future will not only 

rely on one’s ability to learn provided material but rather the ability to seek 
366 | Modernizing Learning

out new information, determine its accuracy and relevance, and assimilate it 

in a manner accessible and translatable to the real world. We will need to educate and empower people to distinguish accurate from falsified data, manage 

data saturation and information overload, and cultivate persistent energy for 

lifelong learning. However, individuals’ abilities to engage in effective informal learning vary. Hence, it’s important to foster individuals’ self-regulation 

capabilities and facilitate their active engagement in self-directed learning, 

for instance, by providing access to resources, making learning content more 

easily “findable” (such as via metadata), or encouraging learning through personalized prompts. 

Of the various informal learning modalities, social learning seems to be particularly poignant, as well as practically supportable. It’s all about collaboration—bringing in some of the social aspects that enable individuals to share 

and learn from each other. Enabling collaboration, the sharing of information, 

When you take away the performance aspect, people act differently. 

In “practice” settings they’re freer to make mistakes without 

someone reviewing and judging them. When they’re being assessed, 

though, people bring to bear a different mindset and focus. If we 

replace more traditional assessment with “stealth” assessment, 

will we introduce a paradigm that’s counter to a growth mindsets 

and to how learning happens best? If they have to be always “on”

that could be a really challenging dynamic for our learners.

Michelle Barrett, Ph.D.

Vice President of Research Technology, 

Data Science, and Analytics, ACT
Strategic Planning | 367 

and co-creation of ideas are important, whether in professional or academic 

settings, and not all interactions need to be formally organized. Side conversations at the water cooler and informal feedback have surprisingly significant 

impacts on how work is accomplished. 

Already, many commercial vendors are developing solutions to support and 

integrate such informal opportunities. The learning and development industry 

also seeks to gather related analytics that can help provide learners with personal, relevant, and engaging social learning experiences. Nonetheless, many 

research questions remain that the scientific community still needs to address, 

including maturing our understanding of how to develop individuals’ self-regulated learning capabilities, how to best support informal learning in applied 

organizational contexts, and how to quantify the range of formal-to-informal 

learning.

4. Improve assessment

Assessments, along with the learning data they generate and evaluations they 

enable, play foundational roles in training and education. In the future, with 

the increased emphasis on personalization and data-driven systems, assessments will only grow in importance. However, the nature of the assessments 

will change. 

At the K–12 level, the number and use of standardized assessments currently 

poses several challenges to learning. Today, in the U.S. system, students are 

required to take numerous standardized tests; the results from these are used 

to identify struggling students or, in aggregate, to uncover underperforming 

school systems. In both cases, the assessments serve as accountability devices. Once a child or school is shown wanting, more assessments are used to 

focus and monitor their remediation. While this sounds logical, in practice, 

time spent on such detailed work can be emotionally and cognitively taxing 

as well as a drain on overall learning time. Emphasis on such high-stakes 
summative testing has been shown 

to shift the focus away from true 

learning and instead to encourage 

superficial “teaching to the test”—

tests that typically emphasize cognitive abilities to the exclusion of the 

“full-spectrum” competencies described above.12

In the future, assessments for secondary and postsecondary education need to focus on feedback and 

“feed forward” support—across all 

developmental dimensions. To the 

extent possible, and with balance so 

that individuals aren’t continuously 

being monitored, a rhythm of automatic assessments in forms more 

closely resembling integrated formative assessments, stealth assessments, portfolio evaluations, and 

experiential trials should be considered. Hence, significant attention needs to 

be paid to understanding new ways to prove capabilities beyond the current 

forms of assessments, articulation of grades, or standardized testing methods. Our concepts of assessment also need to expand in scope. For instance, 

assessments of large-scale outcomes, such as mission success and task accomplishment, can provide significant, valid data for determining competency. Such organizational assessments, however, must be linked to the learning 

institutions that can foster such performance or respond to gaps in it. We can 

no longer consider assessments of learning and performance as sequential, 

causal, chronological occurrences; rather, both become inextricably linked 

and interdependent. 

The problem is, with all the 

assessments we have to 

bombard students with, we don’t 

have time to do the project-based 

work. That, in and of itself, is 

defeating. Teachers have the 

lessons, but they don’t have 

the time to develop them with 

the students because of all the 

testing. 

Sandra Maldonado-Ross

President, Seminole Education 

Association (Florida)
Strategic Planning | 369 

The relationship between assessment, feedback, and self-driven learning is 

particularly meaningful to consider. To better enable self-regulated learning, 

individuals, groups, and organizations need access to their data, both at the 

discrete level (e.g., data from one assessment) and in aggregate (e.g., across 

learning arcs). Such data can inform better learning and development choices—but they don’t guarantee it. Data alone aren’t enough; data need to be 

presented in ways that support decision-making. However, large volumes of 

data can create complexity and overload, making them undigestible and less 

useful, and “noise” can enter the system, reducing the clarity or true meaning 

of the data. Hence, tools are needed to help individuals turn data into insights 

and actions. Accordingly, big data analyses and accompanying visualizations 

are needed to help learners, learning facilitators, and learning organizations 

navigate modern learning systems. 

5. Up-skill and empower learning professionals

As learning contexts evolve, so too do the roles and requirements for learning 

professionals within them, notably teachers, trainers, educational technologists, and instructional designers. The speed of progress in this sector means 

they’ll need to learn continuously—keeping abreast of the latest research, 

technologies, and regulations. Ongoing professional development to re-skill 

and up-skill learning professionals, using formal and informal methods across 

diverse media, will be critical. 

Learning professionals will also need new teamwork skills. Historically, 

someone could be a great teacher in an isolated classroom, without requiring 

the support of other learning professionals. In the future, teams of specialists—each with unique areas of expertise—will be required. Pedagogues will 

need to work with data scientists; AI developers will need to collaborate with 

media designers; and human resource specialists will need to coordinate with 

education and training leaders. At a personal level, individuals will need to 
370 | Modernizing Learning

develop appropriate collaborative skills, and organizationally, new administrative structures may be needed. For instance, instead of assigning a single 

teacher to design, develop, and implement a course, a composite team may be 

required. Some of these individuals may reside within a centralized “pool” of 

shared talent (say, for the data analysts), while others may be dedicated to the 

given program (for instance, the main teacher). Holistic solutions will include 

cross-training talent management professionals, institutional administrators 

and operational supervisors—all of the 70/20/10 components of learning.

The future learning ecosystem will likely be a highly technical and collaborative environment supporting both micro- and macro-level instructional 

strategies, maybe even leveraging the “in-between” learning experiences and 

events—between classes, courses, and life events—to adapt to learners’ interests, needs, prior knowledge, and resources. As we begin to look at learning 

across the lifetime, leveraging big learning data and new learning strategies, 

learning professionals will need new knowledge and skills. This has driven 

efforts to define the concept of a learning engineer (see Chapter 16), to close 

the gaps between technology and instructional design, and between isolated 

instructional events and larger-scale learning systems. We’ll need new conceptual models that define learning engineering, their professional practices, 

certification and skills, professional development processes, and integration 

into teams and organizations.

6. Plan for integration across learning 

and personnel functions

The future learning ecosystem vision views learning as an integral and ongoing aspect of life, woven throughout work and personal contexts. This has 

unique implications for employers, who will no doubt leverage it as a learning 

and performance ecosystem that “enhances individual and organizational effectiveness by connecting people and supporting them with a broad range of 
I just did a survey on issues for teachers asking them what their biggest 

issues in the classrooms are. Four major issues in the survey were found. 

First, academic freedom doesn’t exist anymore: “Learn as you live is gone.” 

Some of the other big issues were about the evaluations. They pressure 

teachers and administrators. In any other job, you’re evaluated on what’s 

seen, but people don’t go into surgeries and second guess everything 

the surgeon does. That doesn’t happen in a regular job; they don’t get an 

evaluation that nitpicks every possible thing they do to ensure it fits into the 

rules of what they’re told is important. The administrators, many of them 

don’t like how strenuous and stressful the process is. 

The third was stress in the classroom and stress in the working conditions. 

If there isn’t strong contract language then when there’s an issue it’s tough 

to fix. 

Finally, there are professional development trainings. Education is changing 

so quickly, but how are you expecting me to go to professional development 

training on top of 60 hours of working? If you don’t go to the trainings, you 

can’t learn what new stuff is available, but if you do go, then you’re missing 

the classroom work.

Sue Carson

President, Seminole Education Association (Florida)
372 | Modernizing Learning

content, processes, and technologies to drive performance.” 13 In other words, 

we imagine employers will seek to leverage it for their talent management, 

performance support, knowledge management, access to experts, social networking and collaboration, and structured learning functions. 

From these components, organizations can craft an infinite number of dynamic solutions for developing and employing individuals, and for optimizing 

their institutions, writ large. For instance, organizations will be able to better 

select and place individuals, shifting away from gross measures of someone’s 

capacity (such as a degree title) and towards competency composites. Psychological and behavioral analytical data will aid developmental recommendations, identification of talent, and connection across employers and educational experiences. Those same data can be used to improve task assignments or 

encourage higher rates of retention. 

Organizational processes will, therefore, need to evolve to support greater 

multidirectional integration among training, education, human resources, and 

talent management systems. The Federal Human Capital Business Reference 

Model could be used as a guide. This model was developed jointly as a public-private partnership mixing human resources, policy, and industry experts 

to create a streamlined and simplified HR system. The model denotes functions, sub-functions, authority, and policy. It also clarifies the Human Capital 

Management lifecycle government-wide. Ultimately, this model directly informs how HR practitioners plan for, work with, and organize people, policy, 

process, service delivery, and data categorization and reporting.14

In the future, we expect to see greater churn across roles, companies, and careers. As workers increasingly value flexibility, fluid work/life structures, and 

personal experiences, we may also see more “gig economy” careers, where individuals or teams are available for project-based work or consulting services 

but don’t work directly for a single company. Correspondingly, there may need 

to be greater permeability across the workforce, encouraging people to move 

into and out of formal learning, full-time jobs, and personal developmental ex-
Strategic Planning | 373 

periences. Continuous up-skilling and re-skilling of the workforce will become 

paramount. Taken together, this implies individuals’ competencies will likely 

need to constantly evolve, meaning someone skilled at learning will be highly 

prized, and organizations will need to better accommodate a variety of lifelong learning mechanisms notably nonlinear, informal, and nonformal learning options. We may also need new social paradigms, for example, for things 

like re-employment insurance that can be used to fill the gaps between careers. 

7. Facilitate a mindset shift

The success of the future learning ecosystem concept is, in large part, predicated on culture change. A significant mindset shift will need to accompany 

any advancement from the Industrial Age of learning towards the future learnIf we’re going to align learning with employer needs, we 

need to deal with job descriptions and postings, and how 

they’re organized on the web. Advancements in data 

standards now allow us to create structured, dynamic data 

on the web. Thus our goal is to: 1. Extend and improve data 

schemas for jobs, and 2. Link it to the web semantically. 

Structured, linked data maximizes our ability to search, 

discover, and compare data about jobs, and to notify anyone 

instantly when a job has changed and in what way. By 

organizing jobs-data in this way, we can create an entirely 

new labor market information system, directly from the 

hiring systems employers use.  

Jason Tyszko

VP, Center for Education and Workforce, U.S. Chamber of Commerce
374 | Modernizing Learning

ing ecosystem vision. Incremental changes 

or mere additions to the current system won’t 

suffice; stakeholders, which includes nearly 

everyone in our society, need to willing embrace the new paradigm. 

We need to change how we typically perceive 

education and training. Very broadly speaking, today’s systems tend to emphasize formal learning. Individuals largely progress 

through prescriptive and similar paths, most 

commonly based on time factors. Courses 

(including technology-based offerings) are 

often authority- or instructor-centric; learners’ roles are to receive the experts’ knowledge and then engage in predefined practice 

activities. In the future, we must be willing to 

embrace more flexible and personalized outcome-focused learning that happens across 

diverse places, time, and modalities. 

We also need to shift our perceptions of teachers and trainers, from the sources of learning 

to its facilitators, and in turn, more genuinely 

emphasize learner-centric methods. For instance, beginning with primary education, 

this mindset shift might mean transforming 

formal education spaces from places where 

students receive information to places where 

they co-create it. In the secondary school years, students might take greater 

control of their own learning journeys, which may mean more lenient rules for 

required courses and greater encouragement for self-directed learning. 

I think the challenge for the 

Department of Defense is 

that we’re using the same 

outlooks and perspectives 

that we’ve been using since 

1947. The Department has to 

change its thinking. My own 

personal answer to that is 

borne out through my work 

on the Force of the Future 

reports; the only way you’re 

going to change how the 

Department thinks is to bring 

in people who look different. 

It’s fundamental; we need 

to increase the intellectual 

diversity of the Department.

Morgan Plummer 

Director, MD5 National Security 

Technology Accelerator

U.S. Department of Defense
Strategic Planning | 375 

Closely related, we’ll need to embrace mastery learning and nonlinear, tailored 

learning. While such concepts have been touted for decades, most systems—

whether formal schools or workplace development programs—still tend to 

emphasize time factors and minimum achievement standards. To move ahead, 

we’ll need to let go of the idea of “minimally acceptable” as an advancement 

criterion. Similarly, we’ll need to allow more flexibility in systems, moving 

away from amassed education and training approaches, with predefined linear curricula, and instead towards more nonlinear, personalized trajectories.

Finally, we’ll need to change how we approach the “ownership” of learning. 

Today, we have separate silos of learning and separate “owners” of those silos—who usually also claim ownership of the data within them. The future 

learning ecosystem represents a dramatic shift away from a single organization trying to meet all education and training needs with a top-down design. In the future, separate entities will need to negotiate within a shared 

“marketplace” of learning that has no single owner, leverages the power of 

self-discovery by placing tools into learners’ hands, and relies on integration 

across a system-of-systems. Without careful plans, we may find increased 

separation and artificial “walls” between segments of the ecosystem, as different commercial vendors or learning institutions try to sell proprietary solutions or promote systems that are intentionally cumbersome to export data or 

move away from, so called “vendor lock.” Changing mindsets (and incentives) 

to embrace this new model may prove challenging, both for individuals and 

organizations. 

8. Enable learning at scale, technologically 

and methodologically 

To make the learning ecosystem practical, particularly from a resource perspective, it will need to support a large number of learners and organizations. Similarly, it will need to be future-proofed: designed to meet today’s 
376 | Modernizing Learning

requirements but also with a structure 

that can evolve to meet future needs 

and advancements. Technologically, 

such a complex system can’t be built 

from a single blueprint; it can only be 

achieved through an open systems architecture approach. This necessarily 

emphasizes interoperability, modular 

designs, common technical specifications, shared data standards, and negotiated data rights as well as extensibility 

in all components to help grow the system over time. A long-term strategy, including broad community coordination 

for training and education technologies, 

data and metadata policies, and collective technical governance, is needed. 

Paradoxically, this learning at scale will 

accompany an increase in the intimacy of learning, as the same technology 

that enables access also increasingly supports the “mass customization” of 

tailored, personal experiences.

Similarly, various social and organizational structures will need to be reconceptualized, as these changes in learning will create wide-ranging impacts, 

from the way our K–12 schools function to the nature of work across society. For instance, the timing for students attending school may change; the 

movement into and out of employment organizations may change and grow 

in frequency. Widening the aperture and access to learning might also change 

the nature of traditional trade schools and colleges, and it will likely create 

new markets for different educational experiences. For example, new entities 

may appear in this learning market to provide “credit” to “students” who take 

part in experiences, ranging from something like rock climbing and travel 

When I got a job here, I 

wanted to change people’s 

minds about math. I could 

do it in the dozens or maybe 

hundreds. Now, educators 

are more empowered 

because it’s not just who 

they can teach in the 

classroom—now they can 

reach thousands.

Ralucca Gera, Ph.D.

 Associate Provost for Graduate 

Education and Professor of 

Mathematics

Naval Postgraduate School
Strategic Planning | 377 

excursions to bootcamps and competency-based micro-degrees. 

From a methodical standpoint, there’s an imperative to conduct deliberate 

investigations across the range of new learning approaches, including those 

outside the dominant paradigm. For example, how do contextual factors, including culture, social context, instruction, and time of life, impact learning? 15 How does technology influence the psychology of learners, and what 

are the design requirements for nonlinear lifelong learning? How can learners 

aggregate and make sense of learning across multiple experiences, while minimizing cognitive friction? There’s much to consider. Our models of learning 

and teaching must evolve, both in theory and practice, and be translated into 

reference sets, use cases, and other formal representations to inform the design and delivery of learning at scale. To support that, however, we first need 

the technological backbone and the mindset shift. These questions can’t be 

answered within the current system because it impedes access to, integration 

of, and sufficient measurement of learning—all of which are necessary for 

addressing these questions. Thus, the only option is to create a system-of-systems approach that supports its own continuous evolution.

Once realized, the technological architecture doesn’t just allow for improved 

access to status-quo learning opportunities—it creates an entirely new capability. Metaphorically, consider the components of a car (the steering wheel, 

tires, pistons, and so on); separately, they’re functional objects, but when connected together, they can produce an entirely new capability—transportation. 

Similarly, the future learning ecosystem, by the aggregate nature of the systems 

that comprise it, will create unimaginable new capacities, more than merely 

the sum of its parts or the incremental expansion of today’s learning paradigm.

9. Design for convenience and equity of access

Usability is often the limiting factor of technical systems. No matter how 

brilliant a new application or hardware solution, if real people in real-world 
There are 417 national parks and monuments. We’re all over the globe. We’re in 

remote locations. You can lose your cell service. It says on our website that we had 

340,000 volunteers in 2016. How do we train all those people? We don’t have any 

subject-matter experts in our Washington office—the expertise is out in the parks. 

How do we get that knowledge into our system and then out to the workforce? 

We created the Common Learning Portal. It’s a web portal—a marketplace for 

training. It opens April 2019. The government’s cybersecurity processes (FedRAMP) 

kept us from opening the doors sooner; it’s been in pilot-project mode, but 

operational, for two years now. It provides a comprehensive learning performance 

ecosystem, a holistic view of learning. The system enables us to put information, 

people, and other learning resources in places where people can find them, even on 

a mobile device. We hope our personnel and volunteers who have been out in the 

field for work can go back to their offices and do their training, which they have to 

do at the beginning of every cycle. Already, we have over 500,000 page views and 

4,000 registered users—without even formally launching. It’s caught on by word of 

mouth. Some trainers got excited. We had support from leadership and people. It 

was a grassroots effort. 

So that’s where we are going tomorrow. We’re not throwing away formal learning, 

but we’re trying to pull in performance support, microlearning, and things that allow 

us to better do our jobs.

Courtesy of Dale Carpenter

Superintendent (Acting), National Park Services
Strategic Planning | 379 

contexts can’t use it, it won’t achieve its goals. At this most obvious level, 

this means system usability—across its various user interfaces and user experiences—plays a major role in its success. It’s necessary to put a focus on 

UI/UX, making all aspects of the system as intuitive, modern, and effective 

as possible, to increase adoption and facilitate its customization to unique 

requirements across the broad stakeholder community. 

Similarly, issues of network connectivity and technical access are equally important and extend beyond technology, touching on social and societal considerations. Access issues already limit educational opportunities for children in 

many rural or underserved areas. As more of our learning becomes digitized 

and networked, we must carefully ensure equity in access to it—not only for 

ethical reasons but also to maximize the diverse capabilities of society and 

enable all to realize their unique potentials. If not, we risk widening the education gap, creating greater disparity in access to quality education and training, and potentially creating a bifurcation between the “haves” and the “havenots.” In other words, we could inadvertently build a divide between those 

with access to open, unstructured junk information versus those with access 

to higher-fidelity, semi-automated methods of transmitting quality knowledge 

within and across communities. 

The advent of the learning ecosystem could affect populations, workforces, 

wealth distribution and other social factors. Until such time that the ability 

to adapt to the pace of change becomes irrelevant, adaptation will become 

increasingly ever more important, and that will rest squarely upon the ability 

to learn. We must carefully consider not only the social implications of establishing the learning ecosystem but also its impact on those unable to fully 

benefit from it. We have an ethical imperative to consider if, and how, access 

to learning is protected and enabled throughout society—and perhaps even 

across the globe. If managed effectively, however, “education for humanity” 

becomes a real possibility, bringing knowledge and elevating the capabilities 

of our entire world.
380 | Modernizing Learning

10. Ensure laws, policies, and governance keep pace

Holistic solutions will demand holistic governance, as well as new law and 

policies. These may span broad areas of consideration from technical frameworks and interoperability standards to content and data exchange processes, 

and equity, ethics, and fairness of use. Below are a few considerations, but this 

discussion will require a much more extensive treatment, as well as interorganizational coordination, to fully outline. 

Starting with K–12 education, new policies and processes are needed in several areas. For example, moving to competency-based learning methods will 

be key for fostering a learner-centered, development-oriented system. In the 

U.S., the Common Core Standards, in theory, allow for a similar kind of coordination. In practice, however, these standards have become inflexible requirements added to already overloaded schedules. Transitioning to a competency-based model would better allow teachers to personalize learning and let 

students earn credit for knowledge acquired outside the classroom. Teachers 

will need policy to support them in exercising the academic freedom required 

by this model, to be able to adjust content and methods to meet each student’s 

unique developmental needs. Additionally, as indicated above, competency 

goals will need to be augmented to incorporate social, emotional, metacognitive, and physical elements for “whole person” development. Additionally, 

the Every Student Succeeds Act focuses on providing funds to schools that 

use evidence-based practices, yet teachers and administrators rarely receive 

formal training on research design and statistics. Expanding existing governmental programs (e.g., the Education Innovation Programs, U.S. Department 

of Education) that aid in closing this research-practice gap will help optimize 

learning and can also support the up-skill and re-skilling of learning professionals (as described in Recommendation 5, above).

For post-secondary and workforce education, policies must address key 

cross-community issues, including funding allocation, data sharing, and data 
Strategic Planning | 381 

usage rights. The diversity of learning venues obviates any unitary solution, 

but certain characteristics will be common, such as privacy, continual assessment, and security. 

As we progress to a digitized nation, we will need to ensure ethical use of 

learning data, both for students and employees. As such, policy must be written that ensures individuals can own their own data, with updated laws designed to protect them in current and evolving contexts. Existing laws, such 

as the Family Educational Rights and Privacy Act, provide some measure of 

protection, but weren’t designed for the kinds of data-rich, technology-enabled learning that’s emerging. They also tend to focus on the education “silo” 

rather than a lifelong perspective. A balance across stakeholders, notably between the public and private sectors, will be needed, particularly given the 

commercial value of data as a sharable resource. Such laws and policies need 

The big thing here, that everyone’s dealing with, is the challenge of: How 

do you transition into a performance organization? How do you support 

an organization trying to become a performance-based one, and what 

are the other things that wrap around that structure? For example, talent 

management systems are important, but the way HR people manage now 

is mainly through “box-checking,” like the things that you’re required to 

do for mandatory training. We’ll have to rework so many things—HR, the 

compliance stuff, and assignment decisions...Conceptually, though, it’s 

always been the same thing: How do you choose the right people? 

Michael Freeman

Consultant, Training and Learning Technologies
382 | Modernizing Learning

to be reconsidered for the future context and designed in a way that balances 

privacy with functionality.

New policy considerations will likely also concern accreditation standards 

and assessment validation. For example, consider a medical student who’s 

gained competencies through means other than formal education (e.g., an internship as a teenager, combined with lifeguard training, volunteering as a 

medic for local disasters, and online personal study). She could, in theory, 

graduate from medical school earlier than her peers—however, only if valid 

assessments can be used to comprehensively evaluate her capabilities across 

the full-spectrum of necessary competencies. Beyond developing such assessWe know we have a shortage of talent (human capital) for certain positions 

but we can’t just “up” our recruiting tactics....We’ve got to transform 

how we put them into that pipeline. The big thing I want is to integrate 

government with industry and universities—to formalize partnerships 

sooner, so we can get smarter about what people need to be ready to 

participate in our workforce. Right now, the people we need aren’t coming 

out of the universities; so, we’ve got incentive to work together. We could 

start small with a representative government agency that could be our 

champion and also couch the program as a college internship or co-op. We 

have to figure this problem out—we have to grow our workforce.

Anne Little, Ph.D.

Vice President, Training Solutions Development, SAIC
Strategic Planning | 383 

ments (as mentioned in Recommendation 4), who will validate them, update 

them, and accredit their use across learning and workforce systems? Further, 

how will schools that provide degrees based on mixed methods for competency attainment be formally accredited or ranked? To continue our example, 

laws involving medical insurance, malpractice determinations, and formal licensure might be impacted. 

Broadly, there’s also a need to develop interoperability specifications. Professional organizations, such as the IEEE Learning Technology Standards 

Committee or ISO IT for Learning, Education and Training, help formalize 

technical standards across communities. However, this still only applies to 

the interface or data layers. Each organization still makes many independent 

decisions about instructional media, new technologies, and their technical 

and programmatic factors. While organizations should retain their autonomy, 

there’s opportunity to increase coordination, give collective guidance, and at 

least within organizations or alliances, to create shared processes. A federated 

data system is an extensive endeavor. We have to constantly ask, how can we 

protect this system and yet keep information as open as possible? Thus, additional governing agreements in the areas of cybersecurity, privacy, and identity, as well as considerations for copyrights and data ownership, are important. 

For example, for the U.S. Department of Defense DoD Instruction 1322.26 

(“Distributed Learning”) provides guidance on best practices for distributed 

learning and permissions to collect, aggregate, and assess data. This is one 

of many policies that could be reviewed to encourage greater unity of action 

across the military services and other defense components as well as the U.S. 

Government, writ large. 

We need to develop effective forms of governance for a diverse and disparate community of practice including government, academic, and industry 

partners. This macro concern for governance is a mirror of what must occur 

within the future learning ecosystem, itself: As components federate to attain 

a capability the need for agile partnerships will grow, enabling rapid aggrega-
384 | Modernizing Learning

tion (or disaggregation) of federated capabilities. Such collective governance 

will need to set approaches, policies, and management strategies that education and training stakeholders can adopt to enable effective learning—not 

only within a given silo—but across the composite, collective system. 

Finally, there is a broader imperative that frames our approaches—policy and 

methodology—for moving towards the future learning ecosystem. We do not, 

and cannot, fully appreciate the impacts of exponential technological change, 

particularly as we approach the point beyond which it’s impossible to fathom 

(the “singularity”). Ethical considerations must be an innate characteristic of 

our process methodologies, or we will sacrifice the human nature of progress. 

Simultaneously, our span of regard will grow to include machine learning as 

an essential and constant complement to human learning and employment; areas that we cannot approach reductively. Consequently, process perspectives 

and non-linear contexts will characterize the evolution of the future learning 

ecosystem—perhaps the final step away from Industrial Age thinking.

CONCLUSION

In this chapter, we’ve offered several recommendations for the advancement 

of learning. Throughout this process, we’ve assumed that technologies, notably automation and data analytics, will continue to advance. In other words, 

we felt it safe to assume that such capabilities are (or would be) technologically feasible. The challenge lies not in developing the technologies but in their 

validation, effective integration into learning systems, and consideration for 

the corresponding social, organizational, and societal changes they’ll produce. 

It’s not feasible, nor frankly advisable, though to plan out every piece of this 

future learning ecosystem; the rapid pace of change and its complexity necessarily require its design to be dynamic, flexible, and collaborative. However, 

we’ve attempted to apply systems-thinking approaches to the planning pro-
Strategic Planning | 385 

cess, considering the comprehensive “talent development system,” including 

formal and informal training and education. We’ve also tried to harmonize 

across the principles of learning science, learning technology, data science, 

organizational dynamics, and policy, and consider a lifelong learning continuum to include K–16, the public and private workforce, military service, 

and self-directed learning. Specific solutions should be grounded within this 

larger tapestry so that when implemented, they’re most likely to work in concert across technology, design, commitment, governance, policy, and human 

infrastructure factors. 

The immediate and enduring relevance of this discussion is clear; we’re conducting basic research now that will provide knowledge to reframe our future 

paradigms, bounding the unknowable to both enable and constrain future 

choices. We recognize whatever choices we make will have consequences, 

but learning, itself, is essential for making those future choices. The confluence of learning and technology—the evolution from traditional schools, 

to distributed learning, and now to “ubiquitous learning”—is driving us towards the need for learning across time, space, and function using tools and 

techniques from across diverse venues to enable seamless lifelong learning, 

whether training, education, or experience, as part of a holistic approach to 

empowering human potential. Interdisciplinary stewardship will be essential 

to extend and connect learning science, policy, and technology to address today’s challenges and be prepared for the unknowable future.

If we don’t like the rules, 

why don’t we change the rules? 

– Reese Madsen, Senior Advisor for Talent Development, U.S. OPM

Chief Learning Officer, OSD (Intelligence and Security)

End Matter
388 | Modernizing Learning

ENDNOTES FOR SECTION 1

(FOUNDATIONS)

Chapter 1 Endnotes

1 Atkinson, R.C., & Shiffrin, R.M. (1968). Human 

memory: A proposed system and its control processes. Psychology of Learning and Motivation, 2, 

89–195.

2 Baddeley, A.D. (1966). The influence of acoustic 

and semantic similarity on long-term memory for 

word sequences. The Quarterly Journal of Experimental Psychology, 18(4), 302–309.

3 Many classic studies in cognitive psychology 

could be listed here; see, for example:

Anderson, M.C. & Neely, J.H. (1996). Interference 

and inhibition in memory retrieval. In E. L. Bjork 

& R.A. Bjork (Eds.), Memory: Handbook of perception and cognition (2nd ed.). San Diego: Academic Press. 

Bjork, R.A. (1989). Retrieval inhibition as an adaptive mechanism in human memory. In H.L. Roediger, II & I.M. Craik (Eds.), Varieties of memory & 

consciousness: Essays in honor of Endell Tulving

(pp. 309–330). Hillside, NJ: Erlbaum.

Sohn, M.H., Goode, A., Stenger, V.A., Jung, K.J., 

Carter, C.S., & Anderson, J.R. (2005). An information-processing model of three cortical regions: 

Evidence in episodic memory retrieval. Neuroimage, 25(1), 21–33.

Sweller, J. (1994). Cognitive load theory, learning 

difficulty, and instructional design. Learning and 

Instruction, 4(4), 295–312.

4 National Academies of Sciences, Engineering, and 

Medicine (2018). How people learn II: Learners, 

contexts, and cultures. Washington, DC: The National Academies Press.

NOTE: A digital version of this book is openly and 

publicly available at nap.edu/24783

5 See, for example:

Manuti, A., Pastore, S., Scardigno, A.F., Giancaspro, M.L., & Morciano, D. (2015). Formal and 

informal learning in the workplace: A research review. International journal of training and development, 19(1), 1–17.

Mocker, D.W. & Spear, G.E. (1982). Lifelong 

learning: Formal, nonformal, informal, and 

self-directed (ERIC No. ED220723). https://eric.

ed.gov/?id=ED220723

6 Friedman, T. L. (2016). Thank you for being late: 

An optimist’s guide to thriving in the age of accelerations. New York: Farrar, Straus and Giroux. 

7 Ibid. National Academies (2018). Endnote 1-4.

Chapter 2 Endnotes

1 See Niper, S. (1989). Third generation distance 

learning and computer conferencing. In Mason, R. 

and Kaye, A. (eds.), Mindweave: Communication, 

Computers and Distance Education (pp. 63–73). 

Oxford: Pergamon Press. 

Also, James Taylor expanded Niper’s model, dividing its third generation into a fourth generation 

focused on flexible online learning delivery and 

fifth that adds intelligent automation to the flexible 

web-based delivery. See Taylor, J.C. (2001). Fifth 

generation distance education. Instructional Science and Technology, 4(1), 1-14.

2 Simpson, M. & Anderson, B. (2012). History and 

heritage in open, flexible and distance education. 

Journal of Open, Flexible, and Distance Learning, 

16(2), 1–10.

3 Moore, M.G. & Anderson, W.G. (Eds.). (2003). 

Handbook of distance education. Mahwah/London: Lawrence Erlbaum Associates.

4 Saettler, P. (1990). The Evolution of American Educational Technology. Englewood, CO: Libraries 

Unlimited, Inc.

5 Spector, J.M., Merrill, M.D., Elen, J., & Bishop, 

M.J. (Eds.). (2014). Handbook of research on educational communications and technology. New 

York: Springer.

6 Molenda, M. (2013). Historical foundations. In 

Spector, J.M., Merrill, M.D., Elen, J., & Bishop, 

M.J. (Eds.). Handbook of research on educational 

communications and technology (pp. 3–18). New 

York: Springer.

7 See discussion and original citations from ibid. 

Molenda, 2013, p. 16. Endnote 2-6.

8 Pask, G. (1960). The teaching machine as a control 

mechanism. Transactions of the Society for Instrument Technology, 12(2), 72–82.

Pask, G. (1982). SAKI: Twenty-five years of adaptive 

training into the microprocessor era. International 

Journal of Man-Machine Studies, 17(1), 69–74.

9 Kulik, J.A. (1994). Meta-analytic studies of findings on computer-based instruction. Technology 

assessment in education and training, 1, 9–34.

10 For more information on this historic adaptive instructional systems, see, for example: 
Endnotes | 389 

Lesgold, A. (1988). SHERLOCK: A coached practice environment for an electronics troubleshooting job (ERIC No. ED299450). https://eric.ed.gov

/?id=ED299450

Anderson, J.R., Conrad, F.G., & Corbett, A.T. 

(1989). Skill acquisition and the LISP tutor. Cognitive Science, 13(4), 467-505.

Brown, J.S., Burton, R., & de Kleer, J. (1982). 

Pedagogical, natural language and knowledge engineering techniques in SOPHIE I, II, and Ill. In D. 

Sleeman & J.S. Brown (Eds.), Intelligent Tutoring 

Systems. Academic Press.

11 For meta-analyses on intelligent tutors, see: 

Dodds, P.V.W., & Fletcher, J.D. (2004). Opportunities for new “smart” learning environments enabled by next generation web capabilities. Journal 

of Education Multimedia and Hypermedia, 13(4), 

391–404. 

Kulik, J.A., & Fletcher, J.D. (2016). Effectiveness 

of intelligent tutoring systems: A meta-analytic review. Review of Educational Research, 85, 171–204.

12 U.S. Office of Technology Assessment. (1988). 

Power On!-New Tools for Teaching and Learning 

(OTA-SET-379). Washington, DC: U.S. Government Printing Office.

13 U.S. Congress, Office of Technology Assessment 

(1989). Linking for learning: A new course for education (OTA-SET-430). Washington, DC: U.S. 

Government Printing Office. NOTE: Refer to page 

27 for the blockquote and page 26 for the in-text 

quote following it.

14 Molenda, M. (2003). In search of the elusive ADDIE 

model. Performance improvement, 42(5), 34–36.

15 Ibid. Molenda (2013). Endnote 2-6.

16 Ibid. Kulik (1994). Endnote 2-9. See page 18. 

17 Peters, O. (1994). Distance education and industrial production: A comparative interpretation in 

outline (1967). Otto Peters on distance education: 

The industrialization of teaching and learning (D. 

Keegan, Ed., pp. 107–127). See page 111.

18 Two useful articles to begin with are, e.g.:

Sweller, J. (1988). Cognitive load during problem 

solving: Effects on learning. Cognitive science, 

12(2), 257–285.

Sweller, J. (2008). Cognitive load theory and the 

use of educational technology. Educational Technology Publications, 48(1), 32–35.

19 Bloom, B.S. (1984). The 2 sigma problem: The 

search for methods of group instruction as effective as one-to-one tutoring. Educational Researcher, 13(6), 4–16.

20 For example, see:

Piaget, J. (1964). Part I: Cognitive development in 

children: Piaget development and learning. Journal 

of Research in Science Teaching, 2(3), 176–186.

Vygotsky, L.S. (1978). Mind in society: The development of higher psychological functions. London: 

Harvard University Press.

Also, for an historical overview, see: Matthews, W. 

J. (2003). Constructivism in the classroom: Epistemology, history, and empirical evidence. Teacher 

Education Quarterly, 30(3), 51–64.

21 Gamoran, A., Secada, W.G., & Marrett, C.B. 

(2000). The organizational context of teaching and 

learning. In M.T. Hallinan (Ed.), Handbook of the 

Sociology of Education (pp. 37-63). New York: 

Kluwer Academic/Plenum Publishers.

22 For a review, see: Au, K.H. (1998). Social constructivism and the school literacy learning of students of diverse backgrounds. Journal of Literacy 

Research, 30(2), 297–319.

23 For Xerox’s NoteCards, see: Halasz, F. (1988). Reflections on Notecards: Seven issues for the next 

generation of hypermedia systems. Communications of the ACM, 31(7), 836–852.

For Carnegie-Mellon University’s Andrew, see 

for example: Morris, J.H., Satyanarayanan, M., 

Conner, M.H., Howard, J.H., Rosenthal, D.S., & 

Smith, F.D. (1986). Andrew: A distributed personal computing environment. Communications of the 

ACM, 29(3), 184–201.

24 Scardamalia, M., Bereiter, C., McLean, R.S., Swallow, J., & Woodruff, E. (1989). Computer-supported intentional learning environments. Journal of 

Educational Computing Research, 5(1), 51–68.

25 Hiltz, S. R. (1994). The virtual classroom: Learning without limits via computer networks. Intellect 

Books. Norwood, NJ: Ablex Publishing Corporation. See pages 5–6.

26 Bell, M. W. (2008). Toward a definition of “virtual worlds.” Journal For Virtual Worlds Research, 

1(1). See page 2.

27 Naimark, M. (1997). A 3D moviemap and a 3D 

panorama. Proceedings of SPIE 3012, Stereoscopic Displays and Virtual Reality Systems IV. doi.

org/10.1117/12.274471

28 Morningstar, C., & Farmer, F.R. (2008). The lessons of Lucasfilm’s habitat. Journal For Virtual 

Worlds Research, 1(1).
390 | Modernizing Learning

29 For example, see: Schatz, S., Nicholson, D., & 

Dolletski, R. (2012). A system’s approach to simulations for training: Instruction, technology, and 

process engineering. In P. J. Mosterman (Series 

Ed.) Real-time Simulation Technologies: Principles, Methodologies, and Applications (pp. 371–

388). Boca Raton, FL: CRC Press.

30 Web-based Education Commission (2000). The 

power of the internet for learning: Moving from 

promise to practice. Washington, DC: Web-Based 

Education Commission. www2.ed.gov/offices/AC/

WBEC/FinalReport/index.html See pages 75–77.

31 For example, see:

Beaumont, I., & Brusilovsky, P. (1995). Educational applications of adaptive hypermedia. In Human-Computer Interaction (pp. 410–414). Springer, Boston, MA. 

Brusilovsky, P., Pesin, L., & Zyryanov, M. (1993). 

Towards an adaptive hypermedia com-ponent for 

an intelligent learning environment. In L.J. Bass, 

J. Gornostaev, & C. Unger (Eds.), International 

Conference on Human-Computer Interaction (pp. 

348-358). Springer, Berlin, Heidelberg.

32 For example, see:

Koedinger, K.R., Anderson, J.R., Hadley, W.H., 

& Mark, M.A. (1997). Intelligent tutoring goes to 

school in the big city. International Journal of Artificial Intelligence in Education, 8, 30–43. 

Ritter, S., Anderson, J.R., Koedinger, K.R., & Corbett, A. (2007). Cognitive Tutor: Applied research 

in mathematics education. Psychonomic Bulletin & 

Review, 14(2), 249–255.

33 Ibid. Kulik & Fletcher (2016). Endnote 2-11.

34 Emerging in the 1990s, the learning gains fostered 

by intelligent tutors became approximately equivalent to those from human tutors. See reviews by:

Graesser, A.C., Rus, V., Hu, X. (2017). Instruction 

based on tutoring. In R.E. Mayer and P.A. Alexander (Eds.), Handbook of Research on Learning and 

Instruction (pp. 460–482). New York: Routledge. 

VanLehn, K. (2011). The relative effectiveness of 

human tutoring, intelligent tutoring systems and 

other tutoring systems. Educational Psychologist, 

46, 197–221

35 Picard R.W. (1997). Affective computing. The MIT 

Press, Cambridge, MA.

36 For example, see:

Graesser, A.C., Wiemer-Hastings, K., WiemerHastings, P., Kreuz, R., & the Tutoring Research 

Group. (1999). AutoTutor: A simulation of a human tutor. Cognitive Systems Research, 1, 35–51. 

Lester, J.C., Towns, S.G., & Fitzgerald, P.J. (1998). 

Achieving affective impact: Visual emotive communication in lifelike pedagogical agents. International Journal of Artificial Intelligence in Education, 10, 278–291.

37 D’Mello, S.K., Picard, R., & Graesser, A.C. (2007). 

Toward an affect-sensitive AutoTutor. IEEE Intelligent Systems, 22, 53–61. 

Kort, B., Reilly, R., & Picard, R.W. (2001). An affective model of interplay between emotions and 

learning: Reengineering educational pedagogy—

Building a learning companion. In T. Okamoto, R. 

Hartley, Knshuk, & J.P. Klus (Eds.), Proceedings 

of the IEEE International Conference on Advanced 

Learning Technologies (pp. 43–46). 

38 For example, see: Calvo, R.A., & D’Mello, S. K. 

(2010). Affect detection: An interdisciplinary review 

of models, methods, and their applications. IEEE 

Transactions on Affective Computing, 1, 18–37.

39 Mayer, R.E. (1997). Multimedia learning: Are we 

asking the right questions. Educational Psychologist, 32, 1–19.

40 For example, see: Garrison, D.R., Anderson, T., & 

Archer, W. (2003). A theory of critical inquiry in 

online distance education. In M. Moore and G. Anderson (Eds.), Handbook of Distance Education. 

(pp.113–127). New York: Erlbaum.

41 Garrison, R. (2000). Theoretical challenges for 

distance education in the 21st century: A shift from 

structural to transactional issues. The International Review of Research in Open and Distributed 

Learning, 1(1). doi.org/10.19173/irrodl.v1i1.2

42 Peters, O. (1993). Distance education in a postindustrial society. In D. Keegan (Ed.), Theoretical 

Principles of Distance Education (pp. 39–58). 

London: Routledge. See page 40.

43 Daniel, J. (1996). Mega-universities and Knowledge Media: Technology Strategies for Higher Education. London: Kogan Page. See page 5.

44 Ibid. Web-based Education Commission (2000). 

Endnote 2-30. Refer to page 1.

45 Ibid. Web-based Education Commission (2000). 

Endnote 2-30. Refer to page 23.

46 Youngblut, C. (1998). Educational uses of virtual 

reality technology (No. IDA-D-2128). Alexandria 

VA: Institute for Defense Analyses.

47 Page, E.H., & Smith, R. (1998). Introduction to 

military training simulation: A guide for discrete 

event simulationists. In D.J. Medeiros, E.F. Watson, J.S. Carson, & M.S. Manivannan (Eds.), 

Winter Simulation Conference Proceedings (pp. 

53–60). Piscataway, NJ: IEEE.
Endnotes | 391 

48 Henninger, A.E., Cutts, D., Loper, M., Lutz, R., 

Richbourg, R., Saunders, R., & Swenson, S. 

(2008). Live virtual constructive architecture roadmap (LVCAR) final report. Alexandria VA: Institute for Defense Analyses.

49 El Kaliouby, R. & Robinson, P. (2005). Generalization of a vision-based computational model of 

mind-reading. In International Conference on Affective Computing and Intelligent Interaction (pp. 

582-589). Springer, Berlin, Heidelberg.

50 Director for Readiness and Training (1999, April 

30). Department of Defense Strategic Plan for Advanced Distributed Learning (Report to the 106th

Congress). Washington, DC: U.S. Office of the 

Deputy Under Secretary of Defense for Readiness. 

apps.dtic.mil/dtic/tr/fulltext/u2/a470552.pdf. 

51 Ibid. El Kaliouby & Robinson (2005).Endnote 2-49. 

52 Motlik, S. (2008). Mobile learning in developing 

nations. The International Review of Research in 

Open and Distributed Learning, 9(2). www.irrodl.

org/index.php/irrodl/article/view/564/1039

53 Crompton, H. (2013). A historical overview of 

m-learning: Toward learner-centered education. In 

Z.L. Berge & L. Muilenburg (Eds.), Handbook of 

Mobile Education. Hoboken: Taylor and Francis.

54 Watson, J., Murin, A., Vashaw, L., Gemin, B., & 

Rapp, C. (2010). Keeping Pace with K-12 Online 

Learning: An Annual Review of Policy and Practice, 2010. Evergreen Education Group.

55 For example, see: 

Muoio, A. (2000, October). Cisco’s Quick Study. 

Fast Company. www.fastcompany.com/41492/ciscos-quick-study

Seufert, S. (2001). E-learning business models, 

framework and best practice examples. In M.S. 

Raisinghani (Ed.), Cases on Worldwide E-Commerce: Theory in Action (70–94). New York: Idea 

Group.

56 Fletcher, J. D. (2009). Education and training technology in the military. Science, 323(5910), 72–75. 

Wisher, R. A. & Khan, B. H. (Eds.), Learning on 

demand: ADL and the Future of e-Learning. Washington DC: Department of Defense.

57 Fletcher, J. D. (2005). The Advanced Distributed 

Learning (ADL) vision and getting from here to 

there (No. IDA/HQ-D-3212). Alexandria VA: Institute for Defense Analyses. apps.dtic.mil/dtic/tr/

fulltext/u2/a452053.pdf. See page 7.

58 Rehak, D., Dodds, P., & Lannom, L. (2005, May). 

A model and infrastructure for federated learning 

content repositories. Paper presented at the 14th 

World Wide Web Conference, Chiba, Japan.

59 For example, see: Cavanagh, S. (2018, November 

21). Ed. Dept. pulls plug on ‘Learning Registry,’ an 

Obama-Era tech initiative. EdWeek Market Brief.

marketbrief.edweek.org

60 Johnstone, S. M. (2005). Open educational resources serve the world. Educause Quarterly, 28(3), 15.

61 Howe, J. (2006, June 2). Crowdsourcing: A definition. Wired. ww.wired.com/2006/06/crowds

62 Siemens, G. (2005). Connectivism: A learning 

theory for the digital age. International Journal of 

Instructional Technology and Distance Learning, 

2(1), 3–10.

63 National Research Council (2000). How people 

learn: Brain, mind, experience, and school: Expanded edition. Washington, DC: The National 

Academies Press. 

NOTE: A digital version of this book is openly and 

publicly available at doi.org/10.17226/9853.

64 Anderson, L., & Krathwohl, D.E. (2001). A Taxonomy for learning teaching and assessing: A revision of Bloom’s taxonomy of educational objectives. New York: Addison.

65 Merrill, M.D. (2002). First principles of instruction. Educational Technology Research and Development, 50(3), 43–59.

66 Fiore, S.M., & Salas, E.E. (2007). Toward a science of distributed learning. Washington, DC: 

American Psychological Association.

67 Pashler, H., Bain, P., Bottge, B., Graesser, A., 

Koedinger, K., McDaniel, M., & Metcalf, J. (2007). 

Organizing instruction and study to improve student learning (NCER 2007-2004). Washington, 

DC: National Center for Education Research, Institute of Education Sciences, U.S. Department of 

Education. http://ncer.ed.gov

68 Magoulas, G.D., & Chen, S.Y. (Eds.). (2006). 

Advances in web-based education: personalized 

learning environments (ERIC No. ED508909). 

Hershey, PA: Information Science Publishing.

69 Woolf, B.P. (2009). Building intelligent tutoring 

systems. Burlington, MA: Morgan Kaufman.

70 King, A. (1993). From sage on the stage to guide 

on the side. College teaching, 41(1), 30–35.

71 O’Flaherty, J., & Phillips, C. (2015). The use of 

flipped classrooms in higher education: A scoping review. The Internet and Higher Education, 25, 85–95.

72 Ibid. Pashler et al. (2007). Endnote 2-67.

73 Kelley, P. (2008). Making minds. New York: Routledge. See page 4

74 Ibid. Graesser et al. (1999). Endnote 2-36. 
392 | Modernizing Learning

Graesser, A. C. (2016). Conversations with AutoTutor help students learn. International Journal of Artificial Intelligence in Education, 26(1), 124–132. 

Nye, B.D., Graesser, A.C., & Hu, X. (2014). AutoTutor and family: A review of 17 years of natural 

language tutoring. International Journal of Artificial Intelligence in Education, 24(4), 427–469.

75 Rowe, J.P., Shores, L.R., Mott, B.W., & Lester, J. 

C. (2010). Integrating learning and engagement in 

narrative-centered learning environments. In: V. 

Aleven, J. Kay, & J. Mostow (Eds.), International Conference on Intelligent Tutoring Systems. ITS 

2010. Lecture Notes in Computer Science, vol 6095 

(pp. 166–177). Berlin, Heidelberg: Springer.

76 Johnson, W.L., & Valente, A. (2009). Tactical language and culture training systems: Using AI to 

teach foreign languages and cultures. AI Magazine, 

30(2), 72.

77 Ibid. Pashler et al. (2007). Endnote 2-67.

78 Roediger, H.L., and Karpicke, J.D. (2006). The 

power of testing memory: Basic research and implications for educational practice. Perspectives on 

Psychological Science, 1(3), 181–210.

79 For example, see: Landauer, T.K., Laham, D., & 

Foltz, P.W. (2003). Automatic essay assessment. 

Assessment in Education: Principles, Policy & 

Practice, 10(3), 295–308.

80 Siemens, G. (2006). Knowing knowledge. www.

knowingknowledge.com

81 Baker, R.S.J.D. & Yacef, K. (2009). The state of educational data mining in 2009: A review and future visions. Journal of Educational Data Mining, 1, 3–16.

82 For example, see: Baker, R.S.J.D. & Inventado, P.S. 

(2014). Educational data mining and learning analytics. In J.A. Larusson & B. White (Eds.), Learning Analytics (pp. 61-75). New York: Springer.

83 Evans, D. (2011, April). The Internet of Things: 

how the next evolution of the internet is changing 

everything. CISCO white paper. www.cisco.com

84 Gómez, J., Huete, J.F., Hoyos, O., Perez, L., & 

Grigori, D. (2013). Interaction system based on internet of things as support for education. Procedia 

Computer Science, 21, 132–139.

85 For example, see: Bower, M., & Sturman, D. 

(2015). What are the educational affordances of 

wearable technologies? Computers & Education, 

88, 343–353.

86 D’Mello, S.K., Kappas, A., & Gratch, J. (2018). 

The affective computing approach to affect measurement. Emotion Review, 10(2), 174–183.

87 For example, see: Hampson, R.E., Song, D., Robinson, B.S., Fetterhoff, D., Dakos, A. S., Roeder, B. 

M., et al. (2018). Developing a hippocampal neural 

prosthetic to facilitate human memory en-coding 

and recall. Journal of Neural Engineering, 15(3), 

036014.

88 See www.gifttutoring.org

Also refer to, for example:

Sottilare, R.A., Goldberg, B. S., Brawner, K.W., 

& Holden, H.K. (2012). A modular framework to 

support the authoring and assessment of adaptive 

computer-based tutoring systems (CBTS). In Proceedings of the I/ITSEC. Arlington, VA: National 

Training and Simulation Association.

89 Sinatra, A., Graesser, A., Hu, X., & Brawner, K., 

(2019). Design Recommendations for Intelligent 

Tutoring Systems: Artificial Intelligence (Volume 

6). U.S. Army Research Laboratory.

90 IEEE Competency Data Standards Work Group 

(CDSWG20 P1484.20.1). (2018). sites.ieee.org/

sagroups-1484-20-1

91 Yang, I. (2014, October 30). Grading adults on life 

experience. The Atlantic. www.theatlantic.com

92 For example, see: Anderson, L. (2018). Competency-based education: Recent policy trends. The 

Journal of Competency-Based Education, 3(1).

doi: 10.1002/cbe2.1057

93 Kazin, C. (2017, August 15). Microcredentials, micromasters, and nanodegrees: What’s the big idea? 

The EvoLLLution. www.evolllution.com

94 Dede, C., Richards, J. & Saxberg, B. (2018). Learning Engineering for Online Education. Routledge.

95 As quoted by Blake-Plock, S. (2018, January). 

Learning engineering: Merging science and data 

to design powerful learning experiences. Getting 

Smart. www.gettingsmart.com

96 Saxberg, B. (2016, July). “Learning engineering” 

making its way in the world. Getting Smart. www.

gettingsmart.com

Chapter 3 Endnotes

1 Allen, I.E., & Seaman, J. (2016). Online Report 

Card: Tracking Online Education in the United States (ERIC No. ED572777). Babson Park, 

MA: Babson Survey Research Group. eric.ed.gov

/?id=ED572777

2 U.S. Department of Education, National Center for 

Education Statistics. (2018). Digest of Education 

Statistics 2016 (NCES 2017-094). nces.ed.gov/

pubs2017/2017094.pdf. See Table 311.15. 
Endnotes | 393 

3 Association for Talent Development Research. 

(2017). Next Generation E-Learning: Skills and 

Strategies (Product Code 191706). Alexandria,VA: 

ATD Research.

4 Shah, D. (2018, March 10). A product at every 

price: A review of MOOC stats and trends in 2017. 

MOOC Report by Class Central. www.class-central.com/report/moocs-stats-and-trends-2017

5 World Bank (2018). World Development Report 

2018: Learning to Realize Education’s Promise. 

Washington, DC: World Bank. doi: 10.1596/978-

1-4648-1096-1. See page 16.

NOTE: This report, which include a notable amount 

of useful empirical data, is freely available under 

the Creative Commons Attribution license.

6 Ibid. National Academies (2018). Endnote 1-4. 

7 Peggy Ertmer and Timothy Newby offer a highly 

readable comparison of these theories in the context of instructional design in their 1993 article: 

Ertmer, P.A. & Newby, T.J. (1993). Behaviorism, 

cognitivism, constructivism: comparing critical 

features from an instructional design perspective 

(reprint). Performance Improvement Quarterly, 

6(4), 50–72. 

8 Ibid. Merrill (2002). Endnote 2-65.

9 Ambrose, S. A., Lovett, M., Bridges, M. W., DiPietro, M., & Norman, M. K. (2010). How learning 

works: Seven research-based principles for smart 

teaching. San Francisco, CA : Jossey-Bass.

10 Chi, M.T.H., & Wylie, R. (2014). The ICAP framework: Linking cognitive engagement to active 

learning outcomes. Educational Psychologist, 

49(4), 219–243. See page 220.

11 Winne, P.H. (2011). A cognitive and metacognitive analysis of self-regulated learning. In D.H. 

Schunk, & B. Zimmerman (Eds.), Handbook of 

self-regulation of learning and performance (pp. 

15-32). Ney York: Routledge.

12 Ibid. Pashler et al. (2007). Endnote 2-67.

13 Graesser, A.C. (2009). Cognitive scientists prefer 

theories and testable principles with teeth. Educational Psychologist, 44(3), 193–197.

14 Jarvis, P. (2012). Towards a comprehensive theory 

of human learning, Vols. 1–3. New York: Routledge. 

15 Saettler, P. (1990). The evolution of American educational technology. Englewood: Libraries Unlimited.

16 Mayer, R. E. (2017). Using multimedia for e-learning. Journal of Computer Assisted Learning, 33(5), 

403–423.

17 For example, see: Puentedura, R. (2014). Learning, 

technology, and the SAMR model: Goals, processes, and practice. Ruben R. Puentedura’s blog. hippasus.com/blog/archives/127

18 Khan, B.H. (2003). The global e-learning framework. STRIDE Handbook, 42–51.

19 Farid, S., Ahmad, R., & Alam, M. (2015). A hierarchical model for e-learning implementation challenges using AHP. Malaysian Journal of Computer 

Science, 28(3), 166–188.

20 Aguti, B., Wills, G.B., & Walters, R.J. (2014). An 

evaluation of the factors that impact on the effectiveness of blended e-learning within universities. In 

Proceedings of the International Conference on Information Society (pp. 117–121). Piscataway: IEEE.

21 Roscoe, R.D., Branaghan, R., Cooke, N.J., & Craig, 

S.D. (2017). Human systems engineering and educational technology. In R.D. Roscoe, S.D. Craig & 

I. Douglas (Eds.), End-user considerations in educational technology design. (pp. 1–34). New York: 

IGI Global.

22 Sohoni, S., Craig, S.D. & Vedula, K. (2017). A 

blueprint for an ecosystem for supporting high 

quality education for engineering. Journal of Engineering Education Transformation, 30(4), 58–66.

23 Dick, W., Carey, L., & Carey, J.O. (2011). The systematic design of instruction (7th ed.). Upper Saddle River, NJ: Pearson.

24 Douglas, I. (2006). Issues in software engineering 

of relevance to instructional design. TechTrends, 

50(5), 28–35.

25 Cooke, N.J. & Hilton, M.L. (2015). Enhancing the 

effectiveness of team science. Washington, D.C.: 

National Academies Press.

Fiore, S.M., Graesser, A.C., & Greiff, S. (2018). 

Collaborative problem-solving education for the 

twenty-first-century workforce. Nature Human Behaviour, 2, 367–369.

26 For an informative and lighthearted discussion 

on software development teams see: Fitzpatrick, 

B., & Collins-Sussman, B. (2012). Team geek: a 

software developer’s guide to working well with 

others. Sebastopol, CA: O’Reilly Media, Inc.

27 www.merlot.org

The Multimedia Educational Resource for Learning and Online Teaching (MERLOT) project began 

in 1997, when the California State University Center for Distributed Learning developed and provided free access to open educational resources.
394 | Modernizing Learning

28 www.oercommons.org

These resources can range from lectures that are 

publicly available on websites like YouTube, to 

electronic books, to entire online courses.

29 ies.ed.gov/ncee/wwc

30 stemedhub.org

31 www.nap.edu

32 www.dtic.mil

Chapter 4 Endnotes

1 Students need non-instructional assimilation 

time—for example, see: Baepler, P., Walker, J. D., 

& Driessen, M. (2014). It’s not about seat time: 

Blending, flipping, and efficiency in active learning 

classrooms. Computers & Education, 78, 227–236.

...varied experiences to aid comprehension—for 

example, see: Kolb, D. A. (2014). Experiential 

learning: Experience as the source of learning and 

development. FT press.

...learning is context-based—for example, see: 

Berns, R. G., & Erickson, P. M. (2001). Contextual teaching and learning: Preparing students for 

the new economy. The Highlight Zone: Research@ 

Work No. 5 (ERIC No. ED452376). Washington, 

DC: Office of Vocational and Adult Education. 

eric.ed.gov/?id=ED452376

2 For example, the U.S. Every Student Succeeds 

Act of 2015, Pub. L. No. 114-95 § 114 Stat. 1177 

(2015–2016). www.ed.gov/essa

3 Ibid. Sweller, J. (1994). Endnote 2-18.

4 Mathers, C.D., Stevens, G.A., Boerma, T., White, 

R.A., & Tobias, M.I. (2015). Causes of international increases in older age life expectancy. The Lancet, 385(9967), 540–548.

5 Bialik, C. (2010, September 4). Seven careers in 

a lifetime? Think twice, researchers say. The Wall 

Street Journal. www.wsj.com

6 For example, see: Park, D.C. & Reuter-Lorenz, 

P.A. (2009). The adaptive brain: Aging and neurocognitive scaffolding. Annual Review of Psychology, 60, 173–196.

7 Bryck, R.L., & Fisher, P.A. (2012). Training the 

brain: Practical applications of neural plasticity 

from the intersection of cognitive neuroscience, 

developmental psychology, and prevention science. American Psychologist, 67(2), 87–100.

8 Power, J.D., & Schlaggar, B.L. (2017). Neural 

plasticity across the lifespan. Wiley Interdisciplinary Reviews: Developmental Biology, 6(1), e216.

9 Ibid. National Academies. (2018). Endnote 1-4.

10 Medel-Añonuevo, C., Ohsako, T., & Mauch, W. 

(2001). Revisiting lifelong learning for the 21st century (ERIC No. ED469790). Hamburg (Germany): 

United Nations Educational, Scientific, and Cultural Organization. eric.ed.gov/?id=ED469790

11 Ibid. National Academies (2018). Endnote 1-4. See 

page 3. 

12 Ibid. For representative examples of Piaget’s and 

Vygotsky’s work, see Endnote 2-20. 

13 Ibid. Sweller, J. (1994). Endnote 2-18.

14 For example, see:

Erikson, E.H. (1950). Growth and crises of the 

“healthy personality.” In M.J.E. Senn (Ed.), Symposium on the healthy personality (pp. 91–146). 

Oxford, England: Josiah Macy, Jr. Foundation.

Bronfenbrenner, U. (1979). The ecology of human 

development. Harvard university press.

15 For instance, as demonstrated by the Diagnostic 

and statistical manual of mental disorders (DSM5®), published 2013.

16 Jones, D.E., Greenberg, M., & Crowley, M. (2015). 

Early social-emotional functioning and public 

health: The relationship between kindergarten social competence and future wellness. American 

Journal of Public Health, 105(11), 2283–2290.

17 Saarni C. (2011). Emotional development in childhood. In R.E. Tremblay RE, M. Boivin, R. Peters, 

& M. Lewis (Eds.), Encyclopedia on Early Childhood Development [online]. www.child-encyclopedia.com/emotions/according-experts/emotional-development-childhood

18 casel.org

19 Schore, A.N. (2015). Affect regulation and the origin of the self: The neurobiology of emotional development. New York: Routledge.

20 For evidence that intra- and interpersonal capabilities can be enhanced through learning see, for 

example: 

Durlak, J.A., Weissberg, R.P., Dymnicki, A.B., 

Taylor, R.D., & Schellinger, K.B. (2011). The impact of enhancing students’ social and emotional 

learning: A meta-analysis of school-based universal 

interventions. Child development, 82(1), 405–432.

Sklad, M., Diekstra, R., Ritter, M. D., Ben, J., & 

Gravesteijn, C. (2012). Effectiveness of schoolbased universal social, emotional, and behavioral 

programs: Do they enhance students’ development 

in the area of skill, behavior, and adjustment? Psychology in the Schools, 49(9), 892–909.
Endnotes | 395 

21 See, for example: 

Vogel-Walcutt, J.J., Ross, K.G., & Phillips, J.K., 

(2016). Instructional design roadmap: Principles 

to maximize learning across developmental stages. 

In Proceedings of the I/ITSEC. Arlington, VA: National Training and Simulation Association.

Vogel-Walcutt, J.J., Ross, K.G., Phillips, J.K., 

& Knarr, K.A. (2015). Marine Corps instructor 

mastery model: A foundation for Marine faculty 

professional development. In Proceedings of the 

I/ITSEC. Arlington, VA: National Training and 

Simulation Association.

Morrison, J.G., Kelly, R.T., Moore, R.A., & 

Hutchins, S.G. (1998). Implications of decisionmaking research for decision support and displays. 

In J.A. Cannon-Bowers & E. Salas (Eds.), Making 

decisions under stress: Implications for individuals 

and teams (pp. 375–406). Washington, DC: APA

22 Payne, V.G., & Isaacs, L.D. (2017). Human motor 

development: A lifespan approach (9th ed.). London/New York: Routledge.

23 Edwards, L.C., Bryant, A.S., Keegan, R.J., Morgan, K., & Jones, A.M. (2017). Definitions, foundations and associations of physical literacy: a systematic review. Sports medicine, 47(1), 113–126.

24 For example, see: Swan, M. (2013). The quantified 

self: Fundamental disruption in big data science 

and biological discovery. Big data, 1(2), 85–99.

25 Ghayvat, H., Liu, J., Mukhopadhyay, S.C., & Gui, 

X. (2015). Wellness sensor networks: A proposal 

and implementation for smart home for assisted 

living. IEEE Sensors Journal, 15(12), 7341–7348.

26 Ibid. Edwards et al. (2017). Endnote 4-23.

27 Laal, M. (2011). Lifelong learning: what does it 

mean? Procedia – Social and Behavioral Sciences, 

28, 470–474.

28 Kalz, M. (2015). Lifelong learning and its support 

with new technologies. International Encyclopedia of the Social & Behavioral Sciences (2nd ed.), 

93–99. 

29 For example, see: Weinberg, R.S., & Gould, D.S. 

(2018). Foundations of sport and exercise psychology (7th ed.). Human Kinetics.

30 Sae Schatz and her colleagues have dubbed this 

integrated approach “Industrial Knowledge Design” and outline it in an article, see: Schatz, S., 

Berking, P., & Raybourn, E. M. (2017). Industrial knowledge design: an approach for designing 

information artefacts. Theoretical Issues in Ergonomics Science, 18(6), 501–518.

Also refer to Chapter 5 in this volume.

Chapter 5 Endnotes

1 von Clausewitz, C. (1989). On War. (M. Howard 

and P. Paret, Eds. and Trans.) Princeton University 

Press. (Original work published 1832)

Or refer to the O. Jolles translation (1943)

2 Viegas, S. Antunes Teixeira L.A., Andrade, D.F. & 

Moreira Silva, J.T. (2015). The information overload due to attention, interruptions and multitasks. 

Australian Journal of Basic and Applied Sciences, 

9(27): 603–613.

3 Hemp, P. (2009). Death by information overload. 

Harvard Business Review, 87(9), 82-9.

Spira, J.B., & Feintuch, J.B. (2005). The cost of not 

paying attention: How interruptions impact knowledge worker productivity. Report from Basex, Inc.

4 Brown, T. E. (2006). Attention deficit disorder: The 

unfocused mind in children and adults. Yale University Press. (As discussed in ibid. Viegas et al., 

2015, Endnote 5-2.)

5 Benedek, A. (2013). Mobile multimedia-based 

knowledge transfer: A toolkit and a 3.0 Reference 

model. In L. Gómez Chova, A. López Martínez, 

& I. Candel Torres (Eds.), Proceedings of EDULEARN 13 (pp. 1926-1936). IATED.

6 Kilgore, W. (2018, December 27). UX to LX: The 

rise of learner experience design. EdSurge. Retrieved from www.edsurge.com.

7 Floor, N. (2018, March 5). Learning experience 

design is NOT a new name for instructional design 

[blog post]. Learning Experience Design.com. Retrieved from www.learningexperiencedesign.com.

8 Weigel, M. (2015, April 2). Learning experience 

design vs. user experience: Moving from “user” 

to “learner” [blog post]. Six Red Marbles. www.

sixredmarbles.com

9 Schatz, S. Berking P. & Raybourn, E.M. (2017). 

Industrial knowledge design: An approach for designing information artefacts. Theoretical Issues in 

Ergonomics Science, 18(6), 501–518.

10 Battarbee, K., & Mattelmäki, T. (2003). Meaningful product relationships. In Design and Emotion, 

Vol. 1 (pp. 337–344).

11 For example, see: 

Pine, B.J. & Gilmore, J.H. (1998). Welcome to the 

experience economy. Harvard business review, 76, 

97–105.

Pullman, M.E. & Gross, M.A. (2004). Ability of experience design elements to elicit emotions and loyalty behaviors. Decision Sciences, 35(3), 551–578. 
396 | Modernizing Learning

Schmitt, B. (1999). Experiential marketing. The 

Free Press.

12 Berry, L.L., Carbone, L.P., & Haeckel, S.H. 

(2002). Managing the total customer experience. 

MIT Sloan Management Review.

13 Ibid. see Pullman & Gross (2004) and Berry et al. 

(2002), respectively. Endnote 5-11.

14 Kolb, D. (1984). Experiential learning: experience 

as the source of learning and development. Prentice Hall. See page 41.

15 Kolb, A.Y. & Kolb, D.A. (2009). The learning 

way: Meta-cognitive aspects of experiential learning. Simulation & Gaming. 40(3), 297–327. See 

page 298.

16 National Society for Experiential (2013, December 

9). Eight principles of good practice for all experiential learning activities. National Society for Experiential Education. www.nsee.org/8-principles

17 Ibid. Schmitt (1999). Endnote 5-11.

18 Ariely, D. (2008). Predictably irrational: the hidden forces that shape our decisions. New York: 

HarperCollins.

19 Cialdini, R. (2009). Influence: Science and practice. Boston, MA: Pearson Education.

20 Thaler, R.H., & Sunstein, C.R. (2009). Nudge: Improving decisions about health, wealth, and happiness. London: Penguin.

21 See, for example: 

Moss, M. (2013, August 27). Nudged to the produce aisle by a look in the mirror. The New York 

Times. www.nytimes.com

Bannister, J. (2010, July 19). NMSU researchers 

shop around for healthier grocery carts [press release]. New Mexico State University. newscenter.

nmsu.edu

22 Kerwin, W.T., Blanchard, G.S., Atzinger, E.M., & 

Topper, P.E. (1980). Man/machine interface – a 

growing crisis (DTIC Accession No.ADB071081). 

U.S. Army Material Systems Analysis Activity.

23 Garrett, J.J. (2010). Elements of user experience, 

the: User-centered design for the web and beyond. 

Pearson Education.

24 Pew, R.W., Mavor, A.S., et al. (2007). Human-System Integration in the system development process: 

A new look. National Academies Press. See page 191. 

NOTE: A digital version of this book is openly and 

publicly available at nap.edu/11893

25 Interaction Design Foundation (2017). Learning 

experience design: The most valuable lessons 

[blog post]. The Interaction Design Foundation. 

www.interaction-design.org.

26 Rifai, N., Rose, T., McMahon, G.T., Saxberg, B., 

& Christensen, U.J. (2018). Learning in the 21st

century: Concepts and tools. Clinical chemistry, 

64(10), 1423–1429.

27 Norman, D. (2005). Emotional design: Why we 

love (or hate) everyday things. Basic Books.

ENDNOTES FOR SECTION 2 

(TECHNOLOGY)

Chapter 6 Endnotes

1 Siemens, G., Dawson S. and Eshleman, K. (2018, 

October 29). Complexity: A leader’s framework for 

understanding and managing change in higher education. EDUCAUSE Review, 53(6). educause.edu

2 Rainie, L. & Anderson, J. (2017, May 3). The future of jobs and jobs training. Pew Research Center. www.pewinternet.org

3 Johnson-Freese, J. (2012) The reform of military 

education: Twenty-five years later (ADA570086). 

Philadelphia, PA: Foreign Policy Research Inst. 

apps.dtic.mil/docs/citations/ADA570086

4 Task Force on Defense Personnel (2017). The 

building blocks of a ready military: People, funding, tempo. Washington, DC: Bipartisan Policy 

Center. bipartisanpolicy.org

5 lrmi.dublincore.org

6 schema.org

7 schema.org/Course

8 U.S. Office of Personnel Management (2018). 

OPM strategic plan fiscal years 2018–2022. Washington, DC: OPM. www.opm.gov

9 Growth Engineering (2017, May). Informal learning: What is the 70:20:10 model? [blog post]. 

Growth Engineering. www.growthengineering.

co.uk/70-20-10-model

10 Blackman, D.A., Johnson, S.J., Buick, F., Faifua, 

D.E., O’Donnell, M., & Forsythe, M. (2016). The 

70:20:10 model for learning and development: An 

effective model for capability development? In 

Academy of Management Proceedings, Vol. 2016 

(p. 10745). Briarcliff Manor, NY: Academy of 

Management.
Endnotes | 397 

11 Tyszko, J.A., Sheets, R.G., Reamer, A.D. (2017). 

Clearer signals: Building an employer-led job 

registry for talent pipeline management. Washington, DC: U.S. Chamber of Commerce Foundation. 

www.luminafoundation.org

Chapter 7 Endnotes

1 For example, see: Cybrary.it. Free hacking training. www.cybrary.it/freehackingtraining

2 Fu, H., Liao, J., Yang, J., Wang, L., Song, Z., 

Huang, X., et al. (2016). The Sunway TaihuLight 

supercomputer: system and applications. Science 

China Information Sciences, 59(7), 072001.

Hruby, D. (2018). Putting China’s science on the 

map. Nature, 553(7688).

3 Gordon, L.A., Loeb, M.P., Lucyshyn, W., & Zhou, 

L. (2015). Externalities and the magnitude of cyber 

security underinvestment by private sector firms: a 

modification of the Gordon-Loeb model. Journal 

of Information Security, 6(1), 24–30.

4 Rustici Software. (n.d.) The layers of Experience 

API. xapi.com/the-layers-of-experience-api-xapi

5 Ramirez-Padron, R. (2017, July) Pushing xAPI 

statements in real time: Part 3. tlacommunity.com/

pushing-xapi-statements-in-real-time-part-3

6 Ibid. Ramirez-Padron (2017). Endnote 7-5.

7 Perrow, C. (1984). Normal accidents: Living with 

high-risk technologies. New York: Basic Books.

8 Ibid. Perrow (1984). Endnote 7-7.

9 Bambauer, D.E. (2014). Ghost in the network. 

University of Pennsylvania Law Review, 162, pp. 

1011–1091.

Lally, L. (2005). Information technology as a target 

and shield in the post 9/11 environment. Information 

Resources Management Journal, 18, pp. 14–28.

10 Massachusetts Institute of Technology. (2018). 

Kerberos: The network authentication protocol. 

web.mit.edu/kerberos

Chapter 8 Endnotes

1 Sandia National Laboratories is a multimission 

laboratory managed and operated by National 

Technology & Engineering Solutions of Sandia, 

LLC, a wholly owned subsidiary of Honeywell 

International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration 

under contract DE-NA0003525.

2 Cavoukian, A. (2009). Privacy by design. Ontario, 

Canada: Information and Privacy Commissioner of 

Ontario. http://www.ontla.on.ca/library/repository/

mon/23002/289982.pdf

3 Knijnenburg, B. P. (2015). A user-tailored approach to privacy decision support (Doctoral dissertation, UC Irvine).

4 Westin, A.F., Harris, L. et al. (1981). The Dimensions of privacy: A national opinion research survey of attitudes toward privacy. New York: Garland.

5 Chellappa, R.K. & Sin, R.G. (2005). Personalization versus privacy: An empirical examination of 

the online consumer’s dilemma. Information Technology and Management. 6(2), 181–202.

6 Malhotra, N.K., Kim, S.S. & Agarwal, J. (2004). 

Internet users’ information privacy concerns 

(IUIPC): The construct, the scale, and a nomological framework. Information Systems Research. 

15(4), 336–355.

7 Knijnenburg, B.P. & Cherry, D. (2016, June). Comics as a medium for privacy notices. Paper presented at the SOUPS 2016 workshop on the Future of 

Privacy Notices and Indicators, Denver, CO.

8 Wisniewski, P.J., Knijnenburg, B.P., & Lipford, 

H.R. (2017). Making privacy personal: Profiling 

social network users to inform privacy education and nudging. International Journal of Human-Computer Studies. 98, 95–108. 

9 Ibid. Wisniewski et al. (2017). Endnote 8-7.

10 Narayanan, A. & Shmatikov, V. (2008). Robust 

de-anonymization of large sparse datasets. In IEEE 

Symposium on Security and Privacy (pp. 111-125).

11 Ibid. Chellappa & Sin (2005). Endnote 8-4.

12 Gootman, S. (2016). OPM hack: The most dangerous threat to the federal government today. Journal 

of Applied Security Research. 11(4), 517–525. 

13 Kobsa, A., Cho, H., & Knijnenburg, B.P. (2016). 

The effect of personalization provider characteristics on privacy attitudes and behaviors: An elaboration likelihood model approach. Journal of the 

Association for Information Science and Technology, 67(11), 2587–2606.

14 Knijnenburg, B.P., Sivakumar, S., & Wilkinson, 

D. (2016). Recommender systems for self-actualization. In S. Sen & W. Geyer (Eds.), Proceedings 

of the 10th ACM Conference on Recommender Systems (pp. 11–14). New York: ACM.
398 | Modernizing Learning

15 Page, X., Knijnenburg, B.P., & Kobsa, A. (2013). 

FYI: Communication style preferences underlie 

differences in location-sharing adoption and usage. 

In F. Mattern & S. Santini (Eds.), Proceedings of 

the 2013 ACM international joint conference on 

Pervasive and ubiquitous computing (pp. 153–

162). New York: ACM.

16 Teltzrow, M. & Kobsa, A (2004). Impacts of user 

privacy preferences on personalized systems: a 

comparative study. In C.M. Karat, J. Blom, & J. 

Karat (Eds.), Designing Personalized User Experiences for eCommerce (pp. 315–332). Norwell, 

MA: Kluwer Academic Publishers.

17 Compañó, R. & Lusoli, W. (2010). The policy 

maker’s anguish: Regulating personal data behavior between paradoxes and dilemmas. In T. Moore, 

D. Pym, & C. Ioannidis (Eds.), Economics of Information Security and Privacy (pp. 169–185). Boston, MA: Springer.

18 Nissenbaum, H. (2011). A contextual approach to 

privacy online. Daedalus. 140(4), 32–48. 

19 Kay, M. & Terry, M. (2010). Textured agreements: 

Re-envisioning electronic consent. In L.F. Cranor 

(Ed.), Proceedings of the Sixth Symposium on Usable Privacy and Security (pp. 13:1–13:13). New 

York: ACM. 

20 Wisniewski, P., Islam, A.K.M.N., Knijnenburg, 

B.P., & Patil, S. (2015). Give social network users the privacy they want. In D. Cosley & A. Forte 

(Eds.), Proceedings of the 18th ACM Conference 

on Computer Supported Cooperative Work & Social Computing (pp. 1427–1441). New York: ACM.

21 Ibid. Compañó & Lusoli (2010). Endnote 8-16.

22 Spiekermann, S., Grossklags, J., & Berendt, B. 

(2001). E-privacy in 2nd generation e-commerce: 

Privacy preferences versus actual behavior. In 

M.P. Wellman & Y. Shoham (Eds.), Proceedings of 

the 3rd ACM Conference on Electronic Commerce 

(pp. 38–47). New York: ACM.

23 Ibid. Knijnenburg (2015). Endnote 8-2.

24 Ibid. Knijnenburg (2015). Endnote 8-2.

Chapter 9 Endnotes

1 For those curious about the similarities and differences between Educational Data Mining and 

Learning Analytics, Ryan Baker and Paul Salvador 

Inventado authored a detailed chapter, see: Ibid. 

Baker & Inventado (2014). Endnote 2-82.

2 Siemens, G. & Baker, R.S.J.D. (2012). Learning 

analytics and educational data mining: towards 

communication and collaboration. In S.B. Shum, D. 

Gasevic, & R. Ferguson (Eds.), Proceedings of the 

2nd International Conference on Learning Analytics 

and Knowledge (pp. 252–254). New York: ACM.

3 The phrase “big learning data” is a nod to Maise, E. 

(Ed.). (2014). Big Learning Data. Alexandria, VA: 

American Society for Training and Development.

4 Crowder, M., Antoniadou, M., & Stewart, J. 

(2018). To BlikBook or not to BlikBook: Exploring student engagement of an online discussion 

platform. Innovations in Education and Teaching 

International, 1–12.

5 Mazza, R. & Dimitrova, V. (2004) Visualising student tracking data to support instructors in web-based 

distance education. In S. Feldman & M. Uretsky 

(Eds.), Proceedings of the WWW Alt. ’04: 13th International World Wide Web Conference on Alternate Track Papers and Posters. New York: ACM.

6 Arnold, K.E., & Pistilli, M.D. (2012). Course signals at Purdue: Using learning analytics to increase 

student success. In Proceedings of the 2nd international conference on learning analytics and knowledge (pp. 267–270). New York: ACM. 

NOTE: However, readers are cautioned that Course 

Signals may have produced undesirable effects, 

possibly encouraging under-performing students 

to withdrawal. This highlights the importance of 

considering, not just analyzing data, but how to effectively use it to achieve the desired outcomes.

7 For a succinct overview of learning analytics and 

educational data mining, as well as several usecase examples, see: Charlton, P., Mavrikis, M., & 

Katsifli, D. (2013). The potential of learning analytics and big data. Ariadne. www.ariadne.ac.uk/

issue/71/charlton-et-al

8 For additional examples and a useful historical 

account of educational data mining and learning 

analytics, see: Ferguson, R. (2012). Learning analytics: drivers, developments and challenges. International Journal of Technology Enhanced Learning, 4(5/6) pp. 304–317.

9 For a systematic review of learning analytics and 

educational dashboards, from scholarly articles 

published between 2011 and 2015, see: Schwendimann, B.A., Rodriguez-Triana, M.J., Vozniuk, 

A., Prieto, L.P., Boroujeni, M.S., Holzer, A., et 

al. (2017). Perceiving learning at a glance: A systematic literature review of learning dashboard 

research. IEEE Transactions on Learning Technologies, 10(1), 30-41.
Endnotes | 399 

10 “Open-learner” (or “open-student”) models provide learning visualizations of the underlying user 

model within a system the learners, themselves. 

Open-learner models appear to enhance outcomes 

(particularly for lower performing students) and 

increase motivation, improve self-awareness, and 

support self-directed learning. Robert Bodily and 

colleagues recently published a review comparing 

learning analytics dashboards with open learner 

models: 

Bodily, R., Kay, J., Aleven, V., Jivet, I., Davis, D., 

Xhakaj, F., & Verbert, K. (2018). Open learner 

models and learning analytics dashboards: a systematic review. In A. Pardo, K. Bartimote-Aufflick, G. Lynch (Eds.), Proceedings of the 8th

International Conference on Learning Analytics 

and Knowledge (pp. 41–50). New York: ACM.

Some other useful sources include: 

Mitrovic, A., & Martin, B. (2002). Evaluating the 

effects of open student models on learning. In 

P. De Bra, P. Brusilovski, & R. Conejo (Eds.), 

International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems (pp. 

296–305). Berlin, Heidelberg: Springer, 

Chou, C.Y., Tseng, S.F., Chih, W.C., Chen, Z.H., 

Chao, P.Y., Lai, K.R. et al. (2017). Open student 

models of core competencies at the curriculum 

level: Using learning analytics for student reflection. IEEE Transactions on Emerging Topics in 

Computing, 5(1), 32–44.

Brusilovsky, P., Somyürek, S., Guerra, J., Hosseini, 

R., & Zadorozhny, V. (2015, June). The value 

of social: Comparing open student modeling 

and open social student modeling. In F. Ricci, 

K. Bontcheva, O. Conlan, & S. Lawless (Eds.), 

User Modeling, Adaptation and Personalization. UMAP 2015. Lecture Notes in Computer 

Science, Vol. 9146 (pp. 44–55). Cham: Springer.

11 For background information on Kappa see: github.

com/milinda/kappa-architecture.com

The original paper describing this paradigm came 

out of work done at LinkedIn, see: Kreps, J. (2014, 

July 2). Questioning the Lambda Architecture: The 

Lambda Architecture has its merits, but alternatives are worth exploring. O’Reilly. www.oreilly.

com/ideas/questioning-the-lambda-architecture

Also, for a deeper dive, see the presentation by 

Martin Kleppmann at youtu.be/fU9hR3kiOK0

12 kafka.apache.org

13 The full quote is, “There are three kinds of lies: 

lies, damned lies, and statistics.” This is often attributed to Mark Twain who, in turn, attributes it to 

the British prime minister Benjamin Disraeli. It’s 

not entirely clear who invented the witticism, but 

it’s sentiment is obvious. For more delightful etymology about it, see: Velleman, P. F. (2008). Truth, 

damn truth, and statistics. Journal of Statistics Education, 16(2).

Chapter 10 Endnotes

1 Ibid. Kulik & Fletcher (2016). Endnote 2-11.

2 Raybourn, E.M., Deagle, E., Mendini, K., & 

Heneghan, J. (2005). Adaptive thinking and leadership simulation game training for special forces 

officers. In Proceedings of the I/ITSEC. Arlington, 

VA: National Training and Simulation Association.

3 Steenbergen-Hu, S., & Cooper, H. (2014). A meta-analysis of the effectiveness of intelligent tutoring systems on college students’ academic learning. 

Journal of Educational Psychology, 106(2), 331.

4 VanLehn, K. (2011). The relative effectiveness of 

human tutoring, intelligent tutoring systems, and 

other tutoring systems. Educational Psychologist, 

46(4), 197-221.

5 Ibid. Kulik & Fletcher (2016). Endnote 2-11.

6 Murray, T. (1999). Authoring intelligent tutoring 

systems: An analysis of the state of the art. International Journal of Artificial Intelligence in Education, 10, 98–129.

7 Koedinger, K.R., McLaughlin, E.A., & Stamper, J. C. (2014). Data-driven learner modeling to 

understand and improve online learning: MOOCs 

and technology to advance learning and learning 

research Ubiquity, 2014(May), 3.

8 Goldberg, B., Schatz, S., & Nicholson, D. (2010). 

A practitioner’s guide to personalized instruction: 

Macro-adaptive approaches for use with instructional technologies. In D. Kaber & G. Boy (Eds.) 

Proceedings of the 2010 Applied Human Factors 

and Ergonomics Conference: Advances in Cognitive Ergonomics (pp. 735–745). Boca Raton, FL: 

CRC Press.

9 Young, J.R. (2018). Keystroke dynamics: Utilizing keyprint biometrics to identify users in online 

courses. (Doctoral dissertation, Brigham Young 

University).

10 Beck, J.E., & Woolf, B.P. (2000, June). High-level 

student modeling with machine learning. In International Conference on Intelligent Tutoring Systems (pp. 584–593). Berlin, Heidelberg: Springer.
400 | Modernizing Learning

11 Sottilare, R.A., Brawner, K.W., Goldberg, B.S., & 

Holden, H.K. (2012). The generalized intelligent 

framework for tutoring (GIFT). Orlando, FL: US 

Army Research Laboratory–Human Research & 

Engineering Directorate. gifttutoring.org

12 Nye, B. D. (2016). Its, the end of the world as we 

know it: Transitioning AIED into a service-oriented ecosystem. International Journal of Artificial 

Intelligence in Education, 26(2), 756–770.

13 Folsom-Kovarik, J.T., Jones, R.M., & Schmorrow, 

D. (2016). Semantic and episodic learning to integrate diverse opportunities for life-long learning. 

In Proceedings of MODSIM World. Arlington, VA: 

National Training and Simulation Association.

14 Weinstein, Y., & Roediger, H. L. (2012). The effect of question order on evaluations of test performance: how does the bias evolve? Memory & 

Cognition, 40(5), 727–735

15 For review of data quality, see: Pipino, L.L., Lee, 

Y.W., & Wang, R.Y. (2002). Data quality assessment. Communications of the ACM, 45(4), 211–218. 

For a discussion on data fairness, see also Brun, 

Y., & Meliou, A. (2018). Software fairness. In 

Proceedings of the 26th ESEC/FSE (pp. 754–759). 

New York: ACM.

16 Soh, L.K., & Blank, T. (2008). Integrating casebased reasoning and meta-learning for a self-improving intelligent tutoring system. International 

Journal of Artificial Intelligence in Education, 

18(1), 27–58.

17 Folsom-Kovarik, J.T., Wray, R.E., & Hamel, L. 

(2013). Adaptive assessment in an instructor-mediated system. In Proceedings of the International 

Conference on Artificial Intelligence in Education 

(pp. 571–574). Berlin, Heidelberg: Springer.

18 Krening, S., Harrison, B., Feigh, K.M., Isbell, C. 

L., Riedl, M., & Thomaz, A. (2017). Learning from 

explanations using sentiment and advice. IEEE 

Transactions on Cognitive and Developmental 

Systems, 9(1), 44–55.

19 Aleven, V., Xhakaj, F., Holstein, K., & McLaren, 

B.M. (2016). Developing a teacher dashboard for 

use with intelligent tutoring systems. In IWTA@ 

EC-TEL (pp. 15-23). New York: ACM.

20 Czarkowski, M., & Kay, J. (2006). Giving learners 

a real sense of control over adaptivity, even if they 

are not quite ready for it yet. In M. Czarkowski, & 

J. Kay (Eds.), Advances in Web-based education: 

Personalized learning environments (pp. 93–126). 

Hershey, PA: IGI Global.

21 Schmorrow, D., Nicholson, D., Lackey, S.J., Allen, 

R.C., Norman, K., & Cohn, J. (2009). Virtual reality in the training environment In P.A. Hancock, 

D.A. Vincenzi, J.A. Wise, & M. Mouloua (Eds.), 

Human Factors in Simulation and Training. Boca 

Raton, FL: CRC Press. 

ENDNOTES FOR SECTION 3 

(LEARNING SCIENCE)

Chapter 11 Endnotes

1 NOTE: The views expressed in this chapter are entirely those of the author, a contractor with Metis 

Solutions, and do not necessarily reflect the views, 

policy, or position of the United States Government, Department of Defense, United States Special Operations Command, or the Joint Special Operations University. 

2 Shute, V. & Ventura, M. (2013). Stealth assessment: Measuring and supporting learning in video 

games. Boston, MA: MIT Press.

3 Hattie, J. (2009). Visible learning: A synthesis of 

over 800 meta-analyses relating to achievement. 

Abingdon, UK: Routledge. See page 24.

4 Hatfield, S. (2009). Assessing your program-level 

assessment plan. Number 45 in IDEA paper series. 

Manhattan, KS: IDEA Center. See page 1.

5 Sadler, D.R. (1989). Formative assessment and the 

design of instructional systems. Instructional Science, 18, 119–144. See page 121.

6 Sadler, D.R. (2010). Beyond feedback: Developing 

student capability in complex appraisal. Assessment & Evaluation in Higher Education, 35(5), 

535–550.

7 Hattie, J. & Timperley, H. (2007). The power of 

feedback. Review of Educational Research, 77(1), 

81–112.

8 For examples of stealth assessment, see: 

Shute, V. & Spector, J. M. (2008). SCORM 2.0 

white paper: Stealth assessment in virtual worlds. 

Unpublished manuscript.

Ibid. Shute & Ventura (2013). Endnote 11-2.

9 Thille, C. (2015, November 19). Big data, the science of learning, analytics, and transformation of 

education [Video file]. Presented at media X conference Platforms for Collaboration and Productivity, Stanford University. https://youtu.be/cYqs0Ei2tFo
Endnotes | 401 

10 Ibid. Thille (2015). Endnote 11-9.

11 Dron, J. (2007a). Designing the undesignable: Social software and control. Educational Technology 

and Society, 10(3), 60–71.

Dron, J. (2007b). The teacher, the learner and the 

collective mind. AI & Society, 21, 200–216.

12 Ibid. Dron (2007a). Endnote 11-11. See page 61.

13 Ibid. Hattie & Timperley (2007). Endnote 11-7.

14 Scott, S. (2013). Practicing what we preach: Towards a student-centered definition of feedback. 

Teaching in Higher Education, 19(1), 49–57.

15 Boud, D. & Molloy, E. (2013a). Feedback in higher and professional education: Understanding it 

and doing it well. Abingdon: Routledge.

Boud, D. & Molloy, E. (2013b). Rethinking models 

of feedback for learning: The challenge of design. 

Assessment & Evaluation in Higher Education, 38 

(6), 698–712.

16 Carey, B. (2014). How we learn: The surprising 

truth about when, where, and why it happens. New 

York: Random House.

17 Wiggins, G. & McTighe, J. (2005). Understanding 

by design, 2nd edition. Alexandria, VA: Association 

for Supervision and Curriculum Development.

18 Michael, N. & Libarkin, J. (2016). Understanding 

by design: Mentored implementation of backward design methodology at the university level. 

Bioscene, 42(2), 44–52.

Reynolds, H. & Kearns, K. (2017). A planning tool 

for incorporating backward design, active learning, 

and authentic assessment in the college classroom. 

College Teaching, 65(1), 17–27.

19 Ibid. Boud & Molloy (2013b). Endnote 11-15.

Hattie & Timperley (2007). Endnote 11-7.

20 Wiliam, D. (2018). Embedded formative feedback. 

Bloomington, IN: Solution Tree Press. See page 29.

Chapter 12 Endnotes

1 Siemens, G. (2006). Connectivism: Learning theory or pastime of the self-amused? Unpublished 

manuscript.

2 Del Moral-Pérez, E., Cernea, A., & Villalustre, L. 

(2013). Connectivist learning objects and learning 

styles. Interdisciplinary Journal of e-Skills and 

Lifelong Learning, 9, 105–124.

Siemens, G. (2008). Learning and knowing in networks: Changing roles for educators and designers. 

ITFORUM for Discussion, 27, 1–26.

Ibid. Siemens (2006). Endnote 12-1.

3 Wiggins, G., & McTighe, J. (1998). What is backward design? Understanding by Design, 1, 7–19.

4 Akdeniz, C. (2016). Instructional strategies. In C. 

Akdeniz (Ed.), Instructional Process and Concepts 

in Theory and Practice (pp. 57–105). Singapore: 

Springer.

5 Jonassen, D.H., Grabinger, R.S., & Harris, N.D. 

C. (1990). Analyzing and selecting instructional 

strategies and tactics. Performance improvement 

quarterly, 3(2), 29–47.

6 Refer to Chapter 11 in this volume

See also Ibid. Dron, J. (2007a & b). Endnote 11-11. 

7 Marr, D.(1982). Vision. San Francisco: W. H. Freeman. See pages 19–20.

8 Ibid. Hattie, J. (2009). Endnote 11-3.

9 Ertmer, P. A., & Newby, T. J. (2013). Behaviorism, 

cognitivism, constructivism: Comparing critical 

features from an instructional design perspective. 

Performance Improvement Quarterly, 26(2), 43–71.

10 Ibid. Del Moral-Pérez et al. (2013). Endnote 12-2.

11 Dabbagh, N., Marra, R.M., & Howland, J.L. 

(2019). Meaningful online learning: Integrating 

strategies, activities and learning technologies for 

effective designs. New York: Routledge.

12 Ibid. Dabbagh et al. (2019). Endnote 12-11.

Chapter 13 Endnotes

1 Air Force Instruction 36-2201, Air Force Training 

Program, see paragraph 4.1.2.

2 Bloom’s taxonomy identified the following general 

categories associated with mastery in the cognitive 

domain (starting with the simplest and moving to 

the most complex): knowledge, comprehension, 

application, analysis, synthesis, and evaluation. 

Bloom, B.S., Engelhart, M.D., Furst, E.J., Hill, 

W.H., Krathwohl, D.R. (1956). Taxonomy of educational objectives: The classification of educational goals. Handbook I: Cognitive domain. New 

York: David McKay Company.

3 Doyle, T. (2008). Helping students learn in a learner-centered environment: A guide to facilitating 

learning in higher education. Sterling, VA: Stylus.

4 See the Direct Assessment: Competency-Based 

Educational Programs policy from the Southern 

Association of Colleges and Schools Commission 

on Colleges. www.sacscoc.org/pdf/081705/DirectAssessmentCompetencyBased.pdf
402 | Modernizing Learning

5 U.S. Office of Personnel Management. (n.d.). Assessment & selection: Competencies. Retrieved 

January 13, 2019 from www.opm.gov

6 U.S. Air Force (2014). Institutional Competency 

Development and Management (Air Force Manual 

36-2647). www.e-publishing.af.mil

7 Bramante, F., & Colby, R. (2012). Off the clock: 

Moving education from time to competency. Thousand Oaks, CA: Corwin. See page 65.

8 Spencer, L. & Spencer, S.M. (1993). competence 

at work: Models for superior performance. New 

York: John Wiley & Sons.

9 Lucia, A.D., & Lepsinger, R. (1999). The Art and 

Science of competency models: Pinpointing critical success factors in organizations. San Francisco: Jossey-Bass/Pfeiffer.

10 Ibid. Bramante & Colby (2012). Endnote 13-7.

11 Ward, S.C. (2016, February 1) Let them eat cake 

(competently). Inside Higher Education. www.insidehighered.com

12 Hollenbeck, G. & Morgan, M. (2013). Competencies, not competencies: Making global executive 

development work. Advances in global leadership 

(pp. 101–119). Emerald Group Publishing Limited.

13 U.S. Department of Energy (2013). U.S. Department of Energy Leadership Development Programs 2013–2014: Readings by Executive Core 

Qualifications. www.opm.gov

14 Krauss, S.M., (2017). How Competency-Based 

Education May Help Reduce Our Nation’s Toughest Inequities (Issue Paper). Indianapolis: Lumina 

Foundation. www.luminafoundation.org

15 www.luminafoundation.org/priorities

16 Voorhees, R.A. Competency-based learning models: A necessary future. New directions for institutional research, 2001(110), 5–13.

Chapter 14 Endnotes

1 Stodd, J. (2015, October 30). An introduction to 

Scaffolded Social Learning [blog post]. Julian 

Stodd’s Learning Blog. Retrieved August 28, 2018 

from julianstodd.wordpress.com

2 Foster, R.E. & Fletcher, J.D. (2002). Computer-based aids for learning, job performance, and 

decision-making in military applications: Emergent technology and challenges (IDA Document 

D-2786). Alexandria, VA: Institute for Defense 

Analyses.

3 Raybourn, E., Schatz, S., Vogel-Walcutt, J.J., and 

Vierling, K. (2017). At the tipping point: Learning 

science and technology as key strategic enablers 

for the future of defense and security. In Proceedings of the I/ITSEC. Arlington, VA: National Training and Simulation Association. 

4 Stodd, J. (2018, July 10). Context of the Social Age 

[blog post]. Julian Stodd’s Learning Blog. Retrieved 

August 28, 2018 from julianstodd.wordpress.com

5 The Economist. (2017, January 14). The return of 

the MOOC: Established education providers v new 

contenders. The Economist. Retrieved February 2, 

2018 from www.economist.com

6 Stodd, J & Reitz, E.A. (2016). Black swans and the 

limits of hierarchy. n Proceedings of the I/ITSEC. 

Arlington, VA: National Training and Simulation 

Association.

7 Brafman, O., & Beckstrom, R. A. (2006). The 

starfish and the spider: The unstoppable power of 

leaderless organizations. Penguin.

8 Stodd, J. (2017, January 17). 10 Tips for designing 

effective social learning [blog post]. Julian Stodd’s 

Learning Blog. Retrieved August 28, 2018 from 

julianstodd.wordpress.com

9 Ibid. Stodd (2015). Endnote 14-1.

10 O’Neil, H.F., Perez, R.S., & Baker, E.L., Eds. 

(2014). Teaching and measuring cognitive readiness. New York, NY: Springer.

11 Jarche, H. (2014, January). The seek-sense-share 

framework. Inside Learning Technologies. jarche.

com/2014/02/the-seek-sense-share-framework

12 St. Clair, R.N., Thome-Williams, A.C., & Su, L. 

(2005). The role of social script theory in cognitive 

blending. In M. Medina & L. Wagner (Eds.), Intercultural Communication Studies, 15(1), 1–7.

Chapter 15 Endnotes

1 Yarnall, L., Remold, J., & Shechtman, N. (2018, 

October). Developing employability skills: Harvesting ideas from the field. Presentation at the 

annual principal investigators’ conference of the 

Advanced Technological Education program, National Science Foundation, Washington, DC.

2 Winters, F.I., Greene, J.A., & Costich, C.M. 

(2008). Self-regulation of learning within computer-based learning environments: A critical analysis. 

Educational Psychology Review, 20(4), 429–444.

3 Azevedo, R. (2014). Issues in dealing with sequential and temporal characteristics of self- and socially-regulated learning. Metacognition Learning, 9, 

217–228. 
Endnotes | 403 

4 Marsick, V.J., & Watkins, K.E. (2001). Informal 

and incidental learning. New Directions for Adult 

and Continuing Education, 2001(89), 25–34.

5 Hu, H., & Driscoll, M.P. (2013). Self-regulation in 

e-learning environments: A remedy for community 

college? Educational Technology & Society, 16(4), 

171–184. 

Sitzmann, T., & Ely, K. (2011). A meta-analysis of 

self-regulated learning in work-related training and 

educational attainment: What we know and where 

we need to go. Psychological Bulletin, 137(3), 421.

6 Zimmerman, B.J. (1990). Self-regulating academic learning and achievement: The emergence of a 

social cognitive perspective. Educational Psychology Review, 2, 173–201.

7 Ibid. Sitzmann & Ely (2011). Endnote 15-5.

8 Bandura, A. (1991). Social cognitive theory of 

self-regulation. Organizational Behavior and Human Decision Processes, 50, 248–287.

Rotter, J. B. (1990). Internal versus external control of reinforcement: A case history of a variable. 

American Psychologist, 45(4), 489.

9 Zimmerman, B.J. (2000). Attaining self-regulation: 

A social-cognitive perspective. In M. Boekaerts 

& P.R. Pintrich (Eds.), Handbook of Self-Regulation (pp. 13–39). New York: Academic Press.

10 For example, see: Moos, D.C. & Azevedo, R. (2008). 

Exploring the fluctuation of motivation and use of 

self-regulatory processes during learning with hypermedia. Instructional Science, 36(3), 203–231.

11 Baker, L. & Brown, A.L. (1984). Metacognitive 

skills and reading. In P.D. Pearson (Ed.), Handbook of reading research (pp. 353–394). Mahwah, 

NJ: Erlbaum.

12 Zimmerman, B. J. (1998). Developing self-fulfilling cycles of academic regulation: An analysis of 

exemplary instructional models. In D.H. Schunk 

& B.J. Zimmerman (Eds.). Self-regulated learning: From teaching to self-reflective practice. New 

York, NY: Guilford Press. 

Zimmerman, B.J. (2000). Attaining self-regulation: 

a social cognitive perspective. In M. Boekaerts, 

P.R. Pintrich & M. Zeidner (Eds.), Handbook of 

Self-Regulation. San Diego: Academic Press.

13 Pintrich, P. R. (1991). A manual for the use of the 

Motivated Strategies for Learning Questionnaire 

(ERIC No. ED338122). Ann Arbor, MI: National Center for Research to Improve Postsecondary 

Teaching and Learning. eric.ed.gov/?id=ED338122

14 Lan, W.Y., Bremer, R., Stevens, T., & Mullen, G. 

(2004, April). Self-regulated learning in the online 

environment. Paper presented at the annual meeting American Educational Research Association, 

San Diego, California.

15 Zimmerman, B.J., & Pons, M.M. (1986). Development of a structured interview for assessing student 

use of self-regulated learning strategies. American 

Educational Research Journal, 23(4), 614–628.

16 González-Torres, M.C., & Torrano, F. (2008). 

Methods and instruments for measuring self-regulated learning. In Handbook of Instructional Resources and their Applications in the Classroom 

(pp. 201–219). New York, NY: Nova Science.

17 Organisation for Economic Co-operation and Development. (2000). Literacy in the information 

age: Final report of the International Adult Literacy Survey. Paris: OECD. www.oecd.org

18 Ibid. Chi & Wylie (2014). Endnote 3-10.

19 Ibid. OECD (2000). Endnote 15-17.

20 Freed, M., Yarnall, L., Spaulding, A., & Gervasio, M. (2017). A mobile strategy for self-directed 

learning in the workplace. In Proceedings of the 

I/ITSEC. Arlington, VA: National Training and 

Simulation Association.

ENDNOTES FOR SECTION 4 

(ORGANIZATION)

Chapter 16 Endnotes

1 Januszewski, A. & Molenda, M. (2008). Educational technology: A definition with commentary. 

New York: Taylor & Francis Group.

2 Glover, J. and Ronning, R. (1987). Historical 

Foundations of Educational Psychology. New 

York, Plenum Press.

3 For example, see: 

Dick, W., & Carey, L. (1978). The systematic design of instruction (1st ed.). Chicago: Scott, Foresman and Company.

Ibid. Dick, Carey, & Carey (2001). Endnote 3-23.

4 Ibid. Glover and Ronning (1987). Endnote 16-2.

5 Gustafson, K. L., & Branch, R. M. (1997). Revisioning models of instructional development. Educational Technology Research and Development, 

45(3), 73–89.
404 | Modernizing Learning

6 Gagné, R.M., Wager, W.W., Golas, K.C., Keller, 

J.M. (2005). Principles of instructional design (5th

Ed.). Belmont, CA: Wadsworth/Thomson Learning.

7 See, for example:

Darling-Hammond, L. (2005). Teaching as a profession: Lessons in teacher preparation and professional development. Phi Delta Kappan, 87(3), 

237–240. 

Gage, N. L. (1989). The paradigm wars and their 

aftermath: A “historical” sketch of research on 

teaching since 1989. Educational Researcher, 

18(7), 4–10. 

Seidel, T., & Shavelson, R. J. (2007). Teaching effectiveness research in the past decade: The role 

of theory and research design in disentangling 

meta-analysis results. Review of Educational Research, 77(4), 454–499.

8 See, for example:

Angelo, T.A., & Cross, K.P. (1993). Classroom 

assessment techniques: A handbook for college 

teachers (2nd Ed.). San Francisco: Jossey-Bass, Inc.

Shulman, L.S. (2004). Visions of the possible: 

Models for campus support of the scholarship of 

teaching and learning. In Becker, W.E & Andrews, 

M.L. (Eds.), The Scholarship of Teaching and 

Learning in Higher Education: Contributions of 

Research Universities (9–24). Bloomington, IN: 

Indiana University Press. 

Sorcinelli, MD., Austin, A.E. Eddy, P.L. & Beach, 

A.L. (2006). Creating the future of faculty development. San Fransisco: Jossey-Bass, Inc.

9 Campbell, K., Schwier, R.A., & Kenny, R.F. 

(2009). The critical, relational practice of instructional design in higher education: An emerging 

model of change agency. Educational Technology 

Research and Development, 57(5), 645–663.

10 Rothwell, W.J. & Kazanas, H. C. (1998). Mastering 

the instructional design process: A systematic approach (2nd Ed.). San Francisco: Jossey-Bass/Pfeiffer.

11 McDonald, J. K. (2011). The creative spirit of design. TechTrends, 55(5), 53–57.

12 Cross, N. (1982). Designerly ways of knowing. 

Design studies, 3(4), 221–227. See page 224.

13 As quoted from Willcox, K.E., Sarma, S., & Lippel, P. (2016). Online education: A catalyst for 

higher education reforms. Cambridge: MIT. 

For the original, see:

Simon, H.A. (1967) “Job of a college president.” 

Educational Record 48(1), 68–78. Washington, 

D.C.: American Council on Education. 

14 Dede, C., Richards, J., & Saxberg, B. (2018). 

Learning engineering for online education: Theoretical contexts and design-based examples. New 

York: Routledge.

15 Ibid. Dede et al. (2018). Endnote 16-14.

16 Ibid. Dede et al. (2018). Endnote 16-14. See page 29.

17 For more information, see: www.opm.gov/policy-data-oversight/classification-qualifications/

general-schedule-qualification-standards/

18 Dede. C. (2018, October 19). The 60 year curriculum: Developing new educational models to serve 

the agile labor market. The EvoLLLution. evolllution.com (Included with permission).

Chapter 17 Endnotes

1 Gall, J. (1975). General Systemantics: An Essay on 

how Systems Work, and Especially how They Fail, 

Together with the Very First Annotated Compendium of Basic Systems Axioms: a Handbook and 

Ready Reference for Scientists, Engineers, Laboratory Workers, Administrators, Public Officials, 

Systems Analysts, Etc., Etc., Etc., and the General 

Public. General Systemantics Press. See page 71.

Chapter 18 Endnotes

1 Godin, S. (2016, March 6). Stop stealing dreams 

[Blog post]. Medium. medium.com

2 Bordia, P., Hunt, E., Paulsen, N., Tourish, D., & 

DiFonzo, N. (2004). Uncertainty during organizational change: Is it all about control? European 

Journal of Work and Organizational Psychology, 

13(3), 345–365.

3 See, for example:

Lane, I. F. (2007). Change in higher education: 

Understanding and responding to individual and 

organizational resistance. Journal of Veterinary 

Medical Education, 34(2), 85–92. 

Zell, D. (2003). Organizational change as a process 

of death, dying, and rebirth. The Journal of Applied 

Behavioral Science, 39(1), 73–96.

4 Clarke, J.S., Ellett, C.D., Bateman, J.M., & Rugutt, J.K. (1996). Faculty Receptivity/Resistance 

to Change, Personal and Organizational Efficacy, 

Decision Deprivation and Effectiveness in Research 1 Universities (ERIC No. ED402846). Paper presented at the Annual Meeting of the Association for the Study of Higher Education, Memphis, 

TN. eric.ed.gov/?id=ED402846
Endnotes | 405 

5 Riley, W. (1989). Understanding that resistance 

to change is inevitable [Monograph]. Managing 

Change in Higher Education, 5, 53–66.

6 Susskind, R., & Susskind, D. (2016, October 11). 

Technology will replace many doctors, lawyers, 

and other professionals. Harvard Business Review. 

hbr.org

7 Bordia, P., Hobman, E., Jones, E., Gallois, C., & 

Callan, V.J. (2004). Uncertainty during organizational change: Types, consequences, and management strategies. Journal of business and psychology, 18(4), 507–532.

8 Ibid. Lane (2007). Endnote 18-3.

9 Mulholland, B. (2017, July 14). 8 critical change 

management models to evolve and survive. Process.st. www.process.st

10 Sinek, S. (2011). Start with why how great leaders 

inspire everyone to take action. New York: Portfolio/Penguin.

11 Gawande, A., & Gawande, A. (2010). The checklist manifesto: How to get things right. New York: 

Henry Holt. 

See also: Shane Parrish interview of Dr. Atul Gawande on “The Learning Project with Shane Parrish” [Podcast]. (2018, October 2). 

12 Teller, A. (2016, April 20). Celebrating Failure Fuels Moonshots [Audio blog interview]. Retrieved 

October 8, 2018 from ecorner.stanford.edu/podcast/celebrating-failure-fuels-moonshots/

13 This truism is often attributed as an “African proverb,” which is most likely inaccurate and certainly 

overly unspecific. It’s been often quoted by various well-known speakers, including by Al Gore 

when accepting his accepting his Nobel Peace 

Prize; yet, it’s unclear where the saying originally 

derives from. Regardless, it’s still relevant, no matter the source! (jezebel.com/on-the-origin-of-certain-quotable-african-proverbs-1766664089)

Chapter 19 Endnotes

1 For example, see: Allen, I. E., & Seaman, J. (2016). 

Online report card: Tracking online education in 

the United States. Babson Survey Research Group.

2 Statista (2018, December). Total training expenditures in the United States from 2012 to 2018 

(in billion U.S. dollars). www.statista.com/statistics/788521/training-expenditures-united-states/

3 Epstein, E.A. (2010, April 11). Chew on this: Inside McDonald’s Hamburger University—the 

“Harvard of the fast food biz.” The Daily Mail.

www.dailymail.co.uk

4 McDonalds (n.d.). Hamburger University. corporate.mcdonalds.com

5 Starbucks (n.d.) Future leaders start here. www.

starbucks.com

6 Bureau of Labor Statistics (2017, August 24). Number of jobs, labor market experience, and earnings 

growth among Americans at 50: Results from a longitudinal survey (USDL-17-1158). www.bls.gov

7 Bureau of Labor Statistics (2018, September 20). 

Employee tenure summary (USDL-18-1500). 

www.bls.gov

8 Aldrich, C. (2003). Simulations and the future of 

learning. San Francisco: Wiley. See page 7.

9 Loughran, D.S. (2014). Why is veteran unemployment so high? Santa Monica, CA: RAND Corporation. www.rand.org

10 Snyder, T.D. (2018). Mobile Digest of Education 

Statistics, 2017 (NCES 2018-138). U.S. Department of Education. Washington, DC: National Center for Education Statistics. nces.ed.gov/

pubs2018/2018138.pdf

11 Ibid. Blackman et al. (2016). Endnote 6-10.

12 Zimmerman, B.J. & Dibenedetto, M.K. (2008). 

Mastery learning and assessment: Implications for 

students and teachers in an era of high-stakes testing. Psychology in the Schools, 45(3), 206–216.

13 Rosenberg, M. (2014, October 14). Marc my words: 

In learning and performance ecosystems, the 

whole is greater than the sum of the parts. Learning Solutions. www.learningsolutionsmag.com

14 U.S. OPM (n.d.). HR line of business: HC Business Reference Model. www.opm.gov

15 Ibid. National Academies (2018). Endnote 1-4.
406 | Modernizing Learning