ARL-TR-7954 ● MAR 2017

US Army Research Laboratory

Assessing Consequential Scenarios in a 

Complex Operational Environment Using 

Agent-Based Simulation

by Alex James, John Pierowicz, Mike Moskal,

Timothy Hanratty, Dan Tuttle, Bob Sensenig, and Bill Hedges

Approved for public release; distribution is unlimited.
NOTICES

Disclaimers

The findings in this report are not to be construed as an official Department of the 

Army position unless so designated by other authorized documents.

Citation of manufacturer’s or trade names does not constitute an official 

endorsement or approval of the use thereof.

Destroy this report when it is no longer needed. Do not return it to the originator.
ARL-TR-7954● MAR 2017

US Army Research Laboratory

Assessing Consequential Scenarios in a 

Complex Operational Environment Using 

Agent-Based Simulation

by Alex James, John Pierowicz, and Mike Moskal

Data Science and Information Fusion, CUBRC, Buffalo, NY

Timothy Hanratty

Computational and Information Sciences Directorate, ARL

Dan Tuttle, Bob Sensenig, and Bill Hedges

Requirements Determination Directorate, USAICOE, Fort Huachuca, AZ

Approved for public release; distribution is unlimited.

FOR OFFICIAL USE ONLY (delete if not FOUO)
ii

REPORT DOCUMENTATION PAGE Form Approved

OMB No. 0704-0188

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the 

data needed, and completing and reviewing the collection information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing the 

burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302. 

Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently valid 

OMB control number.

PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS.

1. REPORT DATE (DD-MM-YYYY)

March 2017

2. REPORT TYPE

Technical Report

3. DATES COVERED (From - To)

1 June 2015–30 September 2016

4. TITLE AND SUBTITLE

Assessing Consequential Scenarios in a Complex Operational Environment 

Using Agent-Based Simulation 

5a. CONTRACT NUMBER

W911NF-15-5-0043

5b. GRANT NUMBER

5c. PROGRAM ELEMENT NUMBER

6. AUTHOR(S)

Alex James, John Pierowicz, Mike Moskal, Timothy Hanratty, Dan Tuttle, 

Bob Sensenig, and Bill Hedges

5d. PROJECT NUMBER

5e. TASK NUMBER

5f. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

US Army Research Laboratory

ATTN: RDRL-CII-T

Aberdeen Proving Ground, MD 21005-5067

8. PERFORMING ORGANIZATION REPORT NUMBER

ARL-TR-7954

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR'S ACRONYM(S)

11. SPONSOR/MONITOR'S REPORT NUMBER(S)

12. DISTRIBUTION/AVAILABILITY STATEMENT

Approved for public release; distribution is unlimited.

13. SUPPLEMENTARY NOTES

14. ABSTRACT

In an operational environment characterized by interactive complexity, emergence, and near constant change, this report

examines whether agent-based or game-theoretic capability can improve the military decision-making process by providing 

additional rigor, speed, or flexibility during tactical US Army wargaming. The report considers methodologies to improve 

analysis of the human domain, identifies frameworks for modeling such a capability, and compares them against current Army 

wargaming practices. The report is composed of 5 sections: State of Wargaming, Agent-Based Modeling in Tactical 

Wargaming, Modeling Human–Computer Decision-Making, Existing Research and Applied Development of Agent-Based 

Modeling, and Plan for Research Development, Technology, and Training. The report concludes with a recommended 

technology and research path for the Army.

15. SUBJECT TERMS

agent-based modeling and simulation, complex operational environment, wargaming, military decision-making process, 

human domain

16. SECURITY CLASSIFICATION OF:

17. LIMITATION

OF ABSTRACT

UU

18. NUMBER

OF PAGES

220

19a. NAME OF RESPONSIBLE PERSON

Timothy P Hanratty

a. REPORT

Unclassified

b. ABSTRACT

Unclassified

c. THIS PAGE

Unclassified

19b. TELEPHONE NUMBER (Include area code)

410-278-3084

Standard Form 298 (Rev. 8/98)

Prescribed by ANSI Std. Z39.18
Approved for public release; distribution is unlimited.

iii

Contents

List of Figures viii

List of Tables xi

Acknowledgments xii

Executive Summary xiii

Study Composition xiv

Recommendation xv

1. Introduction 1

1.1 Operational Problem 2

1.2 Known 2

1.3 Knowable 2

1.4 Complex 2

1.5 Chaotic 3

1.6 Why an Agent-Based Capability, and Why Now 4

1.7 Study Overview 5

2. State of Wargaming 7

2.1 Army Wargaming 7

2.1.1 Personnel and Roles 9

2.1.2 Wargaming Process and Tools 11

2.2 Conceptual Notion of Wargaming 16

2.3 History of Army Wargaming 20

2.4 Historical Evolution of Wargaming 23

2.5 Historical Applications of Tactical Wargaming 25

2.6 Analytical Frameworks 28

2.6.1 PMESII-PT 29

2.6.2 ASCOPE 31

2.6.3 METT-TC 33
Approved for public release; distribution is unlimited.

iv

2.7 Challenges of Modern Wargaming 34

3. Agent-Based Modeling in Tactical Wargaming 37

3.1 Agent-Based Modeling 37

3.2 Identify Rigor 40

3.3 Model of the Human Domain 42

3.4 Social Cognitive Theory 43

3.4.1 Environment 44

3.4.2 Behavior 45

3.4.3 Agent State 45

3.5 Applicability of ABM to Wargaming 46

3.6 Identifying an Increase in Wargaming Rigor 49

3.7 Tactical Agent-Based Wargaming 53

3.7.1 Using an Agent-Based Capability during Tactical 

Wargaming 55

3.7.2 Appropriate Tactical Level for Agent-Based Wargaming 57

3.8 Summary 59

4. Modeling Human–Computer Decision-Making 60

4.1 Artificial Intelligence 60

4.1.1 Sensory–Motor System 61

4.1.2 Hierarchies of Perception, Representation, and Action 62

4.1.3 Predicting and Planning 63

4.1.4 Learning 63

4.1.5 Reasoning and Representation 64

4.1.6 Language 65

4.2 Decision-Making 66

4.2.1 Decision-Making Frameworks 68

4.2.2 Human and Artificial Intelligence Interaction 72

4.3 Computational Social Science 77

4.3.1 Emergence at the Social Behavioral Level 77

4.3.2 Emergence at the Social Aggregate Level 78

4.3.3 Emergence at the Institutional Level 78

4.3.4 Creating Computational Social-Science Models 79

4.3.5 Agent-Based Modeling 79
Approved for public release; distribution is unlimited.

v

4.3.6 Process for Implementing Agent-Based Models 81

4.3.7 ABM and Design Patterns 82

4.3.8 Types and Attributes of Agent-Based Model Design 

Patterns 84

4.3.9 Evaluating Agent-Based Models 86

4.3.10 Limitations and Challenges of ABM 88

5. Existing Research and Applied Development of ABM 90

5.1 ABM in Existing Defense Capabilities 90

5.1.1 Tools and Capabilities 91

5.1.2 Policon/Senturion 92

5.1.3 LODESTONE 93

5.1.4 Synthetic Environment for Analysis and Simulation (SEAS) 

/Reference World Information and Simulation Environment 

(RWISE) 93

5.1.5 Conflict Modeling, Planning, and Outcomes Experimentation 

Program (COMPOEX) 94

5.1.6 Joint Non-Kinetic Effects Model (JNEM)/Athena 95

5.1.7 Peace Support Operation Model (PSOM) 95

5.1.8 National Operational Environment Model (NOEM) 96

5.1.9 TRADOC’s Irregular Warfare Tactical Wargame (IW TWG) 97

5.1.10 IW TWG: Cultural Geography Model 97

5.1.11 IW TWG: Semi Automated Force (SAF) 98

5.1.12 IW TWG: Military Information Support Operations (MISO)

Planner 98

5.1.13 Complex Operational Environment Model (COEM) 99

5.1.14 COEM: Complex Military Mission Environment (CM2E) 99

5.1.15 Worldwide-Integrated Crisis Early Warning System (WICEWS) 99

5.1.16 Virtual Strategic Analysis and Forecasting Tool (V-SAFT) 100

5.1.17 Versatile Multiscale Strategist (vmStrat) 102

5.1.18 COA Analysis by Integration of Decision and Social Influence 

Modeling with Multi-Agent System (CADSIM) 102

5.2 Evaluating Agent-Based Technologies: Maturity Level and the Human 

Domain 103

5.2.1 Evaluation of Maturity Level 103

5.2.2 Human Domain Elements 104
Approved for public release; distribution is unlimited.

vi

5.3 Sampling Agent-Based Technologies That Resemble Human Domain 

Elements 107

5.3.1 Social Technologies 107

5.3.2 Cultural Technologies 109

5.3.3 Physical Technologies 110

5.3.4 Informational Technologies 111

5.3.5 Psychological Technologies 112

5.4 Sampling and Analysis of Agent-Based Research 114

5.5 Sampling of Foreign Agent-Based Research 117

5.5.1 Chinese Agent-Based Research 117

5.5.2 Russian Agent-Based Research 121

5.6 Summary 122

6. Plan for Research, Development, Technology, and Training 123

6.1 Recommendations 124

6.1.1 Research and Develop an Army-Centric Agent-Based 

Modeling Capability 124

6.1.2 Form a Stateside Repository 125

6.1.3 Implement an ABM Capability as a Multilayered Distributed 

Application 126

6.1.4 Provide Applicable Training to Ensure Appropriate and 

Knowledgeable Proficiency 126

6.2 Potential and Limitations 127

6.3 Risks 128

6.4 Comparable Technologies 129

6.5 Technology Path 129

6.5.1 Research Recommendations 131

6.5.2 Army-Centric Agent-Based Modeling Capability: Near 

Term 131

6.5.3 Army-Centric ABM Capability: Medium Term 132

6.5.4 Army-Centric ABM Capability: Long Term 133

6.5.5 Stateside Repository: Near Term 134

6.5.6 Stateside Repository: Medium Term 135

6.5.7 Stateside Repository: Long Term 136

6.5.8 Distributed Application: Near Term 137

6.5.9 Distributed Application: Medium Term 137
Approved for public release; distribution is unlimited.

vii

6.5.10 Distributed Application: Long Term 138

6.5.11 Training and Proficiency: Near Term 139

6.5.12 Training and Proficiency: Medium Term 139

6.5.13 Training and Proficiency: Long Term 139

6.6 Capability Recommendations 140

6.6.1 Army-Centric ABM Capability 140

6.6.2 Stateside Repository 142

6.6.3 Distributed Application 143

6.6.4 Training and Proficiency 144

6.7 Training and Resource Implications 144

6.7.1 Curriculum 145

6.7.2 Theory 146

6.7.3 Practice 147

6.7.4 Proficiency 148

6.8 Summary 148

7. Conclusion 149

8. References 150

Appendix A. Survey of Existing Agent-Based Modeling Frameworks 181

Appendix B. Sampling of Computational Social Science Society of the 

Americas (CSSSA) Conference Papers 193

List of Symbols, Abbreviations, and Acronyms 198

Distribution List 202
Approved for public release; distribution is unlimited.

viii

List of Figures

Fig. 1 An illustration of the Army’s MDMP identifies and associates key 

inputs and outputs with each step of the process. ..................................8

Fig. 2 Table of Army warfighting functions and associated example 

considerations therewithal. Commanders issue guidance on items 

appropriate and applicable to the particular mission. ............................9

Fig. 3 An expanded illustration of the MDMP wargaming step, with 

identified inputs, outputs, and the process involved............................12

Fig. 4 Visual representations of each wargaming method are as follows: belt, 

avenue-in-depth, and box. The upper figures illustrate how they are

applied in spatial areas of operations whereas the lower figures 

illustrate how they are applied to distinct lines of effort (illustrations 

adapted from illustrations in FM 5-0 and FM 101-5).........................13

Fig. 5 A template of a synchronization matrix; the second and third rows 

identify most likely enemy and civilian actions. Rows annotated as 

“Warfighting Functions” focus on specific tactical warfighting 

functions, with additional rows, as appropriate, added for subordinate 

commands (illustration adapted from FM 5-0)....................................14

Fig. 6 A template of a sketch note that identifies critical attributes of 

operations associated with decision points. Locations are marked on 

the map, and wargaming participants identify expected actions, 

reactions, and counteractions pertaining to each sketch note. 

(illustration adapted from FM 5-0 and ATTP 5-0.1). ..........................14

Fig. 7 Illustration adapted representation of Table 9-5, Effective Wargaming 

Results..................................................................................................15

Fig. 8 An illustration of the decision-making process using the maximin 

process. In this example, outcomes of each course of action 

combination are identified. The selection process identifies the worstcase outcomes for each COA; in this case, from the perspective of 

Commander A. From those outcomes, Commander A selects the bestcase scenario. .......................................................................................19

Fig. 9 An illustration of the decision-making process using the minimax 

process. In this example, outcomes of each course of action 

combination are identified. The selection process identifies the bestcase outcomes for each COA; in this case, from the perspective of 

Commander A. From those outcomes, Commander A selects the 

worst-case scenario. .............................................................................20

Fig. 10 The original specification of the MDMP, first documented in FM 101-

5, 1932 (figure adapted from Table 1: Format of the Estimate). .........21

Fig. 11 A crosswalk matrix between PMESII and ASCOPE variables. The 

figure illustrates examples of each variable combination....................33
Approved for public release; distribution is unlimited.

ix

Fig. 12 This figure illustrates Social Cognitive Theory in its triadic reciprocal 

state. The Human Domain concept is compared to and overlaps with 

Social Cognitive Theory. Both theories describe the same concepts, 

but Social Cognitive Theory offers an agentic form of interactive 

complexity............................................................................................44

Fig. 13 This figure illustrates Social Cognitive Theory and the overlap 

between Risk Management and Critical Factor Analysis 

methodologies. Similar to Fig. 12, each methodology overlaps, but 

Social Cognitive Theory offers a more comprehensive expression of 

the interactive complexity....................................................................48

Fig. 14 Nilsson’s categorizations of the field of artificial intelligence. The 

figure also identifies ABM as falling within the scope of Predicting 

and Planning.........................................................................................61

Fig. 15 Wang and Ruhe’s cognitive framework of decision and rational 

decision-making strategies. A decision tree illustrates paths of selected 

strategies, which depend on the individual’s perspective of the 

situation................................................................................................67

Fig. 16 A notional JDL Data Fusion Model is composed of 6 levels, which 

represent abstract notions of data representation and interpretation. 

Levels 4–6 have recently identified interaction with external entities, 

dealing with issues of resource management, human interaction, and 

multiagent coordination. ......................................................................69

Fig. 17 An illustration of the functions that occur at each level and node within 

each JDL Data Fusion level. ................................................................70

Fig. 18 Boyd’s OODA loop used to illustrate the way one can create 

opportunities to act in ways that opponents will see as irregular and 

disorienting. .........................................................................................71

Fig. 19 Endsley’s model of situational awareness for dynamic decisionmaking. While similar to Boyd’s OODA loop, it explicitly identifies 

projections of future status and identified individual factors 

influencing the implementation of actions...........................................72

Fig. 20 Illustration of each of the 3 proposed human–machine interaction 

concepts from a Data Fusion perspective. In each variation, the point 

at which the human influences the interaction is highlighted..............76

Fig. 21 An adapted representation of Oluyomi’s proposed design and 

modeling process .................................................................................83

Fig. 22 A listing of the attributes considered while creating and agent-based 

model based on Oluyomi’s proposed design and modeling process ...85

Fig. 23 Oluyomi’s list of identified design patterns categorized by their type of 

interaction and stage within the agent-based, model-creation-process

model....................................................................................................86

Fig. 24 A quad chart illustrating 4 quadrants of tactical analysis. This study’s 

emphasis on applications of ABM to tactical wargaming thus limits its 

scope to the bottom-left quadrant, “Non-Kinetic/Social Science”. .....91
Approved for public release; distribution is unlimited.

x

Fig. 25 An estimated timeline of tools embodying ABM capabilities. Dates in 

the illustration are approximate. ..........................................................92

Fig. 26 The V-SAFT architecture and implementation phases. The V-SAFT 

capability is used within W-ICEWS to create and configure countryspecific models...................................................................................101

Fig. 27 Axtell and Epstein’s Empirical Relevance Framework describes the 

maturity of agent-based models, ranging from realistic caricatures to 

quantitatively characterized phenomena at the microlevel. This 

illustration also incorporates docking, system engineering, 

policy/regulation design, and estimated data and time required........104

Fig. 28 A heat map identifying whether papers submitted to CSSSA 

conferences (2012, 2013, and 2015) exhibit characteristics identified 

by the Human Domain. Dark-gray cells indicate research that the 

Army could potentially use to develop Army-specific tactical ABM 

capabilities. ........................................................................................115

Fig. 29 Seven cities throughout China, identified numerically on this map, 

represent the locations of published agent-based research. ...............118

Fig. 30 An enumeration of military-relevant or other topical agent-based 

research produced by Chinese organizations. Locations, organizations, 

and names of papers are identified.....................................................118

Fig. 31 Three cities throughout Russia, identified numerically on this map, 

represent the locations of published agent-based research. ...............121

Fig. 32 An enumeration of military-relevant or other topical agent-based 

research produced by Russian organizations. Locations, organizations, 

and names of papers are identified.....................................................121

Fig. 33 Timeline of recommend capability developments identified in Table 

5..........................................................................................................131
Approved for public release; distribution is unlimited.

xi

List of Tables

Table 1 The result of using the 8 metrics of rigorous analysis to identify an 

estimated level of rigor in wargaming analysis. The left column is an 

estimation of wargaming rigor at the tactical level, as is currently 

done. The right-hand column introduces a hypothetical agent-based 

capability, which in this report is purposely under-specified; however, 

estimated level of rigor is used to illustrate the plausibility of improved 

rigor and the areas where such a capability would be most useful......52

Table 2 Endsley and Kaber’s levels of automation, expanding upon Endsley’s 

original 5 levels of automation ............................................................75

Table 3 Endsley and Kaber’s taxonomy for human and algorithmic systems 

operating in dynamic, multitask scenarios...........................................75

Table 4 Axtell’s comparison of traditional modeling methods, techniques, and 

processes with newer, computational social-science techniques.........81

Table 5 Table of recommended technology capabilities the Army should 

develop for a tactical agent-based capability in terms of time 

(i.e., near-, medium-, and long-term) and type ..................................130
Approved for public release; distribution is unlimited.

xii

Acknowledgments 

The authors would like to acknowledge and thank the following individuals for 

their helpful contributions, commentary, discussions, and insights:

Aaron Frank, Brian Kettler, Bruce West, Craig Stansifer, Dale Walsh, Dave Hall, 

Ed Waltz, Eric Heilman, Jim Llinas, John Coles, Kevin Barry, Kevin Corby, Kira 

Hutchinson, Matt Koehler, Moises Sudit, Phil Hanselman, and Stephen Guerin.

Research was sponsored by the US Army Intelligence Center of Excellence and the 

US Army Research Laboratory (ARL) and was accomplished under Cooperative 

Agreement Number W911NF-15-5-0043. The views and conclusions contained in 

this document are those of the authors and should not be interpreted as representing 

the official policies, either expressed or implied, of the US Army Intelligence 

Center of Excellence, ARL, or the US Government. The US Government is 

authorized to reproduce and distribute reprints for Government purposes 

notwithstanding any copyright notation herein.
Approved for public release; distribution is unlimited.

xiii

Executive Summary 

Operational art—the creative thinking used to design strategies, campaigns, and 

major operations and to organize and employ military force—allows commanders 

to understand challenges facing them and to conceptualize approaches for 

achieving the nation’s strategic objectives. This study asks the question, Could the 

art of mission planning and campaign design, where critical and creative thinking 

are used to understand the fundamental nature of a complex military problem, 

benefit from new opportunities provided by mathematics and science? That is, 

might recent advancements in computational social science, agent-based modeling 

(ABM) and simulations, and complex adaptive systems offer commanders and their 

planning staffs new opportunities to do the following:

1) Rigorously explore and make sense of interactively complex operational 

environments.

2) Unveil the nature of ill-structured military problems.

3) Move beyond logical limitations of linear, reductionist methods used in the 

intelligence preparation of the battlefield (IPB).

Future warfare will likely remain characterized by uncertainty. Politicians will send 

US forces to war while having varying depths of understanding of the specific 

complex factors contributing to violence, fear, and depravity. Commanders, their 

planning staffs, and intelligence analysts will encounter military problems that exist 

in 4 different knowledge states, simultaneously: known, knowable, complex, and 

chaotic.

Known: The behavior of a uniformed enemy is somewhat predictable because it is 

based on doctrine, movement rates, and the limitations of weapon systems and on 

the interaction of system components that are not individually purposeful. This state 

is amenable to the present-day IPB.

Knowable: Cause and effect can be discerned over time using wide area 

surveillance, pattern of life techniques, big data collection and mining, or simply 

by talking to the right people at the right time.

Complex: Interactive military problems run the risk of oversimplification in 

Washington, DC and at higher levels of command. Interactive emergence creates 

unpredictable situations, and solutions to murky problems may have totally 

unexpected consequences.
Approved for public release; distribution is unlimited.

xiv

Chaotic: Systems suggest that greater complexity induces less-predictable 

emergent behavior. There is no presumption of rational choice, of order, or of a 

unitary and stable threat.

This study evaluates the capabilities of agent-based analytics and game theory with 

the goal of identifying potential investment areas for US Army Science and 

Technology (S&T) targeted for research and development relative to automated and 

semi-automated analysis. For its purposes, the authors examined ABM as a means 

to improve wargaming rigor and minimize decision-making risk in the Human 

Domain, specifically through its application to tactical battalion-, brigade-, and 

division-level wargaming.

Study Composition

This report is composed of 5 sections: 1) State of Wargaming, 2) Agent-Based 

Modeling in Tactical Wargaming, 3) Modeling Human–Computer DecisionMaking, 4) Existing Research and Applied Development of Agent-Based 

Modeling, and 5) Plan for Research Development, Technology, and Training.

State of Wargaming begins with a summarization of Army wargaming, followed 

by an introduction to the history of wargaming applications. This section identifies 

analytical frameworks the Army uses to communicate strategic and operational 

situational understandings to tactical commanders and identifies challenges to 

current Army wargaming practices.

Agent-Based Modeling in Tactical Wargaming introduces the concept of 

ABM, especially with respect to the Human Domain. It explores the potential 

application of ABM to tactical wargaming, examining how it can improve the rigor 

of the current wargaming process.

Model Human–Computer Decision-Making begins with a survey of artificial 

intelligence and explores how ABM relates to broader artificial intelligence 

capabilities. This chapter discusses several decision-making frameworks, working 

from the assumption that decision-making is a traditional human-cognitive process 

and artificial intelligence is a capability that can assist the human. It then discusses 

agent-based design principles and patterns, in which each agent embodies their own 

decision-making capabilities.

Existing Research and Applied Development of Agent-Based Modeling

reviews current state-of-the-art ABM capabilities, exploring tools that exhibit 

agent-based capabilities, identifying a framework for determining the maturity of 

such research, and reviewing how current agent-based research could assist the 

Army in creating a future tactical wargaming capability.
Approved for public release; distribution is unlimited.

xv

Plan for Research Development, Technology, and Training presents the study 

recommendations, beginning with a holistic Army perspective, then addressing 

specific technologies and capabilities and identifying training and education 

recommendations, specific to the implementation of agent-based capabilities.

Recommendation

The study makes 4 recommendations: 1) begin research and development of an 

Army-centric ABM capability, 2) form a stateside repository, 3) implement an 

ABM capability as a multilayered, distributed application, and 4) provide 

applicable training to ensure appropriate and knowledgeable proficiency. The 

authors recommend that a combination of each of the aforementioned is essential 

to the success of the Army mission. Furthermore, the risk of not having a tactical 

ABM wargaming capability in the future would be greater than the status quo.

The first recommendation, that the Army S&T community begins researching and 

developing Army-centric agent-based models, comprises 4 subelements: 1) 

research and development of resources that model tactical Human Domain 

attributes, 2) research and development of automated reasoning processes that 

support rigorous analysis, 3) research and development of a cognitive-social 

framework, and 4) evaluation of resources and models in terms of a model maturity 

framework. The study identifies short-, medium-, and long-term research goals for 

each of these elements.

As stated earlier, the second recommendation is to form and maintain a stateside 

repository of worldwide applicable models, agents, and resources; that is, a 

so-called Army Wargaming Intelligence Center. The authors base this 

recommendation on “Request for Support” functionality that the Joint Improvised 

Threat Defeat Agency’s Counter-Improvised Explosive Device

Operations/Intelligence Center supported with its “Attack the Network” operations. 

The recommendation particularly intends the development of a system and set of 

processes and personnel who support timely (e.g., 6 h, 24 h, 14 days, etc.) requests 

for information while interacting with liaisons representing a whole-of-government 

to coordinate wargamed courses of action.

To implement the third recommendation, the Army agent-based capability as a 

multilayered and distributed application, the authors recommend that the Army 

require the user interface implementation to be a web-based application, compatible 

with the Distributed Common Ground System–Army Program of Record. 

However, in light of possible network disconnections at the tactical level, the 

web-based application would need to run as a standalone, hybrid web application 

on a client machine.
Approved for public release; distribution is unlimited.

xvi

The fourth recommendation involves incorporating applied topics of culture, tribes, 

complex adaptive systems, game theory, and applied statistics into specialized 

education and training curriculums, such as those provided at the Command and 

General Staff College and Warrant Officer training at the Combined Arms Center.

Some risk is associated with a tactical Army agent-based capability. Getting ahead 

of the science is the foremost of the concerns; it is vital to avoid letting the 

engineering process of developing an agent-based capability outpace the science. 

If engineering outpaces science, there can be the illusion of contextual 

oversimplification that results in users not trusting the capabilities that agent-based 

wargaming otherwise contributes. Another concern is whether validating the 

models and associated data availability is sufficient to ensure proper model validity. 

Finally, the study recommends the Army ruthlessly minimize the amount of time 

tactical users need to interact with and provide input to the produced tools and 

capabilities.

ABM is not an optimal or precision tool; rather, ABM is a robust capability that 

communicates a range of possibilities, including associated warning indicators with 

estimated possibilities of occurrence. The emphasis of the capability is first on 

understanding and modeling the situation within the larger context of human 

interactions and then working within the available Human Domain population.

Researching and developing an Army-centric ABM capability is a nontrivial task. 

The Army will need to devote significant person-hours to perform leading research 

and development, adequate verification and validation, subsequent model creation, 

and resource revisions. Similarly, a nontrivial amount of money, appropriated 

throughout the range of Basic Research–Operational System Development money, 

and people, serving in both stateside and training positions, will be required. 

However, little or no Army doctrine change is likely to be necessary.
Approved for public release; distribution is unlimited.

1

1. Introduction

Operational art, the creative thinking used to design strategies, campaigns, and 

major operations and organize and employ military force, allows commanders to 

understand the challenges facing them and to conceptualize approaches for 

achieving the nation’s strategic objectives. The underlying thought process helps 

commanders and their staffs to lessen the ambiguity and uncertainty of a complex 

operational environment, understand the military problem facing them, and 

visualize how best to effectively employ military capabilities to accomplish their 

mission (Joint Chiefs of Staff 2011, p. 1–5).

This report asks the question, Could the art of mission planning and campaign 

design, where critical and creative thinking are used to understand the fundamental 

nature of a complex military problem, benefit from new opportunities provided by 

mathematics and science? Might recent advancements in computational social 

science, agent-based modeling (ABM) and simulations, and complex adaptive 

systems offer Commanders and their planning staffs new opportunities to do the 

following:

• Rigorously explore and make sense of interactively complex operational 

environments.

• Unveil the nature of ill-structured military problems.

• Move beyond logical limitations of the linear, reductionist methods used in 

the intelligence preparation of the battlefield (IPB).

In other words, in an operational environment characterized by interactive 

complexity, emergence, and near-constant change, in institutions where there is no 

such person as a “subject matter expert” (SME) able to wrap their mind around 

hundreds of operational variables (i.e., political, military, economic, social, 

information, infrastructure, physical environment, and time [PMESII-PT]),

subsystems, and their interrelationships, can the Science and Technology (S&T) 

community deliver an agent-based or game-theoretic capability that helps 

commanders and their planning staffs to more rigorously wargame and explore 

relevant “what if” scenarios before selecting friendly courses of actions? Could 

technology developers design an agent-based capability relevant to staffing and 

other “situational understanding” enablers resident within a brigade combat team 

(BCT)? Alternatively, would such a capability need to reside at a higher level of 

operations?
Approved for public release; distribution is unlimited.

2

Thus, the central purpose of this report is to explore the relevance of agent-based 

models and research, and related game-theoretic concepts, to the improvement of 

tactical wargaming rigor (Sensenig 2015a, 2015b, 2015c).

1.1 Operational Problem

Future warfare will likely remain characterized by uncertainty. Politicians will send 

US forces to war while having varying depths of understanding of the specific 

complex factors contributing to violence, fear, and depravity. Commanders, their 

planning staffs, and intelligence analysts will encounter military problems that exist 

in 4 different knowledge states, simultaneously: known, knowable, complex, and 

chaotic.

1.2 Known

Behavior of a uniformed enemy is somewhat predictable because it is based on 

doctrine, movement rates, and the limitations of weapon systems and on the 

interaction of system components that are not individually purposeful. Observation 

of a single system component can lead to reasonable conclusions about the behavior 

of the larger system. This knowledge state is amenable to the present-day IPB.

1.3 Knowable

Before a conflict, US forces appreciate the need for a complete understanding of 

adversary behavior and the local population. Nevertheless, cause-and-effect can be 

discerned over time using wide area surveillance, pattern of life techniques, big data 

collection and mining, or simply by talking to the right people at the right time. 

There is general agreement among decision-makers from the White House to the 

BCT about the nature of the military problem, goals, and objectives.

1.4 Complex

Interactively complex military problems run the risk of oversimplification in 

Washington, DC and at higher levels of command. Behavior of various individuals, 

groups, and networks is not directly observable, not all interactions are understood, 

emergence creates unpredictable situations, and solutions to murky problems may 

have totally unexpected consequences. Cause-and-effect becomes coherent only in 

retrospect and does not necessarily repeat. Complex military problems make it 

difficult for tactical Commanders to link missions to strategic goals.
Approved for public release; distribution is unlimited.

3

1.5 Chaotic

Chaos theory suggests that greater system complexity induces less predictable 

emergent behavior. There is no presumption of rational choice, of order, or of a 

unitary and stable threat. Some experts have described the chaotic knowledge state 

as the domain of the inconceivable—the black swan.

The impetus behind this study is more than theoretical. Summing up a consensus 

of lessons-learned literature over the past 15 years, “Military professionals describe 

the volatile mix of factors (e.g., mission, enemy, terrain, troops available, time, and 

civilian considerations [METT-TC]; area, structures, capabilities, organizations, 

people, and events [ASCOPE]; sewer, water, electricity, academic, trash – medical, 

safety and other [SWEAT-MSO]; and PMESII-PT) as being ambiguous, complex, 

uncertain, and ill-structured. When trouble appears, there is no consensus about 

what the fundamental problems are, how to solve them, what the desired ‘end state’ 

should be, and whether an ‘end state’ is achievable or not” (Perez 2011).

In a 2012 analysis titled “COIN [counterinsurgency operations] is Dead – Long 

Live Transformation,” the authors described the difficulty of understanding the 

complexity of actors in Helmand Province, Afghanistan:

Even after extensive British and coalition efforts to improve the collection, fusion, 

and assessment of a wide range of intelligence and information sources, the 

linkages between actors at the strategic and tactical level, between the Taliban, al 

Qaeda, local communities and national governments such as Pakistan have proven 

hard to identify, and their impact on local outcomes in Helmand have proven 

difficult to influence. Even in late 2011, the former International Security 

Assistance Force commander General Stanley McChrystal noted a frighteningly 

simplistic view of the country remained and was crippling the NATO war effort 

(Ford et al. 2012).

Closer to home, MG Michael Flynn’s “Fixing Intel: A Blueprint for Making 

Intelligence Relevant in Afghanistan” (2010) asserted:

Eight years into the war in Afghanistan, the US intelligence community is only 

marginally relevant to the overall strategy. Having focused the overwhelming 

majority of its collection efforts and analytical brainpower on insurgent groups, 

the vast intelligence apparatus is unable to answer fundamental questions about the 

environment in which US, and allied forces operate and the people they seek to 

persuade. Ignorant of local economics and landowners, hazy about who the 

powerbrokers are and how they might be influenced, incurious about the 

correlations between various development projects and the levels of cooperation 

among villagers, and disengaged from people in the best position to find answers 

– whether aid workers or Afghan soldiers – US intelligence officers and analysts 

can do little but shrug in response to high level decision-makers seeking the 

knowledge, analysis, and information they need to wage a successful 

counterinsurgency.
Approved for public release; distribution is unlimited.

4

Finally, a RAND National Defense Research Institute analysis from 2014, 

“Modeling, Simulation, and Operations Analysis in Afghanistan and Iraq:

Operational Vignettes, Lessons Learned, and a Survey of Selected Efforts”, cites a 

relevant US Army Training and Doctrine Command (TRADOC) Analysis Center 

Irregular Warfare (IW) Working Group Report (Connable et al. 2014). In 2014, 

following 13 years of Army experience in Afghanistan and Iraq, researchers from 

the TRADOC Analysis Center examined methods, modeling, and analysis in 

support of IW. The study had 3 goals:

1) Determine the analytic community’s, not just the intelligence community’s, 

ability to support the commander’s decisions during irregular warfare.

2) Identify gaps in Department of Defense IW analysis and analytics.

3) Recommend solutions to mitigate the gaps.

The working group’s key findings are directly pertinent to this report: The group 

concluded that the highest risk gaps occur in protracted campaigns, in decision

analytics associated with the civilian populations, understanding interactions 

between actors, in psychological operations, and in civil–military operations. The 

study identified 56 analytic capabilities used to support IW decisions, and identified 

35 major capability gaps in IW decision analytics. Of those 35 gaps, 20 required 

social-science expertise.

This 2014 study told us something that most of us having years of experience in 

Iraq and Afghanistan knew all too well: The Army lacks decision-support tools 

designed to help commanders and planning staffs understand interactively complex 

situations involving hundreds of PMESII-PT subsystems and their impact on 

mission variables.

1.6 Why an Agent-Based Capability, and Why Now

Agent-based capabilities are integrated with emerging computational social 

sciences and practiced Army wargaming techniques to reveal emergent properties 

and behavior of militarily relevant individuals, groups, and networks in any 

operational environment marked by interactive complexity. 

The behavior of individual parts results in interactive complexity. The greater the 

freedom of action of each individual part and the more linkages among the 

components, the greater is the system’s interactive complexity (Department of the 

Army [DA] TRADOC 2008).

Agent-based capabilities begin with software agents, which represent battlefield 

entities that exhibit feelings and emotions (psychology), display behaviors 
Approved for public release; distribution is unlimited.

5

(sociology), have goals and intentions, and either act or do nothing. The research 

and engineering challenge is to design and develop an agent-based capability adept 

at meeting the demands of a brigade planning staff. The authors recommend the 

brigade, or the BCT, because the brigade is the lowest level of command that can 

operate independently as part of a joint task force (JTF).

The BCT’s unique capabilities—not found at battalion and below—allow strategic 

and operational military planners to employ the BCT in a variety of conditions, 

circumstances, and influences. The BCT is a modular organization that provides 

the land component or JTF commander with close-combat capabilities across the 

full spectrum of conflict. BCTs are the Army’s tactical combat power-building 

blocks for maneuver and are the smallest combined arms units that can be 

committed independently. The challenge facing the S&T community is to design 

an agent-based capability of sufficient complexity and granularity, sufficient for 

surprising and useful results but not so complex that it becomes confusing, 

generates “noise in the analysis”, or takes too much time, given time limitations 

that exist as a BCT staff conducts deliberate planning in preparation for war. 

The recent developments of cheap computational power and software capable of 

representing agents with reactive and adaptive behavior provide engineers, 

scientists, and capability developers with a virtual laboratory. Legacy campaign 

simulations are a misfit of algorithmic, homogenous statistical models that cannot 

accurately represent interactively complex and diverse battlefield entities. 

Computational social science is now mature enough to revolutionize tactical 

Army-specific agent-based capabilities.

1.7 Study Overview

This report is composed of 5 sections: State of Wargaming, Agent-Based Modeling 

in Tactical Wargaming, Modeling Human–Computer Decision-Making, Existing 

Research and Applied Development of Agent-Based Modeling, and Plan for 

Research Development, Technology, and Training.

Section 2, State of Wargaming, begins with a summary of Army wargaming, 

followed by an introduction to the history of wargaming application. This section 

identifies analytical frameworks the Army uses to communicate strategic and 

operational situational understandings to tactical commanders and identifies 

challenges to current Army wargaming practices.

Section 3, Agent-Based Modeling in Tactical Wargaming, introduces the 

concept of ABM, especially with respect to the Human Domain. It explores the 
Approved for public release; distribution is unlimited.

6

application of ABM to tactical wargaming, examining how it can improve the rigor 

of the current wargaming process.

Section 4, Modeling Human–Computer Decision-Making, begins with a survey 

of artificial intelligence and explores how ABM relates to broader artificial 

intelligence capabilities. This section discusses several decision-making 

frameworks from the perspective that decision-making is a traditional 

human-cognitive process and artificial intelligence is a capability that can assist the 

human. It then discusses agent-based design principles and patterns, in which each 

agent embodies their own decision-making capabilities.

Section 5, Existing Research and Applied Development of Agent-Based 

Modeling, reviews current state-of-the-art ABM capabilities, exploring tools that 

exhibit agent-based capabilities, identifying a framework for determining the 

maturity of such research, and reviewing how current agent-based research could 

assist the Army in creating a future tactical wargaming capability.

Section 6, Plan for Research, Development, Technology, and Training, presents 

the authors’ recommendations, beginning with a holistic Army perspective, then 

addressing specific technologies and capabilities and identifying training and 

education recommendations, specific to the implementation of agent-based 

capabilities.
Approved for public release; distribution is unlimited.

7

2. State of Wargaming

Wargaming is a process of thinking of and visualizing events that could occur 

during a possible course of action. Initially intended as an asymmetric battlefield 

planning capability, wargaming has matured into a formalized process over the past 

200 years. In recent history, wargaming has become a standardized part of the Army 

military decision-making process (MDMP), helping staff to consider and evaluate 

actions and corresponding counteractions at strategic, operational, and tactical 

levels. 

In this section, the report identifies and investigates what wargaming is, 

specifically, as the Army defines and applies it at the tactical level. The section 

begins with a review of the theoretical and conceptual notions of wargaming and 

connects them to the Army’s wargaming practices. The study summarizes the 

evolution of Army wargaming, and other countries’ application of wargaming, to 

provide the reader with context. Then, it reviews current analytical frameworks the 

Army uses to express operational requirements and situations. These frameworks 

illustrate the breadth of variables the Army considers during tactical analysis of 

courses of actions, or wargaming. Often, these frameworks are used to 

communicate inputs and outputs to and from the decision-making process. Finally, 

the report identifies challenges with current tactical Army wargaming.

2.1 Army Wargaming

The Army MDMP is an iterative planning process that integrates the activities of 

the commander, staff, subordinate headquarters, and other partners. An overview 

of the 7-step MDMP process is outlined in Fig. 1. Its objective is to allow these 

parties to understand the current situation and mission, develop and compare 

courses of action, decide on a course of action that would best accomplish the 

mission, and produce an operation plan or order (Headquarters [HQ] DA, FM 5-0 

2010). According to doctrine, commanders use the full MDMP process if they have 

enough planning time and the staff to support an examination of 2 or more course 

of actions (COAs) and fully develop a synchronized plan or order (HQ–DA ATTP 

5-0.1 2011). In a tactical context, developed COAs include both the most dangerous 

and most likely enemy COAs.
Approved for public release; distribution is unlimited.

8

Fig. 1 An illustration of the Army’s MDMP identifies and associates key inputs and outputs 

with each step of the process (HQ–DA FM 5-0 2010, p. B-3; ADRP 5-0 2012).

Wargaming is the fourth step of MDMP and, as discussed earlier, understood as the 

systematic analysis of possible courses of action. A COA is a plan to accomplish a 

military objective, using friendly strengths and dispositions, known enemy assets, 

and possible resultant friendly and enemy courses of action within the context of 

an area of operations (HQ–DA ATTP 5-0.1 2011, p. 4–22; ADP 3-0 2011; ADRP 

2-0 2012). Once stakeholders can visualize a system and its logic, they may gain 

insight into anticipating future behavior, identifying second and third order effects, 

accurately conceptualizing risk, and potentially influencing outcomes (ADRP 6-0 

2014). However, wargaming does not guarantee selection of the best scenario; it is 

only an evaluation of identified COAs.

Wargaming tests, identifies, and adjusts identified COA deficiencies based on 

identification of unexpected events, tasks, requirements, or problems. During the 

wargame, staff transforms each COA into a detailed plan, while at the same time 

noting its strengths and weaknesses. Staff evaluates all forms of operations, 

including offensive, defensive, stability, and civil support. Within each COA, staff 

evaluates each critical event in terms of the action, reaction, and counteraction 

methods of interaction. The result is a refined set of actions, each with their 

respective synchronization matrixes. Completed decision-support templates and 

synchronization matrices portray both key decisions and potential actions, which 

are likely to arise during execution.
Approved for public release; distribution is unlimited.

9

The wargaming of each COA allows staff to synchronize tactical tasks across the 

warfighting functions. For example, staff could maximize combat power while 

protecting friendly forces, anticipate operational events, determine conditions and 

resources for success, and identify the most flexible COA (HQ–DA FM 101-5 

1997, p. 5–16). Figure 2 enumerates a nonexhaustive list of considerations 

illustrating the breadth of consideration involved in evaluating COAs.

Fig. 2 Table of Army warfighting functions and associated example considerations 

therewithal. Commanders issue guidance on items appropriate and applicable to the 

particular mission (HQ–DA FM 5-0 2010, p. D-1–D-4).

Participants involved in wargaming should remain objective and not allow external 

pressure to influence decisions, particularly while evaluating a COA they 

personally developed. Doctrine states that staff should record advantages and 

disadvantages as they become apparent and avoid premature conclusions and 

comparisons between COAs before wargaming is complete. Staff also continually 

assesses the feasibility, acceptability, and suitability of each COA, where feasibility 

is the actual practicality of executing the COA, suitability is the likelihood that the 

situation will result in the desired effect, and acceptability is the evaluation of 

whether the results are worth the cost.

2.1.1 Personnel and Roles

Army doctrine identifies a number of staff members who have roles to play during 

wargaming. Each of the warfighting functions has assigned staffs who provide 

relevant input (HQ–DA FMI 5-0.1 2008; FM 6-0 2016, p. 9–15). Identified staff 

include the Executive Officer, Assistant Chief of Staff, G-1 through G-9 (as 

appropriate, S-1 through S-9), Chief of Fires, Chief of Protection, Surgeon, Red 
Approved for public release; distribution is unlimited.

10

Team Officer, Staff Judge Advocate, Operations Research and Systems Analysis 

(ORSA) Officer, and Safety Officer. However, many of these defined positions do 

not exist as wargaming is used at increasingly tactical levels (e.g., division, brigade, 

and battalion units). As such, the breadth of participation, analysis, and contribution 

expected within doctrine is not realized.

The following are summarized descriptions of specific roles, which Army doctrine 

identifies and highlights:

• The Executive Officer (XO) coordinates actions of staff during the 

wargame, ensuring that staff stays on schedule. In time-constrained 

situations, the XO ensures the most decisive operation is wargamed (HQ–

DA ATTP 5-0.1 2011, p. 4–24).

• The Assistant Chief of Staff and G-2 (S-2, intelligence) are responsible for 

the “intelligence” warfighting function. They role-play the enemy 

commander and capture the results of enemy actions and evaluate strengths 

and weaknesses of friendly and enemy units. They try to win each wargame 

for the enemy. Enemy decision points, enemy reactions to friendly actions, 

and projections of enemy losses are products of this effort. Recommended 

intelligence requirements and a refined event template with decision points 

are produced.

• The Assistant Chief of Staff, G-3 (S-3, operations), and G-5 (S-5, plans) are 

responsible for the “movement and maneuver” warfighting function and 

role-play the friendly maneuver commander. They execute friendly 

maneuvers as identified in the course-of-action sketch and the statement 

provided as input to the wargame. They also assess warfighting 

requirements, develop plans and orders, and identify possible branches; 

they also coordinate and synchronize warfighting functions. A 

decision-support template and matrix, annotated with action rationale, are 

products of this effort.

• The Chief of Fires is responsible for the “fires” warfighting function and 

develops metrics measuring effectiveness for fire support and evaluation. 

Named and targeted areas of interest are identified, along with a proposed 

high-priority list, target selection standards, and attack guidance.

• The Chief of Protection is responsible for the “protection” warfighting 

function, which develops a method of protection for each wargame. This 

role develops risk control measures and identifies critical and defended 

assets. Products include synchronizing recovery coordination, air and 
Approved for public release; distribution is unlimited.

11

missile defense, and area operational security (HQ–DA ATTP 5-0.1 2011, 

p. 4–24, 4–25). 

• The Assistant Chief of Staff, G-1 (S-1, personnel), G-4 (S-4, logistics), and 

G-8 (financial management) are responsible for the “sustainment” 

warfighting function. The G-1 identifies potential shortfalls in maintaining 

staffing. The G-4 assesses the status of logistics and movement functions. 

If the G-4 identifies shortfalls, they also recommend actions to minimize 

the effects. The G-8 assesses partner relationships, special funding 

situations, and procurement process to identify resource deficiencies.

• The Assistant Chief of Staff, G-6 (S-6, signal), G-7 (S-7, inform and 

influence activities), G-9 (S-9, civil affairs operations), Red Team Officer, 

Staff Judge Advocate, ORSA Officer, and Safety Officer are responsible for 

the “mission command” warfighting function. The G-6 assesses and 

recommends actions to minimize any negative effects of the 

electromagnetic spectrum, and evaluates network and information 

operations and protection. The G-7 assesses how actions will affect 

information operations, along with other consequences of interacting with 

the population. The G-9 considers how tactical and sustainment operations 

may affect public order and safety, disaster relief, evacuation, emergency 

services, and the protection of cultural sites. The Red Team Officer provides 

the commander and G-2 with the independent capacity to explore 

alternatives from the perspective of adversaries, partners, or others. The 

Staff Judge Advocate advises on matters pertaining to law, policy, 

regulation, good order, and discipline. The ORSA Officer provides 

quantitative support to planning, a quality control capability, and measures 

the effectiveness of operations. The Safety Officer develops a risk 

management strategy to help minimize accidents (HQ–DA ATTP 5-0.1 

2011, p. 4–25, 4–26).

2.1.2 Wargaming Process and Tools

The Army wargaming process is composed of 8 steps, which includes gathering 

tools, listing all friendly forces, listing known assumptions, listing known critical 

events and decision points, selecting the wargaming method, selecting a recording 

tool, and finally, wargaming the course of action. Figure 3 and the following text 

describes each of these steps in more detail.
Approved for public release; distribution is unlimited.

12

Fig. 3 An expanded illustration of the MDMP wargaming step, with identified inputs, 

outputs, and the process involved (HQ–DA FM 5-0, 2010, p. B-21)

The first step is for staff to gather tools necessary for the wargame (HQ–DA STP 

34-350F-SM-TG 2006; FM 5-0 2010, p. B-25; ATP 2-01.3 2015). The XO directs 

staff to gather the necessary tools, materials, and data. Tools generally include 

running estimates, event templates, and a means to display enemy and friendly 

units, and other organizational tools. Resources traditionally include maps, sand 

tables, computer simulations, and other visual and interactive tools. Once gathered, 

staff places the individual course of action on a map, displayed within the area of 

operations.

The second step is to identify and display all units that could be committed to the 

operation. Additionally, staff should include assets from all involved parties, 

paying attention to logistical constraints.

The third step is for the commander and staff to review assumptions and ensure 

they are still valid and necessary.

In the fourth step, staff list all known critical events and decision points. Critical 

events are those that would directly influence mission success or would cause the 

commander to make a significant decision. Alternatively, this step includes actions 

requiring study. Reactions from civilians, whose actions could affect operations or 

require allocation for secondary population stability missions, are included. A 

decision point is a point in space and time when the commander must make a 

decision about a specific course of action. Decisions may be associated with 

friendly forces or with the status of operations, or they may result from a 

Commander’s Critical Information Requirements request. However, decision 

points do not specify what the decision will be.

The fifth step is to select a wargaming method that helps staff consider both the 

area of interest and the participants who can affect the outcome (HQ–DA FM 5-0 

2010, p. B-26). Three common methods exist: 1) belt, 2) avenue-in-depth, and 3) 

box, each of which may describe either a spatial area of operations or a lines-ofeffort, event-based timeline. Figure 4 illustrates these methods.
Approved for public release; distribution is unlimited.

13

Fig. 4 Visual representations of each wargaming method are as follows: belt, avenue-indepth, and box. The upper figures illustrate how they are applied in spatial areas of operations 

whereas the lower figures illustrate how they are applied to distinct lines of effort (illustrations 

adapted from illustrations in FM 5-0 [2010 p. B-26 – B-29] and FM 101-5 [1997 p. 5-18 – 5-

21]).

The belt method works well for operations crossing terrain, dividing the area of 

operations into a series of parallel vertical strips. For example, this method is 

especially useful when there is a progression of sequential spatial steps, such as 

incremental navigation over terrain, or when there is a stratified deployment of 

enemy units. Alternatively, staff may use lines of effort in place of a spatial layout; 

in this case, each belt segment may include more than one critical objective or 

event, with each critical event derived from distinct lines of effort.

The avenue-in-depth method works well for decisive action, such as an offensive 

action. While this technique may use a spatial area of operations, it may 

alternatively illustrate specific lines of effort according to objectives, events, and 

interrelated relationships. 

The box method is the contextually isolated means of analysis. Detailed analysis of 

a specific area works best during time-constrained situations or planning operations 

in noncontiguous areas of operation. In this technique, staff purposefully isolate 

spatial areas and focus on the critical events within their bounds.

The sixth step is to select a technique to record and display results (HQ–DA FM 

5-0 2010, p. B-29). Results provide a record from which to build task organizations, 

synchronized activities, decision-support templates, event templates, plans and 

orders, and COA comparisons. Records can take the form of synchronization 

matrices or sketch notes. Synchronization matrices coordinate a course of action 

across time, space, objectives, and decision points in terms of each warfighting 

function. Comparatively, sketch notes identify pertinent information concerning 

specific spatial locations. If lines of effort are used, sketch notes identify a specific 
Approved for public release; distribution is unlimited.

14

task or purpose. Sketch notes have a designated sequence number and identify 

information in terms of specific events, actions and reactions, and decision points 

rather than by each warfighting function. Figures 5 and 6 outline a template 

synchronization matrix and sketch notes, respectively.

Fig. 5 A template of a synchronization matrix; the second and third rows identify most likely 

enemy and civilian actions. Rows annotated as “Warfighting Functions” focus on specific 

tactical warfighting functions, with additional rows, as appropriate, added for subordinate 

commands (illustration adapted from FM 5-0 [2010, p. B-29]).

Fig. 6 A template of a sketch note that identifies critical attributes of operations associated 

with decision points. Locations are marked on the map, and wargaming participants identify 

expected actions, reactions, and counteractions pertaining to each sketch note. (Illustration 

adapted from FM 5-0 [2010, p. B-31] and ATTP 5-0.1 [2006, p. 4-32]).

During the seventh step, staff wargame each course of action independently (FM 

5-0 2010, p. B-30). They assess its individual results, strengths, and weaknesses. 
Approved for public release; distribution is unlimited.

15

Staffs then consider the range of options for each action, reaction, and 

counteraction; this occurs for all participants, including the civilian population. 

They identify necessary tasks and required assets, including tasks for one-echelon 

lower and using assets up to 2 echelons lower. The wargame tests the effects of 

action for both intended and unintended consequences. Participants consider all 

possible actors, such as friendly forces; enemy forces, including forces outside the 

area of operations; civilians; and media. Ideally, staff involved in the wargame 

analysis should have been originally involved in the development of the analyzed 

courses of actions (ATTP 5-0.1 2006, p. 4–33).

The eighth step is an optional step, consisting of briefing results to all stakeholders 

(FM 5-0 2010, p. B-33). The presentation acts as a quality control function and 

ensures proper communication and analysis. Content included within the briefing 

includes orders and the commander’s intent, an updated IPB, assumptions, and the 

analysis technique used. Presentation includes critical events, possible actor actions 

and reactions, impact on the civilian population, possible media perception, 

strengths, weaknesses, and results. If identified during the wargame, the 

presentation may also include modifications to courses of actions. Figure 7 

illustrates the inputs, considerations, and outputs of the wargaming process.

Fig. 7 Illustration adapted representation of Table 9-5, Effective Wargaming Results (FM 

6-0 2012, p. 9–35)

Each illustrated wargaming analysis produces an increased understanding of the 

complex necessities of the battlefield, such as requirements for deception and 

surprise, timing for concentrating and beginning attacks, identifying and 

coordinating movement timelines, and estimating durations of operations. 

Additional elements include projections of casualty rates, an identification of the 

minimal essential tasks, allocation, and prioritization of assets to appropriate subcommanders, and possible resulting media coverage.
Approved for public release; distribution is unlimited.

16

2.2 Conceptual Notion of Wargaming

As described earlier, wargaming is an interactive process of identifying, thinking 

about, and visualizing plausible events that could occur during a given course of 

action. Wargaming considers, evaluates, and projects involved actors, resources, 

environments, and their combined interactions. Some games and analyses simulate 

a similar type of holistic interaction, such as free-form games and game-theoretic 

analysis. However, in each of these examples, results of interactions are 

nondeterministic. As such, the outcome is not entirely predictable. However, one 

can characterize the nature of the output produced by these interactions. For 

example, one of the premises of wargaming is that if one makes specific 

assumptions, one gains decision-making insight that helps delimit the set of 

possible outcomes and their likelihoods of occurring, provided assumptions are and 

remain valid.

Several types of games exhibit properties applicable to wargaming. Two examples 

are so-called rigid and free-form games (Kretchik 1991, p. 5). Rigid games, such 

as chess or poker, identify specific rules before the game starts, and because a finite 

number of choices and moves exist, an optimum solution can be determined. For 

example, in chess, only one piece can move at a time. The benefit of such a game 

is that one can probabilistically identify the possibility of future actions. 

Free-form games loosely define rules and participants are interactive. Consensus 

between participants often forms and frequently determines the game’s result. The 

game of tag is an example where participants adjust the rules of the game according 

to environmental conditions. Compared to rigid games, free-form games better 

model realistic situations by embodying the flexibility of how actors interpret 

situations, interact with each other, select personal and group strategies, and make 

decisions. Because of the dynamic nature of interactions, emergent outcomes often 

result.

Game theoretic analysis is a branch of mathematics suited to the analysis of social 

situations and interactions. It is a technique modeling actor interaction, each with 

interrelated goals and, as such, it is useful for wargaming. Interaction manifests 

through the dynamics of friendly actions, enemy reactions, and friendly 

counteractions. The traditional game theoretic assumption is that each player 

selects an optimal strategy, without knowing the other player’s strategies, and tries 

to obtain the best possible outcome. Game theoretic approaches make 3

assumptions: a rational actor, rational control, and strategies.

A rational actor considers its own interests and pursues a strategy to achieve a set 

of goals. The rational actor works against moves by opponents that prevent it from 
Approved for public release; distribution is unlimited.

17

accomplishing its goals. Alternatively, an irrational actor exhibits properties such 

as not knowing what it wants, failing to define goals, and not seeking attainment of 

objectives.

Rational control assumes consistent rules and principles guide actors. Both rigid 

and free-form games are examples of such control. Both are consistent, in terms of 

their definitions, but express different phenomena.

The third assumption of game theoretic approaches is that actors will use strategies. 

Two ways of evaluating strategies exist: the first considers the spectrum of 

possibilities that any actor may take, and the second uses a selected strategy. If an 

actor is using a selected strategy, that strategy may create indicators that can cue 

opposing actors to the possibility of the decision. During wargaming, participants 

evaluate whether actors are able to execute a given strategy and whether a given 

actor has already made a decision. Wargaming participants evaluate these decisions 

by making intelligence requests for symptoms, identifying named areas of interest, 

and identifying the appropriate decision points (ATTP 5-0.1 2006, p. 4–24). 

Another type of game combines rigid and free-form games; these are serious games. 

Serious games are an explicit and carefully thought-out educational learning 

experience (Abt 1968; Shaffer et al. 2005). This does not limit the enjoyment of 

serious games, but that is not their primary objective. The difference between casual 

and serious games are that the latter have a specific learning objective, engage 

participants with interactive media, and have some form of gaming aspect. Early 

forms of serious games used behavioral models, but recent games have sought to 

incorporate experiential, situated, and social–cultural models. Serious games are 

useful when there is some nominal amount of uncertainty or ambiguity. However, 

the utility of serious games is minimized if a high degree of uncertainty exits 

(Protopsaltis 2011).

Multiple models of learning are included during gaming, including constructive and 

experiential learning. Constructive learning enables participants to contrast mental 

models to understand the presented situation. Participants reinforce learning by 

explaining insights. In contrast, experiential learning is the result of action. The 

participant is immersed within a situation or task and actively works to resolve the 

situations. Different models for learning exist but many often follow a similar 

process: model of a virtual event occurs, participants interpret, reflect upon, and 

learn from the situation, and finally take action.

Regardless of the type of game, learning assessment remains difficult because of 

the open-ended nature of these types of activities. For example, skills used within 

the activity may include the ability to innovate, collaborate, think critically, produce 

a resulting product, and perform system thinking (Shaffer et al. 2005). However, 
Approved for public release; distribution is unlimited.

18

the means to collect information on each individual skill is limited. Thus, some 

researchers have concluded that an evaluation of the process used may provide a 

better means of measuring value than the grading of factual results.

In wargaming, participants follow a rigid game, but there is a variety of actors, 

environments, decisions, and interactions, so free-form results can still emerge. 

Army doctrine ensures staff uses a consistent process to produce output and 

specifies outputs expected from wargaming. The first expectation is to list 

advantages and disadvantages as they become apparent during wargaming analysis. 

The second expectation is for all wargaming participants to remain unbiased and 

avoid influences by other members. The third expectation is for participants to 

evaluate the feasibility of each individual course of action and determine whether 

it meets mission objectives and requirements. The fourth expectation is for 

participants to avoid comparing any course of action with another during the 

wargaming stage; course-of-action comparison is a process that takes place after 

the wargaming process is complete, after which the commander makes a decision 

in a subsequent MDMP step. Finally, the last expectation is to avoid premature 

conclusions; instead, staff should present facts for each course of action 

independently (FM 6-0 2016, p. 9–36).

Several theoretical perspectives and resulting methodologies guide wargaming 

analyses, each dependent on the individual participants’ perspective of risk (Wang

and Ruhe 2007). One approach analyzes a set of possible courses of action, 

evaluates the enemy’s capabilities, and chooses the least risky option to friendly 

objectives. Another is similar to the first but analyzes possible courses of actions 

from an assumed enemy perspective. As such, the second approach embodies an 

optimistic approach because participants attempt to model enemy intentions and 

choices before action has actually begun. In both perspectives, participants assume 

rational actors. However, the underlying assumptions of each participant influence 

the wargame, actor choices, and analysis. Selection of one approach over the other 

is a function of how much risk each participant estimates the commander will 

accept.

From the standpoint of theoretical methodology, the first perspective models the 

game theoretic maximin selection methodology (Haywood 1951). Participants 

express analyses as an evaluation of the detected enemy capabilities. Wargaming 

participants assume rational actors in both friendly and enemy forces, which imply 

that both forces will establish objectives, pursue distinct courses of action, and 

attempt to block each other’s opposing actions. Moreover, once analyses are 

complete, the participant assumes the commander would like to select a course of 

action that has more advantages than disadvantages.
Approved for public release; distribution is unlimited.

19

Army doctrine specifies several wargaming products, one of which is a comparison 

matrix of possible friendly courses of action versus enemy COAs. In this matrix, 

each COA comparison has an associated predicted outcome, such as the likely 

success of the friendly COA. Maximin theory states that a temporary set of worstpossible outcomes for each friendly COA is created, from which the best-possible 

outcome is selected as the appropriate COA. The selected COA is ideally the most 

robust selection but is not optimal, since the methodology assumes that friendly 

forces will act first, which is a theoretical disadvantage. An example of the maximin

selection process in shown in Fig 8.

Fig. 8 An illustration of the decision-making process using the maximin process. In this 

example, outcomes of each course of action combination are identified. The selection process 

identifies the worst-case outcomes for each COA; in this case, from the perspective of 

Commander A. From those outcomes, Commander A selects the best-case scenario (Kretchik 

1991, p. 8).

Comparatively, the second perspective models the game theoretic minimax 

selection methodology. Participants analyze the situation from the enemy’s 

perspective and try to identify the enemy’s intentions. As in the first perspective, 

participants assume rational actors. However, in the second, participants develop 

COAsto take advantage of suspected enemy-first moves, and predict future actions.

Similar to the first technique, a comparison matrix is produced comparing friendly 

and enemy COAs with their estimated results. However, this second approach is 

not without its risks—confusing indicators result from deception, random enemy 

activity, and the fog of war. When conflicting indicators appear, the recommended 

process is to weigh each indicator, placing more weight on indicators that are 

indicative of intent. Minimax theory states that a temporary set of best-possible 

outcomes for each friendly COA is created, from which the worst possible outcome 

is selected as the appropriate COA. Using this methodology allows the participant 

to select a theoretically more optimal COA than the first technique because it 
Approved for public release; distribution is unlimited.

20

assumes enemy intent is discernable prior to action. An example of this alternative 

minimax selection process is shown in Fig. 9.

Fig. 9 An illustration of the decision-making process using the minimax process. In this 

example, outcomes of each course of action combination are identified. The selection process 

identifies the best-case outcomes for each COA; in this case, from the perspective of 

Commander A. From those outcomes, Commander A selects the worst-case scenario.

2.3 History of Army Wargaming

Formal wargaming in the United States began in the late 1800s with a book called 

The American Kriegsspiel: A Game for Practicing the Art of War upon a 

Topographical Map, published by MAJ WR Livermore (1882) in the US Army 

Corps of Engineers.∗ This book was an American application of German 

wargaming techniques, originally based on the writing of von Tschischwits and 

translated to English by CPT Baring (Young 1956, p. 14).

American Kriegsspiel suggested playing the game using 3 maps, each in different 

rooms, to segregate each team’s information and knowledge. Two of the rooms 

contained personnel representing friendly and enemy forces, respectively; each 

team knew its own state but did not have a comprehensive view of the entire 

battlefield. The third room recorded the entire game (i.e., ground truth) on a board 

called the Firing Board, which an umpire used. The umpire used lookup tables and 

charts detailing the rates of movement under different conditions, calculations of 

losses, and effects of fire. The game modeled the effects of a variety details, such

as fire, fatigue, level of training, morale, and terrain variations, on actions and 

interactions during the game.

After publishing his book, MAJ Livermore proposed the game to the Army. 

However, the Army’s Chief of Staff at the time, GEN Sherman, did not approve. 

 ∗Another individual, LT Totten, also produced a similar type of game. However, LT Totten published his 

version after MAJ Livermore’s (Young 1956, p. 16). A major difference between the 2 games was LT 

Totten’s version specified 2 levels of difficulty, beginner and advanced.
Approved for public release; distribution is unlimited.

21

He justified his decision by stating that the wargame “depicted men as if they were 

blocks of wood rather than human beings seized by fear and sustained by 

leadership” (Caffrey 2000). This was because the game’s rules did not account for 

psychological breaking points. For example, the game depicted attrition as units 

fighting to the last man. However, after several years of effort, in 1899 the Army 

established a war college and included wargaming in the curriculum.

Over time, the Army improved on the original game, changing from wooden blocks 

to transparent overlays, which documented and recorded moves for subsequent 

analysis, among other changes. The Army also created a standardized operations 

order format, given to the umpires, to adjudicate actions. 

Creating estimates of the situation became Army doctrine in 1910 and first 

published in FM 101-5 in 1932 (Fig. 10). This publication documented the origins 

of the military decision-making process, which originally consisted of 5 major 

steps: Mission, Opposing Forces, Enemy Situation, Own Situation, and Decision. 

Elements of wargaming were included in this process and identified as “Analysis 

of Plans”.

Fig. 10 The original specification of the MDMP, first documented in FM 101-5, 1932 (figure 

adapted from Table 1: Format of the Estimate [Michel 1990, p. 5]).

Little else changed until the pre-World War II era. Wargames were primarily used 

for instruction and the training of field maneuvers. However, over time free-form 

wargames (also known as Free Kriegsspiel) increased in Army importance and use.

In 1927, von Neumann suggested the theory and potential of a rigid type of 

wargame (i.e., game theoretic analysis). In 1944, von Neumann and Morganstern 

published an account of game theory in their book Theory of Games and Economic 

Behavior (Young 1956, p. 23). They described situational consequences as the 

result of combining actions of friendly decision-makers with those of the opponent. 
Approved for public release; distribution is unlimited.

22

In the 1940s, the Army began to improve the rigor of wargaming. Historians have 

lauded the Army’s Chief of Staff, GEN Marshall, for the improvement. He liked 

wargames from the time he was a junior officer and, with the likelihood of war 

growing, he directed that wargames be executed as field exercises. 

During World War II, both the British and Germans used an intentions-based 

technique for their wargaming. When America entered the war, it was using a 

similar technique; as such, the United States began using British intelligence for 

strategic and tactical information. However, after a short time the Army 

discontinued its practice of estimating enemy intentions because they learned the 

danger of underestimating the enemy (Lewis 2004). After the war, the War 

Department agreed with the decision and concluded that estimating enemy 

intentions had not substantially improved intelligence results.

Once World War II ended, much of the Army’s wargaming effort ceased until the 

Army realized it might have to fight the Soviet Union. This renewed realization of 

the value of wargaming came from debriefing German officers, as American 

personnel learned the value of the Germans’ wargaming practices. One of the 

lessons learned included identifying enemy intentions, a practice the Army had 

originally discarded because of the risks it encouraged. However, the Army once 

again adopted the procedure of estimating and including enemy intent into its 

wargaming.

In 1951, a RAND study by COL Haywood identified the applicability of game 

theory to military situations. His proposal that decisions should be based on an 

estimate of enemy intentions rather than on their capabilities eventually became 

integrated into modern Army wargaming doctrine. 

Haywood also suggested sequential action and counteraction matrices be developed 

during wargaming. These matrixes are nearly identical to 2-person, zero-sum 

games identified in game theory. As such, those games facilitated use of both the 

maximin and the minimax decision-making methodologies. However, the zero-sum 

game introduced an assumption that both friendly and enemy forces were rational 

and employed identical concepts of military worth. It also assumed decisions, based 

on estimates of enemy intentions, would result in outcomes at least as favorable as 

those identified by the alternative assessment methodology using enemy 

capabilities.

During the 1960s, helicopter enthusiasts used wargaming to develop the concept of 

the air-mobile division (Caffrey 2000). This was similar to how Germany used 

wargaming to develop Blitzkrieg tactics, techniques, and procedures. Defense 

Secretary McNamara directed the Army to follow through with the concept. 
Approved for public release; distribution is unlimited.

23

However, in this case, once the Army deployed its first helicopter division, it 

quickly found combat was different from wargaming.

Decision-makers also discovered wargaming results had not anticipated the 

political consequences of decisions. As a result, in 1964 the Advanced Research 

Projects Agency funded efforts to produce a wargame that depicted all political, 

psychological, and economic ramifications of an insurgency. Despite the 

availability of these improvements, the defense planning community continued to 

use traditional attrition-based wargames.

In 1974, the Army became the first service to buy a commercial wargame, the 

tactical ground-combat simulation “Fire Fight” (Caffrey 2000; Dunnigan 2005). In 

1980, the Army opened the National Training Center (NTC), which employs an 

instrumented range, technology similar to laser tag, and a credible aggressor force 

to produce a realistic ground-combat environment. Wargaming became more 

popular once each maneuver base established wargaming centers. Commanders 

realized it took less time to set up wargames than to set up other forms of training.

During the first Gulf War, NTC used wargaming exercises to prepare Soldiers for 

deployment. Individual Army units began to wargame their own responsibilities 

with respect to attack plans. In one instance, an Army unit used a commercial 

wargame for training. However, similar to other historical applications, wargames 

still had the risk of misleading commanders. For example, the predicted number of 

causalities during the first Gulf War was higher than was actually realized. This 

caused commanders to configure transport aircraft for medical airlift. However, 

once the expectation was realized as invalid, commanders reconfigured transport 

aircraft to transport fuel, which turned out to be the actual need.

2.4 Historical Evolution of Wargaming

The following describes the historical context of wargaming, illustrating several of 

the similarities and differences between current and historical practices. It also 

identifies examples of how militaries have used wargaming throughout history.

Informal wargaming, or the consideration of actions, reactions, and counteractions, 

has likely existed since the beginning of human history. Historical first-generation 

games such as chess and Go are abstract representations of conflict and war, likely 

developed years ago in the Far East to help players consider, visualize, and practice 

decision-making and learn consequences of courses of action. In combat, it was 

historically the commander’s responsibility to visualize situational contexts and 

make the appropriate decisions. Anecdotes, such as Napoleon’s use of toy soldiers 
Approved for public release; distribution is unlimited.

24

to visualize battle sequences, illustrate this level of responsibility and forward 

planning.

Historians credit the Prussian Baron von Reisswitz with the development of the 

second-generation wargame. In 1811, he constructed a table-sized model of terrain 

and used blocks to represent units; this was the predecessor to the Army sand table. 

Players would give orders to an umpire, who in turn updated the terrain table, 

resolved combat, and informed players of the knowledge they had of the battlefield. 

Umpires derived casualties by using lookup tables, the roll of a die, and contextual 

attributes like range and terrain.

The Prussians developed the idea of training professional staff officers, and 

wargames evolved to help them visualize battles from start to finish. Reisswitz’s 

game was adapted to topological maps in 1824, and thereafter wargaming became 

standard in Prussian Army practice. As early as 1828, then LT von Moltke 

advocated use of wargames. Nine years later, GEN von Moltke became Chief of 

Staff of the Prussian Army and ordered an increase in the use of wargaming. He 

also used wargaming performance as an evaluative criterion for acceptance into the 

War College, where wargaming was a regular part of the curriculum.

Periodically, GEN von Moltke would take the entire War College student body out 

to a possible invasion corridor. He would describe the most likely first clash to all 

the students, then turn to the most junior student and ask for his plan of battle. Next, 

he asked the second most junior and so forth until all students provided their plans. 

After coming to a consensus for a battle plan, GEN von Moltke designated the next 

senior ranking general after him to command the invading forces and the next 

ranking general after the first to command the Prussian forces. He split the students 

into 2 equal teams and assigned one team as invading commanders and the other as 

Prussian commanders. Once the game had been completed, GEN von Moltke would 

direct the local garrison commander to march several hundred troops to gather 

marching times and other details of the plan. Once complete, the plan became the 

actual Prussian defense plan for an invasion along the exercised corridor.

In the 1860s, the Prussians began using wargaming as a decision-making aid. Over 

time, GEN von Moltke won a series of wars against opponents who sometimes had 

larger forces, and other militaries began to adopt the Prussian wargaming technique. 

However, Prussia/Germany continued to adapt its wargaming techniques.

For example, CPT Naumann published rules for identifying breaking points in 

1877. These rules identified criteria for determining at what casualty levels units 

would cease functioning. This was an improvement over the existing process, 

which used attrition tables to simulate interactions. Interestingly, these rules were 
Approved for public release; distribution is unlimited.

25

missing from the early American wargaming adaptations in the late 19th century 

and were one of the primary reasons GEN Sherman rejected the technique.

The second adaptation was Free Kriegsspiel, a controversial modification 

developed in the mid-1870s. Proponents of Free Kriegsspiel advocated a radically 

different type of wargame, which allowed experienced participants to substitute 

their personal military judgment for many of the time-consuming adjudication 

rules. Initially, the Free Kriegsspiel technique worked well, but over time, 

participants could not adjudicate results accurately due to a lack of actual combat 

experience.

Additionally, using a Free Kriegsspiel methodology, ranking participants began to 

override umpire-adjudicated results based on their professional judgment. The 

immediate result of this modified wargame was a more popular game. Participants 

could complete wargames faster and they would play wargames more often and 

gain confidence in their results. However, in practice this methodology led to 

negligent analysis and overconfidence, which contributed to significant military 

defeats.

2.5 Historical Applications of Tactical Wargaming

The British military experimented with wargaming briefly around the turn of the 

19th century. The British Parliament listed wargaming as a hobby, and a civilian 

naval wargame was the reason for the development of the annual reference book, 

cataloging the world’s military warships, Jane’s Fighting Ships. However, the 

popularity of wargaming subsided as the British discovered it could not address the 

psychological or political dimensions of the Boer War.

The Russians also began adopting wargaming practices but chose to use the Free

Kriegsspiel methodology. For example, one wargame during 1914 demonstrated a 

successful means of advance against German forces. However, at a critical point 

where terrain made communication difficult, the commanding Russian generals 

overrode the negative, adjudicated results and said the strategy would be successful. 

Four months later, the same scenario and plan as that within the wargame played 

out, and the Russians made the same progress as suggested by the wargame. 

However, when the Russians encountered the same difficult terrain simulated in the 

wargame, German forces prevailed against the Russians.

Prior to World War II, Germany continued to improve their wargaming technique. 

Since wargaming could model attrition but neglected other aspects, such as political 

effects, the German government began running strategic-level wargames to 

compensate for the political deficiency. Limited to their post-World War I situation, 
Approved for public release; distribution is unlimited.

26

Germany decided it could wargame with military forces it did not possess. Based 

on historical lessons, the Germans were able to derive new theories and develop 

unique military doctrine. This new doctrine was wargamed to see whether it might 

work; some of it did and was eventually termed mobile-operations or Blitzkrieg. 

Interestingly, when Adolf Hitler came to power, he halted strategic-level wargames 

(Caffrey 2000).

Throughout World War II, the United States, Russia, Germany, and Japan each 

wargamed possible battlefield situations, each with varying degrees of tactical 

success. The Navy used wargaming to develop the carrier attack-group formation. 

The Marines used and refined its wargaming techniques to the point where they 

could accurately predict battle outcomes and logistical needs; it became so accurate 

that it eventually detected the change in Japanese strategy, which emphasized loss 

of life. Similarly, Russian GEN Zhukov wargamed Germany’s battlefield tactics 

accurately. Japan wargamed the emerging tactical World War II situation to decide 

whether to enter the war; however, the wargame did not include political 

considerations. Japan later ran wargames indicating it would lose the war, and that 

prompted Japanese leaders to adopt a new strategy.

Despite resistance to strategic wargaming, Germany continued to use tactical 

wargaming extensively throughout World War II. In early 1944, the Germans 

conducted a wargame of a possible Allied invasion of France that focused on 

logistical preparation. When reconnaissance spotted preparations across the 

Channel from Normandy, the Germans conducted a wargame of an Allied landing 

at Normandy. While the German opinion was that an Allied invasion was a feint, 

the Germans concluded that if the feint were successful, the Allies might make that 

their primary attack. Initial reinforcements, resulting from this wargaming 

conclusion, made the D-Day invasion on Omaha Beach one of the most difficult 

battles of the Normandy invasion. Ironically, assisting the Allies’ invasion, another 

German wargame was taking place when the D-Day invasion began. It was 

evaluating the possibility of an Allied invasion of Normandy, with all their 

reinforcements in place. However, because the wargame took place during the 

actual invasion, it had the effect of keeping key German commanders from 

participating during the initial stages of the Normandy invasion.

Another tactically applied German wargame was during the Battle of the Ardennes, 

in the fall of 1944; the Fifth Panzer Army conducted a wargame of an American 

attack on their position. Coincidently, during the wargame, the Americans attacked. 

However, Field Marshall Model did not stop the wargame; instead, movements 

from the field stimulated actions within the wargame. The Germans used the 

opportunity to wargame each order before actually executing it (Young 1956, p.

21).
Approved for public release; distribution is unlimited.

27

After World War II, countries continued to develop their specific versions of 

wargaming practices. For example, American wargaming changed from a 

commander’s task to a staff function informing the commander and helped 

institutionalize COA analysis. This contrasted with Britain’s and Germany’s 

approach, where staff was responsible for developing COAs but the commander 

was exclusively responsible for their analysis.

Both pros and cons arose from these changes. The American version could cause 

the decision-making process to take more time, due to additional analysis, but the 

advantage was the ability to include multiple perspectives on the wargamed 

situation. This approach also required additional effort to identify the most 

plausible outcome. Britain and Germany’s processes reverted to free-form 

wargaming, which made wargaming easier for the commander but reintroduced the 

possibility of “subjective expert analysis bias”, which had been so disastrous in the 

past.

Other differences continued to emerge over time. For example, the American 

approach emphasized developing optimal courses of actions based on enemy intent.

It focused on the possible choices and motivations of an intelligent adversary. 

Conversely, the British and Germans used capability-based assessments, consisting 

of straightforward adversarial models, and they simply assumed that during a 

conflict adversaries would use all capable assets. For example, a German 

commander would use information regarding the enemy’s capabilities to formulate 

his plan. Trying to discern the enemy’s intentions was a secondary priority. 

Friendly-to-enemy force ratios were calculated, but the commander would mentally 

estimate conflict effectiveness of friendly-to-enemy interactions. Similarly, British 

commanders calculated friendly-to-enemy force ratios and estimated conflict 

effectiveness. However, the British replaced the discernment of enemy intentions 

with the identification of a most dangerous enemy COA.

Each change had its advantages and disadvantages, but all had unique potential to 

skew wargaming conclusions. Identifying enemy intent and a resulting course of 

action, if chosen correctly, has the potential to lead to better friendly outcomes. 

However, the danger with this approach is that predictions can be wrong and 

inappropriate action can introduce additional risk. Comparatively, the alternative 

capability-based approach suffers from the personal estimation of effectiveness. As 

in free-form wargaming, this methodology introduces the possibility of subjective 

expert analysis bias.

Like the British and German wargaming processes, the Soviet intelligence officer 

provided doctrinal information without participating in wargames. The Soviet 

process included consideration of the mission, instruction from superiors, and 
Approved for public release; distribution is unlimited.

28

known factors, including available decision-making time, current friendly and 

enemy situations, and the capabilities of subordinate commanders. However, Soviet 

commanders used lookup tables to calculate capabilities of friendly and enemy 

units, whereas British and German wargaming estimated each conflict’s 

effectiveness. Moreover, the Soviets assumed that their forces would have enough 

substance to fight through any conflict. Identifying enemy intent was ideal, but it 

was susceptible to deception and such analysis was limited by time.

Differing from the American wargaming process, British, German, and Soviet 

variants did not define specific wargaming rules. Of the 3, the German process was 

the most specific and included a 3-step process, comparing friendly and enemy 

combat effectiveness, examining changes in relative strength, and comparing 

friendly COAs to their chance of success.

On the other hand, the formalized MDMP became the surrounding context of 

American wargaming. The original MDMP was a 5-step process, first documented

in 1932 (Michel 1990). In the 1950s, the United States adopted the staff function 

of wargaming, which included concepts from Game Theory and used concepts of 

rational actors with the goal of choosing optimal actions (Kretchik 1991, p. 15). In

1960, the MDMP expanded to include all command and staff actions needed to 

develop and execute a course of action (Shoffner 2000, p. 6).

In the 1960s, Soviet researcher Dr Lefebvre proposed the origins of what would 

become reflexive control theory (Thomas 2004). Reflexive control is an 

incremental adaptation to Game Theory. It observed the interactions of actors with 

their environments and suggested that one could influence the environment such 

that an adversary would willfully make a predetermined choice (Shemayev 2007). 

The Soviets began incorporating this theory into their wargaming and military

decision-making doctrine as early as the 1960s. Soviet military interest further 

increased during the 1970s as 2 books were published, Mathematics and Armed 

Conflict and Problems of Military System Engineering (Chotikul 1986).

In this section, the historical evolution of wargaming illustrates the context of 

current Army wargaming. Historical examples, such as those demonstrated by GEN

von Moltke and more recently during World War II, illustrate both wargaming 

utility and limitations.

2.6 Analytical Frameworks

The Army uses analytical frameworks to communicate assessments of the situation 

to and within tactical commands; for example, a PMESII assessment provided with 

an operations order. These analytical frameworks, while useful in describing the 
Approved for public release; distribution is unlimited.

29

state of different situations, represent a structural representation of situations. 

However, the Army wargaming process uses data contained within analytical 

framework assessments as inputs to the interactive wargaming process.

Army Field Manuals advocate at least 3 analytical frameworks, which describe 

strategic, operational, and tactical situations. These frameworks include 1) political, 

military, economic, social, information, and infrastructure (PMESII); 2) area, 

structures, capabilities, organizations, people, and events (ASCOPE); and 3) 

mission, enemy, terrain, troops available, time, and civilian considerations (METTTC) (HQ–DA FM 3-24.2 2009, p. 4–14; FM 5-0 2010, p. 1–5; ATTP 5-0.1 2011, 

p. 5–7). These frameworks encourage consistent and comprehensive 

communication between stakeholders at all levels and help them understand and 

communicate situations and environments. These methods also identify substeps 

within the larger analytical process. For example, ADRP 2-0, Intelligence (2012, p. 

2-5) states one can use PMESII, ASCOPE, or METT-TC during the IPB, part of 

the larger MDMP.

2.6.1 PMESII-PT

PMESII, and its evolutionary descendent PMESII-PT, which adds physical 

environment and time, is a strategic and operational-level analysis framework. 

Currently, the Army uses PMESII-PT to convey an operational context and the 

understanding of a situation and mission to lower-level tactical commands (FM 3-

24.2 2009, p. 1–3). Planners use PMESII-PT variables to describe the specific 

operational environment of operations (Hartley 2008, 2010). Tactical commanders 

receive PMESII-PT assessments from their higher-level headquarters and are 

responsible for translating them into relevant and actionable tasks.

The Army identifies and defines each of the PMESII-PT variables as summarized 

in the following (DA TRADOC G-2 2012, p. 55):

• Political: Describes the distribution of responsibility and power at all levels 

of governance—formally constituted authorities, and informal or covert 

political powers. It asks the questions, “Who are the tribal leaders in the 

village?” and “Which political leaders have popular support?” 

Subvariables include attitude toward the United States, centers of political 

power, type of government, government effectiveness and legitimacy, 

influential political groups, and international relations.

• Military: Explores the military and/or paramilitary capabilities of all 

relevant actors (e.g., enemy, friendly, and neutral forces) in a given 

operational environment. It asks the question, “What is the force structure 

of the enemy?” Subvariables include military forces, government 
Approved for public release; distribution is unlimited.

30

paramilitary forces, nonstate paramilitary forces, unarmed combatants, 

nonmilitary-armed combatants, and military functions.

• Economic: Encompasses individual and group behaviors relating to 

producing, distributing, and consuming resources. It asks the question, 

“What is the unemployment rate?” Subvariables include economic 

diversity, employment status, economic activity, illegal economic activity, 

and banking and finance.

• Social: Describes the cultural, religious, and ethnic makeup of an 

operational environment and the beliefs, values, customs, and behaviors of 

society members. It asks the question, “What is the ethnic composition of 

the operational environment?” Subvariables include demographic mix, 

social volatility, education level, ethnic diversity, religious diversity, 

population movement, common languages, human rights, centers of social 

power, and basic cultural norms and values.

• Information: Explains the nature, scope, characteristics, and effects of 

individuals, organizations, and systems that collect, process, disseminate, 

or act on information. It asks the question, “How much access does the local 

population have to the news media or the Internet?” Subvariables include 

public communications media, information warfare, intelligence, and 

information management.

• Infrastructure: Details the composition of the basic facilities, services, 

and installations needed for the functioning of a community or society in 

the operational environment. It asks the question, “What are the key modes 

of transportation?” Subvariables include construction patterns, urban zones, 

urbanized building density, utilities, and utility level.

• Physical Environment: Depicts the geography and manmade structures 

and the climate and weather in the operational environment. It asks the 

question, “What types of terrain or weather conditions in the area of 

operations favor enemy operations?” Subvariables include terrain, natural 

resources, and climate and weather.

• Time: Describes the timing and duration of activities, events, or conditions 

within an operational environment in addition to how various actors in the 

operational environment perceive timing and duration. It asks the question, 

“What is the cultural perception of time in the operational environment?” 

Subvariables include knowledge of the area of operations, cultural 

perception of time, information offset, tactical exploitation of time, and key 

dates, time-periods, or events.
Approved for public release; distribution is unlimited.

31

In spite of the way it is currently used, the Army did not create the PMESII 

framework to communicate understanding. Rather, PMESII was designed to target 

nation-building systems and to address concerns that the existing ASCOPE 

framework was not sufficient (Ducote 2010, p. 6). At the operational level, 

PMESII-PT is a linear analysis tool that addresses “what” questions but does not 

answer “why” questions with respect to a complex environment. As such, the 

PMESII framework limits the level of understanding required by the recent Army 

Design methodology (McLamb 2009; HQ–DA ADP 5-0 2012).

2.6.2 ASCOPE

Compared to PMESII, which identifies operational- and strategic-level items of 

interest, the ASCOPE framework identifies tactical-level civil considerations. FM 

3-24.2 (2009, p. 1–8) defines and identifies ASCOPE as a tool assisting counterinsurgent operations. Six categories make up its composition: areas, structures, 

capabilities, organizations, people, and events (US Army Combined Arms Center 

1997). The Army identifies and defines each category as follows:

• Areas: Identifies geographical locations and regions of interest areas. For 

example, these areas could include political boundaries, social, political, 

religious, or criminal enclaves. It asks the question, “What is the 

relationship between people and where they live?”

• Structures: Identifies how locations, functions, and capabilities can either 

support or hinder operations. Traditionally, high-value structures include 

bridges, communications towers, power plants, and dams. Historically, 

significant sites include churches, mosques, national libraries, hospitals, 

cemeteries, historical ruins, religious sites, and other cultural areas. Other 

sites of interest have included jails, warehouses, toxic industrial materials, 

television and radio stations, and printing plants. It asks the question, “Why 

are the natural and manmade structures important?”

• Capabilities: Identifies the ability and effectiveness of local authorities or 

leaders to provide key functions and services. Priorities are ordered 

according to items necessary to save, sustain, or enhance life. Traditionally, 

public health, security, public works and utilities, economics, and 

commerce are included. It asks the question, “Who is capable and 

responsible for providing people basic services?”

• Organizations: Considers nonmilitary groups and institutions within the 

area of operations. Traditional types of groups have included tribes, political 

instantiations of insurgent groups, nongovernmental organizations, private 

companies, other government agencies, contractors, and media sources. It 
Approved for public release; distribution is unlimited.

32

asks the question, “What are the different groups of people in the 

operational environment?”

• People: Includes nonmilitary personnel whom military forces may 

encounter along with how and where each person communicates. 

Historically in the Middle East, this has included sheiks, city council 

leaders, imams, professionals, displaced persons, human sources, and tribal 

leaders. Forms of communication have included verbal, satellite, graffiti, 

and the Internet. Examples of meetings include gatherings at mosques, tribal 

council meetings, and cafes or teashops. It asks the question, “How do 

people communicate?”

• Events: Includes routine, cyclical, planned, or spontaneous activities. 

Examples have included elections, anniversaries, religious events, funerals, 

and political rallies. It asks the question, “When are events occurring?”

Both tactical and operational analyses are interconnected and related. For example, 

both ASCOPE and PMESII describe essential observables of a situation. Both 

frameworks help elicit situation conditions, which could cause the commander risk

(Ntuen 2008; HQ–DA ATP 3-57.50 2013).

A helpful acronym of the ASCOPE Capabilities category is encapsulated by 

SWEAT-MS, which represents sewer, water, electricity, academic, trash, medical, 

and security (HQ–DA FM 3-24.2, 2009, p. 1–13; ADRP 1-02 2015). The acronym 

helps analyze personalities who control essential resources in a community. For 

example, the local government includes the mayor and council, who may in turn 

control the sewer. Historically, a tribal leader controls and protects water wells. In 

the Mideast, the government and imams influence academic curriculums. Private 

businesses and local merchants produce trash. Tribal doctors may provide medical 

assistance. Local police or tribal militia may influence local security.

Figure 11 displays an intersection of variables that influence both frameworks; 

PMESII variables are listed in the top row, and ASCOPE variables are identified in 

the first column.
Approved for public release; distribution is unlimited.

33

Fig. 11 A crosswalk matrix between PMESII and ASCOPE variables. The figure illustrates 

examples of each variable combination.∗

2.6.3 METT-TC

Acting in a complementary role to operational PMESII variables and tactical 

ASCOPE variables, METT-TC helps analysts to describe mission-specific 

variables. METT-TC is a framework that focuses on specific elements of an 

operational environment and how they might affect a mission (Baillergeon and 

Sutherland 2014). It identifies 6 variables: mission, enemy, terrain and weather, 

troops and support, time available, and civil considerations. The Army has 

identified and defined each of the METT-TC variables as summarized in the 

following (ADRP 5-0 2012, p. 1–9):

• Mission: Identifies variables in terms of impact on objectives and is the 

first variable commanders consider. It asks the question, for the given 

operation, “Who, what, when, where, and why?”

• Enemy: Identifies dispositions (e.g., organizations, strengths, locations, 

and tactical mobility), doctrine, equipment, capabilities, vulnerabilities, and 

probable courses of action. It asks the question, “What is the state of the 

enemy?”

• Terrain and Weather: Identifies terrain (e.g., rivers, mountains, cities, 

airfields, bridges) and weather features (e.g., visibility, wind, precipitation, 

cloud cover, temperature, humidity). The acronym OAKOC (Observation 

and fields of fire, Avenues of approach, Key and decisive terrain, Obstacles, 

 ∗Information and variables adapted from HQ–DA ATP 3-57.50 (2013, p. B-1 through B-8), Civil Affairs 

Civil Information Management, Appendix B, Civil Analysis Example of the Interrelationship of Operational

Variables and Civil Considerations and, ATP 2-01.3 (2015, p. 4-29 through 4-35), Intelligence Preparation of 

the Battlefield
Approved for public release; distribution is unlimited.

34

Cover and concealment) supports identification (Army ROTC 2008). It asks 

the question, “How will the environment affect the mission?”

• Troops and Support Available: Identifies the number, type, capabilities, 

and condition of friendly troops and support; includes supplies, services, 

and support of joint-military organizations, host nation, and partners. 

Support from civilians and contractors employed by military organizations, 

such as the Defense Logistics Agency and the US Army Materiel 

Command, are included. It asks the question, “What resources are 

available?”

• Time Available: Identifies the amount of time available for planning, 

preparing, and executing tasks and operations. This includes time required 

to assemble, deploy, and maneuver units in relationship to the enemy’s 

location and other conditions. It asks the question, “When are the 

deadlines?”

• Civil Considerations: Identifies activities of civilian leaders, populations, 

and organizations. It asks the question, “How can the local population 

impact the success of the mission?”

Other analysis frameworks used in Joint Intelligence analysis include DIME-FIL 

(Diplomatic, Information, Military, Economic, Financial, Intelligence, and Law 

Enforcement) and JIPOE (Joint Intelligence Preparation of the Operational 

Environment). Both of these frameworks support recent whole-of-government 

approaches (Joint Special Operations University 2013).

DIME-FIL identifies the variables and national components of diplomatic, 

information, military, economic, financial, intelligence, and law enforcement 

(McDonnell 2009, p. 4; Shellman et al. 2011). These frameworks serve as 

coordination linkages between agencies with individual specialties so that they can 

participate in and contribute to overall objectives.

JIPOE is the joint-force’s version of the IPB (Joint Chiefs of Staff 2013, p. I-17). 

Both frameworks help analysts to identify current and desired conditions, actors, 

actors’ relationships, and their functions.

2.7 Challenges of Modern Wargaming

Despite the successful current state of wargaming and the value of the associated 

tools and involved processes, several challenges exist within the practice of 

wargaming. Challenges range from training to identification of enemy doctrine to 
Approved for public release; distribution is unlimited.

35

individual participants’ cognitive biases (US Government 2009; Hodge 2012; DA 

TRADOC Pamphlet 525-3-1). 

One significant challenge of wargaming is the ambiguity in the knowledge, skills, 

and abilities (KSA) required for each role. For example, the relationship between 

the role of the Army’s intelligence officer and the role play of the enemy 

commander is ambiguous. On the surface, this is a reasonable task; the job of the 

intelligence officer is to know the enemy. However, this is a naïve assumption; it 

does not communicate the experience or training required (DA 2014). For example, 

the further down the tactical chain of command, the younger the analyst is likely to 

be. In the worst case, it may be the analyst’s first assignment. 

Moreover, the hostile commander’s culture influences perceptions, objectives, and 

methodologies. Assignments of intelligence analysts do not necessary consider this 

nuance. As such, even if an analyst is experienced, he or she may not have any 

familiarity with the culture or particular situation (Peck 2003; USAID 2006; 

Wunderle 2007; TRADOC Pamphlet 525-2-1; Salmoni and Holmes-Eber 2011). 

Despite best efforts, the result of both scenarios is a naïve or biased and possibly 

misrepresented situation presented to the commander (Childs and Gallivan 2013). 

Unfamiliarity in any number of significant attributes can cause wargaming 

participants to become overconfident, misinterpret situations, or provide skewed 

results of actions, by either under- or overestimating capabilities or assets. 

However, in both scenarios, if appropriate KSAs were identified and participants 

were to meet stated qualifications, there would be less risk of wargaming 

conclusions being invalid.

A second challenge concerns the amount of rigor within current Army wargaming 

practices. Realities of a situation play a role during wargaming. Specific contextual 

situations may allow a minimal amount of time, such as planning for contingencies. 

The lack of appropriate or sufficient indicators of enemy intentions can reduce 

wargaming rigor. Moreover, the fog of war can cause the commander to focus on 

the wrong priorities.

Alternatively, minimally experienced personnel can result in low rigor. Research 

indicates that analytical methodology changes with the amount of experience 

analysts have. For example, when coming to conclusions, less experienced analysts 

seek to confirm information about their hypothesis. However, experienced analysts 

will seek information to disconfirm their beliefs.

As a result, there is a degree of variability in wargaming analysis. Army doctrine 

states that staff should evaluate at least 2 courses of action, the most likely and most 

dangerous cases. However, evaluating these 2 estimated situations does not 
Approved for public release; distribution is unlimited.

36

demonstrate thorough analysis of the situation. Moreover, analysis of 2 courses of 

actions does not maintain the standards of rigor promoted by the larger intelligence 

community (ODNI 2015).

A third challenge concerns the knowledge of enemy doctrine and its application 

during wargaming. The practice of using enemy doctrine to inform wargaming 

analysis became popular while the possibility of conflict existed with the Soviet 

Union. During this time, Army doctrine writers assumed foreign military doctrines

were appropriate expectations. However, adherence to stated doctrine may not be 

so rigorously applied in future conflicts. Recent state-on-state conflict has 

illustrated the willingness to create and implement hybrid warfare strategies, which 

purposely blurs lines between formalized and unconventional conflict (US Army 

Special Operations Command 2015). 

Moreover, nonstate actors are an extension of doctrine ambiguity. In some cases, 

doctrine may not exist for nonstate organizational actors. A nonstate actor may not 

know what they will do until an action has actually taken place (DA TRADOC 

Pamphlet 525-8-5). Using doctrine helps wargaming participants to model the 

psychology of opponents, but when doctrine does not exist, predicting future 

actions may actually increase the risk of current practices (Robel 2004).

To model thinking opponents, current practice (based on principles of Game 

Theory) assumes rational actors. Justification for this assumption is reasonable in 

many cases; it is the desire to win the conflict. However, the theoretical rationalactor concept may not be an appropriate model in all circumstances, and other 

forms of actors, such as low-rationality actors, have emerged. Rational actors are a 

projection, a result of wargaming participants’ individual perceptions, and may 

differ from reality (Axtell 2006a). For example, hybrid warfare takes advantage of 

this assumption by crossing limits and boundaries of social expectations and laws, 

while operating below the threshold of traditional war (Anderson et al. 2015; Moore 

2015). In other cases, friendly and adversarial expectations, values, procedures, and 

objectives may differ in both interpretation and perspective; consider, for example, 

the use of suicide bombers as a military tactic.

Finally, wargaming is an evaluation and analysis of the interactions of many actors 

within the context of a course of action. Such interaction has the ability and 

tendency to exhibit emergent phenomena. However, it is difficult to use traditional 

deductive analysis to identify potential emergent phenomenon. Moreover, in 

situations of high uncertainty, wargaming provides little useful value. Thus, the best 

way to increase the rigor of tactical wargaming may be to identify and improve the 

process and methodology (and actual implementation) of wargaming.
Approved for public release; distribution is unlimited.

37

3. Agent-Based Modeling in Tactical Wargaming

As described in the prior section, Army staffs at division, brigade, and battalion 

levels often plan for operations. As such, analysts consider the impact and potential 

consequences of actions taken. MDMP dictates identification and evaluation of 

possible enemy COAs; however, nonstate actors often do not exhibit the same level 

and consistency of planned actions that the MDMP was originally designed to 

anticipate. ABM and its resulting emergent behavior is a potential solution to model 

terrain in terms of the Human Domain and improve the results and rigor of the 

traditional wargaming process.

In this section, the report examines whether an agent-based capability can improve 

the MDMP process by providing additional rigor, speed, or flexibility during 

tactical Army wargaming. The study considers methodologies to improve analysis 

of the Human Domain, identifies a framework for modeling such a capability, and 

compares current Army wargaming practices with those if a hypothetical capability 

were to exist.

3.1 Agent-Based Modeling

ABM helps the modeler understand and simulate how a “whole of a system”

responds to change over time. A complete agent-based model provides a 

fundamentally decentralized method of looking at the world where individual 

agents are instantiated within an environment and empowered to make their own 

decisions (Bonabeau 2002). While top-down or command-and-control 

relationships can be instantiated between agents in the system, ABM agents are 

fundamentally capable of autonomously making decisions without any centralized 

form of decision making or command and control (Odell 2002). Agents make their 

decisions based on their individual perceptions of other agents and their 

environment (Parunak 1996). Classic agent-based models run in simulated time and 

the system, analyzed as a whole, tends toward a steady state or falls into a 

predictable pattern of oscillating between extremes (Carmichael and Hadzikadic 

2015).

Traditional model design and engineering techniques tend to focus on problem 

definition and decomposition for testing and design. Once the system goal is 

identified, engineers generally decompose larger problems into smaller,

well-defined subproblems (Software Engineering Process Management Program 

2010). Subcomponents are individually resolved by creating subsystems that act in 

predictable and controlled ways. Once engineers verify subsystems, they compose 

and integrate subsystems to create a composite system that solves the identified 

problem. Several factors influence the engineering process; for example, the degree 
Approved for public release; distribution is unlimited.

38

of specification in the original problem will influence nature of the created 

subsystems. Other factors include the quality of verification and validation of 

individual subsystems. Numerous other aspects of the engineering effort, including 

planning, development, and configuration management, also contribute to the 

overall quality and value of the overall effort.

Creating an appropriate agent-based model requires a different style of engineering

(Chaturvedi et al. 2004, 2008, 2014; Doursat et al. 2013). The difference between 

traditional engineering and creating agent-based models is analogous to the 

difference between structural and interactive complexity (DA TRADOC Pamphlet 

525-5-500, 2008). Structural complexity is based on the number of parts within a 

system, the larger the number, the greater the complexity. Structural systems tend 

to exhibit the properties of linearity, proportionality, replication, additivity and 

repeatable cause and effect. A small input will result in a small output and a 

proportionally larger input will result in a proportionally larger output. It also means 

the system will respond the same, regardless of when input is stimulated. Finally, 

analysis of a structural system is valid; analysts can systematically evaluate 

progressively smaller components of the system for their properties.

Comparatively, interactive complexity is dependent on behavior of all parts and the 

resulting interactions between them; the greater the freedom of action for each part 

and the more linkages among them, the greater the interactive complexity (Chen 

2011; Green 2011). However, analysis of individual components within interactive 

systems is difficult because interactive components do not exhibit the same 

properties as structural components (Crutchfield and Wiesner 2010). The ability of 

individual agents in an agent-based model to make “decisions” independent of the 

other agents can give the impression of instability and fragility. This is because 

each individual component, when duplicated and placed in an environment with 

other interactive components, tends to exhibit properties such as interactions, 

dynamics or flows, and temporal consequences (Bonabeau 2002).

Historically, differential equations have used model system dynamics (e.g.,

Monte-Carlo simulations) providing a predictable and repeatable way of looking at 

a snapshot of a system and perturbing initial conditions slightly to provide an 

estimate of the outcome. Unfortunately, using a single equation to model entire 

components of a complex system (e.g., population sentiment when the number of 

police increases in a nation-building context) tends to provide a false sense of 

clarity and predictability for system behavior that is significantly different from 

how events may evolve in the real world (Bonabeau 2002; Helbing and Balietti 

2011). ABM provides a more flexible and adaptive framework to tease out of the 

complexity of nuanced and unpredictable elements of a system by allowing 
Approved for public release; distribution is unlimited.

39

decomposition of such complex systems into smaller subsystems with individual, 

often stochastic, simulated decision behavior.

To develop an agent-based model, the modeler creates a set of heterogeneous agent 

types with their own rule sets, and places instantiations of these agents into an 

environment (Bonabeau 2002). Modeling emphasesis on creating and altering rules 

of individual agent types rather than identifying and creating rigid system-level 

interactions. Each agent has the ability to exist in one or more states, to represent 

fundamental shifts in behavior or interact with the environment. Additionally, 

agents have a state-based “decision-making” ability constructed as a set of 

mathematical equations to represent some level of decision-making capability or 

artificial intelligence. Depending on the level of system decomposition, this set of 

equations may capture complex decisions (e.g., whether or not a person should 

organize a riot) or simple decisions (e.g., should the agent look for a food source). 

Depending on the instantiation of the agent, each instance may have slight 

differences in their characteristics governed by an initial distribution (e.g., height 

variation in a population). Additionally, each rule has a probability distribution 

attached to it, which is used to determine whether rules are invoked (e.g., will a 

person respond violently to a perceived threat). This provides a method of capturing 

outcomes or behaviors that are outside of the scope of the model, or that may be 

too complex to replicate, given the current understanding of the real-world 

equivalent. The use of varying initial conditions/parameters for decision-making 

structures of each instantiated agent ensures variance within the totality of systems’ 

agents’ choices (Miller 1996; Barry and Koehler 2004; Nakayama 2012). Once an 

iteration of the model is complete, the modeler simulates and evaluates whether 

changes result in an appropriate outcome. The term “appropriate outcome” is used 

because model output is not precise and is represented by a possible representation 

of the output. By analogy, in programming, the equivalent is compiling and testing 

source code whereas in modeling, it is instantiating a set of agents with random 

initial conditions and running a Monte-Carlo series of tests evaluating whether 

expected system phenomena is observed at various points in simulation time.

ABM works comparatively well when compared to average or Gaussian statistical 

techniques because individual agents can exhibit different types of logic, for 

example, individualized preference for decision-making thresholds, different 

degrees of rationality, conditional rules, and use of memory and learning. Similarly, 

in the context of agent interactions, communication networks where all agents are 

not fully connected are another situation where ABM can simulate emergent 

phenomena. Both types of situations are common in social situations and ABM can 

model these situations.
Approved for public release; distribution is unlimited.

40

3.2 Identify Rigor

Traditional structured problem solving and resulting engineering generally creates 

precise solutions. However, most situations the Army deals with are not precise or 

structured. Rather, interactive social systems strongly influence outcomes of Army 

missions; for example, friendly personnel, adversarial personnel, noncombatant 

personnel, the civilian population, and the resources they use, control, and affect. 

As such, emergent situations and outcomes regularly occur. Given this context, an 

agent-based capability could improve the rigor of tactical Army wargaming. 

However, 3 points are necessary to support this argument. First, identify a 

comparable baseline of rigor for wargaming and ABM, independently from each 

other. Second, examine the applicability of ABM to wargaming and determine 

whether its application can increase wargaming rigor. Third, determine whether 

application of ABM is possible at the tactical level and if so, what it might look 

like.

Generically, rigor is the quality of being extremely thorough, exhaustive, or 

accurate (Oxford 2016). Alternatively, rigor is a function of the quality of thinking, 

not the quantity (Rand and Rust 2011). High expectations are important and include 

effort on behalf of the learner. Often, deep immersion in the studied topic includes 

the context of real-world settings and working with an expert. Generally, it includes 

depth or care of study, and use of a process, such as the scientific method, to 

evaluate hypotheses. It may also take the form of a thoughtful literary analysis, with 

sufficient depth and attention to detail and accuracy (Department of Instruction 

2011).

Army doctrine writers have embodied and applied rigor throughout the wargaming 

process, while trying to keep it flexible and adaptable to a variety of unanticipated 

scenarios. For example, the MDMP process contextualizes wargaming within its 

larger 7-step process, which tasks and integrates activities of all stakeholders. Staff 

thoughtfully considers future decision points and links them to a synchronization 

matrix; together, both the MDMP process and synchronization efforts balance 

individual warfighting functions. As staff analyzes a given course of action, they 

identify and record strengths and weaknesses. These observations empower staff to 

make unbiased analyses, interpretations, and decisions regarding findings.

However, based on the authors’ observation of Army wargaming, the practicality 

of tactical wargaming is not as rigorous as stated in doctrine. Many times, there is 

limited time available for a proper application of the MDMP and its substep 

wargaming process. The authors understand this is a reasonable and expected 

military condition but its consequences lead to less analytical rigor. In another 

example, doctrine identifies a number of roles and stakeholders who should 
Approved for public release; distribution is unlimited.

41

participate in wargaming. However, in tactical situations, not all identified people 

can participate because some roles do not exist at lower levels of tactical command. 

For example, only the S2, S3, and S5 (i.e., intelligence, operations, and plans, 

respectively) officers may be involved in wargaming. In this case, civil and other 

local population perceptions and biases may not be fully considered. In general, 

doctrine broadly applies rigor to the Army wargaming process, but in certain cases, 

implementation is not realistic to the extent described.

In practice, commanders require 2 COAs to be identified and wargamed: the most 

dangerous and most likely. Unfortunately, 2 scenarios are likely too few options to 

evaluate situations broadly, especially in an interactively complex situation. Using 

a small number of possibilities (i.e., COAs) to make a decision is a common 

decision-making mistake. Similarly, making judgments based on how easily 

information is accessible, or for expected situation outcomes, decreases decisionmaking rigor. People are often overconfident in their beliefs, assuming what they 

believe will happen more often than it actually does (Kass 2002). Together these 

biases increase variance in the rigor that wargaming provides to the commander.

Finally, creating and analyzing courses of actions based on assumed enemy courses 

of action increases the potential for overly risky consequences. As discussed in 

Section 2, there are 2 major methods of developing courses of action. The first is a 

traditional capabilities-based method that uses intelligence to ascertain adversarial 

assets and capabilities. Wargaming helps identify whether identified capabilities 

affect that particular friendly course of action and if so how. The second method is 

to identify adversarial assets, capabilities, and intents and develop courses of action 

that embody adversarial intents (Caffrey 2000). Using the second method, if staff 

identifies adversarial intent correctly, there is a theoretical chance of a better 

friendly outcome. However, if staff chooses incorrectly, the risk to a positive 

friendly outcome can be higher than using the first method. The Army endorsed the 

second method after World War II, once conflict with the Soviet Union became a 

possibility. Staff used their perception of Soviet doctrine as the basis for identifying 

adversary intent. However, nonstate actor doctrine is less formal and may be 

nonexistent; a prescient and decision may not exist, right up to the actual decision 

and subsequent actions occur. Moreover, recent history and actions of both state 

and non-state actors seem to contradict expectations of knowing preaction intent. 

ABM rigor depends on what the modeler is looking for, specifically, in correlation 

with the asked question and nature of the solution desired. For a simple model, the 

modeler calibrates model output to observed data and uses scale normalization to 

estimate an interpretation of model output. Interactions between the models’ agents 

drive output and feedback loops that control the resulting emergent properties

(Heylighen 1999). In a more complicated case, creating a simple model is still the 
Approved for public release; distribution is unlimited.

42

likely starting point. Once an initial model is working as expected, the modeler 

slowly increases the complexity by adding additional agent types, rules, probability 

distributions, and interactions. Complex models are simulated just like simpler 

cases but are compared quantitatively to collected data that helps evaluate whether 

the model represents the real world.

Since modelers can develop a wide range and variety of models, a common 

evaluation framework would help define criteria used to describe the rigor of 

individual models. In the mid-1990s, Drs Rob Axtell and Joshua Epstein proposed 

an evaluation framework to provide such guidance (Axtell and Epstein 1994). The 

evaluation framework graded models at 4 progressively difficult levels of maturity 

and rigor. Level 0 models produce a caricature of reality allowing a user to visualize 

agent motion. A Level 1 model produces qualitative agreement with empirical 

macrostructures and suggested verification by plotting distributional properties of 

the agent population. Level 2 models produce quantitative agreement with 

empirical macrostructures, as established through statistical estimation. Level 3 

models exhibit quantitative agreement with empirical microstructures, as 

determined from cross-sectional and longitudinal analysis of the agent population. 

Using this framework, one can identify the rigor an individual agent-based model 

provides.

The study suggests if a modeler can describe and model human interactions 

abstractly, in a way that is effective for Army wargaming, then using the abstract 

model to evaluate individual circumstances will provide a more rigorous approach 

to enemy perception than evaluation by intuition (Beeker et al. 2010). Consistent 

use of a model is a technique to limit the impact of psychological biases on 

decisions including biases of anchoring, overweighting, and order of evaluation 

(Kass 2002). Moreover, consistent with the definition of rigor, even if a novice is 

using the agent-based capability, an expert has built the utilized model in the first 

place. If the novice is using the model in a realistic setting and the user is working 

with an expert, via the model, then both attributes are consistent with the definition 

of rigor and the overall result is an increase in wargaming rigor.

3.3 Model of the Human Domain

Special Operations Command (SOCOM) and its Army variant, Army Special 

Operations Forces (ARSOF), have identified a fundamental shift in the character 

and nature of warfare that involves a shift toward irregularity, hybridity, and 

diversification of participants and threats. As such, ARSOF may be responsible for 

future hybrid campaigns mixing conventional, unconventional, and cyber that 

could influence operations through relationships with local security and governance
Approved for public release; distribution is unlimited.

43

(Ryan 2014; Raymond 2015). That said, there is a need to understand the local 

population within the context of the operational environment to take appropriate 

action to influence human behavior toward intended outcomes (Coles 2006; Franks 

2013; Brecher 2014). Similarly, the Army needs to understand dynamics of civil 

populations within the context of it missions to be able to effective execute the 

wargaming portion of the MDMP (Joint Chiefs of Staff 2015a).

Human Domain is a concept SOCOM proposed to focus on people as individuals, 

groups, and populations who exercise agency within an area of operation in a way 

that can affect the US, Partner, or adversary interests. It is composed of 5 elements 

that represent attributes of human decision-making and behavior and provide 

insight into the local culture. These 5 elements are social, cultural, physical, 

informational, and psychological, each with subelements specific to their 

designation (Brecher 2014). The Human Domain concept is a work in progress and 

proposes a comprehensive discipline to identify, understand, and influence relevant 

individuals, groups, and populations across the range of military operations. The 

social element describes the structure and relationships among groups and 

institutions, involving competition and efforts to impose interest and perspectives

(MacCalman et al. 2013). The cultural element involves the impact of a society’s 

beliefs, customs, and way of life on people’s behavior. The physical element is 

composed of how the natural and manmade worlds shape individual actor’s 

priorities. The informational element is composed of information sources, 

availability, and substance of data. Finally, the psychological element identifies 

how people evaluate and act upon information, including patterns of judgment and 

reasoning in response to available facts.

To begin improving the rigor of modeling human behavior, the authors believe the 

Human Domain concept is a good starting point. It appropriately identifies 

major-need categories and identifies relevance between military objectives, 

conflict, and the larger contextual perspective of the local population. However, the 

authors believe the Human Domain concept is a structural representation of the 

problem and is more appropriate as a set of requirements rather than a model. To 

model emergent human behavior, instances of agents need to interact with each 

other and their environment.

3.4 Social Cognitive Theory

In the 1980’s, Dr Albert Bandura (1999) proposed an agentic model of human 

psychology and behavior that he termed Social Cognitive Theory (Pajares 2002;

Matei 2010). Since then, he has clarified his theory in a series of papers and books 

for over 30 years. Social Cognitive Theory expresses individual human agency 
Approved for public release; distribution is unlimited.

44

using 3 components: agent state, behavioral patterns, and the environment. Agent 

state embodies motivation, affective cognition, and biological attributes of 

individual actors. Symbolic modeling, adoption of determinants and social network 

diffusion, encompasses behavioral patterns. The environmental component is 

composed of imposed environmental constraints, selected environmental aspects, 

and constructed attributes. The authors suggest this theory is an appropriate 

theoretical starting point to model the Human Domain and maps directly to the 

ABM paradigm.

Figure 12 illustrates the similarity and difference between the Human Domain 

concept and the Social Cognitive Theory. In general, the Social Cognitive Theory 

identifies each of the major Human Domain elements. However, an emphasis of 

several Human Domain attributes includes those at the group or high level of 

aggregation. While Social Cognitive Theory identifies application to agenticgroups, the authors suspect this aspect may need additional investigation and 

specification to model appropriately. Additionally, interaction and behaviors of 

economic activity may also be an area of additional specification. However, in 

general, it appears Social Cognitive Theory is a reasonable starting point and 

abstract model applicable to the Human Domain.

Fig. 12 This figure illustrates Social Cognitive Theory in its triadic reciprocal state. The 

Human Domain concept is compared to and overlaps with Social Cognitive Theory. Both 

theories describe the same concepts, but Social Cognitive Theory offers an agentic form of 

interactive complexity.

3.4.1 Environment

The environmental component is a dynamic representation of the agents’ 

environment and is composed of 3 aspects. The first aspect consists of imposed 

circumstances such as the agent’s social structure, including family, clan or tribe, 
Approved for public release; distribution is unlimited.

45

and government. Imposed physical environment includes conditions such as the 

terrain, communications, and war zones. The second aspect is selected 

environmental aspects including agent associations, activities, and surroundings. 

The agent also selects the social environment and nature of cultural of interactions. 

The third aspect consists of constructed environmental conditions; this includes the 

social environment and created economic or monetary resources.

3.4.2 Behavior

Agent behavior also is composed of 3 aspects. The first aspect is a symbolic model, 

which embodies the knowledge structure, or process models, roles, and strategies 

representing learned processes. Different styles of thinking form these knowledge 

structures; for example, behaviors modeled by other agents, the outcome of 

exploratory activities, and verbal instruction. Bandura suggests learned behavior is 

the result of a cognitive synthesis process resulting from acquired knowledge and 

influences by observations and behavior modeled by other agents.

Adoption of behavioral determinants is the second aspect. The agent’s perception 

of its own self-efficacy to master requisite competencies influences whether the 

agent adopts certain behaviors. Agents adopt behavioral patterns based on 

perceived costs and benefits resulting from such actions. As such, an agent may 

learn a new behavior but may never use it because of perceived impediments. 

Similarly, agents use opportunity derived from behaviors to estimate the cost and 

benefits of actions.

The third subaspect is the diffusion of social networks. Psychosocial factors 

influence diffusion, structural interconnectedness, and interaction of the agents’ 

social network.

3.4.3 Agent State

Agent state is the internal representation and perception of each agent and is 

composed of 4 sub-aspects: motivation, cognition, affective and biological. The 

first aspect is motivation, which is composed of a process consisting of personal 

challenges, then feedback or evaluation of performance, and finally a self-influence 

cycle. Goal setting is a primary influence on motivation and results in a set of 

hierarchically structurally goals. These aspirations are the result of efficacy beliefs, 

which are products of expectancy, resilience, and thresholds to settle for outcomes 

not fully meeting objectives. Motivation also helps regulate activities by using 

forethought or projected future situations. Agents examine plausibility by 

considering likely consequences of actions, setting goals and planning courses of 

action. Agents also evaluate planned courses of action by considering the potential 

to produce desired outcomes while avoiding detrimental outcomes.
Approved for public release; distribution is unlimited.

46

The second part of agent state involves the cognitive aspect of the agent state. It 

guides the reasoning processes by using a combination of conception matching, 

perceived level of skill and translation of conception to actions. The conception 

matching process includes corrective adjustments, standards, and actions. 

Differences in perceived levels of skill both constrain and generate different 

situational conditions. Finally, the conceptualization process includes a feedback 

process that included detecting and correcting mistakes. The cognitive aspect also 

includes enactive experiences, social guidance, and modeling influences.

The third part of agent state is the affective attribute that contributes to expressing 

emotion. An example of an emotion is depression, which is a result of low 

perceived efficacy and unfulfilled aspirations or devaluation of attainments that 

helps weaken the efficacy cycle. Coping efficacy is an ability of the agent to 

transform the threat environment to a benign circumstance. It also helps to lower 

the agent’s stress and anxiety levels. Mitigation is a function of the agent’s coping 

efficacy; if the agent cannot cope, alternative means to deal with the situation 

include relaxing, engaging in activities, receiving assurance from friends and 

family, and think reassuring thoughts. Depressing trains of thought are the 

responses to rejection, loss, failure, or setback. Depression also is a function of the 

agents’ supportive relationships, if a low sense of social efficacy exists. However, 

the agent can overcome depression by modeling affective strategy, demonstrating 

perseverance, positive incentives, or availability of other resource to support 

efficacy.

The fourth part of the agent state is the biological attribute. While Social Cognitive 

Theory identifies a biological component, it does not sufficiently identify constitute 

components. Perhaps a logical application of biological attributes is modeling the 

interconnection between physical skill, agent state, and behavior.

3.5 Applicability of ABM to Wargaming

Army missions range in breath of objectives, from offensive to defensive to civil 

support. At the tactical level, wargaming personnel often focus on maneuvers and 

force-on-force considerations. However, during recent years counterinsurgency 

circumstances have forced personnel to consider the impact of direct interpersonal 

interactions with the local populations. In this section, the study examines 

applicability of ABM to wargaming and determines whether its application could 

increase wargaming rigor.

Wargaming staff and intelligence-analyst colleagues use different forms of analysis 

in their consideration of the battlefield, including the contextual physical and 

human terrain (Hanratty et al. 2014; Herbert 2014) . Intuitive analysis, case-based 
Approved for public release; distribution is unlimited.

47

reasoning, game theoretic methods, and critical factor analysis are examples of 

several analytic forms. Intuitive analysis and informal application of induction is 

among the most popular and commonly used methods of intelligence analysis 

(Folker 2000). Intuition is a feeling or instinct that does not use a demonstrative 

reasoning process, and analysts may not be able to explain their theory with 

available evidence.

Case-based reasoning, or solving new problems based on solutions of similar past 

problems, is another common and intuitive method. However, this method is 

slightly more rigorous in that it is composed of 4 steps. The first step is retrieving

cases relevant to solving the problem. Second is translating solutions from prior 

examples, adapting the former solution to the current problem. Third is testing the 

modified solution to the current problem and revising as appropriate. Fourth, once 

the solution is successful, is storing the resulting solution as a new case to draw 

upon in the future (Kolodner 1992).

Game Theory is a technique that is more rigorous and involves analyses of players’ 

decision-making and choices, which can potentially affect interests of other players 

during conflict or cooperation (Turocy 2002; Price 2003). Complementary to Game 

Theory is a selection methodology, which is applicable to the decision-making 

process involved in selecting a course of action; these are the maximin and minimax 

selection heuristics discussed in Section 2.

Another structured form of analysis is Critical Factors Analysis, otherwise known 

as Center of Gravity and Critical Vulnerabilities developed by Dr Joe Strange 

(Strange 2005; Strange and Iron 2004) in the 1990s. The Army adopted this form 

of analysis at the strategic level and used the term “decisive points” to attack enemy 

Centers of Gravity. Center of Gravity analysis as an extension of traditional 

Clausewitzian war theory to help communicate military objectives and is used to 

communicate the Army operational design and planning process (Rueschhoff and

Dunne 2011). Center Factors Analysis is an analysis of the progression of 

dependencies: 1) Center of Gravity, 2) Critical Capabilities, 3) Critical 

Requirements, and 4) Critical Vulnerability. By analyzing the enemy and its 

Critical Capabilities, one can identify the chain of dependencies from Center of 

Gravity through Critical Vulnerability. If one can destroy a Critical Vulnerability, 

then the enemy will both lose Critical Capabilities and degrade the power of the 

Center of Gravity (Schnaubelt et al. 2014). 

Figure 13 illustrates interaction between Social Cognitive Theory and Center of 

Gravity Analysis; the technique can describe individuals and social groups but in 

this case, the study focuses on the interaction from the perspective of an individual. 

The figure describes an arbitrary Center of Gravity; it could be an agent or another 
Approved for public release; distribution is unlimited.

48

notional item. However, the Critical Capabilities are those behaviors the agent 

possesses. Furthermore, both the agent’s perceived efficacy and the perception of 

the agent’s efficacy of such behavior affects the value and effectiveness of the 

Center of Gravity. Critical Requirements are composed of the agent’s 

environmental factors; some factors are imposed, while the agent chooses and 

constructs others. Both physical resource and notional items, such as social 

networks or monetary resources, are part of the selected and constructed 

environment. Finally, the figure describes an arbitrary Critical Vulnerability but 

could, as an example, overlap with environmental items friendly forces has access 

to.

Fig. 13 This figure illustrates Social Cognitive Theory and the overlap between Risk 

Management and Critical Factor Analysis methodologies. Similar to Fig. 12, each 

methodology overlaps, but Social Cognitive Theory offers a more comprehensive expression 

of the interactive complexity.

Formalized risk assessment is another structured form analysis and approaches 

individual risks or problems from the perspective of plausibility to affect to the 

whole system (DHS 2010). This form of analysis attempts to understand the 

systematic nature of each problem and make an informed assessment as to the 

consequences that could result. Analysts using this method decompose each 

identified risk into 3 primary parts: vulnerability, impact, and threat (Cloppert 

2013). Vulnerability is the exposure to the risk. However, vulnerability is 

changeable and specific to time; stakeholders can reduce vulnerability. Impact is 

the realized result of risk. Threat is a component of risk that is potentially 

discoverable by using intelligence and composed of 3 aspects: adversary 

opportunity, adversary capability, and adversary intent. The adversarial actor 
Approved for public release; distribution is unlimited.

49

controls intent, and is similar to the previous discussion of enemy courses of action. 

Opportunity relates to the timing and knowledge of the adversary. Capability 

describes the ability of the adversary to achieve intended goals and opportunities 

and is analogous to possible enemy capabilities-based courses of actions. In 

addition, adversary skills and resources influence capability.

Figure 13 also illustrates the interconnection between Social Cognitive Theory and 

risk management, specifically the “Threat” section of model, which represents the 

interaction between the specific risk and external agents. External agents are, in this 

case, any agent operating within the simulated agent-based model. Risk opportunity 

models the environmental factors in the simulated model. Agents’ physical 

constrains are limiting factors. For example, access for a physical space or 

possessing appropriate resources to enable access to the identified risk. 

Risk capability is a manifestation of an agent’s behavioral patterns. Efficacy of an 

agent’s skills and learned behaviors act as constraints on the number of agents who 

could plausibly affect the given risk. Lastly, risk intent is modeled by agents’ self 

or mental state.

The intersection of Social Cognitive Theory, Risk Management, and Center of 

Gravity Analysis illustrates the interconnection between the described modes of 

analysis. It identifies that an agent-based model is a generic enough technique so if 

a modeler describes an appropriate model, the technique can notionally represent 

equivalent results as those resulting from structural analysis approaches. However, 

while an equivalent agent-base model could provide analogous result, it does not 

necessarily mean an agent-based model could increase the rigor of wargaming. 

Each of the aforementioned techniques have different emphasizes on interaction 

and plausibility. Similar to Game Theory, ABM represents actors and their 

environment from an interactive perspective. This allows for the case of unexpected 

but emergent results of actions taken. Additionally, these findings suggest an 

appropriate agent base model can represent the emphasis and attributes of each 

structured analysis technique; specifically, the agentic form of Social Cognitive 

Theory model discussed in the previous section.

3.6 Identifying an Increase in Wargaming Rigor

Sensemaking is a technique to examine an arbitrary piece of analysis and determine 

the rigor of a product. David Moore (2011) proposed the idea in his book titled 

Sensemaking: A Structure for an Intelligence Revolution when he examined several

types and instances of intelligence failures along with the current practice and 

challenges of intelligence analysis. In his book, he adopted a series of eight metrics 

proposed by Zelik, Patterson, and Woods who proposed, in their 2007 paper titled 
Approved for public release; distribution is unlimited.

50

“Understanding Rigor in Information Analysis”, that rigor applied to information 

analysis is an assessment of process quality, affording communication about the 

process, rather than the product, of analysis (Zelik et al. 2007). In other words, rigor 

represents a mechanism, which reveals the observability of analysis (Woods et al. 

2002).

The 8 metrics of rigor were Hypothesis Exploration, Information Search, 

Information Validation, Stance Analysis, Sensitivity Analysis, Information 

Synthesis, Specialist Collaboration, and Explanation Critiquing—necessary 

attributes of rigorous analysis. They define each of these 8 metrics as the following

(Zelik et al. 2009):

• Hypothesis Exploration: Constructing and evaluating potential 

explanations for collected data

• Information Search: Focused collection of data bearing upon the analysis 

problem

• Information Validation: Critically evaluating data with respect to the 

degree of agreement among sources

• Stance Analysis: Evaluating collected data to identify the relative 

positions of sources with respect to the broader contextual setting

• Sensitivity Analysis: Evaluating the strength of an analytical assessment 

given possible variations in source reliability and uncertainty

• Information Synthesis: Extent to which an analyst goes beyond simply 

collecting and listing data in putting things together into a cohesive 

assessment

• Specialist Collaboration: Extent to which substantive expertise is 

integrated into an analysis

• Explanation Critiquing: Critically evaluating the analytical reasoning 

process as a whole, rather than in the specific details

In Sensemaking, Moore applied these metrics to evaluate strategic analysis 

products. He compared a National Intelligence Estimate example to 2 other similar 

versions, one academic and the other created with the assistance of a software tool. 

He found no one report satisfied all 8 criteria of rigorous analysis. However, the 

combination of the 3 reports resulted in the most rigorous analysis.

Wargaming is the analysis of several independent courses of action. As such, 

wargaming is an information-based analysis process. Therefore, the 8 attributes of 

analysis rigor apply to wargaming, regardless of whether analysis is strategic, 
Approved for public release; distribution is unlimited.

51

operational, or tactical. Using these criteria, Table 1 illustrates an evaluation of 

whether a hypothetical wargaming situation would benefit from an agent-based 

capability and if so, how might these benefits realized. To apply these attributes, 

the study uses evaluation criteria identified in Zelik, Patterson, and Wood’s (2009) 

paper, “Measuring Attributes of Rigor in Information Analysis”.
Approved for public release; distribution is unlimited.

52

Table 1 The result of using the 8 metrics of rigorous analysis to identify an estimated level of 

rigor in wargaming analysis. The left column is an estimation of wargaming rigor at the 

tactical level, as is currently done. The right-hand column introduces a hypothetical agentbased capability, which in this report is purposely under-specified; however, estimated level 

of rigor is used to illustrate the plausibility of improved rigor and the areas where such a 

capability would be most useful.

Metric of

rigor Current Army wargaming

Estimated value to Army 

wargaming derived from a 

hypothetical agent-based 

modeling capability

Hypothesis 

exploration

Low: there is little consideration of 

alternatives; standard practice is 

identification of most dangerous and 

most likely course of action

Low: little direct value from agentbased capability; alternate value 

could be derived from automated 

reasoning or other automated 

monitoring and alerting

Information

search

Moderate: at the tactical level, time, 

level of access, and connectivity are all 

issues and realistically prevent 

exhaustive searches

Low: both user and reach-back 

support are required to develop and 

find appropriate information

Information 

validation

Low/moderate: general acceptance of 

information with healthy amount of 

consideration as to the trustworthiness 

of reports of the local population

Low: validation of models and 

interaction between actors will help 

improve validation outcomes

Stance

analysis

Low/moderate: incorporates basic 

comparison strategies but includes 

western-society biases in consideration

Moderate: breath of Human 

Domain attributes helps discern 

perspective but is largely dependent 

upon the user

Sensitivity 

analysis

Low/moderate: evaluation of several 

courses of action helps consider 

alternatives and identify boundaries 

through decision points.

High: computationally running 

many orthogonal configurations 

and scenario permutations can help 

the user identify sensitivities and 

decision points

Specialist 

collaboration

Low/moderate: minimal direct 

collaboration with experts but 

interaction of local population draws 

experience from local social networks

High: verified and validated agentbased capabilities had a specialist 

involved during vetting process 

ensuring micro- and macrobehaviors

Information 

synthesis

Low/moderate: depicts events in 

context of 6 warfighting functions but 

lacks efforts to evaluate longer-term 

consequences

High: breadth of considerations are 

involved and described to the user, 

helping provide context and insight 

into consequences

Explanation 

critiquing

Low/moderate: reliance on pre-existing 

modes of critiquing, such as 

supervisory, but alternative views 

incorporated from new personnel 

participation

Moderate: agent-based capability 

helps ensure many perspectives are 

incorporated, availability of 

verified and validated models is 

essential

In reviewing Table 1, several attributes indicate an agent-based capability could 

improve rigor of wargaming; these are rows where the “estimated value” column 

has a higher utility than “current” column. Any single row where the value column 

is greater than the current column is sufficient to identify the possibility of 

improvement. Therefore, this suggests an agent-based capability could increase the 
Approved for public release; distribution is unlimited.

53

rigor of Army wargaming in several aspects. Moreover, value provided from a 

hypothetical agent-based capability is concentrated in analytical areas that are 

among the most difficult for wargaming participants to do themselves, especially 

when significant time constraints or external situational pressure exists.

3.7 Tactical Agent-Based Wargaming

Tactical missions range in variety and breadth, from kinetic to stability to civil 

support operations. Kinetic actions may be necessary in a variety of situations but, 

in and of themselves, may not be sufficient to resolve conflict and fulfill political 

objectives. MG Flynn said, “Simply stated, the lesson of the last decade is that 

failing to understand the human dimension of conflict is too costly in lives, 

resources, and political will for the Nation to bear” (Flynn et al. 2012). Similarly, 

the United Kingdom’s (UK’s) Foreign Secretary David Milliband recently stated, 

“The lesson is that while there are military victories there is never a military 

‘solution’. There’s only military action that creates space for economic and political 

life.” Moreover, in the Human Domain, “tribalism is underneath everything; every 

glance, every knowing look, every payment, every invitation, everything that 

happens is linked to tribal connections” (Stewart 2013). In this section, the study 

examines whether application of agent-based modeling is appropriate at the tactical 

level and, if so, what it might look like (Berzins et al. 1999).

The Army’s experience during the Iraq and Afghanistan conflicts illustrated 

military intelligence personnel are not prepared to read or analyze the cultural 

terrain of the population. As such, without abandoning core warfighting 

capabilities, is it appropriate to view cultural terrain as a necessary parallel 

endeavor (Connable 2009)? Alternatively, does it make sense to use an agent-based 

capability with time constraints imposed by tactical contexts? Would a tactical 

agent-based capability provide value during wargaming, by either providing 

additional insight or by identifying risk-based decision points during identified 

courses of action? Finally, would nonexpert users be able to use and interact with 

the capability?

Recent conflicts, including those in Iraq and Afghanistan and others throughout the 

world, have demonstrated adversaries continue to evolve the form of warfare. An 

example of this evolution is so-called Hybrid Warfare, a technique that employs 

individual aspects of political, military, economic, and criminal tools together in 

such a way that the overall conflict stays below the threshold of conventional war

(Joint Chiefs of Staff 2015b). As this form of warfare changes, it is possible that 

future international relations could be characterized by continuously shifting 

combinations of collaboration, conciliation, confrontation, and conflict (Brecher 
Approved for public release; distribution is unlimited.

54

2014). However, in some published examples of practical application of MDMP, 

staffs demonstrated an inability to use cultural terrain to their advantage, resulting 

in missed opportunities combating insurgencies and terrorist operations (Connable 

2009).

ABM has the potential to improve a spectrum of tasks related to MDMP and 

wargaming. One example includes how using an agent-based capability could help 

understand consequences of the local population’s cultural terrain by helping to 

mitigate some documented failures. In the future, success of upcoming 

deployments in populated areas is likely to depend on both social and combat 

considerations. For example, local biases and decision-making processes are social 

interactions that are likely to be undervalued during traditional wargaming. 

Moreover, local perceptions to Army forces are likely to be different from those 

projected and an agent-based capability could help identify unexpected 

consequences or local-population reactions.

Based on prewar tables of organization and service staffing, there simply were not 

enough trained and experienced Foreign Area Officers (FAOs) to support each 

brigade (Connable 2009). An agent-based capability could act as an independent 

perspective during wargaming, assisting in what would normally be the FAO’s 

responsibility but without saddling the Army with the burden of ensuring enough 

FAOs are available for all parts of the world. Using an agent-based model, acting 

in this type of capacity could help ensure civil inputs, interactions, and 

consequences were available and considered down to brigade-level wargaming.

Alternatively, an agent-based model could assist in population engagement, helping 

to ensure proper interaction with the local people. One engagement with the local 

population that led to successful execution and mission objectives was in Al-Anbar 

where, as early as 2004, Marines who were formerly ill-prepared for cultural terrain 

began to engage with the local tribal, religious, and business leaders (Connable 

2009). They conducted census polling and gathered cultural inputs from patrol 

reports and human-intelligence sources. From this collected information, they

developed specific and local messages meant to engage the population. After 

engaging and interacting with the population for several years, violence within 

Al-Anbar, Iraq was substantially reduced and was eventually returned to Iraqi 

control. Integrating ABM in this process would provide a standard tool for inputting 

that information and providing a model of how the population may respond to 

different threats and interventions.
Approved for public release; distribution is unlimited.

55

3.7.1 Using an Agent-Based Capability during Tactical Wargaming

Usability of an agent-based capability in a tactical environment is a realistic 

concern. To be relevant to a nonexpert user, the agent-based capability would need 

to be simple yet configurable enough to allow a 20-year-old intelligence analyst to 

adjust relative impacts of cultural variables, while operating within the context of a 

friendly COA. Wargaming participants would need to be able to amplify or reduce 

behavioral nuances of the population to explore a number of “what if” scenarios. 

For example, consider the following:

Imagine the US was about to engage peoples in 3 northern Afghan Provinces in 

2010 after ignoring their plight up to that point. No resources have been committed 

to the Northern provinces during the war. Intelligence analysts have learned from 

interrogations, cultural studies, ethnographic studies, lessons-learned from 

Russia's interaction with them during the 1980’s, and open sources that the 

northern Uzbeks and Tajiks have learned not to take sides until resolution is 

imminent. However, the US has reason to believe that Uzbeks and Tajiks are not 

happy about their subjugation to the Quetta and Peshawar Shuras. Even if they 

wanted to fight the US and Afghan government forces, they are limited in the 

ability to collect zakat, recruit suicide bombers, recruit, move, and train IED

(improvised explosive device) emplacers, etc. In addition, the US has reason to 

believe that “Pashtun cruelty is Uzbek kindness.” Scenario planners have dealt 

with the Taliban, Hezb-e Islami Gulbuddin (HIG), and Haqqani enough times to 

begin to understand and predict their reactions to US actions, but they might 

underestimate the ferocity, savagery, and tenacity of the Uzbeks.

In this example, the means by which staff uses ABM during wargaming may be 

different depending on the amount of time available to prepare. The study proposes 

2 methodologies for using an agent-based capability, depending on available time; 

the first is an iterative multistep process whereby staff uses agent-based capabilities 

incrementally to assist in simulation of progressively in-depth and expanded 

questions and scenarios. A second methodology is to use an ensemble of cultural 

models during wargaming in order for a single simulation to help identify a breath 

of possible situations, interactions, and consequences.

In the first methodology, the incremental multistep version, the study identifies the 

steps involved and follows-up with a description of the need and value of each step 

based on the previous scenario; the objective is to use the agent-based tool 

throughout the entire course of action creation and wargaming analysis workflow, 

instead of treating it as a stand-alone tool. To begin, staff would use the agent-based 

tool to examine the culture and interaction tendencies of the local population in the 

designated area of operations. They would use the capability to identify a range of 

possibilities, ranging from action consequences to perceptions of the local 

population, if different actions and scenarios were chosen. For example, if a course 

of action had logistical consequences demonstrating a show of force, what might 
Approved for public release; distribution is unlimited.

56

the perception of the local population be at the time of occurrence versus one week 

later? Alternatively, what might happen if there were an unexpected increase in the 

behavioral savagery in some elements of the population?

Second, once wargaming participants developed possible scenarios, use the 

agent-based capability to help identify what data could collected by the Army or 

another governmental organization to help validate whether predicted courses of 

actions were actually happening, or beginning to happen. For example, if a certain 

type of population-level interaction, such as women and children not observed in 

the streets, could indicate a future decision points, could these data be collected in 

a timely manner to produce actionable insights? Wargaming products, such as the 

synchronization matrix, could change if decision points identified an inflection 

point in circumstances.

Third, once staff identified a range of possible scenarios, use the agent-based 

capability to identify possible friction points and consequences that could 

negatively affect mission objective and success. Friction points could be due to 

physical constraints, such as traveling a route that historically has had more IED 

attacks, or social constraints, such as a decrease in the tenacity of the Uzbeks.

Fourth, once initial wargaming analysis is complete, use the agent-based capability 

to identify a range of alternate courses of action, constrained by initial conditions. 

In these cases, the agent-based capability could suggest alternatives, which 

wargaming participants did not identify because of either cognitive biases or 

context and pressure of the situation. Given appropriate computation power, the 

agent-based capability could model permutations of various culturally, tribal, and 

physiological conditions with the objective of identifying additional mission risk. 

The goal of these permutations is to identify both perceived and actual risks, from 

both parties’ perspectives. Considerations of break-point thresholds, from the 

perspective of both friendly and adversarial forces, are an important consideration 

in this calculation.

Fifth, use the agent-based capability to determine possible, probable, and 

improbable consequences of each course of action. By independently caveating 

each course of action, the commander can make a more informed and rigorous 

choice.

A disadvantage of using the agent-based capability extensively during wargaming 

is a potential increase in needed time. During tactical situations, time will limit the 

ways such a capability may be used; however, the benefits of increased rigor and 

awareness of the cultural terrain are an appropriate contrast with the need to make 

situationally quick decisions (Spencer 2009). In the context of needing to make 

situationally quick decisions, there could be an alternate method for applying an 
Approved for public release; distribution is unlimited.

57

agent-based capability. For example, perhaps it is more useful to use a single-step 

agent-based capability instead of a multistep capability. To model the cultural 

terrain in this case, the agent-based-capability could use an ensemble of cultural 

models to model a wide range of possible interactions with the local population. 

For example, individual agents with the model would have a mixture of cultural 

models, combining characteristics of the local people and geographically and 

socially surrounding cultures. 

A single-step, agent-based capability could reduce the amount of time used by the 

tool while providing a degree of cultural and interaction rigor to the wargaming 

analysis. A disadvantage of this type of modeling is validation. This type of model 

is likely to be hard to verify; interactions, perceptions, biases, and actions may not 

have prior precedent to the simulated scenario. As such, this type of capability could 

provide value as long as staff interpreted output as representative caricatures of how 

the local population may react to proposed courses of actions.

3.7.2 Appropriate Tactical Level for Agent-Based Wargaming

Finally, the study considers the tactical level that is appropriate for an agent-based 

capability. Tactical planning occurs in Army divisions, brigades, and battalions. 

Based on our consideration, the authors believe the brigade is the lowest level that 

staff should use an agent-based capability during wargaming. There are several 

reasons for this conclusion (Barry and Koehler 2005).

First, the brigade is the lowest level of command that can operate independently as 

part of a JTF. It is the lowest level of command that integrates several intelligence 

military occupational specialties, such as all-source, imagery, human intelligence, 

and signals intelligence analysts. Comparatively, battalions and lower tactical 

levels of command do not include these capabilities. Battalions are not capable of 

operating independently and rely heavily on BCTs for guidance and intelligence 

support. Finally, BCTs have just as much intelligence capability as a division and 

the primary different between staff assigned to division and BCTs is rank and 

experience. 

Second, the BCT is a modular organization that provides the land component or 

JTF commander with close-combat capabilities across the full spectrum of conflict

(Greer 2013). BCTs are the Army's tactical-combat power-building blocks for 

maneuver, and they are the smallest combined-arms units that can be committed 

independently. Now and in the future, it is likely a single BCT from any of 10 active 

divisions will deploy as part of a JTF; in the future, the authors believe it is more 

likely for a BCT to be deployed than an entire division.
Approved for public release; distribution is unlimited.

58

Third, when higher headquarters alerts a BCT for deployment or assignment of a 

new mission, its parent division, or JTF staff, is doctrinally responsible for 

providing an Analysis of the Operational Environment (HQ–DA FM 3-90.6 2009). 

However, this analysis is in the context of the operational variables and 

subvariables (e.g., PMESII-PT). The higher headquarters Analysis of the 

Operational Environment may consider aspects of the environment that are too 

broad for the tactical mission tasks of BCT.

According to Army doctrine, the BCT commander and his staff are responsible for 

refining this operational information and variables to its tactical relevance. With 

limited time and analytic capacity, BCT commander focuses his staff on those 

mission and operational sub variables that are mission-relevant. An agent-based 

capability could help staff identify which mission and operational variables will be 

mission-relevant in the end. Incorporating the operational variables into the mission 

analysis would enhance the BCT command and staff’s early understanding of the 

human aspects of the situation; variables such as language, culture, history, 

grievances, education, beliefs, and so forth that traditional mission analysis might 

otherwise not fully consider.

In a hypothetical contingency situation, the division would have approximately one 

week to complete its MDMP, starting from mission receipt and ending with 

production of operations orders and supporting annexes. The next smaller 

command, the brigade, will have approximately two-thirds of that time. The brigade 

would have approximately 4 1/2 days to complete its own MDMP. The same time 

ratio applies to the battalion; it would have two-thirds of the brigade’s time, 

assuming staffs honor the long-standing “one-third–two-thirds” practice, where 

planning at successive levels of command happen in parallel, and higher-level 

headquarters give their subordinate staffs as much time as possible.

Depending on the particular situation, the amount of time used for each step of the 

MDMP varies. In general, staff is not designing a campaign at tactical levels; they 

are nesting and synchronizing tactical courses of action in their area of operations. 

However, the objective is to quantify the amount of time available for wargaming 

and thus identify the limiting amount of time for using an agent-based modeling 

capability. Generally, course of action development will consume the most time 

and wargaming, the next largest amount. As such, the authors estimate the average 

amount of time used by each step as 

• Step 1: Receive the Mission—5%, 

• Step 2: Analyze the Mission/Frame the Problem—10%, 

• Step 3: COA Development—40%, 
Approved for public release; distribution is unlimited.

59

• Step 4: COA Analysis/Wargaming—20%, 

• Step 5: COA Comparison—10%, 

• Step 6: COA Approval—5%, and 

• Step 7: Orders Production—10%. 

Therefore, given these estimates in this particular scenario, a division would have 

approximately 1 1/2 days to wargame all identified courses of actions, a brigade 

would have around 1 day, and a battalion would have a little over half a day. This 

assumes staff is working 24 h a day, once the contingency situation occurs and has 

received their respective higher-level headquarters’ orders. The authors assess one 

day, or use at the brigade, is likely the minimum amount of time a user would need 

for appropriate use.

3.8 Summary

In this section, the report discusses ways an agent-based capability could improve 

the rigor of tactical Army wargaming. The variety and breath of tactical Army 

missions exhibit different cultural, social, and behavioral variables that cause 

emergent situations to occur. These emergent phenomena are the result of 

interactions between actors and their environment. The study identified several 

structured analytical processes that traditionally help improve rigor of analysis and 

demonstrated an agent-based model can embody the objectives that each process 

emphasizes. Then, the study identified how ABM could improve the rigor of 

tactical wargaming. The 8 metrics of rigor help identify areas where ABM has 

potential to improve rigor of Army wargaming practice. Finally, the study identifies 

how ABM could be applied to tactical wargaming, where constraints such as 

limited time and ease of use are essential for operators in the field.
Approved for public release; distribution is unlimited.

60

4. Modeling Human–Computer Decision-Making

This section details the evolution of modeling the human-computing decisionmaking process in 3 subsections: Artificial Intelligence, Decision-Making, and 

Computational Social Science.

Since the 1950s, research in the field of artificial intelligence has illustrated the 

difficulties of computationally representing and modeling the human 

decision-making process. Numerous techniques have been proposed, researched, 

developed, and tested throughout the years. The first subsection details the 

evolution of artificial intelligence and its application to the tactical environment.

The second subsection surveys both the notion of decision-making and several 

decision-making frameworks and discusses how they could assist with 

individualized human decision-making. While decision-making is traditionally 

considered a cognitive human process, computational techniques and frameworks 

are beginning to demonstrate the value of semiautomated decision-making. The 

study therefore also identifies how semiautomated capabilities could potentially 

interact with and augment human decision-making.

The final subsection introduces the concept of computational social science, which 

is a combination of traditional social science and computational processing. 

Highlighted in this section is a computational technique known as agent-based 

modeling (ABM), which can model and realize characterizations of interactions 

within an environment over a simulated period. However, because agent-based 

models can be difficult to conceptualize and create, the authors identify research 

that proposes using design processes to create models simulating interactions. This 

section of the study concludes with a survey of the limitations and challenges of 

ABM.

4.1 Artificial Intelligence

Artificial intelligence is a branch of computer science dealing with simulation of 

intelligent behavior in computers (Merriam-Webster 2016). In 1955, McCarthy et 

al. proposed that artificial intelligence could embody a specification of learning and 

intelligence such that a machine could replicate it (McCarthy et al. 1955; Langley 

2015). Since then, artificial-intelligence research has evolved to consist of a 

multitude of goals and algorithms. Moreover, its application has been involved in 

many interdisciplinary fields of study.

However, since the original artificial-intelligence proposal, the criterion of success 

has been ambiguous. As a proxy, researchers have identified and used tests to 

identify whether developed capabilities actually realize an artificial-intelligence 
Approved for public release; distribution is unlimited.

61

capability. Alan Turing was one of the first people who proposed such a test, which 

involved 3 players: a questioner and 2 answering players (Turing 1950). The 

artificial-intelligence capability would pass the test if the questioner could not 

identify which of the 2 answering players was a computer. Turing suggested that if 

a computer could pass this test, then it would be a thinking agent.

Since then, others have suggested variations to Turing’s original test. Edward 

Feigenbaum suggested that computers are not human brains; instead, artificial 

intelligence should be evaluated by its capabilities. If both a human and an artificial 

intelligence implementation could perform the same acts, then if one agent was 

considered intelligent, the other should be also be considered intelligent (Laufer

2013). Feigenbaum’s test essentially removed Turing’s requirement of casual 

conversation, replacing it with the requirement that a computer could solve 

problems that a human expert could solve (Smith et al. 2006).

Similarly, Nils Nilsson suggested an alternative test that evaluated artificial 

intelligence by means of an “employment test”. Similar to the expert system 

proposed by Feigenbaum, the employment test further specified evaluation of 

artificial-intelligence capabilities in terms of jobs and tasks ordinarily performed 

by humans (Nilsson 2005). As such, Nilsson’s suggestion helped to transfer the 

success criterion from an abstract notion to actual tests. He also summarized the 

broad artificial-intelligence field into notional categories of research, which are 1) 

Sensory–Motor System, 2) Hierarchies of Perception, Representation, and Action, 

3) Predicting and Planning, 4) Learning, 5) Reasoning and Representation, and 6) 

Language. Figure 14 illustrates each of these categories:

Fig. 14 Nilsson’s categorizations of the field of artificial intelligence. The figure also identifies 

ABM as falling within the scope of Predicting and Planning.

The following sections briefly describe and summarize the nature of artificial 

intelligence research in each of Nilsson’s proposed categories.

4.1.1 Sensory–Motor System

The sensory–motor system is the set of physical capabilities associated with 

robotics and includes domains such as visual, tactile, audio, and haptic. Example 

tasks include manipulating physical objects and conceptual concepts, including 

navigation, mapping, and path planning. Navigation includes localization 
Approved for public release; distribution is unlimited.

62

subproblems, such as knowing where the agent is located and knowing where other 

items are located with respect to its current location. Mapping is the ability to learn 

and create an internal spatial map. Path planning is the routing capability that 

identifies motion paths from source to destination locations (Gigras and Gupta 

2012). Learned conceptual knowledge is specific to each agent, who may choose 

to communicate its findings.

4.1.2 Hierarchies of Perception, Representation, and Action

Hierarchies of perception, representation, and action represent respectively 

knowledge abstraction, the process of annotating sensed information, and the 

resulting actions that an agent may take.

Perception is the ability to interpret input, such as data from sensors, which include 

modalities of speech, facial, and object recognition. Robust perception has been a 

challenge for researchers ever since the inception of artificial intelligence. At first, 

researchers tried to develop detailed and complete representations, independent of 

other parts of the system. However, recent studies have emphasized evaluation of 

perceived data within the holistic system to realize the greatest improvement; many 

times, perception is only a single aspect of the overall problem and solution 

(Fitzpatrick 2003).

Representation is commonly associated with a knowledge map consisting of 

objects, properties, and categories, and their interrelations. Several representations 

of perceptions exist, including data structures called frames or semantic networks, 

among others. Representation may also store temporal representations of situations 

and events, such as cause and effect relationships. The relationship between 

perception and representation is primarily complementary, where representation 

embodies the interpreted storage of observations and assertions. Existing 

knowledge bases of context- and domain-specific information can assist with 

annotation of observations (Rajeswari and Prasad 2012). Metadata, or data about 

data, are generally included in the knowledge-representation category.

Actions may be the result of preconceived or reactive analysis, depending on the 

situation and context, but both perception and action components are part of an 

integrated process. For example, several cognitive architectures, such as SOAR 

(state, operator and result) and HiTEC, have demonstrated the utility of 

interconnected processes (Wray and Jones 2004; Haazebroek et al. 2011). With 

either of these architectures, preconceived actions are often the result of a balanced 

analysis, where logic routines have considered the benefits and risks before action. 

However, these architectures have also modelled reactive modes, where a pseudo-
Approved for public release; distribution is unlimited.

63

subconscious methodology improves performance in time-sensitive applications 

but does not consider the implications of the actions.

4.1.3 Predicting and Planning

Predicting and planning capabilities identify a range of possible future states that 

can result from hypothetical actions. From a psychological perspective, predictions 

form the basis of making plans, specifically to achieve goals. Historically, agents 

have assumed planned actions would result in the intended consequences. 

However, these assumptions have evolved. Researchers have begun to consider 

multiagent problems where actors need to observe whether predicted circumstances 

continue to match expectations. Specifically, in cases where multiple actors 

interact, subtle aspects of social intelligence and creativity can influence outcomes.

Emergent behavior is the property and consequence of social systems; the 

interactive nature of groups tends to cause unexpected outcomes. Agent-based 

models are a computational technique that can identify the range of likely possible 

actions (Gilbert 2007). Moreover, the technique can compute a range of possible 

future outcomes (Jennings et al. 1998). Agent-based models are designed by using 

state diagrams that represent transitions between choices. Each transition is 

associated with a distribution, which represents the probabilistic choice of making 

decisions. It is plausible that an artificial-intelligence capability could use an 

agent-based model to assist with planning, to preemptively identify emergent or 

unexpected consequences and identify risks to overall objectives. Based on model 

results, the artificial-intelligence capability could adjust its planned course of action 

appropriately.

4.1.4 Learning

Artificial learning is the ability of algorithms to improve through imitation, 

experience, practice, and education. Two classes of algorithms widely referred to 

as “supervised” and “unsupervised” embody these capabilities; “semi-supervised”

algorithms are a hybrid, a mixture of the 2. Supervised algorithms tend to include 

techniques based on classification, regression, or statistics, whereas unsupervised 

algorithms attempt to infer an unknown structure from data. Regardless of the 

technique used, at some point learned information needs to be represented in the 

aforementioned category—perception, representation, and action hierarchies—for 

subsequent use.

Supervised learning is a technique where an algorithm evaluates provided data and 

the user observes the difference between the expected and computed answers 

(Rojas 1996; Kotsiantis 2007). A feedback process updates algorithmic rules and 

weights depending on the magnitude of the difference. The nature of the feedback 
Approved for public release; distribution is unlimited.

64

process is either corrective or supportive. Classification algorithms are a form of 

inductive learning, where labeled instances create and train the model (Kotsiantis 

2007). For example, a decision-tree classification model is the result of forming 

structured, hierarchical rule sets. Once complete, the decision-tree model classifies 

other similar data into its learned categories. Regression techniques can be as 

simple as a linear model formed by minimizing the least squares error. A Bayesian 

network is a statistical technique in which the probability of classification is 

dependent on conditional interdependencies of features evaluated (Diez and 

Druzdzel 2003; Mooney 2010).

In comparison, several other types of algorithms such as reinforcement, clustering, 

and game-theory-based learning are used for unsupervised learning (Bkassiny, Li, 

and Jayaweera 2012). Reinforcement learning is the result of the way algorithms 

interact with the environment and adjust their own internal weights to improve the 

reproduction of the desired output (Rojas 1996; Ghahramani 2004). For example, 

neural networks exemplify networks of highly interconnected and weighted nodes, 

which produce an arbitrary output from a given set of inputs (Hajek 2005). 

Clustering is a grouping process where subsets somehow reflect the underlying 

structure of the data. However, groups are an artifact of comparison criteria used 

(Moutari 2013). For example, k-means clustering is a method that produces 

k-number of clusters. Game-theory-based learning is a generalized form of 

reinforcement learning. Its primary difference compared to other learning 

algorithms is that the environment is not static. Other agents can sense and interact 

with the environment. The goal of this algorithm is to maximize rewards, both short 

and long term. There has been a significant amount of work done in game theory 

for simple systems, but the dynamic case of multiple adaptive machines remains an 

area of active research.

4.1.5 Reasoning and Representation

Reasoning and Representation is an essential component of artificial intelligence. 

Reasoning is the process of determining information not stated explicitly. 

Traditional forms of logical reasoning include deduction, induction, and abduction. 

Deduction is the inference from the general to the particular. Induction represents 

the inference of the probable from the particular to the general. Abduction is the 

inference from the result to a rule and provides a plausible explanation of 

observations. As demonstrated by Oliver Ray, using these reasoning types together 

can create a powerful artificial-intelligence capability (Ray 2005). 

Additional approaches exist for dealing with uncertain or incomplete information. 

Approaches include Bayesian networks, fuzzy logic, and Dempster–Shafer theory. 

Bayesian networks are directed acyclic graphs that indicate a probabilistic causal 
Approved for public release; distribution is unlimited.

65

structure. Fuzzy logic characterizes relationships between real-world entities and 

specific concepts without explaining the nature of the relationship (Freksa 1994). 

Dempster–Shafer theory is a method that combines evidence and recognizes that 

each fact has a degree of support ranging from full support to no support (Zadeh 

1986).

In addition to these reasoning types, several forms of knowledge representation 

exist, such as schemas, predicates, frames, scripts, situation models, and semantic 

networks (Gow 2003; Crowley 2012). Each type of representation has its utility and 

advantages. Schemas are the oldest form of representing information and developed 

to demonstrate that not all truth was an empirical observation. Predicates use 

relationships to express the structure of entities in structured knowledge form. 

Frames provide a contextual and structured representation for focusing on visual 

interpretation; they use slots (placeholders for entities) and methods (corresponding 

logic to search for entities). Scripts are another data structure used to represent 

sequences of events. Cognitive psychology uses situational models to express 

mental models of human comprehension. Finally, semantic networks are directed 

networks of semantic relations between concepts. For example, an ontology is a 

semantic network that uses known types of objects, relations, and concepts.

4.1.6 Language

Language is a component of artificial intelligence used to communicate among 

other agents, including both human and other artificial agents. For example, natural 

language processing gives machines the ability to read and understand human 

language. Alternatively, machine-to-machine communication uses interfaces to 

convey syntactic and semantic information.

Nilsson’s categories as described previously thus represent the major categories of 

study within the field of artificial intelligence. His “employment test” variant on 

the Turing test, however, is by no means the last word on the subject. Negroponte 

suggests yet another variation of the Turing test, in which he identifies that the 

value of a thinking computer is in its ability to work together with a human, rather 

than just passing a capability test (Smith et al. 2006). Similar to Nilsson’s test, 

Negroponte’s test justifies artificial-intelligence capabilities by their results. 

However, this test reintroduces the human element, evaluating how humancognitive aspects affect whether artificial support actually improves human 

performance. It is this aspect of modeling human cognition that comprises the 

subject of the next section.
Approved for public release; distribution is unlimited.

66

4.2 Decision-Making

Decision-making is a process that chooses a preferred option or a course of actions 

from among a set of alternatives based on given criteria or strategies (Wang and

Ruhe 2007). Yingxu Wang and Guenther Ruhe have identified 2 types of human 

cognitive-decision theories: descriptive and normative. Experimental observation 

and studies form the basis of descriptive theories, whereas rational decision-making 

and behavior comprise normative theories. Additionally, Wang and Ruhe identified 

4 types of cognitive strategies used during decision-making: intuitive, empirical, 

heuristic, and rational. When making decisions, human cognitive processes 

generally use one of the first 2 strategies. Existing rational models do not explain 

these intuitive and empirical types of decisions.

However, when a person is making decisions, heuristic and rational techniques are 

generally used. Heuristics include decisions based upon guiding criteria such as 

principles, ethics, rules of thumb, availability of information, and other anchoring 

biases. In comparison, rational decision-making is the result of either static or 

dynamic decisions. Static decisions are the result of structural or deductive analyses 

of the situation, whereas dynamic decisions involve an estimate of the dynamics of 

multiagent interactions.

Wang and Ruhe identify a generic decision-making tree, which humans use to 

select their decision-making methodology (Fig. 15). Their work concludes that 

while humans may decide to make rational decisions, the selected methodology 

changes depending on the perspective of the situation specific to the time of the 

selection. As might be expected, employing different methodologies often results 

in different conclusions and results. From a tactical wargaming perspective, 

changing methodologies implies variability in the many decisions and conclusions 

identified during wargaming, simply because participants may change which 

decision-making strategies they use. This variability translates into reduced 

wargaming rigor.
Approved for public release; distribution is unlimited.

67

Fig. 15 Wang and Ruhe’s cognitive framework of decision and rational decision-making 

strategies. A decision tree illustrates paths of selected strategies, which depend on the 

individual’s perspective of the situation.

With the advent of artificial-intelligence research, cognitive strategies have become 

augmented with a variety of computationally defined decision-making strategies. 

Many of these techniques have been derived from research in the Operations 

Research field, and they often search for optimal methods of making decisions 

while using multiple attributes to contribute to the decision-making process. 

Several examples of these computational techniques include multiattribute utility 

theory, analytic hierarchy process, outranking methods, and evidential reasoning, 

among others.

The decision-making process itself uses the selected methodology, and both the 

decision-maker and all the stakeholders are ideally involved in the decision-making 

process, to reduce disagreement about the problem’s definition, requirements, 

goals, and criteria (Harris 2012). Janos Fulop summarizes the generic 

decision-making process using the following steps (Fulop 2005):

1. Define the Problem: root causes, limiting assumptions, system and 

organizational boundaries and interfaces, and stakeholder issues

2. Determine Requirements: conditions that any acceptable solution must 

meet

3. Establish Goals: broad statements of intent and desirable values
Approved for public release; distribution is unlimited.

68

4. Identify Alternatives: different approaches for changing the initial 

conditions to the desired conditions

5. Identify Criteria: objective measures of how well each alternative achieves 

stated goals and requirements

6. Select a Decision-making Methodology: the method used to solve the 

decision problem; a strategy is often selected using the aforementioned 

strategy decision tree

7. Evaluate Alternatives Against Criteria: evaluate alternatives using criteria 

and associated weighting and rank alternatives according to value

8. Validate Solutions Against Problem Statement: ensure selected solution 

adequately meets requirements and goals; if the best solution is not 

adequate, repeat the process of identifying alternatives and criteria until an 

appropriate solution is found

4.2.1 Decision-Making Frameworks

Negroponte’s variation of the Turing test mentioned previously includes not only 

the technological artificial-intelligence capability but also its interface with human 

decision-making processes. Depending on the nature of the artificial-intelligence 

capability, either a single person or a group of stakeholders may interact with the 

machine. Several decision-making frameworks have begun to experimentally 

discover the nature of this human–technology interaction. This section reviews 

several of these frameworks and their associated interactions.

4.2.1.1 Data Fusion

Data fusion is one framework used to represent capabilities intended to assist 

human decision-making abilities. The term “data fusion” grew out of work within 

the defense community after the Joint Directors of Laboratories’ (JDL) Data Fusion 

Group published a Data Fusion model. Researchers often realize data-fusion 

capabilities by integrating multiple data sources and knowledge to estimate or 

predict the state of some aspect of the universe (Steinberg et al. 1999).

The original version of the JDL model identified 4 levels of fusion: (Level 1) Object 

Assessment, (Level 2) Situation Assessment, (Level 3) Impact Assessment, and 

(Level 4) Process Refinement. Object Assessment is the estimation and prediction 

of entity states based on observation, which tracks association, continuous state 

estimation, and discrete state estimation. Situation Assessment is the estimate and 

prediction of relations among entities and may include force structure, cross-force 

relations, communications, perceptual influences, and physical context. Impact 
Approved for public release; distribution is unlimited.

69

Assessment is the prediction and estimate of actions and consequences resulting 

from interactions between actors. Process Refinement is the adjustments or changes 

made to observations and processing to support processing objectives.

However, subsequent revisions of the JDL Data Fusion Model included additional 

categories: (Level 0) Sub-Object Data Assessment and (Level 5) User Refinement 

or Cognitive Refinement. Sub-Object Data Assessment is the estimate and 

prediction of the signal or other observable states necessary for data 

characterization and association. User Refinement is the presentation of 

information to the user so they can specify changes to the system. Most recently, 

(Level 6) Mission Management has been added into the Data Fusion model and 

defined as the cost, location, and coordination of sensors and platforms (Blasch

2006). The notional JDL model is shown in Fig. 16.

Fig. 16 A notional JDL Data Fusion Model is composed of 6 levels, which represent abstract 

notions of data representation and interpretation. Levels 4–6 have recently identified 

interaction with external entities, dealing with issues of resource management, human 

interaction, and multiagent coordination.

In addition to each of the 6 fusion levels, the Data Fusion Model identifies the 

notional concept of distinct processing steps taking place within each data-fusion 

level. Figure 17 illustrates each of these steps, which include common referencing, 

data association, and state estimation and prediction.
Approved for public release; distribution is unlimited.

70

Fig. 17 An illustration of the functions that occur at each level and node within each JDL 

Data Fusion level.∗

The first function is common referencing—also known as data alignment—and it 

includes data-preparation processes. Data transformations could be simple or 

complex. A simple example might include a coordinate transformation, whereas a 

difficult example might include converting uncertainty representation (i.e.,

probability and possibility) into a frame. 

The second function is data association, the purpose of which is to assign data to 

the appropriate fusion processes. Three subfunctions comprise data association: 

hypothesis generation, hypothesis evaluation, and hypothesis selection. Hypothesis 

generation is the enumeration of observation sources within the context of the 

problem domain. Hypothesis evaluation is the computation of how close the data 

relate to different hypotheses. Hypothesis selection is an assignment problem, 

sometimes known as the combinatory optimization problem in Operations 

Research.

The third function is state estimation and prediction, the purpose of which is to 

resolve data associated in the prior step and place them into context within the 

fusion level. Both attributive and relational states may exist. Furthermore, aspects 

of each state estimation may be discrete, continuous, or both. For example, the 

result of a Data Fusion, Level 2 state estimation may be the location of object, the 

annotated type of object, or an identified spatial error bounds.

4.2.1.2 Decision-Making Frameworks—Observe, Orient, Decide, and Act (OODA) 

Loop

The OODA loop, as proposed by John Boyd, describes another decision-making 

framework. As commonly referred to, the first 2 steps identify processes that 

perceive, interpret, and project entity statuses. These statuses allow a user to have 

situational awareness and make decisions during the Decide and Act steps. 

However, Boyd best describes the OODA loop as “observe, orient, decide, and act 

 ∗Figure is adapted from “Lesson 2: The JDL Model” (Hall 2008) and “COIC Information Fusion 

Operations and Capabilities: An Initial Assessment” (Del Vecchio et al. 2010).
Approved for public release; distribution is unlimited.

71

more inconspicuously, more quickly, and with more irregularity” (Boyd 1986).

Figure 18 shows Boyd’s diagrammed decision-making model using the following 

illustration (Boyd 1996; Boyd et al. 2010):

Fig. 18 Boyd’s OODA loop used to illustrate the way one can create opportunities to act in 

ways that opponents will see as irregular and disorienting.

Central to the OODA loop is the identification of 5 elements: genetic heritage, 

cultural traditions, previous experience, analysis and synthesis, and new 

information. Boyd states the environment and experience conditions the repertoire 

of opponent psychophysical skills. However, without analysis across a breadth of 

information, insights to develop new actions cannot be created. Boyd suggests that 

the diagram lines identifying Implicit Guidance and Control should be used to guide 

standard activities (Richards 2012). The term “implicit” indicates actions, which 

are executed out of habit or a common understanding of the objective. The lower 

line, identified as Feedback, indicates experimental and thoughtful adjustments to 

existing repertoires. The interactive results exhibited by the environment provide 

guidance for iterative adjustments.

Dr Mica Endsley proposed a similar form of a decision-making framework from an 

information-system perspective (Endsley 1995). Her version kept the last 2 steps of 

the OODA loop (Decide and Act) but adjusted situational awareness to specify 3

steps: 1) Perception of Elements in Current Situation, 2) Comprehension of Current 

Situation, and 3) Projection of Future Status. The model, shown in Fig. 19, also 

identified task and system factors (the upper region of the diagram) and individual 

cognitive factors (the lower region of the diagram) (Paradis et al. 1999).
Approved for public release; distribution is unlimited.

72

Fig. 19 Endsley’s model of situational awareness for dynamic decision-making. While similar 

to Boyd’s OODA loop, it explicitly identifies projections of future status and identified 

individual factors influencing the implementation of actions.

4.2.2 Human and Artificial Intelligence Interaction

The Data Fusion framework appears to correlate to Nilsson’s artificial-intelligence 

category of Hierarchies of Perception, Representation, and Action. Data Fusion 

focuses on sensed modalities and relative contextual knowledge to identify states 

of situations, as best as possible. Once the data-fusion system has identified entities, 

the framework puts emphasis on identifying context around those entities. 

Subsequent JDL Data Fusion Model revisions and the addition of a Level 5 User 

Refinement to the JDL Data Fusion Model have indicated the need for 

artificial-intelligence capabilities to work cooperatively with humans.

Comparatively, both Boyd’s and Endsley’s models identify cognitive aspects in 

their relation to the broader situational context. As such, original versions of these 

2 models correlate with Negroponte’s artificial-intelligence test, where he locates 

the value of artificial intelligence in the success of the capability working together 

with humans.

However, the interaction of an artificial-intelligence capability with a human is not 

trivial (Dillon 1998; Dietvorst et al. 2014; Pfautz et al. 2015). As discussed 

previously, humans use a variety of strategies for decision-making. Moreover, 

Schneiderman (1992) recognized situations where humans and machines have 

different strengths and weaknesses in performing tasks. Algorithms tend to exhibit 

strengths in deductive capabilities, whereas humans tend to have strong 

inductive-reasoning skills. At the same time, current algorithmic capabilities still 

have relatively weak induction capabilities, and humans tend to have difficulty 

cognitively evaluating multiple hypotheses at the same time. A human working 

with an artificial-intelligence capability could benefit the decision-making outcome 
Approved for public release; distribution is unlimited.

73

in a variety of ways. For example, humans identify hypotheses better than a 

computer, but a computer may validate those hypotheses better. As a disadvantage, 

human interaction with a computer may increase the number of problems requiring 

attention (Paradis et al. 1999).

A paper by Breton, Rousseau, and Price suggests that a 3-part model could be useful 

in identifying requirements appropriate for balancing algorithmic and human

capabilities (Breton et al. 2002). In the paper, the authors identify the need for 

Human Factors cognitive engineering, specific to each application. This identifies 

system-level engineering requirements that support and improve the effectiveness 

of interaction. The 3-part model includes the particular task, technology, and 

human. The model recognizes that either deficiencies caused by the machine or the 

human will cause friction within the whole system. For example, an algorithm 

deficiency will cause the human to do extra work to compensate. A tradeoff 

spectrum results, balancing requirements between the artificial-intelligence 

capability and the human. Placement within the spectrum will affect the overall 

human–machine synergy and effectiveness.

In 1999, Endsley developed an automation framework to identify the nature of 

automated capabilities supporting human decision-making. A 5-level framework 

identifies the cognitive level required by an operator. The first level is human 

manual control without any automated assistance. The second level results in a 

decision support system with which users would interact and from which they 

would receive recommendations. The third level is a consensual system, where the 

artificial-intelligence system would implement actions, pending approval of a user. 

The fourth level is an automated system monitored by a user, in which the user 

could veto actions. The fifth level is a fully automated system without any user 

interaction.
Approved for public release; distribution is unlimited.

74

Similarly, the National Highway Traffic Safety Administration recognized and 

identified a method to evaluate how well human and artificial-intelligence 

capabilities are combined (National Highway Traffic Safety Administration 2013; 

Zheng et al. 2013; Li et al. 2014). They identified 5 levels of automated interaction, 

similar to those originally proposed by Endsley, in which different rules or 

regulations would apply. Levels ranged from no automation through full 

automation and were identified as follows: 

• Level 0: No automation, 

• Level 1: Function-specific automation, 

• Level 2: Combined function automation, 

• Level 3: Limited self-driving automation, and 

• Level 4: Full self-driving automation. 

At Level 0, the driver is in complete control and is solely responsible for monitoring 

the roadway. At Level 1, automation involves a specific control function, such as 

electronic stability control or a crash-imminent situation where dynamic brake 

support is used. At Level 2, at least 2 control functions work together, but the 

system can relinquish control without warning. At Level 3, drivers may cede full 

control under certain traffic or environmental conditions. At Level 4, the vehicle 

can perform all functions for an entire trip.

Most comprehensively Kaber and Endsley (2003) developed a 10-level taxonomy 

that better described the spectrum of human–machine interaction. They also 

identified the responsible party and associated roles for monitoring, generating, 

selecting, and implementing functionalities. Tables 2 and 3, respectively, identify 

and summarize each level of automation and the roles assigned to either the human 

or the algorithmic capability.
Approved for public release; distribution is unlimited.

75

Table 2 Endsley and Kaber’s levels of automation, expanding upon Endsley’s original 5

levels of automation

Level of automation Description

(1) Manual control Human performs all tasks (e.g., monitoring, generating options, 

decision-making, implementation)

(2) Action support Human generates and designates action, system assists by performing 

action

(3) Batch processing Human generates and selects tasks, the system executes batches of 

tasks

(4) Shared control Human and system generate options, but the human selects tasks; the 

human and system execute tasks

(5) Decision support System and human generate options, but the human selects tasks; 

system executes tasks

(6) Blended decisionmakingSystem generates and selects options, the human approves; system 

executes tasks

(7) Rigid system System allows human to see and select limited options; system 

executes tasks

(8) Automated decisionmakingSystem generates options; human may suggest options; system selects 

best option and executes tasks

(9) Supervisory control System generates options, selects best option, and executes tasks; 

human monitors system

(10) Full automation System generates options, selects best option, and executes tasks; 

human cannot intervene

Table 3 Endsley and Kaber’s taxonomy for human and algorithmic systems operating in 

dynamic, multitask scenarios

Level of automation Monitoring 

role

Generating 

role Selecting role Implementing role

(1) Manual control Human Human Human Human

(2) Action support Human/system Human Human Human/system

(3) Batch processing Human/system Human Human System

(4) Shared control Human/system Human/system Human Human/system

(5) Decision support Human/system Human/system Human System

(6) Blended decisionmaking Human/system Human/system Human/system System

(7) Rigid system Human/system System Human System

(8) Automated decisionmaking Human/system Human/System System System

(9) Supervisory control Human/system System System System

(10) Full automation System System System System

The Data Fusion community, while primarily focused on the technological 

capability, has proposed several methodologies for describing human–machine 

interaction. These include human-in-the-loop, human-on-the-loop, and humaninside-the-loop (Sudit 2015). Figure 20 illustrates the relative position of the human 

within each said interaction. In terms of Endsley’s levels of automation (LOAs), 
Approved for public release; distribution is unlimited.

76

human-in-the-loop implementations are comparable to LOAs 2–5, human-on-theloop implementations are comparable to LOA 6, and human-inside-the-loop 

implementations are comparable to LOA 8.

Fig. 20 Illustration of each of the 3 proposed human–machine interaction concepts from a 

Data Fusion perspective. In each variation, the point at which the human influences the 

interaction is highlighted.

In human-in-the-loop, the human acts as a “gate” between the Decide and Act 

phases of the OODA loop. For example, “Advanced Chess” is a form of chess that 

embodies this type of interaction (Kasparov 2010). This type of chess game allows 

algorithmic capabilities to assist a human player but requires the human to make 

the final decisions (Drogoul 2007). However, because the computer is assisting the 

human, the human may choose moves that are normally beyond his or her own 

abilities. Interestingly, in some experimental observations cooperation enables the 

human to achieve a performance rating higher than the rating of either the player or 

the standalone algorithmic capability.

Human-on-the-loop switches the roles of the algorithm and the machine. The 

human interacts between the Act and Observe phases of the OODA loop and 

operates in a supervisory role. In this role, the human may veto algorithmic 

decisions and plans. The artificial intelligence algorithm gains a supervised level of 

autonomy while operating within goals specified by the human. For example, 

Hawthorne and Scheidt suggested using swarm behavior algorithms, within 

unmanned vehicles, that implement the mechanics of individual or combined tasks 

while constraining overall action within the human-specified goals (Hawthorne and 

Scheidt 2005). 

Human-inside-the-loop is the third form of interaction design and proposes 

integrating the capabilities of the machine and the human. The human takes on the 

role of an intelligence and cognitive analyst and uses their own knowledge and 

reasoning to prioritize and weight automated decisions. The human assigns weights 

at 2 points, between Observe and Orient and between Orient and Decide. In this 

type of system, maintaining a balance between objectives is necessary. For 
Approved for public release; distribution is unlimited.

77

example, objectives might include minimizing risks, minimizing deviations from

expected outcomes and costs, and maximizing reliability (Ganapathy 2006).

The methods and frameworks described are important representations of 

semiautomated decision-making. However, a majority of these techniques refer to 

individualized entities. The next section investigates the difference between 

individual decision-making and collective decision-making through computational 

social science, specifically targeting the emergent phenomena that result from the 

interactive nature of individual decision-makers.

4.3 Computational Social Science

Computational social science is an emerging field of study that leverages the 

modern capacity to collect and analyze data with unprecedented breadth, depth, and 

scale (Lazer et al. 2009). Similar to complex adaptive system modeling, 

technological integration measures social interactions and has enabled research 

toward a quantitative understanding of complex social systems (Conte et al. 2012). 

As such, computational social science is the intersection and mixing of social

science, computer science, and statistics.

Computational social-science researchers seek to study and identify attributes of 

self-organization, emergence of collective phenomena and behaviors, and 

spontaneous order; such research is thus of great use to increase our understanding 

of collective decision-making (Fromm 2006). Emergence is the property that 

results from the interactions of agents within a system. In computational social 

science, there are 3 levels of emergence: the social behavioral level (microlevel), 

the social aggregate level (macrolevel), and the institutional level.

4.3.1 Emergence at the Social Behavioral Level

At the microlevel, interactions of heterogeneous agents identify social complexity. 

Heterogeneous agents are composed of multiple classes of virtual agents that 

emulate people, and their decisions and actions produce the interactive effect 

(Axtell 2011). Some amount of rationality limits the bounds of decisions each agent 

class makes. Researchers model interactions computationally, often using game 

theoretic principles or heuristic ABM techniques (Singer et al. 2008). Other 

evolutionary methods, such as genetic algorithms and various forms of game 

theory, are alternative techniques modelers can use (Stanford Encyclopedia of 

Philosophy 2009).

One microlevel area of computational social-science research is modeling 

individual behavior. While in some cases, this topic is technically possible because 

many personal actions can be observed and stored (Aviv et al. 2010); ethics and 
Approved for public release; distribution is unlimited.

78

privacy concerns limit such work (Eloranta 2015). For example, how friends 

influence each other’s purchasing behavior is an interesting question to social 

scientists. The same question also interests marketers and policymakers, each for 

their own reasons. Other questions of interest include inferring intent from search 

data and inferring relationships from network data (Watts 2013). Additionally, 

questions of individualistic social behaviors such as altruism and cooperation are 

concerned with whether cooperative behavior emerges and whether it can improve 

performance of collective tasks (Conte et al. 2012). Individual and agentic cognitive 

modeling research is another area of interest in modeling social behavior (Bandura 

2002).

4.3.2 Emergence at the Social Aggregate Level

At the macrolevel, social systems are multilevel. Many layers of reproducible and 

heterogeneous systems may exist (Axtell 2006; Camus et al. 2013; Carley 2013; 

Makowsky and Smaldino 2014). However, the difficultly is that one cannot infer 

the properties of the microagents from results that emerge at the macrolevel (Cordi 

2014). The same is true for macro results; microlevel specifications do not 

necessary indicate macrolevel results. However, one can characterize the 

macrosystem statistically but one will not know the exact outcome of the system, 

at a given point in time.

A macrolevel area of computational social-science research is identifying the way 

groups of individuals interact, such as modeling how a disease spreads among 

different populations and regions. In this example, the result tends to look like an 

S-curve, where the initial spread of the disease is slow, infection speeds up rapidly, 

and finally slows down until it dies out (Schelling 2011). Other social phenomena 

exhibit similar response-caricatures, as in the production and adoption of 

manufactured products. Other related research questions include how spontaneous 

groups form and when the emergence of coalitions begins (Conte et al. 2012).

4.3.3 Emergence at the Institutional Level

Social organization and the modern state exhibit an institutional form of emergence

(Scheutz et al. 2005). Game theoretic rationality theory has provided a basis for 

modeling emerging conventions and social norms (Bueno de Mesquita 2010). 

Axtell has performed research in modeling the emergence of firms in a population 

of agents (Axtell 1999, 2014, 2016). Other related topics of research include 

modeling governance and modeling nationwide economies (Pike and Brown 2011; 

Richardson et al. 2015; Suslov et al. 2015).
Approved for public release; distribution is unlimited.

79

4.3.4 Creating Computational Social-Science Models

The objective of computational social-science models may be either quantitative or 

qualitative, depending on how the modeler plans to use the results. Qualitative 

models may produce a caricature or reveal the nature of the actual observed data. 

Quantitative models statistically replicate nuances observed in the population. 

Axtell and Epstein address the breadth of these differences by proposing an 

evaluative framework to help communicate which type of model to use, and what 

its output specificity should be. The next part reviews this model in more depth (see 

Section 5). Both quantitative and qualitative objectives may be useful in modeling 

decision-making.

Another area of study within computational social science is the modeling of the 

mind, which includes beliefs, perceptions, desires, intentions, value, and processes. 

These types of models are applied representations of real-world populations and 

rely on cognitive science and social-psychology factors. Actors within these models 

make decisions with some degree of uncertainty. The transitions between different 

states are modeled stochastically, using rules with attached probability 

distributions.

One of the most applicable types of computational social science is a model of 

interactive behavior known as ABM.

4.3.5 Agent-Based Modeling

ABM is a technique used to implement computational social science theories. 

Modelers begin with the assumption that they can model aspects or attributes of 

arbitrarily intended behavior and its associated interactions (Macal and North

2006). They also assume that the involved social processes can be described, and 

they can create them from a combination of elemental components and interactions. 

In practice, modelers can emulate social phenomena replicating individualized 

decision-making processes, groups, organizations, and swarms of entities 

interacting together within their environment.

As discussed in Section 3, within this type of model each agent is a discrete and 

autonomous entity with its own goals and behaviors (Adra and McMinn 2010). 

Like people, agents are diverse and heterogeneous. As such, the nature of decision 

rules varies by the type of agent, based on, for example, the sophistication of the 

individual agent’s rules (Axtell, Andrews, and Small 2002). Moreover, the 

cognitive ability of agents varies. In simple cases, if-then-else rules may guide 

decisions, depending on conditions. In complicated behavior, if-then-else rules may 

have probability distributions attached to them, which cause the election of rules to 

exhibit stochastic properties. As this model is developed into the future, individual 
Approved for public release; distribution is unlimited.

80

agents may eventually use cognitive decision-making models, such as those 

demonstrated by ACT-R (adaptive control of thought–rational; [Anderson et al. 

2004]) or SOAR (Axtell 2015).

∗ Future agents may also exhibit the ability to retain 

memory, which would allow agents to recall learned experience and use memories 

as input to their decision-making process (Kope et al. 2013; Ezhov and Terentyeva

2014). Future agents may also employ models of internal perceptions of the external 

world, causing agents to make decisions based on their perceptions of other actors 

and on the environment, rather than using strict comparisons and evaluations.

Modelers use relatively simple rules to create behaviors which, when instantiated 

within a population of agents, create the illusion of a globally organized flocking 

behavior. Such rules can successfully model a group of agents that exhibits apparent 

cohesion-, separation- or alignment-based rules but does not possess overall 

command and control facilities. In the case of flocking behavior, agent-embodied 

cohesion rules change the direction of individual agents toward the average position 

of viewable neighbors. Similarly, separation rules adjust an agent’s position to 

avoid crowding, as perceived by localized neighboring agents. Finally, alignment 

rules adjust the direction of the agent toward the average heading of sensed 

neighbors. The aggregate result of these decentralized rules is the emergence of an 

organized group behavior without a central authority or controller.

In addition to agentic interactions and resulting emergent behaviors, agents also 

interact with the environment. In simple spatial cases, Euclidean 2- or 3-

dimensional (2-D, 3-D) space is used. However, topologies that are more 

complicated may be used instead, such as grids, asymmetric or directional 

networks, and Geographical Information System (GIS)-based systems. Agents may 

use the environment as a “blackboard” communication tool. For example, a 

foraging ant uses the environment to place pheromones, which are subsequently 

detected and used by other ants to identify popularity of recently used routes. Thus, 

a group of ants with no central controller will eventually identify the shortest path 

solution between their nest and a food source.

Similar to other algorithmic capabilities, ABM excels at some applications but not 

as well in others. There are 4 conditions that indicate whether the application of 

agent-based modeling to a designated problem is appropriate: 1) natural 

representations of agents describe the problem; 2) the past is not a useful predictor 

of the future; 3) if the model needs to scale to an arbitrary number of agents; and 4) 

if the scenario needs to exhibit the possibility of unexpected or emergent 

phenomena (Macal and North 2006, 2008).

 ∗R. Axtell presentation at Computational Social Science Society of the Americas 2015 Conference
Approved for public release; distribution is unlimited.

81

ABM exhibits the natural representations of agents when the following criteria are 

met:

• The modeler can identify and define discrete decisions and/or behaviors.

• Agents adapt and change behavior or exhibit objective-based behaviors.

• Agents have transient relationships, interactions, and group memberships.

• Agent behaviors incorporate spatial components into their behaviors and 

interactions.

The application of ABM illustrates a transition from traditionally deductive 

decomposition and composition of system components to the composition of 

individual entities that interact to produce a resultant phenomenon. As illustrated 

in Table 4, Dr Axtell details this progression well in his comparison of traditional 

modeling practices and its results with the newer computational social-science 

modeling field, its associated ABM, and its more expressive results (Axtell 2006b, 

2011, 2015):

Table 4 Axtell’s comparison of traditional modeling methods, techniques, and processes with 

newer, computational social-science techniques

Simple (20th century) Complex (21st century)

Single decision-maker Multiple agents

Scalar value function, first-order conditions, 

numerical solutions

Heterogeneous utilities, purposive behavior, 

agent models and simulation

Decision theory Game theory

Mean field, averages Networks, extremes

Schema, taxonomy Ontology

Equilibrium, fixed-point theorems Volatility, adaption, coevolution

Continuous, smooth mathematics Discrete mathematics, computation

Command and control Bottom-up emergence

4.3.6 Process for Implementing Agent-Based Models

To employ such agent-based models for decision-making processes, it is necessary 

to understand the theoretical process of creating these models, which requires a 

combination of 3 perspectives (Conte et al. 2012). 

The first perspective required is to understand and create a model that emulates the 

necessary behavior (Bozkurt 2015). Once the model is working properly, it should 

characterize the nature of plausible results. Models provide insight into the 

representative processes and interactions of systems. A common use of models is 

to illustrate how systems work, serving as an example that explains a particular 

behavior (Schieritz and Grobler 2002). The model may be abstract or precise but 
Approved for public release; distribution is unlimited.

82

should illustrate how states evolve within a given system. Moreover, it is common 

for the logic of the model to remain constant while properties change.

The second perspective is to understand how to perform qualitative analysis, 

evaluating results of generative science and models that create hypotheses distinct 

from results produced by traditional deductive and inductive techniques. Modelers 

need a better method than arbitrarily selecting rules and seeing if they expect 

models to work properly (Conte et al. 2012). Currently, creations of modeling and 

simulation processes are the result of expert intuition and are therefore a 

challenging task for novices, in part because generative science and qualitative 

analysis is similar to abductive logic (Klugl and Karlsson 2009). ABM faces a few 

challenges with regard to this perspective: how to avoid creation and testing of 

ad hoc rules and how to construct plausible models with conditions appropriate to 

create valid and desired output. The combination of the agents, their interactions, 

and the environment create emergent behavior and “grow” regularly observed 

macroscopic properties.

The third perspective is to understand methodologies and techniques for the 

integration, verification, and validation of heterogeneous models. The combination 

of each of these perspectives will enable a modeler to create valid models using 

ABM techniques. Nevertheless, challenges remain, such as collecting appropriate 

and necessary data and addressing the questionable science of a one-time test.

4.3.7 ABM and Design Patterns

Some of the challenges described within these 3 perspectives are similar to those 

encountered in computer science. For example, in software development, designers 

use a best-practice technique called design patterns to help make software design 

successful. However, some research has indicated that object-oriented patterns are 

not directly transferable to agent-based design patterns. Among other reasons, this 

may be due to differences in implementation languages and to differences in the 

level of abstraction (Oluyomi 2006; Klugl and Karlsson 2009; Klugl and Bazzan

2012). Moreover, the concept and use of individual agents often changes as model 

development progresses.

However, other research suggests agent-based design patterns are reasonable but 

acknowledge that the notion of model reusability is different from traditional 

software patterns. For instance, Ayodele Oluyomi suggests that the value of an 

agent-based design pattern is a function of proper definition and rigor, which 

includes attributes of the completeness of the model specification, clearly 

communicated understanding, identification of trade-offs, and appropriate 

application. He concludes that a 4-step process could be used as the basis of abstract 
Approved for public release; distribution is unlimited.

83

model design: 1) agent-oriented analysis, 2) multiagent system architecture, 3) 

agent internal architecture, and 4) multiagent system realization. Michael Weiss’s 

(2003) research influenced this proposal, suggesting that major components of a 

design pattern included context, problem statement, acting forces, a solution, and 

problem context.

Figure 21 is an adapted representation of Oluyomi’s proposed design and modeling 

process:

Fig. 21 An adapted representation of Oluyomi’s proposed design and modeling process

4.3.7.1 Step 1: Agent-Oriented Analysis

The first aspect of model creation begins with an analysis of the model 

requirements, such as identifying goals, subgoals, and quality metrics used to 

evaluate model objectives.

Second, the model designer identifies all roles within the model. Roles characterize 

and define system-level roles, such as external human and software systems. Roles 

are not unique to individual agents; multiple agents may express a single role, or a 

single agent may express multiple roles.

Third, the model designer characterizes and organizes groups of agents expressing 

similar roles. This is a structural grouping of the roles contained within the model.

Fourth, the model designer identifies role interactions. Results of the initial goal 

analysis ensure that role interactions will reflect the relationships identified. 

Role-to-role relationships might exhibit collaborative, controlling, or competitive 

properties. In the case of a collaborative relationship, for example, a set of roles 

would work together to realize a common goal necessary for the model.

4.3.7.2 Step 2: Multiagent System Architecture

This design step begins with a specification of agents who represent roles within 

the model. Roles describe behaviors and interactions and are translated and 

extracted into agents. 
Approved for public release; distribution is unlimited.

84

Second, once agents are extract, the model designer tests the defined relationships 

and interactions to ensure that the expected interactions and behaviors between 

agents result from the model.

Third, the model designer introduces traditional software-engineering requirements 

with the system-design step. External data and systems may be accessed in this 

step. Additionally, the model designer specifies interfaces with external systems.

4.3.7.3 Step 3: Agent Internal Architecture

The internal agent step identifies where notions of autonomy, proactive behavior, 

reactivity, and social behavior are represented within internal agent logic. 

Depending on the roles agents need to exhibit, different property values may 

increase certain types of behaviors. Sample types of agents may include assistant, 

informational, collaborative, or mobile agents; the agent type determines individual 

architectures and how internal components are structured. Individual types of 

architecture include deliberative, reactive, and hybrid.

4.3.7.4 Step 4: Multiagent System Realization

This final step incorporates the prior 3 steps of design into an implementation. The 

model developer outlines development specifications and selects various 

considerations, such as implementation environment and programming language.

Second, implementation realizes the specified model. Integrated development 

environments may be used to assist the realization of the model. The model 

developer deals with implementation considerations during this step, such as 

ensuring the simulation runs fast enough.

Verification and validation is the final step, which involves examination of both the 

structural and interactive components of the model and proper output. Ideally, this 

step uses an iterative experimental design and testing.

4.3.8 Types and Attributes of Agent-Based Model Design Patterns

Using the aforementioned ABM flowchart design methodology, Oluyomi identifies 

the design process for defining and instantiating an agent-based model. However, 

there are a number of attributes included within each step. Oluyomi recognized 

these differences and identified the following observations in his research.

Figure 22 identifies each of the 4 steps in the left-most column, highlighted in bold. 

Next to each step are categories, describing the nature of the attributes that should 

be considered by the model designer. For example, the first row identifies the 

Agent-Oriented Analysis step, which identifies organizational and interactional 
Approved for public release; distribution is unlimited.

85

categories of attributes. The organizational category describes real-life 

observations and goals for the model, and these attributes act as a way to structure 

components and interactions used in the model. In comparison, the interactional 

category describes the type and nature of the interactions between agents and how 

agents interact with each other. 

Fig. 22 A listing of the attributes considered while creating and agent-based model based on 

Oluyomi’s proposed design and modeling process

Each of the 4 design steps and categorical designations identify attributes 

influencing model considerations. For example, Agent-Oriented Analysis includes 

“interaction objective”, which relates to the purpose of an interaction within a 

virtual organization. In another example, Multi-Agent System Architecture 

identifies “interactional categorization” attributes of “control flow”, “introduced 

agents”, “message exchange rules”, “message format,” and “ontology” that are 

applicable to this part of model design. Oluyomi goes on to identify and define each 

category and attribute in his research (2006).

Using a combination of the 4-step design process and the categorization of 

attributes relevant to each step, Oluyomi reviews 97 agent-oriented design patterns 

studied by 15 authors. Oluyomi selects 28 of those patterns, using an attribute-based 

analysis, and identifies their uses and applicability within his proposed design 

process. Figure 23 identifies and summarizes the results of the analysis and uses 

the same layout as Fig. 22, thus identifying design patterns applicable to categories 

within the design process (Oluyomi 2006).
Approved for public release; distribution is unlimited.

86

Fig. 23 Oluyomi’s list of identified design patterns categorized by their type of interaction 

and stage within the agent-based, model-creation-process model

4.3.9 Evaluating Agent-Based Models

Epstein identified motivations for creating models, such as explaining phenomena, 

guiding data collection, revealing dynamic analogies, discovering new questions, 

identifying central uncertainties, and providing insight into decision tradeoffs 

(Epstein 2008). Models can confirm whether macro-level observations are possible 

and explain the interactional phenomena, from which those observations result. 

However, once created, the modeler tests the model to determine whether it is 

operating as expected.

Paul Borrill and Leigh Tesfatsion (2010) proposed an 8-step process for testing and 

evaluating agent-based models. In their experience, they found researchers 

typically repeat the same process for experiments:

1. Develop an experimental design for the systematic exploration of a 

theoretical issue of interest

2. Construct a computer world, or a culture dish, consisting of a collection of 

constituent agents appropriate for the study of the theoretical issue

3. Configure the computer world in accordance with the experimental design

4. Compile and run the computer world with no external interference and 

record outcomes of interest
Approved for public release; distribution is unlimited.

87

5. Repeat the same experiment multiple times, using different 

random-number-generator seed values, to generate an ensemble of runs to 

derive a distribution of world outcomes 

6. Repeatedly iterate Steps 3–5 until the full range of configurations, specified 

under the experimental design, has been explored

7. Analyze the sample distributions for world outcomes and summarize 

theoretical implications

8. Use theoretical summaries to form hypotheses that can be potentially 

validated with historical or real-time data

Using this procedure, the modeler evaluates performance of the model by collecting 

data to evaluate relevance, simulating and validating performance, and performing 

a sensitivity analysis.

Hypothetical models test whether conditions are likely to produce expected 

phenomena; if so, they provide justification that a specific type of data or unknown 

phenomena may exist. However, collection of the appropriate data, which is 

necessary to validate models, can be difficult, especially when modeling individual 

human decision-making. Furthermore, the nature of collected data may affect how 

well the model can represent a situation. Referring to Axtell and Epstein’s concept 

of “maturity level,” if a model is designated and verified as maturity Level 4 (using 

Axtell and Epstein’s Empirical Relevance Framework, Section 5) but precise data 

on constituent actors, modelled within the simulation, is not available, then the 

output of the model is not likely to be valid. 

Data collection includes collecting model-produced data. Step 7 of the testing 

procedure identifies an analysis of macro- and micro-level characteristics, verified 

by simulated data. Depending on the modeler’s needs, collected data may enable a 

statistical analysis of the system’s characteristics. However, if the modeler’s 

objective is to evaluate individual agents within the model, such as how they make 

decisions and how they interact with other agents, then more in-depth observation 

may be needed.

Simulation helps the modeler understand the consequences of agent interaction, 

while providing data for validation. Experimental design, identified in step one of 

the testing procedure, and the parameters used for a simulation may result in wideranging difference in the outcome of simulation results. For example, differences 

in spatial perception distances may result in widely varying results. Alternatively, 

the number of agents used within a simulation is another subtle variable, especially 

if the modeled phenomenon requires a minimum required number of agents and 

interactions to emerge. Thus, model validation is an essential aspect of the 
Approved for public release; distribution is unlimited.

88

simulation process (Duong 2011). Ideally, the best-case scenario during validation 

is to use model parameters that have identifiable real-world meanings, as this allows 

a modeler to use and test reasonable and understandable ranges of values.

Finally, the evaluation process considers sensitivity of parameter variables. 

Sensitivity helps determine model robustness. Ideally, the modeler should consider 

and discover sensitivity by testing ranges of parameter values, as identified in step 

6 of the testing procedure (Cioppa and Lucas 2007; Chandola et al. 2009; Viana 

2013). Moreover, the modeler should consider the sensitivity of the model design 

itself, for example, as to whether a minor change in the model leads to large changes 

in the overall output. This is because small amounts of noise may actually 

contribute to the formation of unintended and emergent phenomena. As such, 

before the modeler characterizes simulation output, the simulation should to be run 

enough times for a statistical characterization. Each time, the initialization value of 

the random number generator should be changed so the effect of noise is 

constrained within the context of many simulation trials.

Once the modeler computes and analyzes tests, they become useful to other 

researchers if presented properly, ideally in a form that allows other scientists and 

researchers to reproduce results. For example, by distributing source code and/or 

pseudocode along with initial and boundary conditions, the kind of interaction 

network used, and model parameters. In addition, results of a variance analysis help 

convey intramodel dependence of parameters.

4.3.10 Limitations and Challenges of ABM

While ABM has the potential to model and provide insight into a variety of 

scenarios, it also has limitations. One limitation is in how the technique is viewed; 

namely, it is not a means to control social systems. Instead, Dirk Helbing states that 

agent-based modeling is a robust approach, which provides a way to transition from 

a “regulatory approach” to a mechanism that models the system (Helbing 2012). 

He suggests that regulations often correspond to changing boundary conditions but 

that agent-based modeling is a systems approach that models interactions in a way 

that reduces instabilities.

Axtell (2000, 2011) identifies other limitations, described as the inability to create 

high-fidelity models; predictions of point outcomes; small “n” forecasting; the 

quantitative comparison of tactics; and long-run prognostication (Turnley et al. 

2012; Frank 2012, 2015).

• High-fidelity models describe model minutia and grow the simulated 

environment from small parts and their interactions. High-fidelity models 
Approved for public release; distribution is unlimited.

89

compare to the common approach of limiting model scope to the specific 

question asked.

• Prediction of point outcomes identifies the limitation of forecasting a 

specific event occurring at a specific point (e.g., in space and time). Instead, 

agent-based modeling is used to characterize the range of possibilities of 

such an event occurring. 

• Small “n” forecasting describes the problem of running simulations where 

there are not enough data available to verify and validate the model results.

• The quantitative comparison of tactics identifies the inability to compare 

and contrast irregular-warfare tactics with the models available presently.

• Long-run prognostication identifies the difficulty of identifying the future 

state of situations because of the cumulative effect of small perturbations to 

the system over time.

Another challenge to creating valid models is their dependence on the availability 

of appropriate data. A common approach during model development is to limit 

model creation to the scope of the question asked. This bears out Occam’s razor

that a model should be as simple as possible but not simpler (Gigerenzer and Todd 

1999; Axtell and Young 2004; Guerin and Kunkle 2004). However, depending on 

the intended model use, models are ideally evaluated quantitatively at both microand macrolevels. An accurate representation of both these levels, however, 

generally indicates a large amount of specific data. Therefore, while the potential 

of agent-based models is high, a careful consideration and management of 

expectations is appropriate to balance its value.

A future area of development in ABM will focus on modeling behavioral and 

cognitive interactions, such as social process models like tribal dynamics, which 

are currently difficult to model. In the future, challenges will continue to arise as 

methods, tools, ethics, and privacy concerns influence the collection and testing of 

such models. Additional challenges exist in facilitating cooperation between 

complementary fields of study, notably the social sciences and computer science. 

Yet, there are few defined and managed educational paths for computational social 

scientists.
Approved for public release; distribution is unlimited.

90

5. Existing Research and Applied Development of ABM

ABM capabilities have been developed and integrated into several programs and 

tools. This section aims to provide an overview of the state-of-the-art tools for 

ABM, surveying existing defense capabilities that incorporate agent-based 

techniques, reviewing the evaluative criteria for determining research maturity 

level and introducing the concept and components of the Human Domain, and 

surveying published ABM research with respect to the Human Domain.

5.1 ABM in Existing Defense Capabilities

During tactical decision-making, some decisions consider the results of actions, 

which are kinetic, while others are nonkinetic or observational in nature. Kinetic 

courses of action include the employment of individual weapons systems, force 

maneuvers, fires, and analyses of weapon effects. Observational and other 

nonkinetic activities include intelligence, surveillance and reconnaissance, 

engineering operations, and logistics and sustainment. Moreover, in an irregular 

warfare or urban context, nonkinetic courses of actions include analysis of 

information operations and civil operations (Frewer 2007).

Analysts evaluate both kinetic and nonkinetic courses of actions using a 

combination of methodologies, which might be broadly considered as physics- and 

social-science-based methodologies. Physics-based methodologies depend on a 

well-defined mathematical analysis of variables such as terrain features, weapons 

capabilities, and the numbers of personnel and equipment. Social-science 

capabilities may use mathematical methods and variables, but in these 

methodologies values are qualitative rather than quantitative. For example, socialscience analysis might base its computation on the statistical interpretation of 

surveys, on the cultural analysis of experts, or on organizational dynamic analysis. 

Figure 24 illustrates the differences between the earlier types of analytical 

methodologies and objectives.

The combination of kinetic action and physics-based analysis, shown in the 

top-right quadrant of Fig. 24, plays an important part in helping commanders 

evaluate COAs, which concern executing offensive operations. Similarly, analysts 

combine kinetic action and social-science analysis to identify and model secondand third-order consequences and effects of actions. Such an analysis may consider 

the population’s reaction to the removal of a target and to the methods used. 
Approved for public release; distribution is unlimited.

91

Fig. 24 A quad chart illustrating 4 quadrants of tactical analysis (Connable et al. 2014, p. 23).

This study’s emphasis on applications of ABM to tactical wargaming thus limits its scope to 

the bottom-left quadrant, “Non-Kinetic/Social Science”.

Combining nonkinetic and physics-based analysis helps in the allocation of 

intelligence, surveillance, and reconnaissance assets, the prioritization of 

engineering activities, and the identification of risk for logistics and sustainment. 

Finally, analysts evaluate the results of combining nonkinetic and social-science 

analysis with respect to the design of information operations, the coordination of 

whole-of-government approaches, and the communication of cultural 

interpretation. They typically communicate these results using frameworks such as 

PMESII-PT, DIME, and ASCOPE in an ad hoc manner (Counterinsurgency 

Training Center–Afghanistan 2010, 2011). However, “military professionals 

describe this volatile mix of factors (i.e., the variables within each of these 3 

frameworks) as being ambiguous, complex, uncertain, and ill-structured. When 

trouble appears, there is no consensus about what the fundamental problems are, 

how to solve them, what the desired ‘end state’ should be, and whether an ‘end 

state’ is achievable or not” (Connable et al. 2014, p. 29). This suggests that there is 

still a need to understand how variables interact and understand their possible 

consequences (Perez 2011).

5.1.1 Tools and Capabilities

This section surveys state-of-the-art tools and capabilities identified as embodying 

an agent-based capability or other similar type of capability. These tools emphasize 

the nonkinetic and social-science quadrant identified in the previous section. In Fig.

25, the tools are listed in an order that illustrates the progression of and interest in 

agent-based capabilities over time.
Approved for public release; distribution is unlimited.

92

Fig. 25 An estimated timeline of tools embodying ABM capabilities. Dates in the illustration 

are approximate.

5.1.2 Policon/Senturion

Policon is a tool that was developed by Bueno de Mesquita in the late 1970s and 

early 1980s (Larson et al. 2009, p. 47). It uses algorithms from game theory, 

decision theory, spatial bargaining, and microeconomics to analyze political 

dynamics, modeling how competing agents change policy positions over time

(Bueno de Mesquita et al. 1999; Schneider et al. 2010). The tool computes the 

interactions and decisions of actors based on their own perspectives of situations 

and, as such, an individual actor’s perceptions of a situation can vary greatly from 

other agent perceptions of the same situation (Bueno de Mesquita 2010). The tool 

supports strategic-level analysts and decision-makers. 

Input into the model is specified by SMEs and includes key stakeholders, a range 

of policy alternatives, policy preferences, the estimated capability each stakeholder 

may employ, and the estimated importance each stakeholder assigns to each policy

(Fischhoff and Chauvin 2011). Once an analyst computes a baseline interaction, he 

or she explores changes to agents’ positions and importance in an effort to discover 

how to improve the political outcome.

The CIA has reportedly used the Policon model and has concluded that the model 

has been correct in over 90% of the real-world applications for which it has been 

used. In those cases where the model forecasts differed from those of SMEs, it was 

the Policon forecast that was correct (Cioffi-Revilla and O’Brien 2007, p. 32).

Senturion is an evolution of the Policon tool used at the National Defense

University (Kennedy 2014, p. 1). It is a simulation capability that analyzes the 

political dynamics within local, domestic, and international contexts. It predicts 
Approved for public release; distribution is unlimited.

93

how the policy positions of competing interests will evolve over time (Abdollahian

et al. 2006). Senturion analyzes behavior of actors who influence political outcomes 

by examining interactions through simulation. Like Policon, SMEs identify input 

parameters that include stakeholders, their positions, and the estimated importance 

of outcomes. It applies a game theoretic perspective and assumes that all agents 

maximize their interests and seek to form likeminded coalitions. Senturion also 

provides a user interface to visualize results of estimated interactions.

5.1.3 LODESTONE

LODESTONE is a RAND-developed tool that relies on the axioms of spatial 

politics, strategic game theory, expected utility theory, and rational-choice theory 

(Connable et al. 2014, p. 130). The capability originated as a Defense Advanced 

Research Projects Agency (DARPA)-sponsored research and development project 

in the 1980s and later upgraded in 1997. The model exhibits 3 features: 1) the ability 

to identify the most likely outcome, 2) the ability to calculate the expected utility 

(i.e., desired outcome) from changing interactions with another agent, and 3) the 

ability to reiterate agent interaction results until a stopping criterion is met.

5.1.4 Synthetic Environment for Analysis and Simulation (SEAS)

/Reference World Information and Simulation Environment 

(RWISE)

In the early 2000s, the SEAS capability was developed and later modified to 

represent effects-based operations for Joint Forces Command (JFCOM)

(Chaturvedi 2004; 2007; 2009; Snyder and Tolk 2006). Effects-based operations 

evaluate and consider the broader consequences of military operations (Maxwell

and Carley 2008). SEAS does this by using an agent-based modeling technique, 

which integrates socio-psychological and organizational-behavioral models at the 

individual, organizational, institutional, infrastructure, and geographical levels 

(Prevette and Snyder 2004). The simulation uses combinations of heterogeneous 

agents and includes individuals, organizations, institutions, and infrastructure.

Analysts communicate with SEAS using PMESII and DIME constructs, and the 

simulation has been used for strategic and operational communication. For 

example, courses of action defined in terms of DIME variables can model actions 

defined with the PMESII taxonomy and can attempt to quantify the impact of such 

actions. Joint Warfare Systems (JWARS) is a complementary simulation capability 

that models both red and blue forces and quantifies the impacts of military actions. 

For example, JWARS can model up to a million buildings with their own PMESII 

designations. The JWARS simulator and SEAS contain messaging capabilities, so 
Approved for public release; distribution is unlimited.

94

both models can communicate while focusing on modeling their respective and 

specific phenomena. 

RWISE is an evolution of the SEAS capability, with the goal of building a synthetic 

“mirror” of the real world. Automated data sources such as news feeds, opinion 

polls, demographic statistics, economic reports, and social media are fed into the 

system. Automatic calibration algorithms inform the RWISE system which 

scenarios, identified in prior time steps, are no longer valid so RWISE can prune 

those possibilities. In 2006, JFCOM’s Urban Resolve 2015 experiment tested 

RWISE in a synthetic environment (Cerri and Chaturvedi 2006; Anastasiou 2006).

5.1.5 Conflict Modeling, Planning, and Outcomes Experimentation 

Program (COMPOEX)

COMPOEX began in 2004 as a collaborative project sponsored by DARPA and 

JFCOM to develop technologies that could enhance the capability of leaders and 

staffs to plan and execute major operations in a complex environment (Kott and 

Corpac 2007; Waltz 2008). Ideally, such decisions would consider PMESII.

The project was the composition of 4 experiments: 1) effects identification, 2) 

domain visualization, 3) operation planning, and 4) parallel planning (Kott et al.

2010). The effects identification experiment analyzed whether a range of options,

such as actions against certain nodes or changing the timing and sequence of 

actions, would result in the intended actions. Domain visualization tested whether 

PMESII data was effective in helping teams understand complex domain 

information. Operation planning evaluated whether COMPOEX-developed tools 

would enhance creation of an interagency operation plan. Parallel planning 

evaluated whether parallel processes could effectively result in several named 

operational effects.

The underlying technologies of the tested tools in this project were a mixture of 

agent-based models, systems dynamics, Bayes Nets, discrete time models, Petri 

Nets, and Markov models. The primary contractor, BAE Systems, created a library 

of models, each instantiating an aspect of the PMESII model (Waltz 2010). For 

example, systems dynamics and linear programming techniques were used to 

model infrastructure components. However, political components were the only 

components to use agent-based models.

Lessons learned from COMPOEX evaluations mainly focused on human 

interaction and expectation management rather than on technological items. 

Pre-experiment training is a strict necessity and experiment design needs to exhibit 

individual and specific interagency cultures. Moreover, the nature and interaction 
Approved for public release; distribution is unlimited.

95

between interagency members and organizations need to be well understood in 

order for broad collaborations to be successful.

5.1.6 Joint Non-Kinetic Effects Model (JNEM)/Athena

JNEM is a tool developed by NASA’s Jet Propulsion Laboratory (JPL) for US 

Army brigade, division, and corps commanders operating in Stability and 

Reconstruction Operations during 2005 (Ripley 2008). In 2009, JPL released an 

updated version of the tool named Athena, with the goal of creating a single-user 

decision support tool with a multiyear planning horizon. By early 2012, JPL 

released the third version of Athena, which models the interaction between ground

forces, demographics, attitudes, politics, economic and information (Duquette 

2012; Chamberlain and Duquette 2013).

Athena simulates situations at the strategic and operational levels. The analyst is 

responsible for entering all inputs. According to user reports, it takes about 4

months to run a complete cycle, which includes gathering data validated by an SME 

and the potential end-user (Connable et al. 2014). Inputs include defining actors 

and their resources, belief systems, and strategies. They also include civilian, force, 

and organization groups, and their respective satisfaction levels. Models of the 

neighborhood and environment are also defined. After the operator identifies all of 

the inputs, the simulation is run; at the operator’s discretion, “magic” capabilities 

may be interjected, and attrition may be adjusted to account for noncombat deaths, 

attitude changes, and environmental situations.

Athena is a tool included in the Army’s One Semi-Automated Force (OneSAF) 

simulation suite (Boiney and Foster 2013). Within OneSAF, Athena helps represent 

the interaction of PMESII and DIME effects on the operational environment over 

time (Brecher 2014). It helps the analyst understand the impacts of operations, 

including possible second-, third-, and higher-order effects.

5.1.7 Peace Support Operation Model (PSOM)

The UK’s Ministry of Defence, Defence Science and Technology Laboratory 

(DSTL) developed PSOM in 2006 as a simulation-based wargaming tool designed 

to help decision-makers consider populations affected by irregular warfare. As 

such, PSOM incorporates attributes of civil and military components, crisis 

management, and security and stabilization operations (Brecher 2014). DSTL 

created the tool after British participation in the Iraq and Afghanistan wars, 

identifying a need for a better understanding of conflict stability.

DSTL designed PSOM to simulate situations at the strategic and theater levels. 

PSOM uses a combination of 2 processes, a Strategic Interaction Process and an 
Approved for public release; distribution is unlimited.

96

Operational Game, to simulate interactions. The Strategic Interaction Process 

simulates strategic level decision-making, such as those by political and military 

leaders. The Operational Game translates those decisions into campaign effects. 

Both processes simulate military battle groups and civil reconstruction groups 

carrying out their designated activities with the objective of identifying how policy 

decisions, strategic decisions, and operational effects are interrelated.

The PSOM sequence of events is as follows: 1) develop the course of action, 2) 

evaluate the course of action, 3) run a simulation turn, 4) receive the situation 

update, 5) discuss the results, and 6) repeat. PSOM operates with the assumptions 

that a population’s behavior changes slowly and that each time step is one month. 

During the input process, players may interact with each other to simulate 

diplomatic and political activity in the real world. An analyst then feeds the input 

information into the simulation. Inputs consist of the following: 1) all factions in 

the campaign, 2) the environment, 3) the infrastructure, 4) the population, 5) the 

economy and employment, 6) political-level interactions, 7) military units and 

combat, 8) reconstruction and civilian units, and 9) human and other soft factors. 

Plans are scripted and all actors proceed according to plan for the full month; PSOM 

adjudicates outcomes of agent interactions stochastically. Outputs consist of 

1) civilian casualties, 2) security, 3) consent, 4) initiators of kinetic events, and 

5) readiness levels.

According to DSTL, analysts have used PSOM several times to support planning

decisions, finding that the tool was appropriate as a “useful conversation piece” and 

helped stakeholders think through situations. However, PSOM is a time- and dataintensive wargame and requires many different players to represent scenarios, 

which may make it impractical for time-sensitive tactical scenarios.

5.1.8 National Operational Environment Model (NOEM)

NOEM is a strategic analysis/assessment tool that provides insight into complex 

operational environments (Salerno 2010; Sudit et al. 2015). The US Air Force’s

Rome Laboratory developed NOEM in conjunction with Sandia National 

Laboratories, Johns Hopkins University Applied Physics Laboratory, the 

Massachusetts Institute of Technology, and the University of Texas at Austin. 

NOEM supports baseline forecasts by generating plausible futures based on the 

current state. It supports what-if analysis by forecasting ramifications of potential 

“Blue” (i.e., friendly) actions on the environment, including unintended 

consequences in the short and long term. NOEM supports sensitivity analysis by 

identifying possible pressure points and their relative sensitivities to stabilize a 

region or nation.
Approved for public release; distribution is unlimited.

97

NOEM can assist decision-makers, analysts, and researchers with understanding 

the interworkings of a region or nation state, the consequences of implementing 

specific policies, and the ability to test new operational-environment theories. 

NOEM models the environment as a nation-state with the ability to connect 

multiple regions. The model uses stability-operations theory as the basis of its 

interactions and supports DIME and PMESII variables. NOEM models people 

based on age, interest group, affinity, and occupation; it also supports social and 

cultural interaction and understanding.

5.1.9 TRADOC’s Irregular Warfare Tactical Wargame (IW TWG)

IW TWG was developed by TRADOC Analysis Center beginning in 2008 as an 

outgrowth of the PSOM capability. It focuses on human behavior and social and 

cultural factors, analyzing the effect that different courses of action have on civilian 

population attitudes. It is developed in conjunction with the Center for Naval 

Analyses, TRADOC G2, Naval Postgraduate School, Texas A&M University, 

Argonne National Laboratory, Charles River Analytics (CRA), University of 

California at Davis, Army Materiel Systems Analysis Agency, and others (Duong

et al. 2011; Duong and Bladon 2012; Works 2015). The intended users of the tool 

include operational users at the brigade, battalion, and company commander level 

(Brecher 2014). Context includes interactions between ground forces and civilians 

and key leaders and infrastructure.

IW TWG consists of several components: 1) an operational wraparound, 2) 

planning, adjudication, and visualization environment, and 3) cultural geography 

and nexus-network learner models (Duong 2009a, 2009b, 2012, 2013; Duong and

Pearman 2012). The first component simulates command and control function and 

incorporates infrastructure and population atmospherics. The second component 

allows players to plan, access, and record tactical-level inputs such as local leaders, 

population, and infrastructure interaction. The third component provides population 

responses to events. IW TWG is designed to simulate one or more weeks of time, 

with human, social, cultural, and behavioral attributes as domain features.

Data input into the system includes population, key leaders, local infrastructure, 

and population responses to actions. IW TWG then uses SME input to verify inputs. 

Gathering data for the data entry process, including verified inputs, can take around 

9 months. Similar to other simulation models, a specific scenario output is one of 

many possible outcomes and is not an exact forecast.

5.1.10 IW TWG: Cultural Geography Model

In 2006, development began on the Cultural Geography capability, later used within 

the larger IW TWG modeling tool. Like several of the tools discussed, the 
Approved for public release; distribution is unlimited.

98

capability uses agent-based modeling to represent interactions between agents. It 

seeks to address analysis issues within the context of an irregular warfare setting, 

and interactions are based upon Army and Marine COIN and Stability Operations 

doctrine (DA 2011). Agents within the system interpret information through their 

own perceptions of beliefs, values, and interests (Alt and Lieberman 2010; Alt et 

al. 2010). The theory of “narrative paradigm” is used to instantiate agent models to 

include identity and individual agent history (Alt et al. 2009). Analysts use a 

combination of survey data and SME input to derive agent attributes. The advantage 

of using this theory is that agents are semirational actors, in that they rely on their 

beliefs about the world; this is an advantage because modeled situation better 

represents reality (Wakeman 2012). Moreover, the theory of “planned behavior” 

guides individual agent actions, describing the relationship between an agent’s 

individual attitudes toward different behaviors, his or her perception of the 

relationship of social norms to behaviors, the individual efficacy of his or her given 

behavior, and his or her intention to take an action (Ajzen 1991).

5.1.11 IW TWG: Semi Automated Force (SAF) 

Researchers at the University of California at Davis developed another tool for IW 

TWG that automates the selection of “Red” (i.e., enemy) courses of action. The 

Human Social Culture Behavior (HSCB) modeling program developed the 

capability and then it was transitioned into the IW TWG suite. The tool makes 

choices based on historical data about players and on their standard operating 

procedures, the commander’s preference for kinetic versus nonkinetic tasks, and 

the commander’s preferences to automatically select tasks (Boiney and Foster 

2013).

During a wargame, the SAF tool speeds up red task selection to allow more time 

for planning. Input data include the executed tasks of all players, the red players’ 

scheduled tasks, the population’s attitudes and behavior, and the population 

density. These data produce a list of predicted tasks that red forces will undertake 

in the following week.

5.1.12 IW TWG: Military Information Support Operations (MISO) 

Planner

The MISO Planner tool was developed by CRA for the HSCB program and 

transitioned into the IW TWG suite. Originally, CRA developed the capability for 

the Air Force as a program called Organizational and Cultural Criteria for 

Adversary Modeling, which evolved into the Susceptibility and Vulnerability 

Analysis Tool. Military information-support operations, formally known as 
Approved for public release; distribution is unlimited.

99

psychological operations, are one of several courses of action a planner may 

consider while modeling civil interactions.

The MISO tool allows the user to create a computational model of an adversarial, 

friendly, or neutral population. The tool helps the user assess and verify causal 

reasoning, communicate the reasoning decision process, forecast future behaviors, 

and analyze consequences of course of actions (Air Force Research Laboratory 

2011).

The IW TWG Cultural Geography model produces observable attitudes and 

behaviors of populations, which the MISO tool accepts as input, in addition to a list 

of scenario events. The analyst then selects the message for the intended 

demographic. The tool output includes a list of tasks, the message, delivery mode, 

location of delivery, and timeline.

5.1.13 Complex Operational Environment Model (COEM)

COEM was developed out of IW TWG and is intended for modeling theater and 

higher operational levels. COEM attempts to model extended-duration campaign 

effects. It does this by modeling the interactions among PMESII-PT variables rather 

than concentrating on the individual variables themselves. COEM decomposes 

each PMESII-PT variable into a number of subcomponents, based on social science 

theories and SME inputs, to enable wargaming staffs to better understand how 

Human Domain variables interact with its subcomponents and external stimuli 

(Brecher 2014).

5.1.14 COEM: Complex Military Mission Environment (CM2E)

The CM2E extends the COEM model by integrating a stochastic, closed-form 

simulation. This tool alters the human-interaction workflow by producing results 

through simulation instead of relying on the wargaming staff to specify them. 

Environmental states may change based on those simulation results (Brecher 2014).

5.1.15 Worldwide-Integrated Crisis Early Warning System (W-ICEWS)

The W-ICEWS tool was developed by Lockheed Martin’s Advanced Technology 

Laboratories, beginning in 2007 as a DARPA project, and demonstrated the 

possibility of forecasting events of interest in different countries (Ruvinsky et al. 

2014; Kettler 2015; Kettler and Lautenschlager 2016). The tool was developed in 

conjunction with Duke University, Innovative Decisions, Lustick Consulting, 

Pennsylvania State University, Strategic Analysis Enterprises, and Raytheon BBN 

Technologies. The tool is used at the strategic level as a social radar capability

(Maybury 2011).
Approved for public release; distribution is unlimited.

100

The W-ICEWS system monitors open-source information and uses that information 

to monitor, display, and forecast assessments at the national level. It is a composite 

of several components including a data processing and annotation capability 

(iDATA), a regional trend and sentiment capability (iSENT), a situational 

understanding and implication capability (iCAST), and a visualization capability 

(iTRACE). The iDATA annotates the “who, what, where, and when” using a 

combination of shallow and deep natural language-processing technologies (Corby 

2015). Various integrated capabilities identify entities and events from within the 

source data. The iCAST interprets annotated data and forecasts events that 

contribute to instability in identified countries. It uses a combination of statistical 

and agent-based models to generate aggregate forecasts (Mahoney et al. 2011). The 

iSENT measures attitudes and perceptions about issues, people, and events and the 

propagation of attitudes across the Internet. The iTRACE provides a means to 

visualize trends and patterns and allows the user to identify time series and 

relationships. It also allows the user to view the underlying data sources.

The iCAST calibrates agent-based models to the current state of the modeled 

country and characterizes possible output to produce multiple trajectories of 

potential future states for that country. The actions and interactions of the agents 

within the ABM are based on rules and environmental conditions that describe the 

country and its sociopolitical theory. For example, all of the political models stem 

from a generic political model that captures rules based on political theory for 

country-specific actions, such as protest, rebellious activity, and religious violence. 

As planners recognize and comprehend the political theory at play, using the 

country’s environmental conditions to provide input into the initial state of the 

agent-based model, the model enables them to understand a variety of potential 

future states and thus potential conflict futures for the country in question.

5.1.16 Virtual Strategic Analysis and Forecasting Tool (V-SAFT)

An agent-based capability developed by Lustick Consulting to support the DARPA 

ICEWS project, V-SAFT demonstrates the plausibility of forecasting events of 

interest (Lustick 2012; Egeth et al. 2014, p. 344). The tool is a forward-looking 

social radar that estimates the likelihood of events, such as domestic political crisis, 

insurgency, rebellion, and ethnic or religious violence (Alcorn et al. 2012; Chandra 

2012; Garces and Alcorn 2012; Ward et al. 2013).

Figure 26 depicts the V-SAFT architecture, which is composed of 3 phases 

consisting of model building, experimentation, and analysis. During the 

model-building phase, each modelled country’s geographical and administrative 

regions are translated into a pixilated spatial grid. Then, a combination of 

social-science theories and data (e.g., demographic, cultural, electoral, economic, 
Approved for public release; distribution is unlimited.

101

and SME input, such as the elite network) are used as inputs into a Generic Political 

Model. Because it is integrated with W-ICEWS, the tool receives data as input, 

including elections, censuses, and SME reports, every month.

Fig. 26 The V-SAFT architecture and implementation phases. The V-SAFT capability is used 

within W-ICEWS to create and configure country-specific models.

During the Experimentation phase, models are instantiated with between 1,000 and 

4,500 agents per country, followed by a brief initialization period. Once 

initialization is complete, simulated time progresses at approximately weeklong 

steps for one year. This simulation repeats 1,000 times to create distinct simulation 

results. V-SAFT ensures simulation results are unique by randomizing the 

distribution of agent attributes and by introducing small permutations into

individual agent biases.

Finally, the analysis phase displays and compares monthly results to determine 

trends within the modeled country. Results from the current month identify 

probable, plausible, or possible outcomes. Forecasts are provided in a narrative and 

textual form for the analyst to read; additional information is made available if the 

analyst desires it. The objective of these results is not to provide an actual predicted 

future but to provide quantifiable and time-sensitive indications and warnings about 

specific types of events of interest.
Approved for public release; distribution is unlimited.

102

5.1.17 Versatile Multiscale Strategist (vmStrat)

The vmStrat modeling framework is an ABM capability developed under the HSCB 

program. The principal investors originally created one of the more mature ABM

frameworks, Repast, and vmStrat appears to be a related interactive framework for 

modeling actor interactions at multiple scales. For example, actors may represent 

states, alliances, substate institutions, parties, and so forth. It is designed so the user 

can experimentally and interactively create a model representing the current 

scenario. Likely, vmStrat has the potential to model Human Domain interactions 

but would need a mature operational concept, including additional agent attributes 

and interrelations (Brecher 2014).

5.1.18 COA Analysis by Integration of Decision and Social Influence 

Modeling with Multi-Agent System (CADSIM)

In support of the HSCB Program, Perceptronics Enhanced developed CADSIM to 

explore and model the effects of different COAs. It is a turn-based simulation tool, 

which forecasts models of interaction between actors, specifically modeling 

possible responses to COAs. CADSIM models actors and their attitudes, biases, 

and behaviors using Social Influence Network Theory.

SMEs identify inputs for CADSIM. They identify the actors to model, including 

relevant individuals, groups and populations, and political entities such as states, 

international organizations, and nonstate armed and criminal groups. SMEs also 

define the nature and strength of relationships between actors. Moreover, they 

develop each actor’s decision model based on a consideration of the actor’s sociocultural, political, military, interests, strengths, and biases.

CADSIM outputs are ranked probabilistically and indicate a range of outcomes, if 

assumptions and hypotheses are valid. However, CADSIM does not emphasize 

Human Domain considerations. Additionally, author bias is possible because of the 

difficulty in assessing the degree of objectivity in data, design, and COA options. 

While testing the tool, experimental users have noted it may be difficult to 

configure if the operator is not already familiar with how to use the tool (Brecher 

2014).

During CADSIM’s development, the Air Force Targeting Center acted as a 

transition partner and provided operational use cases (Boiney and Foster 2013). The 

goal of the enhanced CADSIM framework is to allow planners to identify critical

points of social interaction, and the likely influence of nth-order communication 

effects on different COAs.
Approved for public release; distribution is unlimited.

103

5.2 Evaluating Agent-Based Technologies: Maturity Level and 

the Human Domain

Agent-based technologies are a combination of pre-existing research and enabling 

technologies, and each initiative has different degrees of maturity according to their 

objectives. This section identifies the criteria the study uses to evaluate and 

categorize research and then summarizes and identifies publically available agentbased research. The study uses a combination of Human Domain definitions, as 

identified by Brecher (2014) for SOCOM, and an agent-based model maturity 

framework identified by Axtell and Epstein (1994).

5.2.1 Evaluation of Maturity Level

Each of the agent-based research and technological capabilities discussed in this 

report has its own degree of maturity. The following evaluation scale provides a 

measure by which the authors graded and ranked identified capabilities. The 

scoring method has 5 possible states:

1. Does not meet the need: In this case, the research or technology is not 

likely to be an appropriate resource in modeling or developing the intended 

Human Domain capability

2. Could be an external reference or example: In this case, the research 

or technology could be a resource in modeling or developing the intended 

Human Domain capability; however, it is not likely to be a strong starting 

point

3. Starting point for modeling the need: In this case, the research or 

technology meets the threshold where it could be used as a starting point for 

additional research, without requiring the development of another 

independent capability, or as a starting point to begin modeling the intended 

Human Domain capability

4. Exceeds the need: In this case, the research or technology exceeds the 

threshold of an initial starting point and somewhat represents a modelled 

Human Domain capability

5. Ideally suited for the need: In this case, the research or technology 

demonstrates a well-modelled Human Domain capability

The authors use the previously stated evaluative criteria alongside the Axtell and 

Epstein maturity model to evaluate research. As discussed in Section 3, Axtell and 

Epstein identify a 4-level Empirical Relevance Framework ranging from Level 0

through Level 3, as shown in Fig. 27. Over time, Axtell has expanded this 
Approved for public release; distribution is unlimited.

104

framework to include comparisons of docking, system engineering, policy, and the 

amount of data and time required (Axtell et al. 1996). For the purpose of this study, 

the authors added 5 evaluation criteria stated within the column “Research 

Relevance to Human Domain.”

Fig. 27 Axtell and Epstein’s Empirical Relevance Framework describes the maturity of 

agent-based models, ranging from realistic caricatures to quantitatively characterized 

phenomena at the microlevel. This illustration also incorporates docking, system engineering, 

policy/regulation design, and estimated data and time required (Egethet al. 2014, p. 290). 

5.2.2 Human Domain Elements

The Human Domain concept is a construct that identifies elements necessary to 

understand human decision-making (Brecher 2014). The concept is comparable to 

other communicative frameworks, such as PMESII or ASCOPE, but, as might be 

expected, better describes the Human Domain (US Special Operations Command 

2015). The study uses this categorization taxonomy to categorize existing agentbased research.

Brecher identifies 5 elements to the Human Domain concept: 1) social, 2) cultural, 

3) physical, 4) informational, and 5) psychological. Social elements are structures 

and relationships among groups and institutions, often involving competition for 

influence and efforts to impose interest and perspectives. Cultural elements are a 

society’s beliefs, customs, and way of life, and their impact on people’s behavior. 

Physical elements are those natural and manmade attributes, which shape 

individuals’ priorities. Informational elements are the sources, availability, and 

substance of data. Psychological elements describe how people evaluate and act 
Approved for public release; distribution is unlimited.

105

upon information, including patterns of exercising judgment and reasoning in 

response to available facts. Each of these elements is described in the following text 

with additional detail by their respective subelements; this text is not 

comprehensive but rather illustrates the intended concepts.

5.2.2.1 Social

The following subelements—public groups, state institutions, local governments, 

civic groups, and societal groups—make up the social element. Public groups are 

gatherings of people drawn together by a specific cause but are not officially 

organized. They are distinct from organized civic groups. A state institution 

includes the state government and its connected entities, such as a Department of 

Motor Vehicles, Taxation, Social Services, and so on. The local government 

includes city, town, and village governments and its connected institutions, such as 

the department of public works, the police, and so forth. Civic groups include 

nongovernment groups who interact with people on specific issues such as poverty, 

welfare of animals, or regarding support of medical cause issues. A societal group 

includes gatherings for religious services and for social well-being (Brecher 2014).

5.2.2.2 Cultural

The following subelements—ideology, tribalism, customs or beliefs, ethnicity, 

religion and rituals, language, and communication—comprise the cultural element. 

Ideology is a set of conscious and unconscious ideas, which help define a person’s 

goals, expectations, and motivations. Often, they are a set of standards considered 

to be normal behavior. Tribalism is a set of organized persons; tribalism may refer 

to a way of thinking or behaving in which people are more loyal to their tribe than 

to their friends, their country, or any other social group. Tribalism also implies the 

possession of a strong cultural or ethnic identity that separates a member of the 

tribal group from the members of another group. Based on strong relations of 

proximity and kinship, members of a tribe tend to possess strong feelings of 

identity. Customs are beliefs or behaviors passed down within a group or society, 

generally with symbolic meaning or special significance and with origins in the 

past. An ethnic group is a socially defined category of people who identify with 

each other based on common ancestral, social, cultural, or national experience. 

Unlike most other social groups, ethnicity is primarily an inherited status. 

Membership of an ethnic group tends to be defined by a shared cultural heritage, 

ancestry, origin myth, history, homeland, language and/or dialect, and symbolic 

systems such as religion, mythology and ritual, cuisine, fashion, art, and physical

appearance. Religion and rituals refers to the system of religious beliefs and rituals 

practiced by a tribe or group of people. Language is the spoken word of a group or 
Approved for public release; distribution is unlimited.

106

groups of people. Communication is the means of exchange of ideas, thoughts, and 

intentions among people (Brecher 2014).

5.2.2.3 Physical

The following subelements—geography, topography, hydrology, urbanization, 

resources, and climatology—make up the physical element. Geography examines 

the natural environment and how organisms, climate, soil, water, and landforms 

produce and interact. Topography is the field of geoscience comprising the study 

of surface shape and features of the earth; it also includes surface shapes and 

features. Topography, in a narrow sense, involves the recording of terrain, the 3-D 

quality of the surface, and the identification of specific landforms. Hydrology 

includes studying the movement, distribution, and quality of water including the 

hydrologic cycle, water resources, and environmental watershed sustainability. 

Urbanization is the population shift from rural to urban areas and the ways in which 

each society adapts to that change. It is the main process by which towns and cities 

are formed and become larger as more people begin living and working in central 

areas. Resources are sources or supplies from which benefit results. Typically, 

resources are materials, energy, services, staff, knowledge, or other assets that 

produce benefit and may be consumed or made unavailable. Climatology is the 

study of climate, defined as weather conditions averaged over a period (Brecher 

2014).

5.2.2.4 Informational

The following subelements—means, message, and audience—make up the 

informational element. Means are the methods of transferring information among 

people, such as the use of the Internet, print media, radio, television, or 

person-to-person communication. Message is the information transferred through 

such means. Audience is the group or groups to whom the message is directed 

(Brecher 2014).

5.2.2.5 Psychological

The following subelements—cognition, awareness, perception, and reasoning—

make up the psychological element. Cognition is the set of all mental abilities and 

processes related to knowledge, attention, memory, judgment and evaluation, 

reasoning, problem solving and decision-making, comprehension and production 

of language, and so on. Awareness is the ability to perceive, feel, or be conscious 

of events, objects, thoughts, emotions, or sensory patterns. Perception is the 

organization, identification, and interpretation of sensory information to represent 

and understand the environment. Perception is active and shaped by learning, 
Approved for public release; distribution is unlimited.

107

memory, expectation, and attention. Reasoning is the capacity to consciously make

sense of things, apply logic, establish and verify facts, and change or justify 

practices, institutions, and beliefs based on new or existing information. In addition, 

reasoning includes judgment and critical thinking. Judgment supports reasoning by 

evaluating evidence to make a decision. Critical thinking involves critique and 

emotion involves the state of feeling (Brecher 2014).

5.3 Sampling Agent-Based Technologies That Resemble Human

Domain Elements

For this section of the report, the study surveyed 72 instances of academic ABM

frameworks (see Appendix A). Models and research, using these frameworks, were 

examined for their resemblance to the described Human Doman elements; that is, 

whether each model exhibits some amount of utility, such as the possibility of being 

an external reference to research, for the purpose of developing future Army- and 

tactically specific agent-based models.

In general, the scope of this survey is limited to academic-level research and, where 

possible, is limited to discussion of models with possible tactical relevance. While 

there is some promising research, the majority of the available ABM research is at 

a basic research level of maturity (Brecher 2014).

5.3.1 Social Technologies

In this section, 3 models—GROWlab (Geographical Research on War Laboratory), 

RebeLand, and AfriLand—provide the reader with an indication of the maturity of 

current social Human Domain research. Project Albert and PAX are other research 

projects that seek socially relevant capabilities (Koehler et al. 2004; Tan 2004; 

Schwarz 2004, 2005; Johnson et al. 2009).

GROWlab is a suite that models countries at the regional and national levels. 

Attributes of the model include country borders, state and local institutions, and 

ethnic groups within countries. However, the primary emphasis is to modelling

conflict within and between countries and modelling social elements at a high level 

of fidelity. A particularly useful aspect of GROWlab is that a number of databases 

on the modeled countries are available. As such, these resources could be useful for 

Army researchers.

GROWlab’s database includes data on 160 countries, covering dates between 1946 

and 2013 (Vogt et al. 2015). Historically, research enabled by GROWlab has 

modeled conflicts between states or national-level conflict within a state, such as a 

civil war. A user interface included with GROWlab helps researchers visualize 

subsets of data and contains information such as settlement patterns. This interface 
Approved for public release; distribution is unlimited.

108

allows the researcher to identify traditional ethnic regions, the amount of access a 

group has to executive power, the population, the Gross Domestic Product, and/or 

terrain data. The user interface also allows the researcher to download queried 

datasets.

By using the available data, GROWlab researchers have been able to find points of 

conflict. For example, in Bangladesh, 4 ethnic groups have divided political power. 

Database records helped researchers to discover that Bengali Muslims were the 

politically dominant ethic group and that the Bengali Hindus, Biharis, and Tribal 

Buddhists were being politically discriminated.

Another agent-based modeling framework used to model conflict, MASON, has 

been developed into a model called RebeLand (Cioffi-Revilla and Rouleau 2009a). 

RebeLand models a fictitious island nation consisting of a national government and 

3 provinces. Each province had a state (provincial) government, population, and 

resources. Researchers used this model to examine political stability to measure the 

impact of different policies. Political stability is the ability of a political system to 

withstand changes given a range of stress, such as social, economic, or 

environmental, and RebeLand acted as a model to help researchers better 

understand conditions that contribute to stability.

Using their experience with RebeLand, the same team developed another model on 

top of MASON called AfriLand, which modelled the government and society of 10

eastern African countries: Sudan, Ethiopia, Somalia, Djibouti, Eritrea, Rwanda, 

Burundi, Kenya, Tanzania, and Uganda (Cioffi-Revilla and Rouleau 2009b). The 

model helped identify conflict within and between the countries, especially through 

examining the effects of transborder and transnational issues, such as refugee 

migration and societal stress. Concepts demonstrated with RebeLand, such as the 

interaction of politics, society, government, and associated issues, were reused in 

the AfriLand model.

Using the empirical relevance framework described earlier, all 3 models are 

assessed at Level 0 maturity (i.e., representing a caricature of reality) in terms of 

their applicability to tactical modeling. In the case of GROWlab, the model 

provides between a Level 0 and Level 1 representation at the strategic or national 

level. GROWlab provides a high level of detail for ethnic populations at the country 

and region levels but also embodies lower-level detail (e.g., city, town, and village). 

Comparatively, the other 2 models, RebeLand and AfriLand, demonstrate a 

representation of both national and local governments; however, it is unclear 

whether the model authors used real-world data observations to calibrate social 

representations and interactions within the model.
Approved for public release; distribution is unlimited.

109

5.3.2 Cultural Technologies

Several agent-based models exhibit cultural attributes: VECTOR, Hispanic 

Population Model, Insurgency Dynamics, and SWARM.

VECTOR is a project that implements cultural knowledge and interaction for 

training within a game-type interactive environment. VECTOR uses cognitive 

agents to create an immersive cultural-awareness-training environment for Soldiers 

(Bell 2005). The game uses agents to model a number of actions that include the 

behaviors of nonplayer characters within the population, realistic interactions of 

virtual actors with the trainee, and the demonstration of emotion in virtual actors.

Another model, Hispanic Population Model, examines Hispanic acculturation 

within the United States (Wallis et al. 2004). The model uses the AnyLogic ABM

framework to examine both individual agent behavior and that of the environment, 

especially people moving in and out of modeled neighborhoods. The model 

included individual person variables for aging, income, education decisions, and 

the choice of whether and when to have children. At the macrolevel, Hispanic 

Population Model models aggregate behaviors, such as the decision to live in a 

dominantly Hispanic area.

The Insurgency Dynamics process model, published by AnyLogic, divides a 

population into 4 groups: government supporters, dissidents, insurgents, and 

removed insurgents. Both dissidents and insurgents interact with government 

supporters, and various amounts of the population are stochastically swayed toward 

one group or the other. The government seeks to minimize the size of the insurgents 

group, and the strength of the effort depends on resources and the size of the 

insurgents group.

A fourth model uses the SWARM ABM framework (Allan 2010) to simulate ethnic 

mobilization, based on migrations in the former Yugoslavia (Srbljinovic et al.

2003). In this model, agents were composed of 3 attributes: ethnic membership, 

affinity to their ethnicity, and social network. This research demonstrates that the 

frequency of messages (e.g., stay at the current location, move to a new location) 

had the effect of increasing the speed of mobilization, while increasing neutral 

appeals slowed mobilization down. Regardless of this finding, mobilization 

increased if the agent had a social network (alternative findings include those by 

Hamill and Gilbert 2008, 2010). Overall, the researchers found that multiple stable 

states could exist and outcomes often depend on the initialization values of agents’ 

mobilization intensity and number of social-network connections.

Using the empirical relevance framework, all 3 models are assessed between Level 

0 and Level 1 maturity (i.e., between a representation of a caricature of reality and 
Approved for public release; distribution is unlimited.

110

a qualitative representation at a macrolevel) in terms of their applicability to tactical 

modeling.

5.3.3 Physical Technologies

Most of the models of physical elements contain spatial attributes and range from 

2-D grids to 3-D GIS spaces. Despite the fact that the majority of the following 

descriptions specify the use of GIS capabilities, many researchers prefer to use 

simple 2-D spaces in their models for convenience and reduced complexity

(Burtsev and Korotayev 2004; Andrade 2010; Johnston 2013). The models 

discussed in the following are 2 versions of GAMA, Acequia-based Agriculture, 

RebeLand, and the Hydrogen Economy model.

GAMA is an ABM framework that enables use of GIS data within models

(Taillandier et al. 2012). GAMA handles the geometries by utilizing traditional 

shapefiles, raster files, or mesh files. These types of files enable GAMA to import 

attributes such as geography, hydrology, and other factors, such as road networks.

Some research has investigated using agent-based models to evaluate policies for 

improving urban mobility (Bathe and Frewer 2008; Fosset et al. 2016). This work 

used the GAMA framework to create a virtual representation of Grenoble, France. 

The model replicated buildings, businesses, streets, traffic, and populations to 

determine the effects of policies on pollution in the city. Shops and businesses were 

considered resources, with populations moving from homes to work and/or 

shopping.

Other research has used GAMA to study the social, economic, and ecological 

impact of water management in the Adour-Garonne Basin, France (Grignard et al.

2013). This particular model simulated the physical elements of the geography, 

hydrology, urbanization, and resources within the greater basin area.

Another model, Acequia-based Agriculture, has investigated situations involving 

water management in New Mexico (Wise and Crooks 2012). GIS data for the 

county of Taos, New Mexico, and its surrounding area was imported from the US 

Geological Survey’s EarthExplorer into an agent-based model using the MASON 

framework. Information included terrain elevation change, shapefiles, and land-use 

data. Land use included data describing urbanization and agriculture areas. The 

overall model replicated real-world conditions of terrain and hydrology (i.e., rivers, 

acequilas) and decisions made by individual landowners. The result of the model is 

a realistic socio-physical system simulating water-supply use, resulting agricultural 

consequences, and the impact to the surrounding communities.
Approved for public release; distribution is unlimited.

111

Similarly, the previously discussed RebeLand model incorporated physical terrain 

features, including topology and vegetation land cover. Moreover, a simple weather 

system simulated climate variability, the impact of droughts, and other natural 

stressor events. The model also included natural resources distributed throughout 

the terrain, such as oil, diamonds, and gold.

Another model is the Hydrogen Economy Model, which models highways and local 

road information of the Los Angeles, California, metropolitan area (Mahalik et al. 

2007). In this model, a 2-D grid and cell representation is used to model the world,

but a GIS-layout of roads informs which cells are “road” cells. Two types of agents 

are included in the model: drivers and investors. Driver agents move their cars 

between demographically assigned home neighborhoods and job sites. Drivers have 

a variety of characteristics, such as income levels, environmental concerns, risk 

aversion, and car type preferences (e.g., conventional vs. hydrogen). Investor 

agents build, own, and operate hydrogen fuel stations based on the investor’s 

estimates for potential profit at available locations. Researchers used the model to 

investigate practical and logistical questions concerning the plausibility of 

hydrogen-based vehicles. 

Using the Empirical Relevance Framework, all 5 models are assessed between 

Level 0 and Level 2 maturity (i.e., between a representation of a caricature of reality 

and a quantitative representation at a macrolevel, in terms of physical elements) in 

terms of application to tactical modeling. The distributions of these models 

illustrate the range of specificity models may use to affect simulations. Current 

research capabilities, if selected, can model some physical features (e.g., using GISannotated data, modeled weather systems) with sufficient accuracy for use by 

tactical military planners.

5.3.4 Informational Technologies

The Human Domain informational element specifies 3 subelements: the 

transmission, the message, and the audience. However, in this survey, the authors 

could not find any models that explicitly examined the modality of communication. 

Each model reviewed in the following—Janus, ACT-R, and Geo-Game—exhibits 

some form of communication (i.e., interaction); one of the premises of ABM is that 

agents interact with each other and their environment. The general agent-based 

interactive process partially represents the transmission and audience subelements, 

such as some agents being aware of or receptive to information communicated by 

other agents. 

The Janus agent-based model, simulating carpooling situations, provides an 

example of how informational subelements may be incorporated (Hussain et al.
Approved for public release; distribution is unlimited.

112

2015). Agents must belong to the same social group (e.g., communication channel) 

to send and receive communications (Gaud et al. 2008). Agent roles are the basis 

for all model interactions, and each agent has a queue for sending and receiving 

messages. Messages are delivered according to recipient roles.

Similarly, in the ACT-R model, agents who are in the same network pass 

information between themselves. However, the means of communication is not 

included in the model. The Geo-Game model exchanges information through 

individual “chats” (Reitter and Lebiere 2011). Similarly, this model does not 

include social-media mechanisms.

Using the Empirical Relevance Framework, these models are assessed at a Level 0

maturity (i.e., a caricature representation of reality), in terms of application to 

tactical modeling. At a tactical level, the means of transmission is essential to 

planners. Information operation messages can fail because of the limitations of the 

modality chosen.

5.3.5 Psychological Technologies

Of all the academic-research models the study identified, 2 stood out in their 

capability to represent psychological subelements. ACT-R and SOAR implement 

various capabilities and maturities of 4 subelements: cognition, judgment, emotion, 

and critical thinking. Each model represents these subelements differently, 

particularly with respect to emotion, but both approaches exhibit positive attributes.

ACT-R is a framework, based on the theory of rational analysis, used in the 

development of cognitive models capable of predicting and explaining human 

behavior. It models human cognition as a series of modules interacting with the 

external environment. The ACT-R architecture is composed of a visual object 

recognition module, a module for controlling extremities, a declarative module for 

retrieving information from memory, and a goal module that tracks goals and 

intentions (Anderson et al. 2004). Each module has an associated buffer that stores 

information. A central production system coordinates information placed in the 

buffers. It also processes buffer information, recognizing patterns and triggering 

module-specific rules that ultimately result in actions. Depending on the 

application, researchers may add additional modules as long as the same basic 

architecture is used.

Research for the Georgia Tech Aegis Simulation Program successfully used 

ACT-R to model cognition, judgment, and critical thinking (Fu et al. 2006). This 

project evaluated a visual display to provide mission-critical information to 

Anti-Air Warfare Coordinators. The task involved acquiring information such as 

aircraft location and speed to classify aircraft in a timely manner. ACT-R modeled 
Approved for public release; distribution is unlimited.

113

2 cognitive processes: first, the conversion of visual inputs from the display screen 

to the coordinator’s memory, and second, the discernment and classification of 

aircraft type. The results of the modeling closely mirrored human times.

The ACT-R framework does not have a built-in module dealing with emotions. 

However, researchers have proposed different means to address this issue; for 

example, Roman Belavkin (2002) suggested an implementation using attributes of 

the conflict resolution component to realize emotions. Although these suggestions 

are not implemented in the framework, they demonstrate a means to deal with 

emotions within the model.

SOAR differs from ACT-R in fundamental ways, using multiple memory and 

processing modules. For example, Procedural, Semantic, and Episodic memories 

are used for short-term memory. Another difference is in handling conflict 

resolution; ACT-R evaluates current information with matching rules to select 

actions, whereas SOAR implements all matched rules in parallel and deliberates on 

the selection and application of operators. SOAR operators are procedures with 

preconditions; in other words, a condition must be true to activate an operator, and 

if it is so, the procedure will run.

SOAR expresses Human Domain psychological subelements within its decision 

cycle. During input, the model acquires data from the environment. In the 

“propose” phase, critical thinking is realized by applying rules to acquired data. 

The next phase, “decide”, applies judgment in making decisions on which rule to 

implement. Once SOAR makes a decision, rules describing how to apply the 

operator are activated.

As with ACT-R, SOAR does not have a built-in module for emotions. However, 

similarly to ACT-R, there has been research suggesting such modules. For example, 

one module implemented within SOAR integrated emotions into the cognitive

model using appraisals. Appraisal theory suggests that emotions result from the 

relationship between goals and situations, along specific dimensions (Marinier et 

al. 2008).

Numerous research has used the SOAR framework since the 1990s (Soar-Related 

Research 2016). The TacAir-Soar agent was one such project that developed a rule 

system for fixed-wing aircraft (Jones et al. 1999; 2009). Project-developed 

innovations included reasoning capabilities, integration with a simulated 

environment, a representation of human-like coordination and communication, and 

situational understanding to drive agent reasoning. The result was a system capable 

of executing most airborne missions that the US military flies in fixed-wing aircraft. 

This research accomplishes these objectives by integrating a variety of capabilities, 

including reasoning about interacting goals, reacting to rapid changes in real time, 
Approved for public release; distribution is unlimited.

114

communicating and coordinating with other agents and humans, maintaining 

situational awareness, and accepting new orders during flight. Because of 

TacAir-SOAR’s success, it can model and implement appropriate tactical behavior 

for a broad variety of such missions routinely used by the US Navy, Air Force, and 

Marines; the UK Royal Air Force; and opponent forces in full-scale exercises.

According to the Empirical Relevance Framework, the ACT-R model operates at 

approximately Level 2 maturity (i.e., a quantitative representation at a macrolevel, 

in terms of psychological elements) in terms of application to tactical modeling.

However, the SOAR model, given the proper adaption to the applied domain, can 

likely operate between Level 2 and Level 3 maturity (i.e., between a quantitative 

representation at a macrolevel and a quantitative representation at a microlevel, in 

terms of psychological elements). These models illustrate the ability to model 

human cognition through a number of scenarios, but both models also have similar 

weaknesses in modeling emotion.

5.4 Sampling and Analysis of Agent-Based Research

While the prior section addressed existing academic agent-based models, this 

section evaluates a selected sample of publically available agent-based research. 

Papers submitted to the Computational Social Science Society of the Americas 

(CSSSA) serve as a sample from which to extrapolate the nature of public research 

applicable to Human Domain elements. The study samples 54 papers submitted to 

and published by the last 3 CSSSA conferences, held in 2012, 2013, and 2015. The 

same level of relevance to Human Domain scale, outlined and used in the prior 

section, is once again used to provide the reader with an indication of the maturity 

and breadth of current research.

The following illustration is a heat map of the categorized research. The surveyed 

research works are enumerated in Appendix B. Their identification (ID) numbers 

correspond to the numbers used in Fig. 28. Each of the 54 papers sampled are listed, 

with recent conference papers starting at line one. One hundred and eighteen unique 

authors and coauthors contributed to these articles and approximately 15% of those 

authors contributed to more than one paper over those 3 submission years. 

Light-gray cells indicate no discussion of the identified Human Domain element 

and/or subelements. A mid-gray color indicates a Level 2 estimate; that is, there is 

some discussion of the designated Human Domain attribute and the paper could be 

used in future research to justify a research direction. A darker gray indicates the 

paper had enough assessed maturity that the research could be a starting point for 

future Army applications. None of the papers appeared to warrant maturity level 
Approved for public release; distribution is unlimited.

115

ratings of 4 or 5, which would be research appropriate for various degrees of 

off-the-shelf integration into tactical Army wargaming application.

Fig. 28 A heat map identifying whether papers submitted to CSSSA conferences (2012, 2013, 

and 2015) exhibit characteristics identified by the Human Domain. Dark-gray cells indicate 

research that the Army could potentially use to develop Army-specific tactical ABM

capabilities.

The objective of this assessment is to provide a holistic sense of the nature and 

maturity of research within computational social science community. It may be 

possible that the assessed value of a specific paper is found incorrect upon review. 

In addition, it should be remembered that this distribution of research is a sample; 

other conference proceedings—such as those provided by Behavior Representation 

in Modeling & Simulation, the Journal of Artificial Societies and Social 
Approved for public release; distribution is unlimited.

116

Simulation, and the Journal of Complex Systems—are likely to have different 

distributions of the nature of presented research and published works.

After categorizing each paper, the authors evaluated results holistically in 2 ways. 

The first type of evaluation involved simply calculating the numeric sum of each 

row. The goal was to find research that demonstrated a broad spectrum of Human 

Domain influences, and this evaluation produced several interesting results. First, 

the research with the highest scores correlated with both maturity and existence of 

contextually larger simulation projects. For example, the top IDs were no. 27 and 

47. These 2 papers corresponded respectively to 1) Deborah Duong’s (2013) paper 

regarding a specific capability used in the Army’s IW TWG analysis and 2) the 

paper by Latek, Rizi, Crooks, and Fraser (2012), which identifies an agent-based 

capability to assist with finding border crossings and determining the possibility of 

detection, sponsored by EADS North America. 

Research with the highest scores correlated to research that was more 

comprehensive than other peer research. For example, Michel and Megerdoomian 

(2015) demonstrated population-physiological interactions (e.g., exhibiting 

characteristics of fear, disregard) in the context of environmental dangers, 

specifically Ebola outbreaks. Alternatively, McCaskill (2013; listed in Appendix 

B) investigated intervention strategies for coordinating military and 

nongovernmental-organization activities. This paper diverged from common 

practice within the computational social-science community—namely, the practice 

of keeping the model as simple as possible to emphasize and demonstrate the 

intended phenomena. However, in McCaskill’s paper, the authors found that, as 

additional context was included in the author’s research, the developed model 

became more compelling. The results of this paper suggest that it is likely necessary 

to keep models simple while developing the “science of the phenomena and/or 

model”, but it is reasonable to associate additional, contextually relevant variables 

thereafter, during 6.2-Applied Research or 6.3-Advanced Technology 

Development.

The second evaluation strategy computed the sum of each column. The 3 most 

common Human Domain subelements referenced in the sampled research were 

Physical (Resources), Social (Societal Groups), and Psychological (Judgement). 

Coming from the perspective of using an ABM approach, this result makes sense. 

In their simplest form, agent-based models are the combination of agents 

interacting with each other and their designated environment. Thus, researchers 

often use resources as a proxy representation of the environment. Societal groups 

often become the lowest common denominator of interacting agents within models. 

Finally, agents within the model enact judgment by making decisions, regardless 
Approved for public release; distribution is unlimited.

117

of whether the decision possibility (i.e., stochastics) was determined a priori or 

learned during each simulation.

Under this evaluation strategy, the aggregate identification and use of Human 

Domain subelements followed a power-law distribution (Cioffi-Revilla and

O’Brien 2007, p. 29). Some subelements are referenced several times more than the 

least-referenced subelements. In contrast, the broad Human Domain elements are 

referenced a proportionally similar number of times. The proportional distribution 

was computed by taking the sum of each column within an element and dividing it 

by the number of included columns, thus representing a normalized number of 

references. Surprisingly, the Cultural Human Domain element and 

subelements (e.g., ideology, tribalism, religion and rituals, and language) were the 

least represented and researched area.

5.5 Sampling of Foreign Agent-Based Research

This section provides a comparison between agent-based models published in the 

United States and those published by organizations in China and Russia, to illustrate 

the nature of comparable research in other countries. As in the previous section, the 

goal is not to provide an exhaustive representation but to provide an overview of 

ongoing research. These samples emphasize military-relevant or other topical 

agent-based research and are organized according to their originating cities and 

institutions.

5.5.1 Chinese Agent-Based Research

The 28 research papers sampled originated from 24 institutions in 7 cities 

throughout the country (Fig. 29). Sampled papers are identified in Fig. 30 and are 

grouped and sorted by the city, the institution, and name of the paper. Duplicate 

titles thus occur when authors from multiple institutions collaborated on the paper

(Kai-jia et al. 2012; Jiangl 2015).
Approved for public release; distribution is unlimited.

118

Fig. 29 Seven cities throughout China, identified numerically on this map, represent the 

locations of published agent-based research.

Beijing, China (Fig. 29, Location 1)

ACADEMY OF MILITARY MEDICAL SCIENCES

“A framework of multilayer social networks for communication behavior with agent-based 

modeling”

BEIJING INSTITUTE OF ELECTRONIC SYSTEM ENGINEERING

“Study on MAS-based Modeling and Simulation of Land Combat”

BEIJING UNIVERSITY

“On Crowd Psychology Behavior Simulation and Security”

CHINA AGRICULTURAL UNIVERSITY

“The Agent-based Architecture and Simulation of Intelligence Traffic”

CHINA DEFENSE SCIENCE TECHNOLOGY INFORMATION CENTER

“Study on MAS-based Modeling and Simulation of Land Combat”

Fig. 30 An enumeration of military-relevant or other topical agent-based research produced 

by Chinese organizations. Locations, organizations, and names of papers are identified.
Approved for public release; distribution is unlimited.

119

Beijing, China (Fig. 29, Location 1 cont.)

CHINESE ACADEMY OF SCIENCES

“Agent-Based Modeling of Netizen Groups in Chinese Internet Events”

COMMUNICATION INSTITUTE OF CESEC

“A Mobile Agent-Based Network Management Architecture and Simulation System for 

Special-Operational Radio Communications Networks”

INSTITUTE OF SOFTWARE

“The Agent-based Architecture and Simulation of Intelligence Traffic”

NATIONAL DEFENSE UNIVERSITY

“Research on Military Simulation System based on Multi-Agent”

“Study on Some Key Issues about Agent-Based Modeling in War Complex System”

“Study on Agent-based Comprehensive-domain War Universe”

“Research and Implementation of the Third Party Intelligence Agent’s Strategy DecisionMaking Behavior Model”

PLA UNIVERSITY OF SCIENCE & TECHNOLOGY

“Evaluating Effectiveness of Information Sharing via Combat Simulation Based on Agent”

“Lanchester combat model in conditions of modernized warfare”

RENMIN UNIVERSITY

“Analysis and Simulation on Mass Psychology by CAS Theory” [includes models of policy 

research]

Nanjing, China (Fig. 29, Location 2)

NANJING ARMY COMMAND COLLEGE

“Model and Method for A Multi-Agent-based Ammunition Consumption Calculating”

“Research on Agent-based Modeling and Simulation of Military Counter System”

“Personality and propagation model of psychological war agent”

“Dynamic Model Research on Ammunition Demand in Combination of Firepower Blow”

NAVY TRAINING EQUIPMENT INSTITUTE

“Agent-based Modeling and Controlling Strategy in Generation of Virtual Battlefield Scene”

PLA UNIVERSITY OF SCIENCE AND TECHNOLOGY

“Study on Combat Agent Model”

SOUTHEAST UNIVERSITY

“Agent-based Modeling and Controlling Strategy in Generation of Virtual Battlefield Scene”

Fig. 30 An enumeration of military-relevant or other topical agent-based research produced 

by Chinese organizations. Locations, organizations, and names of papers are identified 

(continued).
Approved for public release; distribution is unlimited.

120

Wuhan, China (Fig. 29, Location 3)

COMMUNICATION COMMAND ACADEMY

“Dynamic Model Research on Ammunition Demand in Combination of Firepower Blow”

“Study on MAS-based Modeling and Simulation of Land Combat”

NAVAL UNIVERSITY OF ENGINEERING

“A Mobile Agent-Based Network Management Architecture and Simulation System for 

Special-Operational Radio Communications Networks”

SECOND ARTILLERY COMMAND COLLEGE

“Dynamic Model Research on Ammunition Demand in Combination of Firepower Blow”

Bengbu, China (Fig. 29, Location 4)

BENGBU NAVAL PETTY OFFICER ACADEMY

“Recognition-Primed Decision-Making Model Based on Multi-Agent”

BENGBU TANK INSTITUTE

“The Research on Modeling and Simulation of Warfare Decision-Making Behavior”

“Research on Agent-based Modeling and Simulation of Military Counter System”

Changsha, China (Fig. 29, Location 5)

COLLEGE OF ASTRONAUTICS AND MATERIAL ENGINEERING

“Study on Complex Adaptive System and Agent-Based Modeling & Simulation”

NATIONAL UNIVERSITY OF DEFENSE TECHNOLOGY

“A framework of multilayer social networks for communication behavior with agent-based 

modeling”

“Research on Effectiveness Evaluation of Psychological Operations Based on Multiple-agent 

Modeling and Simulation Method”

“Research on modeling method of complicated information system based multiple 

intelligence agents”

“A Multi-Thread Implementation Method of BDI Agent”

Hefei, China (Fig. 29, Location 6)

ARTILLERY ACADEME OF PLA

“Law of Dynamic Consumption for Artillery Ammunition”

Xi’an, China (Fig. 29, Location 7)

XI’AN RESEARCH INSTITUTE OF HI-TECH

“Research on Behavior Modeling Method of Agent-Based CGF” [includes models of brigade, 

battalion, and operational units]

Fig. 30 An enumeration of military-relevant or other topical agent-based research produced 

by Chinese organizations. Locations, organizations, and names of papers are identified 

(continued).
Approved for public release; distribution is unlimited.

121

5.5.2 Russian Agent-Based Research

The 8 research papers sampled represent 6 institutions in 3 cities across Russia (Fig. 

31). The papers are organized in a similar manner to those from China and are 

identified in Fig. 32 (Kotenko 2010; Kotenko et al. 2006; Makarov and Bakhtizin

2010).

Fig. 31 Three cities throughout Russia, identified numerically on this map, represent the 

locations of published agent-based research.

Moscow, Russia (Fig. 30, Location 1)

CENTRAL ECONOMICS AND MATHEMATICS INSTITUTE OF RUSSIAN ACADEMY 

OF SCIENCES

“Agent-based model for simulation of terrorism in Russia’s Caucasus”

RUSSIAN ACADEMY OF SCIENCES

“An Evolutionary Agent-Based Model of Pre-State Warfare Patterns Cross-Cultural Tests”

St. Petersburg, Russia (Fig. 30, Location 2)

ITMO UNIVERSITY

“Multiscale agent-based simulation in large city areas: emergency evacuation use case”

“The Multi-Agent Simulation-Based Framework for Optimization of Detectors Layout in 

Public Crowded Places”

ST. PETERSBURG INSTITUTE FOR INFORMATICS AND AUTOMATION OF 

RUSSIAN ACADEMY OF SCIENCES

“Agent-based modelling and simulation of network cyber-attacks and cooperative defence 

mechanisms”

Fig. 32 An enumeration of military-relevant or other topical agent-based research produced 

by Russian organizations. Locations, organizations, and names of papers are identified.
Approved for public release; distribution is unlimited.

122

St. Petersburg, Russia (Fig. 30, Location 2 cont.)

“Agent-based modeling and simulation of malefactors’ attacks against computer networks”

“Agent-Based Modeling and Simulation of Cyber-Warfare Between Malefactors and Security 

Agents in Internet”

Novosibirsk, Russia (Fig. 30, Location 3)

ACAD. LAVRENTYEVA

“Spatial Aspects of Agent-Based Modeling of Large Economy”

NOVOSIBIRSK STATE UNIVERSITY

“Spatial Aspects of Agent-Based Modeling of Large Economy”

Fig. 32 An enumeration of military-relevant or other topical agent-based research produced 

by Russian organizations. Locations, organizations, and names of papers are identified 

(continued).

5.6 Summary

This section has summarized existing agent-based tools and capabilities. It then 

contextualized published agent-based research in terms of Human Domain 

elements and the maturity level of the research and/or models, including a brief 

enumeration of foreign agent-based research. In spite of the breadth of published 

agent-based research and models, only a few capabilities are immediately 

applicable to tactical wargaming. For agent-based capabilities to become tactically 

useful to the Army, additional research and development is needed, emphasizing 

maturation of existing research to address tactical wargaming needs. The next 

section lays out a plan for such research.
Approved for public release; distribution is unlimited.

123

6. Plan for Research, Development, Technology, and Training

This report has evaluated the capabilities of agent-based analytics and game theory 

with the goal to identify potential investment areas for Army S&T targeted for 

research and development relative to automated and semiautomated analysis. For 

its purposes, the authors examined ABM as a means to improve wargaming rigor 

and minimize decision-making risk in the Human Domain, specifically through its 

application to tactical battalion-, brigade-, and division-level wargaming. A 

detailed study found a wide breath and combination of agent-based research and 

accomplishments. Based upon those findings, this report concludes with several 

recommendations, along with a discussion of the limitation of the technologies, a 

comparison to peer technologies and an Army-relevant technological path forward.

The desire behind using and applying technological capabilities, such as ABM in 

tactical wargaming situations, is to reduce the friction and cost of conflict with 

respect to the broader population. MG Flynn said, “Simply stated, the lesson of the 

last decade is that failing to understand the human dimension of conflict is too 

costly in lives, resources, and political will for the Nation to bear” (Flynn et al.

2012). Advanced technology itself, such as artificial intelligence, ABM, and game 

theoretic techniques, does not provide solutions to human conflict. Similarly, 

unwitting kinetic actions may be necessary but are not sufficient to resolve conflicts 

and fulfill political objectives: “The lesson is that while there are military victories 

there is never a military ‘solution.’ There’s only military action that creates space 

for economic and political life” (Stewart 2013).

After World War I, TE Lawrence profoundly stated, “War upon rebellion was 

messy and slow, like eating soup with a knife,” an appropriate analogy to the 

Army’s current wargaming process as applied to the civilian population. The Army 

is a powerful instrument of political will, built to withstand and defeat enemy 

forces, but it holistically plays an additional role of laying foundations for future 

economic, political, and social stability and peace. This study has evaluated 

agent-based technologies within the context of Army wargaming, knowing that 

while the plan of conflict itself changes the moment it begins, there is also 

conscious time set aside to evaluate decisions, which influence the Human Domain 

beyond kinetic actions. In many instances of the Human Domain, “tribalism is 

underneath everything; every glance, every knowing look, every payment, every 

invitation, everything that happens is linked to tribal connections” (Stewart 2013). 

Human populations—especially their interactive social, cultural, and tribal 

elements—are features of the social environment just as terrain is of the physical 

environment, and both aspects are vital to military, economic, and political 

objectives.
Approved for public release; distribution is unlimited.

124

6.1 Recommendations

This report proposes 4 recommendations: 1) begin researching and developing an 

Army-centric ABM capability, 2) form a stateside repository, 3) implement an 

ABM capability as a multilayered, distributed application, and 4) provide 

applicable training to ensure appropriate and knowledgeable proficiency. A 

combination of each of the aforementioned is essential to the success of the Army 

mission. Furthermore, the risk of not having a tactical ABM wargaming capability 

in the future would be worse than the status quo.

6.1.1 Research and Develop an Army-Centric Agent-Based Modeling 

Capability

The first recommendation is that the Army S&T community begins researching and 

developing Army-centric agent-based models. Four elements make up the basic and 

applied research recommendations: research and development of resources that 

model tactical Human Domain attributes, research and development of automated 

reasoning processes that support rigorous analysis, research and development of a 

cognitive-social framework, and evaluation of resources and models in terms of a 

model maturity framework. This report identifies short-, medium-, and long-term 

research goals for each of the 4 elements, and describes each step in terms of 

usefulness to the Army.

Army Special Forces Command has identified interrelated attributes necessary to 

model the Human Domain, namely social, cultural, physical, informational, and 

psychological elements. Numerous attributes of the Human Domain have been 

studied and documented within the social science field and generally take the form 

of research papers and reports and occasionally as proof-of-concept examples. This 

report recommends adopting and slightly adjusting the proposed Human Domain 

Framework, integrating existing social science research, and addressing 

military-specific gaps in that research. Game theoretic attributes of dynamic culture 

and low-rationality actors are likely to play a complementary modeling role, and 

the study recommends research, which incorporates these conceptual notions 

within a traditional application of agent-based models. Finally, the study 

recommends basic research investigating problems related to the connections and 

interactions between Human-Domain elements.

While agent-based technologies are at the core of the study’s recommendation, a 

complementary need exits for semiautomated reasoning processes to support 

rigorous analysis. Traditional notions of reasoning include concepts of deduction, 

induction, and abduction; the authors see a variety of situations where appropriate 

development of these concepts facilitates successful application of tactical ABM. 
Approved for public release; distribution is unlimited.

125

A narrative capability—research and development of a capability that analyzes 

model output and translates it into a narrative textual format with descriptions and 

interpretations—is one such example. The study recommends research identifying 

points at which human interaction affects operation of models; decision (human in 

the loop), veto (human on the loop), and analysis (human inside the loop) are 

examples of such interaction modes. Moreover, while ABM exhibits emergent 

properties, research of automated deductive capabilities that support translation and 

implementation of courses of action is useful. In addition, development of 

automated monitoring, model-adjustment, and alerting capabilities are appropriate

(Sudit et al. 2013).

ABM and additional supporting technologies have the potential to introduce a thirdgeneration (i.e., cognitive) wargaming capability to the Army, creating a positive 

overmatch decision-making capability. As a means to enable this ability, this study 

recommends researching and developing an understanding of the psychological 

dimension of agent behavior to model microlevel interactions through cognitive 

and interactive social relations. Subsequently, it also recommends investigating any 

resulting macrolevel properties of collections of cognitive-social agents interacting 

with their environments. Social networks, traditional and modern tribal groups, and 

local governments, each with their respective properties, are a possible means of 

measuring whether such a capability is sufficiently mature. Finally, the study 

recommends researching, developing, and evaluating course-of-action analysis 

using cognitive-social agents in terms of military and political objectives. 

Verification and validation of models, actors, and resources will likely be the most 

difficult aspect of developing an Army tactical agent-based capability. The study 

recommends that the Army consider adopting a standardized framework, such as 

the one originally proposed by doctors Rob Axtell and Joshua Epstein, to ensure 

stakeholders communicate with the same language; this is not a standardized data 

schema or ontology but rather an evaluative framework (Egeth et al. 2014). The 

Axtell–Epstein framework identifies 4 levels of maturity: Level 0 as a caricature of 

reality, Level 1 as a qualitative representation of macrostructures, Level 2 as 

quantitative representation of macrostructures, and Level 3 as a quantitative 

representation of microstructures. Model maturity helps end-users identify the 

nature of model results. Moreover, the study recommends the Army delay 

standardization of model interfaces—for example, using specified data schemas or 

ontologies—until individual models and resources reach Level 3 maturity.

6.1.2 Form a Stateside Repository

The second recommendation is to form and maintain a stateside repository of 

worldwide applicable models, agents, and resources; that is, a so-called Army 
Approved for public release; distribution is unlimited.

126

Wargaming Intelligence Center. The authors base this recommendation on the 

“Request for Support” functionality that the Joint Improvised Threat Defeat 

Agency’s (JIEDDO) Counter-IED Operations/Intelligence Center (COIC) 

supported with its “Attack the Network” operations (Del Vecchio et al. 2010). In 

particular, a system and set of processes and personnel who support timely (e.g., 6 

h, 24 h, 14 days) requests for information while interacting with liaisons 

representing a whole-of-government to coordinate wargamed courses of action. 

Moreover, it recommends the Army use the proposed Stateside Repository as a 

wargaming center to ensure consistency of interaction between rotating 

deployments of tactical units and local populations. The stateside center would 

become a repository for current and historical wargaming resources. Several 

existing capabilities exhibit some tactical-level capabilities to Army should 

investigate further (e.g., IW TWG, SEAS/RWISE; see Section 5).

6.1.3 Implement an ABM Capability as a Multilayered Distributed 

Application

The third recommendation is to implement the Army agent-based capability as a 

multi-layered and distributed application. The study recommends the Army require 

the user-interface implementation to be a web-based application, compatible with 

the Distributed Common Ground System–Army (DCGS–A) Program of Record. 

However, in light of possible network disconnections at the tactical level, the webbased application would need to run as a standalone hybrid web-application on a 

client machine. If broader network access is available, distributed, parallel 

processing, and/or cluster resources may be available for additional combinatorial 

scenario processing. Ideally, the agent-based capability would be sharable 

vertically and horizontally throughout the command and support structure, allowing 

individual users to receive, share, and maintain their own specific settings and 

model configurations. Finally, the study recommends models are versioncontrolled through the proposed stateside repository, using a distributed version 

control system such as GIT.

6.1.4 Provide Applicable Training to Ensure Appropriate and 

Knowledgeable Proficiency

The fourth recommendation is to incorporate applied topics of culture, tribes, 

complex adaptive systems, game theory, and applied statistics into specialized 

education and training curriculums, such as those provided at the Command and 

General Staff College (CGSC) and Warrant Officer training at the Combined Arms 

Center. The study also recommends the Army introduce a semiregular certification 

requirement encompassing basic essentials, applied methodology, and application 

of agent-based capabilities during wargaming.
Approved for public release; distribution is unlimited.

127

6.2 Potential and Limitations

While the recommendation is to research and develop an Army-centric ABM

capability, this is a nontrivial task. To research, develop, deploy, and maintain any 

agent-based capability will be a daunting task. The Army will need to devote 

significant person-hours to perform leading research and development, adequate 

verification and validation, subsequent model creation, and resource revisions. 

Similarly, a nontrivial amount of money, appropriated throughout the range of 6.1-

Basic Research to 6.7-Operational System Development money, and people 

serving in both stateside positions and training will be necessary (Fossum et al.

2000). However, little or no Army doctrine change is necessary. In general, an 

agent-based modeling can improve intelligence analysis rigor but the study 

recommends connecting the standalone capability to a “check and balance” 

process; without the combination of the process and the technology, the authors 

foresee the possible misuse or misapplication of the modeling capability in the 

future, resulting in an increase in tactical risk. 

In the immediate and near future, an ABM will not be able to forecast an exact 

future of tactical situations or precise Human Domain outcomes. However, an 

agent-based capability can help minimize risk by improving contextual awareness, 

communicating informed consequences of tactical decisions, and improving the 

study and training of warfare. In general, an agent-based capability requires a 

different type of thinking than structural problem solving. ABM is not an optimal 

or precision tool; rather, ABM is a robust tool that communicates a range of 

possibilities, including associated warning indicators with estimated possibilities of 

occurrence. The emphasis of the capability is on understanding and modeling the 

situation within the larger context of human interactions, and then working within 

the available Human Domain population.

Initially, ABM was developed in the field of mathematics. However, because the 

mathematics community deemed the modeling technique too inexact, it migrated 

to economics where it was discovered to be more rigorous than other alternative 

techniques. Discussion with members in the computational social-science 

discipline has indicated that some disagreement exists within the academic 

community as to whether heuristic results produced by agent-based models and 

their rules are as good and justifiable as the precise sets and results of formal 

mathematics, such as differential equations (Pramukkul et al. 2013). Because of the 

need to address the value of these results, there is potential for a new academic 

discipline of computational social science to emerge in the future, similar to 

emergence of the computer science field.
Approved for public release; distribution is unlimited.

128

If the Army decides it does not want to use an agent-based capability, this report 

recommends at least being prepared for an adversary to use it. The authors see 

agent-based capabilities as modeling the physics of social science; as an analogy, 

an eventual agent-oriented “bomb” capability could be possible (West 2015). ABM

will likely be a military capability of the future; both state and nonstate actors could 

weaponize it. In the commercial domain, agent-based modeling is beginning to be 

applied to exclusive domains. However, over time it is possible to see methods of 

computational social science applied to cross-domain problems.

6.3 Risks

Several risks are associated with a tactical Army agent-based capability. Getting 

ahead of the science is the foremost of the concerns; it is vital to avoid letting the 

engineering process of developing an agent-based capability outpace the science

(Barry, Koehler, and Tivnan 2009). If engineering outpaces science, there can be 

the illusion of contextual oversimplification that results in users not trusting the 

capabilities that agent-based wargaming otherwise contributes. It is important to 

understand the difference between tool precision and robustness (Meyer 2012); 

both types of results provide very different conclusions. 

Another risk is whether the validation of models and availability of the associated 

data is sufficient to ensure proper model validity (Hodges and Dewar 1992). The 

proposed Axtell–Epstein maturity framework identified and discussed in the prior 

section will help mitigate risk. However, the amount of time and effort the Army 

will need for this verification and validation process will be significant and should 

not be underestimated. 

Finally, the study recommends the Army ruthlessly minimize the amount of time 

tactical users need to interact with and provide input to the produced tools and 

capabilities. In a tactical environment, time and simplicity are of the essence. If 

deemed appropriate, the Army could choose to allocate additional time to develop 

models stateside, and provide the models and/or results to the tactical user.

Other technical risks include effectively understanding and translating model 

results to action. Lessons learned from former ABM projects suggest that this risk 

may be addressed by providing guidance to users, so they can interpret what the 

model is communicating. Other risks include balancing idealized “thinness” or 

simplicity of agent rule sets with the broad abstractness of tactical situations the 

Army expects to model. Finally, historical culture within the Army tends to avoid 

planning the human aspect of conflict; plans to alter kinetic actions, due to 

consideration of Human Domain consequences, may be a difficult organizational 

proposition and change.
Approved for public release; distribution is unlimited.

129

6.4 Comparable Technologies

Alternative procedural methods, such as game theory, include a variety of 

techniques to anticipate emergent consequences. Game Theory assumes a rational 

actor and makes assumptions that all players have some common knowledge. In the 

case of wargaming, motivation for winning the war is the argument for a rational 

actor. However, within different cultures and during asymmetric warfare, 

adversaries may not consider winning in the same way as the Army. Behavioral, 

Evolutionary and Low-rationality Game Theory are recent modifications to the 

original theory. In the first case, actor choices do not reflect benefits expected. In 

the second, beliefs and norms change over time. In the third, cognitive limitations 

affect the strategy space, and adaptive learning among strategies replaces the 

traditional utility maximization assumption. Reflexive Control is the former Soviet 

and current Russian evolution of game theory as applied to influencing the 

environment; in this case, the adversary will want to make the intended or 

constructed decision.

In addition to Game Theory, other methods may be useful. The technique of Points 

of Segmentation tries to identify a set of points that distinguishes individuals and 

groups with the goal of identifying possible points of cooperation and conflict. 

Prospect Theory describes the decision-making process of people, namely that 

people tend to be risk-taking with long-shot positive risk but risk averse when 

facing significant loss. Institutional Interactions is another systematic interaction 

model that allows participants to explore roles that influence capabilities and needs, 

similar to the current wargaming process. Morphological Analysis considers the 

entire space of possible implications; the analyst considers information abductively 

to derive judgments that could be useful (Ritchey 2011, 2014).

6.5 Technology Path

The study recommends the Army consider the following short-, medium- and longterm technology path to develop a tactical agent-based capability. For this study, 

the authors identify short term to be around 0–5 years in duration, medium term to 

be around 5–10 years, and long term to be 10 or more years. Table 5 and Fig. 33 

identify individually identified research goals. The first illustration identifies 

broad-brush topics for each of the 4 recommendations. The study categorizes each 

research topic into the applicable recommendation and time horizon and provides 

brief summary descriptions in the following text. The second illustration uses a 

similar matrix of recommendation versus time horizon to identify specific 

technological capability recommendations. Similar to topics identified in the 

technology areas table, brief summarizations describe the capabilities.
Approved for public release; distribution is unlimited.

130

Table 5 Table of recommended technology capabilities the Army should develop for a 

tactical agent-based capability in terms of time (i.e., near-, medium-, and long-term) and type

Recommendation Near-term Medium-term Long-term

Army-centric

agent-based 

modeling capability

Collect existing 

research and models

Develop initial 

capabilities for each 

human domain subelement that 

represents stylized 

facts

Research architecture 

of cognitive models

Identify reusable usecases of integrating 

civil concerns into 

wargaming

Intuitive user-interface 

experimentation

Improve quantitative 

macrolevel human 

domain model 

verification

Applied research of 

cognitive reasoning 

agents

Research interlinkages/interactions 

between human domain 

elements

GIS-integrated urban 

models

Microlevel models, 

stimulated by data 

(augmented 

computation)

Influence of country and 

regional monitoring on 

tactical models

Megacity urban models

Dynamic culture models

Low-rationality agents

Stateside repository

Data acquisition 

sufficient for 

quantitative macro 

models

Collect and document 

best practices with

existing tools

Identify request for 

support workflows

Develop version 

control workflow 

Using GIT technology

Library of models and 

research resources

Ontology identifying 

modular componentlevel data 

communication

Model verification and 

validation

Workflow supported 

area of operations civilpopulation continuity

Data acquisition 

sufficient for modeling 

and verifying 

quantitative micro 

phenomena

Integration application 

program interfaces 

(APIs) enabling access to 

operational data

Model maintenance (revalidation)

Running continuity 

models of areas of 

operations

Distributed 

application

Develop intuitive 

sliders user interface 

capability

Research display 

capabilities for use in

the field

Distributed APIs for 

using AnyLogic 

modeling software

Monitoring and alerting 

of modeled conditions

Web-based (in the 

browser) modelling

Human on the loop 

(veto)

Monitoring and alerting 

when real-world COAs 

deviate from wargamed 

COAs

Research model plugand-play

Human inside the loop 

(analyze)

Integration of capability 

into DCGS–A

Training and 

proficiency

Develop warrant 

officer curriculum

Integration of 

curriculum into the 

schoolhouse

Certification of stateside 

library resources

Certification of using 

ABM capabilities during 

wargaming
Approved for public release; distribution is unlimited.

131

Fig. 33 Timeline of recommend capability developments identified in Table 5

6.5.1 Research Recommendations

Four research recommendations are proposed and grouped into the following areas: 

1) Army-centric ABM capability, 2) forming a stateside repository, 3) 

implementing a distributed application and capability, and 4) providing applicable 

training and proficiency. Each of these 4 recommendations identifies technology 

areas containing individual research topics and categorized into near-, medium-, or 

long-term objectives. Following the table, brief summaries describe each research 

topic.

6.5.2 Army-Centric Agent-Based Modeling Capability: Near Term

Collect Existing Research and Models: The objective of this recommendation 

is to identify and document existing resources from both the academic and the 

government-sponsored communities and to make an index of these resources.
Approved for public release; distribution is unlimited.

132

Develop Initial Capabilities for Each Human Domain Subelement that 

Represents Stylized Facts: There are 5 Human Domain elements, each of which 

has multiple subelements. However, a survey of current research (see Section 5) 

has indicated that only a small number of developed models simulate militaryrelevant conditions. The objective of this recommendation is to use existing 

research to create military-relevant models appropriate to each specific Human 

Domain sub element.

Research Architecture of Cognitive Models: Several cognitive approaches 

exist for agent-based architectures, such as belief–desire–intention and 

social-cognitive theories. The objective of this recommendation is to perform longterm research into the approaches and methodologies that can embody cognitive 

abilities into agents.

Identify Reusable Use-Cases of Integrating Civil Concerns into

Wargaming: Tactical wargaming occurs across the breadth of military missions. 

The objective of this recommendation is to identify and build a library of notional 

or abstract representations of common tactical situations. Use-cases support 

subsequent recommendations, such as appropriate user interface development.

6.5.3 Army-Centric ABM Capability: Medium Term

Intuitive User-Interface Experimentation: A balance exists between model 

specificity, intuitiveness, and configurability. The objective of this 

recommendation is to identify an appropriate level of interaction, or “sliders,” and 

correlated dependencies that a tactical-level analyst should be able to control to 

gain insight into possible situation consequences, without being overwhelmed in 

terms of data or amount of required time.

Improve Quantitative Macrolevel Human-Domain Model Verification:

Models are characterized by their maturity level. According to the Axtell–Epstein 

Empirical Relevance Framework, the lowest level of model represents stylized 

facts or caricatures of macro phenomena. As maturity increases, models improve 

qualitative and quantitative representations of both macro- and microlevel 

phenomena. The objective of this recommendation is to improve individual Human 

Domain models, created in the near term, so they can more accurately represent 

macrolevel phenomena, ideally characterized with a degree of statistical relevance.

Applied Research of Cognitive Reasoning Agents: Agent-based models use 

probability distributions to represent agent decision-making processes. Cognitive 

reasoning has the potential to influence the nature of interactions between agents, 

with the goal of making models of these interactions more realistic or plausible. 

The objective of this recommendation is to incorporate research, performed in the 
Approved for public release; distribution is unlimited.

133

near term, into cognitive processes to create additional versions of Human Domain 

models.

Research Interlinkages/Interactions between Human-Domain Elements: 

SOCOM has identified 5 Human Domain elements, each with multiple 

subelements. However, each subelement is not completely distinct from the others. 

The objective of this recommendation is to begin developing models that 

incorporate several independent Human Domain phenomena into distinct models. 

A challenge that comes with this recommendation is an increased burden on the 

collection, verification, and validation of the resulting model.

GIS-Integrated Urban Models: Modelers use different forms of nearness in 

agent-based models, such as grid-cell distance, Euclidean distance, and geospatial 

distance. For simplicity, many academic models use simple measures of distance, 

such as grid-cell distance or number of hops in a social network. The objective of 

this recommendation is to begin incorporating GIS representations of distance into 

existing models. In addition to incorporating simple GIS attributes, this research 

would also seek to integrate locations of urban components, such as roads and 

locations of buildings, into models.

6.5.4 Army-Centric ABM Capability: Long Term

Microlevel Models, Stimulated by Data (Augmented Computation): In the 

medium term, macrolevel maturity is the recommendation. However, as model 

maturity improves, so does the ability to characterize microlevel interactions and 

phenomena. Additional possibilities emerge as a result; one such possibility is for 

augmented simulation and computation. Augmented simulation models a synthetic 

version of the observed world and predicts future situations, which may emerge 

based on the current state. The objective of this recommendation is to improve the 

maturity of existing models so they can represent microlevel phenomena. A 

challenge accompanying this task is the collection and use of sufficient microlevel 

data.

Influence of Country and Regional Monitoring on Tactical Models:

Initially, modelers build and test agent-based models without external influences or 

biases. At the tactical level, this keeps models simpler and easier to use. However, 

biases based on country and regional level models can provide insight into whether 

certain conditions may exist, such as a high probability of imminent conflict. The 

objective of this recommendation is to modify existing models to incorporate the 

context of biases into agents, interactions, and the environment.

Megacity Urban Models: The Army has identified the possibility of conflicts 

occurring within or around megacities, defined as cities with populations greater 
Approved for public release; distribution is unlimited.

134

than 10 million. The objective of this recommendation is to create models that 

represent phenomena and resources representative of megacities (Ehlschlaeger et 

al. 2014). A challenge accompanying this task is the collection, verification, and 

validation of data, and identified phenomenology.

Dynamic Culture Models: ABM is a computational technique that can represent 

emergent phenomena. Other computational techniques, such as evolutionary game 

theory, are alternative computational means to embody changing states and 

situations over time. The objective of this recommendation is to create models that 

embody the phenomena of cultural evolution and the effect it has on agent 

interactions. 

Low-Rationality Agents: Game theory is a computational technique that can 

identify the probability outcomes where the interest of multiple actors is involved. 

However, game theory assumes that all actors are rationally motivated. Research 

supporting prospect theory suggests rationality is not an appropriate 

decision-making assumption in all scenarios. The objective of this recommendation 

is to investigate how individual agents’ decision-making distributions change 

depending on their specific situations and context.

6.5.5 Stateside Repository: Near Term

Data Acquisition Sufficient for Quantitative Macro Models: One challenge 

during the creation, verification, and validation of models is the use of appropriate 

data ensuring models is operating properly. Collection of data, specifically Human 

Domain data, is an essential task, on which other recommendations, such as model 

creation verified with proper macrolevel phenomena, depend. The objective of this 

recommendation is to identify and collect data that modelers are likely to use while 

creating and calibrating models. A difficulty within this task is to collect enough 

necessary and appropriate data, as required to model a diverse number of Human 

Domain elements and subelements.

Collect and Document Best Practices with Existing Tools: Similar to a 

recommendation within the Army-centric ABM track (see Section 6.5.2 and

“Collect Existing Research and Models”), tools, and capabilities exist throughout 

the government and supporting contractor communities. Several of these tools were 

described in Section 5 (Existing Research and Applied Development of ABM), 

including IW TWG and SEAS/RWISE. The objective of this recommendation is to 

gather and store up-to-date versions of these capabilities for future use and 

reference. Additionally, this recommendation encompasses configuring tools to 

specific tactical Army needs, developing version control configurations and tools, 

and documenting best practices to interact with, configure, and test each capability. 
Approved for public release; distribution is unlimited.

135

In addition, this work should include documenting scenarios and use-cases the tool 

both can and cannot model appropriately.

Identify Request for Support Workflows: During the mid-2000s, JIEDDO set

up COIC to support Request for Information (RFIs) from the warfighter. In many 

cases, requests for information directly supported tactical situations. A primary 

reason for recommending a stateside repository is to support tactical wargaming 

needs. The objective of this recommendation is to review the processes and 

workflows that COIC implemented in supporting its RFI system and to identify 

processes appropriate to support tactical wargaming requests.

Develop Version Control Workflow Using GIT Technology: The 

configurability of agent-based models may present yet another challenge. For 

example, if a stateside analyst is supporting a tactical analyst but the tactical analyst 

wants to make changes to the model, there needs to be a mechanism to enable, 

manage, and control differences and versions. GIT is a modern, distributed 

configuration and version management tool used in software development. The 

objective of this recommendation is to identify a workflow process, using a version 

control capability such as GIT, which allows multiple analysts to configure and 

track multiple versions of agent-based models. Thus, analysts could share and 

distribute models while ensuring they can be independently configured, depending 

on analysts’ preferences.

6.5.6 Stateside Repository: Medium Term

Library of Models and Research Resources: This task is based on a prior 

recommendation (see Section 6.5.2, “Collect Existing Research, and Models”) to 

collect existing research and models. The objective of this recommendation is to 

form a stateside repository that becomes a library of tactical wargaming models and 

research. The burden of identification, use, collection, and versioning of tactical 

wargaming models produced in the field may present a challenge to this 

recommendation. 

Ontology Identifying Modular Component-Level Data Communication:

Ontological representations are useful for inferring unstated information. Thus, 

ontologies could be useful for communicating mature model results, whereas 

incorporating ontologies into the executions of models may be less useful. The 

objective of this recommendation is to develop an ontology that serves as a 

plug-and-play connector, integrating disparate models together. A challenge with 

this task is that the ontology needs to be used as a standardized machine-to-machine 

communication language rather than to compute information.
Approved for public release; distribution is unlimited.

136

Model Verification and Validation: Model maturity is essential if models are to 

become dependable. The objective of this recommendation is for the stateside 

repository to act as a quality control mechanism, with personnel supporting model 

validation. The breadth of the phenomena likely to be modeled may present a 

challenge, and it may also be difficult to ensure that the individuals doing the 

verification and validation are SMEs.

Workflow Supported Area of Operations Civil-Population Continuity: A 

difficulty with tactical-population interaction is the lack of continuity; deployed 

personnel change with regularity. A stateside support element could help provide 

and maintain continuity with local populations. The objective of this 

recommendation is ensure personnel and expectations for specific areas of 

operations are consistent and maintained over time, so each redeployment of 

tactical personnel are not starting over.

6.5.7 Stateside Repository: Long Term

Data Acquisition Sufficient for Modeling and Verifying Quantitative 

Micro Phenomena: In the long term, similar to the need for macrolevel data, 

microlevel data will be needed to model, verify, and validate microlevel 

phenomena. Moreover, the nature of the data collected will depend on the models 

created. The objective of this recommendation is for the stateside repository to 

collect and store microlevel data in anticipation of creating high-fidelity Human 

Domain models.

Integration APIs Enabling Access to Operational Data: As models become 

more capable and integrated within the wargaming process, there will likely be a 

need for timely operational data. Accessing operational data using automated tools 

presents a variety of integration challenges, and, as such, common APIs for 

accessing and parsing such data will likely be useful input to models. The objective 

of this recommendation is to create APIs that will translate operational data into 

appropriate model inputs. Both the breadth of translated data and the task of 

ensuring high quality automated translation will be challenges in implementing this 

recommendation.

Model Maintenance (Revalidation): Over time, models can become inaccurate 

because the assumptions used to create models change over time. Alternatively, the 

nature of the data collected may change over time. The objective of this 

recommendation is to ensure models, examples, documentation, and best practices 

are maintained and updated.

Running Continuity Models of Areas of Operations: In the long term, running 

continuous wargaming models that augment sensed reality is a unique military 
Approved for public release; distribution is unlimited.

137

advantage. As tactical deployments cycle through areas of operations, units would 

have access to ongoing modeling resources and forecasts. The objective of this 

recommendation is for existing and mature wargaming resources to be available for 

use and configuration, with a minimal amount of up-front tactical time 

requirements.

6.5.8 Distributed Application: Near Term

Develop Intuitive “Sliders” User-Interface Capability: Several challenges 

exist for deploying agent-based models into a tactical environment. One of those 

challenges is ensuring models are relevant to tactical situations. Another is ensuring 

that interaction with given models is intuitive and useful for tactical users. The 

objective of this recommendation is to identify and develop user interfaces for the 

tactical user where a minimal number of “sliders” or variables may be altered but 

the model still provides valuable insights. In this case, “sliders” may not be limited 

to direct one-to-one alteration of model variables; for example, a single slide may 

alter the value of multiple variables proportionally.

Research Display Capabilities for Use in the Field: The unique demands of 

the tactical environment may not be best suited to the entire representation of agentbased models and their human–computer interfaces on a computer screen. 

Traditionally, tactical commanders use sand tables to communicate situations and 

plans to larger numbers of people. The objective of this recommendation is to 

investigate means of extending the user interface beyond the computer monitor. For 

example, one could calibrate the orientation of a display and project the scenario 

on the side of a tent to increase the viewing surface. Alternately, other research 

could investigate interacting with the model using touch or motion interfaces.

Distributed APIs for Using AnyLogic Modeling Software: In the short term, 

the study recommends use of the AnyLogic desktop-based, agent-based simulator. 

Other ABM frameworks exist, but AnyLogic is preferred because of its ease of use 

in the process of designing and testing agent-based models (Coles 2015; Grigoryev

2015). The objective of this recommendation is to create distributed processing 

APIs that models use while simulated by the AnyLogic tool. APIs enable the system 

to interact with other third-party systems and data that affect simulated results of 

the wargame.

6.5.9 Distributed Application: Medium Term

Monitoring and Alerting of Modeled Conditions: The goal of an agent-based 

model to provide nonexact but plausible simulations, based on identified scenarios. 

Automated tripwire or monitoring and alert capabilities can monitor wargaming 

states, similar to the way staff uses such capabilities in reality. The objective of this 
Approved for public release; distribution is unlimited.

138

recommendation is to create APIs so monitoring and altering tools can be tested 

and configured using the wargaming scenario and simulation.

Web-Based (in the Browser) Modeling: In the long term, the goal is to run 

agent-based simulations in the browser so that installed programs are not needed. 

The objective of this recommendation is to develop or adapt ABM framework for 

use within the browser. Similar to desktop-based tools, web-based models need to 

exhibit ease of configurability, version control, and remote user interaction. 

Moreover, APIs enabling third-party data feeds need to be available. 

Human on the Loop (Veto): While users are simulating agent-based models, they 

need to be able to control the simulation in a way analogous to a “god’s-eye view” 

mode. For example, while the simulation is running, force specific agents to make 

decisions at specific times or locations. The objective of this recommendation is for 

the user to have control over decisions agents may make within the simulation, 

overriding normally used probability-based decision distributions. A challenge 

within this task is that it will be necessary to implement a “veto” mode, where the 

user does not take certain paths within the simulation without approval.

6.5.10 Distributed Application: Long Term

Monitoring and Alerting When Real-World COAs Deviate from 

Wargamed COAs: The objective of agent-based models, while simulating 

possible future interactions, is not to predict an exact future. Rather, the goal is to 

characterize possible situations that may exist in the future. A prior 

recommendation (see “Monitoring and Alerting of Modeled Conditions”) 

suggested developing a synthetic simulation capability, paralleling the observable 

world. This recommendation expands on that concept, adding additional alteration 

capabilities. Specifically, the objective of this recommendation is to develop a 

monitoring and altering capability that notifies a user if real-world circumstances 

are deviating from the characterization of normally expected situations. One 

challenge to this task is the need to reduce false alarms by ensuring real-time 

observations and input data feeds are high quality.

Research Model Plug-and-Play: The current state-of-the-art procedure is to 

create models are that independent of each other. If a modeler combines 2 or more 

models, the resulting combination needs verification and validation as if it were a 

unique model. The objective of this recommendation is to investigate whether an 

ontology can mediate the combination of independent models in a dynamic fashion. 

It may be difficult to determine how to produce or prove automated verification and 

validation may occur.
Approved for public release; distribution is unlimited.

139

Human inside the Loop (Analyze): In the long term, the authors envision the 

tactical user drawing from an existing library of models, simulating characteristics 

of the situation represented in reality. However, at some point, the stateside 

repository may have created and maintained pre-existing models for designated 

areas of operation. In that case, the tactical user would not need to model situations 

from scratch. The objective of this recommendation is to identify how the user may 

participate within the simulation, providing decision-making biases to agents 

within the simulation.

Integration of Capability into DCGS–A: DCGS–A is a program of record for 

Army intelligence systems. Over time, the goal of these intelligence capabilities is 

to be integrated and maintained within the DCGS–A environment. The objective of 

this recommendation is to ensure a successful transition of agent-based capabilities 

into DCGS–A, once the science and practicality of tactical model capabilities is 

demonstrated.

6.5.11 Training and Proficiency: Near Term

Develop Warrant Officer Curriculum: Wargaming is an essential component 

of the military decision-making process. However, tactical wargaming has been 

primarily taught as a manual process, nesting and synchronizing designated 

operations. Wargaming using agent-based modeling is a new capability to improve 

the rigor of tactical wargaming, specifically as Army operations relate to the civil

population. The objective of this recommendation is to develop a training 

curriculum for the analysts who are most likely to use it at the tactical level. A 

challenge to this task is the limited amount of time available for training and the 

need to streamline communication of agent-based concepts necessary for the 

appropriate use of the capability.

6.5.12 Training and Proficiency: Medium Term

Integration of Curriculum into the Schoolhouse: Once a curriculum is 

developed, it needs formal Army approval and appropriate training tools. The 

objective of this recommendation is to integrate tactical wargame training into the 

warrant-officer schoolhouse and CGSC.

6.5.13 Training and Proficiency: Long Term

Certification of Stateside Library Resources: Once training is complete, a 

certification process ensures stakeholders are cognizant of the available resources. 

The objective of this recommendation is to create a certification that is renewed 

yearly or biyearly in warrant-officer training. Training and certification would be 
Approved for public release; distribution is unlimited.

140

composed of a survey of available resources, their locations, and their abilities and 

limitations. 

Certification of Using ABM Capabilities during Wargaming: Once initial 

training is complete, a follow-up process ensures training is both useful and applied 

in the field. The objective of this recommendation is to create a certification process 

ensuring personnel are aware of and trained regularly on changes to wargame 

modeling. This certification would provide examples of how other analysts used 

resources in realistic settings, based on lessons learned.

6.6 Capability Recommendations

As Office of Secretary Defense’s HSCB Modeling Program neared completion, 

Boiney and Foster (2013) of MITRE, acting in its role of systems engineer, 

produced a summary document identifying social, cultural, and behavior modeling 

efforts (Schmorrow 2013). They categorized the multitude of funded projects into 

4 capability areas: understand, detect, forecast, and mitigate. “Understand” refers 

to the capabilities that support perceptions and comprehension of socio-cultural 

features and dynamics. “Detect” identifies capabilities that discover, distinguish, 

and locate operationally relevant signatures. “Forecast” refers to capabilities for 

tracking and predicting change through sensing and modeling the environment. 

“Mitigate” involves courses of action with respect to information gleaned from the 

social and behavioral sciences.

Boiney and Foster summarize each of the 4 socio-cultural behavioral capabilities 

in terms of capability needs. They also identify recommended research aims for 

each category. This study identifies the following capabilities roadmap, which uses 

those recommendations as a starting point to identify capabilities relevant to the 

Human Domain and ABM.

6.6.1 Army-Centric ABM Capability

Psychological Dynamics of Small Groups: Research identifying small group 

decision-making dynamics exists within the academic community. However, the 

psychological aspect of small groups is not well modeled; as such, a capability 

addressing this psychological aspect needs to be developed. Challenges in 

developing this capability include collecting data to create and verify such a model 

and identifying or modeling the underlying psychological decision-making process 

of individual agents.

Data, Methods, Models for Insurgent Communication: The survey of existing 

Human Domain information (see Section 5, Existing Research and Applied 

Development of ABM) identified a deficiency in the means and methods of 
Approved for public release; distribution is unlimited.

141

communicating information, regardless of the responsible party. Thus, an effective 

communication capability needs to be developed. Historically, these types of means 

and methods have been investigated for state actors to model military reaction times 

and procedures. Challenges in developing this capability include collecting data to 

verify such a model and keeping the model up-to-date with constantly emerging 

forms of communication.

Models of Emergent Social Leaders in Dynamic Social Networks: Models 

exist that represent interaction and projected results of political decision-makers; 

however, these models require copious amounts of time to investigate the relevant 

actors and identify their perspectives and resolutions to specific issues. This 

capability needs to be developed to reduce the amount of time needed to model 

emerging leaders within social networks that constantly change.

Models that Track Narratives over Time: Narrative text describes progressions 

of events through time. However, the underlying phenomena driving such 

interactions are often hidden to the external observer. Thus, models are needed that 

provide proper interpretations of text and the ability to form agents and interactions 

that support formation of the phenomena discussed in text.

Model Growth of Factions and Ideologies: As stated above, the 

decision-making processes of social groups have been investigated. However, 

existing research does not express the attribution of factions or the psychological 

dynamics of ideologies. Research is needed to address the decision-making results 

of populations to appropriately model growth of factions. To contribute to this 

research, sufficient data on decision-making results and population-level data 

regarding beliefs and ideologies need to be collected. Both these types of collected 

data are subject to inaccuracies for a variety of reasons, presenting a challenge to 

the development of accurate models.

Data, Methods, Models for State-Based Decision-Making: Many of the 

capability emphases addressed here concern nonstate actors. However, some recent 

events have demonstrated that state-based personnel can operate under different 

pretenses, suggesting a need to model state-actor-specific elements of decisionmaking (Thomas 2015). It may be difficult to model “informal” forms of command 

and control within the context of the civil population, or to collect observations 

regarding these types of interactions.

Data, Methods, Models for Insurgent Decision-Making: Analysis of 

decision-making is often reserved for the human analyst, so transferring this 

analysis to a more automated process presents challenges, such as the breadth of 

considerations that are incorporated into making decisions and the collection of the 

means and methods to verify proper model creation.
Approved for public release; distribution is unlimited.

142

Models to Predict Social Ties Based on Spatial–Temporal Information:

Research has indicated that movement to different locations over similar timespans 

can indicate the presence of social networks, which may then be modeled 

accordingly (Shen and Cheng 2016). A challenge in developing this capability is 

the collection of data that support model creation and verifying its accuracy. 

Data, Methods, Models for Insurgent Behavior: According to some 

psychological theories, behavior is coupled to the agent’s psychological state, 

suggesting that a model for psychological states may be developed. One challenge 

in developing this capability is developing a psychological agent model that 

accurately models reality. Another challenge is collecting enough data, with high 

enough resolution of insurgent behavior, to create and verify such models.

Behaviors Based on Culture-Specific Variables: Some psychological theories 

propose that culture is a component of the environment, with the result that its 

effects on behavior may be modeled. This capability must address the challenge of 

identifying whether individual behaviors are the result of environmental influences. 

If it is a result of culture, another challenge is identifying whether there is 

relationship between the strength of the culture and the behaviors exhibited.

Data, Methods, Models for State-Based Behavior: Estimation of state-based 

behaviors is often a hallmark of military versus military wargames. Developing a 

capability to estimate these behaviors needs to address the challenge of modeling 

the behavior of state-sponsored activities, which are under the pretense of operating 

outside formal state control.

6.6.2 Stateside Repository

Data-Collection Methodology and Tools: The objective of the stateside 

repository is to provide a reach-back capability to tactical wargamers. This is 

instantiated in several ways, one of which is to collect and publish data to create, 

verify, and validate models. A challenge in developing this capability is to identify 

and collect data that are appropriate for a wide variety of model development.

Extraction of Event Data to Stimulate Augmented-Reality Models: As 

model maturity grows, the expectation is that models will use APIs to external data 

sources to stimulate models. A challenge to this particular capability is to develop

an extraction capability that can parse operational data and provide accurate inputs 

to wargaming model simulations.

Mobile Data-Collection Methodology and Tools: An essential aspect of model 

creation and verification is ensuring appropriate data are used to guide model 

interactions. Some of these needed data are expected to be collected in the field. A 
Approved for public release; distribution is unlimited.

143

challenge in developing this capability is choosing the appropriate methodologies 

and means to collect the needed data.

Data Collection Tools for Attitudes, Sentiment, and Ideology: Sentiment 

analyses are derived from a variety of means including polls and salient textual 

attributes. However, a greater breadth and accuracy of collection information is 

expected to be necessary in the development of accurate Human Domain models.

Models that Simulate Consequences of Kinetic Effects on Populations:

While a majority of the emphasis of recommended models is on Human Domain 

elements, kinetic effects also determine some aspects of population physiology. A 

challenge with modeling kinetic effects on the population capability is modeling 

long-term consequences on individual agents within the population, who as a whole 

population express resultant phenomenon.

6.6.3 Distributed Application

Automated Assistance of Pre-COA Definitions and Scenarios: Wargaming 

is the independent evaluation of several scenarios, each of which requires the 

identification of different situations. Thus, an automated capability is needed, 

which assists users in configuration each scenario before wargaming. This 

capability should be a support tool that effectively addresses the challenge of 

automatically creating and configuring large amounts of the wargame.

Characterize Outcomes of Different COAs: An agent-based model will 

produce a single output each time the simulation is run. However, that output is one 

of many possible outcomes, each of which could be the output of another 

simulation. This capability requires learning to characterize the expected output in 

the least amount of time possible.

Capability to Translate COAs to Narration by Analogy: When agent-based 

models are simulated, the output is generally a set of time-series charts. The user 

then needs to interpret and translate results into possible consequences. This 

capability is to translate the model output into user-understandable consequences. 

Once achieved, the capability should ideally translate that output into analogies 

with which the user is familiar, in order for the “lesson learned” to be 

communicated effectively.

A Reasoning Engine to Explain Culturally Informed Actions: Throughout 

the world, actions communicate different meanings. Culture and context help 

inform meaning. From a computational perspective, a reasoning engine could 

interpret the meaning of actions within different cultures. One challenge facing this 
Approved for public release; distribution is unlimited.

144

capability is the need to address the breadth of cultures and situations required to 

translate the relevance of culturally biased actions.

Ensemble Model Computational Tool: Assuming all models are incorrect but 

some have value, this study suggests the developed wargaming tool run an 

ensemble of models to identify a wide range of possible emergent interactions. 

Since each model may use different assumptions, the capability will also need to 

ensure inputs and outputs, from each model is communicated appropriately.

Demographically Based Information Operations Tool: Once agent-based 

models are run and insights are identified, it would be useful to develop a tool that 

addresses how to use those insights effectively. One possibility is to alter inputs or 

adjust the models to account for demographic differences. The challenge is to 

develop an automated means of making model changes, emulating specific 

demographic attributes, to simulate proposed information operations. This 

capability would likely require a large enough data collection to test proposed 

information operations’ messages with confidence.

6.6.4 Training and Proficiency

Understanding Cost Effectiveness and Effectiveness of Cultural Training:

A baseline of training efficacy is needed to understand whether improvements need 

to be made or have been successfully implemented. This capability should identify 

the goals and objectives of training and identify the proper metrics to measure an 

improvement in effectiveness.

Cross-Cultural Training Program and Tools: Once demographic and 

culture-specific goals and objectives are identified, training tools and resources 

need to be identified and developed. It is essential to communicate the value of 

cultural differences to trainees so they understand the nuances of what they are 

trying to accomplish.

6.7 Training and Resource Implications

Similar to other specialized capabilities and technologies, tactical agent-based 

wargaming requires a proper understanding and skill set to produce expected 

results. Four criteria are essential for an effective training program: curriculum, 

theory, practice, and ongoing proficiency. “Curriculum” identifies what the authors 

believe are essential elements to an appropriate training curriculum. This study 

identifies curriculum resources available from both government and academic 

sources. “Theory” identifies topics of interest and their relevance to ABM. 

“Practice” identifies several of the practical concerns one would need to be aware 
Approved for public release; distribution is unlimited.

145

of when operating with a degree of proficiency. Finally, “proficiency” identifies 

evaluating ongoing knowledge in tactical ABM capabilities.

6.7.1 Curriculum

A wargaming curriculum is composed of several resources and perspectives. 

Firstly, a survey and recounting of wargaming history is essential to newly trained 

personnel. Summarization provides both context and practical rationale for 

wargaming. Despite the amount of effort wargaming requires, it has real value and 

provides stakeholders with advantages once participants have worked through 

scenarios. Ideally, both forms of wargaming—traditional and Free Kriegsspiel—

are taught to trainees in the context of historical use-cases, results, and lessons 

learned (Caffrey 2000).

Secondly, while the Army is a separate organization than the Special Forces, 

ARSOF training perspectives may offer insight into components of wargaming 

curricula. Special Forces personnel are trained and are qualified in both military 

skills and specific regional languages and cultures of the world. This study suggests 

the Army adopt a similar perspective for a wargaming specialization. Some 

wargaming students should specialize in traditional force-on-force wargaming, 

while others should specialize in Human Domain topics specific to different areas 

of the world. A holistic training perspective is likely to allow Army personnel to 

become more proficient and effective. Moreover, a holistic form of training 

supports the modern Army “Design” concept.

Thirdly, several existing Army-centric wargaming resources are available to 

students, namely the following 2 books:

• Student Text 100-9, Techniques and Procedures for Tactical DecisionMaking (1991)

• Student Text, Version 2, Art of Design (2010)

In an academic setting, several institutions offer courses in Computational Social 

Science. George Mason University is one such institution offering courses that 

teach ABM. Resources for such a course are also independently available; Dr Axtell 

(2013) recommends the following texts for CSS 610–Computational Analysis of 

Social Complexity:

• Epstein JM, Axtell R. Growing Artificial Societies: Social Science from the 

Bottom Up (1996)

• Leemis L, Park S. Discrete-Event Simulation: A First Course (2004)
Approved for public release; distribution is unlimited.

146

• Railsback SF, Grimm V. Agent-Based and Individual-Based Modeling: A 

Practical Introduction (2011)

• Schelling TC. Micromotives and Macrobehavior (1978)

• Heppenstall AJ, Crooks AT, See LM, Batty M. Agent-Based Models of 

Geographical Systems (2012)

• Shoham Y, Layton-Brown K. Multiagent Systems: Algorithmic, GameTheoretic, and Logical Foundations (2008)

• Weiss G. Multiagent Systems: A Modern Approach to Distributed Artificial 

Intelligence (1999)

Fourthly, several institutions, including the Army Culture Center and the Naval 

Postgraduate School, would likely provide valuable resources. The Army’s 

TRADOC Culture Center was established during the mid-2000s to improve 

Soldier’s cultural awareness to situations they would likely encounter (Bird 2007; 

Connable 2009). The Naval Postgraduate School offers wargaming training in other 

domains but it is likely that some crossover exists between curriculums (Naval 

Postgraduate School 2015). 

6.7.2 Theory

The theory component comprises several topics relevant to ABM, ranging from 

social sciences to math, and seeks to integrate computational techniques with 

cultural intricacies. This study identifies 6 topics, each of which could be an 

individual course. These include in order of priority, the following: 1) Culture and 

Modern-Day Tribes, 2) Complex Adaptive Systems, 3) Applied Complex Systems, 

4) Survey of Game Theory, 5) Applied Microsoft Excel Analysis, and 6) Applied 

Statistics.

A study of culture and modern-day tribes would identify the need to understand 

culture down to a tribal level. In the Middle East, a clear designation of a tribe may 

exist. However, in Western culture, a tribe may be more analogous to a company 

and the culture that it embodies. Culture both shapes the environment within which 

people operate and influences their perception of the world. Recent conflicts in both 

Iraq and Afghanistan have demonstrated a need to understand the underlying 

culture of an area of operations and conflict.

The study of complex adaptive systems would explore a field of study known as 

complexity science. Complex refers to dynamic interactions, where relationships 

are not aggregations of individual entities. Adaptive refers to change by individual 

entities and, because of interactions, collective behavior changes. Complex 
Approved for public release; distribution is unlimited.

147

adaptive systems are similar to multiagent systems, but complex adaptive systems 

focus on macro behavior whereas multiagent systems focus on individual behavior.

Applied complex systems are a second iteration on the aforementioned complex 

adaptive systems topic. However, this second iteration would emphasize the 

application and exploration of such systems. For example, it would incorporate the 

study of the rules within agents that cause emergent phenomena at the macrolevel.

A survey course of game theory and its derivatives would review related 

mathematical expressions of game theory. Agent-based modeling and game theory 

are slightly different because game theory assumes rational actors and homogenous 

agent information and ABM does not. However, game theory is still useful in 

gaining understanding of dynamics and interaction between agents. Its derivatives 

include behavioral game theory, evolutionary game theory, low-rationality game 

theory, prospect theory, and reflexive control.

Applied Microsoft Excel analysis would be a practical training course and would 

ensure trainees have some familiarity with data analysis and are proficient with 

Excel, so they can use spreadsheets to input data into simulations and use Excel to 

analyze results.

Applied statistics is the last recommended training topic. Statistics are essential to 

ABM because, for example, statistics represent decision-making distributions used 

by agents. It is also useful in characterizing the nature of the output produced from 

agent-based simulations.

6.7.3 Practice

The practice component identifies practical concerns one would need to know to 

use agent-based capabilities and tools. Available ABM tools would ideally use a 

form of version control technology, which allows users to snapshot individual 

iterations of models and their configurations. For example, version control 

technology allows individuals to share models and configurations, allows a third 

party to make changes, and allows the ability to receive and approve all, some, or 

none of the changes. Modeling tools may operate in a connected or disconnected 

mode. In a connected mode, properly built models can use large clusters of 

computers to assist in computations. In a disconnected mode, the user only has their 

local computer to perform computations. As a result, the user may be limited in the 

number of simulations that can characterize model results.

“Practice” also includes techniques used to develop and edit agent-based models. 

Similar to a mechanic working with an engine, the user should ideally have a 

working knowledge of how models operate. Trainees should be aware of existing 
Approved for public release; distribution is unlimited.

148

agent-based models and supporting capabilities. Finally, the trainees need to know 

how to use and interpret results displayed on the modeling tool’s user interface, 

such as maps, charts, and time-series representations. Trainees should also know 

how to analyze model data exported to Excel.

6.7.4 Proficiency

The proficiency component includes a combination of stateside support and regular 

certifications that evaluate whether the user is familiar and proficient with available 

modeling tools and existing agent-based models. Stateside support is a reach-back 

support mechanism available to tactical wargamers. If working properly, stateside 

support should collaboratively support the tactical user’s needs. Certifications 

ensure that the user is familiar with changes in published models and capabilities 

available from the stateside repository.

6.8 Summary

In this section, the report identified and discussed 4 recommendations for using an 

ABM capability to improve the rigor of tactical wargaming. Challenges to 

agent-based wargaming capacity include a wide breadth of research topics. The 

study identifies surveyed and comparable technologies, such as game theory and 

its derivatives, and detailed the agent-based potential, limitations, and risks. 

Finally, the study discusses a recommended technology and research path for the 

Army. The study concludes that ABM would increase the rigor of tactical 

wargaming, especially if the Army would like to consider the consequences and 

interactions of its actions with local populations.
Approved for public release; distribution is unlimited.

149

7. Conclusion

Future warfare will likely remain characterized by uncertainty and commanders, 

their planning staffs, and intelligence analysts will encounter a range of known, 

knowable, complex, and chaotic problems. ABM and its resulting emergent 

behavior is a potential solution to model terrain in terms of the Human Domain and 

improve the results and rigor of the traditional wargaming process.

Some risk is associated with developing a tactical Army agent-based capability. 

Researching and developing an Army-centric ABM capability is a nontrivial task. 

One of the concerns is allowing engineering to get ahead of the modeled science. 

If engineering outpaces science, there can be the illusion of contextual 

oversimplification that results in users not trusting the capabilities that agent-based 

wargaming otherwise contributes.

The Army will need to devote significant person-hours to perform leading research 

and development, adequate verification and validation, subsequent model creation, 

and resource revisions. Similarly, a nontrivial amount of money, appropriated 

throughout the range of Basic Research–Operational System Development money, 

and people, serving in both stateside and training positions, will be required.

Finally, the authors recommend the Army consider using a tactical agent-based 

capability during wargaming, down to the brigade. If the Army decides it does not 

want to use an agent-based capability, this report recommends at least being 

prepared for an adversary to use it.
Approved for public release; distribution is unlimited.

150

8. References

Abdollahian M, Baranick M, Efird B, Kugler J. Senturion: a predictive political 

simulation model. Washington (DC): National Defense University. Defense & 

Technology Papers. 2006 Jul.

Abt C. Serious games. Lanham (MD): University Press of America; 1968. ISBN 

978-0-8191-6148-2.

Adra S, McMinn P. Automated discovery of emergent misbehavior in agent-based 

models. Presentation slides. ABM Workshop; 2010 Jun 15.

Air Force Research Laboratory. Transition: organizational and cultural criteria for 

adversary modeling (OCCAM). 2011 Jan. White paper.

Ajzen, I. The theory of planned behavior. Organizational Behavior and Human

Decision Processes. 1991;50:179–211.

Alcorn B, Garces M, Lustick IS. Chapter XX: granular ABM simulations for 

operational use: forecasting and what-if experiments with models of Kandahar 

and Kunduz. 2012.

Allan R. Survey of agent based modelling and simulation tools. Warrington, UK:

Science and Technology Facilities Council; 2010. Report No.: DL-TR-2010-

007.

Alt JK, Jackson LA, Lieberman S. The cultural geography model: an agent based 

modeling framework for analysis of the impact of culture in irregular warfare.

US Army Training and Doctrine Command Analysis Center–Monterey, 

Monterey, CA. 2009:11–17.

Alt JK, Lieberman S. Agent frameworks for discrete event social simulations.

Proceedings of the 19th Conference on Behavior Representation in Modeling 

and Simulation. Charleston, SC. 2010 Mar 21–24.

Alt JK, Lieberman S, Anderson T. Visualizing the human, social, cultural and 

behavioral components of a complex conflict ecosystem. Naval Postgraduate 

School; Monterey (CA), 2010. White paper.

Anastasiou AB. Modeling urban warfare: joint semi-automated forces in urban 

resolve [master’s thesis]. [Wright-Patterson Air Force Base (OH)]: Air Force 

Institute of Technology; 2006.

Anderson GK, Kooij G, Briggman KLT, Hilbert JE, Lay CT, McNaughton JC.

From cooperation to competition – the future of US-Russian relations: a report 
Approved for public release; distribution is unlimited.

151

on an interdisciplinary wargame conducted by the US Army War College. US 

Army War College, Carlisle, PA. 2015 May.

Anderson JR, Bothell D, Byrne MD, Douglass S, Lebiere C, Qin Y. An integrated 

theory of the mind. Psychological Review. 2004;111(4):1036–1060.

Andrade P. Game theory and agent-based modelling for the simulation of 

geospatial phenomena [doctorate thesis]. [Sao Jose dos Campos (Brazil)] 

National Institute for Space Research; 2010.

Army ROTC. MSL II: foundations of leadership, section 3, introduction to terrain 

analysis. 2008; p. 244–265.

Aviv AJ, Sherr M, Blaze M, Smith JM. Moving targets: geographically routed 

human movement networks. University of Pennsylvania. Philadelphia, PA;

2010 Apr; Paper No. 926.

Axtell R. 120 million agents self-organize into 6 million firms: a model of the US 

private sector. Proceedings of the 15th International Conference on 

Autonomous Agents and Multiagent Systems. International Foundation for 

Autonomous Agents and Multiagent Systems. 2016 May 9–13; Singapore.

Axtell R. Adaptive play of games. Presentation slides. Trento Summer School;

2006a Jul.

Axtell R. What methodology would social scientists adopt if we could start over?,

Key Note Address presented at the Computational Social Science Society of 

the Americas 2015, Santa Fe (NM); 2015 Oct 29–Nov 1.

Axtell R. Agent-based computing in economics. Presentation slides. VII Trento 

Summer School; 2006b Jul.

Axtell R. Complex adaptive systems, computational social science and irregular 

warfare. Presentation slides. ONR Workshop on Human Interactions in 

Irregular Warfare as a Complex System; Atlanta, GA. 2011.

Axtell R. CSS 610: computational analysis of social complexity. Course syllabus. 

Fairfax (VA): George Mason University; 2013.

Axtell R. Pathologies of integrated assessments of climate change: representative 

agents vs. heterogeneous populations, rational response vs. behavioral 

adaptation, homogeneous and static beliefs vs. diverse and dynamic 

perceptions, technological stationarity in a non-stationary world, average 

effects vs. extremes, and neglect of poorly understood scientific issues. George 

Mason University; Fairfax (VA), 2014 Apr 10. White paper.
Approved for public release; distribution is unlimited.

152

Axtell R. The emergence of firms in a population of agents: local increasing returns, 

unstable Nash equilibria, and power law size distributions. Brookings 

Institution; Washington (DC), 1999. Working Paper No.: 3.

Axtell R. Why agents? on the varied motivations for agent computing in the social 

sciences. Brookings Institution; Washington (DC), 2000. White paper.

Axtell R, Axelrod R, Epstein JM, Cohen MD. Aligning simulation models: a case 

study and results. Computational and Mathematical Organization Theory.

1996;1:123–141.

Axtell R, Young HP. The changing face of economics. 2004; p. 251–290.

Axtell RL, Andrews CJ, Small MJ. Agent-based modeling and industrial ecology.

Journal of Industrial Ecology. 2002;5(4).

Axtell RL, Epstein JM. Agent-based modeling: understanding our creations. The 

Bulletin of the Santa Fe Institute. 1994.

Baillergeon R, Sutherland J. Tactics 101 094 – the light infantry platoon maneuver 

techniques. Blog post, http://www.armchairgeneral.com/tactics-101-094-thelight-infantry-platoon-maneuver-techniques.htm. 2014 Mar 12.

Bandura A. Social cognitive theory in cultural context. Applied Psychology: an 

International Review. 2002;51(2):269–290.

Bandura A. Social cognitive theory: an agentic perspective. Asian Journal of Social 

Psychology. 1999;2:21–41.

Barry P, Koehler M. Simulation in context: using data farming for decision support.

Proceedings of the 2004 Winter Simulation Conference; IEEE, 2004 Dec; p. 

814–819. 

Barry PS, Koehler MTK. Leveraging agent based simulation for rapid course of 

action development. Proceedings of the 2005 Winter Simulation Conference;

2005 Dec; p. 1001–1007.

Barry PS, Koehler MTK, Tivnan BF. Agent-directed simulation for systems 

engineering. The MITRE Corporation, McLean (VA), 2009. White paper; 

MITRE 09-0267.

Bathe MR, Frewer L. Modelling peace support operations: an agent-based 

approach. The Cornwallis Croup IX: Analysis for Stabilization and CounterTerrorist Operations. 2008; p. 98–105. ISBN 1896551599.

Beeker E, Bergen-Hill T, Henscheid ZA, Jacyna G, Koehler MTK, Litwin L, 

McLeod A, McMahon M, Mulutzie SK, Rothleder N, Shenoy R, Tivnan BF, 
Approved for public release; distribution is unlimited.

153

Wilk TJ, Turnley JG. COIN 1.0 Formulation. Bedford (MA): MITRE; 2010.

Technical Report No. MTR080315.

Belavkin RV. The role of emotion in problem solving. University of Nottingham; 

Nottingham (UK), 2002. White paper.

Bell B. Cognitive fidelity as a touchstone for on-demand team training. Presentation 

Slides. 14th BRIMS Conference; Universal City, CA. 2005 May 16–19.

Berzins V, Luqi SM, Saluto M, Williams J. Re-engineering the Janus (Army) 

combat simulation system. Monterey (CA): Naval Postgraduate School, US 

Army TRAC-Monterey; 1999. Technical Report. Naval Postgraduate School.

Bird J. TRADOC culture center. Presentation slides. TRADOC. 2007 Nov.

Bkassiny M, Li Y, Jayaweera SK. A survey on machine-learning techniques in 

cognitive radios. University of New Mexico; Albuquerque (NM). Working 

paper; 2012.

Blasch E. Sensor, user, mission (SUM) resource management and their interaction 

with level 2/3 fusion. Wright Patterson Air Force Base (OH) Air Force 

Research Lab; 2006. White paper.

Boiney J, Foster D. Progress and promise: research and engineering for human 

sociocultural behavior capability in the US Department of Defense. Report. 

2013 Jun; MITRE.

Bonabeau E. Agent-based modeling: methods and techniques for simulating human 

systems. PNAS. 2002;99(3):7280–7287.

Borrill PL, Tesfatsion L. Agent-based modeling: the right mathematics for the 

social sciences? Ames (IA); Iowa State University; 2010; Working Paper No.: 

10023.

Boyd J. Patterns of conflict. Presentation slides. 1986. 

Boyd J. The essence of winning and losing. Presentation slides. 1996.

Boyd JR, Spinney C, Richards C, Richards G. The essence of winning and losing.

Presentation slides. Atlanta, GA. 2010 Aug.

Bozkurt I. Developing an agent-based model on how different individuals solve 

complex problems. Journal of Industrial Engineering and Management.

2015;8(1):233–266.
Approved for public release; distribution is unlimited.

154

Brecher JA. Human domain effects in army special operations forces wargaming: 

a survey of modelling and simulation tools. Fort Bragg, NC. 2014 Dec 3; US 

Army Special Operations Command.

Breton R, Rousseau R, Price W. The integration of human factors in the design 

process: a triad approach. Report; 2002 Sep 13; Defence R&D Canada–

Valcartier. Universite Laval, Canada.

Bueno de Mesquita B. A new model for predicting policy choices: preliminary 

tests. Working paper; 2010.

Bueno de Mesquita B, Morrow JD, Siverson RS, Smith A. Policy failure and 

political survival: the contribution of political institutions. Journal of Conflict 

Resolution. 1999;43(2):147–161.

Burtsev MS, Korotayev A. An evolutionary agent-based model of pre-state warfare 

patterns: cross-cultural tests. Russian Academy of Sciences, Moscow, Russia. 

Dec 2004. White paper.

Caffrey M Jr. Toward a history based doctrine for wargaming. Aerospace Power 

Journal. Fall 2000.

Camus B, Bourjot C, Chevrier V. Multi-level modeling as a society of interacting 

models. Universite de Lorraine, France, 2013 Apr. White paper.

Carley KM. Networks and agents: the value of a multi-level approach to agentbased dynamic-network modeling. Carnegie Mellon. 2013 Aug; presentation.

Carmichael T, Hadzikadic M. Predator-prey dynamics and the red queen 

hypothesis: putting limits on the evolutionary arms race. Computational Social 

Science Society of the Americas 2015; White paper; 2015 Oct.

Cerri T, Chaturvedi A. Sentient world simulation (SWS): a continuously running 

model of the real world, a concept paper for comments, Purdue University, 

West Lafayette, IN. Concept Paper; Version 2.0; 2006 Aug 22.

Chamberlain RG, Duquette WH. Athena in 2013 and Beyond. NASA, Pasadena, 

CA. JPL Publication 13-9. 2013 May.

Chandola V, Banerjee A, Kumar V. Anomaly detection: a survey. ACM Computing 

Surveys. 2009;41(3), Article 15.

Chandra K. Constructivist theories of ethnic politics. New York (NY): Oxford 

University Press; 2012.

Chaturvedi A. Living transformation: ISAF-SEAS case study for urgent 

computing. Presentation slides. Purdue University, West Lafayette, IN. 2007.
Approved for public release; distribution is unlimited.

155

Chaturvedi A, Dolk D, Sebastian HJ. Agent-based simulation and model 

integration. Presentation slides. Workshop on Virtual Environment for 

Advanced Modeling; Honolulu, HI. Purdue University, Naval Postgraduate 

School, University of Aachen. Honolulu, HI. 2004 Jan 2–3.

Chaturvedi A. Simulex sample posters. Presentation slides. Simulex Inc. 2009.

Chaturvedi A. Synthetic environment for analysis and simulation, towards theorybased synthetic environment for computational experimentation. Presentation 

slides. JFCOM. 2004.

Chaturvedi A, Armstrong B, Chaturvedi R. Securing the food supply chain: 

understanding complex interdependence through agent-based simulation.

Health Technol. 2014;4:159–169.

Chaturvedi A, Chaturvedi R, Mulpuri M, Mellema S. Modeling stability and 

reconstruction operation using SEAS. Purdue University, Simulex Inc., West 

Lafayette (IN), 2008 Oct. White paper.

Chen S. Varieties of agents in agent-based computational economics: a historical 

and an interdisciplinary perspective. Journal of Economic Dynamics and

Control. National Chengchi University, Taiwan. 2011 Aug 31.

Childs MJ, Gallivan JJ. Organizing the BCT staff to enable reconnaissance and 

surveillance operations. Journal of the Military Intelligence Corps Association 

–the Vanguard. 2013;18(1):16–25.

Chotikul D. The soviet theory of reflexive control in historical and psychocultural 

perspective: a preliminary study. Monterey (CA): Naval Postgraduate School;

1986 Jul. Technical report.

Cioffi-Revilla C, O’Brien SP. Computational analysis in US foreign and defense 

policy. First International Conference on Computational Cultural Dynamics;

2007 Aug 27–28; College Park (MD). p. 26–36.

Cioffi-Revilla C, Rouleau M. MASON RebeLand: an agent-based model of 

politics, environment, and insurgency. George Mason University, Fairfax, VA.

Proceedings of the Human Behavior-Computational Modeling and 

Interoperability Conference; Oak Ridge National Laboratory, Oak Ridge (TN). 

2009a Jun 23–24.

Cioffi-Revilla C, Rouleau M. MASON AfriLand: a regional multi-country agentbased model with cultural and environmental dynamics. George Mason 

University, Fairfax, VA. Proceedings of the Human Behavior-Computational 
Approved for public release; distribution is unlimited.

156

Modeling and Interoperability Conference. Oak Ridge National Laboratory, 

Oak Ridge, TN. 2009b; Oak Ridge (TN).

Cioppa TM, Lucas TW. Efficient nearly orthogonal and space-filling latin 

hypercubes. Technometrics. 2007;49(1):45–55.

Cloppert M. Security intelligence: introduction (part two). SANS Digital Forensics 

and Incident Response. Blog post, 2013 March 22. [accessed 2016 Feb 19]. 

https://digital-forensics.sans.org/blog/2009/07/23/security-intelligence

-introduction-pt-2/.

Coles JP. Incorporating cultural intelligence into joint doctrine. IO Sphere–The 

Professional Journal of Joint Information Operations. Joint Information 

Operations Center; 2006 Spring. p. 7–13.

Coles J. Implementing ABM in practice: demonstration and discussion.

Presentation slides. CUBRC, Buffalo, NY. 2015 Oct 6.

Connable B, Perry WL, Doll A, Lander N, Madden D. Modeling, simulation, and 

operations analysis in Afghanistan and Iraq: operational vignettes, lessons 

learned, and a survey of selected efforts. RAND Corporation, Report; ISBN: 

978-0-8330-8211-4; 2014.

Connable B. All our eggs in a broken basket: how the human terrain system is 

undermining sustainable military cultural competence. Military Review, 

Combined Arms Center, Fort Leavenworth, KS. 2009 Mar–Apr; p. 57–64.

Conte R, Gilbert N, Bonelli G, Cioffi-Revilla C, Deffuant G, Kertesz J, Loreto V, 

Moat S, Nadal JP, Sanchez A, Nowak A, Flache A, San Miguel M, Helbing D.

Manifesto of computational social science. The European Physical Journal 

Special Topics. 2012;214:325–346.

Corby K. Deep learning and data embeddings: representing disparate data in a 

common vector space. Presentation slides. CCRi; Buffalo, NY. 2015 Oct 6.

Cordi M. A Study of the distribution of firm sizes: applying methods of physics on 

team dynamics [master of science thesis]. [Stockholm (Sweden)]: Royal 

Institute of Technology; 2014.

Counterinsurgency Training Center–Afghanistan. A counterinsurgent’s guidebook: 

the application of coin doctrine and theory, version 2. Kabul (Afghanistan): 

Camp Julien; 2011.

Counterinsurgency Training Center–Afghanistan. District stability framework 

quick reference guide. US Agency for International Development; 2010.
Approved for public release; distribution is unlimited.

157

Crowley JL. Intelligent systems: reasoning and recognition. Institut Polytechnique 

de Grenoble, Montbonnot, France. Lesson notes; 2012 Mar 16.

Crutchfield J, Wiesner K. Complexity: simplicity and complexity. Physics World.

2010 Feb;36–38.

Del Vecchio J, Hall D, Hanratty T, Heilman E, James A, Kaura M, Llinas J, 

Moskal M, Tucker S. COIC–future roles of fusion technology to support COIC 

operations. 2010 Sep 30. JIEDDO Report.

Del Vecchio J, Hall D, Hanratty T, Heilman E, James A, Kaura M, Llinas J, 

Moskal M, Tucker S. COIC information fusion operations and capabilities: an 

initial assessment. 2010 Jul 10. JIEDDO report.

[DHS] Department of Homeland Security. DHS risk lexicon. 2010 Sep.

Department of Instruction. So that’s what they mean by rigor. Summer Professional 

Development. Hamblen County, Morristown, TN. 2011 Jul 25. White paper.

Department of the Army. Army intelligence training strategy. 2014.

Department of the Army. The social impact model: a tool for irregular warfare 

adjudication, analysis, and validation. Presentation slides. TRADOC Analysis 

Center–Monterey. 2011 Feb.

Department of the Army. Operational environments to 2028: the strategic 

environment for unified land operations. TRADOC G-2, 2012 Aug 20.

Department of the Army. The United States Army functional concept for 

intelligence 2016-2028. Fort Monroe (VA). TRADOC Pamphlet 525-2-1;

2010 Oct 13.

Department of the Army. The US Army operating concept: win in a complex world 

2020-2040. Fort Eustis (VA). TRADOC Pamphlet 525-3-1; 2014 Oct 31.

Department of the Army. Commander’s appreciation and campaign design, version 

1.0. Fort Monroe (VA). TRADOC Pamphlet 525-5-500; 2008 Jan 28.

Department of the Army. US Army Functional Concept for Engagement. Fort 

Eustis (VA). TRADOC Pamphlet 525-8-5; 2014 Feb 24.

Dietvorst BJ, Simmons JP, Massey C. Algorithm aversion: people erroneously 

avoid algorithms after seeing them err. Journal of Experimental Psychology: 

General. American Psychological Association. University of Pennsylvania, 

Philadelphia (PA). 2014 Sep 25.
Approved for public release; distribution is unlimited.

158

Diez FJ, Druzdzel MJ. Reasoning under uncertainty. In Encyclopedia of Cognitive 

Science. Nadel L (Ed.), London (UK): Nature Publishing Group, 2003. p. 880–

886.

Dillon SM. Descriptive decision making: Comparing theory with practice. 

University of Auckland, Auckland, New Zealand: ORSNZ. 1998.

Dillon S. Descriptive decision making: comparing theory with practice. University 

of Waikato, New Zealand. 1998. White paper.

Doursat R, Read M, Halloy J. Introduction to complex systems and agent based 

modeling. Presentation slides. Awareness: Self-Awareness in Autonomic 

Systems. 2013 Aug.

Drogoul A, When ants play chess (or can strategies emerge from tactical 

behaviors?). Universite Paris Vi, Paris, France. 2007 Nov. White paper.

Ducote B. Challenging the application of PMESII-PT in a complex environment.

AY 2010-1; School of Advanced Military Studies, United Sates Army

Command and General Staff College, Fort Leavenworth, Kansas, 2010 Apr 

26.

Dunnigan JF. The complete wargames handbook. Richard Mataka. Lincoln (NE):

Writer’s Club Press; 2005.

Duong D. Verification, validation, and accreditation (VV&A) of social simulations. 

Augustine Consulting, Inc., Naval Postgraduate School, Monterey (CA); 2011. 

White paper.

Duong D, Bladon C. Chapter XX: Interfacing and validating models of the US 

Army TRAC tactical war game. Agent Based Learning Systems, Birmingham 

(AL). 2012 Sep. White paper.

Duong D, Pearman G, Baez F. Social impact module (SIM) integration. Monterey

(CA): TRADOC Analysis Center; 2011 Sep 16. Technical report.

Duong D, Pearman J. Simulating civilization change with nexus cognitive agents. 

Agent Based Learning Systems, Birmingham, AL. Augustine Consulting Inc., 

Monterey (CA). 2012 Jul.White paper.

Duong DV. Modeling the effects of international interventions with nexus network 

learner. Agent Based Learning Systems. Association for the Advancement of 

Artificial Intelligence 2012 Mar. AAAI Technical Report No.: SS-12-01.
Approved for public release; distribution is unlimited.

159

Duong D. The data absorption technique: coevolution to automate the accurate 

representation of social structure; Agent Based Learning Systems, 

Birmingham (AL). 2013 Aug. White paper.

Duong DV. The design of computer simulation experiments of complex adaptive 

social systems for risk based analysis of intervention strategies. Association 

for the Advancement of Artificial Intelligence. 2011. AAAI Technical Report

No.: SS-12-03.

Duong DV. The nexus cognitive agent social models. Presentation slides. 2009a.

Duong DV. Using complex adaptive systems to simulate information operations at 

the defense department. ACI Edge, Monterey, CA. 2009b; Complex Adaptive 

Systems and the Threshold Effect: Views from the Natural and Social 

Sciences: Papers from the AAAI Fall Symposium (FS-09-03), p. 47–52.

Duquette WH. Toward RESTful desktop applications. 19th Tcl/Tk Conference. Jet

Propulsion Laboratory, California Institute of Technology, Pasadena (CA). 

2012. White paper.

Egeth J, Klein GL, Schmorrow D, editors. Sociocultural behavior sensemaking: 

state of the art in understanding the operational environment. McLean (VA):

MITRE Corporation; 2014 Nov. Technical report.

Ehlschlaeger C, Flynn MT, Batson DE, Bradley E, Brandenberger J, Bullock B, 

Calvin K, Clarke L, Edmonds J, Ellis D, Eslami I, Farry M, Driffin S, 

Hejazi M, Hibbard K, Hill R, Imhoff M, Knotwell J, Leetaru KH, 

McAlinden R, Owen K, Pearson RP, Pfautz J, Pynadath D, Sisco J, 

Thomson A. Understanding megacities with the reconnaissance, surveillance, 

and intelligence paradigm. 2014 Apr. White volume.

Eloranta L. A Summary of computational social science, lecture 8 in introduction 

to computational social science. University of Helsinki, Helsinki, Finland. 

Presentation slides. 2015 May 11. [accessed 2016 Apr 5] 

http://www.slideshare.net/laurieloranta/lecture-8-a-summary-ofcomputational-social-science.

Endsley MR. Toward a theory of situation awareness in dynamic systems. Human 

Factors. 1995;37(1):32–64.

Epstein JM. Why model? Journal of Artificial Societies and Social Simulation.

2008 Oct 31;11(4 12).

Epstein JM, Axtell R. Growing artificial societies: social science from the bottom 

up; ISBN: 9780262550253. 1996.
Approved for public release; distribution is unlimited.

160

Ezhov AA, Terentyeva SS. Agent-based model heuristics in studying memory 

mechanisms. Psychology. 2014 Mar 28;(5):369–379.

Fischhoff B, Chauvin C. Intelligence analysis: behavioral and social scientific 

foundations. Washington (DC): The National Academies Press (US); 2011.

ISBN 978-0-309-17698-9.

Fitzpatrick PM. From first contact to close encounters: a developmentally deep 

perceptual system for a humanoid robot. [doctoral thesis]. [Boston (MA)]: 

Massachusetts Institute of Technology; 2003 Jun.

Flynn MT, Sisco J, Ellis DC. Left of bang: the value of sociocultural analysis in 

today’s environment. PRISM 3. 2012 Sep 4.

Flynn MT, Pottinger M, Batchelor PD. Fixing intel: a blueprint for making 

intelligence relevant in Afghanistan. Center for a New American Security. 

Washington, DC. 2010 Jan.

Folker Jr., RD. Intelligence analysis in theater joint intelligence centers: an 

experiment in applying structured methods. Occasional Paper Number Seven. 

Joint Military Intelligence College; 2000 Jan; Washington (DC).

Ford M, Rose P, Body H. COIN is dead–long live transformation. University of 

Hull. UK Defence Science and Technology Laboratory. Parameters. 2012;32–

43.

Fosset P, Andre-Poyaud I, Banos A, Beck E, Chardonnel S, Conesa A, Lang C,

Leysens T, Marilleau N, Piombini A, Thevenin T. Exploring intra-urban 

accessibility and impacts of pollution policies with an agent-based simulation 

platform: GaMiroD. Systems. France. 2016 Jan 16;4(5).

Fossum D, Painter LS, Williams VL, Yezril A, Newton EM. Discovery and 

innovation: federal research and development activities in the fifty states, 

District of Columbia, and Puerto Rico. RAND Corporation; 2000. ISBN 0-

8330-2844-8.

Frank AB. Agent-based modeling and intelligence analysis. Presentation slides.

Buffalo, NY. 2015 Oct 6.

Frank AB. Agent-based modeling in intelligence analysis [dissertation]. [Fairfax 

(VA)]: George Mason University; 2012.

Franks MD. Institutionalizing interdependence: US Army special operations 

forces/conventional forces ‘no turning back’. Carlisle Barracks (PA): US 

Army War College (US); 2013 May 1.
Approved for public release; distribution is unlimited.

161

Freksa C. Fuzzy systems in AI. Fuzzy Systems in Computer Science. Kruse R, 

Gebhardt J, Palm R, editors. P.155-169 Braunschweig/Wiesbaden, (Germany): 

Vieweg+Teubner Verlag; 1994.

Frewer L. The development of an agent-based model to investigate possible power 

law relationships in peacekeeping operations [doctoral thesis]. [Cranfield, 

England]: Cranfield University; 2007.

Fromm J. On engineering and emergence. Kassel University, Kassel (Germany). 

2006. White paper.

Fu WT, Bothell D, Douglass S, Haimson C, Sohn MH, Anderson J. Toward a realtime model-based training system. Interacting with Computers. 2006;18:1215–

1241.

Fulop J. Introduction to decision making methods. Hungarian Academy of 

Sciences, Budapest (Hungary). 2005 Oct 24. White paper.

Ganapathy S. Human-centered time-pressured decision making in dynamic 

complex systems [doctoral thesis]. [Dayton (OH)]: Wright State University; 

2006 Jun 9.

Garces M, Alcorn B. Nesting a data-driven agent-based model within a theoretical 

framework. Lustick Consulting, Narberth, PA. International Workshop on 

Advances in Computational Social Science; 2012.

Gaud N, Galland S, Hilaire V, Koukam A. An organizational platform for holonic 

and multiagent system. Sixth International Workshop on Programming MultiAgent Systems, Seventh International Conference on Autonomous Agents and 

Multiagent Systems; 2008; Estoril, Portugal.

Ghahramani Z. Unsupervised learning. University College, London, UK. Chapter; 

Advanced Lectures on Machine Learning LNAI 3176; 2004 Sep 16.

Gigerenzer G, Todd PM. Simple heuristics that make us smart. Oxford University 

Press; 1999. ISBN 0-19-512156-2.

Gigras Y, Gupta K. Artificial intelligence in robot path planning. International 

Journal of Soft Computing and Engineering (IJSCE). ITM University, India. 

Banasthali, India. 2012 May;2(2).

Gilbert N. Agent-based models. University of Surry, Guildford (UK). 2007 Nov; 

White paper.

Gow J. Artificial intelligence, 4. Knowledge representation. Course V231.

Presentation slides. Imperial College; London, UK; 2003.
Approved for public release; distribution is unlimited.

162

Green KL. Complex adaptive systems in military analysis. Institute for Defense 

Analysis, Alexandria (VA). 2011 May; IDA Document-4313.

Greer JK, The network vs. the BCT: organizational overmatch in hybrid strategies.

Small Wars Journal. Journal Article. 2013 Nov 18.

Grignard A, Vo DA, Drogoul A, Zucker JD. Using high-level 3-D graphics library 

in agent-based simulation platform. Advances in Smart Systems Research.

2013;3(1):1–6.

Grigoryev I. AnyLogic 7 in three days: a quick course in simulation modeling. 2nd 

Ed. 2015. ISBN 978-1508933748.

Guerin S, Kunkle D. Emergence of constraint in self-organizing systems. Redfish 

Group, Santa Fe, NM. Nonlinear Dynamics, Psychology, and Life Sciences.

2004 Apr;8(2). Society for Chaos Theory in Psychology & Life.

Haazebroek P, van Dantzig S, Hommel B. A computational model of perception 

and action for cognitive robotics. Cognitive Process. 2011 May 20;12: 355–

365. DOI 10.1007/s10339-011-0408-x.

Hajek M. Neural Networks. Book; 2005.

Hall DL. Lesson 2: the JDL model. Presentation slides; 2008 Jul.

Hamill L, Gilbert N. A simple but more realistic agent-based model of a social 

network. University of Surrey, Guildford, UK. 2008 Jul.

Hamill L, Gilbert N. Simulating large social networks in agent-based models: a 

social circle model. University of Surry, UK. E:CO. 2010;12(4):78–94.

Hanratty T, Richardson J, Mittrick M, Dumer J, Heilman E, Roy H, Kase S. Human 

terrain exploitation suite: applying visual analytics to open source information.

ARL, Aberdeen, MD. Conference Paper; 2014 April.

Harris R. Introduction to decision making. Virtual Salt; Online Article; 2012 Jun 

09. [accessed 2016 Jun 20] http://www.virtualsalt.com/crebook5.htm.

Hartley III DS. Comparing validation results for two DIME/PMESII models: 

understanding coverage profiles. Hartly Consulting, Oak Ridge, TN. 

Proceedings of the 2010 Winter Simulation Conference. Johansson B, Jain S, 

Montoya-Torres J, Hugan J, Yucesan E, editors. IEEE 978-1-4244-9864-2. 

2010; p. 428–440.

Hartley D. DIME/PMESII tools: past, present and future. Presentation slides. 

Hartley Consulting; Oak Ridge, TN. 2008 Jan 21.
Approved for public release; distribution is unlimited.

163

Hawthorne C, Scheidt D. Moving emergent behavior algorithms from simulation 

to hardware: command and control of autonomous UxV’s. John Hopkins 

Applied Physics Laboratory; 2005 Laurel, MD. 10th International Command 

and Control Research and Technology Symposium.

Haywood OG Jr. Military doctrine of decision and the von Neumann theory of 

games. RAND Corporation, Santa Monica (CA). 1951 Feb 20. ASTIA 

Document No.: ATI 210383.

Headquarters, Department of the Army. Unified land operations. Washington (DC): 

Headquarters, Department of the Army; 2011 Oct 10. ADP 3-0.

Headquarters, Department of the Army. The operations process. Washington (DC): 

Headquarters, Department of the Army; 2012 May 17. ADP 5-0.

Headquarters, Department of the Army. Terms and military symbols. Washington 

(DC): Headquarters, Department of the Army; 2015 Dec 7. ADRP 1-02.

Headquarters, Department of the Army. Intelligence. Washington (DC): 

Headquarters, Department of the Army; 2012 Aug 31. ADRP 2-0.

Headquarters, Department of the Army. The Operations Process. Washington 

(DC): Headquarters, Department of the Army; 2012 May 17. ADRP 5-0.

Headquarters, Department of the Army. Mission command. Washington (DC): 

Headquarters, Department of the Army; 2014 Mar 28. ADRP 6-0, C2.

Headquarters, Department of the Army. Civil affairs civil information 

management. Washington (DC): Headquarters, Department of the Army; 2013

Sep 6. ATP 3-57.50.

Headquarters, Department of the Army. Commander and staff officer guide.

Washington (DC): Headquarters, Department of the Army; 2011 Sep14. ATTP 

5-0.1.

Headquarters, Department of the Army. Staff organization and operations.

Washington (DC): Headquarters, Department of the Army; 1997 May 31. FM 

101-5.

Headquarters, Department of the Army. Tactics in counterinsurgency. Washington 

(DC): Headquarters, Department of the Army; 2009 Apr 21. FM 3-24.2.

Headquarters, Department of the Army. The brigade combat team. Fort Knox (KY): 

Headquarters, Army Armor Center (US); 2009 Jul. FM 3-90.6.

Headquarters, Department of the Army. The operations process. Washington (DC): 

Headquarters, Department of the Army; 2010 Mar 26. FM 5-0.
Approved for public release; distribution is unlimited.

164

Headquarters, Department of the Army. Commander and staff organization and 

operations. Washington (DC): Headquarters, Department of the Army; 2016

Apr 22. FM 6-0, C2.

Headquarters, Department of the Army. The operations process. Washington (DC): 

Headquarters, Department of the Army; 2008 Mar 14. FMI 5-0.1.

Headquarters, Department of the Army. Intelligence preparation of the 

battlefield/battlespace. Washington (DC): Headquarters, Department of the 

Army; 2015 Mar 26. US Marine Corps, ATP 2-01.3, MCRP 2-3A.

Headquarters, Department of the Army. Soldier’s manual and trainer’s guide for 

MOS 350F, officer foundation standards all source intelligence technician, 

CW2, CW3, CW4, and CW5. Washington (DC): Headquarters, Department of 

the Army; 2006 Apr 20. STP 34-350F-SM-TG.

Helbing D, Balietti S. How to do agent-based simulations in the future: from 

modeling social mechanisms to emergent phenomena and interactive systems 

design. ETH Zurich, Zurich, Switzerland. Santa Fe Institute Working Paper 

2011-06-024; 2011.

Helbing D. Social self-organization: understanding complex systems. 2012. 

Chapter 2, Agent-Based Modeling; p. 25–70. Springer-Verlag Berlin 

Heidelberg. DOI 10.1007/978-3-642-24004-1 2.

Heppenstall AJ, Crooks AT, See LM, Batty M, editors. Agent-Based Models of 

Geographical Systems. 2012. Springer. DOI 10.1007/978-90-481-8926-7.

Herbert M. The human domain: the Army’s necessary push toward squishiness.

Army Capabilities Integration Center, Fort Eustis, VA. Military Review, 2014 

Sep–Oct. p. 81–87.

Heylighen F. The science of self-organization and adaptivity. Free University of 

Brussels (Belgium); 1999 May. White paper.

Hodge G. Wargaming courses of action during other-than-major combat 

operations. US Army Command and General Staff College, Fort Leavenworth, 

KS. Small Wars Journal. 2012 Jun 22.

Hodges JS, Dewar JA. Is it you or your model talking? a framework for model 

validation. RAND Corporation, Santa Monica, CA. 1992. R-4114-AF/A/OSD.

Hussain I, Knapen L, Arsalan KM, Bellemans T, Janssens D, Wets G. Agent-based 

negotiation model for long-term carpooling: a flexible mechanism for trip 

departure times. Hasselt University, Belgium. WIT Transactions on the Built 

Environment. 2015 Apr; vol. 146. DOI:10.2495/UT150371.
Approved for public release; distribution is unlimited.

165

Jennings NR, Sycara K, Wooldridge M. A roadmap of agent research and 

development. Queen Mary and Westfield College, London, UK. Carnegie 

Mellon University, Pittsburgh, PA. Autonomous Agents and Multi-agent 

Systems. 1998;1:7–38. Kluwer Academic Publishers, Boston (MA).

Jiangl Z. Model and method for a multi-agent based ammunition consumption 

calculating. Google Cache; [accessed 2015 Oct 9]. http://en.cnki.com.cn

/Article_en/CJFDTOTAL-HLYZ201305009.htm.

Johnson RT, Lampe TA, Seichter S. Calibration of an agent-based simulation 

model depicting a refugee camp scenario. In: Hill RR, Johansson B, Dunkin 

A, Ingalls RG, editors. Naval Postgraduate School, Monterey, CA. EADS, 

Unterschleissheim, Germany. Bundeswehr Transformation Center, Ottobrunn, 

Germany. Proceedings of the 2009 Winter Simulation Conference; Rossetti 

(MD): 2009; p. 1778–1786.

Johnston KM. Agent analyst: agent-based modeling in ArcGIS. ESRI, Redlands

(CA). EsriPress; 2013. ISBN: 9781589483231.

Joint Chiefs of Staff. Joint concept for human aspects of military operations. Draft 

version 0.5; 2015a Oct 5.

Joint Chiefs of Staff. The United States military’s contribution to national security.

2015b Jun.

Joint Chiefs of Staff. Joint Intelligence. 2013 Oct 22; JP 2-0.

Joint Chiefs of Staff. Joint Operation Planning. 2011 Aug 11; JP 5-0.

Joint Special Operations University. Special operations forces interagency 

counterterrorism reference manual. MacDill Air Force Base (FL): Joint 

Special Operations University Press (US); 2013 Sep.

Jones RM, Laird JE, Nielsen PE, Coulter KJ, Kenny P, Koss FV. Automated 

intelligent pilots for combat flight simulation. AI Magazine. 1999;20(1). 

American Association for Artificial Intelligence. 0738-4602-1999. p. 27–41.

Jones RM, Stensrud BS, Taylor GE. Knowledge-rich intelligent agents for SOF

training. Soar Technology, Inc. Waterville, ME, Orlando, FL, Ann Arbor, MI. 

Special Operations Forces Industry Conference (SOFIC); 2009.

Kaber DB, Endsley MR. The effects of level of automation and adaptive 

automation on human performance, situation awareness and workload in a 

dynamic control task. Theoretical Issues in Ergonomics Science. Taylor & 

Francis Ltd. 2003;1–40. DOI 10.1080/1463922021000054335.
Approved for public release; distribution is unlimited.

166

Kai-jia T, Shuai P, Yong-Qiang Q. Research on military simulation system based 

on multi-agent. National Defence University, Beijing, China. Proceedings of 

2012 International Conference on Mechanical Engineering and Material 

Science; Atlantic Press. 2012 Nov. p.708–710. doi:10.2991/mems.2012.104.

Kasparov G. The chess master and the computer. The New York Review of Books.

Online article, [accessed 2016 Jun 20] http://www.nybooks.com

/articles/2010/02/11/the-chess-master-and-the-computer/; 2010.

Kass S. Decision making: human factors psychology. Presentation slides.

University of West Florida; Pensacola, FL. 2002.

Kennedy R. Making useful conflict predictions: methods for addressing skewed 

classes and implementing cost-sensitive learning in the study of state failure. 

University of Houston (TX); 2014 Mar 20. White paper.

Kettler B. Bringing socio-cultural modeling and forecasting from open source intel 

to the tactical domain. Presentation slides. Lockheed Martin ATL; Buffalo, 

NY. 2015 Oct 6.

Kettler B, Lautenschlager J. Expeditionary modeling for population-centric 

operations in megacities: some initial experiments. Lockheed Martin ATL

Proceedings of the 7th International Conference on Applied Human Factors 

and Ergonomics; 2016 Jul 27–31; Orlando, FL.

Klugl F, Bazzan ALC. Agent-based modeling and simulation. Orebro University, 

Sweden. University of Rio Grande do Sul, Brazil. Association for the 

Advancement of Artificial Intelligence; ISSN 0738-4602. AI Magazine. 2012

p.29–40.

Klugl F, Karlsson L. Towards pattern-oriented design of agent-based simulation 

models. Orebro University, Orebro (Sweden); 2009 Jun 21. White paper.

Koehler MTK, Barry PS, Widdowson BL, Forsyth AJ. Case study: using agents to 

model stability and support operations. Conference on Social Dynamics: 

Interaction, Reflexivity and Emergence. 2004 Oct 7–9; Chicago, IL. p. 667–

676. ISBN 0-9679168-5-2. Cosponsored by Argonne National Laboratory and 

The University of Chicago, Chicago (IL).

Kolodner JL. An introduction to case-based reasoning. Georgia Institute of 

Technology; Atlanta, GA. Artificial Intelligence Review. 1992;6:3–34.

Kope A, Rose C, Katchabaw M. Modeling autobiographical memory for believable 

agents. Proceedings of the Ninth AAAI Conference on Artificial Intelligence 

and Interactive Digital Entertainment; 2013; University of Western Ontario, 
Approved for public release; distribution is unlimited.

167

Ontario, Canada. p. 23–29. Association for the Advancement of Artificial 

Intelligence.

Kotenko I. Agent-based modelling and simulation of network cyber-attacks and 

cooperative defence mechanisms. St. Petersburg Institute for Informatics and 

Automation of Russian Academy of Sciences, St. Petersburg, Russia. Discrete 

Event Simulations, ISBN: 978-953-307-115-2, InTech; 2010.

Kotenko I, Stepashkin M, Ulanov A. Agent-based modeling and simulation of 

malefactors’ attacks against computer networks. Intelligence Systems 

Laboratory, Russia. Security and Embedded Systems. Serpanos DN, Giladi R, 

editors. IOS Press. 2006;139–146.

Kotsiantis SB. Supervised machine learning: a review of classification techniques.

Informatica. University of Peloponnese, Greece. 2007 Jul 16;31:249–268.

Kott A, Hansberger J, Waltz E, Corpac P. Whole-of-government planning and 

wargaming of complex international operations: experimental evaluation of 

methods and tools. The International C2 Journal. 2010;4(3).

Kott A, Corpac PS. COMPOEX technology to assist leaders in planning and 

executing campaigns in complex operational environments. 12th International 

command and Control Research and Technology Symposium; 2007 Jun 19; 

Newport, RI.

Kott A, Corpac PS. Technology to assist leaders in planning and executing 

campaigns in complex operational environments: conflict modeling, planning 

and outcomes experimentation program (COMPOEX). DARPA. 2007; 

Arlington, VA.

Kretchik WE. The manual wargaming process: does our current methodology give 

us the optimum solution? School of Advanced Military Studies, US Army 

Command and General Staff College, Fort Leavenworth, KS. 1991 Dec 22. 

92-32401.

Langley P. Progress and challenges in interactive cognitive systems. Presentation 

slides. University of Auckland; Auckland, New Zealand. 2015 Dec.

Larson EV, Darilek RE, Gibran D, Nichiporuk B, Richardson A, Schwartz LH, 

Thurston CQ. Foundations of effective influence operations: a framework for 

enhancing army capabilities. Santa Monica (CA): RAND Corporation; 2009.

DAPRR05007. ISBN 978-0-8330-4404-4

Latek MM, Rizi SMM, Crooks AT, Fraser M. A spatial multi-agent model of border 

security for the Arizona–Sonora borderland; George Mason University, 
Approved for public release; distribution is unlimited.

168

Fairfax, VA. EADS North America, Herndon, VA. CSSSA 2012 Conference 

Paper. 2012.

Laufer MS. Artificial intelligence in humans. 2013 Oct 30. White paper.

arXiv:1311.0716.

Lazer D, Pentland A, Adamic L, Aral S, Barabasi AL, Brewer D, Christakis N, 

Contractor N, Fowler J, Gutmann M, Jebara T, King G, Macy M, Roy D, 

Alstyne MV. Computational social science. Science 323, 721. 2009 Feb 6;323.

p. 721–723. DOI: 10.1126/science.1167742

Leemis L, Park S. Discrete-Event Simulation: A First Course. Williamsburg (VA): 

The College of William & Mary; 2004.

Lewis D. Predictive analysis: an unnecessary risk in the contemporary operating 

environment. School of Advanced Military Studies, US Army Command and 

General Staff College, Fort Leavenworth, KS. 2004 May 26.

Li W, Sadigh D, Sastry SS, Seshia SA. Synthesis for human-in-the-loop control 

systems. SRI International, Menlo Park, CA. University of California, 

Berkeley, CA. 2014 Feb 11. White paper.

Livermore WR. The American kriegsspiel: a game for practicing the art of war 

upon a topographical map. Cambridge (MA): The Riverside Press; 1882.

Lustick IS. Validating and verifying validation and verification: the methodological 

challenge of a public policy imperative. University of Pennsylvania, 

Philadelphia, PA. American Political Science Association Annual Meeting;

2012 Aug 30–Sep 2; New Orleans (LA).

Macal CM, North MJ. Agent-based modeling and simulation: ABMS examples.

Proceedings of the 2008 Winter Simulation Conference; InterContinental 

Hotel; Miami, FL, USA, December 7–10, 2008; p. 101–112.

Macal CM, North MJ. Introduction to agent-based modeling and simulation.

Argonne National Laboratory; Presentation slides. MCS LANS Informal 

Seminar, Argonne National Laboratory; Argonne, IL. Argonne. 2006 Nov 29. 

MacCalman M, MacCalman A, Wilson G. Visualizing social networks to inform 

tactical engagement strategies that will influence the human domain. Naval 

Postgraduate School, Monterey, CA. United States Military Academy, West 

Point, NY. Small Wars Journal. 2013 Aug 15.

Mahalik MR, Conzelmann G, Stephan CH, Mintz MM, Veselka TD, Tolley GS, 

Jones DW. Modeling the transition to hydrogen-based transportation.

Proceedings of the Agent 2007 Conference on Complex Interaction and Social 
Approved for public release; distribution is unlimited.

169

Emergence; 2007 Nov 15–17; Evanston (IL). Argonne National Laboratory;

Argonne, IL RCF Economic and Financial Consulting, Inc. Chicago, IL.

Mahoney S, Comstock E, deBlois B, Darcy S. Aggregating forecasts using a 

learned bayesian network. Proceedings of the Twenty-Fourth International 

Florida Artificial Intelligence Research Society Conference, Association for 

the Advancement of Artificial Intelligence; 2011 May 18–20; Palm Beach, FL. 

Vienna (VA): Innovative Decisions, Inc., p.626–631.

Makarov VL, Bakhtizin AR. Agent-based model for simulation of terrorism in 

Russia’s Caucasus. Central Economics and Mathematics Institute of Russian 

Academy of Sciences. Moscow (Russia); 2010 Apr 27. White paper.

Makowsky MD, Smaldino PE. The evolution of power and the divergence of 

cooperative norms. John Hopkins University, Baltimore, MD. University of 

California, Davis (CA). 2014 Apr 24. White paper. doi: 

http://dx.doi.org/10.1101/004416.

Marinier RP III , Laird JE, Lewis RL. A computational unification of cognitive 

behavior and emotion. University of Michigan, Ann Arbor, MI. Cognitive 

Systems Research. 2008 Mar 31. doi:10.1016/j.cogsys.2008.03.004

Matei SA. Social cognitive theory, social learning, self-efficacy and social media.

Blog Post. 2010 Jul 22 [accessed 2016 Feb 23]. 

http://matei.org/ithink/2010/07/22/bandura-last-hurrah-an-integrated-social

-psychological-theory-of-mass-communication-with-a-cognitive-twist/.

Maxwell DT, Carley KM. Principles for effectively representing heterogeneous 

populations in multi-agent simulations. Innovative Decisions, Vienna, VA. 

Carnegie Mellon University, Pittsburgh, PA. Complex systems in knowledgebased environments: theory, models and applications. Studies in 

Computational Intelligence. 2008 Jul 14;168:199–227.

Maybury MT. Science and technology for the human domain. Presentation slides. 

Air Force (US); Washington, DC. 2011 Feb 8.

McCarthy J, Minsky ML, Rochester N, Shannon CE. A proposal for the Dartmouth 

summer research project on artificial intelligence. 1955 Aug 31. [accessed 

2016 Jun 20]. http://www-formal.stanford.edu/jmc/history/Dartmouth

/dartmouth.html.

McDonnell JP. National strategic planning: linking DIMEFIL/PMESII to a theory 

of victory [master’s thesis]. [Norfolk (VA)]: Joint Forces Staff College; 2009

Dec 6.
Approved for public release; distribution is unlimited.

170

McLamb WB. The US Army’s design doctrine: a solution to the ills of the 

operations planning processes? Newport (RI): Naval War College. 2009 May 

4. White paper.

Merriam-Webster. Definition of artificial intelligence. 2016. [accessed 2016 Jun 

20]. http://www.merriam-webster.com/dictionary/artificial intelligence. 

Meyer D. Complexity, antifragility, and integrated theories: some initial thoughts 

(and more questions than answers). Presentation slides. NCRG IETF 83; Paris, 

France. 2012 Mar 26.

Michel RR. Historical development of the estimate of the situation. Alexandria 

(VA): Army Research Institute for the Behavioral and Social Sciences (US);

1990 Oct. Research Report No.: 1577.

Michel S, Megerdoomian K. Modeling community resilience for a post-epidemic 

society. George Mason University; Fairfax (VA). 2015 Oct 20. White paper.

Miller JH. Active nonlinear tests (ANTs) of complex simulation models. Carnegie 

Mellon University; Pittsburgh (PA). 1996 Feb 5. White paper.

Mooney RJ. CS 343: artificial intelligence bayesian networks. Presentation slides.

University of Texas at Austin; Austin, TX. 2010.

Moore DT. Sensemaking: a structure for an intelligence revolution. Washington 

(DC): National Defense Intelligence College. NDIC Press; 2011 Mar. ISBN: 

978-1-932946-28-4.

Moore J. America’s new strategic reality: irregular world war. Arlington (VA): 

Muir Analytics. Small Wars Journal. 2015 Jun 24.

Moutari S. Unsupervised learning: clustering. Presentation slides. Queen’s 

University; Belfast, Ireland. 2013 Sep 17.

Nakayama MK. Confidence intervals for quantiles when applying replicated Latin 

hypercube sampling and sectioning. New Jersey Institute of Technology;

Newark (NJ). 2012 Sep 24. White paper.

National Highway Traffic Safety Administration. Preliminary Statement of Policy 

Concerning Automated Vehicles. Washington DC. 2013 May 29.

Naval Postgraduate School Cyber Academic Group. [accessed 2015 Dec 16]. 

http://www.nps.edu/Academics/Schools/GSOIS/Departments/CAG

/Education/degrees.html.
Approved for public release; distribution is unlimited.

171

Nilsson NJ. Human-level artificial intelligence? be serious! American Association 

for Artificial Intelligence, AI MAGAZINE. 25th Anniversary Issue. Winter 

2005. 0738-4602-2005. p. 68–75.

Ntuen CA. The process of sensemaking in complex human endeavors. North 

Carolina A&T State University. International Command and Control Research 

and Technology Symposium; 2008 Jun 17–19; Bellevue, WA.

Odell J. Objects and agents compared. Journal of Object Technology. 2002 May–

Jun;1(1). Published by ETH Zurich, Chair of Software Engineering.

ODNI. Intelligence community directive 203–analytic standards. 2015 Jan 2.

Oluyomi AO. Patterns and protocols for agent-oriented software development [PhD 

thesis]. [Melbourne (Australia)]: University of Melbourne; 2006 Nov.

Oxford Dictionary of difficult words. 2016. [accessed 2016 Mar 7]. http://www

.oxforddictionaries.com/us/definition/english/rigour.

Pajares F. Overview of social cognitive theory and of self-efficacy. Atlanta (GA); 

2002. Emory University. [accessed 2015 Oct 27]. http://www.emory

.edu/EDUCATION /mfp/eff.html.

Paradis S, Brenton R, Roy J. Data fusion in support of dynamic human decision 

making. Defence Research Establishment; Valcartier, Canada. 1999 Jun 14.

White paper.

Parunak HVD. ‘Go to the ant’: engineering principles from natural multi-agent 

systems. Industrial Technology Institute. Annals of Operations Research, 

special issue on Artificial Intelligence and Management Science. Ann Arbor 

(MI): 1996.

Peck M. Successful war games combine both civilian and military traits. NDIA 

National Defense. Online article; 2003 Nov. [accessed 2015 Nov18].

http://www.nationaldefensemagazine.org/archive/2003/November/Pages

/Successful_War3734.aspx.

Perez C. A practical guide to design: a way to think about it, and a way to do it.

Fort Leavenworth (KS): Army Command and General Staff College (US):

Military Review. 2011 Mar–Apr;91(2). p. 41–51.

Pfautz J, Koelle D, Prue B, Lofdahl C. Challenges in the operational use and 

validation of sociocultural techniques, tools, and models. Presentation slides, 

2015 NDIA Human-Systems Integration Conference, Session 4: Social, 

Cultural, Behavioral Understanding; 2015 Feb 10. Alexandria, VA. Boston 

(MA): Charles River Analytics.
Approved for public release; distribution is unlimited.

172

Pike T, Brown EJ. Populations as complex adaptive systems: a case study of 

corruption in Afghanistan. Small Wars Journal. 2011 Aug 26.

Pramukkul P, Svenkeson A, Grigolini P, Bologna M, West B. Complexity and the 

fractional calculus. Denton (TX): University of North Texas. [Arica, Chile]: 

Universidad de Tarapaca. Research Triangle Park (NC): Army Research 

Office. Hindawi Publishing Corporation, Advances in Mathematical Physics.

2013 Mar 28. Volume 2013, Article ID 498789. 

http://dx.doi.org/10.1155/2013/498789.

Prevette M, Snyder D. Synthetic environment for analysis and simulation (SEAS).

Presentation slides. JFCOM/J9. 2004.

Price JC. Game theory and U-boats in the bay of Biscay [thesis]. [Wright-Patterson 

Air Force Base (OH)]: Air Force Institute of Technology; 2003.

Protopsaltis A. Learning theories and serious games. Coventry, UK: Serious Games 

Institute. 2011 Sep. [accessed 2016 Feb 9]. http://www.slideshare.net

/helenaxe/learning-theories-and-serious-games.

Railsback SF, Grimm V. Agent-based and individual-based modeling: a practical 

introduction. ISBN: 9780691136745; 2011.

Rajeswari PVN, Prasad TV. Hybrid systems for knowledge representation in 

artificial intelligence. Kavali, India: Visvodaya Technical Academy. 

International Journal of Advanced Research in Artificial Intelligence. 2012

Nov 10;1(8).

Rand W, Rust RT. Agent-based modeling in marketing: guidelines for rigor.

College Park (MD): University of Maryland. Intern. J. of Research in 

Marketing. 2011. doi:10.1016/j.ijresmar.2011.04.002.

Ray O. Hybrid abductive inductive learning [doctoral thesis]. [London (England)]:

University of London; 2005 Dec.

Raymond D. Human domain mapping in 21st century warfare. Fort Campbell

(KY): 5th Special Forces Group. Small Wars Journal. 2015 Aug 22.

Reitter D, Lebiere C. Towards cognitive models of communication and group 

intelligence. Pittsburgh (PA): Carnegie Mellon University. 2011 Jul 7. p. 734–

739.

Richards C. Boyd’s OODA loop (it’s not what you think). J. Addams & Partners, 

Inc.; Atlanta (GA). 2012 Mar 21.White paper.
Approved for public release; distribution is unlimited.

173

Richardson R, Richiardi M, Wolfson M. We ran one billion agents. Scaling in 

simulation models. University of Oxford; Oxford, UK: Moncaliero, Italy: 

Collegio Carlo Alberto. Ottawa, Canada: University of Ottawa. 2015 Jun 16. 

White paper.

Ripley RS. Converting the JNEM training aid to a forecasting tool. US Army 

Command and General Staff College; Fort Leavenworth (KS). 2008 Dec 5. 

Monograph.

Ritchey T. General morphological analysis: a general method for non-quantified 

modelling. Swedish Morphological Society. 2014 Feb 5. White paper.

Ritchey T. Wicked problems–social messes: decision support modelling with 

morphological analysis. Berlin Heidelberg (Germany): Springer-Verlag; 2011. 

ISBN 978-3-642-19653-9. p. 7–18.

Robel MK. The difference between military and civilian wargames. Strategy 

World; online article; 2004 Jun 18. [accessed 2015 Nov 18]. 

https://www.strategypage.com/dls/articles2004/2004618.asp.

Rojas R. Neural networks: a systematic introduction. Berlin (Germany): SpringerVerlag; 1996.

Rueschhoff JL, Dunne JP. Centers of gravity from the ‘inside out’. NDU Press. 

Joint Forces Quarterly. 2011; 1st quarter (60).

Ruvinsky A, Hingst R, Hoffman M, Kettler B. Chapter 16: Enabling operational 

users to fully exploit the tools, data, and analytics to shape potential futures.

Lockheed Martin Advanced Technology Laboratories. MITRE Case Number 

14-2487. 2014. p. 434–451.

Ryan SE. Concept for cognitive joint force entry. Presentation slides, Version 1.4.

Army Special Operations Command (US). 2014 Nov 4.

Salerno J. The national operational environment model (NOEM): understanding 

the operational environment. Presentation slides. AFRL/RIEF 

EUCOM/AFRICOM S and T Conference; Social Behavioral Working Group.

2010 Jun 16.

Salmoni BA, Holmes-Eber P. Operational culture for the warfighter: principles and 

applications. Quantico (VA): Marine Corps University Press; 2011.

Schelling TC. Micromotives and macrobehavior. WW Norton & Company; 1978.

Schelling TC. Social mechanisms and social dynamics. 2011 Dec 13. p. 32–44
Approved for public release; distribution is unlimited.

174

Scheutz M, Madey G, Boyd S. tMANS–the multi-scale agent-based networked 

simulation for the study of multi-scale, multi-level biological and social 

phenomena. University of Notre Dame; Notre Dame (IN). 2005 Jan 10. White 

paper.

Schieritz N, Grobler A. Emergent structures in supply chains–a study integrating 

agent-based and system dynamics modeling. Mannheim University. 

Proceedings of the 36th Hawaii International Conference on System Sciences;

2002 Oct 1; Mannheim, Germany.

Schmorrow D. Sociocultural behavior analysis and modeling technologies for a 

phase 0 world. Presentation slides for PACOM. Office of the Assistant 

Secretary of Defense (Research and Engineering). 2013 Mar 6.

Schnaubelt CM, Larson EV, Boyer ME. Vulnerability assessment method pocket 

guide: a tool for center of gravity analysis. RAND Corporation, ISBN 978-0-

8330-8689-1; 2014.

Schneider G, Finke D, Bailer S. Bargaining power in the European Union: an 

evaluation of competing game-theoretic models. Political Studies.

2010;58(1):85–103.

Schneiderman B. Designing the user interface: strategies for effective humancomputer interaction. 2nd ed. Boston (MA): Addison Wesley Publishing 

Company; 1992.

School of Advanced Military Studies. Art of design. Student text, version 2.0, 2010.

Schwarz G. Command and control in peace support operations model PAX–

approaching new challenges in the modeling of C2. EADS Dornier GmbH;

Friedrichshafen, Germany. 2004 May 21. White paper.

Schwarz G. Experiments with PAX: a quick guide. Friedrichshafen, Germany: 

EADS Deutschland GmbH, 2005 Jul 6.

Sensenig B. An Army intelligence analyst’s perspective on the patterns of life (PoL) 

problem domain. Presentation slides. Chenega Technical Innovations; Fort 

Huachuca, AZ. 2015a Dec.

Sensenig B. Predicting wicked problems in an interactively complex operational 

environment. Project kickoff presentation slides. US Army Intelligence Center 

of Excellence (US); Fort Huachuca, AZ. 2015b Jun.

Sensenig B. Understanding complex operational environments in support of 

operational design and/or tactical mission planning. Presentation slides Army 
Approved for public release; distribution is unlimited.

175

Intelligence Center of Excellence (US); Fort Huachuca (AZ). Workshop, 

Buffalo, NY. 2015c Oct 6.

Shaffer DW, Squire KR, Halverson R, Gee JP. Video games and the future of 

learning. Phi Delta Kappan, 2005;87(02):104–111.

Shellman SM, Levey B, Leonard HH. Countering the adversary: effective policies 

or a dime a dozen? Williamsburg (VA): The College of William and Mary. 

HSCB FOCUS 2011 Conference; 2011; American Political Science 

Association Meetings, Sep 2011 and the HSCB FOCUS 2011 Conference 

February 2011. 2011 Feb 6. White paper.

Shemayev VN. Cognitive approach to modeling reflexive control in socioeconomic systems. Kyiv, Ukraine: National Defence Academy. Information 

and Security: an International Journal. 2007;(22):28–37.

Shen J, Cheng T. A framework for identifying activity groups from individual 

space-time profiles. London, UK: University College London. International 

Journal of Geographical Information Science. 2016 Jan 27. DOI: 

10.1080/13658816.2016.1139119.

Shoffner WA. The military decision-making process: time for a change. Fort 

Leavenworth (KS): US Army Command and General Staff College. 2000 Sep 

19. Monograph. 

Shoham Y, Leyton-Brown K. Multiagent systems: algorithmic, game-theoretic, and 

logical foundations. 2008. ISBN 978-0-521-89943-7.

Singer HM, Singer I, Herrmann HJ. An agent based model for friendship in social 

networks. Zurich, Switzerland: Eidgenossische Technische Hochschule. 

Preprint. 2008 Mar 24.

Smith C, McGuire B, Huang T, Yang G. The history of artificial intelligence.

Seattle (WA): University of Washington; 2006 Dec.

Snyder D, Tolk A. Exchanging PMESII data to support the effects-based approach.

Suffolk (VA): Booz Allen Hamilton. Norfolk (VA): Old Dominion University. 

2006 CCRTS. 2006 Mar 29.

Software Engineering Process Management Program. CMMI for development, 

version 1.3. 2010. Technical report.

Soar-Related Research–Soar Cognitive Architecture [accessed 2016 Jun 21].

http://soar.eecs.umich.edu/Soar-RelatedResearch, 2016.
Approved for public release; distribution is unlimited.

176

Spencer E, Balasevicius T. Crucible of success: cultural intelligence and the 

modern battlespace. British Columbia, Canada: University of Northern British 

Columbia. Ontario, Canada: Royal Military College of Canada. Canadian 

Military Journal. 2009;9(3).

Srbljinovic A, Penzar D, Rodik P, Kardov K. An agent-based model of ethnic 

mobilization. Journal of Artificial Societies and Social Simulation. Prepared 

for presentation at UCLA Computational Social Science Conference, Lake 

Arrowhead, CA; 2002 May 9–12. 2003;6(1). http://jasss.soc.surrey

.ac.uk/6/1/1.html.

Stanford Encyclopedia of Philosophy. Evolutionary game theory. 2009 Jul 19.

http://plato.stanford.edu/entries/game-evolutionary/.

Steinberg AN, Bowman CL, White FE. Revisions to the JDL data fusion model. 

Arlington (VA): ERIM International. Broomfield (CO): Data Fusion and 

Neural Networks. SPAWAR Systems Center; San Diego (CA), 1999 Jan. 

White paper.

Stewart R. Human domain operations: institutionalizing eating soup. New Zealand. 

Small Wars Journal. 2013 Oct 29.

Strange J. Centers of gravity and critical vulnerabilities: building on the 

Clausewitzian foundation so that we can all speak the same language.

Quantico, VA: Marine Corps War College. Marine Corps University, 

Perspectives on Warfighting. 2005;4(2).

Strange JL, Iron R. Center of gravity: what Clausewitz really meant. Quantico, VA: 

Marine Corps War College. Joint Forces Quarterly. 2004;35:20–27.

Sudit M, Ganger R, Hall D, Graham J. Future IPB through intelligence analystexpert system teaming: final report of findings and recommendations. Buffalo 

(NY): CUBRC. State College (PA): Penn State University. Technical report; 

2013 Mar 14.

Sudit M. Computer prediction is an unrealistic panacea, cognitive prediction is not 

scalable so, computer-human prediction is tactically attainable. Presentation 

slides. CUBRC, workshop; Buffalo, NY. 2015 Oct 6.

Sudit M, Costantini K, Spaulding J. NOEM and GNOME overview. Presentation 

slides. CUBRC workshop; Buffalo, NY. 2015 Oct 6.

Suslov V, Domoghirov D, Ibragimov N, Kostin V, Melnikova L, Tsyplakov A. 

Spatial aspects of agent-based modeling of large economy. Institute of 
Approved for public release; distribution is unlimited.

177

Economics and Industrial Engineering, Novosibirsk State University; 

Novosibirsk (Russia). 2015 Feb. White paper.

Taillandier P, Vo D, Amouroux E. Drogoul A. GAMA: a simulation platform that 

integrates geographical information data, agent-based modeling and multiscale control. Kolkata, India. Principles and Practice of Multi-Agent Systems.

Berlin (Germany): Springer Berlin/Heidelberg; 2012 Apr 19. 7057/2012, p.

242–258. Lecture notes in computer science. Doi: 10.1007/978-3-642-25920-

3 17.

Tan PS. Using agent-based modeling to assess the impact of martial law on a 

representative Iraqi town [thesis]. [Monterey (CA)]: Naval Postgraduate 

School; 2004 Dec.

Thomas T. Russia’s military strategy and Ukraine: indirect, asymmetric–and Putinled. Fort Leavenworth (KS): Foreign Military Studies Office. Journal of Slavic 

Military Studies. 2015 Sep 4;28:445–461. DOI: 

10.1080/13518046.2015.1061819.

Thomas TL. Russia’s reflexive control theory and the military. Fort Leavenworth 

(KS): Foreign Military Studies Office. Journal of Slavic Military Studies.

2004;17:237–256. DOI:10.1080/13518040490450529.

Turing AM. Computing machinery and intelligence. Mind. 1950;49:433–460.

Turnley JG, Henscheid Z, Koehler M, Mulutzie S, Tivnan B. COIN of the sociocultural realm: emphasizing socio-cultural interactions in counterinsurgency 

modeling and simulation. Albuquerque, NM: Galisteo Consulting Group. The 

MITRE Corporation. Small Wars Journal. 2012 Feb 28.

Turocy TL, von Stengel B. Game theory. In Encyclopedia of Information Systems. 

2002.

[USAID] US Agency for International Development. Tactical conflict assessment 

and planning framework (TCAPF). Presentation slides. Office of Military 

Affairs. 2006.

United States Army Combined Arms Center. IPB in counterinsurgency (part 1 

operational environment). Presentation slides. 1997.

United States Command and General Staff College, Techniques and Procedures for 

Tactical Decision Making. Student Text 100-9. 1991.

United States Army Special Operations Command. SOF support to political 

warfare. 2015 Mar 10. White Paper, version 2.3.
Approved for public release; distribution is unlimited.

178

United States Special Operations Command. Operating in the human domain.

Version 1.0. 2015 Aug 3.

US Government. A tradecraft primer: structured analytic techniques for improving 

intelligence analysis. 2009 Mar.

Viana FAC. Things you wanted to know about the Latin hypercube design and were 

afraid to ask. 10th World Congress on Structural and Multidisciplinary 

Optimization; 2013 May 19–24; Orlando, FL. Niskayuna (NY): GE Global 

Research.

Vogt M, Bormann NC, Ruegger, S, Cederman LE, Hunziker P, Girardin L. 

Integrating data on ethnicity, geography, and conflict: the ethnic power 

relations dataset family. 2015 Apr 21. White paper.

Wakeman CC. Discrete event simulation modeling and analysis of key leader 

engagements [thesis]. Naval Postgraduate School; 2012 Jun.

Wallis L, Paich M, Borshchev A. Agent modeling of Hispanic population 

acculturation and behavior. Fort Collins (CO): Decisio Consulting. XJ 

Technologies Company Ltd. 2004 Apr 22.

Waltz E. Anticipatory intelligence analysis: integrating multiple models for joint 

intelligence preparation. Presentation slides. BAE Systems Technology 

Solutions; Arlington, VA. 2010 May 21.

Waltz E. Anticipatory intelligence analysis: integrating multiple models for joint 

intelligence preparation. Proceedings of IEEE Conference on Intelligence 

Security and Informatics, Workshop on Predictive Analytics for Intelligence 

Security Applications; 2010 May 21; Vancouver, BC, Canada. Arlington 

(VA): BAE Systems Technology Solutions.

Waltz E. Situation analysis and collaborative planning for complex operations: the

COMPOEX (conflict modeling, planning, and outcomes experimentation) 

program. Arlington (VA): BAE Systems Technology Solutions. [Bellevue 

(WA)]: 13th International Command and Control Research and Technology 

Symposium: C2 for Complex Endeavors. 2008 Mar 07.

Wang Y, Ruhe G. The cognitive process of decision making. International Journal 

of Cognitive Informatics and Natural Intelligence. 2007 Apr–Jun;1(2):73–85; 

Alberta, Canada: University of Calgary.

Ward MD, Metternich NW, Dorff CL, Gallop M, Hollenbach FM, Schultz A, 

Weschle S. Learning for the past and stepping into the future: toward a new 

generation of conflict prediction. Durham (NC): Duke University. London, 
Approved for public release; distribution is unlimited.

179

England: University of College London. International Studies Review. 2013.

doi: 10.1111/misr.12072

Watts DJ. Computational social science: exciting progress and future directions.

Microsoft Research. National Academy of Engineering of the National 

Academies, The Bridge. Frontiers of Engineering. Winter 2013;43(4). 

Weiss G. Multiagent systems: a modern approach to distributed artificial 

intelligence. ISBN: 9780262731317; 1999.

Weiss M. Pattern-driven design of agent systems: approach and case study. Ottawa 

(Canada): Carleton University. 2003 Mar 28. White paper.

West BJ. Fractional calculus view of complexity: tomorrow’s science. Army 

Research Office. Research Triangle Park (NC): CRC Press, Taylor & Francis 

Group; 2015. ISBN 978-1-4987-3801-9.

Wise S, Crooks AT. Agent-based modeling for community resource management: 

acequia-based agriculture. Fairfax (VA): George Mason University. 

Computers, Environment and Urban Systems. 2012 Aug 29;36:562–572.

Woods DD, Patterson ES, Roth EM. Can we ever escape from data overload? A 

cognitive systems diagnosis. Columbus (OH), Ohio State University. 

Brookline (MA): Roth Cognitive Engineering. Cognition, Technology and 

Work. 2002;22–36.

Works P. Putting innovation into practice: connections 2015. Presentation slides.

TRADOC Analysis Center, Connections; Fort Leavenworth, KS. 2015 Jul 23.

Wray RE, Jones RM, An introduction to soar as an agent architecture. Ann Arbor 

(MI); Soar Technology. 2004 Jun 1. White paper.

Wunderle WD. Through the lens of cultural awareness: a primer for US armed 

forces deploying to Arab and Middle Eastern countries. Fort Leavenworth 

(KS): Combat Studies Institute Press; 2007 Apr 26. ISBN 0-16-077466-7.

Young JP. A brief history of war gaming. Arlington (VA): Armed Services 

Technical Information Agency. Technical report; AD 235 893; 1956 Oct 23.

Zadeh LA. A simple view of the Dempster-Shafer theory of evidence and its 

implication for the rule of combination. Berkeley (CA): University of 

California. AI Magazine. 1986;7(2).

Zelik D, Patterson ES, Woods DD. Understanding rigor in information analysis.

Columbus (OH), Ohio State University. Proceedings of the Eighth 

International NDM Conference; 2007 Apr 2; Pacific Grove (CA).
Approved for public release; distribution is unlimited.

180

Zelik DJ, Patterson ES, Woods DD. Measuring attributes of rigor in information 

analysis. In Patterson ES, Miller J, editors. Macrocognition metrics and 

scenarios: Design and evaluation for real-world teams. Aldershot, UK: 

Ashgate. 2009. ISBN 978-0-7546-7578-5.

Zheng H, Son YJ, Chiu YC, Head L, Feng Y, Xi H, Kim S, Hickman M. A primer 

for agent-based simulation and modeling in transportation applications. 

Tucson (AZ): University of Arizona; 2013 Nov. White paper; Report No.:

FHWA-HRT-13-054.
Approved for public release; distribution is unlimited.

181

Appendix A. Survey of Existing Agent-Based Modeling 

Frameworks 

 

This appendix appears in its original form, without editorial change.
Approved for public release; distribution is unlimited.

182

The following is a sampling of existing agent-based modeling frameworks. For the 

purpose of this Appendix, the study separates the identified frameworks into three 

groups based on an initial assessment of their maturity: most mature, intermediate 

maturity, and low maturity or unmaintained. Most mature indicates the likelihood 

of continual use and a well-developed capability. Intermediate maturity indicates 

still relevant research and capabilities but the concepts may not be fully realized. 

Low maturity, or appears to be unmaintained, identifies available frameworks but, 

at this time, the study does not recommend their use.

Each of the following seventy-two frameworks is identified by its name, its 

estimated use (e.g., Academic-Advanced), estimated last update, location, and a 

brief description. Estimated use is identified as one of the following enumerations: 

Academic-Advanced, Academic-Intermediate, Academic-Basic, Commercial, and 

Government. Descriptions use text from the respective projects’ web site.

A.1 Most Mature Frameworks

ACT-R Academic-Advanced Updated: 2016

http://act-r.psy.cmu.edu/

ACT-R is a cognitive architecture: a theory for simulating and understanding human cognition. 

Researchers working on ACT-R strive to understand how people organize knowledge and 

produce intelligent behavior.

AgentBase Academic-Advanced Updated: 2016

http://agentbase.org/

AgentBase.org allows you to do Agent-Based Modeling (ABM) in the browser. You can edit, 

save, and share models without installing any software or even reloading the page. Models are 

written in Coffeescript, and use the AgentBase library.

AgentScript Academic-Advanced Updated: 2016

http://agentscript.org/

AgentScript is a minimalist Agent-Based Modeling (ABM) framework based on NetLogo agent 

semantics. Its goal is to promote the Agent Oriented Programming model in a highly 

deployable CoffeeScript/JavaScript implementation.

AMP (Agent Modeling Platform) Academic-Advanced Updated: 2015

http://eclipse.org/amp/

The AMP project provides extensible frameworks and exemplary tools for representing, editing, 

generating, executing, and visualizing agent-based models (ABMs) and any other domain 

requiring spatial, behavioral, and functional features.

AnyLogic Commercial Updated: 2016

http://www.anylogic.com

AnyLogic is the only simulation tool that supports all the most common simulation 

methodologies in place today: System Dynamics, Process-centric (AKA Discrete Event), and 

Agent-Based modeling. The object-oriented model design paradigm supported by AnyLogic 

provides for modular, hierarchical, and incremental construction of large models.
Approved for public release; distribution is unlimited.

183

Ascape Academic-Advanced Updated: 2013

http://ascape.sourceforge.net/

Models can be developed in Ascape using far less code than in other tools. Ascape models are 

easier to explore, and profound changes to the models can be made with minimal code changes. 

Ascape offers a broad array of modeling and visualization tools.

Behavior Composer Academic-Advanced Updated: 2015

http://m.modelling4all.org

The Modelling4All Project is building a web-based tool for constructing, running, visualizing, 

analyzing, and sharing agent-based models. These models can be constructed by non-experts by 

composing pre-built modular components called micro-behaviors.

Brahms Academic-Advanced Updated: 2012

http://brahms.sourceforge.net/docs/

BRAHMS is a Modular Execution Framework (MEF) for executing integrated systems built 

from component software processes (a SystemML-ready execution client). Its operation is 

conceptually similar to that of Simulink, for example.

Cougaar Government Updated: 2015

http://cougaar.org

Cougaar is a Java-based architecture for the construction of large-scale distributed agent-based 

applications. It is a product of two consecutive, multi-year DARPA research programs 

(UltraLog) into large-scale agent systems spanning eight years of effort.

CybelePro Commercial Updated: 2014

http://www.i-a-i.com/?core/modeling-and-simulation/distributed-architectures-and-solutions

CybelePro is the underlying agent framework of NASA’s Airspace Concept Evaluation System, 

a distributed gate-to-gate simulator of the ATM operations in the National Airspace. For 

applications in network modeling and simulation our Agent-Based Parallel Discrete Event 

Simulation Framework, enables the execution of large-scale (10,000s of communicating nodes), 

high fidelity, mobile, tactical (primarily wireless) network models in a parallel computing 

environment, where each computing node is a multi-core processor.

GAMA Academic-Advanced Updated: 2016

https://github.com/gama-platform

GAMA is a modeling and simulation development environment for building spatially explicit 

agent-based simulations.

Gambit Academic-Advanced Updated: 2016

http://www.gambit-project.org/

Gambit is an open-source collection of tools for doing computation in game theory. With 

Gambit, you can build, analyze, and explore game theoretical models.

GoldSim Commercial Updated: 2016

http://www.goldsim.com/Home/

GoldSim is a general-purpose simulator that utilizes a hybrid of several simulation approaches, 

combining an extension of system dynamics with some aspects of discrete event simulation, 

and embedding the dynamic simulation engine within a Monte Carlo simulation framework.

iGen Commercial Updated: 2010

http://www.chisystems.com/cognitivemodel.html

iGEN is a patented artificial intelligence engine that mimics the way human experts analyze 

and make decisions in a wide range of situations. iGEN uses a revolutionary psychological 

model of human thought and problem solving called COGNET within an engineering-oriented 

integrated software development environment.
Approved for public release; distribution is unlimited.

184

Insight Maker Commercial Updated: 2016

https://insightmaker.com/

Insight Maker runs in your web-browser. No downloads or plugins are needed. Explore 

powerful simulation algorithms for System Dynamics and Agent-Based Modeling in a truly 

cohesive environment.

JABM Academic-Advanced Updated: 2016

http://jabm.sourceforge.net/

The Java Agent-Based Modelling (JABM) toolkit is a Java framework for building agent-based 

models using a discrete-event simulation framework.

JADE Commercial Updated: 2015

http://jade.tilab.com/

JADE (Java Agent DEvelopment Framework) is a software Framework fully implemented in 

the Java language. Besides the agent abstraction, JADE provides a simple yet powerful task 

execution and composition model, peer to peer agent communication based on the 

asynchronous message passing paradigm, a yellow pages service supporting publish subscribe 

discovery mechanism and many other advanced features that facilitates the development of a 

distributed system.

MaDKit (Multi Agent Development Kit) Academic-Advanced Updated: 2016

http://www.madkit.org/

MaDKit is a lightweight Java library for designing and simulating Multi-Agent Systems 

(MAS). In contrast to conventional approaches, which are mostly agent-centered, MaDKit 

follows an organization-centered approach (OCMAS) so that there is no predefined agent 

model in MaDKit. So, MaDKit is built upon the AGR (Agent/Group/Role) organizational 

model: agents play roles in groups and thus create artificial societies.

MASON Academic-Advanced Updated: 2015

https://cs.gmu.edu/~eclab/projects/mason/

MASON is a fast discrete-event multi-agent simulation library core in Java, designed to be the 

foundation for large custom-purpose Java simulations, and to provide more than enough 

functionality for many lightweight simulation needs.

NetLogo Academic-Advanced Updated: 2016

https://ccl.northwestern.edu/netlogo/index.shtml

NetLogo is a multi-agent programmable modeling environment. Tens of thousands of students, 

teachers, and researchers use it worldwide. It also powers HubNet participatory simulations.

PS-I (Political Science-Identity) Government Updated: 2014

http://ps-i.sourceforge.net/

Ps-i is an environment for running agent-based simulations. It is cross-platform, with binaries 

available for Win32.

Repast Academic-Advanced Updated: 2015

http://repast.sourceforge.net/

The Repast Suite is a family of advanced, free, and open source agent-based modeling and 

simulation platforms that have collectively been under continuous development for over 15 

years. 
Approved for public release; distribution is unlimited.

185

SEAS (System Effectiveness Analysis 

Simulation)

Government Updated: 2015

https://www.teamseas.com/

The System Effectiveness Analysis Simulation (SEAS) is a constructive modeling and 

simulation tool that enables mission-level Military Utility Analysis (MUA). SEAS offers a 

powerful agent-based modeling environment that allows the analyst to simulate the complex, 

adaptive interactions of opposing military forces in a physics-based battlespace. Agents (units 

and platforms) execute programmable behavioral and decision-making rules based on 

battlespace perception. The interaction of the agents with each other and their environment 

results in warfighting outcomes, enabling Military Utility Analysis.

SimPy Academic-Advanced Updated: 2016

https://simpy.readthedocs.org/en/latest/

SimPy is a process-based discrete-event simulation framework based on standard Python. Its 

event dispatcher is based on Python’s generators and can also be used for asynchronous 

networking or to implement multi-agent systems (with both, simulated and real 

communication).

SimX Commercial Updated: 2014

https://github.com/sim-x/simx

SimX is a library for developing parallel, discrete-event simulations in Python. SimX also 

supports process-oriented simulations. Parallel simulations are implemented using distributed 

memory and message passing.

SOAR Academic-Advanced Updated: 2016

http://soar.eecs.umich.edu/

Soar is a general cognitive architecture for developing systems that exhibit intelligent behavior. 

Researchers all over the world, both from the fields of artificial intelligence and cognitive 

science, are using Soar for a variety of tasks. It has been in use since 1983, evolving through 

many different versions to where it is now Soar, Version 9.

Sugarscape Academic-Advanced Updated: 2013

http://sugarscape.sourceforge.net/

Sugarscape is an 'artificially intelligent agent-based social simulation'. Epstein & Axtell's 

implementation came to be known as the Sugarscape model and it is from that work that this 

project derives its name.

Swarm Academic-Advanced Updated: 2013

http://www.swarm.org/wiki/Swarm_main_page

Swarm is a platform for agent-based models (ABMs) that include a conceptual framework for 

designing, describing, and conducting experiments on ABMs; software implementing that 

framework and providing many handy tools; and a community of users and developers that 

share ideas, software, and experience.

Vensim Commercial Updated: 2015

http://vensim.com/vensim

Vensim is simulation software for improving the performance of real systems. Vensim is used 

for developing, analyzing, and packaging dynamic feedback models.
Approved for public release; distribution is unlimited.

186

A.2 Intermediate Maturity Frameworks

AgentService Academic-advanced Updated: 2009

http://www.AgentService.it

AgentService is an agent oriented programming framework based on the Common Language 

Infrastructure (CLI) and the C# language. The key features of AgentService are the new agent 

model that exploits the innovative scheduling features of the CLI and the software platform 

providing the agents with advanced run-time services.

AgentSheets Commercial Updated: 2015

http://www.agentsheets.com/products/index.html

AgentSheets is a revolutionary tool that lets you create your own agent-based games and 

simulations and publish them on the Web through a user-friendly drag-and-drop interface.

AOR simulation Academic-intermediate Updated: 2010

http://oxygen.informatik.tu-cottbus.de/aor/?q=node/2

The Entity-Relationship (ER) and Agent-Object-Relationship (AOR) modeling and simulation 

framework developed at the Brandenburg University of Technology, Germany, provides the 

XML-based simulation languages ERSL and its superset AORSL for basic and agent-based 

discrete event simulation.

BDI4Jade Academic-advanced Updated: 2014

http://www.inf.ufrgs.br/prosoft/bdi4jade/

BDI4JADE is an agent platform that implements the BDI (belief-desire-intention) architecture. 

It consists of a BDI layer implemented on top of JADE.

Cormas (common-pool resources and multiagent systems)Academic-intermediate Updated: 2012

http://cormas.cirad.fr/indexeng.htm

Cormas is intended to facilitate the design of ABM as well as the monitoring and analysis of 

simulation scenarios. It is used by an international community of researchers willing to 

understand the relationships between societies and their environment.

DALI Academic-intermediate Updated: 2016

https://github.com/AAAI-DISIM-UnivAQ/DALI

A Multi Agent Systems Framework built on top of Sicstus Prolog.

ENVISION Academic-advanced Updated: 2015

http://envision.bioe.orst.edu/

ENVISION is a GIS-based tool for scenario-based community and regional integrated planning 

and environmental assessments. It provides a robust platform for integrating a variety of 

spatially explicit models of landscape change processes and production for conducting 

alternative futures analyses.

FLAME Academic-advanced Updated: 2014

http://www.flame.ac.uk/

FLAME is a generic agent-based modelling system, which can be used to development 

applications in many areas. The FLAME framework is an enabling tool to create agent-based 

models that can be run on high performance computers (HPCs).
Approved for public release; distribution is unlimited.

187

GROWlab Academic-advanced Updated: 2008

http://www.icr.ethz.ch/research/growlab

Computer simulation has proved to be a vital adjunct to traditional approaches of understanding 

social phenomena in fully understanding the dynamics of individual and collective actions.

JAMEL (java agent-based macroeconomic 

laboratory)

Academic-intermediate Updated: 2016

http://p.seppecher.free.fr/jamel/

Jamel (Java Agent-based Macro-Economic Laboratory) is an open source agent-based 

framework dedicated to the modeling, the simulation, and the analysis of complex monetary 

economies.

JAMSIM Academic-basic Updated: 2012

https://ideas.repec.org/a/jas/jasssj/2011-26-2.html

JAMSIM (JAva MicroSIMulation) is an innovative synthesis of open source packages that 

provides an environment and set of features for the creation of dynamic discrete-time 

microsimulation models that are to be executed, manipulated, and interrogated by nontechnical, policy-oriented users. Combining the leading open source statistical package R and 

one of the foremost agent-based modelling (ABM) graphical tools Ascape, JAMSIM is 

available as an open source tool, for public reuse and modification.

JANUS Academic-advanced Updated: 2016

http://www.janusproject.io/

Janus is an open-source multi-agent platform fully implemented in Java 1.7. Janus enables 

developers to quickly create web, enterprise, and desktop multi-agent-based applications.

JAS Academic-intermediate Updated: 2006

http://jaslibrary.sourceforge.net/index.html

JAS is a Java toolkit for creating agent-based simulations.

It features a discrete-event time engine, statistical probes with Hypersonic database built-in 

storage capability, Neural Networks and Genetic Algorithms packages, and graph support for 

Social Network Analysis.

JASA (Java auction simulator API) Academic-intermediate Updated: 2014

http://jasa.sourceforge.net/

JASA is a high-performance auction simulator. It is designed for performing experiments in 

agent-based computational economics.

JIAC Academic-advanced Updated: 2016

http://www.jiac.de/

JIAC (Java-based Intelligent Agent Componentware) is a Java-based agent architecture and 

framework that eases the development and the operation of large-scale, distributed applications 

and services. The framework supports the design, implementation, and deployment of software 

agent systems.

LSD (laboratory for simulation development) Academic-intermediate Updated: 2015

http://www.labsimdev.org/Joomla_1-3/

LSD implements discrete-time simulations, expressing the results as series of values for each 

variable in the model. Being based on C++, LSD models are extremely fast and efficient, 

allowing for even huge models. LSD is particularly suited to implement agent-based models.
Approved for public release; distribution is unlimited.

188

MADeM (multi-model agent decision making) Academic-intermediate Updated: 2015

http://www.uv.es/grimo/jmadem/

The MADeM (Multi-modal Agent Decision Making) model provides agents with a general 

mechanism to make socially acceptable decisions. In this kind of decisions, the members of an 

organization are required to express their preferences with regard to the different solutions for a 

specific decision problem.

MASS (multi-agent simulation suite) Commercial Updated: 2012

http://www.aitia.ai/en/web/iaws/mass

The Multi Agent Simulation Suite consists of four applications offering solutions for different 

aspects of modeling. Each of the applications is developed with the intention of providing 

professional tools for users without heavy programming skills. The components include 

functional agent-based language for simulations, visualization module, model exploration 

module, and participatory extension.

MATSim Academic-advanced Updated: 2016

http://www.matsim.org/

MATSim is an open-source framework to implement large-scale agent-based transport 

simulations. Currently, MATSim offers a framework for demand-modeling, agent-based 

mobility-simulation (traffic flow simulation), re-planning, and a controller to run simulations as 

well as methods to analyze the output generated by the modules.

Mesa Academic-intermediate Updated: 2016

https://pypi.python.org/pypi/Mesa/

Mesa is an Apache2 licensed agent-based modeling (or ABM) framework in Python. It allows 

users to quickly create agent-based models using built-in core components (such as spatial grids 

and agent schedulers) or customized implementations; visualize them using a browser-based 

interface; and analyze their results using Python’s data analysis tools. Its goal is to be the 

Python 3-based alternative to NetLogo, Repast, or MASON.

Modgen Government Updated: 2015

http://www.statcan.gc.ca/microsimulation/modgen/modgen-eng.htm

Modgen (Model generator) is a generic microsimulation programming language supporting the 

creation, maintenance, and documentation of dynamic microsimulation models. Several types 

of models can be accommodated, be they continuous or discrete time, with interacting or noninteracting populations.

Pandora Academic-intermediate Updated: 2016

http://www.bsc.es/computer-applications/pandora-hpc-agent-based-modelling-framework

Pandora Library is an ABM framework created by the social simulation research group of the 

Barcelona Supercomputing Centre. This tool is designed to implement agent-based models and 

to execute them in high-performance computing environments.

StarLogo Nova Academic-advanced Updated: 2014

http://www.slnova.org/

StarLogo Nova is the new online iteration of StarLogo, following in StarLogo TNG's footsteps. 

StarLogo Nova builds upon TNG's innovations, with several language refinements and new 

features.
Approved for public release; distribution is unlimited.

189

TNG Academic-Advanced Updated: 2016

http://www2.econ.iastate.edu/tesfatsi/tnghome.htm

Trade Network Game (TNG) is a framework for studying the formation and evolution of trade 

networks among strategically interacting traders (Buyers, Sellers, and Dealers) operating under 

variously specified market protocols.

Xholon Commercial Updated: 2014

http://www.primordion.com/Xholon/

Xholon is a flexible open source tool for multi-paradigm modeling, simulation, design, 

execution, and transformation. Generic Java and XML building blocks are extended into 

multiple domains, and woven into loosely organized systems.

A.3 Low Maturity Frameworks (or Possibly Unmaintained)

ABLE (agent building and learning 

environment)

Commercial Updated: 2005

http://www.freecode.com/projects/agentbuildingandlearningenvironmentforlinux

Agent Building and Learning Enviroment (ABLE) is a Java framework, component library, and 

productivity tool kit for building intelligent agents using machine learning and reasoning. With 

this tool, developers can implement their own AbleBeans and AbleAgents and plug them into 

ABLE's Agent Editor.

AgentBuilder Commercial Updated: 2011

http://www.agentbuilder.com

AgentBuilder is an integrated software toolkit that allows software developers to quickly 

develop intelligent software agents and agent-based applications.

Breve Academic-intermediate Updated: 2008

http://www.spiderland.org

breve is a free, open-source software package which makes it easy to build 3D simulations of 

multi-agent systems and artificial life. Using Python, or using a simple scripting language called 

steve, you can define the behaviors of agents in a 3D world and observe how they interact. 

breve includes physical simulation and collision detection so you can simulate realistic 

creatures, and an OpenGL display engine so you can visualize your simulated worlds.

DigiHive Academic-intermediate Updated: 2011

http://www.swarm.eti.pg.gda.pl/index.htm

The DigiHive environment is an original, artificial, complete, low level, and closed 

environment consisting of space, moving objects, and rules, which govern their interactions.

D-OMAR (distributed operator model 

architecture)

Government Updated: 1999

http://oai.dtic.mil/oai/oai?verb=getRecord&metadataPrefix=html&identifier=ADA364623

Distributed OMAR (D-OMAR) is a new implementation of the Operator Model Architecture 

(OMAR) designed to operate in a distributed computing environment. As in OMAR, D-OMAR 

is designed as a simulation environment in which to create human performance models. It also 

provides the capability to develop the complex multi-tasking agents that are needed in agentbased systems.
Approved for public release; distribution is unlimited.

190

JCA-Sim Academic-intermediate Updated: 2004

http://www.jcasim.de/#Overview

The program system JCASim is a general-purpose system for simulating cellular automata in 

Java. It includes a stand-alone application and an applet for web presentations.

jES (Java enterprise simulator) Academic-intermediate Updated: 2015

http://terna.to.it/jes/

jES, the synthetic name of the Java Enterprise Simulator project, is a frame used to develop 

enterprise simulation models based on the Java version of Swarm both (1) to simulate the 

activities ‐ and the consistent emerging results ‐ of an actual enterprise, and (2) to build virtual 

or hypothetical enterprises.

MAML (multi-agent modeling language) Academic-advanced Updated: 1999

http://www.maml.hu/maml/initiative/index.html

The current version of MAML is an extension to Objective-C (using the Swarm libraries). It 

consists of a couple of 'macro-keywords' that define the general structure of a simulation. The 

remaining is filled with pure swarm-code. A MAML-to-Swarm (named xmc) compiler is being

developed, which compiles the source code into a swarm application. Like every other Swarm 

application, this generated application must be compiled by gcc.

MAS-SOC (multi-agent simulations for the 

SOCial sciences)

Academic-advanced Updated: 2005

http://jasss.soc.surrey.ac.uk/8/3/7.html

One of the main goals of the MAS-SOC simulation platform (MAS-SOC stands for MultiAgent Simulations for the SOCial Sciences) is to provide a framework for the creation of agentbased simulations, which does not require too much experience in programming from users, yet 

allowing users to use state-of-the-art agent technologies. In particular, it should allow for the 

design and implementation of simulations with cognitive agents -- a plethora of platforms for 

reactive agents exists, but that is certainly not the case for agents that are more elaborate.

Mimosa Government Updated: 2014

http://mimosa.sourceforge.net

Mimosa is a modeling and simulation platform, covering the process from building conceptual 

models to running the simulations. The specification use ontologies and an extensible set of 

formalisms for the dynamics, initialization, and visualization. The simulation kernel is based on 

DEVS.

MIMOSE (micro-und multilevel modelling 

software)

Academic-basic Updated: 1999

http://userpages.uni-koblenz.de/~moeh/projekte/mimose.html

The main purpose of the MIMOSE project was the development of a modelling language, 

which considers special demands of modelling in social science, especially the description of 

nonlinear, quantitative, and qualitative relations, stochastic influences, birth, and death 

processes, as well as micro and multilevel models.

MobiDyc Academic-intermediate Updated: 2006

http://w3.avignon.inra.fr/mobidyc/index.php/English_summary

Mobidyc is a software project that aims to promote Individual-Based Modelling in the field of 

ecology, biology, and environment. It is the acronym for MOdelling Based on Individuals for 

the DYnamics of Communities.
Approved for public release; distribution is unlimited.

191

Moduleco Academic-basic Updated: 2001

http://www.gemass.fr/dphan/moduleco/english/moduleco00.htm

Moduleco is a multi-agent modular framework for the simulation of network effects and 

population dynamics in social sciences, markets, and organizations.

MOOSE (multimodeling object-oriented 

simulation environment)

Academic-intermediate Updated: 1997

http://www.cise.ufl.edu/~fishwick/moose.html

MOOSE is the next generation of SimPack which was initiated in 1990 for providing a 

general-purpose toolkit of C and C++ libraries for discrete-event and continuous simulation.

SimAgent Academic-intermediate Updated: 2014

http://www.cs.bham.ac.uk/research/projects/poplog/packages/simagent.html

Like ACT-R, COGENT, and the original SOAR, SimAgent is primarily designed to support 

design and implementation of very complex agents, each composed of very different interacting 

components (like a human mind) where the whole thing is embedded in an environment that 

could be a mixture of physical objects and other agents of many sorts.

TerraME Academic-advanced Updated: 2013

http://www.terrame.org/doku.php

TerraME is a programming environment for spatial dynamical modelling. It supports cellular 

automata, agent-based models, and network models running in 2D cell spaces. TerraME 

provides an interface to TerraLib geographical database, allowing models direct access to 

geospatial data.

VisualBots Academic-basic Updated: 2008

http://www.visualbots.com/index.htm

VisualBots for Excel is an educational tool for exploring the fascinating world of multi-agent 

systems and their emergent behaviors. The VisualBots simulator can be used to create virtual 

worlds of programmable agents that interact with one another through time.

ZEUS Commercial Updated: 2013

https://sourceforge.net/projects/zeusagent/

Zeus provides a graphical environment to build distributed agent systems. A rule engine, 

planner and visualization tools are included.
Approved for public release; distribution is unlimited.

192

INTENTIONALLY LEFT BLANK.
Approved for public release; distribution is unlimited.

193

Appendix B. Sampling of Computational Social Science Society of 

the Americas (CSSSA) Conference Papers
Approved for public release; distribution is unlimited.

194

Fifty-four papers submitted to the CSSSA serve as a sample from which to 

extrapolate the nature of public research applicable to Human Domain elements. 

Papers were published throughout the last 3 CSSSA conferences (2012, 2013, and 

2015).

1. Lauf T, Gawel E, Frank K. The spatial allocation of renewable power 

infrastructure–an economic assessment of energy landscapes with an agentbased modelling approach; 2015.

2. Clark R, Kimbrough SO. The spontaneous emergence of language variation 

from a homogeneous population; 2015.

3. Bodo P. How much abuse can markets take? on the structural effects corporate 

income redistribution; 2015.

4. Baseman E, Jensen D. Exploring collective behavior in social computation 

through relational statistical models; 2015.

5. Gulden T, Koehler M, Scott S, Henscheid Z. Evidence for allometric scaling 

of government services in American cities; 2015.

6. Sakahira F, Terano T. Generating anthropological and archaeological 

hypotheses in Okinawa through agent-based simulation; 2015.

7. Sandoval-Felix J, Castanon-Puga M, Gaxiola-Pacheco C. Using an urban 

growth simulator for Ensenada city public policy analysis; 2015.

8. Dixon D, Stith S, An agent-based model of innovation in organ transplant data; 

2015.

9. Garibay I, Akbas MI, Gunaratne C, Ozmen O, O’Neal T. An agent-based 

approach to study incubation in innovation ecosystems; 2015.

10. Barkoczi D, Galesic M. Social learning strategies, network structure and the 

exploration-exploitation tradeoff; 2015.

11. Aydogmus O, Çagatay H, Gurpinar E, Oguz F. The effect of expertise in norm 

formation; 2015.

12. Bassett J, Cioffi C. Evolutionary computation applied to agent-based 

simulation modeling of climate and social dynamics; 2015.

13. Oviedo L, Koehler M, Taylor M. Exploring organizational learning and 

structuring; 2015.
Approved for public release; distribution is unlimited.

195

14. Michel S, Megerdoomian K. Modeling community resilience for a postepidemic society; 2015.

15. Yang Z. An agent-based dynamic model of politics, fertility and economic 

development; 2015.

16. Gonnering R, Lein J. Political speech: poem, preaching, performance? a pilot 

computational study; 2015.

17. Carbone G, Giannoccaro I. Collective decision making on complex 

landscapes; 2015.

18. Leibzon W. Modeling competitive beliefs on social network; 2015.

19. Adiga A, Chu S, Marathe A, Vullikanti A. Behavioral modeling for epidemic 

planning and response; 2015.

20. Henderson H. Modern value chains and the organization of agrarian 

production; 2015.

21. Taghawi-Nejad D. An agent-based model to understand the effect of 

interactions between federal, state and local governance on the distribution of 

educational resources; 2015.

22. Abdollahian M, Yang Z, Coan T, Yesilda B. Human development dynamics: 

an agent-based simulation of social systems and heterogeneous evolutionary 

games; 2013.

23. Baggio JA. Synchronization of management strategies; 2013.

24. Bloomquist KM, Koehler M. A large-scale agent-based model of taxpayer 

reporting compliance; 2013.

25. Bramson A, Grim P, Singer DJ, Fisher S, Sack G, Berger W, Flocken C.

Measures of polarization and diversity; 2013.

26. Dixon DS, Mozumder P, Vásquez WF. An information entropy approach to 

salience for survey-driven simulation; 2013.

27. Duong D. The data absorption technique: coevolution to automate the accurate 

representation of social structure; 2013

28. Galloway E, Mappus RL IV, Briscoe E. The role of networks in durable goods 

technology adoption; 2013.

29. Garibay I, Hollander CD, Ozmen O, O’Neal T. Towards modeling economic 

ecosystems: an initial model and preliminary validation; 2013.
Approved for public release; distribution is unlimited.

196

30. Ge J. Endogenous formation and collapse of housing bubbles; 2013.

31. Golman R, Hagmann D, Miller J. Polya’s bees: decentralized decision making 

with quorum-based strategies; 2013.

32. Herrmann J, Rand W, Schein B, Vodopivec N. An agent-based model of urgent 

diffusion in social media; 2013.

33. Kennedy WG, Harrison JF. Towards representing disasters in computational 

social simulations; 2013.

34. Kim Y. Dynamics of the silence of majority from the perspective of social 

dilemma; 2013.

35. Krejci CC, Beamon BM. Modeling the impacts of farmer coordination on food 

supply chain structure; 2013.

36. McCaskill JR. Complex humanitarian intervention simulation; 2013.

37. Nasrallah WF, Cheaib KA, Yassine AA. A dynamic equilibrium model of how 

regulative and normative institutions influence economic behavior and growth; 

2013.

38. Waring TM, Goff S, Smaldino PE. Evolving the core design principles: the 

coevolution of institutions and sustainable practices; 2013.

39. Zhang H, Putra HC, Andrews CJ. Modeling real estate market responses to 

climate change in the coastal zone; 2013.

40. Flocken C, Carmichael T, Hadzikadic M. The role of uninformed individuals 

in making the right group decisions; 2012.

41. Geller A, Latek M, Rizi SMM, Fournier S, Prenot-Guinard F. Contexts as 

reasoning and action frames for multi-agent societies; 2012.

42. Gonnering RS. Modeling the spread of a “cultural meme” through an 

organization; 2012.

43. Griffin WA, Li X. Unsupervised learning of dyadic processes: models, 

methods, and simulation; 2012.

44. Gulden T. Modeling selective violence in the Guatemalan civil war; 2012.

45. Hagmann D, Tassier T. Spatial coordination games with agent turnover; 2012.

46. Hailegiorgis A, Crooks AT. Agent-based modeling for humanitarian issues: 

disease and refugee camps; 2012.
Approved for public release; distribution is unlimited.

197

47. Latek MM, Rizi SMM, Crooks AT, Fraser M. A spatial multi-agent model of 

border security for the Arizona–Sonora borderland; 2012.

48. Lawless WF. Incompleteness, uncertainty and autonomy: intelligent systems; 

2012.

49. Mappus RL IV, Briscoe E, Hutto CJ. Effects of exogenous input on adoption 

rates in social networks; 2012.

50. Marlin B, Sohn H. Demand based balance of flow manpower modeling for 

education policy in Afghanistan; 2012.

51. Parunak HVD, Brooks HS, Brueckner S, Gupta R. Dynamically tracking the 

real world in a CSS model; 2012.

52. Schoon M, Baggio J, Salau K, Janssen M. Modeling decision-making across 

habitat patches: insights on large-scale conservation management; 2012.

53. Wise S, Scott S. The puck stops here: evolving social norms of helmet usage 

in the National Hockey League; 2012.

54. Yang Z, Abdollahian M, Nelson H, Close B. Getting to yes: the sustainable 

energy modeling project (SEMPro) model of infrastructure siting; 2012.
Approved for public release; distribution is unlimited.

198

List of Symbols, Abbreviations, and Acronyms

2-D 2-dimensional

3-D 3-dimensional

ABM agent-based modeling

ACT-R adaptive control of thought–rational (often refers to a 

cognitive architecture)

API application program interface

ARSOF Army Special Operations Forces

ASCOPE area, structures, capabilities, organizations, people, and 

events

BCT brigade combat team

CADSIM COA Analysis by Integration of Decision and Social 

Influence Modeling with Multi-Agent System

CGSC Command and General Staff College

CM2E complex military mission environment

COA course of action

COEM complex operational environment model

COIC Counter-IED Operations/Intelligence Center

COIN counterinsurgency operations

COMPOEX Conflict Modeling, Planning, and Outcomes 

Experimentation Program

CRA Charles River Analytics

CSSSA Computational Social Science Society of the Americas

DA Department of the Army

DARPA Defense Advanced Research Projects Agency

DCGS-A Distributed Common Ground System–Army

DIME diplomatic, information, military, and economic
Approved for public release; distribution is unlimited.

199

DIME-FIL diplomatic, information, military, economic, financial, 

intelligence, and law enforcement

DSTL Defence Science and Technology Laboratory, Ministry of 

Defence, United Kingdom

FAO Foreign Area Officer

GAMA agent-based modeling framework

GIS geographical information system

GIT version control system

GROWlab Geographical Research on War Laboratory

HIG Hezb-e Islami Gulbuddin

HQ Headquarters

HSCB Human Social Culture Behavior

ID identification

IED improvised explosive device

IPB intelligence preparation of the battlefield

IW irregular warfare

IW TWG irregular warfare tactical wargame

JDL Joint Directors of Laboratories, often refers to the JDL data 

fusion model

JFCOM Joint Forces Command

JIEDDO Joint Improvised Threat Defeat Agency

JIPOE Joint Intelligence Preparation of the Operational 

Environment

JNEM Joint Non-Kinetic Effects Model

JPL Jet Propulsion Laboratory

JTF joint task force

JWARS Joint Warfare Systems

KSA knowledge, skills, and abilities
Approved for public release; distribution is unlimited.

200

LOA level of automation

MDMP military decision-making process

METT-TC mission, enemy, terrain, troops available, time, and civilian 

considerations

MISO military information support operations

NOEM national operational environment model

NTC National Training Center

OAKOC Observation and fields of fire, Avenues of approach, Key 

and decisive terrain, Obstacles, Cover and concealment

OneSAF One Semi-Automated Force

OODA observe, orient, decide, and act

ORSA Operations Research and Systems Analysis

PMESII political, military, economic, social, information, and 

infrastructure

PMESII-PT political, military, economic, social, information, 

infrastructure, physical environment, and time

PSOM peace support operation model

RAND American nonprofit global policy think tank 

RebeLand an agent-based model built upon the MASON agent-based 

framework

RFI Request for Information

RWISE reference world information and simulation environment

S&T Science and Technology

SAF semi automated force

SEAS synthetic environment for analysis and simulation

SME subject matter expert

SOAR state, operator and result (often refers to a cognitive 

architecture)

SOCOM Special Operations Command
Approved for public release; distribution is unlimited.

201

TRADOC US Army Training and Doctrine Command

UK United Kingdom

vmStrat Versatile Multiscale Strategist

V-SAFT virtual strategic analysis and forecasting tool

W-ICEWS worldwide-integrated crisis early warning system

XO Executive Officer
Approved for public release; distribution is unlimited.

202

1 DEFENSE TECHNICAL

(PDF) INFORMATION CTR

DTIC OCA

2 DIRECTOR

(PDF) US ARMY RESEARCH LAB

RDRL CIO L

IMAL HRA MAIL & RECORDS

MGMT

1 GOVT PRINTG OFC

(PDF) A MALHOTRA

1 DIR USARL

(PDF) RDRL CII T

T HANRATTY