A DoD CCRP/NATO Collaboration

This major revision to the Code of Best Practice for

C2 Assessment is the product of a NATO Research

and Technology Organisation (RTO) sponsored

Research Group (SAS-026). It represents over a

decade of work by many of the best analysts from

the NATO countries. A symposium (SAS-039) was

hosted by the NATO Consultation Command Control

Agency (NC3A) and provided the venue for a rigorous

peer review of this code.

This publication is the latest in a series produced by

the Command and Control Research Program (CCRP)

under the auspices of the Assistant Secretary of

Defense (C3I). The CCRP has demonstrated the

importance of having a research program focused on

the national security implications of the Information

Age. The research sponsored and encouraged by the

CCRP contributes to the development of the

theoretical foundations necessary to support the

Information Age transformation of the Department.

Other CCRP initiatives are designed to acquaint

military and civilian leaders with emerging issues

related to transformation. This CCRP Publication

Series is a key element of this effort.

Check our Web site for the latest CCRP activities and publications.

This code is also available on our Web site.

www.dodccrp.org
DoD Command and Control Research Program

ASSISTANT SECRETARY OF DEFENSE (C3I)

&

CHIEF INFORMATION OFFICER

Mr. John P. Stenbit

PRINCIPAL DEPUTY ASSISTANT SECRETARY OF DEFENSE (C3I)

Dr. Linton Wells, II

SPECIAL ASSISTANT TO THE ASD(C3I)

&

DIRECTOR, RESEARCH AND STRATEGIC PLANNING

Dr. David S. Alberts

Opinions, conclusions, and recommendations expressed or implied

within are solely those of the authors. They do not necessarily

represent the views of the Department of Defense, or any other U.S.

Government agency. Cleared for public release; distribution unlimited.

Portions of this publication may be quoted or reprinted without

further permission, with credit to the DoD Command and Control

Research Program, Washington, D.C. Courtesy copies of reviews

would be appreciated.

Library of Congress Cataloging-in-Publication Data

NATO code of best practice for command and control assessment.—Rev. ed.

 p. cm.

 ISBN 1-893723-09-7 (pbk)

 1. Command and control systems—North America—Evaluation. 2. Command

and control systems—Europe—Evaluation. 3. North Atlantic Treaty Organization—

Armed Forces—Equipment—Quality control. I. Title.

 UB212 .N385 2002

 355.3’304180973--dc21

 2002013927

October 2002
NATO Code

of

Best Practice

for

C2 Assessment

Revised 2002 Reprinted

by
i

Table of Contents

Chapter 1—Introduction ...................................... 1

Chapter 2—Preparing For Success: Assessment

Participants, Relationships, and Dynamics ..... 21

Chapter 3—Problem Formulation ..................... 53

Chapter 4—Solution Strategies ......................... 69

Chapter 5—Measures of Merit ........................... 89

Chapter 6—Human Organizational Factors ..... 127

Chapter 7—Scenarios ...................................... 163

Chapter 8—Methods and Tools ....................... 187

Chapter 9—Data ............................................... 229

Chapter 10—Risk and Uncertainty .................. 247

Chapter 11—Products ...................................... 267

Annex A—References .......................................A-1

Annex B—Acronyms .........................................B-1

Annex C—The MORS Code of Ethics ..............C-1

Annex D—Human And Organisational Issues

Checklist ............................................................D-1

Annex E—Alternate Definitions of Risk and

Uncertainty.........................................................E-1

Annex F—SAS-026 History ............................... F-1
iii

List of Tables

and Figures

Table 1.1—Comparison of Symmetric,

Conventional Warfare and OOTW Missions &

Principles Mission ................................................ 6

Figure 2.1.—Example of the Mapping of Roles

onto Players—for a Complex C2 Project (the

Evaluation of Immediate Reaction Task Force

(Land) C2 Concept) ............................................ 27

Table 2.1.—Knowledge, Capabilities and Skills

Needed by OR/OA Cell Team Members ............ 33

Figure 2.2—C2 Assessment Process ............... 36

Figure 3.1—The Formulated Problem............... 59

Figure 3.2—Problem Formulation ..................... 64

Figure 4.1—From Problem Formulation to Solution Strategy ....................................................... 71

Figure 4.2—Solution Strategy ........................... 77

Figure 4.3—Overall Study Plan ......................... 79

Figure 5.1—Relationships of Measures of

Merit .................................................................... 93

Table 5.1—Validity Criteria of Measures .......... 99

Table 5.2—Reliability Criteria of Measures .... 100

Table 5.3—Other Useful Criteria ..................... 100
iv

Table 5.4—Examples of Time- and AccuracyBased Measures ............................................... 103

Table 5.5—Examples of MoPs and MoCEs .... 104

Table 5.6—Normality Indicators ...................... 108

Figure 6.1—Representation of the C2 System,

based on Mandeles et al. (1996) ...................... 149

Figure 6.2—Decisionmaking Drivers .............. 152

Figure 7.1—The Scenario Framework ............ 175

Figure 7.2—Intersections in Hierarchy ........... 179

Figure 7.3—Segmentation Hierarchy Range .. 180

Table 8.1—Some Recommended Tools by

Category Type .................................................. 190

Figure 8.1—Model Linkage .............................. 195

Figure 8.2—Model-Tool Hierarchy .................. 205

Figure 8.3—Modelling Guidelines ................... 208

Figure 9.1—Data Taxonomy ............................ 231

Figure. 10.1—Illustration of the Variety of

Variables Relevant to Decisionmaking ........... 258
1

CHAPTER 1

Introduction

So the principles which are set forth

in this treatise will, when taken up by

thoughtful minds, lead to many

another more remarkable result; and

it is to be believed that it will be so

on account of the nobility of the

subject, which is superior to any

other in nature.

—Galileo Galilei (1638)

Command and Control

Assessment Challenges

N

ATO and its member nations are in the midst of a

revolution in military affairs. There are three major

dimensions to this revolution—a geopolitical

dimension, a technological dimension, and a closely

coupled conceptual dimension. This multidimensional

revolution poses significant new challenges for

analysis in general and for command and control

assessment in particular.

The changed geopolitical context is characterised by

a shift from a preoccupation with a war involving NATO

and the Warsaw Pact to a concern for a broad range

of smaller military conflicts and Operations Other Than
2 NATO Code of Best Practice for C2 Assessment

War (OOTW). Analysts will increasingly be called upon

to provide insights into these non-traditional operations.

Advances in technology, particularly informationrelated technologies, offer military organisations

unprecedented opportunities to significantly reduce the

fog and friction traditionally associated with conflict.

At the same time, they may prove to be challenges in

themselves across a wide variety of realms—technical,

organisational, and cultural.

To the extent that they can be achieved, significantly

reduced levels of fog and friction offer an opportunity

for the military to develop new concepts of operations,

new organisational forms, and new approaches to

Command and Control (C2), as well as to the

processes that support it. Analysts will be increasingly

called upon to work in this new conceptual dimension

in order to examine the impact of new informationrelated capabilities coupled with new ways of

organising and operating.

Definition of Command and Control

C2 has been defined by NATO as Military Function

01: “The Organisation, Process, Procedures and

Systems necessary to allow timely political and military

decisionmaking and to enable military commanders

to direct and control military forces.” (NATO 1996) C2

systems are further defined in NATO documents to

include: headquarters facilities, communications,

information systems, and sensors & warning

installations. (NATO 1998).

Other terms are used in NATO member nations that

are synonymous with, or closely related to, C2. These
Chapter 1 3

include Command, Control, and Communications or

Consultation (C3), and Computers (C4), and

Intelligence ([C3I] or [C4I]), and Surveillance and

Reconnaissance (C4ISR). The term CIS is sometimes

used to refer to command information systems. More

recently the term “C2” has referred to the collaborative

and consultative processes that are an inherent part

of coalition operations.

For the purposes of this Code of Best Practice (COBP),

the term C2 is intended to be an umbrella term that

encompasses the concepts, issues, organisations,

activities, processes, and systems associated with the

NATO definition of C2 as well as the other terms

enumerated above.

Uniqueness of C2 Analyses and Issues

The focus of military research and analysis has

predominantly been on the physical domain. C2 issues

differ in fundamental ways from physics dominated

problems. C2 deals with distributed teams of humans

operating under stress and in a variety of other operating

conditions. C2 problems are thus dominated by their

information, behavioural, and cognitive aspects that

have been less well researched and understood. This

focus creates a multidimensional, complex analytic

space.

Military operations involve multi-sided dynamics

encompassing friendly, adversary, and other actors

including:

• Action-reaction dynamics;
4 NATO Code of Best Practice for C2 Assessment

• Tightly connected interaction among subjective

elements such as cultures, morale, doctrine,

training, and experience and between those

subjective elements and the combat arena;

• Non Governmental Organisations (NGO);

• Private Volunteer Organisations (PVO);

• International organisations;

• International corporations; and

• Transnational, subnational, criminal, and

terrorist organisations.

C2 issues are difficult to decompose and recompose

without committing errors of logic. Moreover, the

composition rules, by which the various factors that

impact C2 interact, are poorly understood except in

arenas that have been previously studied in detail.

Finally, the C2 arena is weakly bounded by issues

that on initial examination appear quite finite, but prove

to be linked to very high-level factors. For example,

tactical performance may be tied to national culture.

Analyses of C2 are also often constrained by factors

that are beyond the boundaries of the research. For

example, security policies may restrict data availability

and otherwise constrain the analysis. The availability

of data often limits the scope of an analysis. Moreover,

the time and resources available to conduct an analysis

are often severely constrained because the decision

processes being supported are being driven by outside

planning, operational, or budget and decision

processes. This should be seen as a challenge rather

than a problem. Uncertainty and risk associated with a
Chapter 1 5

lack of appropriate data need to be embraced as part

of the analytical approach. It is unreasonable to expect

that data would be available for the performance of

future systems and processes that do not yet exist. An

experimental component and a modelling and

simulation component need to be integrated into modern

C2 analyses in order to close the gap in knowledge

and data.

Finally, because of the complexity of C2 processes

and systems, analysis in this area requires the ability

to understand how Dimensional Parameters (DP),

Measures of Performance (MoP), Measures of C2

Effectiveness (MoCE), Measures of Force

Effectiveness (MoFE), and Measures of Policy

Effectiveness (MoPE) are linked and impact on one

another. The cumulative set of these measures is

denoted as Measures of Merit (MoM) in the COBP.

Determining the precise nature of these relationships

nearly always proves to be an analytic challenge.

Taken together, all these factors mean that C2 modelling

and analysis are more uncertain and therefore more

prone to risk than their equivalents in conventional

weapon and platform analyses. Indeed, C2 issues have

long been regarded as difficult to analyse. Many

operational analysis (OA) studies have simply assumed

perfect C2 in order to focus on other variables. As a

result of these characteristics of C2 analysis, these

endeavours will require a heavy element of research

within each analysis. This COBP is intended to assist

the community in dealing with, and overcoming, the

barriers to effective analysis of C2.
6 NATO Code of Best Practice for C2 Assessment

Table 1.1. Comparison of Symmetric, Conventional Warfare and

OOTW Missions & Principles Mission

In symmetric conventional warfare, the mission tends

to be relatively stable, there is a clear focus on the

enemy, and the military has a common understanding

and commitment.1 Conversely, in OOTW the mission

is often more dynamic. This is captured by the often

pejorative term “mission creep.” In many of the

operations in question there is no “enemy.” This is

obviously true for operations such as humanitarian

assistance and disaster relief. In addition,

peacekeeping activities involve protagonists who must

be treated even-handedly if the operation is to be

Differences Across the Mission Spectrum

There are significant differences among the different

parts of the mission spectrum (e.g., MoM) that the

assessment team needs to take into consideration.

Table 1.1 highlights the differences between traditional

combat and OOTW.
Chapter 1 7

successful. In the latter missions, political-military

ambiguities frequently result in uncertain

understanding of the goals and objectives of the

mission and a limited commitment.2 (Starr, Haut &

Hughes, 1997)

Principles

Military theorists have frequently propounded basic

principles of conventional warfare. Three often cited

principles include the need for unity of command, the

importance of hierarchical decisionmaking, and the

criticality of achieving surprise in operations. A recent

book has proposed alternative principles for OOTW.

(Alberts & Hayes, 1995) It cites the need for unity of

purpose, consensus decisionmaking, and

transparency of operations.

Information

In conventional warfare, the issue of information

gathering and management focuses on the issue of

“known unknowns” (e.g. Where are the enemy’s

battalions?). For that case, the key question is how to

get the needed information (e.g. What are the key

signatures for the targets in question? What sensors

should we task to exploit those signatures?). Clearly,

the focus is on the enemy military and one objective is

to assemble a complete, timely, and accurate common

picture of the air-land-sea situation. The result is a

very large, time-sensitive database, but one that is

relatively well structured (e.g. enemy order of battle).

Conversely, in OOTW, the problem of information

gathering and management is dominated by “unknown

unknowns.” Thus, the primary question to address is
8 NATO Code of Best Practice for C2 Assessment

what information to get. The information focus is much

more diffuse because of the myriad of military, political,

economic, and social factors that must be considered.

Consequently, situation awareness is much more

complex. Political considerations often make it prudent

to limit the dissemination of information, creating a

tension between the desire to create shared

awareness by increasing information sharing and the

need, for political and/or security reasons, to limit

information sharing. The resulting databases are

frequently larger and less structured.

Analysis

Over many years, the military operations research

community has become relatively adept at analysing

key aspects of symmetric conventional warfare. As an

illustration, analyses of ground warfare often focus on

battalion-level operations and techniques have emerged

to integrate across those results to derive insights into

campaign outcomes. The focus is on military systems

and organisations, and the techniques in question

involve a broad set of methods (e.g., mathematical

programming, decision theoretic approaches) and tools

(e.g., models and simulations). Analyses of C2 issues

remain among the most challenging, even in warfare

contexts. In addition, analyses of OOTW often require

consideration of individual behaviour. It has proven very

difficult to integrate across these results to derive a

comprehensive understanding of the problem. The issue

is compounded by the many factors that have to be

considered in the analysis process (e.g. military, political,

economic, social). This has led to the application of

“softer” analytic approaches (e.g., extensive reliance

on expert elicitation). Moreover, the very nature of
Chapter 1 9

warfare appears to be changing. For example,

asymmetric threats are becoming more common,

information technologies are impacting C2 processes,

and organic structure and dynamics are changing

rapidly and in ways we do not fully understand.

Types of C2 Assessments

The assessment team could be called upon to support

a wide variety of sponsors (e.g. acquirers of C2

systems, long range planners and programmers,

developers of requirements, operational commanders,

and trainers). These sponsors will bring different

problems to the assessment team (e.g. assessment

of alternative systems or concepts, identification and

selection of alternative courses of action in an

operational context). Some of these will deal with a

specific mission (e.g. air defence) while others will need

to deal with the entire mission spectrum from forward

presence to high intensity conventional war.

Specific problems that the team may be called upon

to address:

Requirements Analysis

• Derivation of specific C2 requirements from

broad statements of objectives; and

• The establishment of a minimum, standard, or

expected level of performance.
10 NATO Code of Best Practice for C2 Assessment

Assessment of Alternatives

• Comparison and selection of alternative systems

that may be very dissimilar but are designed to

achieve a similar purpose;

• Assessment of the utilisation of a system in a new

or unexpected application domain or mission;

• Trade-offs between C2 systems and combat systems;

• Analysis of the impact of an organisational change;

• Determination of the most cost-effective

approach to achieving the desired objective; and

• Comparison of a replacement system or

components of a system.

Research Issues

• Effectiveness of human decisionmaking as a

function of system performance or other factors;

• Effectiveness of C2 training; and

• Impact of collaboration on C2 quality.

Support to Operations

• Course of action analysis;

• Real time assessment of mission effectiveness; and

• Rehearsal assessment.
Chapter 1 11

Purpose and Scope of the COBP

This COBP offers broad guidance on the assessment

of C2 for the purposes of supporting a wide variety of

decisionmakers and the conduct of C2 research

described above. It should be noted that this COBP is

focused upon the assessment challenges associated

with the nature of C2 and does not attempt to

specifically address the unique properties and

constraints associated with each of the many C2-

related problem domains.

Given the increasing interdependence among the

elements of a mission capability package3

(organisation; doctrine; C2 concepts, processes,

systems; materiel; education; training; and forces), C2-

related analysis cannot easily be done in isolation from

a more comprehensive mission analysis. This COBP

is meant to support analyses that go beyond the

traditional boundaries of C2 analyses.

This new version of the COBP for C2 assessment was

developed by SAS-026 building upon the initial version

of the COBP produced by SAS-002. This new COBP

is a synthesis of decades of expertise from various

countries and hundreds of analyses. The COBP was

developed using a set of case studies to test out the

varied advice and guidance received, and incorporates

feedback from users of the initial version. Lastly, SAS039 provided a peer review of the final draft product.

The earlier version focused on the analysis of ground

forces at a tactical echelon in mid to high intensity

conflicts. Consequently, the initial version of the COBP

did not completely address the full range of important

issues related to C2. In developing this new version of
12 NATO Code of Best Practice for C2 Assessment

the COBP, SAS-026 explicitly focused upon OOTW,

the impact of significantly improved information related

capabilities, and their implications for military

organisations and operations. In addition, SAS-026 was

cognisant of the fact that NATO operations are likely to

include coalitions of the willing, which might involve

Partnership for Peace (PfP) nations, others partners

outside of NATO, international organisations, and

NGOs. NATO operations may also be “out of area.”

Feedback from users of the original COBP also

identified a number of ways in which the original COBP

could be improved. These areas were addressed

during the development of this version of the COBP.

Cost analyses continue to be explicitly excluded for

two reasons. First, cost analysis is a mature discipline

that experienced operational analysts already practice.

Hence, C2 issues are not unique in the arena. Second,

most nations have already developed approaches to

cost analysis and cost effectiveness that are consistent

with their national approaches to accounting. Because

these national practices differ among NATO members,

no single approach would be appropriate.

As this COBP is being drafted, novel experiments with

new information-related capabilities, particularly

networking and ways to accomplish their assigned tasks

abound. Advances in technology are expected to

continue at an increasing rate and spur both sustaining

and disruptive innovation in military organisations. It is

to be expected that this COBP will need to be

periodically revisited in light of these developments.
Chapter 1 13

Overview of COBP Assessment

Philosophy

The COBP assumes that the objective of a C2 system

is to exercise control over its environment, through

either adaptive or reactive control mechanisms, or

some combination of those two approaches. This

provides the context and point of departure for the

assessment of C2.

Analysis of C2 should consider all the relevant actors,

military command levels, and functions involved and

should investigate issues of integration across

disparate organisations, military command levels, and

functional domains over time. Consideration should

also be given to the robustness and security of

information systems and to human computer interface

issues. Human behavioural, physiological, and

cognitive factors, along with organisational and

doctrinal issues, must be considered in C2 analyses.

C2 assessments must also consider a range of

missions, adversary capabilities, and adversary

behaviours. Moreover, it must be understood that

adversaries will use asymmetric tactics and techniques

to deny or exploit differences in technology, force size,

information systems, or cultural factors. Hence,

scenarios and analyses that deal with an appropriate

set of all these dimensions should be considered in

either the main research design or in the excursions

to assess risks and uncertainty.

The evaluation of C2 issues depends in important ways

on both distinguishing and linking dimensional

parameters, measures of performance, measures of
14 NATO Code of Best Practice for C2 Assessment

C2 effectiveness, and measures of force and policy

effectiveness. Modelling and other tools must be

designed to support this requirement.

Tools and data used in C2 analysis should conform to

good OA processes and practices and, to the extent

feasible, should be subject to Model Verification,

Validation, and Accreditation (VV&A) and to Data

Verification, Validation, and Certification (VV&C).

Interoperable analytical infrastructures (e.g. data

dictionaries, glossaries, models, tools, data sets) are

necessary to facilitate the efficient proliferation and

reuse of study results and data within the broader

interdisciplinary research community.

Because the complexity of C2 and the requirements

for its analysis are often underestimated by

decisionmakers, a continuing dialogue between

analysts and those decisionmakers is necessary both

to scope the problem properly and to ensure that the

analytic results are properly understood. Part of this

process includes performing sensitivity analyses and

other common practices designed to ensure the validity

and reliability of the results.

Changes to C2 systems will often lead to changes in

military concepts, command approaches, doctrine,

Tactics, Techniques, and Procedures (TTP), and related

factors, which must also be considered in the analysis.

Current State of Practice in C2 Analysis

Assessment of C2 issues typically employs classic

tools of OA. Relatively few specialised tools and

methods have been developed for C2. Moreover, those
Chapter 1 15

specialised tools generated to deal with the unique

aspects of C2-focused research are generally not as

well understood as those used in more traditional

warfare modelling domains. C2 analysts will often find

themselves having to develop tools and approaches

appropriate for their research agendas. However, a

general analytic process can be identified that will

enhance the likelihood that an OA analyst can conduct

successful analyses.

Organisation of the COBP

This COBP is organised into four themes. The first

theme deals with study dynamics, problem formulation,

and the development of a solution strategy. The

second theme identifies and discusses in depth the

essential elements of assessment: measures of merit,

scenarios, human and organisational issues, data, and

tools. The third theme addresses issues related to risk

and uncertainty while the final theme describes the

range of assessment products.

This represents a significant enhancement of the initial

COBP. In particular the first, third and fourth themes

were not treated in detail in the initial version of the

COPB. In addition, material has been added to the

second theme to address the unique assessment

challenges associated with OOTW.

Brief History of SAS-026

SAS-026 builds upon almost a decade of work that

began with the formation of the Ad Hoc Working Group

on the Impact of C3I on the Battlefield by Panel 7 of the
16 NATO Code of Best Practice for C2 Assessment

NATO Defence Research Group in 1991 to assess the

state of the art in C2 analysis. Based on the

recommendations of the Ad Hoc Working Group, Panel

7 constituted Research Study Group-19 (RSG-19) to

address issues of methodology, measures of merit, and

tools and analysis. The panel also addressed issues of

improving a nation’s capability to examine C2 acquisition

and decisionmaking. At the October 1995 RSG-19

planning meeting, the group determined that the primary

product of RSG-19 was to be a Code of Best Practice

for assessing C2. As part of selected RSG-19 meetings,

workshops would be conducted to support the

development of the major sections of the COBP.

Workshops were conducted on Measures of Merit

(Canada), Scenario Development (Netherlands), C3I

Systems, Structures, Organisations, and Staff

Performance Evaluations (Norway), and Models Used

for C3 Systems and Analysis (US/UK). Representatives

from the nations in parentheses took the lead in

organising the workshops and summarising the results.

The minutes of the workshops provide further

illustrations of the techniques presented in the COBP.

At the October 1996 meeting, the group took up a

request by Panel 7 to conduct a symposium on

modelling and analysis of C3I, which was scheduled

at the July 1997 meeting for January 1999. This

symposium was a forum for presentation and

discussion of the COBP and related topics.

At the July 1997 meeting, in response to a query by

Panel 7, the group discussed, acknowledged, and

agreed on the need for a follow-on group to SAS-002.

An exploratory group on organisational change (SASE05) was formed to recommend a way ahead.
Chapter 1 17

SAS-E05 recommended the formation for a follow-on

activity to SAS-002 to accomplish four objectives:

• Demonstrate and assess the initial version of

the COBP;

• Revise and extend the COBP;

• Identify research areas; and

• Facilitate the adoption of the COBP.

The SAS panel concurred in May 1999 and approved

the formation of SAS-026, which began its 2 1/2-year

plan of work in a symposium in January 2000.

Chapter 1 Acronyms

C2 – Command and Control

C3(I) – Command, Control, Communications (and

Intelligence)

C4(I) – Command, Control, Communications, and

Computers (and Intelligence)

C4ISR – Command, Control, Communications, Computers,

Intelligence, Surveillance and Reconnaissance

CIS – Command Information Systems

COBP – Code of Best Practice

DP – Dimensional Parameters

MoCE – Measures of C2 Effectiveness

MoFE – Measures of Force Effectiveness

MoM – Measures of Merit
18 NATO Code of Best Practice for C2 Assessment

MoP – Measures of Performance

MoPE – Measures of Policy Effectiveness

NGO – Non Governmental Organisations

OA – Operational Analysis

OOTW – Operations Other Than War

PfP – Partnership for Peace

PVO – Private Volunteer Organisations

RSG-19 – Research Study Group-19

TTP – Tactics, Techniques, and Procedures

VV&A – Verification, Validation, and Accreditation

VV&C – Verification, Validation, and Certification

Chapter 1 References

Alberts, D. (1996). The unintended consequences of

information age technologies: Avoiding the

pitfalls, seizing the initiative. Washington, DC:

National Defense University.

Alberts, D., & Hayes, R. E. (1995). Command

arrangements for peace operations.

Washington, DC: National Defense University.

North Atlantic Treaty Organisation (NATO) Bi-MNC C2

plan part 2 – Command and control

requirements. (1998).

North Atlantic Treaty Organisation (NATO) Annex B

to MC Guidance for Defence Planning. MC-299/

5. (November 1996)
Chapter 1 19

1As an illustration, General Colin Powell, then Chairman, Joint

Chiefs of Staff, summarised the mission in Desert Storm by

stating that “First, we will cut off the enemy and then we will kill

it.” (Pentagon Briefing, Wednesday January 23, 1991.)

2As an example, the US Congress has continually sought to

impose an arbitrary deadline for US forces to withdraw from

Bosnia.

3Mission capability packages include all of the elements

necessary for an operation. (Alberts, 1996).

Starr, S.H., Haut, D., and Hughes, W. (1997)

“Developing Intellectual Tools to Support C4ISR

Analyses for Operations Other Than War,”

Proceedings of the Third International

Symposium on Command and Control

Research & Technology, National Defense

University, Fort McNair, Washington, DC, pp.

600-608, June 17 – 20, 1997.
21

CHAPTER 2

Preparing For

Success:

Assessment

Participants,

Relationships,

and Dynamics

For hypotheses ought...to explain the

properties of things and not attempt to

predetermine them except in so far as

they can be an aid to experiments.

—Isaac Newton (1687)

We have run out of Money—now we

have to think.

—Ernest Rutherford (1871-1937)

Overview

T

his chapter is organised into three parts. The first

discusses the roles played by the significant players

associated with a Command and Control (C2)
22 NATO Code of Best Practice for C2 Assessment

assessment and how these roles affect the design and

conduct of the assessment. The second part identifies

the major phases of a C2 assessment and their iterative

nature. The concluding section addresses the subject

of professional ethics and standards of conduct.

Assessment Participants

Like their subject, the organisation of C2 studies

involves complex interrelationships. It is crucial for the

analytical team to establish which individuals and

organisations are involved at an early stage of the

study. It is prudent for the analytical team to map the

roles described below onto the individuals and

organisations involved and to understand their

interrelationships. An example of such a mapping is

at Figure 2.1. Appendix 1 to this chapter provides a

brief explanation of the organisations involved.

Due to the dynamic nature of such projects, those

involved should not be surprised if the nature of the teams

involved might have to expand or change with time.

Assessment Team

The assessment team is working for the sponsor or

client (sponsor). The team consists of a senior team

leader (who may also be referred to as the project

manager), a core set of analysts, subject matter

experts including military officers, and supporting staff

who are working on the study on a day to day basis.

The team provide the legitimacy and authority for the

study. The sponsor will provide the terms of reference,

access to needed information, and identify the desired

products. It is important for the analytical team to
Chapter 2 23

understand exactly why the sponsor wants the study

and what the sponsor wants to do with the results.

Decisionmakers or Problem Owners

The decisionmakers are the individuals or

organisations that are expected to make decisions

wholly or partially based on the output or findings of

the study. If there is no decision to be made (i.e. this

is an exploratory study) then the decisionmakers could

be referred to as problem-owners. It is important for

the assessment team to understand exactly what type

of assessment the decisionmakers want the study to

support. The decisionmakers may or may not be in

command of or part of the sponsor’s organisation.

Complex problems may arise when the decisionmaker

is several steps laterally away in the organisation from

the sponsor and study team.

Stakeholders

The stakeholders are the persons or organisations that

are directly or indirectly affected by the study outcome.

Stakeholders may also play other roles. The

assessment team must be aware of the potential for

conflict when the stakeholders do not include the

sponsor or decisionmaker. Complex problems may arise

in the provision of data for the study, as it is the

stakeholders who may have to provide the data, set

the security or releasibility of that data (and hence the

study), and/or agree that the data are representative.

For these reasons it is essential that the analysts

establish a working relationship with the stakeholders

early in the process.
24 NATO Code of Best Practice for C2 Assessment

Bill Payer

The bill payer is the organisation or individual official

paying for the study. It is important for the assessment

team to know the level of resources available. Bill

payers will normally have a direct interest in the

outcome of the study and may be one or more of the

other players. Contractual authorities have the legal

authority to let contracts on behalf of the bill payer.

Existing Study Teams

The assessment team must be aware of and sensitive

to the existence of teams in other related study areas.

Should such teams exist, the assessment team should

endeavour to exploit the work done and available skills

and techniques. Such external teams may also be

appropriate for membership in peer reviews.

Future Study Teams

The assessment team must be aware of and sensitive

to the needs of future analyses and assessments. Data

collection, method documentation, and the archiving

of data, methods, models and results are fundamental

responsibilities of all professional analysts. Method and

data should be (as far as is practicable) disseminated

and published.

Peer Reviewers

Outside experts brought in to look at the work and

provide constructive criticisms are called peer

reviewers. Peer review teams could be composed of

specialists and other study teams in related subject
Chapter 2 25

areas and should include representatives from all key

disciplines in the assessment.

Data Providers

Data providers are the individuals and organisations

that possess data and information useful to the

assessment team. Many of these will be stakeholders.

The motivation to provide data to the study must be

developed by the analytical team and the sponsor.

Assumption Providers

Assumption providers are the individuals or

organisations that can provide “givens” such as future

doctrine, performance data, force mixes,

organisational structures, and scenarios. Creation of

a positive relationship with these organisations is

important to the study.

Data Collectors

In some C2 analyses, where data must be extracted from

real world experiences, exercises, experiments, and

wargames, teams of data collectors and subject matter

experts will be required. The identification of people with

the appropriate background and training as data

collectors is an important element of such studies.
26 NATO Code of Best Practice for C2 Assessment

Relationships Among Participants

and the Conduct of the Assessment

Relationships

Figure 2.1 illustrates how complicated the participant

roles and relationships can be in a real C2 assessment.

This particular figure represents the organisations

involved in the recent Immediate Reaction Task Force

(Land) (IRTF(L)) C2 Concept evaluation that

completed at the end of 2001 (Candan & Lambert,

2002). Appendix 1 to this chapter provides a brief

explanation of the organisations involved. Although not

all projects will be this complex, many important C2

assessments will.

Through the prudent act of mapping the roles of the

participants of the study, the potential conflicts of

interest and complex interactions can be identified.

One method to mitigate these is to present or conduct

this activity openly and discuss with all involved so

that all potentially affected participants are aware of

the possibility of future conflict and the fact that all

participants fall into one or more roles within a project.

In the event of conflict with other participants in the

project the assessment team should address the

issues in a neutral and independent manner.
Chapter 2 27

Figure 2.1. Example of the Mapping of Roles onto

Players—for a Complex C2 Project (the Evaluation of

Immediate Reaction Task Force (Land) C2 Concept)

Understanding the Context of the

Assessment

The relationship among the assessment team, the key

sponsor, and the stakeholders is of paramount

importance and perhaps, more than any other single

factor, will influence the course and success of the

effort. Accordingly, adequate attention needs to be paid

to understanding the situation facing the key sponsor

and stakeholders as much as the subject under study.

The assessment team should be aware that the

different participants will have divergent perspectives

and may have divergent agendas.

Therefore the assessment team should understand the

background of the individuals involved, their

organisational settings, roles and responsibilities, their
28 NATO Code of Best Practice for C2 Assessment

history, and their current situation. Contact with analysts

who have worked with this sponsor and review of prior

analyses for this sponsor facilitate this objective.

It is good practice to build and maintain long-term

relationships with the sponsor and stakeholder

organisations. This will yield substantial dividends in

the form of easier communication, greater trust, and

stronger support.

A Continuing Dialogue

It is important that a dialogue with the sponsor and

stakeholders is maintained by the assessment team

throughout the study. As there is no single “language”

that will describe the study problem, it is important to

spend time at the beginning of a study to establish a

common “language” that both the assessment team

and the sponsor and stakeholder can understand. This

point may seem obvious in a NATO setting in which

the participants speak many different natural

languages. However, it is equally important in a single

language setting because common words and phrases

have different meanings for different organisations,

services, and even individuals within a single

organisation. Regular meetings and contact will

minimise misunderstandings. From a professional

point of view, Operations Research (OR) and

Operations Analysis (OA) analysts will always wish to

inform the sponsor and stakeholders of key

developments and/or challenges as the study unfolds.

Regular and routine interactions need to be built into

the project plan. If there are multiple sponsors and

stakeholders and other key actors, the assessment

team should try to meet them jointly, particularly when
Chapter 2 29

decisions need to be made. Separate meetings will

often lead to inconsistent guidance and will place the

assessment team in a position of trying to

accommodate differing interests.

The development of a collectively agreed upon Study

Glossary that captures the definitions of words, phrases

and acronyms used in the study is a useful tool.

Terms of Reference

A good “term of reference” covers goals, scope,

products, schedule, and resources. These will

determine the focus of the assessment and establish

limits or freedoms granted to the assessment team

within the sponsor’s and stakeholder’s organisations.

Letters of introduction and instructions to actors within

the sponsor and stakeholder organisations may also

be useful.

Understanding How the Output of the

Study Will be Used

It is important to know at an early stage in the project

what the products of the study are to be used for by

the sponsor and stakeholder organisations. The

expected end product will set the tone and relative

importance of the project in the eyes of the sponsor,

stakeholder, and other actors. The assessment team

needs to establish and understand the products

needed or desired by the sponsor and stakeholder.

For example, a study could be used to create a

significant impact on the stakeholder’s domain, gain a

greater understanding of the issues, produce an
30 NATO Code of Best Practice for C2 Assessment

improved capability to perform future work, and/or

make contributions to the body of knowledge.

Budget

The sponsor will have limited resources with a study

budget in mind. When the sponsor’s resources are

limited to a level below what is required to support a

quality study, the assessment team will need to

suggest strategies to address the shortfall. Alternative

approaches include decomposing the problem and

only undertaking the core part of the study that is

affordable, linking the sponsor to other actors that have

an interest in the same or a similar problem and who

could contribute resources, and/or stretching the

project over a longer time so that resources from future

budget cycles become available. In developing

strategies that involve doing only a part of the study to

satisfy budget constraints, care must be taken to

ensure that the product that will be produced provides

a meaningful answer or contribution and does not

depend upon a follow-on effort that may or may not

be funded.

It may take a complete iteration of the assessment

phases of the project to establish the complete scope

of the project and the resources required. Therefore it

is good practice in large C2 projects to allow the

assessment team to perform a rapid first pass of all

the phases of the project to help establish the budget

required. This is contrary to the usual practice of setting

the budget in stone immediately following the initial

Problem Formulation or Solution Strategy phase (see

Section 2-E below).
Chapter 2 31

Relationships

Figure 2.1 illustrates how complicated the participant

roles and relationships can be in a practical C2

assessment. This particular figure represents the

organisations involved in the recent Immediate

Reaction Task Force (Land) (IRTF(L)) C2 Concept

evaluation that completed at the end of 2001 (Candan

& Lambert, 2002). Although not all projects will be this

complex, many important C2 assessments will be.

Building an Assessment Team

Skills Available to the Assessment Team

Following initial problem formulation (Chapter 3), the

precise skills and experience required by the

assessment team will need to be established. Typically,

the assessment team will need to be interdisciplinary.

The wide range of skills and experience required can

be allocated between a full-time core team and a

collection of consultants or part-time team members.

As an example, an ideal breakdown of the skills available

to the assessment team involved in the evaluation of

the Immediate Reaction Task Force (Land) C2 concept

study1 is given below:

Skills: Core Team

• Project management;

• OR/OA skills: simulation, wargaming,

mathematical programming, database creation

and management, brainstorming and problem
32 NATO Code of Best Practice for C2 Assessment

structuring, scientific/military report writing/

editing;

• Cross military experience—i.e., OR/OA

personnel with military experience or military

personnel with OR/OA knowledge;

• Organisational theory; and

• Data collection (e.g., questionnaire and form

design).

Skills and Experience: Consultants and Part-time

Team Members

• Military (or access to practical experience of

problem under study);

• Training and exercise planning (if an exercise is

to be used as a vehicle for the study);

• Communications and information systems

specialists for the systems of the organisation

under study;

• Human computer interface expertise;

• Operations Other Than War (OOTW) related

issues (e.g., C2/Headquarters [HQ], media, civilmilitary cooperation—theory and practice);

• Social scientists (e.g., political, psychological,

economic, cultural, legal);

• Military history;

• Command experience;

• Deployment analysis;

• Intelligence/threat/area of operations expertise; and
Chapter 2 33

• Legal/contracts/administration expertise.

As another example of the skills required for C2

Assessment Studies, the skills required to provide OR

support to C2 elements (such as OR/OA support to

an operational HQ) is also analogous, as illustrated in

Table 2.1 (RTO, 1999).

Table 2.1. Knowledge, Capabilities and Skills Needed

by OR/OA Cell Team Members

Background of the Assessment Team

Building a C2 assessment team with this full breadth

of knowledge, capabilities, and skill requires a longterm commitment by the mother OR/OA organisation

to prepare a corpus of potential team members through

recruitment, education, training, and opportunities for

appropriate field experience.

Following the identification of the skills required for

the team, those analysts made available for the team

should ensure that they leave a basic understanding

of the military fields under consideration. Gaps in

experience should be rapidly filled through background

reading, short courses, field experience, or additional/
34 NATO Code of Best Practice for C2 Assessment

alternative analysts with the appropriate experience

and skills.

Forming the Assessment Team

In a study that involves dispersed and disparate

organisations and teams, the effort to command and

control the study group must be recognised and effort

and time built into the study plan. This can be, for

example, through maintenance of distributed working

environments such as web portals, information

campaign material and travel time to meetings. In these

cases the senior team leader will revert to a role more

akin to a project manager.

It is one thing to assemble a group of people, quite

another to forge them into a coherent effective team.

Sufficient time and a facilitating process should be built

in to the project plan for the group of individuals to

coalesce into a team.

Interdisciplinary Assessment

Team and Outside Relations

It follows that C2 analysis, particularly for OOTW issues,

should be done by an interdisciplinary assessment

team. Experienced analysts know that their work owes

success in no small measure to efficient working

relationships within the assessment team and with the

customer of analysis. Building good working

relationships among representatives of different

scientific cultures, such as OR/OA and IT analysts

grounded in (hard) physical sciences and mathematics

on one hand and (soft) social scientists on the other,

requires sufficient mutual understanding of
Chapter 2 35

methodologies and tools. In fact, differences in scientific

cultures can outweigh differences in natural cultures

provided that all members of the assessment team have

sufficient command of a common language. Therefore,

in addition to leadership and project management skills,

the head of the assessment team must have a good

general idea of the current state of all disciplines involved

in order to compose an efficient team and facilitate

interdisciplinary cooperation throughout the analysis.

Good personal and working relationships with the

customer of the analysis are essential for

understanding every aspect of the problem and being

able to arrive at a problem structure and solution

strategy that meets the customer’s immediate needs

in the light of the strategic objectives of the respective

OOTW. Knowing the customer’s position in the

command hierarchy and the degree of influence he/

she wields through informal relationships over

stakeholders and actors, the co-operation of which

might be essential for an implementation of analysis

results, and understanding the respective

consequences associated with alternative solutions is

important for assessing their acceptability and

organisational risk.

It is equally important for the assessment team to

establish working relationships with the potential

subjects of study in the early stages. This is essential

for capturing the nature and problem relevance of formal

and informal relations between all organisations, groups,

and individuals that are subjects of the study, finding

out about their motivations and agendas, and eliciting

firsthand information that is critical for solving the

problem such as their capabilities and the conditions

attached for their employment. However, the analyst
36 NATO Code of Best Practice for C2 Assessment

should be careful not to allow this effort to gain greater

understanding of the problem to introduce bias.

Assessment Phases, Process,

and Dynamics

It is important to realise that all of the elements of the

C2 assessment are interrelated. Hence Problem

Formulation, Solution Strategy, Measures of Merit,

Scenarios, Human/Organisational Factors, Models/

Tools/Data, and products are all interdependent.

Figure 2.2 illustrates the major phases and iterative

nature required for C2 assessments. The Assessment

Process diagram was the most difficult thing for the

SAS026 team to agree upon. In essence this diagram

is at the heart of the COBP (Starr, 2001). The

remainder of this chapter discusses the key points in

this diagram.

Figure 2.2. C2 Assessment Process
Chapter 2 37

Problem Formulation

The output of Problem Formulation (Chapter 3) specifies

the “what” of the assessment. C2 studies tend to be

complex and feature multiple attributes, some of which

may not be apparent at the start of the study. Neither

the assessment team nor the sponsor should be

surprised if the issues initially presented for study are

replaced by other issues that are closer to the underlying

causes of the initial problem or, in some cases,

symptoms presented. A consequence of the dynamic

nature of problem formulation is that the solution strategy

and any of the other elements of the assessment may

change as the study progresses. Problem formulation

should therefore be consciously and routinely iterated

during a study—especially when new attributes start to

appear. As a minimum an iteration should occur

immediately following the establishment of the initial

solution strategy and the assessment of study risk.

Additionally, the sponsor should be aware that the

nature of the assessment team, sponsor, or assumption

provider teams might also have to expand or change

with time. This has implications for planning, budgeting,

and tasking.

In nearly all C2 studies the assessment team will study

only a subset of the whole problem space due to the

sponsor’s sphere of interest. This fact must be

recognised by the assessment team. An initial study

of the complete problem space is essential to establish

this realisation. This will help the assessment team to

understand the context of the study and provide advice

to the sponsor on the actual underlying causes to his

problem and consequently the requirement to involve

other participants.
38 NATO Code of Best Practice for C2 Assessment

Solution Strategy

The next step is to develop a Solution Strategy

(Chapter 4) that specifies the “how” of the assessment.

Arising from the Solution Strategy agreed upon and

adopted by the sponsor are a set of terms of reference

(e.g., Statements of Work [SOW] for contracts) that

will determine what work is to be conducted, the

contractual obligations, deadlines, and resources.

Although these must be established as an

experimentation campaign plan3 and study

management plan (project plan) before work on the

project begins in earnest, flexibility must be built-in due

to the iterative nature of C2 assessment. The

assessment team must be aware of any preconceived

“solutions” that have been proposed by the sponsor,

stakeholders, and/or decisionmakers and explicitly

deal with these as appropriate, avoiding another

pressure to be steered in a particular direction. The

assessment team must note if its results are being

steered in a particular direction and follow ethical

behaviour in performing the study (see the end of this

chapter). In many cases a risk-based approach to C2

assessments can usefully complement the more

traditional cost-effectiveness approach. In particular,

this helps decisionmakers to deal with the uncertainties

of the real problem.

From a professional point of view analysts should always

defer the selection of a particular method until the problem

has been formulated and a solution strategy has been

defined. Recognise and beware of “preconceived”

solutions that could influence the assessment.
Chapter 2 39

Review

Once there has been a preliminary formulation of the

problem and development of a solution strategy, it is

imperative that an initial review be conducted. This

review should be conducted from multiple perspectives

(e.g. with respect to the sponsor’s initial problem, the

feasibility with respect to resources including team

skills and schedule, soundness of the proposed

analytic approach). As a result of this review, changes

will usually be made in both the problem formulation

and the solution strategy.

Measures of Merit, Scenarios, and

Human/Organisational Factors

At this stage the assessment team must specify the

hierarchy of Measures of Merit (MoM) (Chapter 5),

incorporate and identify relevant human and

organisational factors (Chapter 6), and specify the

appropriate scenarios (Chapter 7). As suggested by

the diagram, there is no unique sequence for doing

these tasks. Iteration is required to ensure that these

tasks are done in a coherent, consistent fashion. When

all of these tasks have been completed, the team has

specified the key variables to the necessary level of

detail with adequate considerations for assessment

validity and reliability.

When developing the MoM it is very valuable to involve

the sponsor in establishing the linkages between the

MoMs and the hierarchy of MoM. This is because then

the sponsor will then appreciate the dynamics of the

problem and the requirement for breadth in the study.

Although a full set of MoM must be derived in
40 NATO Code of Best Practice for C2 Assessment

accordance with the best practice noted in Chapter 5,

the MoM should be prioritised to focus on providing

support to the objectives of the study and be practical

and cost effective.

When selecting appropriate scenarios it is good

practise to utilise scenarios (if they exist) from a

standard set of scenarios approved for use within the

assessment and sponsor organisation. The sponsor

must always be approached for approval of the

scenarios. It is bad practice to design a scenario to

prove a point.

Models, Tools, and Data Requirements

The next step is to iteratively identify the methods and

tools (Chapter 8) and data (Chapter 9) required to

perform the assessment. One of the major challenges

of the assessment is to identify and gain access to

models, tools, and data that are appropriate for

exploring the issues of interest. The challenges come

in several dimensions:

• First, there is a limited set of tools that deal

effectively with the C2 dimension of the problem.

• Second, for even this limited set, it is often

difficult to access and modify the tools to reflect

the variables of interest.

• Third, there is often a paucity of useful data and

previously validated parameters.

As a result of the establishment of the MoM for the

study and the data that underpins those MoM and

models, a data collection and analysis plan should be

formulated. The sponsor should also be made aware
Chapter 2 41

of the difficulties associated with getting appropriate

data, cost of the data collection and analysis plan, and

the implications to the study if the required resources

are not set aside and budgeted to collect, collate,

process, and analyse the data.

Assess Study Risk

At this point in the process the assessment team

should take a look at the risks and uncertainties

(Chapter 10) associated with the decisions made with

respect to all of the tasks performed to date (e.g.,

consistency between the scenarios and the data,

models and availability of data, tools and analysis).

The sponsor must be made aware of these risks and

uncertainties and the strategies developed by the team

to mitigate them. If the risks associated with the

successful completion of the study are perceived as

being too high, the solution strategies should be

revisited and adjusted accordingly.

Peer Review

When the risk and uncertainties are perceived as

manageable, a peer review should be conducted. Peer

reviews are not used enough because they tend to be

time-consuming, seen as raising costs, or perceived

as threatening. In addition, research teams often want

to perfect their results and methods before revealing

them. The key is to build a peer review into the study

from the outset. The sponsor should be informed as

to the importance of the peer review. Peer reviews

should be built into the budget and reviewers invited

to look at the terms of reference, interim products, and

draft reports so that they are knowledgeable about
42 NATO Code of Best Practice for C2 Assessment

the effort and motivated to support the project. In later

stages of the study, the peer review can improve

presentation and also act as a mechanism to make

the results known to the professional communities.

Over time the assessment team should develop a

relationship with high quality peers and use them as a

pool of reviewers.

Peer reviews are not a luxury but a necessity.

Conduct of the Study

At this point we are in a position to execute the

assessment. The assessment team leader should

keep a study notebook or journal in which all

assumptions and decisions are documented so that

they are available for detailed discussion. Detailed

administrative records need to be kept regarding the

data, metadata, models, and analytical and

documentation tools. This will enable replication of

parts of the C2 analysis should the need arise. An

effort should be made to create data sets (not just the

project results) that will be available to other

researchers. The resources required to make such

data available to external bodies needs to be made

clear to the sponsor. The conduct of the study will not

usually be linear. It should be anticipated that multiple

iterations will be conducted and that lessons learned

from initial data collection and analysis efforts will

inform subsequent activities.

Study Products

The team must recognise the importance of presenting

the results of the assessment in a clear and
Chapter 2 43

comprehensive manner, taking into consideration the

style of the decisionmaker (Chapter 11). It is

particularly important that these results illuminate

rather than obscure the uncertainties associated with

the assessment.

Ethics

Professional operations research organisations, such

as the Military Operations Research Society (MORS),

have developed professional codes of ethics (Annex

C). The assessment team should also be guided by a

set of professional ethics and standards of conduct to

ensure the integrity and quality of the analysis. This

means that the assessment team should, inter alia:

• Maintain an open and honest dialogue with the

sponsor and other key players within the project

in order to minimise misunderstandings;

• Ensure that C2 assessments are organised and

conducted in a balanced fashion that adequately

identifies and represents all perspectives,

options, and relevant evidence;

• Inform the sponsor and other key players of:

• any constraints, assumptions, or

circumstances that threaten a balanced

assessment; and

• the risks and uncertainties associated with the

methods and data used in the project, and

• strategies to minimise the risks.
44 NATO Code of Best Practice for C2 Assessment

Chapter 2 Acronyms

ACE Resources – Allied Command Europe Resources

– (Part of SHAPE)

AF(N) – Regional Command (North)

AMF(L) – ACE Mobile Force (Land)

ARRC – ACE Rapid Reaction Corps.

C2 – Command and Control

CDE – Concept Development and Experimentation

CPX – Command Post Exercise

FTX – Field Training Exercise

HB(A) – UK Historical Branch (Army)

HQ – Headquarters

IRTF(L) – Immediate Reaction Task Force (Land)

JCSC – Joint Sub-Regional Command South Centre

JCSE – Joint Sub-Regional Command South East

JFCOM – US Joint Forces Command

KIBOWI – NL Army Exercise Driver

MND(C) – Multinational Division (Central)

MoM – Measures of Merit

MORS – Military Operations Research Society

NC3A – NATO C3 (Consultation, Command &

Control) Agency

NL MOD – Netherlands Ministry of Defence
Chapter 2 45

OOTW – Operations Other Than War

OA – Operational Analysis

OR – Operations Research

PRL – SHAPE Policy Requirements Land

SACLANT OA – Supreme Allied Command Atlantic

Operational Analysis Cell

SFS – Strike Force South

SHAPE – Supreme HQ Allied Powers Europe

SOW – Statements of Work

WPC – Warrior Preparation Center (Ramstein Germany)

Chapter 2 References

Research and Technology Organization (RTO) SAS-003

report on IFOR and SFOR data collection & analysis

(6 AC/323(SAS)TP/11). [Technical Report]. (1999).

Neuilly-Sur-Seine Cedex, France: North Atlantic

Treaty Organisation (NATO) RTO. Available from

http://www.rta.nato.int/RDP.asp?RDP=RTO-TR006

Starr, S. H. (n.d.). (2001). Lessons recorded from

applying the North Atlantic Treaty Organisation

(NATO) code of best practice (COBP) for C2

assessment to operations other than war

(OOTW). McLean, VA: MITRE Corporation.

Available from http://www.mitre.org/support/

papers/tech_papers_01/starr_nato/

starr_nato.pdf
46 NATO Code of Best Practice for C2 Assessment

Candan, U. and Lambert, N. J. (2002). Methods used

in the Analysis and Evaluation of the Immediate

Reaction Task Force (Land) Command and

Control Concept. TN 897. NATO C3 Agency,

PO Box 174, 2501 CD, The Hague, The

Netherlands.

Appendix 1 to Chapter 2: Participant

Mapping of the Evaluation of the

Immediate Reaction Task Force

(Land) C2 Concept—An Explanation

of Figure 2.1

Background

The Immediate Reaction Task Force (Land) (IRTF(L))

command and control concept was proposed in 1998

as a mechanism to modernise the ACE Mobile Force

(Land) (AMF(L)). The IRTF(L) concept is predicated

on the enlargement of AMF(L) from brigade size up to

division size with a single streamlined headquarters

and a chain of command using embedded mini-Task

Group HQ cells. This was evaluated between 1999

and 2001 as a test case for the NATO Concept

Evaluation and Experimentation (CDE) process using

a series of FTX, CPX, wargames, simulations and

historical analyses.

Assessment Team

In the case of the IRTF(L) study the assessment team

was led by NC3A OR Division, with contracted experts,

analytical and military support from KS Consultants
Chapter 2 47

and UK DERA. Free analytical support was also made

available at peak periods from US JFCOM and

SACLANT Operational Analysis (such as the exercises

of the experimentation campaign).

The sponsor was SHAPE Policy and Requirements Land—

who were tasked with the evaluation of the C2 Concept.

It was clear to the assessment team as to why the

sponsor wanted the study—a straightforward

evaluation of the military utility (to NATO) of the C2

Concept. However at the end of the study the results

were combined with other issues, and decisions were

made on the future of the unit under study. This was

something that was not foreseen by any of the

participants at the start of the project.

Decisionmakers or Problem Owners

The sponsor’s task was to provide advice up the chain

of command to SACEUR and ultimately the Military

Committee on the efficacy of the C2 Concept. Although

the HQ ACE Mobile Force (Land) was the subject of

the study it was also party to any decision regarding

its own future modernisation. It is commanded directly

by SACEUR via SHAPE.

SACLANT CDE, however, was not in the command

chain, but was seen as a decisionmaker within the

context of the study as it was interested in the

experience of the team in conducting the study as a

test case to illustrate the value of NATO Centred CDE

to the Alliance.
48 NATO Code of Best Practice for C2 Assessment

Stakeholders

The ACE Mobile Force (Land) was the main

stakeholder as it was the subject of the study. As a

decisionmaker, data and assumption provider and also

possible member of a future study team it was in a

very powerful position, and was approached and

treated with much respect by the assessment team.

After a shaky start (where neither side was sure of the

other’s intentions) a good working relationship was

established over the period of the project.

The Netherlands MOD—in the form of the Royal

Netherlands Army—was the provider of the Command

Information System (ISIS) used as the digitisation

vehicle for the evaluation of the concept. As such it

was directly affected by the exercise program used

for the experimentation and also the future of the

concept and AMF(L).

The Military Committee was also a stakeholder,

representing the Nations of NATO that contribute

troops and staff to the AMF(L), and these nations would

be directly affected by any decision on the concept.

ACE Resources at SHAPE were also a stakeholder as

they were required to sanction and organise any manning

changes proposed for the HQ—including the temporary

additional manning required for the evaluation.

Bill Payer

Monies were mostly provided from the slice of the

NC3A Scientific Program Of Work (SPOW) controlled

by SHAPE PRL. In the initial stages of the project

additional monies were also provided by SACLANT.
Chapter 2 49

Monies also had to be sought from the SHAPE

Exercise budget to pay for movement of the exercise

observers in order to attend the exercises.

Throughout the project the NC3A was the contractual

authority to let contracts on behalf of the bill payer.

Existing Study Teams

An extensive literature search was conducted for the

study—with the majority of recent references occurring

within the UK and US. Exploratory trips (organised

through US JFCOM) to US Battle labs and UK facilities

(through UK DERA) revealed the current state of

knowledge with respect to measuring C2 performance

in exercises and evaluating new C2 concepts. In

response to this the data collection methodology was

based initially on the Fort Leavenworth, US Army

Research Institute ACCES method.

Future Study Teams

It was identified at an early stage that there could be

future related projects. In particular those relating to

expeditionary and initial entry forces. The probable

NATO organizations that could be involved in such

studies were NC3A, SHAPE PRL, AMF(L),

Multinational Division (Central), Strike Force South and

the ACE Rapid Reaction Corps. Of course there would

probably be future study teams within the nations—

but these plans are not visible to NATO. Consequently,

as the assessment team was very likely to be involved

in such future work; all data was archived and routinely

written up and published.
50 NATO Code of Best Practice for C2 Assessment

Peer Reviewers

The assessment team were not able to arrange a

formal Peer Review of the solution strategy adopted.

This mechanism does not yet exist for NATO centred

studies. What was achieved the submission of the

problem to the SAS026 panel as an example for testing

the coverage of the revised COBP. This yielded some

practical advice and helped the assessment team

better understand the dynamics of the project.

Data Providers

Most of the data for the evaluation were derived from

NATO training exercises run by or for AMF(L),

MND(C), SFS and Joint Sub Regional Commands

South East and South Centre. In all cases relationships

had to be curried by the assessment team and sponsor

to allow access to the HQ and Exercise Control for

the exercise observers, and for background materiel.

In two cases national exercise training centres and

exercise drivers were hired by the assessment team

to support command post exercises (Warrior

Preparation Center, and the KIBOWI exercise driver).

Historical data for the study was also provided from

the UK Historical Branch (Army)—which was

approached via the contracted UK members of the

assessment team.

Assumption Providers

The assessment team was in the fortunate position to

actually be one of the assumption providers—through

NC3A’s and the sponsor’s involvement in the NATO

Defence Requirements Review. The owner of the C2
Chapter 2 51

1As developed independently by the SAS-026 panel in February

2001 in response to a presentation in the IRTF(L) project. In

fact, this was fairly close to what was available to the team.

2Linear programming, dynamic programming, queuing theory,

inventory control, network analysis with PERT, game theory and

simulation.

3Required if the C2 Assessment makes use of a series of linked

events such as seminars, wargames, command post exercises

(CPX), field training exercises (FTX), etc.

Concept however remained HQ AMF(L) itself, and

therefore remained the authority as to its conceptual

and physical implementation.

Data Collectors

In the case of the IRTF(L) study, data was largely

extracted through observation of HQ activities during

exercises and team-in-the-loop wargames. In all of

these exercises Subject Matter Experts (SME) were

used to observe functional and cross functional

activities in the HQ. Most of the military SMEs were

provided by Regional Command AF NORTH and its

subordinate commands across Allied Command

Europe (ACE). Additional data collectors were also

provided by the German University of the Federal

Armed Forces and US JFCOM. UK DERA provided

military analysts to lead some of the activities involved

in capturing the HQ processes.
53

CHAPTER 3

Problem

Formulation

First find out what the question is—

then find out what the real question is.

—Vince Roske

Definition of Problem Formulation

E

ffective problem formulation is fundamental to the

success of all analysis, but particularly in Command

and Control (C2) assessment because the problems

are often ill-defined and complex, involving many
54 NATO Code of Best Practice for C2 Assessment

dimensions and a rich context. Problem formulation

involves decomposition of the analytic problem into

appropriate dimensions such as structures, functions,

mission areas, command echelons, and C2 systems.

Problem formulation is an iterative process that evolves

over the course of the study. It is essential even for

small studies or where time is short—it will save time

later and help ensure quality.

The problem formulation phase should identify the context

of the study and aspects of the problem-related issues.

The context of the study includes:

• Geopolitical context that bounds the problem space;

• Political, social, historical, economic, geographic,

technological environments;

• Actors;

• Threats;

• Aim and objectives of the analysis, including the

decisions to be supported;

• Generic C2 issues;1

• Relevant previous studies; and

• Stakeholders and their organizational affiliation

(including both stakeholders of the problem and

stakeholders of the study).

The aspects of the problem include:

• Issues to be addressed;

• Assumptions;
Chapter 3 55

• High-level Measures of Merit (MoM);

• Independent variables (controllable and

uncontrollable);

• Constraints on the values of the variables

(domain and range);

• Time constraints on delivery of advice to the

decisionmaker; and

• Whether this is a single decision or (possibly one

of) a chain of decisions to be made over time.

The problem is not formulated until the assessment

team has addressed each aspect of the problem. In

simple terms, problem formulation can be seen as an

iterative process. First, the team must identify the

variables that bound the problem space. Then they

must determine which of these are outputs (dependent

variables) and which of these are inputs (independent

variables). The team proceeds by iterating to build an

understanding of how these relate to each other. It

should be viewed as a voyage of discovery. In most, if

not all, cases of C2 assessment, the knowledge

domain under study is in fact a system characterised

by rich interaction and feedback among all the factors

or variables of interest. The choice of dependent

variables results from a clear specification of the issues

and products needed to satisfy the terms of reference.

Independent and intervening variables are also chosen

based on the purpose of the analysis.

In the initial problem formulation iteration, it is critical

to begin with an understanding of the REAL problem

rather than a determination to apply readily available

tools, scenarios, and data.
56 NATO Code of Best Practice for C2 Assessment

Within the NATO context, a number of documents are

available or under development that may assist in

understanding the study context. They are listed at

the end of this chapter.

Principles of Problem Formulation

There is no universally acceptable approach to

problem formulation. However, best practices exist that

can be applied. The principles associated with problem

formulation are addressed in two categories: those that

are appropriate for all C2 assessments and those that

are appropriate for assessments of C2 for Operations

Other Than War (OOTW).

Principles Appropriate for C2 Assessments

Explicit problem formulation must precede construction

of concepts for analysis or method selection. This is

not a trivial exercise, especially in C2 assessments.

Proper resourcing of problem formulation activities will

improve the overall efficiency and quality of the study.

An understanding of the decisions to be supported by

the analysis and the viewpoints of the various

stakeholders (e.g., customers, users, and suppliers)

is essential to clarifying the study issues. This

understanding should be fed back to the stakeholders.

A careful review of previous and current work must be

carried out as a valuable source of ideas, information,

and insight. This review should also serve to identify

pitfalls and analytic challenges.

Problem formulation must not only provide problem

segments amenable to analysis, but also a clear and valid
Chapter 3 57

mechanism for meaningful synthesis to provide coherent

knowledge about the original, larger problem. The

formulated problem is thus an abstraction of the real problem

that can be defined in terms of dependent variables that

relate to this real problem and coherent settings for the

independent variables that can be interpreted in terms of

decisions and actions by the customer.

Problem formulation must be broad and iterative in

nature, accepting the minimum of a priori constraints

and using methods to encourage creative and multidisciplinary thinking, such as proposing a number of

hypotheses for the expression of the problem. It must

be recognised that change is inevitable in many

dimensions (e.g., understanding of the problem,

requirements, technologies, co-evolution of concepts

of operation, command concepts, organisation,

doctrine, systems). Thus the assessment process must

anticipate and accommodate this change.

Practical constraints such as data availability, study

resources (including time), and limitations of tools

should be treated as modifiers of the problem

formulation rather than initial drivers. Such constraint

may, in the end, drive the feasible solutions, but it is

important to recognise this as a compromise rather

than an ideal. Proper problem formulation takes

substantial time and effort!

It is important that problem formulation address risk

from multiple perspectives. In addition to sensitivity

analysis of the dependent variables, risk analysis

techniques should be used to directly explore options

to mitigate risk (Chapter 10).
58 NATO Code of Best Practice for C2 Assessment

C2 assessment often involves impacts on defense

business outside the context of a particular campaign

or operation. The study must address these impacts.

Principles Appropriate for OOTW C2

Assessments

Problem formulation must address the geopolitical

context of the problem and seek to identify the “broad”

C2 issues contained within the terms of reference for

the study. There are no universal societal “norms.”

Therefore, care must be taken in attempting to transfer

the experience in one OOTW to another.

OOTW C2 assessments often involve policy-related

impacts outside the context of a particular military

operation. Therefore, MoM hierarchies must contain

measures of policy effectiveness.

An historical perspective is critical to understanding

OOTW because social conflict and structures often

have roots far back in history. However, it must be

remembered that present-day social behaviour is not

driven by historical events themselves, but by presentday perceptions, processes, and prejudices which

have evolved from the past.

A key risk in complex OOTW studies is allowing the

problem formulation process to focus prematurely on

subsets of the problem because they are: a) interesting;

b) familiar; c) pre-judged to be critical; or d) explicitly

called out by the customer. This requires great discipline

by the study team, especially where the team’s previous

experience is biased in favour of particular parts of the

problem space. The assessment team needs access

to subject matter experts from a broad range of
Chapter 3 59

disciplines (e.g., social scientists, historians, and

regional experts in OOTW assessment).

Problem Formulation Process

During the early stages of problem formulation it is

important to quickly cover the whole problem and

produce an initial formulation (i.e., an explicit

expression of the problem) (See Figure 3.1). This

prevents premature narrowing of the assessment and

serves as an aid to shared situation awareness within

the study team.

Figure 3.1. The Formulated Problem

The process begins with the sponsor presenting the

assessment team with a problem to assess and an

articulation of broad constraints (e.g., schedule,

resources). Based on a preliminary assessment of the

problem, the team identifies the key issues to address.
60 NATO Code of Best Practice for C2 Assessment

This identification of key issues leads to a

characterisation of the context for the study (e.g.,

relevant geopolitical factors, identification of the key

actors and threats, identification of generic C2 issues,

review of prior studies). Based on the results of this

characterisation, the analysis team identifies what it

perceives as the real issues to address. It is vital for the

team to engage in a dialogue with the key sponsor and

stakeholders to get “buy in” for these issues. Once that

is achieved, the team must identify and characterise

the remaining elements of the problem formulation

phase. To facilitate that activity, the analysis team should

identify/create and apply selected problem formulation

tools and techniques (e.g., brainstorming, Delphi

analyses, directed graphics, influence diagrams). The

results of that activity will include a summary of the

assumptions, high-level MoM, independent variables

(both controllable and uncontrollable), and constraints

on the variables. Once it is coordinated with the sponsor

and stakeholders, the end product documents what is

to be done in the analysis. The next key activity will be

to develop a solution strategy that describes how the

study is to be done.

Bounding the Problem/Issues and

Assumptions

In dealing with fuzzy or uncertain boundaries, the

problem formulation process needs to explore and

understand the significance of each boundary before

making (or seeking from customers) assumptions

about it. This involves keeping an open mind, during

the early stages of problem formulation, about where

the boundaries lie and their dimensional nature. This
Chapter 3 61

is difficult because it makes the problem modelling

process more complicated. A call for hard specification

too early in the problem formulation process must be

avoided. In the end, of course, the problem must be

formulated in order to solve it, but formulation should

be an output from the first full iteration, not an early

input to it.

In formulating an OOTW problem, we are trying to

bound a complex system. This is partly a process of

understanding boundaries which exist in reality (e.g.,

mission statements, geographical areas and the timing

of a procurement process) and partly imposing artificial

boundaries in order to illuminate the structure of the

problem and constrain the scope of the analysis. To

avoid the trap of over-specification, boundaries

(especially self-imposed ones) should be kept porous,

allowing for cause and effect chains to flow through

the external environment of the portion of the complex

system that the boundaries define.

While clear definitions and hard conceptual boundaries

are ultimately necessary in order to create a

manageable problem space, care must be taken to

avoid coming to closure prematurely.

High-Level MoM

Identification of high-level MoM should start with ideal

measures of the desired benefits before considering

what can be practically generated by analysis (the latter

may force the use of surrogate MoM, but these must

be clearly related to the desired measures).

A structured analysis of potential benefits2 should be

carried out as a basis for constructing appropriate
62 NATO Code of Best Practice for C2 Assessment

MoM. Mapping techniques, such as cognitive and

causal mapping, are a good way to express the various

relationships within the problem space and to identify

‘chains’ of analysis (i.e. links among the independent

variables and between the independent and dependent

variables). These lead to resultant structure in terms

of independent and dependent variables, and hence

to high-level MoM.

Problem Formulation Tools

It is useful to identify, develop (if necessary), and apply

appropriate tools to support problem formulation.

Representative tools and techniques include:

techniques for supporting expert elicitation, influence

diagrams, causal maps, system dynamic models, and

agent-based models.

Problem Formulation is fundamentally a social process

of developing a shared understanding. People skills

such as the ability to facilitate a ‘brainstorming session’

or to elicit information and context, are thus important.

‘Throwaway models’ (which may be simple simulation

models, causal maps, system dynamic models, etc.)

may be developed as part of the process, and then

discarded as insight is gained.

Tools and approaches used for problem formulation

must be consistent with other tools and techniques

likely to be considered for the subsequent analysis in

order to produce a sensible ‘multi-methodology’

approach to the entire problem and its solution.
Chapter 3 63

Constraints on the Variables

The formulation of the problem is completed when the

constraints on either the independent or dependent

variables have been identified. Constraints on the

dependent variables represent “acceptable” thresholds

or limits. For example, one could place a constraint

on blue loss, time to accomplish a mission, collateral

damage, or some combination of factors. Constraints

on the independent variables represent either feasible

or acceptable limits on such factors as human

performance, C2 system performance, or even

supplies. They also could represent doctrinal or legal

processes that act as constraints.

The Next Step

The next step in the C2 assessment process is the

development of a solution strategy. It should be noted

that the team is not finished with problem formulation

at this point but is now ready to proceed to build a

solution strategy. As work progress on the

development of a solution strategy, it will also certainly

be necessary to revisit the specification of high-level

MoM and the constraints. This chapter concludes with

a discussion of the products of problem formulation.

Products of Problem Formulation

Figure 3.2 depicts the essential elements of the

formulated problem.
64 NATO Code of Best Practice for C2 Assessment

Figure 3.2. Problem Formulation

A checklist can be used to ensure that all the aspects

described in the definition have been covered. These include:

• Precise statements of the question being researched;

• A list of independent variables;

• A list of high-level MoM; and

• A list of assumptions and constraints.

Diagrams

Typically, the problem formulation phase should also

produce a number of diagrams such as influence maps

which summarise the key issues and interactions.
Chapter 3 65

Data Glossary

The problem formulation phase must begin to create

a glossary of key data elements, metadata,

information, and terms.

Chapter 3 Acronyms

C2 – Command and Control

MoM – Measures of Merit

OOTW – Operations Other Than War

TTP – Tactics, Techniques, and Procedures

Chapter 3 References

Checkland, P. (1999). Systems thinking, systems practice.

Chichester, UK: John Wiley and Sons Ltd.

De Maio, J. (Ed.). (2000). The challenges of

complementarily [Report on the fourth workshop

on protection for human rights and humanitarian

organisations]. Geneva, Switzerland: International

Committee of the Red Cross.

Defence Science and Technology Laboratory (DSTL).

(2000). A guide to best practice in C2 assessment:

Version 1. Farnborough, United Kingdom.

Fisher, R. A. (1951). Design of experiments (6th ed.).

Ontario, Canada: Oliver and Boyd, Ltd.

Flood, R. L., & Jackson, M. C. (1991). Creative problem

solving: Total systems intervention (ISBN

0471930520). Chichester, UK: John Wiley and

Sons Ltd.
66 NATO Code of Best Practice for C2 Assessment

Holt, J., & Pickburn, G. (2001). OA techniques for the

future [Defence Evaluation and Research

Agency (DERA) Unclassified Report].

Farnborough, United Kingdom: Defence

Science and Technology Laboratory (DSTL).

Keppel, G. (1973). Design and analysis—A

researcher’s handbook (ISBN 0-13-200030-X).

Englewood Cliffs, NJ: Prentice-Hall.

Midgley, G. (1989). Critical systems and the problem of

pluralism. Cybernetics and Systems, 20, 219-231.

Moffat, J., & Dodd, L. (2000). Bayesian decisionmaking

and catastrophes. Defence Science and

Technology Laboratory (DSTL) [Unpublished

Report] Available from ksglasgow@dstl.gov.uk.

Moffat, J., & Witty, S. (2000). Changes of phase in the

command of a military unit. Defence Science

and Technology Laboratory (DSTL)

[Unpublished Report] Available from

ksglasgow@dstl.gov.uk.

NATO Open Systems Working Group (2000). NATO

C3 Technical Architecture. AC/322(SC/5)WP/

31, Brussels

NATO Consultation, Command and Control Board

(2000). NATO C3 System Architecture

Framework. AC/322-WP/0125, Brussels

NATO Consultation, Command and Control Board

(2000). NATO Interoperability Management

Plan (NIMP). AC/259-D/1247(rev), Brussels
Chapter 3 67

NATO Study Group on Modelling and Simulation

(1998). NATO Modelling and Simulation Master

Plan, Version 1.0. AC/323(SGMS)D/2, Brussels

Pidd, M. (1996). Tools for thinking—Modelling in

management science. Chichester, UK: John

Wiley and Sons Ltd.

Prins, G. (1998). Strategy, force planning and

diplomatic/military operations. London: The

Royal Institute of International Affairs.

Rosenhead, J. & Mingers, J. (Ed.). (2001). Rational

analysis for a problematic world revisited:

problem structuring methods for complexity,

uncertainty and conflict (2nd ed.). Chichester,

UK: John Wiley and Sons Ltd.

Windass, S. (2001). Just war and genocide—The

Ottawa Papers. Woodstock, United Kingdom:

Foundation for International Security.

Witty, S., & Moffat, J. (2000). Catastrophe theory and

military decisionmaking. Defence Science and

Technology Laboratory (DSTL) [Unpublished

Report] Available from ksglasgow@dstl.gov.uk.

The following additional documents are under development:

NATO C3 System Baseline Architecture

NATO C3 System Overarching Architecture

1Broad C2 issues include key systems, doctrine, Tactics,

Techniques, and Procedures (TTP), organisational structures,

and key assumptions (e.g., system performance parameters).

2The structured analysis of benefits is a logical process that

seeks causally to map lower level MoM that can be related to

investments or other actions to higher level MoM that can be

valued directly by decisionmakers.
69

CHAPTER 4

Solution Strategies

Operations research is a scientific method.

Executives have often in the past used some

of the techniques...to help themselves arrive

at decisions...But the term “scientific methods”

implies more than sporadic application and

occasional use of a certain methodology; it

implies recognized and organized activity

amenable to application to a variety of

problems and capable of being taught.

—Philip M. Morse and George E Kimball,

Methods of Operations Research
70 NATO Code of Best Practice for C2 Assessment

The Study Plan

A

 conscientious effort is required to create and follow

 a study plan that guides data collection and

analyses and prepares for the use of the insights and

data to be collected to contribute to a solution to the

problem at hand. The study plan consists of two interrelated parts—the formulated problem (the What) and

the solution strategy (the How). The output of the initial

problem formulation provides the assessment team

with an operating definition of what needs to be done.

The output of the solution strategy phase provides the

team with an operating definition of how this will be

accomplished. As the project unfolds, there will usually

be a significant amount of iteration that modifies both

the problem formulation and the solution strategy.

Without a study plan, it is unlikely that needed efforts

will be properly scoped, prioritised, scheduled, and

resourced. Even if the way ahead seems clear, the

articulation of a formal Solution Strategy is necessary.

The objective of this phase of the study is to develop

a feasible approach to go from the specification of what

is to be done to how it is to be done. This involves

developing an approach that will result in the team’s

ability to collect the data necessary to determine the

values of the Measures of Merit (MoM) for specified

values of independent variables. The characteristics

of data collection instruments and analysis tools and

techniques will determine the resources required, the

time needed, and the risks inherent in the solution

approach. When compared to the study constraints

and the problem formulation, it will be determined

whether the solution approach is both feasible and

satisfies the requirements of the problem formulation
Chapter 4 71

(e.g., measures the right things). Figure 4.1 depicts

what is involved in moving from a problem formulation

to a solution strategy.

Figure 4.1. From Problem Formulation to Solution Strategy

Key Definitions

Solution Strategy

A solution strategy consists of the specification of a

set of sequential and parallel analytical steps, often

involving several methodologies and tools. The

solution strategy is designed to begin with what is

known, and by execution of the specified steps, leads

to what one desires to know—an illumination of the

issues. The strategy can be:

• simple—calculate mortgage payments by finding

the input values for the payment equation and

then evaluating the result;
72 NATO Code of Best Practice for C2 Assessment

• moderately complicated—define input variables,

output variables and precision requirements,

create a designed experiment and run the

experiment with appropriate measurements

including the regression analysis; or

• extremely complex—identify the relevant

variables and systems of variables, specify how

they might be measured, hypothesize how they

are related, and design research strategies that

allow for complex adaptive systems or other

“messy” structures or processes.

The solution strategy must take the outputs from

problem formulation, refine and operationalise them,

and develop a plan to collect and analyse appropriate

data (including the development and/or selection of

models, the design of collection instruments, and the

selection of analysis tools) to understand the

relationships among the relevant variables associated

with MoM, the scenarios, and human and

organisational factors. (Figure 4.2)

Measures of Merit (MoM)

MoM are a set of variables that focus the assessment

on the issues of interest. In most analyses, these are

the dependent variables. In many cases there are

significant inter-relationships among the MoM.

Human Factors

Human factors consists of a set of variables that

characterise concepts including beliefs, cultural norms,

stress, fatigue, fear, arousal, morale, intelligence, and
Chapter 4 73

level of experience. In Command and Control (C2)

assessments these are typically independent or

intervening variables.

Organisational Factors

Organisational factors consist of a set of variables that

characterise organisations, such as cohesion,

command structure, explicit and tacit relationships,

information flows, and organisational cultures. These

are also typically independent of intervening variables

in C2 analyses.

Scenarios

Scenarios consist of the evolution in time of several

elements: a context (e.g., a characterisation of a

geopolitical situation), the participants (e.g., intentions,

capabilities of blue, red, others), and the environment

(e.g., natural, weather and manmade, mines). In C2

assessments, the purpose of scenarios is to ensure

that the analysis is performed within the appropriate

range of opportunities to observe the relevant variables

and their interrelationships.

Model

A model is a physical, analogue, or symbolic

representation of relevant aspects of reality for a

purpose. It is an abstraction of reality. A model

emphasises particular aspects (a subset) of reality.

The assessment model of primary interest is the

assessment team’s model (conceptual or in some

analytic manifestation) of the C2 problem including the
74 NATO Code of Best Practice for C2 Assessment

variables of interest, their hypothesised relationships,

and any prior assumption about their values and

linkages. The assessment team may also employ or

develop other models or simulations in order to perform

analysis or explore risks and uncertainties. Some teams

will employ more than one of these analytic tools.

Tool

A tool facilitates the exploration of relationships among

model variables and/or develops “solutions” (e.g.,

maximise value subject to constraints). A tool may be

as simple as a checklist or an algorithm, or it may be

an extremely large simulation. A simulation is the

instantiation of a model that serves to facilitate the

exploration of the relationships among the variables –

it generates data for analysis and generally

emphasises the passage of time. Models and

simulations are frequently subdivided into categories

of constructive, virtual, and live.

Occasionally, the distinctions among a tool, model,

and data are subtle. For example, in a linear program

the model is the set of formulas that specify the

objective function and the constraints. The tool is the

simplex method (or similar algorithmic solution

method). The data is an instantiation of the formulas

(provides values for the coefficients and constants).

In the case of a simulation, the simulation environment

and simulation engine are the tools, the coded

simulation embodies the model, and the input values

to the simulation comprise the data. Often, the

simulation code and the data, together, are required

for a complete definition of the model.
Chapter 4 75

Data

Data are the values associated with the variables. Data

may be ratio, interval, ordinal, or nominal in scale. Data

may originate from empirical observation; be derived from

models, simulations, or analyses; be established from

subject matter experts; or be established by assumption.

Developing a Solution Strategy

The development of a solution strategy is an iterative

process that strikes an artful balance between what

the team would like to do and what, given the state of

the art, the available data, tools, schedule, and

resources, is possible to do.

Prerequisites

The solution strategy should not be designed before

the problem formulation process is substantially

complete (Figure 4.1) and the problem formulation

products specified in Chapter 3 are available to the

team. This means that:

• The “real” question to be answered is known;

• The assumptions have been articulated;

• The high level MoM have been identified;

• The independent variables have been identified; and

• The constraints associated with the variables

have been identified.
76 NATO Code of Best Practice for C2 Assessment

However, the assessment team should always

remember the inherently iterative nature of the

process. Adjustments may prove necessary in the

problem formulation as the solution strategy matures.

Steps in Developing a Solution Strategy

As an initial step, the team should elaborate on the

MoM to specify the detailed MoM that are to be

evaluated. This is sometimes referred to as developing

operational definitions for the MoM—definitions that

specify the metric to be used, the instrument, and the

context in which the measurement is to take place.

Often the value of a particular MoM can not easily be

observed or measured and one or more surrogate

measures are used in its place. In any event, the

development of the set of MoM to be used in the study

anchors the process that will eventually lead to a

solution strategy.

This process (Figure 4.2) revolves around the

conceptual model that the assessment team builds,

and is at the heart of that process. It is best practise to

make this model explicit and have it serve as the

common picture that develops a high quality of shared

understanding among the team, sponsors,

stakeholders, and other key study participants. The

initial conceptual model consists simply of the MoM, a

first cut of the hypothesised relationships among them,

assumptions about variables and their relationships,

and constraints. Later iterations include additional

independent variables that are known or assumed to

affect the values of the MoM or the nature of the

relationship among them, increasingly detailed

specifications of relationships, and specific values or

ranges for the independent variables.
Chapter 4 77

Figure 4.2. Solution Strategy

The identification of human and organisational factors

that impact model variables and relationships serves

to flesh out the basic conceptual model generated in

the problem formulation phase.

Scenarios then need to be derived to provide

opportunities in an appropriate context for data

collection and exploration of the variables and

relationships contained in the conceptual model. The

data the study requires are, in large measure, a

derivative of the scenarios utilized and the design of

the assessment.

The design of an assessment also requires

specification of methods and tools and how they will

be employed. Methods and tools are required to

explore the relationships among the independent

variables and between the independent and dependent

variables. Complex solution strategies may be

necessary. In these cases, multiple analyses will be
78 NATO Code of Best Practice for C2 Assessment

implied. The problem must be divided into parts, each

part requiring analysis with its own set of tools.

Frequently the tools that are available do not provide

interfaces from one part of the analysis to the next.

Taken together, the detailed specification of the MoM,

the development of a conceptual model including the

relevant human and organisational factors, the

specification of a set of scenarios, and a data collection

and analysis plan (that consists of the methods and

tools to be used) constitute a solution strategy.

The solution strategy developed needs to be tested to

see if it can be expected to address the issues at hand,

within schedule and resource constraints, and with

accepted levels of uncertainty and risk. However,

uncertainty and risks are being continually assessed

throughout the process of developing a solution

strategy. The team should also consider the form of

study output and its relevance to the decisionmaker.

Iteration of these ideas with the stakeholders

throughout the study helps to avoid surprises and to

ensure that the basic assumptions underlying the study

have not changed.

Iterating the Study Plan

Figure 4.3 depicts the iterative nature of the process

involved in developing the Overall Study Plan, linking

problem formulation and solution strategies together

with the inputs from study sponsors and stakeholders.
Chapter 4 79

Figure 4.3. Overall Study Plan

A first order feedback loop is shown between problem

formulation and solution strategy, with both processes

having iterative internal processes. An analysis of the

study of risk and uncertainty provides the control

mechanism that drives the iterative process to an

acceptable result.

Study Management Plan

The team should also create and maintain a Study

Management Plan (SMP) to guide the direction,

management and co-ordination of the project team. The

SMP should include a detailed, time-phased execution

plan for the study and a Work Breakdown Structure

(WBS).1 The SMP should show the requirements for all

the team products and their delivery dates thereby

creating delivery milestones for the execution of the

study. It should show the planned dates for all scheduled
80 NATO Code of Best Practice for C2 Assessment

meetings including progress meetings and technical

interchange meetings. The plan should also include a

time-phased manning plan identifying the types,

quantities, and period of performance for all members

of the study team. The SMP should include details of

the controls that will be applied to supervise any

contractor performance. The team should maintain a

current version of the SMP during the study period of

performance. The SMP would typically include the

associated supporting plans:

• Study glossary;

• Analysis plan;

• Tool deployment and modelling and simulation plan;

• Data collection/engineering plan;

• Configuration management plan;

• Explicit consideration of risk and uncertainty;

• Quality assurance plan;

• Security plan;

• Review plan; and

• Plan of deliverables.

The elements of an ideal SMP are defined and

discussed below.

Study Glossary

The assessment team should create and maintain a

study glossary comprising all relevant definitions

needed in the study. It should aim to create a general
Chapter 4 81

study glossary that is improved by every study that

uses this glossary. As a starting point, the NATO AAP6, “NATO Glossary of Terms and Definitions,” or the

Joint Publication 1-02, “DOD Dictionary of Military and

Associated Terms,” should be used.

Analysis Plan

The assessment team should also create and maintain

an analysis plan for the study. The analysis plan should

describe the analyses in detail. This description should

include the analysis methodology, the tools to be used

for analysis, the input data requirements, the Essential

Elements of Analysis (EEA), the MoM to be used to

evaluate the results, and any analysis assumptions.

Tool Deployment and Modelling and

Simulation Plan

The team may create and maintain a tool deployment

and modelling and simulation plan covering the needs

of each task for numerical simulations or other applied

means of operations analysis (OA). It should describe

the use of tools and models and simulations in the

feasibility study (FS). The plan should include a

description of each tool to be used, a list of the key

assumptions and caveats for each tool, an analysis of

the suitability of each tool in addressing the functionality

and performance issues in the FS, the source of input

data for each tool, the available output from each tool,

and should detail any changes to these tools that are

intended. The plan should indicate how data

traceability from one tool to another will be maintained.
82 NATO Code of Best Practice for C2 Assessment

The modelling and simulation plan may be included

as part of the analysis plan in simple studies.

Data Collection/Engineering Plan

The team should create and maintain a Data

Collection/Engineering Plan (DCEP) which covers data

and metadata necessary to describe:

• The scenario;

• The essential elements of analysis in the analysis plan;

• The MoM to be used to evaluate the result (also

in the analysis plan); and

• The input and output parameters of tools to be

used within the study.

The DCEP describes and documents who owns the

data, where the data can be found (including open

sources like the Web), necessary methodologies and

procedures to prepare the data, and assumptions and

constraints connected to generated data, etc. The data

definitions used in the DCEP have to be harmonised

with the study glossary.

Configuration Management Plan

The team should create and maintain a configuration

management plan. The plan should ensure

identification, traceability and control of the descriptions

of the system elements, interfaces and architectures

considered in the FS, as well as associated

documentation. The plan should show how the study

database must be controlled and updated. The plan

should follow NATO STANAG 4159, “NATO Materiel
Chapter 4 83

Configuration Management Policy and Procedures for

Multi-National Joint Projects” (1992), as a guideline.

The plan should also ensure that the description of

the system configuration being simulated or analysed

can be precisely identified, as well as any system or

technology improvements considered, with respect to

an identified baseline configuration.

Study Risk Register

The team should also identify and assess the technical

and schedule risks concerned with the successful

completion of the FS. The list of risks identified should

be maintained in a study risk register, which shows

the probability of occurrence of each risk and its impact

on the FS. The register should include the risk

mitigation activity for each risk and the expected

improvements to time and performance. The study risk

register should be regularly maintained, should be

available to the assessment team by arrangement, and

should be presented at each progress meeting. The

Generic Risk Register (GRR) developed during the

SAS-026 efforts (Chapter 10) is recommended.

Quality Assurance Plan

The team should create and maintain a quality

assurance plan. The plan should declare all relevant

quality standards and procedures that are to be applied

in the course of the study, and should describe the

quality organisation to be used, including the principal

quality officers and their lines of authority. The policy

and requirements for quality assurance in NATO are

given in the following documents, which should be used

as guidelines:
84 NATO Code of Best Practice for C2 Assessment

• STANAG 4107, “Mutual Acceptance of

Government Quality Assurance and Usage of

the Allied Quality Assurance Publications

(AQAPs),” 6th Edition (March 1998).

• AQAP 100, “General Guidance on NATO Quality

Assurance,” 2nd Edition (March 1995) or

equivalent/comparable national standards.

The Security Plan

The team should create and maintain a security plan.

This plan should contain the approach to the utilisation,

storage, publication, dissemination, and control of

classified and unclassified materials.

Review Plan

The team should create and maintain a review plan.

For every critical phase of the study, preferably marked

by respective milestones, reviews of the study have

to be planned and executed. Participants should go

beyond the members of the study team to include peer

reviews. The review results should be documented.

Plan of Deliverables

The team should also create and maintain a plan of

deliverables for each phase of the study. This includes

what is to be delivered, when it is to be delivered, to

whom, in what form and format, and how many copies.

Chapter 4 Acronyms

C2 – Command and Control
Chapter 4 85

DCEP – Data Collection/Engineering Plan

EEA – Essential Elements of Analysis

FS – Feasibility Study

GRR – Generic Risk Register

MoM – Measures of Merit

OA – Operations Analysis

SMP – Study Management Plan

WBS – Work Breakdown Structure

Chapter 4 References

With most textbooks, the bulk of the text is concerned

with teaching about the available tools, not with how

to use them as an ensemble. Each of the following

references provides some information concerning

solution strategies.

Cleman, R. T. (1990). Making hard decisions: An

introduction to decision analysis. Pacific Grove,

CA: Duxbury Press.

Gass, S.I., & Harris, C.M. (Eds.), Encyclopedia of

operations research and management science

(2nd ed.). New York: Kluwer Academic.

Hartley, D. S., III. (1996). Operations other than war:

Requirements for analysis tools research report

(K/DSRD-2098). Oak Ridge, TN: Lockheed

Martin Energy Systems. Available from http://

www.msosa.dmso.mil/ootw.html
86 NATO Code of Best Practice for C2 Assessment

Hartley, D. S., III. (2001). Military operations other than

war. In S. I. Gass & C. M. Harris (Eds.),

Encyclopedia of operations research and

management science (2nd ed.). New York:

Kluwer Academic.

Hillier, F. S., & Lieberman, G. J. (1990). Introduction

to operations research (5th ed.). New York:

McGraw-Hill.

Hoeber, F. P. (Ed.). (1981). Military applications of

modelling: Selected case studies. Alexandria,

VA: Military Operations Research Society.

Hughes, W. P., Jr. (Ed.). (1989). Military modelling

(2nd ed.). Alexandria, VA: Military Operations

Research Society.

Jaiswal, N. K. (1997). Military operations research:

Quantitative decisionmaking. New York: Kluwer

Academic.

Olae, N. G., Roy, J., & Wetter, M. (2000). Performance

drivers—A practical guide to using the balanced

scorecard (ISBN 0-471-49542-5). Chichester,

UK: John Wiley and Sons Ltd.

Quade, E. S. (Ed). (2000). Analysis for military

decisions. Alexandria, VA: Military Operations

Research Society.

Rosenhead, J. (2001). Problem structuring methods.

In S. I. Gass & C. M. Harris (Eds.), Encyclopedia

of operations research and management

science (2nd ed.). New York: Kluwer Academic.
Chapter 4 87

Wagner, H. M. (1975). Principles of management

science (2nd ed.). Upper Saddle River, New

Jersey: Prentice-Hall.

1A WBS is a decomposition of the effort into its constituent parts

(or tasks) and the assignment of assets to the component tasks.

Assets may be people’s time, facilities, or other elements

required to complete the task. WBS are common in construction

and engineering projects.
89

CHAPTER 5

Measures of Merit

It’s best to know what you are looking

for, before you look for it.

—Winnie the Pooh, from A. A. Milne

I

n order to understand the impact of Command and

Control (C2), it is necessary not only to analyse and

measure the effect of C2 on military operations, but

also the effects on the components of the constituent

systems. No single measure or methodology exists

that satisfactorily assesses the overall effectiveness

of C2. Therefore, a multi-faceted and sometimes multiphased approach is necessary. The benefits of C2

should be evaluated through their impact on the
90 NATO Code of Best Practice for C2 Assessment

fulfilment of the military and policy objectives, and the

impact of C2 should be measured in terms of specific

qualities that are relevant to these objectives. A set of

scenarios provides the contexts in which Measures of

Merit (MoM) are determined.

MoM Challenges

During the last two decades, many new automated

C2 systems have been developed and fielded.

However, the determination of both the performance

and the effectiveness of these systems has proven to

be a complex problem. Recognising this, the Military

Operations Research Society (MORS) has sponsored

several workshops on MoM since 1985. The

workshops have led to the development of an analysis

framework, Modular Command and Control Evaluation

Structure (MCES), for the measurement of

performance and effectiveness within a conceptual

model for C2. Based on the MORS workshops, the

US Army’s Training & Doctrine Command Analysis

Center (TRAC) developed the C2 Measures of

Effectiveness (MoE) Handbook in 1990. This

document and the measurement tools developed for

the Headquarters Effectiveness Assessment System

(HEAT) and the Army Command and Control

Evaluation System (ACCES) represented the then

established best practices.

The AC/243 Panel 7 Ad Hoc Working Group (AHWG)

on the Impact of Command, Control, Communications

& Intelligence (C3I) on the Battlefield acknowledged that

the specification of measures of effectiveness is difficult.

The 1992 final report recommended that a hierarchy of

measures be established as an important step in
Chapter 5 91

understanding overall system effectiveness, and that

systems be analysed at different levels of detail. The

types of measures were grouped relating to C2 system

performance, force/commander effectiveness, and

battle outcome. To quote from the final report,

“Measures...are often inadequate and too model or

scenario specific. In addition, they have often been

generated in ad hoc ways, suggesting a lack of formal

analysis in their development.” Since then, RSG-19,

SAS-002 and SAS-026 have canvassed the field, and

have brought together the best ideas and practices in

order to support MoM development and applications

within C2 assessment.1 This version of the Code of Best

Practise (COBP) extends this thinking and includes the

Operations Other Than War (OOTW) domain.

Definitions

It has been recognised that a single definition for

measures of performance (MoP) and effectiveness

(MoE) does not exist. MoM is recommended as a

generic term to encompass different classes of

measures. The measures are defined in hierarchical

levels related to each other, each in terms of its own

boundary. From the conceptual viewpoint, it is

important to keep in mind the level of analysis and the

context in which the measurements are made.

Within the MCES framework, MORS has developed a

four-level hierarchy of measures from high-level force

effectiveness to low-level rudimentary measures of

physical entities, which were adopted by RSG-19. In

the context of OOTW, a fifth level is added, Measures

of Policy Effectiveness (MoPE), to characterise the

contribution of military actions to broader policy and
92 NATO Code of Best Practice for C2 Assessment

societal outcomes. For OOTW, political factors are

paramount and considerations such as media

coverage, local regional stability, and sustainment of

community societal standards must be taken into

account. Military missions may not directly achieve

policy objectives, although they often strive to provide

an environment more conducive to these objectives.

However, MoE of military tasks should quantify

performance against military missions, not the overall

political aspirations.

The Code of Best Practice has adopted the following

five levels of MoM:

• Measures of Policy Effectiveness (MoPE), which

focus on policy and societal outcomes;

• Measures of Force Effectiveness (MoFE), which

focus on how a force performs its mission or the

degree to which it meets its objectives;

• Measures of C2 Effectiveness (MoCE), which

focus on the impact of C2 systems within the

operational context;

• Measures of Performance (MoP), which focus on

internal system structure, characteristics and

behaviour; and

• Dimensional Parameters (DP), which focus on

the properties or characteristics inherent in the

physical C2 systems.

Figure 5.1 emphasises the diminishing impact of a

particular MoM as the circle widens.
Chapter 5 93

Figure 5.1. Relationships of Measures of Merit

Measurement Objectives

The important issues raised by decision-makers

require a sense of the degree to which C2 performance

may improve force and policy effectiveness. Therefore,

C2 assessments are often called upon to provide

convincing evidence of the expected improvements

in mission effectiveness that can be attributed to

improved C2. The ideal approach may be to define a

single measure that reflects the military and political

objectives of the missions under consideration.

However, the determination of such a measure is

generally not feasible, although not necessarily
94 NATO Code of Best Practice for C2 Assessment

impossible for particular classes. In most

decisionmaking problems, it is necessary to define

several measures that together provide the necessary

insights. A major reason for this is that a single

measure may not provide sufficient scope and/or detail

to analyse the impact of specific C2 elements,

particularly second and third order effects or

unintended consequences. Many analyses are

conducted precisely in order to enable trade-off

between important equities which can only be seen if

a set of MoM is generated for analysis. The set of

MoM selected must be comprehensive to ensure that

all factors are considered.

MoM are used to compare different options on equal

terms, and serve a wide range of purposes, including:

• Establishing a standard or expectation of

performance (for new requirements);

• Establishing the bounds of performance of a system

as well as the effects of imposed constraints;

• Comparing and selecting alternative systems that

may be very dissimilar but are designed to

achieve a similar purpose;

• Assessing the utilisation of a system in new or

unexpected application domains or missions;

• Identifying potential weaknesses in specific areas

of an organisation or system (areas of high error

potential or high user workload);

• Analysing the impacts of organisational changes;

• Analysing training effectiveness;
Chapter 5 95

• Determining the most cost-effective approaches

to achieve desired objectives;

• Comparing a replacement system, or

components of a system, against predecessors

or competitors;

• Assisting in generating and validating

requirements and deriving specific C2

requirements from broad statements of objectives;

• Evaluating the effectiveness of human

decisionmaking in the C2 cycle;

• Determining the degree of mission success; and

• Determining the return to normality in OOTW.

Characteristics of MoM

The development of an operational definition of the

measure, the development of instruments, the

application of the instruments to collect of appropriate

data, and the establishment of relationships among a

group of MoM vary in difficulty, effort, cost, precision,

generalizability and other characteristics. These

differences are often related to the type(s) of MoM

involved and the domain in which the measurement

needs to take place.

C2 Assessments involve measurement of variables

that exist in the physical, information, and cognitive

domains.2 In general, the development of operational

definitions, instrumentation and data collection for

variables to be measured in the physical and

information domains are more straightforward and

require less effort and expense than dealing with
96 NATO Code of Best Practice for C2 Assessment

variables that are measured in the cognitive domain.

Furthermore, the former can be measured more

“precisely,” are easier to comprehend, and are less

subject to interpretation than the latter.

In general, DP, MoP related to systems, and MoFE

tend to be measured in the physical domain (e.g.,

bandwidth, computing capacities, time to accomplish

a task, force exchange ratios) while MoP related to

measures of C2 effectiveness (quality of awareness,

shared awareness, and trust) and MoPF (will of an

adversary, public opinion) tend to be measured in the

cognitive domain.

As one goes up the hierarchy of MoM, the measures

tend to become more context, task, or mission specific.

For example, the performance characteristics of systems

(DP) apply to systems in general but MoFE are usually

limited to a set of tasks or missions. MoFE for combat

are very different from MoFE for various OOTW. If done

well measures of C2 effectiveness will be scenario

independent so one can compare C2 effectiveness

across a range of missions and circumstances.

Except for DP any of the MoM can be either an

independent or dependant variable in a given

assessment with any of the independent variables

being either “controllable” or not. The difficulty in

establishing relationships among the MoM varies as

a function of the level of the independent variable.

The Assessment Team should recognize and plan for

the difficulties associated with using various MoM and

should avoid substituting easier to deal with but less

relevant MoM. It is always better to try to measure

(estimate, approximate) MoM that reflect first order
Chapter 5 97

effects than to precisely measure MoM that do not

adequately reflect key aspects of the problem.

Measuring MoM

This section outlines measurement theory concepts

that apply to ensure that the right measuring

instruments are selected and applied correctly. By

definition, measurement is the assignment of values

to observation units that express properties of the units.

Four levels of measures relate numbers to properties

of interest: nominal (e.g., artillery vs. infantry), rank or

ordinal (e.g., worst to best), relative or interval value

(e.g., change in temperature), and absolute value or

ratio (e.g., 2 kilobits per second). Analysts must ensure

that the specific MoM adopted are at the appropriate

levels of measurement.

The key properties for quality assurance are reliability

and validity. Other significant properties include

practical issues, such as the effort required to collect

appropriate data and the convenience of measurement

(e.g., whether the collection process itself interferes

with the conduct of an exercise or experiment). Ideally,

measurements should be easy to capture and easy to

apply. There are clearly trade-offs to be made between

MoM that may closely track the property of interest

and that are costly and/or difficult to measure and those

that are less strongly related to the property of interest

but that are easier to measure. The effort required for

collection bears no direct relationship with validity, but

reliability may be related to cost. Reliable

measurements require repeated observations and

appropriate sample sizes. Reliability represents

accuracy and consistency. A cost-effective
98 NATO Code of Best Practice for C2 Assessment

measurement plan provides enough data for useful

and definitive conclusions. However, cost and/or

convenience of measurement may be an overriding

factor in system evaluation.

Failure to take validity and reliability into account raises

the risk of generating false conclusions. Validity and

reliability are not absolutes, but matters of degree.

Validity is the degree to which a measure characterises

the attribute of interest and only that attribute. Complex

concepts often require multiple measures to provide

valid information. In order to make a valid link between

the performance of a system as a whole against

performance of its components, the measures must

correspond to critical tasks. Reliability represents

accuracy and repeatability. A measure may be reliable

but not valid, or it may be valid but not reliable.

Validity

The properties of validity may be categorised into five

types: internal, construct, statistical, external, and expert.

• Internal validity is defined as the establishment of

causal relationships between variables of

interest. This is necessary to accept a

hypothesis that a given measure is responsible

for a specific effect on another measure;

• Construct (also referred to as content) validity

means that the target objects, and only the target

objects, are measured;

• Statistical validity implies that sufficient sensitivity

is involved in order to determine relationships

between independent and dependent variables.
Chapter 5 99

Statistical tests control two types of errors in

measurement. Type I, or alpha, is the probability

of rejecting a claimed hypothesis that is true,

Type II, or beta, is the probability of accepting an

hypothesis that is not true;

• External validity implies that the results may be

extended to other populations or environments; and

• Expert validity refers to the degree to which

measures are accepted by those knowledgeable

in the field.

A MoM should meet the validity-related criteria outlined

in Table 5.1.

Table 5.1. Validity Criteria of Measures

Reliability

Reliability involves the expectation of errors associated

with measurements. It is defined as the accuracy of a

measurement, as reflected in the variance of repeated

measurements of the same phenomenon. The key

principles of reliability are consistency (repeatability)

and accuracy. The variance associated with

measurement must be known or estimated to interpret

results and to discriminate between real effects and

measurement effects.
100 NATO Code of Best Practice for C2 Assessment

The reliability-related criteria that MoM should meet

are outlined in Table 5.2. Additional criteria that have

proven usual in the past are outlined in Table 5.3.

Table 5.2. Reliability Criteria of Measures

Table 5.3. Other Useful Criteria

Practical MoM Issues

The assessment of C2 requires the application of a

framework to yield values for appropriate MoM.

Analyses of C2 systems and processes often reveal a

complex hierarchical composition. A structured

resolution/functional decomposition approach may be

related to the organisational structure to yield

performance measures for the organisation as a whole,

individual components within the organisation, and

specific tasks within the organisational cells.

If the analyst assumes that C2 effectiveness is positively

correlated with overall military unit effectiveness, MoM

could be obtained by addressing the outcomes or

products of such unit activities. Goal-level evaluation

attempts to define the ability of the specific military

formation to make the system state match the goal

(directive) provided by the superior headquarters. These

are measures of force effectiveness. The degree to

which the system state matches the desired goal states
Chapter 5 101

indicates a level of effectiveness. Alternatively, C2

effectiveness may be viewed as dependent on the

functional processes of the C2 system, with measures

obtained mainly at the task level.

A C2 assessment framework encompasses several

factors that must be considered iteratively, as

discussed in the introductory chapter. Typical factors

important for the identification and selection of MoM

include the:

• Assessment configuration, e.g., storyboard,

testbed, constructive simulation, field trial;

• Assessment goal or purpose;

• Context, assumptions, and constraints;

• Scenarios or stimuli;

• Collection means, e.g., subject matter experts,

automatic data logging;

• Analysis plan; and

• Interpretation of results.

Categories of Measures

A common thread in the approaches for C2

assessment is the functional decomposition of the C2

cycle. C2 effectiveness depends upon the functional

processes of the C2 system, and the evaluation of

functions may be determined by data measured at the

task level.

The evaluation of tasks provides the most detailed

insight into C2 activities. The primary measures are
102 NATO Code of Best Practice for C2 Assessment

expressed in terms of time consumed and accuracy.

Task analysis must be performed prior to evaluation,

with the identification of task definition and the critical

elements for successful task completion.

Measures of a C2 system’s behaviour may thus be

reduced to measures based on time, accuracy, or a

combination which may be interdependent. Time

based measures are quantitative, while accuracy

measures may be quantitative or qualitative.

For C2 tasks, time-based metrics include the:

• Time taken to react to an event (time to notice

process and act upon new information);

• Time to perform a task (time to make decision);

• Time horizon for future for predictive analysis; and

• Rate of performing tasks (tempo).

Metrics for accuracy include:

• Precision of the observed system(s) performance;

• Reliability of the observed system(s) performance;

• Completeness (known unknowns, unknown unknowns);

• Errors (alpha, beta, omission, transposition,

severity); and

• Quality of information produced.

Some accuracy measures may be calculated in units

of time, e.g., the time taken to detect an error. Quality

of decisions is difficult to evaluate objectively, except

by focusing on outcomes. The processes involved may

have to be examined to obtain objective measures, or
Chapter 5 103

subject matter experts may be consulted to make an

evaluation. Accuracy of information implies both the

accuracy of the data and the accuracy of the

interpretation of the data.

Time based and accuracy measures often bear an

inverse relationship, implying a trade-off between speed

of performance and accuracy of performance. Speed

of performance must be specified in terms of minimum

desired accuracy or completeness, and accuracy

measurements in terms of time available. Therefore,

the specification of thresholds or standards for metrics

must be referenced in terms of imposed constraints.

Examples of time and accuracy based measures are

compiled in Table 5.4. Table 5.5 provides some

additional examples, specifically MoP and MoCE.

Table 5.4. Examples of Time- and Accuracy-Based Measures
104 NATO Code of Best Practice for C2 Assessment

Table 5.5. Examples of MoPs and MoCEs

Example Headquarters C2 MoM

C2 measures may also be divided into sets

corresponding to the sequential steps of the C2 cycle.

These include:

• Monitoring and understanding: information

transmission, values, times, effect, comprehension;

• Planning: information exchange, coordination,

impact, flexibility, process quality; and

• Directing and disseminating.

The MoM for C2 can also be focused on four levels: a

network of headquarters, a single headquarters, the

individual cells within the headquarters, and

performance of specific tasks within the cells.
Chapter 5 105

OOTW MoM

While national NATO policies require that military

forces be prepared for high intensity conflict, forces

have been increasingly involved in low-intensity

conflicts and C2 analyses for OOTW are therefore

becoming important. OOTW include force deployment

to create or maintain conditions for a political solution

in order to avoid escalation into hostilities. Threats to

international and national security may also unfold from

natural disasters, terrorism organised crime, civil

unrest, migration, or other territorial intrusions. Most

OOTW are inherently joint or combined operations.

While the determination of MoM has been stated as

difficult to obtain, OOTW offer even greater challenge

for MoM. Traditional MoFE such as loss exchange

ratios, combat effectiveness, or duration of the

campaign are rarely applicable to OOTW. In such

operations, military forces may play important roles

but political concerns may limit the scope of imposable

solutions. Public and political pressures may result in

shifts in the selection of criteria for MoM e.g., more

emphasis may be placed on personnel casualties and

less on equipment losses.

OOTW MoPE

While MoFE and MoCE provide measures of success

for military operations, MoPE measure the degree of

attaining political objectives. In some cases, such as

humanitarian assistance or nation building MoPE may

measure the degree of improvement in the quality of

life of the populace.
106 NATO Code of Best Practice for C2 Assessment

MoFE usually were the highest MoM used within the

analysis of Article V missions3 assuming that

effectiveness is directly related to the higher level

MOPE such as “winning the war.” However, such an

assumption may not always apply to OOTW. For

example, military actions that would be highly effective

in accomplishing mission objectives in war might be

quite counterproductive in OOTW. In fact, the value

of military actions in OOTW is not so much a question

of physical effects, but rather how military actions and

their physical effects are perceived by the various

actors and the population in the theatre, how the

military actors interpret the behaviour of the other

actors and the critical mission task conditions such

as, for example, political interest and media attention.

McCafferty and Lea developed low-level military-related

measures (MoCE) to cover OOTW (McCafferty and Lea,

1997). The MoM, which are classified as MoCE, include the:

• Time between the arrival of friendly forces in the

area and their deployment;

• Time between deployment of friendly forces and

contact with adversary forces;

• Length of time adversary forces were under

observation without posing a threat to friendly forces;

• Length of time friendly forces are in potential

danger (i.e., adversary forces have the

opportunity to fire on friendly forces); and

• Time horizon of friendly C2 processes (how far

into the future they are focused).
Chapter 5 107

Mobility may be important for OOTW, as well as

sustainability and self-sufficiency in theatre, with the

implication of emphasising measures of reliability and

maintainability. Moreover, the perception of the

capabilities of deployed forces acts as deterrence or

coercion on the parties in conflict.

Some examples identified by McCafferty and Lea are:

• Opportunities to employ forces, which reflects the

range of military capabilities available;

• Strategic deployment, which is related to

deploying and recovering the right force to

theatre efficiently and in time;

• Endurance, to maintain an effective force in

theatre for an extended time;

• Mission objectives, to measure the success of

achieving military objectives in OOTW; and

• Successful termination, to deal with progress to

the desired end state (the criteria may be political

and thus not measured by military activities).

One class of effectiveness indicators in OOTW is

provided by transition measures, which focus on the

progress in the transfer of responsibilities to the followon military force or civil agency. Transition measures

focus on the degree that follow-on organisations

assume tasks and responsibilities.

Progress toward success may be tracked by normality

indicators, which are indirect measures of the effects

of military involvement in OOTW, although causal

relationships are difficult to prove (Lambert, 2000).
108 NATO Code of Best Practice for C2 Assessment

These MoPE may be obtained by evaluating the extent

to which conditions have been restored.

Normality indicators measure the level of improvement

in the quality of life of the general population, and may

be defined as “relative measures of the state of

normalcy characterising an element of the civil

environment, through data collected on a regular basis

and assessed to have the frequency, quantity,

consistency and coverage required to make a useful

objective assessment of the changes occurring in the

civilian populace.” (Department of National Defence,

Canada, 1999). Normality indicators can be grouped

in categories and adapted to meet the changing

requirements: political, socio-economic levels of

development, cultural, legal and technological.

Table 5.6. Normality Indicators

Limitations of normality indicators include:

• Inexperienced personnel available for data

collection and analysis. A mix of inexperienced

civil and military personnel are often assigned to

collect data in fields foreign to them; training may

be necessary to assure reliable and valid data;

• Temporary effect due to military presence for data

collection. The mere presence of military personnel

collecting data may affect normality measurements;

• Difficulties in obtaining valid and reliable data

calibrated against baselines. It may be difficult to
Chapter 5 109

establish the threshold for “normality” if archive

data is not available;

• Extrapolation in space and time from a specific

locality. It may be inappropriate to extrapolate civil

progress to an entire region. Sampling is important;

• Limited resources and constraints for collection

and analysis. Data must be collected

consistently but may be occasionally

unobtainable due to physical inaccessibility or

lack of personnel;

• A “snapshot” which may not provide trends if

infrequently obtained. Trend analysis requires

sufficient data and sampling rates; and

• During OOTWs, the relevant MoM may change

over time, particularly the lower level MoCE and

MoP. For example, during the earliest phases of

NATO operations in Bosnia, tracking weapon

systems and knowing how many of them were in

cantonments was a major MoM. Once the forces

were separated and most weapon systems under

control, emphasis shifted to the disruptive activities

such as road blocks and ethnic harassment. As

these behaviours became less frequent, NATO’s

emphasis shifted to normality indicators.

MoM Hierarchies: Some Examples

Modular Command and Control Evaluation System

Evaluation of C2 effectiveness requires a

comprehensive approach for the preparation of the

evaluation process, the collection of data, and its
110 NATO Code of Best Practice for C2 Assessment

interpretation. MCES addresses both the managerial

and analytical aspects of evaluation and was originally

developed for the systematic comparison of C2

systems. The objective of MCES is to guide analysts

in the identification of appropriate measures for

estimating the effects of C2 on combat.

MCES prescribes a process of measurement, but does

not identify either a measurement system or a set of

measures. Similarly, while calling for the collection of

data, MCES does not provide details on how data are

to be collected. MCES does provide guidance on how

good measures and good collection procedures are

characterised, but leave the details of the

measurement, data collection, and analysis plans to

the analyst.

MCES considers C2 as consisting of three components:

• Physical entities (equipment, software, people);

• Structure (interrelationships of entities); and

• Processes (C2 functions).

The boundary of a C2 system here is defined as a delineation

between the system studied and the environment. The US

Army’s Training and Doctrine Command (TRADOC) C2

MoCE Handbook adds mission objective as the top layer

of the hierarchy of C2 components.

MCES focuses on measures as opposed to models,

but includes the cybernetic loop model of generic C2.

It consists of seven procedural steps.
Chapter 5 111

HEAT

While originally developed for theatre-level combat

applications, the HEAT system has proven robust. For

example, it has been used to assess US Department

of State crisis task force performance, military

operations in Grenada, Panama, and Haiti, and several

exercises focused on peacekeeping, and humanitarian

assistance missions. The underlying C2 process of

monitoring, understanding, and developing alternative

actions; predicting the consequences of each

alternative for each possible future under

consideration, decision; developing and promulgation

of plans and directives and requests for support;

seeking information; making reports; and responding

to inquiries are all relevant across the ranges of OOTW

missions. HEAT has been modified since 2000 to

include measure of the quality of collaboration, which

is often a key process in OOTW.

ACCES

ACCES is a derivation of HEAT, which was developed

primarily for joint theatre-level operations. ACCES

reorganised HEAT concepts into army doctrinal

language and doctrine, but shares the same

philosophy. ACCES has been applied to numerous

division and corps command centre assessments. It

represents a comprehensive set of practical and

objective performance measures for C2 activities. The

primary focus of ACCES is the overall performance of

a command centre or network of command centres,

at various stages of the C2 process, from the collection

of data to the conversion of data to intelligence to the

implementation of plans and directives. The underlying
112 NATO Code of Best Practice for C2 Assessment

approach to ACCES is that C2 comprises

interdependent sub-processes which can be observed

and measured. ACCES considers C2 as an adaptive

control process, where information collected from the

outside is processed internally to generate plans that

may be adapted to reflect new information. ACCES

takes the view that the overall effectiveness of a

command centre can be judged by the viability of its

plans. A good plan is one that can be executed without

the need for modification beyond the contingencies

stated in the plan and that remains in effect throughout

its intended life.

Multi-Attribute Utility Theory (MAUT)

MAUT is similar to ACCES in the sense that both use

functional decomposition and function-specific

evaluation metrics. The major differences are that MAUT

can be used with any set of metrics (including those

from ACCES), which must be specified by the analyst.

MAUT assigns weights to the MoM at each level of the

MoM hierarchy and utility values or scores at the lowest

level. MAUT then aggregates upwards the weighted

scores to provide composite scores of effectiveness.

MAUT, if properly used with appropriate application of

judgmental weights, will allow integrated analyses based

on multiple MoM. While this is often satisfying to

decisionmakers (it provides a single index of quality),

analysis should always monitor the components of such

indices as they may provide insight into strength and

weaknesses of the C2 system. For example, many

OOTW C2 problems involve a variety of objective

functions and trade-offs. MAUT results should always

be assessed by sensitivity analyses.
Chapter 5 113

Many applications of MAUT assume additive

composition of MoM. However, this is a very restrictive

assumption that needs to be validated in each case.

In addition MoM may interact suggesting that the

aggregation of MoM is at partly multiplicative (Keeny

and Raiffa, 1976; Sarin, 2000).

Collaboration C2 Metrics

The following collaboration metrics have evolved out

of work done by Evidence Based Research, Inc. for

the United States Office of Naval Research. These

collaboration metrics focus on individual and team

cognitive/awareness, team behaviour, and team

products. Individual cognitive metrics measure

collaboration, team members’ understandings about

their mission and their team, and team cognitive

metrics apply the individual cognitive metrics to

quantify the level of awareness in a team. There are

four classes of these metrics:

• Averages of the understanding among team members;

• Extent of alignment of these understandings;

• Maximum level of understanding anywhere within

the team; and

• Presence of gaps in understanding throughout

the entire team.

Team behaviour metrics measure the key behaviours

indicative of effective teams. These behaviours include

smooth and efficient synchronisation, efficient

information exchange, adaptability, effective workload

distribution, and team member engagement. Team

product metrics measure product quality and team
114 NATO Code of Best Practice for C2 Assessment

efficiency and are the bottom-line “proof of the pudding”

metrics, applicable whether a team or a single

individual produces the product.

Other metrics under consideration are: task

performance, workload, level of engagement (buy-in),

synchronisation, information needs workload and

handling, workload awareness and handling, and

problem awareness and handling.

Other Considerations in the

Selection and Interpretation of MoM

Effects of Uncertainty

In order to state a level of confidence in the

interpretation of MoM, the underlying assumptions

must be clearly stated and uncertainties recognised.

Uncertainties manifest themselves in several ways that

may affect MoM. They may be grouped as follows:

• Study assumptions—(uncertainties in the

scenario, model input);

† Relevance to the purpose of the evaluation,

uncertainties in the military objective,

knowledge of enemy concept of operations,

intentions, capabilities, weapon performance,

uncertainties in terrain data, etc.

• Modelling assumptions—(uncertainties in the

model, structural uncertainty); and

† Human performance, parameters, objects,

attributes, processes, effects of constraints,

effects of aggregation and de-aggregation,
Chapter 5 115

deterministic (usually high hierarchical level

but low resolution) versus stochastic models,

especially in OOTWs; and

† Uncertainties about the implications of value

changes in lower-level MoM with respect to

the values of higher-level MoM (e.g.,

increases in arrest rates at late stages of an

OOTW may be negative, while increases in

arrest rates at early stages may be positive).

• Model sensitivity—(uncertainties in the outcome)

† Hypersensitivity to input variations, (instability

or chaos theory), effects of model nonlinearities and non-monotonic behaviour

(effects of thresholds), decisionmaking for

local versus global optimisations, etc.; and

† Sensitivity analysis may be applied to identify

uncertainty. By varying the assumptions and

input data within the plausible ranges,

excursions in the analysis verified by the

subject matter experts provide insight into the

effects of uncertainty.

Impact of Technological Changes

The rapid pace of technological change involving

information systems is causing major changes in the way

C2 is perceived and executed, leading to potential changes

in the way war fighting commands are organised. For

example, the last decade has seen the emergence of

collaborative technologies, which enable new ways to

command and control military forces and for them to

interface with other actors. To keep pace with and to
116 NATO Code of Best Practice for C2 Assessment

evaluate the impacts of these changes, the nature of these

changes and impacts need to be understood so that the

appropriate MoM can be developed.

One approach to doing this postulates those

differences likely to occur between today’s C2 and

future C2, and then describes an evaluation

methodology, including MoM, to measure the impact

of the changes. Other approaches are Social

Construction of Technology (SCOT) (Bijker, 1989),

Socio-Technical Networks (Elzen, Enserink and Smit,

1996) and Actor Network Theory (ANT) (Latour, 1997).

Conclusions

No single measure or methodology exists that

satisfactorily assesses the overall effectiveness of C2

systems. As a minimum, the following factors must be

considered in conducting an analysis of C2:

• Determine the appropriate levels of MoM hierarchy;

• Identify specific MoM which are practically obtainable;

• Specify means of collection of MoM;

• Assure the validity and reliability of measures for correct

interpretation with quantifiable levels of confidence;

• Be aware that variation in measurements (e.g.,

due to human factors) may well cause

unacceptable levels of uncertainty. Hence the

analyst must pay particular attention to

measurements related to the human element;

• Consider that while MoPE and MoFE may provide

the most persuasive measures from the military
Chapter 5 117

perspective, MoCE and MoP are the most readily

derivable by operations analysts; and

• Account for the principles of reliability and validity

to avoid the risk of generating false conclusions.

Recommendations for Generation and

Selection of MoM

The principal objective for MoM is to determine

judgements of the degree to which C2 or changes to

C2 may improve force effectiveness and to provide

convincing arguments for the improvements. It is

important to stress that the purpose is to assess the

contributions of C2 in terms of how C2 improves the

effectiveness of military missions, and not the quality

of the C2 process itself. However, to arrive at these

assessments and assign attribution to the C2 system,

the C2 system must be included in the analysis. To

achieve this objective, the following steps are required:

• The objectives for the assessment must be

established and clearly stated;

• Selection of MoM should not be done in isolation

from consideration of the assumptions,

constraints, models, tools, scenarios, or other

elements of the analytic plan and assessment

process. The assumptions used in the model

and/or evaluation must be stated along with their

potential impact on the results;

• A detailed assessment of reliability and validity of

the selected measures needs to be made in

order to determine a level of confidence in

measures; and
118 NATO Code of Best Practice for C2 Assessment

• For C2 acquisition analyses, the generation of

measures should occur in parallel with the

development of the system, so that as the

system is being matured, developers can know

the standards to which they are being held.

Summary of the Challenges and Issues

in the Evaluation of C2

• Correlation of MoPE and MoFE with C2 process

measures (e.g., battle outcome against lowerlevel measures) is difficult;

• Separation and linkage of the respective

relationships between C2 and users,

organisations and military objectives requires

some effort;

• Aggregated measures (e.g., as obtained from

ACCES or MAUT) have limitations in the

diagnosis of C2 success or failure. A careful

analysis is required to provide a comprehensive

assessment of highly complex C2 systems

based on a small number of summary measures

of outcome and process;

• The assessment of the reliability of measures in

an environment where sample sizes are small

will remain difficult and may require the use of

non-parametric statistics;

• The analyst must pay attention to the complex

task of establishing and measuring control

variables in order to achieve correlation of

measures against a wide spectrum of scenarios

and staff;
Chapter 5 119

• Defining criteria to differentiate measures must

be established;

• Verifying measurement criteria (e.g.,

discrimination) must be ensured;

• For the near future, collecting data to support C2

measures will remain labour intensive because

C2 processes remain human intensive;

• Many of the measures for information processing

concern completeness of the information.

Deciding what makes information complete

requires coordination and cooperation between

the assessor and the user;

• The relationship between outcome and process

may be complex because C2 is an integrated

system with continuous feedback;

• The analysis of uncertainties and measures of

central tendency and dispersion are both

significant when examining C2 issues; and

• A Command Post Exercise (CPX) is a useful

venue for evaluation of C2. However, the costs

involved generally preclude conducting a CPX

solely to evaluate the C2 process or a C2 system.

Cost control is increasingly leading to the use of

laboratory and human in the loop experimentation

to develop knowledge and insights.

Summary of the Challenges and Issues

for OOTW MoM

• Cost/benefit: cost in time and effort for data

collection and analysis may outweigh benefit;
120 NATO Code of Best Practice for C2 Assessment

• Standardisation of data collection, evaluation,

and analysis with many diverse nongovernmental organizations (NGOs), UN

agencies, and militaries;

• Factors outside military control: other agencies,

policy restrictions;

• Creation of a rich and comprehensive set of MoM

to preclude reliance on a limited number of MoM

as the key to success;

• Availability and consistency of information (e.g.,

in OOTW, a possible consequence of different

factions controlling geographic areas);

• Merging of strategic, operational, and tactical

domains; and

• Clear recognition of roles and responsibilities of

all participants.

Chapter 5 Acronyms

ACCES – Army Command and Control Evaluation System

AHWG – AC/243 Panel 7 Ad Hoc Working Group

ANT – Actor Network Theory

C2 – Command and Control

C3I – Command, Control, Communications & Intelligence

COBP – Code of Best Practise

CPX – Command Post Exercise

DP – Dimensional Parameters
Chapter 5 121

HEAT – Headquarters Effectiveness Assessment System

MAUT – Multi-Attribute Utility Theory

MCES – Modular Command and Control Evaluation Structure

MoCE – Measures of C2 Effectiveness

MoE – Measures of Effectiveness

MoFE – Measures of Force Effectiveness

MoM – Measures of Merit

MoP – Measures of Performance

MoPE – Measures of Policy Effectiveness

MORS – Military Operations Research Society

NGO – Non-Governmental Organization

OOTW – Operations Other Than War

SCOT – Social Construction of Technology

TRAC – TRADOC Analysis Center

TRADOC – US Army Training & Doctrine Command

Chapter 5 References

This chapter is based on the results of the SAS-002 workshop

on Measures of Merit. Additional references include:

Adelmam, L. (1992). Evaluating decision support and

expert systems. Chichester, UK: John Wiley

and Sons Ltd.

Bijker, W.E. (1990). The Social Construction of

Technology. PhD Thesis University of Twente,

Eijsden, The Netherlands.
122 NATO Code of Best Practice for C2 Assessment

Bornman, L. G., Jr. (1993). Command and control

measures of effectiveness handbook (C2 MoCE

handbook) (TRAC-TD-0393). Fort Leavenworth,

KS: U.S. Army Training & Doctrine Command

(TRADOC) Analysis Center.

Brown, R. V., Kahr, A. S., & Peterson, C. (1974).

Decision analysis: An overview. New York: Holt,

Rinehart and Winston.

Canadian Department of National Defence Document

B-GG-005-004/AF-023. (1999). Civil-Military

cooperation in peace, emergencies, crisis and

war. Available on-line at http://www.dnd.ca/eng/

index.html

Defence Systems. (1982). Theatre headquarters

effectiveness: Its measurement and relationship

to size, structure, functions, and linkages; Vol.

1: Measures of effectiveness and the

headquarters effectiveness assessment tool.

McLean, VA: Author.

Defining and Measuring Success in the New Strategic

Environment [Technical Cooperation Program

Panel JSA-TP-3] Proceedings Report (1999)

Ottawa, Ontario, Canada.

Duff, J. B., Konwin, K. C., Kroening, D. W., Sovereign,

M. G., & Starr, S. H. (1992). In T. J. Pawlowski

(Ed.), Command, control, communications,

intelligence, electronic warfare measures of

effectiveness [Final Report]. Fort Leavenworth,

KS: Military Operations Research Society

(MORS) workshop.
Chapter 5 123

Elzen, B., Ensenrink, B. & Smit, W.A. (1996). SocioTechnical Networks: How a Technology Studies

approach may help to solve problems related

to technical change. [Social Studies of Science,

vol 26, pp.95-141]

Engel, R., & Townsend, M. (1991). Division Command

And Control Performance Measurement.

Toronto, Ontario, Canada: Defence and Civil

Institute of Environmental Medicine (DCIEM).

Fliss, R., Feld, R., Geisweidt, L., Kalas, W. D., Kirzl,

J., Layton, R., et al. (1987). Army command

and control evaluation system user’s handbook

(MDA903-96-C-0407). Ft. Leavenworth, KS:

Army Research Institute for the Behavioral and

Social Science.

Halpin, S. M. (1992). Army command and control

evaluation system (ACCES): A brief description

(Working paper LV-92-01). Ft. Leavenworth,

KS: US Army Research Institute for the

Behavioral and Social Sciences.

Hawkins, G. (1984). Structural variance and other

related topics in the SHAPE Armour/AntiArmour study. In R. K. Huber (Ed.), Systems

analysis and modeling in defense—

development, trends, and issues. New York:

Plenum Press.

Human Systems. (1993). Survey of evaluation

methods for command and control systems.

Ontario, Canada.
124 NATO Code of Best Practice for C2 Assessment

The impact of C3I on the battlefield Panel 7 on the

defence applications of operational research

(AC243 panel-7 TR/4). [North Atlantic Treaty

Organisation (NATO) unclassified technical

report] (1994). Ad Hoc Working Group on the

impact of C3I on the battlefield. Brussels,

Belgium: NATO Defence Research Group.

Available from www.nato.int

Keeny, R. L., & Raiffa, H. (1976). Decisions with multiple

objectives: Preferences and value tradeoffs.

Chichester, UK: John Wiley and Sons Ltd.

Lambert N. J. (1998). Operation Firm Endeavour:

Campaign Monitoring; Factional compliance

and the Return to Normality in Bosnia and

Herzegovina. Operational Analysis Branch, HQ

ARRC, Queens Avenue, JHQ Rheindahlen,

41179 Moenchengladbach, Germany. Also

available from the British Library Document

Supply Centre, Boston Spa, Wetherby, West

Yorkshire, LS23 7BQ, United Kingdom,

Reference DSC Shelfmark: 99/20361 and 99/

20367 (annexes).

Lambert, N. J. (1998). Operational analysis in the field:

The utility of campaign monitoring to the

implementation force (IFOR) operation in

Bosnia and Herzegovina 1995—1996.

Proceedings from Cornwallis III, Analysis for

Peace Operations. (ISBN 1-896551-22-X). The

Lester B Pearson Canadian International

Peacekeeping Training Centre, Cornwallis

Park, PO Box 100, Clementsport, NS BOS

1EO, Canada.
Chapter 5 125

Lambert, N. J. (in press). Measuring the Success of

the North Atlantic Treaty Organisation (NATO)

Operation in Bosnia and Herzegovina 1995—

2000. European Journal of Operations

Research, Special 2000 Edition.

Latour, B. (1997). Bruno Latour’s keynote speech: On

recalling ANT. In Actor network and after. Centre

for Social Theory and Technology (CSTT),

Department of Sociology and Social Anthropology,

Keele University, United Kingdom. Available from

http://www.comp.lancs.ac.uk/sociology/

stslatour1.html

Latour, B. (1999). On Recalling ANT. In J. Law & J.

Hassard (Eds.), Actor network theory and after

(pp. 15-25). Oxford, United Kingdom: Blackwell

Publishers.

Leroy, D., & Hofmman, D. (1996). Criterion of

command, control, communications,

intelligence (C3I’s) effectiveness. 1996

Command & Control Research & Technology

Symposium Proceedings. Washington, DC:

National Defense University.

McCafferty, M. C., & Lea, S. C. (1997). Measures of

effectiveness for operations other than war

(CDA/HLS/R9714/1.1). Farnborough, United

Kingdom: Center for Defence Analysis.

National Research Council. (1998). Modeling human

and organizational behavior—Application to

military simulations (ISBN 0-309-06096-6).

Washington, DC: National Academy Press.

Available from http://www.nap.edu
126 NATO Code of Best Practice for C2 Assessment

1The following reports can be ordered from http://

www.rta.nato.int, RTO-MP-029 AC/323(SAS)TP/9. Workshop on

C3I Modelling (April 1999), RTO-MP-038 AC/323 (SAS) TP/12

Modelling and Analysis of Command and Control (June 1999),

RTO-TR-9 AC/323 (SAS) TP/4 COBP on Command and Control

(March 1999).

2David S. Alberts, John J. Garstka, Richard E. Hayes, and David

A. Signori, Understanding Information Age Warfare (Washington,

DC: CCRP Publication Series, August 2001).

3Article V of the North Atlantic Treaty, which deals with classic

military attacks on members.

Olae, N. G., Roy, J., & Wetter, M. (2000). Performance

drivers—A practical guide to using the balanced

scorecard (ISBN 0-471-49542-5). Chichester,

UK: John Wiley and Sons Ltd.

Sarin, R. K., (2001). Multi-attribute utility theory. In S.

I. Gass & C. M. Harris (Eds.), Encyclopedia of

operations research and management science

(Centennial ed.). Boston: Kluwer Academic.

Sovereign, M., Kemple, W., & Metzger, J. (1994).

Command, control, communications,

intelligence evaluation workshop (C3IEW).

Measures Workshop, Phalanx, United

Kingdom.

Sweet, R., Metersky, M., & Sovereign, M. (1985).

Command and control evaluation workshop

[Military Operations Research Society (MORS)

C2 MoCE Workshop]. Monterey, CA: Naval

Postgraduate School.

Tolk, A. (1995). Zur Reduktion struktureller Varianzen

[On the reduction of structural variances].

Unpublished doctoral dissertation, University of

the Federal Armed Forces of Germany, Pro

Universitate Verlag, Sinzheim.
127

CHAPTER 6

Human

Organizational

Factors

If I had time...to study, I think that I

should concentrate almost entirely on

the ‘actualities of war,’ the effect of

tiredness, hunger, fear, lack of sleep,

weather...It is the actualities that make

war so complicated and so difficult, and

are usually neglected by historians.

—Field Marshall Archibald Wavell, 1883–1950,

Author of Soldiers and Soldiering
128 NATO Code of Best Practice for C2 Assessment

Importance of Human and

Organisational Factors

T

he human dimension largely distinguishes

command and control (C2). Key differences

between C2 analyses and traditional military

operations analysis (OA) applications include the need

to deal not only with military organisations, but also

with distributed military teams (and organisations)

under stress and their decisionmaking behaviour as

well. Moreover, in operations other than war (OOTW),

consideration must be paid to the behaviour of and

interaction with non-military organisations, political

groupings, and amorphous groups such as crowds and

refugees. Thus, the formulation of the problem and

the development of solution strategies cannot be

completed without explicit consideration of both human

and organisational issues.

The human factors of interest fall into three major categories:

• Human behaviour related to performance

degradation, such as stress and fatigue, and as

a consequence of social interactions among

individuals and members of groups;

• Decisionmaking behaviour (cognitive questions)

including the cognitive complexity of the issues

and the capacities of the commanders or other

decisionmakers of interest; and

• Command style.

By contrast, organisational factors deal with

relationships among groups of individuals, including

connectivity, roles, and organisational structures.
Chapter 6 129

Since both human and organisational factors can affect

C2 performance, the operations analyst must consider

their impact early in the research design process and

review a priori assumptions about them in an iterative

manner throughout the entire analytical process.

Human and organisational factors must be considered

as part of structuring the problem, selecting measures

of merit (MoM), defining scenarios, developing solution

strategies, and selection of tools.

The first key consideration when structuring the

problem is whether individual decisionmaking and

behaviour of individuals or groups is important to the

C2 processes under analysis. If the research question

can be answered without considering differences

between individual decisionmakers and groups then

the additional complexity that issue introduces should

be avoided. For example, in addressing a C2 issue

that deals, all other things being equal, with a simple

change in connectivity (which headquarters will have

which linkages to others), human behaviour may not

be important to the analysis at least as long as combat

missions are involved. However, the same change in

connectivity might affect relations with non-military

organisations and individuals essential for mission

success in OOTW suggesting their response must be

accounted for in an appropriate manner. Thus,

deciding on the importance of human issues in OOTW

and identifying the issues at stake requires a good

general understanding of human behaviour and its

underlying motivations beyond military common sense

as well as knowledge about relevant cultural factors

and the interests of the parties involved.
130 NATO Code of Best Practice for C2 Assessment

Human Factors

Human Performance and Behaviour

Human performance affects behaviour and vice versa.

Human performance depends on psycho-physiological

variables (e.g., stress, fatigue, sleep deprivation, hunger,

and alertness) and on ergonomic and external factors

limiting performance and behavioural freedom.

Individual and group behaviour is the result of social

interaction. It includes interactions by military

commanders and their troops, underlying psychological

processes and factors (e.g., fear, morale, and values),

and the cultural, educational, and religious background

of individuals. There is significant historical evidence

that inferior combat potential as measured in terms of

numbers of personnel and weapon systems may be

compensated for by superior human performance in

battle (Dupuy, 1979).

Any time human performance and/or behaviour are at

issue, parameters and/or models will be needed to

reflect those issues. For example, systems that involve

human activity, such as watch or command centres,

need to be studied in ways that reflect differences in

C2 performance that can be traced to human

performance/behaviour issues. In addition, differences

can arise from experience or training, coalition features

(e.g., language, national doctrine, command style), or

service/branch-unique doctrine and practice. These

kinds of individual performance and behaviour issues

may be modelled in two ways. They can be treated

stochastically, in a manner that reflects their

occurrence in “real” systems depending on situational

factors (black box approach), or in terms of process-
Chapter 6 131

oriented behavioural models, describing the

psychological processes behind the observable

behaviour of individuals or groups in a given situation

(Long Term Scientific Study SAS-017 on Human

Behaviour Representation, Final Report, Chapter 2).

The decision that human performance and behaviour

may vary meaningfully will have a clear impact on the

choice of models and analytic approaches (e.g.,

stochastic processes or action-theoretic models such

as Norman’s Activation Trigger Scheme (ATS) for

simulating the dynamics of actions or reactions)

(Norman, n.d.).

Where human performance is considered to be a

meaningful factor (e.g., C2 within command centres

when wearing chemical protection gear), some

experimentation may be necessary to develop realistic

parameters for the impact on error rates or the pace

of work. In other cases, such as simple fatigue, human

factors specialists may be able to provide valid

parameters from work in other contexts. Such

specialists are often valuable members of research

teams. In any case where the workflow and work rate

within command centres is relevant, human

performance parameters must be considered. In

addition, human performance issues will have some

effect on decisionmaking. Error rates increase as

people become tired and overloaded, altering the way

they work and the information they consider.

However, research on how to represent human

behaviour and its impact on performance in models is

still in an early stage. This is particularly true in the

context of OOTW. Those operations require the cooperation of non-military actors. In addition, military

mission objectives include providing security and
132 NATO Code of Best Practice for C2 Assessment

assistance in the reconstruction of conditions in the

theatre of operations and helping to stabilise the

situation to a degree that ultimately permits the military

forces to leave. The use of force in OOTW is limited in

degree and kind to responding appropriately to threats

for defensive and protective purposes and to coercive

actions for enforcing compliance with agreements.

Military actions that would be highly effective in

accomplishing mission objectives in war might be

counterproductive in OOTW. In fact, the value of

military actions in OOTW is not so much a question of

physical effects, but rather how military actions and

their physical effects are perceived by the various

actors and the population in the theatre, how the

military actors interpret the behaviour of the other

actors, and the critical mission task conditions (e.g.,

political interest and media attention). Thus, a careful

analysis of mission tasks by human science experts

is indispensable for modelling and assessment of C2

options in OOTW (Baeyer, 2001).

It follows that, in addition to the traditional composition

operations research/operations analysis (OR/OA)

assessment teams, assessors of C2 for OOTW must

possess hands-on experience with such operations

and relevant non-military organisations (e.g., private

aid organisations). The assessment team must also

have access to experts from the fields of political

science, cultural anthropology, demography, sociology

(including media impact research), social psychology,

and individual psychology. These experts will

contribute the expertise for diagnosing the relevance

of, and differences in, performance and behaviour of

actors and for the formulation of hypotheses for

assessing the “human issue risk” of analysis results.
Chapter 6 133

In addition, they may contribute to the modelling of

behavioural processes in analysis tools used to test

C2 system sensitivity via parametric variations of

human performance and behaviour parameters.

However, well-documented empirical knowledge on

human performance in military operations is scarce

and little is known about its relevance in circumstances

other than those prevailing when the underlying data

were collected. Similarly, experience and

systematically compiled data on behaviour and

response of individuals and groups to actions and

situations in OOTW are still limited and theories on

human behaviour are mostly untested in the context

of military operations. Therefore, dealing with human

issues in C2 analyses reduces to the problem of

addressing decisions under risk and uncertainty when

each of the C2 design options is tested for the range

of possible hypotheses on the implications of human

issues for C2 effectiveness.

Human Decisionmaking

Increasingly, assessment teams have to deal with

issues where individual decisions are important. This

is especially true for OOTW in which even tacticallevel decisions by a lower level military leader may

have strategic implications because of media

presence. This represents a major challenge because

the variety of human behaviours involved makes

modelling decisionmaking very difficult. Fortunately,

there are some approaches that can be used to cope

with these difficulties. The correct choice, however,

will depend on the research issue(s).
134 NATO Code of Best Practice for C2 Assessment

In some cases the analyst is asked to assume that

decisionmaking will follow established doctrine and

tactics, techniques, and procedures (TTPs). In these

cases, the challenge is to craft a set of rules or look up

tables that reflect the existing guidance correctly. Hence,

a model that replicated the “correct” set of decisions

would be useful for assessing simple C2 issues such

as the impact of new communication and information

technology or changes in connectivity or supporting

relationships within the force. However, maximising the

benefits of technical and organisational changes in a

C2 system might require an appropriate adaptation of

doctrine and TTPs. Therefore, testing the impact of

established rules should be part of the analysis.

Testing the impact of rules is an indispensable

prerequisite for models that have built-in sets of rules

that are not driven by approved TTPs, but rather by

opinions of subject matter experts or modellers whose

rationales have been neither validated nor accredited.

Considerable knowledge exists on how to organise

and validate such expert elicitation. Here again,

specialised team members may be helpful. Simple

adoption of models developed from subject matter

experts will put the assessment team at considerable

risk of accepting false conclusions. When such models

must be adopted, they should be explored in detail to

uncover their driving assumptions and subjected to

sensitivity analyses (Chapter 9). Where this cannot

be done, these models are best avoided when C2

assessments are performed.

From human factors research on stimuli that influence

human decisionmaking we have learned that human

decisionmaking capability is degraded in some
Chapter 6 135

situations and enhanced in others. These stimuli may

originate from, inter alia:

• Biological/physiological processes (e.g., physical

overexertion [fatigue], use of bio-chemical

substances, and/or sensory deprivation);

• Cognitive sources (e.g., confusion arising in

unfamiliar or unknown situations);

• Psychological processes (e.g., processes

causing emotions and stress);

• Social processes (e.g., group dynamics that

coerce or reinforce individual decisionmaking

depending on accepted social norms,

organisational infrastructure and procedures);

• Environmental factors (e.g., darkness, austere

and/or uncomfortable environments); and

• The decision support tools and technologies that

humans use (e.g., information displays and decision

support software). It is important to look beyond

technology at whether or not human decisionmaking

is improved, or even constrained in some cases, by

using computerised decision aids depending upon

their functionality and configuration.

Types of Decisions

The nature of the decisions being supported by C2 systems

will also enable the assessment team to make intelligent

decisions about how they influence the analysis. Three

useful decision types can be distinguished:
136 NATO Code of Best Practice for C2 Assessment

• Automatable decisions;

• Contingent decisions; and

• Complex decisions.

Automatable Decisions

Automatable decisions fall into the category of “simple

decisions.” The range of decision options is finite and

known, and the criteria for selecting among them are

clear. Basic sensor-to-shooter decisions are simple

decisions that are usually automated for the sake of

timely response (e.g., anti-aircraft or missile defence

systems). Similarly, the selection of patrol routes,

inspection strategies, and many logistics decisions

relevant for OOTW can be automated. For example,

scheduling can be seen as an optimisation problem in

which time, space, and priorities are traded off to

generate a “best” answer. Even though the decision

environment is constantly changing due to factors such

as weather and mechanical problems, scheduling

decisions are characterised by rules and algorithms.

Models of automatable decisions of this kind can be

built relatively easily.

However, where the C2 system employs humans to

make these choices, some error rate parameters will

be needed if the results are to be meaningful. For

example, error rates may increase if the time available

to make a decision is insufficient or physical demands

induce fatigue. Even where the operational concept

calls for the use of automated systems, the analyst

should explore the quality of the data, information, or

knowledge used to drive the process and the likelihood

that humans will be involved in collection or fusion.
Chapter 6 137

In these fully automatable decisions the assumption

is that “to know is to decide.” In these cases, if

uncertainty were adequately reduced, the correct

course of action or decision would be obvious. In that

case, decision theory classifies the problem as a

“decision under certainty.” These problems are trivial

except for the determination of the utility function in

case there are more than two selection criteria that

need to be considered and/or when constraints need

to be accounted for to complete the analysis (Keeny

and Raiffa, 1976).

Contingent Decisions

The next level of decisionmaking complexity is best

thought of as contingent decisions. These are cases

where the commander has thought through the situation

and developed a set of alternative actions or decisions

that are appropriate to the situation, but further information

on the operational environment will be needed to

determine which is the proper course of action. In other

words, “to know is to decide, but knowing is not yet

possible.” In some NATO countries the research

community terms this “opportunistic decisionmaking.”

In most cases a lack of clear, precise knowledge is

unavoidable. For example, the commander in a

defensive posture may recognise that the adversary

has several potentially viable options. The adversary

may not even know which alternative he will choose. In

such a case, the defending commander would both

develop courses of action to meet likely contingencies

and also undertake a variety of information collection

activities designed to provide as much warning as

possible when the attacker selects a main attack option.
138 NATO Code of Best Practice for C2 Assessment

Modelling contingent decisions is much more difficult

than modelling automatable decisions, but is similar

in that an underlying set of rules or algorithms still

drives the process. The added complexity comes from

the need to find the time when information is adequate

to select one of several actions. The best models for

that purpose are essentially hypothesis testing models.

They align information about the operational

environment against a finite set of alternative futures

and perform probability calculations to determine when

the commander has enough confidence to act or to

estimate the information gain in terms of the expected

value added to decisions as new information arrives

(Sherrill, 1996). Information entropy is also a valuable

measure of the information state of the commander. It

can also be extended to a measure of information

dominance (Perry and Moffat, 1997).

Complex Decisions

Finally, “complex” decisions are very difficult to model.

These require the decisionmaking system to:

• Recognise when a decision needs to be made;

• Identify the relevant set of options;

• Specify the criteria by which they will be judged; and

• Determine when the decision will be made.

Examples of complex decisions include the definition

of missions at the operational level, decisions to

change the fundamental activity of the organisation

(e.g., shift from the offence to the defence), and the

process that creates courses of action in response to

events on the battlefield or in OOTW. Except when
Chapter 6 139

doctrinal answers are available, complex decisions are

very difficult to model and even more difficult to validate

or accredit. Most successful efforts dealing with

complex decisions have used “human-in-the-loop”

techniques and relied on the quality and variety of

experts employed for reliability and validity. Some

promising research on modelling complex decisions

in military operations has been completed in the UK

and this is in the process of being incorporated as the

core of the next generation of closed form simulation

models of conflict being developed by Defence Science

and Technology Laboratories (DSTL) (Moffat, 2000;

Moffat, 2002) Similar research on tactical decision

automata is going on in Germany to improve the

capability of simulation systems for analysis as well

as training and staff exercises support (Hofmann and

Hofman, 2000, von Baeyer, 2001).

Command Style

Assessment teams often encounter the argument that

decisionmaking depends on the “commander’s style.”

Moreover, they are told, systems must be designed to

support commanders with differing styles. Because it

is an elusive and multi-dimensional concept, command

style represents a challenge to modelling. However,

this factor can be accommodated if the analyst is able

to develop a clear concept of the alternative command

styles that must be recognised and their consequences

for military decisionmaking.

Attributes of the Commander

Differences in command style may be reflected by

appropriate attributes such as the background and
140 NATO Code of Best Practice for C2 Assessment

training of commanders, their decision and order style,

risk tolerance, and operational experience. For

example, in conjunction with field experience, the

background and training of commanders affect the

richness of their understanding of the military situation

and their capacity to influence it.

Organisational Style

Another not totally unrelated topology deals with the

degree to which the commander uses a formal

decomposition of the situation versus a holistic, integrated

vision. The decomposition style of management is

associated with hierarchical and segmented work, as in

the Napoleonic or classic German general staff. This

heavily structured process allows centralised control and

tight coupling between the structure of the problem, the

structure of the supporting staff, and the flow of

information within and between command centres. The

classic centralised commander imposes his style on the

C2 process and impacts key organisational issues as

well as decision style.

The alternative command style is an open and holistic

one in which senior staff and commanders from related

command levels are directly involved in a broad

development of courses of action and implementation

plans. This more open process also has implications

for the information flow within and between command

centres. While decisions are still made authoritatively

at the centre (by the commander or senior staff), they

tend to generate loose guidance (mission type orders)

and to enable lower level commanders and their staffs

more latitude in implementation.
Chapter 6 141

Risk Style

Another topology applied by some practitioners is the

degree to which individual commanders (and

doctrines) are risk averse versus attracted to risk. Most

military enterprises have some properties that impel

commanders to minimise risk. The fact that lives,

national treasure, and serious national interests are

involved in warfare suggests that risk averse strategies

will tend to dominate. However, some military

commanders are more comfortable with greater risk.

Indeed, outnumbered or otherwise disadvantaged

forces must often take risks in order to prevail. To the

extent that the relative risk aversion of commanders

is relevant to the C2 analyses underway, assessment

teams will need to define and model variables that

represent this factor (Schultz, n.d.).

Recent research on Bayesian decisionmaking had

indicated that the perception of risk as measured by utility

loss relative to a goal value and perception of future

outcomes can in combination give insight into the way

these affect command decisionmaking (Moffat, 2002).

Orders Style

Commanders, and national command styles, have also

been shown to differ in the degree of detail contained

in directives to subordinates. At one end of the

spectrum is the commander who issues detailed orders

that specify what is to be done, how it is to be

accomplished, and when and where the specified

activities are to occur. At the opposite end of the

spectrum is the commander who issues “mission type

orders” which simply specify the mission to be

accomplished and leave decisions about the detailed
142 NATO Code of Best Practice for C2 Assessment

objectives, forces to be employed, critical terrain, and

timing up to the subordinate commanders. In between

are those who specify a series of linked objectives

(cross the river, take the high ground in the north, and

be prepared to defend or carry the attack north-east

into the valley) and supporting detail (e.g., forces

available, rough timetable keyed to the objectives) but

leave subordinates with considerable discretion within

that guidance. Both the speed of the C2 process and

the distribution of C2 work across command centres

(particularly planning and operations management) will

vary greatly depending on the commander’s style on

this dimension. National doctrine and practice may also

influence this factor.

Other Typologies

Other typologies of command styles are, of course,

possible and may be more relevant to particular C2

analyses. Human behaviour experts (e.g., cognitive

and organisational psychologists and anthropologists)

should be recruited to the project team if novel

categories are developed. However, the most

important issue when dealing with command style is

whether it is included in the analysis at all. The C2

research related hypotheses under analysis should

dictate the forms of command style examined.

However, the impact of command style should only

be examined because it appears to be necessary to

answer the analytic question(s) of interest. Otherwise

it tends to introduce a level of complexity that may

confound the other analyses underway.
Chapter 6 143

Organisational Factors

All elements of the C2 system are ultimately related to

one another. The linkage between human and

organisational issues, however, is particularly direct

and close. Properly done, organisational design

reflects the interaction among the tasks to be done,

the people available to perform them, and the systems

or tools that support those people. Hence, the “proper”

organisation of C2 depends in large measure on the

capabilities, training, and experience of the people in

the C2 system.

Organisation is a serious subject in military analyses.

For centuries the military has sought to implement

unambiguous relationships and responsibilities. Unity

of command is a central principle of war. When it has

been lost or comes into question, as in OOTW, the

professional militaries of the world have found

themselves very uncomfortable. Fortunately, military

organisational issues are driven by a fairly small and

finite list of principles. Assessment teams asked to

work on C2 issues can use the known list of factors as

a checklist about organisational differences to

determine whether they need to build organisational

matters into their research designs. This includes the

issue of informal relationships that may have evolved

in order to overcome organisational deficits and thus

streamline day-to-day operations. In fact, an

organisational design and command style that are

supportive of building informal relationships may

provide the flexibility for efficiently handling the

manifold demands facing commanders in OOTW.

Also, it should be recognised that organisational

implications which are perceived as detrimental to the
144 NATO Code of Best Practice for C2 Assessment

interests of the affected individuals and groups

inevitably would jeopardise cooperation, technical and

procedural improvements in C2 notwithstanding.

Organisational Differences

The principal differences between military

organisations are related to structure, function, and

capacity. Any change or innovation that can be

introduced in a C2 organisation falls into one of these

three categories which, therefore, may be used to

guide analysts when structuring a problem.

Structural differences include:

• The number of echelons or layers in the

command structure;

• The span of control for nodes in the

command structure;

• The pattern of linkages between those nodes

(e.g., hierarchical, spokes of a wheel, multiconnected, networked);

û Permanent versus transitory organisational

relationship; and

û Formal versus informal relationships.

Functional differences include:

• The distribution of responsibility: where functional

activities are located (e.g., intelligence, logistics,

command, civil military cooperation [CIMIC]);

û The distribution of authority (ideally colocated with responsibility);
Chapter 6 145

û The distribution of information;

û Functional specificity (e.g., fire support vs.

infantry or close air support vs. defensive

counter air) vs. general and integrated

military capabilities (mission tailored task

forces); and

• Degree of ambiguity in command relationships.

Difference in capacity is related to differences in:

• Personnel (e.g., quality, training, experience);

• Communications systems and architectures;

• Information processing systems and

architectures; and

• Field training and operational experience.

All these dimensions can be modelled, some more

easily than others. However, the assessment team’s

challenge is to identify those organisational factors that

are relevant to the C2 analyses underway. This issue

must also be addressed knowing that organisational

factors are interrelated: changing one may change

others. For example, the decision to eliminate a level

of hierarchy within a military organisation may have a

profound influence on the span of control. Similarly,

changing the distribution of information so that it no

longer follows the chain of command may have

profound implications for the ambiguity of command

relationships. Similar effects can be expected when

coalition operations involve ad hoc members.
146 NATO Code of Best Practice for C2 Assessment

Treatment of Organisational Factors

Because of the large numbers of organisational

variables that may be relevant to the analysis of C2

issues, they must be approached carefully and

systematically. When possible, organisation theory

expertise should be brought into the assessment team.

Review of organisational issues is treated in a twostep process guided by a hypothesis testing logic. The

first step should assess whether any organisational

variable is being manipulated directly. For example, a

decision to move from warfare domain task forces1 to

mission tailored task forces with air and land units

planning and operating together under a joint

commander would be seen as a direct manipulation

of organisational factors and should be studied as

such. The second step, a search for indirect effects of

organisational factors, may be more difficult and will

require that the assessment team use the list of

possible factors as a checklist and think through

whether they may be altered in a prepositional (if, then)

logic. An assessment team that posits a relationship

between the C2 analysis and an organisational issue

should be able to make a clear statement of the

hypothesis and the causality anticipated. This will

enable the research design to cover not only the gross

effect anticipated, but also the underlying causal

mechanism(s) that will be present if the proposition is

correct. Adopting this hypothesis approach is also a

safeguard against assuming that organisational issues

are easily or well understood and can be treated by

assumption. In fact, the organisational arena, like that

of human factors, is one of the most difficult in C2

analysis and must be approached with care and rigour.
Chapter 6 147

For example, the small group literature makes a clear

prediction that multi-connected groups will be able to

generate better answers to complex problems, but will

take more time to do so than either hierarchies or star

shaped groups. The causal mechanism in that theory is

greater dialogue and the representation of more

independent viewpoints. Moreover, these richer

discussions are expected to take more time. All other

things being equal, multi-connected groups that are found

to generate better answers to complex problems should

also engage in more dialogue and be found to have

considered more information and or solutions. Modellers

who want to take advantage of this factor to explore

alternatives to traditional hierarchical military

decisionmaking must also include the negative features

in their C2 models (e.g., demands for more time from

already overburdened staffs and slower decisionmaking).

Roles

The concept of a role comes from sociology. A role is

a set of behaviours expected by the self and others.

For most military systems the roles of commander and

key staff are well understood and arise from a

combination of tradition, training, experience, and

rational planning. Because of their origin, roles are

often a convenient way of capturing the doctrine about

responsibilities within the C2 system.

Roles can be used to capture “syndromes” or sets of

related attributes within a C2 system. For example,

an object oriented program might have different

functional organisations and their leaders might be

defined as having different attributes that reflect their

decisionmaking responsibilities and the information
148 NATO Code of Best Practice for C2 Assessment

they would receive or be able to obtain from the

information network. When assessing new C2

systems, analysts will often need to search for potential

role gaps or role overlaps. Either of these would be

dysfunctional in military operations over time. Changes

in information structures also have considerable

potential for creating problems of this type.

Role gaps, role overlaps, and even role conflicts may

be more of a problem in OOTW because military

organizations have to assume new roles not

embedded in their tradition and experience and

requiring them to cooperate with a variety of nonmilitary actors and organizations pursuing their

particular objectives and the roles of which are ill

defined in many cases. Thus, rather than being a

source of friction, changing or adapting existing

information structures may be part of the decision

problems that the C2 system must address in order to

eventually bring about a secure and stable situation in

the theatre of operations through overarching “unity

of command.”

Human and Organisational Issues

and Technology

The foregoing discussion of human and organizational

issues revealed that human performance and behaviour

as well as the organizational design of C2 systems,

and therefore the effectiveness of C2, depend on the

available communication and information technology.

In fact, a C2 organization resembles a system

composed of interacting human, organizational, and

technological elements as depicted schematically in

Figure 6.1 adapted from Mandeles et al. (1996).
Chapter 6 149

Figure 6.1 is meant to illustrate that the character and

performance of a C2 system may change as anyone of

elements in these three categories changes. Moreover,

since the human, organizational and technological

elements are closely linked in most cases, optimising

each one of them at a time under ceteris paribus

assumptions for the other two rarely ever results in an

efficient C2 system (Schot and Rip, 1996).

Figure 6.1. Representation of the C2 System, based

on Mandeles et al. (1996)

In particular, the assessment of the human-technology

relationship is a critical requirement that implies

challenges that can be both social and technical in nature.

Without adapting human thought and behaviour patterns

and organizational structures it may be impossible to

exploit the potential of new technology. On the other hand,

the performance of new communication and information

technologies may exceed human capabilities of

processing information (information overflow) and thus

result in a degradation of human performance and overall
150 NATO Code of Best Practice for C2 Assessment

effectiveness of a C2 system, the improvement of

technical parameters notwithstanding.

The challenges of adapting technological capabilities

to meet human capabilities and the requirements of the

social interaction processes of commanders and staff,

and non-military actors and populations in OOTW,

require socio-technical assessment approaches of the

kind that evolved in the fields of science and technology

studies (STS), technology assessment (TA) (Rip.1995),

and constructive technology assessment (CTA) (Van

de Poel, 1999).

Integrated Analyses

Because the issues arising from human and

organisational factors are so complex and so tightly

coupled, C2 assessment teams often use integrating

tools to define the key dimensions relevant to their

analyses and explore the relationships between and

among them. Integrating tools are those that use

selected key factors with powerful influence to cut

through the clutter and detail implied by trying to study

everything and concentrate instead on the most

important elements in the problem. These key driving

factors are used to conduct a simpler analysis that

can then be augmented by sensitivity analyses and

analytic excursions to ensure that the problem has

been fully and properly understood.

For example, Figure 6.2 has been used to illustrate

the relationship between the time available to make a

decision, the complexity of the decision, and the

uncertainty of the information available about the

situation. These three factors also reflect the risk or
Chapter 6 151

opportunity inherent in a military situation. The more

complex a situation, the less time available, and the

greater the uncertainty of the available information,

the greater the risks (and opportunities) present.

While each of these three dimensions can be examined

independently, considerable insight can be derived

from examining them as a related set. This examination

normally begins as an exercise in hypothesis

generation, but can, as research is accomplished, be

converted into a component of a knowledge base. That

is, as evidence confirming or calling for revising the

key hypotheses is generated, the graphic becomes a

way of conveying known relationships and generating

new propositions about regions or subspaces that have

not yet been examined empirically.

In some sense, one corner of this cube represents the

worst of all C2 worlds—almost no time available, an

enormously complex problem, and considerable

uncertainty about the situation. Past research suggests

that when these conditions exist the decisionmaker has

no choice except to use “best professional judgement”

to match the operational situation to some class of wellunderstood military situations and act accordingly.

(Hayes, 1994). However, decisionmaking theory also

indicates that the wise commander will take short-term

actions designed to create more time and/or more

information and thereby relocate the problem to a

“better” portion of the space. A “risk averse” commander

will clearly attempt this transformation of the situation.

However, a more risk oriented leader may attempt to

cut through the fog of war with decisive action.
152 NATO Code of Best Practice for C2 Assessment

Figure 6.2. Decisionmaking Drivers

The opposite corner of this analytic space, defined as

ample decision time available, limited complexity, and

low uncertainty, provides the ideal situation for

decomposition of the problem and development of

“optimal” military plans. Many innovations in C2

systems are designed to move the situations facing

commanders of friendly forces toward this region.

Indeed, Van Creveld’s analysis of C2 defines it as a

search for greater certainty (Van Creveld, 1985).

This cube also emphasises an imperfectly understood

dimension of C2 systems and the decisionmaking they

imply. That dimension is the speed at which the situation

is changing (the pace of operations) in relation to the

time required to make and implement a military decision

(the speed of the C2 system). Where the speed of the

C2 system is faster, proactive decisions are possible.

When the pace of operations is faster, decisions must

be reactive. The commander who is capable of making

decisions that transform the operation from reactive to
Chapter 6 153

proactive is rare and enjoys vision not only about what

is, but also about what is possible.

This key relationship (pace of operations to speed of

the C2 system) is the driving force behind the observe,

orient, decide, act (OODA) loop and the resulting

guidance to seek to “turn inside the enemy’s C2 loop.”

However, C2 analysts must constantly discipline their

analyses away from assuming that speed alone is a

desirable attribute of a C2 system or organisation.

Making and implementing bad decisions quickly will

result in more rapid failures, not military success. As

is discussed in detail in Chapter 5: Measures of Merit,

multiple dimensions of performance need to be

analysed whenever C2 systems are assessed.

However, this requirement to look at multiple

dimensions in order to assess C2 does not obviate

the value of performing integrated analyses of human

factors and organisational issues.

Conclusions

• Issues of human performance and behaviour

should be incorporated in models used to

analyse issues that require human activity either

in the form of performance parameters or

appropriate sub-models on behaviour.

• Decisionmaking that is rule or algorithmically

based can be modelled directly, but error rates

should be estimated if humans are involved in

the relevant decisionmaking.

• Simple decisions are programmable (with

appropriate error rates), but also require

estimations of when decision would be made.
154 NATO Code of Best Practice for C2 Assessment

• Complex decisions can be treated with “human in

the loop” tools and techniques, but new

techniques are being developed and applied

(see Chapter 5).

• Style of command and decisionmaking should

be considered in C2 analyses that focus on

specific decisionmaking.

• Organisational issues can be decomposed into

constituent elements for analysis.

• Hypotheses or propositional structures are often

the most useful approach to human factors and

organisational issues.

• Integrated analyses involving roles or selected

aspects of a problem space often provide a

cohesive approach to the complexity inherent in

human factors and organisational issues.

• Research in organisations and human factors is

expanding and analysts are advised to consult

the available literature. Experts in this area

should be included on the interdisciplinary C2

assessment teams.

• Operational knowledge of human issues is still

weak in many areas. Systematic effort is

required for organising a consistent program for

experiments on human issues.

Recommendations

• Human and organisational issues are not closed

topics and should be considered early in the
Chapter 6 155

process of C2 analysis when the problem is

formulated and a strategy is adopted.

• Test the impact of established decisionmaking rules

that reflect existing guidance as part of an analysis.

This is an indispensable prerequisite for models.

• The assessment team should explore the quality

of data, information, or knowledge used to drive

the automatable decisionmaking process.

• Early on, the assessment team should establish

working relationships with the potential subjects

of the study but be careful not to allow this to

introduce a bias.

• Human factors and organisational expertise

should be included in all C2 assessment teams;

at least until a decision can be made that they

are not major elements of the analysis.

• Separate human performance issues (e.g.,

stress and fatigue) from cognitive issues (e.g.,

decisionmaking) when possible, but recognise

that they interact.

• Use a checklist (Annex D) and hypothesis-testing

logic for reviewing human and organisational

issues. Remember that human and organisational

issues may interact as do the structural,

functional, and capacity arenas of organisations.

• Integrated analytic tools that focus on key

variables that drive human factors and

organisational issues will often prove useful in

simplifying analysis.
156 NATO Code of Best Practice for C2 Assessment

• Sensitivity analyses are particularly important

when working with human factors and

organisational issues.

• Experiments for testing hypotheses on human

behaviour underlying the C2 analysis are

strongly recommended.

Chapter 6 Acronyms

ATS – Norman’s Activation Trigger Scheme

C2 – Command and Control

CIMIC – Civil-Military Cooperation

CISS – Center for Information Systems Security

CTA – Constructive Technology Assessment

DISA – Defence Information Systems Agency

DSTL – Defence Science and Technology Laboratories

MoM – Measures of Merit

OA – Operations Assessment

OODA loop – Observe, Orient, Decide, Act

OOTW – Operations Other Than War

OR/OA – Operations Research/Operations Assessment

STS – Science and Technology Studies

TA – Technology Assessment

TTP – Tactics, Techniques, and Procedures

VTC – Video Teleconference
Chapter 6 157

Chapter 6 References

Alberts, D. S. (1996). The unintended consequences

of information age technologies: Avoiding the

pitfalls, seizing the initiative. Washington, DC:

National Defense University.

Alberts, D. S., & Hayes, R. E. (1995). Command

arrangements for peace operations.

Washington, DC: National Defense University.

Baeyer, A. V. (2000). Human Factors in militärischen

Führungsprozessen für die simulation [Human

factors in military guidance processes for the

simulation] (B-IK 3103/01). München, Germany:

Industrieanlagen-Betriebsgesellschaft (IABG).

Available from http://www.unibw-muenchen.de/

campus/itis/projekte.html

Baeyer, A. V. (2001, June). Task analysis, training and

simulation for operations other than war

(OOTW). Paper presented at the North Atlantic

Treaty Organisation (NATO) Specialists

Meeting on Human Factors in the 21st Century,

Paris.

Darilek, R. (2001). Measures of effectiveness for the

information age army. Santa Monica, CA: Rand

Arroyo Center. Available from http://

www.rand.org/publications/MR/MR1155/

Depuy, T. (1979). Numbers, prediction and war: Using

history to evaluate combat factors and predict

the outcome of battles. Indianapolis, IN: BobsMerrill.
158 NATO Code of Best Practice for C2 Assessment

Deutsch, R., & Hayes, R. E. (1990). Personality and

social interaction of effective expert analysts:

The issue of cognitive complexity [technical

paper].

Deutsch, R., & Hayes, R. E. (1992). The war story in

warfighting situation assessment and

decisionmaking [technical paper].

Dodd, L. (1995). Adaptive C2 modeling for RISTA

effectiveness studies. Second International

Command & Control Research & Technology

Symposium Proceedings. Washington, DC:

National Defense University.

Hayes, R. E. (1994). Systematic assessment of C2

effectiveness and its determinants. 1994

Symposium on C2 Research and Decision Aids.

Monterey, CA: Naval Post Graduate School.

Hofmann, H. W., & Hofman, M. (2000). On the

development of command and control modules

on battalion down to single item level. In New

information processing techniques for military

systems (pp. 8.1–8.12). North Atlantic Treaty

Organisation (NATO) Research & Technology

Organisation (RTO) Meeting Proceedings 49.

Available from ftp://ftp.rta.nato.int/PubFulltext/

RTO/MP/RTO-MP-049/MP-049-$$TOC.pdf

Jaques, E. (1976). A general theory of bureaucracy.

Portsmouth, NH: Heinemann Educational

Books.

Keeny, R. L., & Raiffa, H. (1976). Decisions with multiple

objectives: Preferences and value tradeoffs.

Chichester, UK: John Wiley and Sons Ltd.
Chapter 6 159

Long term scientific study SAS-017 on human

behaviour representation [Final Report, Chapter

2]. (n.d.). Available, with clearance, from: http:/

/www.rta.nato.int/RDP.asp?RDP=RTO-TR-047

Mandeles, M.D., et al. (1996). Managing “command

and control” in the Persian Gulf war (ISBN 0-

275-952614). Westport, CT: Greenwood

Publishing Group.

Moffat, J. (2000). Representing the command and

control process in simulation models of conflict.

Journal of the Operational Research Society,

51(4), 431-439. Available with registration at

http://www.palgrave-journals.com/jors/

Moffat, J. (2002). Command and control in the

information age—Representing its impact.

Norwich, United Kingdom: The Stationery

Office. Available from www.tso.co.uk

National Research Council. (1998). Modeling human

and organizational behavior—Application to

military simulations (ISBN 0-309-06096-6).

Washington, DC: National Academy Press.

Available from http://www.nap.edu

Noble D., & Flynn, W. (1993). Recognized primed

decisions (RPD) tool concept document.

Vienna, VA: Engineering Research Associates.
160 NATO Code of Best Practice for C2 Assessment

Noble, D., Flynn, W., & Lanham, S. (1993, June). The

TADMUS recognized primed decisions (RPD)

tool: Decision support for threat assessment

and response selection. Paper presented at the

10th Annual Conference on Command and

Control Decision Aids.

Norman, D.A. (n.d.). Organization of action slips.

Psychological Review, 88, 1-15.

Perry, W. L., & Moffat, J. (1997). Developing models

of decisionmaking. Journal of the Operational

Research Society, 48(5), 457-470. Available

with registration from http://www.palgravejournals.com/jors/

Perry W. L., & Moffat, J. (1997). Measuring consensus

in decisionmaking: An application to maritime

command and control. Journal of the

Operational Research Society, 48(4), 383-390.

Available with registration from http://

www.palgrave-journals.com/jors/

Perry W. L., & Moffat, J. (1997). Measuring the effects

of knowledge in military campaigns. Journal of

the Operational Research Society, 48(10), 965-

972. Available with registration from http://

www.palgrave-journals.com/jors/

Poel, I. (1998). Changing technologies. A comparative

study of eight processes of transformation of

technical regimes. Dissertation. Enschede, the

Netherlands: Twente University Press.
Chapter 6 161

Rip, A. (1995). Introduction of a new technology:

Making use of recent insights from sociology

and economics of technology. In Technology

analysis & strategic management. Journals

Oxford, 7(4), 417-430.

Schot, J., & Rip, A. (1996). The past and future of

constructive technology assessment. In

Technological forecasting and social change,

54, 251-268. New York: Elsevier Science.

Schultz, J. V. (n.d). A framework for military

decisionmaking under risks [Prospect theory

related to military decisionmaking].

Sherrill, E. T., & Barr, D. R. (1996). Exploring

relationships between tactical intelligence and

battle results. Military Operations Research, 2(3).

Available to order from http://www.mors.org/

publications/mor/vol02.htm

Van Creveld, M. (1985). Command in war. Cambridge,

MA: Harvard University Press.

Van Lente, H. (1993). Promising technology; The

dynamics of expectations in technological

developments. Unpublished doctoral dissertation,

Twente University Press, The Netherlands.

Available from h.vanlente@geog.uu.nl

1Land warfare with one commander, air warfare with another,

each reporting directly to a joint commander.
163

CHAPTER 7

Scenarios

Before beginning operations you must,

without any indulgence or selfdeception, examine objectively every

step that the enemy might undertake

to thwart your plan, and consider in

each conceivable case what means

are open to you to fulfill your goal. The

more you anticipate the difficulties in

advance, the less surprised you will be

should you encounter them during the

campaign. Besides, you have already

thought about these obstacles

deliberately, and with composure you
164 NATO Code of Best Practice for C2 Assessment

have perceived the means of avoiding

them, so nothing can surprise you.

—Frederick the Great

Plans that assume the likelihood of

one particular world run the risk of

being seriously wrong.

—James A. Dewar, Carl H. Builder, William M. Hix,

Morlie H. Levin from Assumption-Based Planning—

A Planning Tool for Very Uncertain Times

Purpose of Scenarios

T

he report of the NATO Panel 7 Ad Hoc Working

Group on the Impact of C3I on the Battlefield gave

extensive consideration to the role of scenarios in

operations analysis (OA). This chapter builds on that

material to discuss the role of scenarios in Command

and Control (C2) analysis. Figure 2.2 in Chapter 2

shows the role played by scenarios in the overall C2

assessment process. The analysts craft a set of

scenarios to provide the context or environment for

the conduct of the operational analysis. The scenarios

bound the arena of the analysis and are used by the

analyst to focus the analysis on central issues.

Definitions

These definitions are of particular relevance to this

Code of Best Practice (COBP):

Scenario

A description of the area, the environment, means,

objectives, and events related to a conflict or a crisis
Chapter 7 165

during a specified time frame suited for satisfactory

study objectives and the problem analysis directives.

As described in Chapter 4, scenarios consists of four

elements—a context (e.g., a characterisation of a

geopolitical situation), the participants (e.g., intentions,

capabilities of blue, red, others), the environment (e.g.,

natural—weather and manmade—mines), and the

evolution of events in time. In C2 assessments, the

purpose of scenarios is to ensure that the analysis is

informed by the appropriate range of opportunities to

observe the relevant variables and their interrelationships.

Approved Scenario

In order to support the analysis, the use of an approved

scenario is more relevant than to analyse a current

situation or future plans. The impact of all the elements

is simpler to analyse.

In some countries these scenarios are mainly

developed by a Strategic Committee and are designed

to meet the government strategy. The first step is to

implement a steering group, composed of

representatives of the government and members of

the Armed Forces, which has the responsibility to

describe all the commitments or the precise regions

in which any major event may have an impact on the

foreign politic or on the economy. This basic work is

then forwarded to a larger group or a next military level

which shares the responsibility for implementing the

main recommendations in a more strategic field. These

scenarios specify the elements that must be kept in

mind in the generic planning process or in the

operational studies in relation with the use of military
166 NATO Code of Best Practice for C2 Assessment

forces. These scenarios also describe the global threat,

the geographic areas, the political and military

objectives and the level of force commitment.

With this basic approach done, the analyst or his

military counterpart has only to adapt those guidelines,

to compare his study to the scenarios and to propose

the more relevant alterations to his sponsor and his

assessment team (See Assessment Team paragraph

in Chapter 2).

At this stage the decision to keep this scenario, to

diverge or to choose another one will be discussed

among the stakeholders. The products of this process

are called approved scenarios.

After that, at any time the analysts may develop

vignettes using the approved scenarios as a base in

order to focus on a specific issue. These vignettes

can be used as small scenarios to explore a particular

topic. The use of prepared scenarios must not be

considered as a limitation for customers or analysts

because all the data could be adapted by the customer

or the assessment team, the only limitation is to avoid

the re-use of this particular scenario for a study not in

close relation with the original one.

Planning Scenario

A planning scenario is one in which these elements

are defined:

• Time frame of the analysis;

• Geographic area;

• Meteorological environment;
Chapter 7 167

• Political, historical, economic, and social context;

• Mission objectives and constraints;

• Level of threat;

• Friendly forces (and links between each

members of the coalition);

• Adversary forces (e.g., enemy order of battle);

• Neutral or uncommitted forces;

• Non-combatant (non-government organisations

[NGOs], international organisations, etc.) and

other relevant actors; and

• Media (when relevant).

Operational Scenario

An operational scenario contains additional details

especially with respect to threats, orders of battle,

tactics, rules of engagement, courses of action,

deployment, end state and reserves.

Vignette

The term “vignette” is sometimes used for a scenario

that is not approved. The term is also primarily used

for smaller scenarios, particularly as excursions from

the main scenario.

Role of Scenarios in C2 Analysis

In general, the ideal OA is scenario independent. All

relevant factors can be identified and dealt with
168 NATO Code of Best Practice for C2 Assessment

empirically and algorithmically across a range of military

contexts. However, C2 involves human behaviour,

organisations, missions, and other complex

phenomena. Human behaviour is very difficult to put

into equations (see “Human Decisionmaking” in Chapter

6). There is no single linear dimension for organisations

or human issues. Moreover, military missions do not

form simple dimensions. Therefore, for most C2

analyses, the context must be defined. This is the role

for which the analyst defines the scenarios.

The formulation of the original problem dictates the

contents of the scenarios. There are no overall

scenarios that are independent of a specific problem.

Scenarios are never truly generic, but rather are

customised if only by the assumptions built into them.

The boundaries of the scenario space should be

defined in part by the issues unique to the problem

under analysis.

Organisational issues include the involvement of

various levels of military and non-military hierarchies,

including different command levels. This requires that

scenarios accommodate analysis across different

echelons of command. Information processing and the

characteristics of information must also be

accommodated. Human factors include the

decisionmaking process and supporting staff activities.

The analyst will need to design or select scenarios to

address C2 under a broad range of circumstances.

This taxonomy of C2 analysis might include:

• Defense-planning;

• Force structure and organisation;
Chapter 7 169

• Mission analysis;

• Doctrine/tactics development;

• Cost-benefit/effectiveness analysis;

• Training and education;

• Balancing C2 systems and weapon/sensor systems;

• C2 system procurement, which will often require

more detailed, task specific scenarios to cover

the range of relevant system uses; and

• Non-combatant actor’s access to C2 systems

and integration into the information flow, typically

through liaison officers, but increasingly through

a variety of other means.

In essence, the role of a scenario is to define a set of

conditions and restrictions to enable “good” analysis

as well as to create a structure within which the results

of the analysis can be understood and interpreted.

Understanding and Interpreting the

Results of Operational Analysis

The analyst uses scenarios to understand and interpret

the value of OA study results for the focus of the

analysis. The scenarios provide the context in which

the C2 system will be assessed. It should reflect the

scenarios envisaged by the originator of the

requirement for the system. Often artificial constraints

must be introduced into the scenarios, due to cost

considerations, to properly focus the analysis, or both.

The scenario developer must have an appreciation of

the objectives of the simulation, experiment, or
170 NATO Code of Best Practice for C2 Assessment

exercise analysis plan in order to determine the artificial

constraints necessary to facilitate the analysis. The

analyst needs to be aware of scenario assumptions

and artificial constraints.

There are several essential questions in C2 analysis

that should be addressed in the scenario considerations:

• operational benefits of C2 to be translated in

the definitions of Measures of Policy

Effectiveness (MoPE) and Measures of Force

Effectiveness (MoFE);

• required or desired performance thresholds and

nominal Measures of Merit (MoM) values; and

• the impact of an improved volume, accuracy, and/

or quality of information on the final outcome.

Developing and Specifying Scenarios

Prerequisites in Scenario Definition

Several prerequisites are essential before using

scenarios for C2 analysis:

• Approval: the analyst should strive for the

creation of a family of approved scenarios. In

creating a family of approved scenarios, which

reflect the mission objectives and force

capabilities and cover all significant warfare

areas, the analyst facilitates the scenario

development process to a great extent, because

references to basic assumptions and conditions

can be made. This will also increase the validity

of the analysis in the eyes of the client and
Chapter 7 171

facilitate comparison of the results from different

studies and analyses that use the same

approved scenarios;

• Breadth: a scenario should reflect those factors

that are hypothesised to have a significant

impact on C2 issues;

• Capability: a scenario should stress C2

capabilities, including human and organisational

factors (military and/or civilian) where

appropriate; and

• Credibility: scenarios should include logical

assumptions about the environment under analysis.

Scenarios should represent plausible real world

situations. The synthetic scenario environment should

be consistent across OA studies. The scenarios will

gain credibility if a broadly based scenario team is

involved in the process from the outset. This team

should include a variety of perspectives and expertise,

such as:

• OA analysts for defining the required scenario

information, to avoid biasing the analysis by

selecting an inappropriate scenario space, and

for working out the process of framing of the

scenario space;

• Defence concept planners to propose options

and highlight the critical factors;

• Policy makers to ensure that the strategic

decision points and the alternative options for

each of these decision points are clear and

consistent with Defence Policy; and
172 NATO Code of Best Practice for C2 Assessment

• Subject matter experts to credibly explore the range of

possibilities (scenario space) and foster discussions.

C2 Organisation Infrastructure and

Operating Environment

Organisation infrastructure and environment are often

pre-set conditions and not the subject of the study. They

include C2 concepts of operation, decision hierarchy of

the units under consideration, degree of technological

competence relative to that of the adversary,

requirements or objectives placed upon the system in

terms of speed, accuracy, flexibility, etc., and the

impacts of terrain, weather, and adversary activities.

C2 Processes

The scenarios need to provide for the realistic

execution of the C2 processes. These include the span

of control of the various military command levels and

civilian authorities, information management schemes,

information flows, the elements of the decision cycle,

the decision processes (course of action development,

planning, directing), and the communications

processes and capabilities (data update rates,

throughput, reliability, accuracy, etc.). All too often

these issues are characterised by simple performance

indicators and not examined in detail. They may also

be an important subject of the study. These factors

need to be explicitly built into the scenarios.

C2 Systems

Characteristics of C2 systems are directly related to

system improvements. They include system
Chapter 7 173

performance parameters, command and control

information systems (CCIS), data availability,

intelligence functions (fusion, correlation, aggregation,

etc.), surveillance, targeting and acquisition (STA),

communications systems, throughput, and so forth.

Human Factors

C2 studies are complex in nature. One of the

complicating factors is the involvement of human

beings and their interpretation of a situation, order, or

rule of engagement. These factors can be covered by

the “aggregation/de-aggregation” phenomena in the

command chain. Human factors have to be included

in the analysis and modelling activities, but guidelines

on how to integrate the “human in the loop” are partly

defined in the scenarios. (Chapter 6—Human

Decisionmaking discusses these in some detail.)

Miscellaneous

C2 studies usually cover various levels of hierarchy.

However, the nature of C2 issues does not materially

change for the various command levels. The analyst

may need to perform a cost/benefit analysis on the

inclusion of lower level C2 issues in a closed simulation

model. The analyst needs to consider to what extent

these issues (e.g., performance of a logistic information

system) need to be converted to enabling factors (e.g.,

sustainability, operational delays) at the higher levels.

These issues are mentioned to illustrate that in the scenario

definition a great deal of attention is required to ensure

that the scenario enables the proper C2 issue to be

addressed in problem definition. The elements are often
174 NATO Code of Best Practice for C2 Assessment

dependent on each other. For example, some

shortcomings in C2 systems can be compensated for by

alterations in C2 processes. Similarly, inefficiencies in C2

processes can be met by an adaptable C2 organisation.

The relationships between these issues should be

recognised and taken into account in the scenarios.

Approach to Scenario Development

This section describes a framework for the definition

of a scenario. Then, based on this framework, some

specific aspects of actually using scenarios for C2

analysis are addressed.

Scenario Structure

The general scenario framework developed from the

NATO Panel 7 Ad Hoc Working Group on the Impact

of C3I on the Battlefield has been adopted (Figure 7.1).

At the first two levels, a description of the external

factors and the capabilities of the actors, including

national security interests; the political, historical, and

military situation; and the acting assumptions,

boundary conditions, and limitations related to

adversaries, threats, risks, coalition partners, warfare

domains etc. are given. Very often a reference to an

approved scenario will suffice.

At the third level, the mission environment is defined.

Whether it is a generic, virtual geographic environment

or a specific geographical area is not important. What is

essential is that the mission environment be addressed.
Chapter 7 175

Figure 7.1. The Scenario Framework

The intermediate level is the most challenging one. The

actual military problem has to be projected on the

mission, the military forces and capabilities, the civilian

capabilities, and the resources available. A scenario

must be developed by coherently aggregating a number

of components or dimensions with their attached values,

taking into account of the problem formulation. It should

address at least these components:

• Geopolitical situation, including historical aspects;

• Geographical area;

û Availability/usability of civil infrastructure;

û Terrain and climate;

• Political/economical/military objectives;

û Level of violence, type of warfare areas, and

preparation times;

• Mission context and objectives;

û Mission tasks and goals;
176 NATO Code of Best Practice for C2 Assessment

û National contributions and roles within the

coalition;

û Order of battle;

û Doctrines, procedures, (range of acceptable)

rules of engagement, and concepts of operation;

û Temporal factors (e.g., anticipated duration

of operations);

û Desired end states;

• Opposition/threat/risks;

û Adversary forces and their organisation;

û Other actors (neutral and uncommitted

forces, non-combatants, refugees, IOs,

NGO’s, Red Cross, etc.);

û Level of threat and risk;

û C2 structures;

û Interaction between friendly, adversary, and

other information systems; and

û Assumptions/hypotheses/axioms about level of

technology, impact on information defence, etc.

The actual military problem will be placed in the context

of the friendly and adversary military forces involved, e.g.:

• Force organisation, C2 structure, force components;

• Doctrines, tactics, rules of engagements;

• Courses of action;

• Information systems; and
Chapter 7 177

• Logistics.

The framing of scenarios begins with the identification of

the key dimensions relative to the problem being

addressed. The search of these dimensions is delicate

and requires thorough reflection, for instance, on the

results of a structural analysis. Once identified, each key

dimension or factor must be characterised by a range of

possible values or sectors, which enables the set up of

the sector-factor matrix. This sector factor matrix is then

analysed through a morphological analysis (construction

and reduction of the morphological space or space of

“all possible scenarios”), which provides the user with a

set of appropriate scenarios. The scenarios are

dependent upon the problem and the objective of the

study and can range from generic to very specific. The

analyst will select the level of detail required to drive and

focus the model.

Using Scenarios in C2 Assessment

It has already been noted that a well-formulated OA

problem definition, guidelines and directives for how

to approach the analysis should accompany a

scenario. Emphasis in this COBP is given to C2

elements in relation to:

• Mission scope: as stated before, one of the

characteristics of C2 is that it can not be studied

in isolation. C2 is an integrating and enabling

factor. As a consequence there is a tendency to

consider the entire mission, covering all the

arenas connected to it (from logistics to

manoeuvre, from artillery support to close

combat, from security/police issues to refugee
178 NATO Code of Best Practice for C2 Assessment

management). Therefore, the typical mission

scope for C2 analysis will be broad;

• Levels of hierarchy: the C2 chain is not limited to

a special hierarchical level; information and

command flows are running from the lowest

levels to the higher echelons and vice versa. As

a consequence, there is a tendency to cover a

wide set of hierarchy levels in considering a C2

problem. Single layer analyses do not represent

the dynamics of military problems adequately to

answer most C2 issues; and

• Aggregation/disaggregation: the aggregation of

data flows, data fusion in support of intelligence

processes, merging C2 items to more abstract

levels, etc. is difficult but manageable. The

process of deleting, merging, and combining

information is reasonably well understood. The

integration of soft factors (e.g., human and

organisational) is less well understood and makes

the problem more difficult to simulate, study, or

analyse. In particular, the effects of interactions

between functional groups and echelons must be

considered when decomposition is undertaken for

analytic purposes.

Mission Scope versus Levels of Hierarchy

Experience with prior modelling efforts indicates that

the scenario developer needs to understand the

relationships between the scope of the mission, the

hierarchy assigned to conduct the mission, and the

level of detail at which the echelon operates. In general,

as shown in Figure 7.2, the broader the mission scope,
Chapter 7 179

the higher the echelon required to conduct the mission.

Also, higher command levels tend to use information

at higher levels of aggregation, or less detail, than lower

levels. The scenario developer should attempt to

operate in the shaded area of the diagram, matching

the echelon levels to the proper aggregation level and

mission scope.

Figure 7.2 Intersections in Hierarchy

In many complex problems, the analyst is required to

subdivide the problem into smaller parts, perhaps even

using different models or model federations to

represent different levels. The scenario developer

should work in conjunction with the analyst to ensure

that the hierarchical intersections and interactions (e.g.,

organisation and infrastructure, C2 processes, and C2

resources) are properly represented, that the models

are consistent, and their inputs and outputs are

properly linked. Figure 7.3 graphically depicts this
180 NATO Code of Best Practice for C2 Assessment

relationship, with the rectangles X, Y, and Z

representing different models.

Figure 7.3 Segmentation Hierarchy Range

Aggregation/De-Aggregation: Non-Causality

Between C2 Issues At Different Levels

Analysis of C2 issues generally requires assessing

the effects of events and actions across command

levels to determine causality between actions at one

echelon and events at another. For example, the

analyst will want to know if the capability to make faster

decisions at battalion level has an impact at brigade

level or higher. Some C2 items have an impact

throughout the entire hierarchical range, some affect

only one other level, and some are purely local, i.e.,

level-specific. The scenario developer will need to be

aware of the analyst’s requirements in this area in order

to design the proper linkages between events.
Chapter 7 181

Additional Aspects

Additional Areas of Consideration:

• Specification: the purpose of the C2 analysis will

influence the kind of scenarios to be used. There

are no universal generic scenarios for C2

analysis. Some nations have “validated”

scenarios that can be taken off the shelf and

modified to support particular analyses;

• Merging mission operations areas: mission areas

for the various levels of intensity of conflict and

various level of civilians involvement are not

discrete, and it may be necessary to include

elements of more than one type into a scenario;

• Decomposition: sometimes it might be useful to

decompose a scenario into two or more detailed

scenarios that each deal with a certain subset of

C2 issues. This is more or less analogous to the

decomposition into one or more mission

operations areas. It may be necessary to add

some vignettes to look in greater detail such that

all C2 issues of interests are covered and that

the whole range of relevant OOTW is explored;

• Adversary-friendly interaction: the attention given

to adversary and friendly C2 processes should

be balanced in those cases where adversary

activities are germane to the problem, especially

if counter C2 (information defence) is part of the

analytic focus;

• Assumptions and guidelines: the scenarios are

part of the process in developing an analysis (See
182 NATO Code of Best Practice for C2 Assessment

Figure 2.3 in Chapter 2). In the scenario phase, all

scenario assumptions, guidelines, and boundaries

for the study should be revisited. Each study is

executed within the framework of the scenario,

and therefore the study findings are only valid

within the limitations of the various assumptions

and artificial constraints of the scenario;

• Traceability: analysts should understand which

scenario assumptions and/or boundary

conditions are driving factors in the analysis. A

detailed description of past use of scenarios

should be maintained on a national level in order

to avoid duplication. Such a repository should

also contain Verification, Validation, and

Accreditation (VV&A) information on the

scenarios; and

• Awareness: create awareness of the robustness of

the overall conclusions and decisions and be

aware of the degrees of uncertainty in the scenario.

Conclusions

• Using a scenario for C2 analysis is only one part

of a larger analytical methodology. The context

provided by the scenario impacts in other areas,

and the scenario in turn is affected by those

same areas;

•Six prerequisites should be in place before using

a scenario for C2 analysis:

û It should be approved for the assessment;

û It should reflect the factors that have

significant impact on C2 needs;
Chapter 7 183

û It should stress C2 issues;

û It should be militarily credible;

û It should be credible in terms of civilmilitary objectives;

û It should facilitate the design process;

• At least three C2 elements should be reflected in a

scenario in order to make it useful for C2 analysis:

û The C2 organisation and infrastructure,

including human issues;

û The C2 processes;

û The C2 systems;

• Scenario guiding directives should indicate how

the scenario has been used in a hierarchy of

scenarios (interpretation of input and output

events, etc.);

• The actual C2 analysis problem usually will be

broader in scope than OA will allow. Hence

Scenario Analysis in combination with military

and civilian judgement (including lessons

learned) must bridge this gap; and

• Analysts need to use multiple scenarios; no

single scenario is sufficient.

Recommendations

Practice

• Organise a set of scenarios and vignettes that

allow the analysis to cover or sample the

interesting problem space for the C2 analysis;
184 NATO Code of Best Practice for C2 Assessment

• Create a (national) base of approved scenarios

and vignettes reflecting the civil-military

objectives within the national hierarchy of

operations and thus the required spectrum of

military missions including OOTW capabilities;

• Explicitly identify and describe the scenarios prior

to the execution of a study. However, it might be

necessary to revisit the scenario definition during

the conduct of the study;

• Information and hypotheses on threats,

adversary forces, and non-combatants should be

addressed in the scenario;

• Explicitly identify the C2 aspects under

consideration within the problem definition; and

• During the analysis, the key scenario assumptions

should be identified and documented.

Challenges

• Standards for judging the applicability and

accreditation of (existing) models should be

developed; and

• For coalition C2 assessments the scenarios should

be developed or adapted by teams including

representatives from all participating nations.

Chapter 7 Acronyms

C2 – Command and Control

CCIS – Command and Control Information Systems

COBP – Code of Best Practice
Chapter 7 185

IO – International Organisation

MoFE – Measures of Force Effectiveness

MoM – Measures of Merit

MoPE – Measures of Policy Effectiveness

NGO – Non-Government Organisation

OA – Operational Analysis

STA – Surveillance Targeting and Acquisition

VV&A – Verification, Validation, and Accreditation

Chapter 7 References

Almèn, A., DeBayser, O., Eriksson, P., Jarlsyik, H.,

Lascar, G. & Ritchey, T. (1999). Models for

evaluating peace support operations. A FOA—

DGA/DSP /Centre d’Analyse de Défense Joint

Workshop.

Coyle, R. G., & Yong, Y. C. (1996, April). A scenario

projection for the South China Sea. Further

experience with field anomaly relaxation.

Futures, 28(3).

Entretiens Sciences & Défense. (1998). Nouvelles

avancées scientifiques et techniques, tome 1

[New scientific and technical advances, Volume

1]. DGA/CHEAr/D.R.E.E.

Godet, M. (1998). Manuel de prospective stratégique,

tome 1, Une indiscipline intellectuelle, tome 2,

L’art et la méthode [Strategic prospective

handbook, Volume 1. An intellectual
186 NATO Code of Best Practice for C2 Assessment

indiscipline, Volume 2. In The method and art].

Paris: Dunod.

Jantsch, E. (1968). La prévision technologique [The

technology forecast]. Paris: OCDE.

Schwartz, P. (1993). La planification stratégique par

scénarios [Strategic planning by scenario].

Futuribles, 176. Available from: http://

www.futuribles.com

Starr, S. H. (1996). Developing scenarios to support

C3I analyses. Proceedings of the Cornwallis

Group, Pearson Peacekeeping Center,

Clementsport, Nova Scotia, Canada.
187

CHAPTER 8

Methods and Tools

The reasonable course of action in any

use of arms starts with calculation.

Before fighting, first assess the relative

sagacity of the military leadership, the

relative strength of the enemy, the size

of the armies, the lie of the land, and

the adequacy of provisions. If you send

troops out only after making these

calculations, you will never fail to win.

—Liu Ji, 1310-1375, Lessons of War

War is essentially a calculation of

probabilities.

—Napoleon
188 NATO Code of Best Practice for C2 Assessment

If the only tool that you have is a

hammer, you tend to see every

problem as a nail.

—attributed to a Harvard anthropologist

T

he purpose of this chapter is to consider the best

methods for representing Command and Control

(C2) systems, processes, organisations, and their

interaction in order to support assessments of C2 over

the full spectrum of operations, to include Operations

Other Than War (OOTW). The key objective is to

establish an ‘audit trail’ from data or information

collected, through its processing, presentation,

dissemination, and use to the performance of C2

processes and organisation as well as to high-level

measures of their effects on battle or operations

outcome.

This chapter of the revised COBP extends the code to

cover method and tool considerations regarding

OOTW. The material in this chapter is a distillation of

the best approaches and ideas being considered in

current NATO research for representing C2 across the

full spectrum of operations in all models/tools.

Types of Methods and Tools

This chapter covers all tools (simulations or other

quantitative or qualitative techniques), whether used

for analysis, training, or operational purposes that can

be used to assess C2 processes, performance, and

effectiveness. Available methods and tools can be

categorised into four distinct groups:
Chapter 8 189

• Data collection/generation: methods and tools

used to either collect or generate subjective or

objective data for subsequent analysis from live,

virtual, or constructive sources, whether past,

present, or future;

• Data organisation/relationship: methods and

tools used to organise data in some logical way,

or used to establish relationships between data.

These methods and tools, rather than providing

a mathematical solution to problems, tend to be

more qualitative, subjective, and exploratory

techniques based on expert opinion, judgement,

and interaction, whether obtained directly or

through role playing. Although, in some cases,

these tools/techniques may totally solve the

problem at hand, more often they will illuminate

other associated or sub-problems, determine

areas for further analysis, or provide expert input

“data” for more quantitative “solving” tools;

• Solving: methods and tools which have been

typically associated with operations research,

business, mathematics, computer science,

information science, engineering, or

management science which tend to be

quantitative in nature and which usually consist

of techniques providing mathematically derived

solutions, even if the data analysed is subjective

in nature; and

• Support: methods and tools used to collect,

organise, store, and explore typically large sets

of empirical data.
190 NATO Code of Best Practice for C2 Assessment

Table 8.1 provides some recommended methods and

tools that fall into each category. The intent here is

not to provide an exhaustive list or to debate into which

category a specific method or tool should be included,

but to give the range of methods and tools available

to the analyst for C2 assessment. While the range and

scope of potential methods and tools is broad, clearly

the emphasis of this chapter of the COBP, and of the

analysis community at large over recent years, is on

constructive modelling and how best to enhance,

orchestrate, and apply it to the assessment of C2

impacts on battle/operations outcome.

Table 8.1. Some Recommended Tools by Category Type
Chapter 8 191

Issues

The NATO analytic community has had many

challenges in the past analysing the effectiveness of

C2 related systems and determining what it is that sets

it apart from other types of operational analysis. The

added complexity and number of confounded variables

in OOTW make analysis of these operations even

more challenging. The key to the problem, no matter

where on the spectrum of conflict the analysis is

located, lies in making a properly quantified linkage

between C2 Measures of Performance (MoP), such

as communication system delays, C2 Measures of

Measures of C2 Effectiveness (MoCE), such as

planning time, and their resultant impact on higher level

Measures of Force Effectiveness (MoFE) or Measures

of Policy Effectiveness (MoPE), which capture the

effects on battle or operations outcome. These higher

level MoFE/MoPE are required in order to be able to

trade off investment in C2 systems against investment

in combat systems such as tanks or aircraft. At present,

there is no routine way of making this linkage, nor any

one tool that can be applied to generate the required

measures. Hence, all analyses of C2 issues demand

a high level of creative problem structuring and

approach, and selection and application of a range of

available analysis tools, to overcome this challenge.

The range of issues is very broad and challenging.

Particularly challenging are the following issues that

will be addressed in more detail later in this chapter:

• Representation of human behaviour (e.g. rulebased, algorithmic, or “human-in-the- loop”);

• Homogeneous models versus hierarchies/

federations;
192 NATO Code of Best Practice for C2 Assessment

• Stochastic versus deterministic models;

• Adversarial representation;

• Verification, Validation, and Accreditation

(VV&A); and

• The conduct of sensitivity analysis and other

ways of dealing with uncertainty.

Of particular importance for analysis of OOTW:

• Selecting an orchestrated set of tools that

generate the required MoM;

• Scoping the analysis considering tool availability;

• Considering the human dimension in tool

selection early in the process; and

• Ensuring tools selected have the trust and

confidence of the decisionmaker(s).

Representation of Human Behaviour:

Rule-Based, Algorithmic, or “Human-inthe Loop”

In developing methods and tools that represent the

process and performance of C2 explicitly, most

approaches until very recently have been founded on

the artificial intelligence (AI) methods of expert

systems. These represent the commander’s

decisionmaking process (at any given level of

command) by a set of interacting decision rules. The

advantage of such an approach is that it is based on

sound AI principles. However, in practice it leads to

tools which are large, complex, and slow. The decision
Chapter 8 193

rules themselves are, in many cases, very scenario

dependent and, as noted in Chapter 6 – Human

Decisionmaking, human factors and organisational

expertise may be needed on a project team to treat

these issues correctly.

These factors were not a problem when the Cold War

prevailed. There was sufficient time to complete

extended analyses, and one key scenario dominated.

However, in the post-Cold War environment, such

certainties have evaporated. Indeed, uncertainty is now

one of the key drivers of analysis. There is an

increasing requirement to consider large numbers of

scenarios and to perform a wide range of sensitivity

analyses. This has led to a requirement for ‘lightweight,’

fast running tools, that can easily explore a wide range

of scenarios, yet still appropriately represent C2. Some

have begun to explore advanced algorithmic tools

based on Bayesian mathematics, catastrophe theory,

and complexity theory. Such approaches to the

representation of C2 are at the core of a new

generation of closed form constructive simulation

models that are beginning to be used for analysis.

(Moffat, 2002; Moffat, 2000).

Many analyses employ “human-in-the-loop” techniques

in order to ensure realistic human performance or to

check assumptions and parameters. However, “humanin-the-loop” techniques are expensive and require the

inclusion of soft factors and their attendant MoM. The

introduction of “human-in-the loop” introduces a source

of variances and uncertainty. The increased cost,

complexity, and uncertainty of “human-in-the-loop”

methods often requires analysts to limit the use of these
194 NATO Code of Best Practice for C2 Assessment

techniques rather than employ them as the primary

analytical method.

Homogeneous Model-Tools versus

Hierarchies/Federations

In order to build an “audit trail” that traces the

interrelationships among individual C2 systems,

processes, and organisations, as well as their impacts

on mission operational outcomes, there is a need to

represent the key detailed processes involved, such as

the transmission of communications across the

battlefield and the impact of logistics on decisionmaking.

Taking this as an example, the question then arises as

to whether all the transmission media (radio, satellites,

etc.), with their capacities, security level,

communications protocols, etc., should be represented

explicitly, or whether these details should be split out in

a supporting model. Similarly, the details of logistics

could be undertaken as part of the main model or in a

specialised supporting model. Supporting models could

be run off-line, providing sets of input data to the main

model (giving rise to a model hierarchy) or they could

be run in real time interaction with the main model (giving

rise to a model federation). In the off-line mode, the

main model would generate demands on the

communications and logistics systems. The supporting

models would check if these demands could be

satisfied. If not, communication delays and logistics

constraints in the main model would be increased, and

the main model re-run. This would have to be done a

number of times to bring the main and supporting

models into balance. However, such an approach can

generate valuable analytical insights.
Chapter 8 195

Figure 8.1 shows the main model-tool producing (in

addition to its MoFE) a detailed set of dynamic

demands on the communications (such as capacity

required of different communications systems as a

function of simulated time), and logistics processes

(demands for transport and key consumables), in order

to achieve the assessed levels of MoFE. These are

then fed back into detailed model-tools of the

communications and logistics infrastructure. Those

supporting model-tools can then be matched against

the dynamic demand placed on the communications

and logistics infrastructure to the available capacity. If

there is a mismatch, the assumptions in the main

model-tools are adjusted iteratively to bring the two

model-tools into balance. This approach is more

flexible and reactive for a large set of C2 assessments.

However, this approach increases the complexity of

the architecture (number of linked sub-processes,

failure of the sub-model, etc.)

Figure 8.1. Model Linkage
196 NATO Code of Best Practice for C2 Assessment

A similar approach can be applied to concepts of

operation. In some models, it is possible to describe a

concept of operations as a sequence of standard

missions (e.g. attack, defend, move). These missions

can then be analysed to determine the demands they

place on the supporting infrastructures. This can be

tested off-line to see if the infrastructure can cope.

Again, this would have to be iterated a number of times,

but provides an ability to relate, in an understandable

way, the infrastructure capacity to its ability to support

a defined concept of operations (and hence battle

outcome). In addition to the use of such hierarchies of

supporting models in an off-line mode, it is possible to

create real-time federations of such models to

represent, inter alia, combined or joint operations.

Stochastic versus Deterministic Models

Stochastic and deterministic models differ in how they

treat variables. Stochastic models incorporate the

attributes of the probability density (or distribution)

functions associated with model variable into run time

calculations. The actual values of model variables will

therefore differ each time the model is run. Thus, the

output of a stochastic model will yield different results

each time even when a set of inputs is “fixed.”

Deterministic models utilise point estimates for the

values of model variables, and hence for any given

set of inputs the model will produce a single output.

Stochastic and deterministic models each have their

advantages. The selection of which to use or how to

employ each type of model in an assessment depends

upon the nature of the problem and other elements of

the solution approach. Whether the model is time step
Chapter 8 197

driven or event sequence driven will also impact on

the selection of the appropriate models(s).

Deterministic Models

The merits of a deterministic approach are that run-times

are reduced, and there is a single ‘thread’ connecting

the input data and the results, making the analysis of the

tool output potentially easier. The justification for using

deterministic (expected value) tools is usually based on

the assumption that, due to the large number of stochastic

processes involved in combat on higher levels, their

results converge rather quickly to the mean values

obtained from stochastic models.

Stochastic Models

Stochastic models do provide significantly more

information than deterministic tools. They are deemed

to be indispensable for an assessment of the

robustness of the results as well as providing the “data”

needed for risk analysis. These additional benefits

come at the cost of significantly higher run time

requirements and a more complex analysis task.

Chaos theory shows that structural variance (or

‘deterministic chaos’) can occur when sets of decision

rules interact in the simulation of a dynamic process.

Small changes in initial conditions can lead to very

different trajectories of system evolution. Any

simulation model of combat, with a representation of

C2, has to face this kind of problem. The merits of a

deterministic approach are that run times are reduced

and there is a single ‘thread’ connecting the input data

and the results, making analysis of the model output
198 NATO Code of Best Practice for C2 Assessment

potentially easier. However, the representation of the

C2 process (whether using decision rules or not) gives

rise to a number of alternative decision options at any

given moment, and can thus potentially give rise to

such ‘deterministic chaos’. If such effects are likely to

arise, one solution is to use stochastic modelling. The

use of stochastic sampling in the model, together with

multiple replications of the model, gives rise to a

distribution of outcomes, which is much more resistant

to such chaotic effects.

US Army Training And Doctrine Command (TRADOC)

has been experimenting with Deterministic Combat

Models. “A potential alternative solution, when the issue

under study warrants, is to conduct analysis of multiple

runs of a deterministic model where the initial states of

information systems are varied.” (Bailey, 2001).

Representing Adversary Forces

Historically, adversary capabilities and behaviours

were often fully scripted or heavily constrained. This

was more appropriate in Cold War contexts than it is

today. However, it was never ideal for C2 analysis

because the dynamic interaction among friendly,

adversary, and other forces is a critical element of C2

representation. Today, much more robust adversary

representation of operational capabilities and choices

are employed and indeed are necessary. Analysts

must consider not only a range of scenarios, but also

the range of possible adversary actions and reactions.
Chapter 8 199

Verification, Validation, and

Accreditation (VV&A)

VV&A has historically been a challenge for model

development efforts, but is particularly challenging for

C2 modelling. This is due to the variability inherent in

most C2 processes, especially those that involve the

human aspects of information processing and

decisionmaking. The approach to VV&A needs to be

carefully considered, particularly in light of the need to

assess future C2 systems and capabilities in association

with new concepts of operation, new organisational

forms, new doctrine and asymmetrical adversaries.

Selecting an Orchestrated Set of Tools

The natural tendency of an analyst is to simplify a

problem. Part of that simplification is to select a tool,

preferably only one, which will meet the analysis

requirements. In the analysis of C2 of combat

operations, this may be possible if the analysis is

properly scoped. In the analysis of OOTW C2, the

issues are typically too numerous, the variables too

confounding and the scope too broad for one tool to

satisfy all analysis requirements. An orchestrated set

of complementary tools will normally be required.

Scoping the Analysis Considering

Tool Availability

An analyst must always scope the analysis during

problem formulation to enable it to be accomplished

within available resource constraints. During problem

formulation, however, the consideration of available
200 NATO Code of Best Practice for C2 Assessment

tools has typically not been a driving factor. Also, in

the analysis of C2 combat operations this may not be

a problem because selection of the tool(s) to be used

is more obvious based on past experience. However,

for OOTW C2 analysis, the availability of tools, and

their orchestration, requires more consideration early

in the process (i.e. during problem formulation).

Consideration of the Human Dimension

C2, by its very nature, is closely linked to human

behaviour, and its analysis requires careful consideration

and inclusion of the human dimension. Often, as a way

of simplifying the analysis, the C2 assessment team

eliminates these considerations by assuming that human

commanders and their staffs are not affected by their

environment and will always make the best decision, and

the same decision, given the required information. This

is unsafe and not good practice. This has been especially

true for the analysis of OOTW C2, even though this

analysis is often more impacted by the human dimension

than C2 for combat operations.

Ensuring Trust and Confidence in the Tools

Analysts select and apply the tools of their trade based

on what those tools can do for them in accomplishing

their analysis objectives. Over time they become

comfortable with certain tools and the customers for

their analysis also develop a trust in the tools and

confidence that they will produce valid results for them.

For OOTW C2 analysis, given that the tools are more

numerous, must be orchestrated to work together, and

are sometimes unknown or not understood by the
Chapter 8 201

customer, the development of trust and confidence in

these tools is difficult to achieve.

New Methods and Emerging Practices

Given the inherent problems associated with the issues

described previously, a best practice for the application

of analysis tools for C2 analysis is still emerging. This

best practice is described below, first for tools in

general, then for models, both at the model level itself

and at the algorithm level.

Selection of Methods and Tools

The selection of tools to apply to C2 analysis should

be based both on evaluation of the candidate tools

themselves against a set of evaluation criteria and on

consideration of the type of study to be undertaken.

The evaluation criteria for tool selection includes criteria

related to the functionality of the candidate tool and to

the performance of the candidate tool. The following

are established evaluation criteria for tool selection.

They are as applicable to tool selection for OOTW C2

as for combat C2 analysis.

• Functionality-related tool selection criteria:

û Resolution: the level of detail in

representation of entities within the tool;

û Completeness/scope: the extent to which the

tool is able to address analysis issues;

û Functionality: the extent to which the tool

represents the full range of functions;
202 NATO Code of Best Practice for C2 Assessment

û Explicitness: the ability of the tool to explicitly

represent required entities;

û MoM Generation: the ability of the tool to

generate the MoM required;

û VV&A: the determination of whether the tool

has been verified, validated, and/or

accredited for it intended use; (Note: Not all

NATO member nations recognise the term

“Accreditation.” Accreditation here refers to

some form of formal approval to use the

model for the analysis intended.)

• Performance-related tool selection criteria:

û Responsiveness: the amount of time between

request and receipt of information;

û Simplicity: the ease of preparation and use

the tools;

û Preparation/use time: the length of time

necessary to prepare and use the tool;

û Data availability and parameters: the ease in

acquiring or generating the necessary data or

parameters for tool use;

û Interoperability: the ability of the tool to

interoperate with other tools;

û Resource requirements: the amount of

resources (time, personnel, and funds)

required; and

û Credibility: the extent to which the customers

and users accept tool results.
Chapter 8 203

Using Models

The potential for exploiting recent advances in

mathematics in order to create fast running modeltools was noted earlier. Such models have exploited

emerging approaches, such as complexity theory,

chaos theory, catastrophe theory, and game theory in

order to produce a ‘good enough’ representation. They

can be used to complement more complex, detailed

models of the problem area. In many cases, a tailoring

of models, or other tools, will be required to properly

address the analysis issues at hand. (Moffat, 2002;

Moffat, 2000).

Model Federations

A number of new approaches share a key set of

characteristics. First, an object-oriented approach

within the model-tool allows different objects to be

brought together to represent the complete command

process, rather like ‘LegoTM bricks.’ Such a philosophy

also encourages the development of model-tools

based on holistic and evolutionary principles. In other

words, always capture a complete model of the

process, including the parts whose representation is

still unclear. As understanding develops, improve those

parts (or objects) which were rudimentary at the start.

At the next level up, the use of run-time interfacing

allows different model-tools to be brought together to

create a federation to represent the process under

study. This federation then may also have to be

integrated with the use of a mix of tools, which include

techniques other than modelling, to fully address the

study issues.
204 NATO Code of Best Practice for C2 Assessment

Agent-Oriented Modelling

A second key aspect is the description and

representation of the C2 process through agent

modelling and programming techniques. Modelling of

the C2 process as a group of agents, based on artificial

intelligence concepts, favours the capture of the

cognitive nature of command tasks. Agents can be

implemented, in an object-oriented environment, as

either objects (e.g., actor or “applet” type of agents),

or aggregates of objects (coarse-grain agents). Such

agents interact with each other through a messaging

infrastructure. The term “agent-oriented modelling” is

suggested as a way of capturing this idea.

Linking of Performance Model-Tools to

Effectiveness Tools

This third idea, used in a number of NATO countries,

uses a structured hierarchy of model-tools to create

an audit trail from C2 systems, processes, and

organisations through to battle operations outcome.

The idea is to create supporting performance level

model-tools of particular aspects of the process (e.g.,

communications, logistics) which can be examined at

the performance level. These then form inputs to higher

level force on force models. This ensures that the

combat models themselves do not become overly

complex. We have already discussed this in the sense

of hierarchies of support models.

For example, in Figure 8.2, a detailed model of the

intelligence system can be very complex, if we wish to

take account flow of intelligence requirements, tasking,

collection processes, fusion processes, and

intelligence products. In order to analyse the impact
Chapter 8 205

of intelligence, it is important to have all of this detail,

but it does not necessarily have to be represented

explicitly in the main model. A supporting model-tool

which captures all of this detail can be created (or used

if one already exists) in order to produce outputs at

the MoCE level, such as speed and quality of

intelligence. These can then form inputs to the main

simulation model. The main model-tool then takes

these into account in producing its own outputs. These

will now be at the MoFE level. Examples are friendly

casualty levels, adversary attrition, and time to achieve

military objectives.

Figure 8.2. Model-Tool Hierarchy

Scanning Scenario Space

The use of very fast model-tools to scan the overall

space of possibilities and to identify areas of concern

for further analysis appears to give a good balance

between the use of simple and complex modelling
206 NATO Code of Best Practice for C2 Assessment

approaches. Fast models, which are simpler but may

have less analytic depth, allow the analyst to scope

the problem and determine the degree of complexity

a model-tool must represent in order to conduct the

desired level of analysis.

Decisionmaking Process

It is important to have a proper representation of the

decisionmaking process in order to establish the link

from C2 performance through MoE to overall MoFE

and to represent information operations (IO) effects

such as Counter C2 or digitisation of the battlespace.

Representation of the decisionmaking process itself,

however, remains difficult because of the difficulty in

representing human performance, command styles,

and organisational relationships.

Parameter Development Context

Finally, it may become necessary to generate new or

additional data to validate new or existing model-tools

to incorporate C2 factors. This may be especially true

in the case of integrating soft factors into C2 analysis.

Possible methods include field trials, “model-testmodel”, or advanced warfighting experiments. Field

trials are used if uncertainty revolves around

measurable factors that are only observable in the field

or are not reproducible in the laboratory. Model-TestModel (M-T-M) or Model-Exercise-Model (M-E-M) is

used as part of an iterative process to develop and

apply systematically more in-depth and sophisticated

model-tools and, in some cases, more simplistic

model-tools to increase their validity. The original

model-tool is executed, modified based on results of
Chapter 8 207

the test or experiment, and executed again until it has

developed a sufficient representation of a complex

process. Experiments, like advanced warfighting

experiments, are useful in modelling new, large scale,

and complex interactions for which little data or few

validated tools exist. Each approach requires additional

time and resources, and the data sets may not be

validated for some time.

C2 Modelling Guidelines

A number of common ideas have emerged which are

worth consideration for new modelling and tool

developments. Figure 8.3 shows how each of the

following common ideas are represented within an

ideal command and control model:

• Understanding of adversary intent can be

represented by having a number of prescribed

intents or options, which are updated in an

advanced data architecture, or Bayesian way, as

more information becomes available;

• Representing headquarters explicitly in the modeltool allows proper representation of Information

Warfare (IW) effects such as counter C2;

• Explicit representation of the “recognised picture”

within each headquarters (HQ) allows the modeltool to run based on different perceptions by each

individual unit on each side represented. This

allows the effects of aspects such as deception,

shock, and surprise to be explicitly considered;

• Represent information as a commodity. This

consideration is the most critical and difficult to
208 NATO Code of Best Practice for C2 Assessment

implement, but is the foundation for the other

guidelines, as well as for the model itself.

Information should be considered as a resource

that can be collected, processed, and

disseminated. It includes information about

adversary, friendly, and other forces,

considerations such as political-military factors and

rules of engagement (ROE), as well as

environmental information such as weather and

terrain. Information should posses dynamic values

such as accuracy, relevance, timeliness,

completeness, and precision. These values should

in some way affect other activities within the model,

to include, when appropriate, combat functions;

Figure 8.3. Modelling Guidelines

• Represent the realistic flow of information

throughout the operational environment.

Information has a specific source, and that source

is usually not the end user of the information. A

requirement exists, therefore, to move information

from one place to another in the operational
Chapter 8 209

environment. Communications systems of all

forms exist to accomplish this movement. These

systems can be analogue or digital. Information

can be lost and/or degraded as it flows around the

operational environment. The model-tool should

represent the communications systems and

account for these degradation factors as it

represents information flow;

• Represent the collection of information from

multiple sources and tasking of information

collection assets. This guideline applies equally

to adversary, neutral, and friendly and other

force information. For the collection of adversary

and other force information, the model-tool

should represent a full suite of sensors and

information collection systems, and the ability of

these systems to be tasked to collect specific

information. For the collection of friendly

information, this consideration is just as critical.

Knowledge of one’s own capability in combat, as

well as that of the adversary and other forces, is

essential for effective decisionmaking;

• Represent the processing of information.

Information is rarely valuable in its original form. It

usually has to be processed in some way. Typical

processing requirements include filtering,

correlation, aggregation, disaggregation, and

fusion of information. These processes can be

accomplished by either manual or automated

means. The ability, or inability, to properly process

information, and the time it takes can have a

direct bearing on combat operational outcome;
210 NATO Code of Best Practice for C2 Assessment

• Represent C2 systems as entities in the

operational environment. C2 systems perform

information collection, processing, dissemination,

and display functions. They should be explicitly

represented as entities that can be targeted,

degraded, and/or destroyed by either physical or

non-physical means. Additionally, the model-tool

should account for continuity of operations of

critical functions during periods of system failure

or degradation;

• Represent unit perceptions built, updated, and

validated from the information available to the

unit from its information systems. This is a

critical requirement. Each unit should have its

own perceptions, gaining knowledge from

superior, subordinate, or adjacent units only

when appropriate;

• Represent commander’s decisions based on the

unit’s perception of the battlefield. Each unit

should act based on what it perceives the

situation to be informed by its commander’s

intent of mission, goals, constraints, and biases,

not based on ground truth available within the

model. When a unit takes action based on

inaccurate perceptions, it should suffer the

appropriate consequences; and

• Represent IO for each/all combatants. With

information so critical to combat operations

outcome, the model-tool should be able to

represent the deliberate attack and protection of

information, information systems, and decisions.

This applies to all sides represented in the model.
Chapter 8 211

As shown in Figure 8.3, explicit representation of these

information operations elements within a command

and control will facilitate the assessment of human

factors, psychological operations, deception, C2

systems effectiveness, and staff structure issues.

Conclusions

The following conclusions are made regarding the

strengths and weaknesses of current C2 tools/

modelling approaches.

Strengths in Current C2 Tools/Modelling

An assessment of current C2 modelling approaches

employed against the guidelines above show that,

while they may not yet be satisfied, there are some

strengths in the C2 tools/modelling approaches

currently being implemented:

• There is a common understanding of issues. This

has not always been the case. With the inherent

complexity of C2, as well as the challenges in

modelling such a complex subject, there has

been a tendency in the past to ignore the

subject, or just to accept it as something that is

too complex to address. This does not seem to

be the case now. Perhaps through the

emergence of new technologies and through the

work of groups such as the NATO SAS groups,

the analysis and modelling of C2 is now

considered possible. Member nations now seem

to have both an understanding of C2 and its

importance to combat and OOTW operations
212 NATO Code of Best Practice for C2 Assessment

and a common understanding of the modelling

challenges that exist.

• There is wide application of C2 modelling.

Member nations now apply C2 modelling and

analysis to a wide range of issues. These issues

include those associated with investment,

requirement identification, force structuring and

operational support. In all of these areas, there is

understanding of the sensitivity and criticality of

C2 to the proper analysis of combat operations

outcome. Selection of the model-tools to apply to

a particular problem should be based on

evaluation of specific criteria, as discussed

previously in this chapter.

• Although each nation develops its models for

different purposes and tailors their models and other

tools for specific issues, there exists a commonality

of approaches in different nations that serves to

strengthen their collective merits. These common

approaches are an outgrowth of the modelling

technologies now available, but also result from

shared experiences by member nations.

• Most of the progress and success in C2

modelling has been with regard to high-intensity

combat. This is perhaps due to the belief by

many that a high-intensity combat scenario is still

most appropriate for the analysis of combat,

particularly for analysis of primary combat

systems. Progress, therefore, has been focused

on embellishing high-intensity combat analysis

with C2 improvements to models. Unfortunately,

low-intensity combat and OOTW modelling and
Chapter 8 213

analysis have not received the same level of

attention until recently.

• There is wide use of evolutionary development

approaches. After many years of neglect, a

problem as complex and difficult as C2 modelling

requires years of focused research and

development. There are no simple fixes to the

problem. It is evident that member nations

recognise this and are willing to approach the

problem in an incremental manner, applying

evolutionary approaches.

• There is progress in linking and federating

models. Significant progress has been made by

several nations in linking performance modeltools with combat effectiveness models, either

directly or through off-line approaches.

Additionally, creating federations of models

through standard interface protocols has

significantly improved the use, and reuse, of

existing models and has provided a promising

approach for future modelling. The inherent

difficulties in federating models, given today’s

state of the art, must be considered when

contemplating this approach.

• There is progress in modelling “soft factors.”

Several nations have made real progress in

modelling phenomena that have non-physical, or

soft impact, on combat operations outcome.

Among these factors are morale, fatigue, and

training proficiency. These and other soft factors

have increased importance on combat

operations outcome as C2 modelling improves

combat models.
214 NATO Code of Best Practice for C2 Assessment

• Standard interface protocols, data standards, and

other standards either now exist, or are under

development. These standards serve to make this

difficult task easier. Continued development of such

standards is envisioned for the future is essential.

• There is widespread use of Commercial Off The

Shelf (COTS) products. These products are

generally available to all member nations. This

use of COTS has served to help further

standardise individual modelling approaches and

will continue to do so in the future.

• There is application of new information

technologies. New technologies, such as those

supporting animation, have been applied to the

challenge of C2 modelling simulation, and

analysis. Additional technological advancements

will no doubt continue and will be similarly

applied to this problem.

Weaknesses in Current C2 Tools/Modelling

An assessment of current C2 modelling approaches

employed against the guidelines above show that

weaknesses exist in current approaches. These

weaknesses, rather than being enumerated here, are

expressed as challenges below.

Recommendations

It is recommended that analysts take advantage of the

strengths available in current approaches and in the

new methods that are evolving. They should also be

aware of the challenges that must still be resolved and
Chapter 8 215

should attempt to play their part in helping address those

challenges both through study activities and research.

Challenges: OOTW

As discussed previously, recent world events and

current projections call for an emphasis on OOTW.

C2 in these environments can be quite different and

may require fresh C2 tool/modelling approaches to link

C2 to outcome in these environments. The following

challenges with tools-models for C2 Assessment exist:

• Orchestrating a set of applicable tools. Because

there is no one universally accepted tool that will

satisfy OOTW C2 analysis requirements, a set of

tools must be selected, based on evaluation of

potential tools against selection criteria, and

applied to the analysis. Orchestrating a set of

tools that complement the strengths and

weaknesses of each to satisfy the analysis

requirements is difficult. Proper consideration of

assumptions and constraints during tool

selection, and careful scoping the analysis

issues, will help to simplify the orchestration of

tools. Additionally, it must be remembered that

not only the tools, but trained, skilled users of the

tools are required.

• Breadth of tool application. Because of the

complexity of OOTW analysis, the full set of

potential tools should be considered for

application throughout the study process, from

problem formulation through sensitivity analysis.

Additionally, the more subjective nature of

OOTW dictates that tools, heretofore not
216 NATO Code of Best Practice for C2 Assessment

typically applied to the analysis of C2, be

considered for use throughout the study process.

• Relationship of tools to data availability and MoM

generation. Tools must be selected for OOTW

C2 analysis that have necessary data available,

or able to be generated or obtained from the

application of other tools. Reliance on more

subjective data, especially for higher C2

echelons, may be necessary. Additionally, tools

must be selected that generate the MoMs that

will help answer the study issues at hand. The

OOTW C2 analyst will be challenged to be

creative in establishing the strong relationship

between data, tools, and MoM required for a

successful OOTW C2 analysis. It must be

remembered that analysts, not the tools

themselves, answer the analysis issues.

• Consideration of M-E-M or M-T-M. Capitalisation

on testing or training events for C2 analysis

purposes can be highly beneficial, both from a

resource and analytic point of view, but doing so

can be difficult. Such approaches are more

subjective, take more human involvement, and

are inherently more complex than more classical

analysis tools/approaches. The advantages of

having live players/subjects in the analysis,

however, generally outweigh the disadvantages.

These approaches are becoming more

commonplace and more accepted as a

legitimate analysis approach, especially for more

subjective issues such as those associated with

OOTW C2.
Chapter 8 217

• Sharing of tools between different communities.

Given the nature of analysis of OOTW C2 and the

emergence of M-E-M and M-T-M approaches, the

boundaries between the testing, training, analytic,

and operational communities are blurring and tools

once considered for use in only one of these

communities are finding application in another. The

sharing of available tools among these communities

is considered even more appropriate now with the

rise in importance of complex OOTW C2 analysis

requirements. This sharing, however, is difficult due

to differences in terminology, as well as cultural

differences between the different communities.

• Reuse of operational schemas and data models.

OOTW typically involves many more

organisational entities than combat operations,

making interoperability of organisations and

systems critical to successful operations. C2

analysts, therefore, are challenged to ensure

they use, to the extent possible, existing

operational schemas, such as orders and

reports, and data models used for C3I systems

integration, whenever possible to further

standardization and interoperability goals.

• Analysis of architectures. Technical, operational,

and system architectures are sometimes

developed in order to facilitate integration and

interoperability of C3I systems. For NATO,

required architecture frameworks are contained

in the NATO C3 Interoperability Management

Plan (NIMP), Volume II. Considerable challenges

exist in developing and applying analytic tools for
218 NATO Code of Best Practice for C2 Assessment

evaluation architectures, to include analysis of

alternative architectures and their implications.

• Management of customer expectations and

relations. OOTW C2 analysis is so difficult that it

is necessary for the analyst to ensure the

customer for the analysis understands the

inherent difficulty and to attempt to manage the

customer expectations for the analysis. This

includes informing the customer of the tools to be

employed and to gain an understanding and trust

in those tools by the customer. It also implies a

special relationship between analyst and

customer for OOTW C2 analysis. This is even

more critical when the customer is a subject of the

analysis, such as when a commander has his/her

own command analysed.

Challenges: Modelling C2

The following challenges with modelling C2 exist:

• Better representation of cognitive processes. C2

can be incorporated at one level of resolution in

combat tools through representation of the

effects of particular decisions. At another level,

representation of the decision process itself is

desirable. It would enable alternative

decisionmaking styles and the effects of soft

factors such as stress, training level, fatigue, and

morale to be more easily assessed. These

factors become more important as the full range

of IO representation is attempted;

• Long standing challenges associated with both

stochastic and deterministic models. The
Chapter 8 219

advantages and disadvantages associated with

stochastic and deterministic modelling

approaches will remain as C2 modelling

improves. The objective is to select the best

modelling approach for the issue at hand.

Recognising the inherent advantages and

disadvantages, and then capitalising on the

advantages while minimising the disadvantages,

are the challenges;

• Better standard definition of C2 terms. This

challenge has plagued the C2 community for

many years, because of both the scope and

complexity of the subject. This is particularly true

across service boundaries and across the

international community. A standard set of

definitions would greatly simplify the C2

modelling challenge;

• The definition and application of C2 scope. This

challenge, related to the previous challenge, is

especially critical to modellers. The C2 of a

fighter aircraft or a carrier group is very different

from the C2 of an army corps or an army squad.

On the other hand, there are C2 aspects of each

of these combat elements that are similar.

Modelling of C2, however, can be vastly different

in each case. The scope of each modelling

undertaking must be properly considered and

discipline must be applied throughout model

development to focus on the proper scope. Once

the scope is established, a mix of tools may be

required to address the full scope of the analysis;

• Multiple application of C2 model-tools to analysis,

training, and operational requirements. C2
220 NATO Code of Best Practice for C2 Assessment

phenomena are relatively constant whether they

exist within the analytic, training, or operational

environment, and they should be consistently

modelled in each environment. This fact, as well

as the obvious need to conserve expensive

model-tool development resources wherever

possible, leads to the challenge of developing C2

models, or at least component software modules

that can be used to support analysis, training,

and operational requirements;

• Level of resource application to the breadth and

depth of C2 modelling. Because of the large

scope of C2, there has been a tendency by

some to model C2 at great breadth (multiple

applications), at the expense of modelling C2

phenomena at a corresponding depth. In a

constrained resource world, sufficient resources

are not usually available for both. The challenge

is to either apply sufficient resources or to

recognise the shortfall and to level the available

resources across the breadth and depth of the

problem. Models and other tools must, therefore,

be tailored to the extent possible to fit the study

issues being addressed;

• Differences in the level of modelling of friendly

and adversary forces. Many combat models do

not represent adversary forces to the same level

of resolution as friendly forces. In the past, there

may have been good reasons for this. Besides

the obvious resource savings, the lack of C2

representation often precluded further

representation of adversary forces. Valid

representation of C2, to include full play of IO
Chapter 8 221

such as deception and psychological operations,

will require equal representation across both

adversary and friendly forces, as well as any

other supporting or neutral forces in the

simulation. All discussion and recommendations

in the COBP, therefore, are equally applicable to

modelling of adversary forces as it is to

modelling friendly force C2. This represents a

significant challenge to many modelling efforts;

• Continuing lack of “soft factor” representation and

data. As discussed previously, a robust C2

representation in combat models will permit soft

factors to be better represented. The bigger

challenge, perhaps, may not be the modelling

methodology itself, but the acquisition of data to

support it. The effects of such things as stress,

training proficiency, morale, fatigue, and shock, for

example, necessitate new data generation

approaches, which will take some years to

implement. The tasks of VV&A and Verification,

Validation, and Certification (VV&C) are most

severe in their soft factor arena. The certification of

soft factor data, as well as most all C2-related data,

is particularly difficult to achieve. Innovative and

focused C2 data VV&C programs are required;

• VV&A of C2 model-tools and the parameters that

drive them. This is always a challenge for modeltool development efforts, but is particularly

challenging for C2 modelling, due to the variability

inherent in most C2 processes, especially those

that involve the human aspects of information

processing and decisionmaking; and
222 NATO Code of Best Practice for C2 Assessment

• Sensitivity Analysis. The challenges associated

with the proper conduct of sensitivity analysis of

C2 is as great as, or perhaps greater than, that

associated with other analyses. This is because

of the uncertainty associated with C2 itself, and

the relatively immature modelling of C2 that

exists today. Innovative, yet cost-effective

approaches to sensitivity analysis are required.

Chapter 8 Acronyms

ACCESS – Army Command and Control Evaluation System

AI – Artificial Intelligence

C2 – Command and Control

CPX – Command Post Exercise

FTX – Field Post Exercise

HEAT – Headquarters Effectiveness Assessment System

HQ – Headquarters

MAPEX – Map Exercise

M-E-M – Model-Exercise-Model

MoCE – Measures of C2 Effectiveness

MoFE – Measures of Force Effectiveness

MoP – Measures of Performance

MoPE – Measures of Policy Effectiveness

M-T-M – Model-Test-Model

ROE – Rules of Engagement
Chapter 8 223

OOTW – Operations Other Than War

STAFFEX – Staff Exercise

TRADOC – US Army Training And Doctrine Command

VV&A – Verification, Validation, and Accreditation

VV&C – Verification, Validation, & Certification

Chapter 8 References

This chapter is based in part on the results of previous

workshops on models used for C3I systems and

analyses. Additional Useful references include:

Alberts, D. S., & Czerwinski, T. J. (1997). Complexity,

global politics and national security. Washington,

DC: National Defense University.

Allen, P. M. (1988). Dynamic models of evolving systems.

System Dynamics Review, 4, 109-130. Available

to purchase from http://www.albany.edu/cpr/sds/

publications.htm

Bailey, T. J. (2001). Multiple Runs of a Deterministic

Combat Model (Technical Document 01-001).

Fort Leavenworth, KS: U.S. Army Training &

Doctrine Command (TRADOC) Analysis Center.

Bailey, T. J., & Clafin, R. A. (1996). Information

operations in force on force simulations (US

Army TRAC Technical Memorandum TM-0196).

Fort Leavenworth, KS: U.S. Army Training &

Doctrine Command (TRADOC) Analysis Center.
224 NATO Code of Best Practice for C2 Assessment

Bares, M. (1996). Pour une prospective des systèmes

de commandements [For a futurology of the

systems of commands]. Polytechnica.

Bellman, R. E., & Zadeh, L. A. (1970). Decision-making

in a fuzzy environment. Management Science,

17(4), 141-164.

Code of Best Practice (COBP) on the assessment of

command and control (Research & Technology

Organisation (RTO) Technical Report 9AC/

323(SAS)TP/4). (1999). Neuilly-Sur-Seine

Cedex, France: North Atlantic Treaty

Organisation (NATO) RTO. Available from http:/

/www.rta.nato.int/RDP.asp?RDP=RTO-TR-009

Hartley, D. S., III. (1996). Operations other than war:

Requirements for analysis tools research report

(K/DSRD-2098). Oak Ridge, TN: Lockheed

Martin Energy Systems. Available from http://

www.msosa.dmso.mil/ootw.html

Hartley, D. S., III. (2001). Battle modeling. In S. I. Gass

& C. M. Harris (Eds.), Encyclopedia of operations

research and management science (2nd ed.).

New York: Kluwer Academic.

Klein, G. A. (1989). Recognition primed decisions.

Advances in Man-Machine Systems Research,

5, 47-92.

Kroening, D., & Moffat, J. (Eds.). (1999). Workshop on

C3I modeling (AC/323SASTP/9). North Atlantic

Treaty Organisation (NATO) Research &

Technology Organisation (RTO) Meeting

Proceedings 29. Available for order from

www.rta.nato.int
Chapter 8 225

Lauren, M. K. (2000). Modelling combat using fractals and

the statistics of scaling systems, 5(3), 47-58.

Alexandria, VA: Military Operations Research Society.

Lauren, M. K. (2001). Beyond Lanchester: A fractal

based approach to equations of attrition.

Manuscript submitted for publication.

Libicki, M. C. (1997). Defending cyberspace and other

metaphors. Washington, DC: National Defense

University.

Modeling and analysis of command and control (AC/323(SAS)TP/

12). (1999). North Atlantic Treaty Organisation (NATO)

Research & Technology Organisation (RTO) Meeting

Proceedings 38. Available from http://www.rta.nato.int/

RDP.asp?RDP=RTO-MP-038

Moffat, J. (1996). The system dynamics of future warfare.

European Journal of Operational Research, 90(3),

609-618. Available with registration from http://

www.palgrave-journals.com/jors/

Moffat, J. (2000). Representing the command and control

process in simulation models of conflict. Journal

of the Operational Research Society, 51(4), 431-

439. Available with registration at http://

www.palgrave-journals.com/jors/

Moffat, J. (2002). Command and control in the

information age—Representing its impact.

Norwich, United Kingdom: The Stationery Office.

Available from www.tso.co.uk
226 NATO Code of Best Practice for C2 Assessment

Moffat, J., & Passman, M. (2002). Metamodels and

emergent behaviour in models of conflict.

‘Proceedings of the Simulation Study Group Two

Day Workshop, March 2002.’ The Operational

Research Society, UK. Available on

www.orsoc.org.uk

Moffat, J., & Prins, G. (2000, July). A revolution in military

thinking? [Issues from the 1999 Defence

Evaluation and Research Agency (DERA) Senior

Seminars]. Journal of Defence Science, 5(3).

Available by request from ksglasgow@dstl.gov.uk

Moffat, J., & Witty, S. (2001). Bayesian decisionmaking

and military command and control. Manuscript

submitted for publication.

Moffat, J., Witty, S., & Dodd, L. (2002). Bayesian

decisionmaking and the precise use of force. ‘Journal

of Defence Science 7(1).’ Available from

ksglasgow@dstl.gov.uk

Nowak, M., & May, R. (1992, October). Evolutionary

games and spatial chaos. Nature, 359.

Perry, W. L., & Moffat, J. (1997). Developing models of

decisionmaking. Journal of the Operational Research

Society, 48(5), 457-470. Available with registration

from http://www.palgrave-journals.com/jors/

Perry W. L., & Moffat, J. (1997). Measuring consensus in

decisionmaking: An application to maritime command

and control. Journal of the Operational Research

Society, 48(4), 383-390. Available with registration

from http://www.palgrave-journals.com/jors/
Chapter 8 227

Perry W. L., & Moffat, J. (1997). Measuring the effects of

knowledge in military campaigns. Journal of the

Operational Research Society, 48(10), 965-972.

Available with registration from http://

www.palgrave-journals.com/jors/

Pidd, M. (1996). Tools for thinking—Modeling in management

science. New York: John Wiley and Sons.

Richards, R., Neimeier, H. A., Hamm, W. L. (n.d.). Analytical

modelling in support of C4ISR mission assessment

(CMA).

US Department of Defense (DOD) model and simulation

master plan. (1995). http://www.msiac.dmso.mil/

ms_needs_documents/MS_Master_Plan.PDF

Wallace, G. J., & Moffat, J. (1997, January). The use of

Petri Nets to model satellite communications

systems. Journal of Defence Science, 2(1).

Available from ksglasgow@dstl.gov.uk
229

CHAPTER 9

Data

Know the enemy and know yourself; in

a hundred battles you will never be in

peril. When you are ignorant of the

enemy but know yourself, your

chances of winning or losing are

equal. If ignorant both of your enemy

and of yourself, you are certain in

every battle to be in peril.

—Sun Tzu, The Art of War
230 NATO Code of Best Practice for C2 Assessment

Definitions

D

ata are factual information that are organised for

analysis and in a form suitable for machine

processing. Data are usually thought of as anchoring

an epistemological scale with understanding or wisdom

anchoring the other end. Information is data that have

been put into context.

Metadata are “information about information.”

Metadata can describe data as well as metadata,

therefore there are several levels of metadata possible.

An example for metadata on a higher level is the

reliability of the source that data have been derived

from. In general, metadata are documentation of the

attributes of data directly attached to the data, and

therefore can be archived along with the data.

Role of Data

The role and importance of data in C2 assessment is

underestimated by many people, often including the

decisionmakers and the assessment team itself. Figure

9.1, Data Taxonomy, lays out a number of the types

of data including broad categories of sources that will

be of interest to the analyst. The ability to determine

the needed data and the ability to assemble or collect

this data determine the solution strategy. The capability

to obtain or collect the appropriate data:

• Will often reflect the difference between the desired

or “ideal” measures of merit (MoM) and the set of

MoM actually available and used in the assessment;
Chapter 9 231

• Frequently either constrains or determines the

scenario or set of scenarios that are used;

• Is a key or major factor in determining the set of

tools appropriate for the assessment; and

• Often acts as a schedule and cost driver for

the analysis.

Figure 9.1. Data Taxonomy

Reuse of Data

Because of the centrality of data to the assessment,

there is increasing interest in data reuse. While the

amount of data potentially available for a given study

is growing exponentially, the real opportunities for the

reuse of data have proven to be limited because:
232 NATO Code of Best Practice for C2 Assessment

• The rapid change of technical data derailing

performance of systems or sub-systems must be

identified by version and data;

• Assessment teams do not always know that data

exist and have no easy way to find out about

such “legacy” data;

• The data that are available are seldom in an

easily accessible form;

• The conditions under which the data were

collected are not documented;

• The definitions, languages, and measurement

instruments used in different analyses vary

widely; and

• Restrictions arising from security considerations.

Despite these barriers, an effort should be made to

find and reuse data. Sources that should be considered

include “official sources” such as the customer, open

sources such as those available on the Internet, and

prior studies on similar topics. Because the data

needed will seldom be available in a form and format

ideal for the assessment, data engineering is often

needed to gather, organise, and transform the

available data.

Data generated by a given assessment phase may, in

itself, be valuable as an input later in the research or

to other project teams. To enable appropriate

reusability of data, every modification, constraint,

assumption, etc. has to be documented adequately.

To facilitate the reuse of data, it is best practice to use

metadata for this purpose.
Chapter 9 233

In order to facilitate the reuse of data across community

boundaries, alignment of the processes and methods

for data and metadata modelling on the mid term—

resulting in shareable data and metadata models and

a common ontology on the long term—is necessary.

Furthermore, the issue of verification, validation, and

certification (VV&C) for data becomes an issue of

increasing significance. It is good practise to use

certified data whenever possible.

Definition of Data Domains

Data can be directly connected to the other sections

of the Code of Best Practice (COBP) by respective

data domains. These data domains categorise what

the information is about—and what data are

available—as well as the assessment needs of the

study—and what data are required. Some of these

categories are:

• Scenario data: the set of data describing the

scenarios and vignettes;

• Human organisational issues data: the set of

data describing the scenarios and vignettes;

• System performance data: the set of data

describing system performance in different

scenarios; and

• Tool data: the set of additional data used for the

tools that are not covered by any other category.

Hard-wired assumptions belong here as well as

study-specific configuration parameters used for

technical calibration of tools. It is good practice
234 NATO Code of Best Practice for C2 Assessment

to be aware of the hidden data as well as the

input parameters for each tool to be used within

the study, especially when it is planned to build

federations of tools.

Data Sources

Data can be obtained from various sources. The most

common sources include:

• Official sources: sources such as military

databases, governmental data, data owned by

the United Nations, etc. The customer, or other

stakeholders, controls the access to this data or

is at least aware of the existence and structure of

the data. It can be assumed that the customer

will support the analyst in getting access to the

data or a sanitised version, in case the original

data are not releasable to the study.

• Open sources: data sources that are neither

influenced nor controlled by the customer. The

Internet, commercial organisations, as well as

open data sources of non-participating

organisations are examples.

• Legacy study results: data sources derived from

other studies conducted by the operations

analysis and operations research (OA/OR)

community, including political, psychological, and

sociological studies and Command, Control,

Communications, Computers, Intelligence,

Surveillance, and Reconnaissance (C4ISR) tests

and evaluations. They may be the delivered as
Chapter 9 235

the result of a former study as well as having

been an intermediate result.

If needed data are not available from empirical sources,

subject matter experts should be used as sources to

estimate needed values.

Data Classes

Data classes describe the technical aspects of datum

format, datum types, and stage of processing. Data classes

identified in the context of this section are the following:

• Raw data are unprocessed. Raw data may come

from observations of reality or artificial reality, the

product of an instrumented reality, experimental

situations, or selected artificial realities.

• Processed data and information are the result of

transforming one or more raw data elements into

another variable. For example, one or more

radar returns are transformed into a track

(friendly or not).

• Aggregate data and information are properties of

a collection of elements. For example, the

movement of individuals on a battlefield versus

the movement of a squad or a platoon.

• Statistical values1 are assessed on a sample of a

population and characterised by this population.

These include mean, modes, medians, standard

deviations, and kurtosis. Statistics are often used

as parameters in assessments.
236 NATO Code of Best Practice for C2 Assessment

• Derived data and information are outputs from a

formula or simulation model that implicitly incorporates

a set of assumptions. For example, loss-exchange

ratios given sensor and weapons assumptions.

• Intermediate data and information are the

products of one phase or component of the

assessment that provides input to another phase

or component.

Some C2 assessments use assumptions or preset

parameters in place of data or statistics.

Use of Metadata

Data that are collected without adequate

documentation are frequently viewed as suspect. To

avoid having good data discarded due to a lack of

documentation, acceptable community standards for

documentation should be employed. The data must

be clearly described in a manner that is understandable

to a subject matter expert, not directly involved with

conducting the study. The data description must be

robust enough to inspire user confidence in the data.

It is good practice to record these considerations in

the metadata associated with the data to facilitate the

data use and reuse.

Data may be in any of the relevant levels of

measurement—ratio, interval, ordinal, or nominal. In C2

assessments, some significant factors may be nominal.

It is good practise to record all of the assumptions,

constraints, and limitations in the metadata.

In general, every time data are modified or processed

to create new data, appropriate metadata
Chapter 9 237

documentation should be provided to ensure

traceability of results, validation, verification, and

certification of respective data, and reuse of data in

later phases or within other studies. This also allows

dealing with the challenge of multiple instances of data

in different studies.

Data and Problem Formulation

The initial data available will often be vague, uncertain,

incomplete, and contradictory. Analysts usually prefer

data to be sharp, certain, complete, and consistent. It

is necessary to be explicit about the assumptions

inherent in this transformation.

If the assessment team finds it necessary to transform

“soft” data to “hard” data in order to use tools that

require “hard” data, it is best practice to record the

source, perceptions etc. in the metadata associated

with the data.

Obtaining Data

It may be safely assumed that not all of the required

data will be available pre-packaged for the study

question. Some relevant data will likely be submerged

among a pool of irrelevant information.

Not all data will be under military/government control.

Data belong to the stakeholders who are not

necessarily connected to (or even friendly toward) the

customer of the study. A lot of information is available

from open sources. The challenge is to find, organise,

verify, process, and convert it into the data needed.
238 NATO Code of Best Practice for C2 Assessment

The team should be aggressive and persistent in the

pursuit of required data.

The data needed will seldom be available in its raw

form. Often, data have been transformed and

aggregated. When tools have to be applied to derive

data, the derived data should be tagged with

explanatory metadata to record that information in a

form that will facilitate its reuse.

If the data are not available and can neither be

aggregated nor derived from the available sources, it

is good practice to use the knowledge of subject matter

experts to generate the necessary data. Increasingly,

when C2 assessment teams are tasked with exploring

new concepts of operation, empirical data can not be

expected to exist. It is further best practice to document

this in respective metadata and replace such

assumption based data as early as possible with

empirical data, e.g., as soon as another study delivers

the needed inputs. It is recommended to check the

respective study results if subjectively generated data

are replaced, particularly when the study result has

been shown to be sensitive to this data element.

The team needs to know:

• What data are needed/structure of data;

û Preferred data (independent variables within

the MoM);

û Necessary data (to be able to drive the tools);

û Available data (derivation, extraction,

collection, etc.);

• Who owns this data;
Chapter 9 239

• Security issues, possibility of declassifying or

otherwise obtaining release may be another issue;

• Costs to;

û Buy data;

û Collect data (people, time, resources); and

û Generate data.

It is essential to make the value of data clear to all levels

of decisionmakers and operational planers to ensure

that data collection issues are included in all phases of

the study. The data collection and engineering plan, as

introduced in section 4-E-4, as part of the solution

strategies, has to take this into account.

The process of data acquisition is important, but data

acquisition should not be emphasised over data interpretation.

There is an increasing urgent need for data describing

operations over the complete mission spectrum. It is

good practise to collect and exploit operational data

whenever possible. It is therefore recommended to

synchronize respective data collection and engineering

plans, ensuring from the start that the desired data

will be collected and archived appropriately.

The Use of Data within the Study

The archiving of data in retrievable form is essential.

This is necessary both to support the ongoing study

and also to be of value for future study efforts. The

team should establish and adopt process models that

ensure the build-up of archives within a respective

infrastructure. Metadata have a critical role to play in

data archiving and retrieval.
240 NATO Code of Best Practice for C2 Assessment

The discussion of data, and the development of a

community database, must be driven by agreed upon

definitions of data that are sufficient to support the

community. This should be a high priority item.

Data can be described as the “glue” that holds together

the different phases of a study. As the results from

one phase are transported to the next it is unlikely

that both will use identical processes and procedures,

therefore harmonised data formats will allow for

smooth transition and continuity of effort.

The analyst will be faced with data in various forms

and formats. In order to consider all available data,

find it if necessary, and format it in the required manner

(e.g., as an input parameter for the MoM or a model)

data must be stored appropriately. One standard for

storing data is the Information Resource Dictionary

Systems (IRDS) [ISO 1990].

IRDS is a layered database that not only comprises

the data, but also the metadata describing the meaning

of the data, the format, the constraints, the intended

use, the source, degree of certainty, vagueness and

reliability, etc. It comprises data as well as metadata.

A common understanding of the problem between the

customer and the study group as well as among the

interdisciplinary study team is essential for the success

of the study. Therefore, a study glossary—based on a

general and evolving OR glossary (e.g., the NATO

JOINT Pub 1-02 [US DoD 1999])—is needed. The data

definitions stored as metadata have to be aligned with

the definitions found in the study glossary. This provides

a basis for standardised documentation of study results
Chapter 9 241

in a highly reusable form that can be manipulated easily

and reliably, perhaps by automated systems.

Using the right toolkit for the management of data

within the IRDS, such as the Shared Data Environment

(SHADE), can create the initial state of every future

data driven application. (DISA, 1996; Krusche and

Tolk, 2000). The same techniques and tools can and

should be used for information systems delivering the

needed functionality to the warfighter and

decisionmaker. (Tolk, 2000).

Conclusions and Recommendations

Data transcend all phases of an assessment and may

be seen as the “glue” that holds the phases of an

assessment together. Given its importance, resources

must be committed to ensure effective data acquisition,

management, and availability for reuse.

There is an urgent need to agree on standards for

data, metadata, and data management, including the

conditions under which the data are collected, data

element definitions, metrics, etc. It is good practise to

use established standards, where appropriate, such

as the Source for Environmental Representation and

Interchange (SEDRISTM), or to use de facto standards

like Digital Terrain Elevation Data (DTED).

As the data being used today by the analysts will be

the data needed tomorrow by systems engineers,

decisionmakers, and commanders for their operations,

alignment of the standardisation processes and the

respective toolsets as early as possible with the

command and control systems community is good
242 NATO Code of Best Practice for C2 Assessment

practice. A significant first step in such an alignment

would be using the same IRDS [NATO 1999]. A

common C4I and Modelling and Simulation (M&S)

community is needed, to make visions like integrated

alternative course of action analyses become a reality.

Chapter 9 Acronyms

C2 – Command and Control

COBP – Code of Best Practice

C4ISR – Command, Control, Communications, Computers,

Intelligence, Surveillance, and Reconnaissance

IRDS – Information Resource Dictionary Systems

MoM – Measures of Merit

M&S – Modelling and Simulation

OA/OR – Operations Analysis and Operations Research

SHADE – Shared Data Environment

Chapter 9 References

Defense information infrastructure (DII) shared data

environment (SHADE), Capstone document.

(1996). Arlington, VA: U.S. DoD, Defense

Information Systems Agency (DISA). Available

from http://www.disa.mil/coe/srs/srs_base/

srs_csa_PDF/zsrscsa.pdf

The Federation of American Scientists (FAS) Web

Page on Digital Terrain Elevation Data: http://

www.fas.org/irp/program/core/dted.htm
Chapter 9 243

IS10027: 1990 an information resource dictionary

system (IRDS) framework. (1990).

Krusche, S., & Tolk, A. (2000, October). Information

processing as a key factor for modern

federations of combat information systems.

Paper presented at the North Atlantic Treaty

Organisation (NATO) Research & Technology

Organisation (RTO) Information Systems

Technology Panel (IST) Symposium: New

information processing techniques for military

systems, Istanbul, Turkey. Available in PDF

format from http://home.t-online.de/home/

Andreas.Tolk/literat.htm

Lambert, N.J, (1997). “Operation Firm Endeavour:

Campaign Data, Its Collection and Use.”

Presented at the Directorate of Land Warfare

Historical Co-ordination Group Conference,

Trenchard Lines, DGD&D, Upavon, UK, 28-29

April 1997. Also available from the British

Library Document Supply Centre, Boston Spa,

Wetherby, West Yorkshire, LS23 7BQ, United

Kingdom, Reference DSC Shelfmark: 99/

20363.

Mathieson, G. L., Tolk, A., Sundfoer, H. O., & Sinclair,

M. (2001). Data the Glue to Assessment.

Washington, DC: Cooperative Research

Program (CCRP).

North Atlantic Treaty Organisation (NATO)

Consultation, Command and Control Board

(NC3B), Information Systems Sub-Committee

(ISSC), NATO Data Administration Group

(NDAG). (1999). Feasibility study for the NATO
244 NATO Code of Best Practice for C2 Assessment

C3 information resource dictionary system,

(NC3 Information Resource Dictionary System

(IRDS), AC/322SC/5-WG/3WP4). Brussels,

Belgium. Available from http//www.nato.int/

docu/home.htm

The report of the data working group. (1998). In

Simulation technology-1997 (SIMTECH 97, Vol.

III). Alexandria, VA: Military Operations

Research Society.

Sinclair, M., Tolk, A. (2002, April). Building up a

Common Data Infrastructure NATO

Symposium SAS039 on the “Analysis of the

Military Effectiveness of Future C2 Concepts

and Systems” NATO C3 Agency, The Hague,

The Netherlands.

The Source for Environmental Representation and

Interchange (SEDRISTM) Home Page: http://

www.sedris.com

Tolk, A. (2000, October). Functional Categories of

Support to Operations in Military Information

Systems. Proceedings of the North Atlantic

Treaty Organisation (NATO) Regional

Conference on Military Communication and

Information Systems, Zegrze, Poland. Available

in PDF format from http://home.t-online.de/

home/Andreas.Tolk/literat.htm

Tolk, A. (2001, March). Bridging the data gap -

Recommendations for short, medium, and long

term solutions, (01S-SIW-011). Proceedings of

the Simulation Interoperability Workshop Spring

2001, Orlando, Florida. Available in PDF format
Chapter 9 245

from http://home.t-online.de/home/Andreas.Tolk/

011_0301.pdf

Tolk, A., & Kunde, D. (2000, June). Decision support

systems – Technical prerequisites and military

requirements. Proceedings of the 2000

Command and Control Research and

Technology Symposium (CCRTS), Monterey,

CA. Available in PDF format from http://

www.dodccrp.org/6thICCRTS/Cd/Tracks/

Papers/Track6/038_tr6.pdf

US Department of Defense (DOD), Joint Chiefs of

Staff. (1999). Dictionary of military and

associated terms. Washington, DC: Greenhill

Books. Available from http://www.dtic.mil/

doctrine/jel/doddict/index.html

1Statistics is a branch of mathematics dealing with the collection,

analysis, interpretation, and presentation of masses of numerical

data; alternatively, statistics is a collection of quantitative data.

(Webster Online Dictionary).
247

CHAPTER 10

Risk and

Uncertainty

We may at once admit that any

inference from the particular to the

general must be attended with some

degree of uncertainty, but this is not

the same as to admit that such

inference cannot be absolutely

rigorous, for the nature and degree of

the uncertainty may itself be capable

of rigorous expression.

—R.A. Fisher, The Design of Experiments
248 NATO Code of Best Practice for C2 Assessment

T

here are risks associated with the decisionmaker’s

situation that are an inherent part of the analysis.

There are also risks related to the conduct of the

analysis itself. This chapter deals with both of these

sets of risks, focusing upon reducing uncertainty and

other contributors to risk as well as the mitigation of

their effects. Failure to deal effectively with risks will

jeopardise the accomplishment of the goals of the

study, namely to provide high quality decision support.

It should be noted that, by the end of the assessment,

the team will generally be aware that the initial study

plan will have been changed, whereas flaws in the

conceptual model or in the assessment design may

remain hidden for some time from both the analyst

and the decisionmaker.

While the adoption of the guidance contained in this

Code of Best Practice (COBP) will help minimise the

risks, it will not eliminate them. This chapter discusses

a number of issues related to the risks and

uncertainties that the assessment team needs to

explicitly address.

Risk

Risk is commonly defined as the possibility of suffering

harm or loss; in other words, an exposure to harm or

loss.1 This includes an opportunity loss. The difference

between risk and loss is that risk inherently involves

before the fact probability, while loss is an after the

fact certainty. Insurance is about estimating and

covering an exposure of value to uncertainty. Risk often

has a negative connotation, yet “taking risks” can also

be a positive act when a proper balance or trade-off is
Chapter 10 249

made between good and bad outcomes and the

respective probabilities associated with the outcomes.

Perceptions of risk can substantially differ from more

objective assessments of probability and impact. This

is particularly important in command and control (C2)

problems, which have a high sociological content.

Uncertainty

Uncertainty can be generally defined as an inability to

determine a variable value or system state (or nature)

or to predict its future evolution. Uncertainty is inherent

in risk. Even if a person is certain about the possible

outcomes and the probabilities associated with these

outcomes, there is uncertainty about which outcome

will in fact occur. When this situation applies we can

say that we have a known risk. For example, the

possible outcomes and the probabilities associated

with the toss of a coin (and many other popular forms

of gambling) are known. What is uncertain is the

outcome. Hence there is a risk, in this case a known

risk, associated with obtaining a particular outcome.

There are other types of uncertainty, including

uncertainty about the possible outcomes (or their

values) and uncertainty about the probabilities

associated with these outcomes. Uncertainties arise

from other uncertainties, namely uncertainties about

the potential actions of others or what is referred to as

“states of nature” and their associated probabilities.

In C2-related assessments, in particular, perceptions

of uncertainty may substantially differ from a more

objective assessment of lack of knowledge. There are

also a number of uncertainties that are associated with
250 NATO Code of Best Practice for C2 Assessment

the assessment itself. These are discussed later in

this chapter.

Dealing with Risk

There are three basic ways to deal with risk. The first is

to reduce the uncertainty that underlies the risk. The

second is to mitigate risk by developing and selecting

risk-averse strategies. The third is to effectively

communicate the nature of the risk involved.

Reducing Uncertainties

In effect, all methods for reducing the uncertainties

that underlie risk involve the collection or analysis of

information. This is, of course, a major focus of an

assessment. Clearly the assessment team needs to

identify the most important uncertainties—a

combination of the degree of uncertainty and the

consequences of that uncertainty. However, there will

always be some significant residual uncertainties. The

aim of the assessment team, at this point, is to reduce

the risk associated with the study by learning more

about the robustness (or lack thereof) of the study

findings and conclusion. Sensitivity analysis is one

method to accomplish this purpose.

Sensitivity Analysis

The goals of sensitivity analysis are fourfold:

• To establish the regions for which established

results are valid;
Chapter 10 251

• To isolate those factors that contribute most to

the uncertainties that exist and identify the risks

associated with these uncertainties as they

relate to study findings and conclusions;

• To make study results more robust for decision

support by allowing the decisionmaker to see the

immediate consequences as external factors are

changing; and

• To give the decisionmaker a richer understanding

of the decision problem, highlighting the

consequences of limited changes relative to a

solution proposed by the study.

The team should take a three-step approach:

• Identify the variables that are associated with the

greatest combination of risk and uncertainty;

• Perform sensitivity analyses that vary across the more

likely region of key parameters (often the

neighbourhood of the initial estimate) to see how the

result is influenced. A sensitivity analysis investigates

the main region of the output space; and

• Investigate the extremes of the same output space,

focusing most strongly on the negative regions.

The assessment team also partially takes an inverted

approach, looking for possible failures, and then

seeking possible sources of each kind of failure.

Together, these three steps should provide the team

with a better understanding of residual uncertainties

and associated risks. The need for and results of

sensitivity analyses must be stressed in discussions
252 NATO Code of Best Practice for C2 Assessment

with the decisionmakers. This will help avoid the underresourcing of this critical activity.

Uncertainty and Risk in C2

Assessments

Uncertainty and the risks associated with it can not be

totally eliminated in any real world C2 assessment

problem. Moreover, in most real-life problems, one

cannot even identify all of the unknowns. This has

given rise to the term the “unknown unknowns.”

Strategies to minimise risk have evolved to handle

these facts of life. Therefore, even the best possible

assessment approach will result in residual uncertainty

and risks.

In general, assessments should be judged by their

ability to reduce uncertainty so that the decisionmaker

is in a better position (less risk) after the assessment

than before the assessment. Looking at the absolute

uncertainty that remains is not as useful a measure

since it may say more about the nature of the problem

than the success of the assessment.

Dealing with Uncertainty

It is important to treat uncertainty consistently and

explicitly. This allows information from two given

sources or results to be fused, (e.g., by taking the most

precise assessment of each factor from the two

results). Thus the resulting knowledge will be better

than either of the two separate results. On the other

hand, if uncertainty is not treated explicitly or

consistently, the best one can do is to pick the single
Chapter 10 253

result that seems best. This makes it more difficult for

a study to add value to a decisionmaker.

C2 issues are complex, and it is an understatement to

say they are incompletely understood. C2

requirements and solutions tend to depend heavily on

the nature of the operating environment. Thus, a C2-

related study will rely on factors that are imprecisely

determined and change frequently over time. Almost

regardless of the assessment effort, parts of the

problem-space will have been investigated less

thoroughly than would be ideal. Sensitivity analyses

are required to give high quality study results.

C2 assessment problems (particularly in operations other

than war (OOTW) contexts) generally have many

interacting factors. It is unwise to rely upon single factor

sensitivity analysis (i.e., testing sensitivity to one factor

at a time). Multi-factorial experimental design methods

are good practice in such circumstances. (Keppel, 1973).2

OOTW studies typically have less well-formed

quantitative factors and more qualitative factors. The

“softer” nature of these factors makes assessment

more difficult. The types of factor seen in OOTW

problems include:

• Social and political activity impacting the tactical level;

• A strong influence of negotiation and persuasion

as opposed to coercion;

• Non-optimal performance of military capabilities

from a technical perspective due to their poor fit

to the problem;

• Severe rules of engagement (ROE) constraints; and
254 NATO Code of Best Practice for C2 Assessment

• Unclear or evolving goals and objectives.

The team must be aware of the assumptions and

limitations included in models, scenarios, and data

structures, which should be captured in the metadata.

In particular, it should be noted that humans involved

in C2 experiments (as analysts or subjects) always

bring assumptions with them. These need to be

identified and collected to form an audit trail. This is

one place where the project leader’s log of

assumptions and decisions made during the

assessment pays off. Uncertainty over the validity of

these assumptions and limitations provide a source

of uncertainty in study results to which study

conclusions must be made robust.

In C2 assessments, in particular, all aspects of a study

and study-problem may be connected to uncertainty.

Thus, different sorts of uncertainty should be

addressed explicitly at appropriate stages in the study.

Examples of these are:

• Parameter value uncertainty—many of the

parameters and factors in C2 assessment are

difficult to evaluate confidently;

• Model-based uncertainty—i.e., over whether

underlying models are valid and representative;

• Uncertainty of focus—i.e., over whether the

assessment covers all of the important factors

and/or issues (this includes uncertainty of

scenarios); and

• Complexity of uncertain factors (i.e., their

dimensionality)—when a sufficiently complex

factor (e.g., scenarios or future technology) is
Chapter 10 255

uncertain, the team can not expect to overview

the set of all possible true states.

Uncertainty in complex factors, such as scenarios,

should be addressed thoroughly. Even though it is

philosophically impossible to know everything about

a problem, an adequately complete knowledge can

be better assured by explicit use of checklists that

highlight the breadth of factors typically involved in C2

assessments, such as:

• Technology—disruptive uncertainty and

disruptive innovation;

• Organisational use of technology;

• Scenarios, tasks, and nature of operations;

• Data;

• Context or environment of the assessment; and

• Co-evolution of factors (e.g., summarised in

doctrine, organisation, training, material,

leadership, personnel, and facilities

[DOTMLPF]).

Nonetheless, the team should not rely completely on

checklists, but rather complement them with critical

thinking in the specific study context.

Within electronic systems, organisations, and battle

concepts, there is a lot of opportunity for disruptive

technology, producing substantial uncertainty. In

human-in-the-loop experiments there is a greater

danger of bias in subjective judgements.3
256 NATO Code of Best Practice for C2 Assessment

In considering sensitivity analysis, it is important not

to associate it only with statistical variance in parameter

values. Qualitative consideration, such as variation of

model, perspective, or assumptions (i.e., categorical

variations) should also be used to test for and assess

sensitivity. Variations in ranking (ordinal variation) can

also be a powerful tool for sensitivity analysis. Other

analytic tools and constructs relevant to sensitivity

analysis are:

• Non-parametric statistics (Siegal and Castellan, 1988);

• Belief-functions (as an alternative to probability);

• Judgmental uncertainties (Wilson and Corlett, 1990);

• Fuzzy numbers, theories of semiorders, scoring

criteria (Siegal and Castellan, 1998);

• Multi-dimensional scaling; and

• Mathematics applied to non-ordinal scales.

Semiorders constitute the intermediate level between

ordinal information and value. They apply to many

fields but in this context especially to scored data or

preference data—in that account is taken of the

intervals of imprecision around the measuring systems

used. Analysts should be aware of the thresholds

associated with the collected data above which

differences can be legitimately distinguished to

produce values. This idea is very useful when the

measurements cannot be repeated (as in statistical

theory) (Prilot and Vincke, 1997).

Keeping C2 assessment rigorous and robust in the

face of the many uncertainties and complexities of the
Chapter 10 257

subject matter, as well as the need to use a rich

combination of methods, can be difficult. Again, it is

good practice to use checklists and structured

appraisal, in this case to maintain an objective view of

study rigour. The choice of checklist or appraisal

structure can depend upon personal preferences, but

Annex F lists a number of structures, which have

proved useful in the experience of the nations

contributing to this COBP. These include:

• Repeatability, independence, grounding in reality,

objectivity of process, uncertainty and

robustness (RIGOUR); and

• Strengths, weaknesses, opportunities, and

threats (SWOT).

Where an assessment uses experimentation or

observation of exercises it is important to identify

independent and dependent variables and, for the

former, which are controllable, which are measurable,

etc. Figure 10.1, for example, illustrates the variety of

variables that need to be considered when studying

decisionmaking.

A thorough understanding of the variables of a study

is essential for effective treatment for uncertainty.
258 NATO Code of Best Practice for C2 Assessment

Figure. 10.1. Illustration of the Variety of Variables

Relevant to Decisionmaking

Risk-Based Analysis

In cost-benefit or cost-effectiveness analyses there has

historically been a tendency to focus on the single most

likely value (its expected value) for each input factor

(including scenario and course of action) and to “solve”

the problem based upon these point estimates. This

can lead to fragile solutions, which do not provide

decisionmakers with help in dealing with the

uncertainties and associated risks inherent in the real

problem. A risk-based approach can overcome some

major pitfalls by adding a focus on the multiplicity of

possible outcomes and opening up the possibility of

richer solutions involving portfolios of action that produce

robustness rather than narrow optimality. Portfoliobased solutions can be associated with cost-benefit

approaches, but this has not been common in practice.

The subject of C2 assessments is typically both highly

uncertain and opaque in nature. The team should,
Chapter 10 259

therefore, expect a complex and partly hidden set of

risks to C2 studies and C2-related decisions. It is

recommended that serious efforts be made to illuminate

the risks, and it is good practice to include an explicit

risk-based analysis in the study and in study planning.

Different people have different worldviews and different

approaches to risk taking. Analysts should seek to find

out about how risks are traded for expected gain by

the study sponsor so that their worldview can be

appropriately represented in the assessment.

Risk-based analysis needs metrics for risks and failure

as well as success and benefits, which means that

one needs a way of expressing various levels of all of

these dimensions.

In C2 assessments, analysts need to be particularly

alert to the possibility of chaotic behaviours arising from

dynamic interactions. Human and organisational

factors are particularly prone to this type of instability.

A sound and explicit treatment of boundaries and

system definitions during problem formulation is

essential to managing this aspect of the assessment.

Holistic systems thinking and complexity-based

analysis may be needed for this purpose.

When dealing with problems involving human

decisionmaking the analyst must be aware of the

diversity of courses of action that are possible as the

situation evolves. The analyst must ensure that these

courses of action are represented in a way that allows

for the possibility of a discontinuous set of possible

future states. If wide divergence in course of action

selection is possible over the timescales under

consideration, then the treatment of scenarios may
260 NATO Code of Best Practice for C2 Assessment

be particularly problematic and may require explicit

consideration under the risk and sensitivity heading.

Managing Study Risk

C2 assessments are inherently complex. They often

contain poorly understood study problems. These

factors enhance the level of risk in the design and

conduct of the assessment.

It is good practice to try to make a complete list of

risks and then treat them in appropriate detail. A topdown approach may be useful in assuring a certain

degree of completeness. Independently of the depth

of each concrete study, it is good practice to use a

risk perspective to explicitly assess the robustness of

a conclusion or recommendation. If possible, one

should try to keep track of which mechanisms underlie

each risk, the probability that it will occur and how it

can be mitigated, then consider cost of mitigation and

cost of risk impact, before taking a cost-benefit

approach in managing the study risk level.

C2 problems are often weakly bounded. There is a

particular risk associated with problem formulation and

the identification of factors. Annex F lists a number of

checklists, which have proved useful in ensuring an

adequately complete treatment of the multiple risk areas.

The Generic Risk Register

The generic risk register for C2 assessment (GRR) is

a good starting point for a risk-based analysis of a

study project (not the decision problem supported by

the study). The lists of risks, consequences of impact,
Chapter 10 261

and mitigation recommendations are directly derived

from the COBP. They are, therefore, generic in their

form and should only be taken as a starting point for a

project-specific risk analysis. The GRR has

functionality, which allows it to be used as a basic tool,

or the list of risks can be copied into a more elaborate

tool used for risk handling in the project.

An illustrative example is the case study undertaken

by the SAS-026 study group, when a brief journey of

only an hour through the generic risk register turned

out very useful. Although the study team was well

aware of the advice and possible pitfalls in advance,

explicitly addressing them with a risk perspective led

to the recognition of two significant flaws:

• The low number of planned iteration had the

potential to lead to a risk of an inefficient and

unfocused study with possibly misleading

results; and

• The relatively narrow selection of methodological

approaches entailed a risk of misleading

conclusions. There could be important

consequences of varying the C2 system that

were not reflected in the study, and the possibly

biased representation would represent a hidden

flaw in conclusions.

The example illustrated both the usefulness of making

the risk assessment explicit, since these problems were

actually known to all participants prior to the risk

management session, and also illustrated that even

surface scratching (as was the case here) may lead to

significant results. It is, therefore, advisable not to skip

risk analysis even when time and resources are limited.
262 NATO Code of Best Practice for C2 Assessment

Communication of Risk and Uncertainty

The high level of uncertainty (and hence risk) in C2

problems and their assessment mean that the

communication of risk and uncertainty to study

customers, sponsors, and stakeholders is of particular

importance. The value of a quality assessment is that it

provides decisionmakers with the evidence they need to

make better decisions. The nature and quality of evidence

required depend upon the decisionmaker’s approach to

and tolerance for risk-taking and his/her level of prior

knowledge of the problem area being assessed.

Communication is about giving the receiver of a

message a right impression, not about formulating a

statement that is formally correct on its own. This might

seem obvious, but in communicating uncertainties and

risk, it should be given particular attention, since the

complexities of the subject, human limitations in

reflecting on uncertainty, and the lack of a common

set of concepts (and also hidden agendas) often will

make communication far less than perfect.

As discussed earlier, some uncertainty can be reduced

by analysis. However, some uncertainty is inherent in

the problem and needs to be exposed to the

decisionmaker. A failure to do this can lead to false

confidence in study conclusions. C2 assessments, in

particular, will contain many areas of unresolvable

doubt and uncertainty. These should be openly and

honestly communicated to decisionmakers to avoid

misinterpretation of study conclusions. Support to

decisionmaking under uncertainty is a vital

complementary activity to C2 assessment.
Chapter 10 263

Different ways of framing results and uncertainties may

strongly influence the way results are perceived. This

should be considered thoroughly to assure compliance

with ethical standards. One should be aware that

stakeholders (including customers) might have a tendency

to gloss over or alternately over-focus on uncertainties.

An analyst should take care in communicating an objective

impression of risks and uncertainty.

Limitations and shortcomings in a study are a crucial

part of the study result and should be communicated as

effectively as possible. This enables alignment of study

results and background knowledge on the problem.

Human ability to understand and reason on uncertainty

is limited. (Kahnmann et al., 1982). These limitations

should be given particular attention when

communicating risk and uncertainty to stakeholders

and decisionmakers.

People with different backgrounds will have different

concepts of uncertainty (e.g., people without some

mathematical background won’t necessarily intuitively

understand Bayesian concepts). Thorough dialogue

may be needed to find a common language.

Visualisation techniques will be helpful in this regard

since they are usually more powerful than verbal

reference to abstract concepts.

One should be careful not to overwhelm an audience

with details on uncertainties and possible

shortcomings. However, a continuing dialogue about

uncertainty will facilitate a common understanding.

Also, the analysis team should be aware of the

possibility that residual uncertainties may make it

impossible to draw robust conclusions.
264 NATO Code of Best Practice for C2 Assessment

Conclusions

The explicit treatment of risk and uncertainty is best

practice in all studies and is of particular importance

in C2 assessment. A variety of dimensions and aspects

of C2 assessments carry risk and uncertainty,

particularly because they are liable to include complex,

interacting factors. Even when study resources are

limited, it is best practice to include both an assessment

of most likely outcome (result), to do sensitivity

analyses looking for other likely outcomes, and to take

a risk-based approach looking for the more extreme

possible outcomes (in particular failures).

The use of checklists is recommended to ensure a

rigorous treatment of risk and uncertainty. The best

choice of checklist depends upon personal preference,

but a number of examples are presented that have

been found useful by the nations contributing to this

COBP. Additionally, the GRR has proved useful as an

aid to C2 study risk management.

Chapter 10 Acronyms

C2 – Command and Control

COBP – Code of Best Practice

DOTMLP – Doctrine, Organisation, Training, Material,

Logistics, Personnel

DOTMLPF – Doctrine, Organisation, Training, Material,

Leadership, Personnel, Facilities

GRR – General Risk Register
Chapter 10 265

METT-TC – Mission, Enemy, Troops, Terrain, Troops,

Time, and Civil considerations

OOTW – Operations Other Than War

PESTLE – Political, Economic, Social, Technological,

Legal, and Environmental

RIGOUR – Repeatability, Independence, Grounding in

reality, Objectivity of process, Uncertainty, and Robustness

ROE – Rules of Engagement

SWOT – Strengths, Weaknesses, Opportunities, Threats

Chapter 10 References

Kahnmann, D., Slovik, P., & Tversky, A. (Eds.). (1982).

Judgement under uncertainty: Heuristics and

biases. New York: Cambridge University Press.

Keppel, G. (1973). Design and analysis—A

researcher’s handbook (ISBN0-13-200030-X).

Englewood Cliffs, NJ: Prentice-Hall.

Kruskal, J. B., & Wish, M. (n.d.). Multidimensional

scaling. In E. M. Uslaner (Ed.), Quantitative

Applications in the Social Sciences (Series 07–

011, ISBN 0-8039-0940-3). Beverly Hills, CA:

SAGE Publications.

Mathieson, G. L., Tolk, A., Sundfoer, H. O., & Sinclair,

M. (2001). Data the glue to assessment.

Washington, DC: Cooperative Research

Program (CCRP).
266 NATO Code of Best Practice for C2 Assessment

Pirlot & Vincke. (1997). Semiorders: Properties,

representations, applications (ISBN 0-7923-

4617-3). New York: Kluwer Academic.

Siegal & Castellan. (1998). Non-parametric statistics

for the behavioral sciences (2nd ed., ISBN 0-

07-100326-6). New York: McGraw-Hill.

Wilson, J., & Corlett, E. (Eds.). (1990). Evaluation of

human work: A practical ergonomics

methodology (2nd ed.). London: Taylor &

Francis.

Winer, B. J. (1962). Statistical principles in

experimental design. New York: McGraw-Hill.

1This interpretation of risk is used throughout this section.

However, other common definitions have been included at Annex

E as an aid to common understanding.

2Tom Lucas’ work in Project Albert.

3These types of experiment are typically unrepeatable due to

resource constraints.
267

CHAPTER 11

Products

We are what we repeatedly do.

Excellence, then, is not an act, but

a habit.

—Aristotle

T

he purpose of study products is threefold: to

communicate results to sponsors and

stakeholders; to provide a lasting record of what went

into the planning; and to establish credibility within the

technical community. Verbal communication and

progress reports may be necessary, especially for a

short study, however a thorough written record is

essential to the credibility and longevity of study results.
268 NATO Code of Best Practice for C2 Assessment

Study products that are delivered to the customer

generally include a study plan, periodic status/progress

reports, and a final report. Several other products may

be produced and delivered to the customer. Some

study products are created and maintained primarily

for internal study support. These products are not

unique to C2 OOTW studies or to general C2 studies.

Products that are typically produced from a study include:

Study Plan

The study plan described here is a subset of the Study

Management Plan of section 4-E. It generally includes

the initial problem formulation and solution strategy,

emphasizing a general understanding of the problem,

deliverables, budget, time line, and solution approach.

The study plan includes at a minimum:

• Statement of the problem (problem

formulation)—“the what”;

• Solution strategy “the how and the when”;

û Tasks and their relationships; and

û Milestones.

Periodic Status/Progress Reports

Periodic status reports describe the activities of the most

recent period and the expected activities of the next

period, and link the activities to the tasking statement.

Status reports also contain cost information and track

adherence to the planned schedule. One of the most

important sections of the status report is the “Problems
Chapter 11 269

Encountered” section. This section should include

technical problems, budgetary problems, and most

importantly problems relating to sponsor/assessment

team relations. These reports may be delivered:

• To the sponsor and other stakeholders; and

• To the peer review team.

Final Report

The final report contains sections that address the following:

• Objectives (customer question and the

problem formulation);

• Scope and assumptions;

• Approach (solution strategy);

• Findings/conclusions (with caveats);

• Recommendations (optional);

• Future challenges (optional);

• Appendices;

û Data collection instruments or discussion of

instrumentation (optional);

û Data dictionary (optional);

û Data (optional);

û Glossary of terms/acronyms; and

û References.
270 NATO Code of Best Practice for C2 Assessment

The findings/conclusions and recommendations (if

present) should address each of the objectives.

The final report should be produced as an archivable

document that can be readily accessed by the

community. This means that the document is produced

in electronic form using commercial standards. Ideally

these documents should be made available through

web sites. However, it is recognised that security

considerations and language will limit the availability

of many documents. Briefings are a useful form of

communication. It is desirable that a briefing

accompany the more formal final report. However,

when only a briefing is produced, it must be annotated.

Command and control (C2) problems tend to be multidimensional and highly complex. Therefore, they pose

unique challenges to the assessment team in

communicating the result effectively to the

decisionmaker. There are several steps that can be

taken to facilitate this communication. First, if the product

is an architecture, it is recommended that the templates

developed by community to depict alternative

architectural perspectives (e.g., operational, system,

and technical architectures) be employed. Second, it

has proven useful to employ “stop light” charts (red,

yellow, and green) to characterise measures of

performance/effectiveness. However, such techniques

are useful only in conveying qualitative insights. A

tendency to rely heavily upon this presentation

technique may result in assessments that do not drive

to quantitative results. Third, there is a need to develop

and employ visualisation techniques to capture the full

richness of the insights, particularly risk and uncertainty

(e.g., depicts the distribution rather than just the
Chapter 11 271

statistical) that are derived in assessments. Preliminary

research is underway in this area. It needs to be

extended and translated into application.

Because C2 data are rare, every effort should be made

to retain the data and make it available to other

recorders. Sometimes others will require “sanitising”

the data to prevent anyone knowing which units or

exercises produced it. When archiving data the

metadata labels that identify the conditions under which

it was collected should also be preserved.

A peer review process is an essential part of producing

a final report. It should begin with the preparation of

the study plan and continue throughout the life of the

assessment. Draft products must be provided to

reviewers in ample time for them to review and

comment on the product and for the team to reflect

their comments. A failure to institute an adequate

review process can compromise the quality, credibility,

and utility of the assessment.

Other Delivered Products

Several products may be created and delivered,

depending on the needs of the project:

• Description of the assessment participants

(including assessment team) and their

relationships (defined in 2-B);

• Description of the budget and time constraints to

provide context information for future studies;

• Human and Organizational Factors Checklist;

• Scenario details
272 NATO Code of Best Practice for C2 Assessment

• Video and audio presentations; and

• Created models, spreadsheets, decision support

tools, etc.

• Simple models and tools (EG for the sponsor and

other parties to interactively explore interrelationships

between the variables of the study);

• Experimentation Campaign Plan (if the C2

Assessment makes use of a series of linked

events such as seminars, wargames, command

post exercises (CPX), field training exercises

(FTX) etc).

Other Products

In addition to study products that are delivered to the

sponsor there are a number of products that best

practice demands are produced and maintained during

the course of a study. These include:

• A project journal;

• Study Management Plan (defined in 4-E)

• Data collection plan;

• Data analysis plan; and

• Study Glossary

Conclusion

A study is generally appraised based on the quality of its

study products. This Code of Best Practice aims to

highlight important areas that will improve both the
Chapter 11 273

assessment process, and the quality, longevity, and utility

of the study products. The goal is to make the state of

practice one and the same with the state of the art.

Of all the communities available to us

there is not one I would want to devote

myself to, except for the society of the

true searches, which has very few

living members at any time...

—Albert Einstein
A-1

ANNEX A

References

Adelmam, L. (1992). Evaluating decision support and

expert systems. Chichester, UK: John Wiley

and Sons Ltd. [Chapter 5]

Alberts, D. S. (1996). The unintended consequences

of information age technologies: Avoiding the

pitfalls, seizing the initiative. Washington, DC:

National Defense University. [Chapter 1]

[Chapter 6]

Alberts, D. S., & Czerwinski, T. J. (1997). Complexity,

global politics and national security.

Washington, DC: National Defense University.

[Chapter 8]

Alberts, D. S., & Hayes, R. E. (1995). Command

arrangements for peace operations.

Washington, DC: National Defense University.

[Chapter 1] [Chapter 6]

Allen, P. M. (1988). Dynamic models of evolving

systems. System Dynamics Review, 4, 109-

130. [Chapter 8] Available to purchase from

http://www.albany.edu/cpr/sds/publications.htm

Almèn, A., DeBayser, O., Eriksson, P., Jarlsyik, H.,

Lascar, G. & Ritchey, T. (1999). Models for

evaluating peace support operations. A FOA–
A-2 NATO Code of Best Practice for C2 Assessment

DGA/DSP /Centre d’Analyse de Défense Joint

Workshop. [Chapter 7]

Baeyer, A. V. (2000). Human Factors in militärischen

Führungsprozessen für die simulation [Human

factors in military guidance processes for the

simulation] (B-IK 3103/01). München, Germany:

Industrieanlagen-Betriebsgesellschaft (IABG).

[Chapter 6] Available from http://www.unibwmuenchen.de/campus/itis/projekte.html

Baeyer, A. V. (2001). Task analysis, training and

simulation for operations other than war

(OOTW). Paper presented at the North Atlantic

Treaty Organisation (NATO) Specialists

Meeting on Human Factors in the 21st Century,

Paris. [Chapter 6]

Bailey, T. J. (2001). Multiple Runs of a Deterministic

Combat Model (Technical Document 01-001).

Fort Leavenworth, KS: U.S. Army Training &

Doctrine Command (TRADOC) Analysis

Center. [Chapter 8]

Bailey, T. J., & Clafin, R. A. (1996). Information

operations in force on force simulations (US

Army TRAC Technical Memorandum TM0196). Fort Leavenworth, KS: U.S. Army

Training & Doctrine Command (TRADOC)

Analysis Center. [Chapter 8]

Bares, M. (1996). Pour une prospective des systèmes de

commandements [For a futurology of the systems

of commands]. Polytechnica. [Chapter 8]
Annex A A-3

Bellman, R. E., & Zadeh, L. A. (1970). Decision-making

in a fuzzy environment. Management Science,

17(4), 141-164. [Chapter 8]

Bijker, W.E. (1990). The Social Construction of

Technology. PhD Thesis University of Twente,

Eijsden, The Netherlands.

Bornman, L. G., Jr. (1993). Command and control

measures of effectiveness handbook (C2 MoCE

handbook) (TRAC-TD-0393). Fort

Leavenworth, KS: U.S. Army Training &

Doctrine Command (TRADOC) Analysis

Center. [Chapter 5]

Brown, R. V., Kahr, A. S., & Peterson, C. (1974).

Decision analysis: An overview. New York: Holt,

Rinehart and Winston. [Chapter 5]

Canadian Department of National Defence Document

B-GG-005-004/AF-023. (1999). Civil-Military

cooperation in peace, emergencies, crisis and

war. [Chapter 5] Available on-line at http://

www.dnd.ca/eng/index.html

Candan, U. and Lambert, N. J. (2002). Methods used in

the Analysis and Evaluation of the Immediate

Reaction Task Force (Land) Command and

Control Concept. TN 897. NATO C3 Agency, PO

Box 174, 2501 CD, The Hague, The Netherlands.

Checkland, P. (1999). Systems thinking, systems

practice. Chichester, UK: John Wiley and Sons

Ltd. [Chapter 3]
A-4 NATO Code of Best Practice for C2 Assessment

Cleman, R. T. (1990). Making hard decisions: An

introduction to decision analysis. Pacific Grove,

CA: Duxbury Press. [Chapter 4]

Code of Best Practice (COBP) on the assessment of

command and control (Research & Technology

Organisation (RTO) Technical Report 9AC/

323(SAS)TP/4). (1999). Neuilly-Sur-Seine

Cedex, France: North Atlantic Treaty

Organisation (NATO) RTO. [Chapter 8]

Available from http://www.rta.nato.int/

RDP.asp?RDP=RTO-TR-009

Coyle, R. G., & Yong, Y. C. (1996). A scenario

projection for the South China Sea. Further

experience with field anomaly relaxation.

Futures, 28(3). [Chapter 7]

Darilek, R. (2001). Measures of effectiveness for the

information age army. Santa Monica, CA: Rand

Arroyo Center. [Chapter 6] Available from http:/

/www.rand.org/publications/MR/MR1155/

De Maio, J. (Ed.). (2000). The challenges of

complementarily [Report on the fourth

workshop on protection for human rights and

humanitarian organisations]. Geneva,

Switzerland: International Committee of the Red

Cross. [Chapter 3]

Defence Science and Technology Laboratory (DSTL).

(2000). A guide to best practice in C2

assessment: Version 1. Farnborough, United

Kingdom. [Chapter 3]

Defence Systems. (1982). Theatre headquarters

effectiveness: Its measurement and relationship
Annex A A-5

to size, structure, functions, and linkages; Vol.

1: Measures of effectiveness and the

headquarters effectiveness assessment tool.

McLean, VA: Author. [Chapter 5]

Defense information infrastructure (DII) shared data

environment (SHADE), Capstone document.

(1996). Arlington, VA: U.S. DoD, Defense

Information Systems Agency (DISA). [Chapter

9] Available from http://www.disa.mil/coe/srs/

srs_base/srs_csa_PDF/zsrscsa.pdf

Defining and Measuring Success in the New Strategic

Environment [Technical Cooperation Program

Panel JSA-TP-3] Proceedings Report (1999)

Ottawa, Ontario, Canada.

Depuy, T. (1979). Numbers, prediction and war: Using

history to evaluate combat factors and predict

the outcome of battles. Indianapolis, IN: BobsMerrill. [Chapter 6]

Deutsch, R., & Hayes, R. E. (1990). Personality and

social interaction of effective expert analysts:

The issue of cognitive complexity [technical

paper]. [Chapter 6]

Deutsch, R., & Hayes, R. E. (1992). The war story in

warfighting situation assessment and

decisionmaking [technical paper]. [Chapter 6]

Dodd, L. (1995). Adaptive C2 modeling for RISTA

effectiveness studies. Second International

Command & Control Research & Technology

Symposium Proceedings. Washington, DC:

National Defense University. [Chapter 6]
A-6 NATO Code of Best Practice for C2 Assessment

Duff, J. B., Konwin, K. C., Kroening, D. W., Sovereign,

M. G., & Starr, S. H. (1992). In T. J. Pawlowski

(Ed.), Command, control, communications,

intelligence, electronic warfare measures of

effectiveness [Final Report]. Fort Leavenworth,

KS: Military Operations Research Society

(MORS) workshop. [Chapter 5]

Elzen, B., Ensenrink, B. & Smit, W.A. (1996). SocioTechnical Networks: How a Technology Studies

approach may help to solve problems related

to technical change. [Social Studies of Science,

vol 26, pp.95-141]

Engel, R., & Townsend, M. (1991). Division Command

And Control Performance Measurement.

Toronto, Ontario, Canada: Defence and Civil

Institute of Environmental Medicine (DCIEM).

[Chapter 5]

Entretiens Sciences & Défense. (1998). Nouvelles

avancées scientifiques et techniques, tome 1

[New scientific and technical advances, Volume

1]. DGA/CHEAr/D.R.E.E. [Chapter 7]

The Federation of American Scientists (FAS) Web

Page on Digital Terrain Elevation Data: http://

www.fas.org/irp/program/core/dted.htm

Fisher, R. A. (1951). Design of experiments (6th ed.).

Ontario, Canada: Oliver and Boyd, Ltd. [Chapter 3]

Fliss, R., Feld, R., Geisweidt, L., Kalas, W. D., Kirzl,

J., Layton, R., et al. (1987). Army command

and control evaluation system user’s handbook

(MDA903-96-C-0407). Ft. Leavenworth, KS:
Annex A A-7

Army Research Institute for the Behavioral and

Social Science. [Chapter 5]

Flood, R. L., & Jackson, M. C. (1991). Creative problem

solving: Total systems intervention (ISBN

0471930520). Chichester, UK: John Wiley and

Sons Ltd. [Chapter 3]

Gass, S.I., & Harris, C.M. (Eds.), Encyclopedia of

operations research and management science

(2nd ed.). New York: Kluwer Academic.

[Chapter 4]

Godet, M. (1998). Manuel de prospective stratégique,

tome 1, Une indiscipline intellectuelle, tome 2,

L’art et la méthode [Strategic prospective

handbook, Volume 1. An intellectual

indiscipline, Volume 2. In The method and art].

Paris: Dunod. [Chapter 7]

Halpin, S. M. (1992). Army command and control

evaluation system (ACCES): A brief description

(Working paper LV-92-01). Ft. Leavenworth,

KS: US Army Research Institute for the

Behavioral and Social Sciences. [Chapter 5]

Hartley, D. S., III. (1996). Operations other than war:

Requirements for analysis tools research report

(K/DSRD-2098). Oak Ridge, TN: Lockheed

Martin Energy Systems. [Chapter 4] [Chapter

8] Available from http://www.msosa.dmso.mil/

ootw.html

Hartley, D. S., III. (2001). Battle modeling. In S. I. Gass

& C. M. Harris (Eds.), Encyclopedia of

operations research and management science
A-8 NATO Code of Best Practice for C2 Assessment

(2nd ed.). New York: Kluwer Academic.

[Chapter 8]

Hartley, D. S., III. (2001). Military operations other than

war. In S. I. Gass & C. M. Harris (Eds.),

Encyclopedia of operations research and

management science (2nd ed.). New York:

Kluwer Academic. [Chapter 4]

Hawkins, G. (1984). Structural variance and other

related topics in the SHAPE Armour/AntiArmour study. In R. K. Huber (Ed.), Systems

analysis and modeling in defense—

development, trends, and issues. New York:

Plenum Press. [Chapter 5]

Hayes, R. E. (1994). Systematic assessment of C2

effectiveness and its determinants. 1994

Symposium on C2 Research and Decision Aids.

Monterey, CA: Naval Post Graduate School.

[Chapter 6]

Hillier, F. S., & Lieberman, G. J. (1990). Introduction

to operations research (5th ed.). New York:

McGraw-Hill. [Chapter 4]

Hoeber, F. P. (Ed.). (1981). Military applications of

modelling: Selected case studies. Alexandria,

VA: Military Operations Research Society.

[Chapter 4]

Hofmann, H. W., & Hofman, M. (2000). On the

development of command and control modules

on battalion down to single item level. In New

information processing techniques for military

systems (pp. 8.1–8.12). North Atlantic Treaty

Organisation (NATO) Research & Technology
Annex A A-9

Organisation (RTO) Meeting Proceedings 49.

[Chapter 6] Available from ftp://ftp.rta.nato.int/

PubFulltext/RTO/MP/RTO-MP-049/MP-049-

$$TOC.pdf

Holt, J., & Pickburn, G. (2001). OA techniques for the

future [Defence Evaluation and Research

Agency (DERA) Unclassified Report].

Farnborough, United Kingdom: Defence

Science and Technology Laboratory (DSTL).

[Chapter 3]

Hughes, W. P., Jr. (Ed.). (1989). Military modelling

(2nd ed.). Alexandria, VA: Military Operations

Research Society. [Chapter 4]

Human Systems. (1993). Survey of evaluation

methods for command and control systems.

Ontario, Canada. [Chapter 5]

The impact of C3I on the battlefield Panel 7 on the

defence applications of operational research

(AC243 panel-7 TR/4). [North Atlantic Treaty

Organisation (NATO) unclassified technical

report] (1994). Ad Hoc Working Group on the

impact of C3I on the battlefield. Brussels,

Belgium: NATO Defence Research Group.

[Chapter 5] Available from www.nato.int

IS10027: 1990 an information resource dictionary

system (IRDS) framework. (1990). [Chapter 9]

Jaiswal, N. K. (1997). Military operations

research:Quantitative decisionmaking. New

York: Kluwer Academic. [Chapter 4]
A-10 NATO Code of Best Practice for C2 Assessment

Jantsch, E. (1968). La prévision technologique [The

technology forecast]. Paris: OCDE. [Chapter 7]

Jaques, E. (1976). A general theory of bureaucracy.

Portsmouth, NH: Heinemann Educational

Books. [Chapter 6]

Kahnmann, D., Slovik, P., & Tversky, A. (Eds.). (1982).

Judgement under uncertainty: Heuristics and

biases. New York: Cambridge University Press.

[Chapter 10]

Keeny, R. L., & Raiffa, H. (1976). Decisions with

multiple objectives: Preferences and value

tradeoffs. Chichester, UK: John Wiley and Sons

Ltd. [Chapter 5] [Chapter 6]

Keppel, G. (1973). Design and analysis—A

researcher’s handbook (ISBN0-13-200030-X).

Englewood Cliffs, NJ: Prentice-Hall. [Chapter

3] [Chapter 10]

Klein, G. A. (1989). Recognition primed decisions.

Advances in Man-Machine Systems Research,

5, 47-92. [Chapter 8]

Kroening, D., & Moffat, J. (Eds.). (1999). Workshop

on C3I modeling (AC/323SASTP/9). North

Atlantic Treaty Organisation (NATO) Research

& Technology Organisation (RTO) Meeting

Proceedings 29. [Chapter 8] Available for order

from www.rta.nato.int

Krusche, S., & Tolk, A. (2000). Information processing

as a key factor for modern federations of combat

information systems. Paper presented at the

North Atlantic Treaty Organisation (NATO)
Annex A A-11

Research & Technology Organisation (RTO)

Information Systems Technology Panel (IST)

Symposium: New information processing

techniques for military systems, Istanbul, Turkey.

[Chapter 9] Available in PDF format from http://

home.t-online.de/home/Andreas.Tolk/literat.htm

Kruskal, J. B., & Wish, M. (n.d.). Multidimensional

scaling. In E. M. Uslaner (Ed.), Quantitative

Applications in the Social Sciences (Series 07

– 011, ISBN 0-8039-0940-3). Beverly Hills, CA:

SAGE Publications. [Chapter 10]

Lambert, N.J, (1997). “Operation Firm Endeavour:

Campaign Data, Its Collection and Use.”

Presented at the Directorate of Land Warfare

Historical Co-ordination Group Conference,

Trenchard Lines, DGD&D, Upavon, UK, 28-29

April 1997. Also available from the British Library

Document Supply Centre, Boston Spa,

Wetherby, West Yorkshire, LS23 7BQ, United

Kingdom, Reference DSC Shelfmark: 99/20363.

Lambert, N. J. (1998). Operational analysis in the field:

The utility of campaign monitoring to the

implementation force (IFOR) operation in Bosnia

and Herzegovina 1995–1996. Proceedings from

Cornwallis III, Analysis for Peace Operations.

(ISBN 1-896551-22-X). The Lester B Pearson

Canadian International Peacekeeping Training

Centre, Cornwallis Park, PO Box 100,

Clementsport, NS BOS 1EO, Canada.

Lambert N J, (1998). Operation Firm Endeavour:

Campaign Monitoring; Factional compliance

and the Return to Normality in Bosnia and
A-12 NATO Code of Best Practice for C2 Assessment

Herzegovina. Operational Analysis Branch, HQ

ARRC, Queens Avenue, JHQ Rheindahlen,

41179 Moenchengladbach, Germany. Also

available from the British Library Document

Supply Centre, Boston Spa, Wetherby, West

Yorkshire, LS23 7BQ, United Kingdom,

Reference DSC Shelfmark: 99/20361 and 99/

20367 (annexes).

Lambert, N. J. (2002). Measuring the Success of the

North Atlantic Treaty Organisation (NATO)

Operation in Bosnia and Herzegovina 1995–2000.

Issue 140/2, pp. 459-481. European Journal of

Operations Research, Special 2000 Edition.

Latour, B. (1997). Bruno Latour’s keynote speech: On

recalling ANT. In Actor network and after.

Centre for Social Theory and Technology

(CSTT), Department of Sociology and Social

Anthropology, Keele University, United

Kingdom. [Chapter 5] Available from http://

www.comp.lancs.ac.uk/sociology/

stslatour1.html

Latour, B. (1999). On Recalling ANT. In J. Law & J.

Hassard (Eds.), Actor network theory and after

(pp. 15-25). Oxford, United Kingdom: Blackwell

Publishers. [Chapter 5]

Lauren, M. K. (2000). Modelling combat using fractals

and the statistics of scaling systems, 5(3), 47-

58. Alexandria, VA: Military Operations

Research Society. [Chapter 8]
Annex A A-13

Lauren, M. K. (2001).Beyond Lanchester: A fractal based

approach to equations of attrition. Manuscript

submitted for publication. [Chapter 8]

Leroy, D., & Hofmman, D. (1996). Criterion of

command, control, communications,

intelligence (C3I’s) effectiveness. 1996

Command & Control Research & Technology

Symposium Proceedings. Washington, DC:

National Defense University. [Chapter 5]

Libicki, M. C. (1997). Defending cyberspace and other

metaphors. Washington, DC: National Defense

University. [Chapter 8]

Long term scientific study SAS-017 on human behaviour

representation [Final Report, Chapter 2]. (n.d.).

[Chapter 6] Available, with clearance, from: http:/

/www.rta.nato.int/RDP.asp?RDP=RTO-TR-047

Mandeles, M.D., et al. (1996). Managing “command

and control” in the Persian Gulf war (ISBN 0-

275-952614). Westport, CT: Greenwood

Publishing Group. [Chapter 6]

Mathieson, G. L., Tolk, A., Sundfoer, H. O., & Sinclair,

M. (2001). Data the Glue to Assessment.

Washington, DC: Cooperative Research

Program (CCRP). [Chapter 9] [Chapter 10]

McCafferty, M. C., & Lea, S. C. (1997). Measures of

effectiveness for operations other than war

(CDA/HLS/R9714/1.1). Farnborough, United

Kingdom: Center for Defence Analysis.

[Chapter 5]
A-14 NATO Code of Best Practice for C2 Assessment

Midgley, G. (1989). Critical systems and the problem

of pluralism. Cybernetics and Systems, 20, 219-

231. [Chapter 3]

Modeling and analysis of command and control (AC/

323(SAS)TP/12). (1999). North Atlantic Treaty

Organisation (NATO) Research & Technology

Organisation (RTO) Meeting Proceedings 38.

[Chapter 8] Available from http://www.rta.nato.int/

RDP.asp?RDP=RTO-MP-038

Moffat, J. (1996). The system dynamics of future

warfare. European Journal of Operational

Research, 90(3), 609-618. [Chapter 8] Available

with registration from http://www.palgravejournals.com/jors/

Moffat, J. (2000). Representing the command and

control process in simulation models of conflict.

Journal of the Operational Research Society,

51(4), 431-439. [Chapter 6] [Chapter 8]

Available with registration at http://

www.palgrave-journals.com/jors/

Moffat, J. (2002). Command and control in the

information age—Representing its impact.

Norwich, United Kingdom: The Stationery

Office. [Chapter 6] [Chapter 8] Available from

www.tso.co.uk

Moffat, J., & Dodd, L. (2000). Bayesian decisionmaking and

catastrophes. Defence Science and Technology

Laboratory (DSTL) [Unpublished Report]. [Chapter

3] Available from ksglasgow@dstl.gov.uk

Moffat, J., & Passman, M. (2002). Metamodels and

emergent behaviour in models of conflict.
Annex A A-15

‘Proceedings of the Simulation Study Group Two

Day Workshop, March 2002. The Operational

Research Society, UK.’ [Chapter 8] Available on

www.orsoc.org.uk

Moffat, J., & Prins, G. (2000). A revolution in military

thinking? [Issues from the 1999 Defence

Evaluation and Research Agency (DERA)

Senior Seminars]. Journal of Defence Science,

5(3). [Chapter 8] Available by request from

ksglasgow@dstl.gov.uk

Moffat, J., & Witty, S. (2000). Changes of phase in the

command of a military unit. Defence Science and

Technology Laboratory (DSTL) [Unpublished Report].

[Chapter 3] Available from ksglasgow@dstl.gov.uk

Moffat, J., & Witty, S. (2001). Bayesian decisionmaking

and military command and control. Manuscript

submitted for publication. [Chapter 8]

Moffat, J., Witty, S., & Dodd, L. (2002). Bayesian

decisionmaking and the precise use of force.

‘Journal of Defence Science 7(1)’ [Chapter 8]

Available from ksglasgow@dstl.gov.uk

National Research Council. (1998). Modeling human

and organizational behavior—Application to

military simulations (ISBN 0-309-06096-6).

Washington, DC: National Academy Press.

[Chapter 5] [Chapter 6] Available from http://

www.nap.edu

Noble D., & Flynn, W. (1993). Recognized primed

decisions (RPD) tool concept document.

Vienna, VA: Engineering Research Associates.

[Chapter 6]
A-16 NATO Code of Best Practice for C2 Assessment

Noble, D., Flynn, W., & Lanham, S. (1993). The

TADMUS recognized primed decisions (RPD)

tool: Decision support for threat assessment

and response selection. Paper presented at the

10th Annual Conference on Command and

Control Decision Aids. [Chapter 6]

Norman, D.A. (n.d.). Organization of action slips.

Psychological Review, 88, 1-15. [Chapter 6]

North Atlantic Treaty Organisation (NATO) Bi-MNC C2

plan part 2—Command and control

requirements. (1998).

North Atlantic Treaty Organisation (NATO) Annex B

to MC Guidance for Defence Planning. MC-299/

5. (1996)

North Atlantic Treaty Organisation (NATO)

Consultation, Command and Control Board

(NC3B), Information Systems Sub-Committee

(ISSC), NATO Data Administration Group

(NDAG). (1999). Feasibility study for the NATO

C3 information resource dictionary system,

(NC3 Information Resource Dictionary System

(IRDS), AC/322SC/5-WG/3WP4). Brussels,

Belgium. [Chapter 9] Available from http//

www.nato.int/docu/home.htm

North Atlantic Treaty Organisation (NATO) Open Systems

Working Group (2000). NATO C3 Technical

Architecture. AC/322(SC/5)WP/31, Brussels

North Atlantic Treaty Organisation (NATO)

Consultation, Command and Control Board

(2000). NATO C3 System Architecture

Framework. AC/322-WP/0125, Brussels
Annex A A-17

North Atlantic Treaty Organisation (NATO)

Consultation, Command and Control Board

(2000). NATO Interoperability Management

Plan (NIMP). AC/259-D/1247(rev), Brussels

North Atlantic Treaty Organisation (NATO) Study

Group on Modelling and Simulation (1998).

NATO Modelling and Simulation Master Plan,

Version 1.0. AC/323(SGMS)D/2, Brussels

Nowak, M., & May, R. (1992). Evolutionary games and

spatial chaos. Nature, 359. [Chapter 8]

Olae, N.–G., Roy, J., & Wetter, M. (2000). Performance

drivers—A practical guide to using the balanced

scorecard (ISBN 0-471-49542-5). Chichester,

UK: John Wiley and Sons Ltd. [Chapter 4]

[Chapter 5]

Perry, W. L., & Moffat, J. (1997). Developing models

of decisionmaking. Journal of the Operational

Research Society, 48(5), 457-470. [Chapter 6]

[Chapter 8] Available with registration from http:/

/www.palgrave-journals.com/jors/

Perry W. L., & Moffat, J. (1997). Measuring consensus

in decisionmaking: An application to maritime

command and control. Journal of the

Operational Research Society, 48(4), 383-390.

[Chapter 6] [Chapter 8] Available with

registration from http://www.palgravejournals.com/jors/

Perry W. L., & Moffat, J. (1997). Measuring the effects

of knowledge in military campaigns. Journal of

the Operational Research Society, 48(10), 965-

972. [Chapter 6] [Chapter 8] Available with
A-18 NATO Code of Best Practice for C2 Assessment

registration from http://www.palgravejournals.com/jors/

Pidd, M. (1996). Tools for thinking—Modelling in

management science. Chichester, UK: John

Wiley and Sons Ltd. [Chapter 3] [Chapter 8]

Pirlot & Vincke. (1997). Semiorders: Properties,

representations, applications (ISBN 0-7923-4617-

3). New York: Kluwer Academic. [Chapter 10]

Poel, I. (1998). Changing technologies. A comparative study

of eight processes of transformation of technical

regimes. Dissertation. Enschede, the Netherlands:

Twente University Press. [Chapter 6]

Prins, G. (1998). Strategy, force planning and diplomatic/

military operations. London: The Royal Institute

of International Affairs. [Chapter 3]

Quade, E. S. (Ed). (2000). Analysis for military

decisions. Alexandria, VA: Military Operations

Research Society. [Chapter 4]

The report of the data working group. (1998). In

Simulation technology-1997 (SIMTECH 97, Vol.

III). Alexandria, VA: Military Operations

Research Society. [Chapter 9]

Research and Technology Organization (RTO) SAS003 report on IFOR and SFOR data collection

& analysis (6 AC/323(SAS)TP/11). [Technical

Report]. (1999). Neuilly-Sur-Seine Cedex,

France: North Atlantic Treaty Organisation

(NATO) RTO. [Chapter 2] Available from http:/

/www.rta.nato.int/RDP.asp?RDP=RTO-TR-006
Annex A A-19

Richards, R., Neimeier, H. A., Hamm, W. L. (1997).

Analytical modeling in support of C4ISR mission

assessment (CMA). Third International

Symposium on Command and Control

Research & Technology, National Defense

University, Fort McNair, Washington DC

[Chapter 8]

Rip, A. (1995). Introduction of a new technology:

Making use of recent insights from sociology

and economics of technology. In Technology

analysis & strategic management. Journals

Oxford, 7(4), 417-430. [Chapter 6]

Rosenhead, J. (2001). Problem structuring methods.

In S. I. Gass & C. M. Harris (Eds.), Encyclopedia

of operations research and management

science (2nd ed.). New York: Kluwer Academic.

[Chapter 4]

Rosenhead, J. & Mingers, J. (Ed.). (2001). Rational

analysis for a problematic world revisited:

problem structuring methods for complexity,

uncertainty and conflict (2nd ed.). Chichester,

UK: John Wiley and Sons Ltd. [Chapter 3]

Sarin, R. K., (2001). Multi-attribute utility theory. In S.

I. Gass & C. M. Harris (Eds.), Encyclopedia of

operations research and management science

(Centennial ed.). Boston: Kluwer Academic.

[Chapter 5]

Schot, J., & Rip, A. (1996). The past and future of

constructive technology assessment. In

Technological forecasting and social change,
A-20 NATO Code of Best Practice for C2 Assessment

54, 251-268. New York: Elsevier Science.

[Chapter 6]

Schultz, J. V. (n.d). A framework for military

decisionmaking under risks [Prospect theory

related to military decisionmaking]. [Chapter 6]

Schwartz, P. (1993). La planification stratégique par

scénarios [Strategic planning by scenario].

Futuribles, 176. [Chapter 7] Available from:

http://www.futuribles.com

Sherrill, E. T., & Barr, D. R. (1996). Exploring

relationships between tactical intelligence and

battle results. Military Operations Research,

2(3). [Chapter 6] Available to order from http://

www.mors.org/publications/mor/vol02.htm

Siegal & Castellan. (1998). Non-parametric statistics

for the behavioral sciences (2nd ed., ISBN 0-

07-100326-6). New York: McGraw-Hill.

[Chapter 10]

Sinclair, M., Tolk, A. (2002). Building up a Common

Data Infrastructure NATO Symposium SAS039

on the “Analysis of the Military Effectiveness of

Future C2 Concepts and Systems” NATO C3

Agency, The Hague, The Netherlands.

The Source for Environmental Representation and

Interchange (SEDRISTM) Home Page: http://

www.sedris.com

Sovereign, M., Kemple, W., & Metzger, J. (1994).

Command, control, communications, intelligence

evaluation workshop (C3IEW). Measures

Workshop, United Kingdom. [Chapter 5]
Annex A A-21

Starr, S. H. (1996). Developing scenarios to support C3I

analyses. Proceedings of the Cornwallis Group,

Pearson Peacekeeping Center, Clementsport,

Nova Scotia, Canada. [Chapter 7]

Starr, S. H. (2001). Lessons recorded from applying

the North Atlantic Treaty Organisation (NATO)

code of best practice (COBP) for C2

assessment to operations other than war

(OOTW). Proceedings of the Command and

Control Research & Technology Symposium,

USNA, Annapolis, MD, June 19-21, 2001.

[Chapter 2] Available from http://www.mitre.org/

support/papers/tech_papers_01/starr_nato/

starr_nato.pdf

Starr, S.H., Haut, D., and Hughes, W. (1997)

“Developing Intellectual Tools to Support C4ISR

Analyses for Operations Other Than War,”

Proceedings of the Third International

Symposium on Command and Control

Research & Technology, National Defense

University, Fort McNair, Washington, DC, pp.

600-608, June 17–20, 1997.

Sweet, R., Metersky, M., & Sovereign, M. (1985).

Command and control evaluation workshop

[Military Operations Research Society (MORS)

C2 MoCE Workshop]. Monterey, CA: Naval

Postgraduate School. [Chapter 5]

Tolk, A. (1995). Zur Reduktion struktureller Varianzen

[On the reduction of structural variances].

Doctoral dissertation, University of the Federal

Armed Forces of Germany, Pro Universitate

Verlag, Sinzheim. [Chapter 5]
A-22 NATO Code of Best Practice for C2 Assessment

Tolk, A. (2000). Functional Categories of Support to

Operations in Military Information Systems.

Proceedings of the North Atlantic Treaty

Organisation (NATO) Regional Conference on

Military Communication and Information

Systems, Zegrze, Poland. [Chapter 9] Available

in PDF format from http://home.t-online.de/

home/Andreas.Tolk/literat.htm

Tolk, A. (2001). Bridging the data gap—

Recommendations for short, medium, and long

term solutions,(01S-SIW-011). Proceedings of

the Simulation Interoperability Workshop Spring

2001, Orlando, Florida. [Chapter 9] Available

in PDF format from http://home.t-online.de/

home/Andreas.Tolk/011_0301.pdf

Tolk, A., & Kunde, D. (2000). Decision support

systems—Technical prerequisites and military

requirements. Proceedings of the 2000

Command and Control Research and

Technology Symposium (CCRTS), Monterey,

CA. [Chapter 9] Available in PDF format from

http://www.dodccrp.org/6thICCRTS/Cd/Tracks/

Papers/Track6/038_tr6.pdf

US Department of Defense (DOD), Joint Chiefs of

Staff. (1999). Dictionary of military and

associated terms. Washington, DC: Greenhill

Books. [Chapter 9] Available from http://

www.dtic.mil/doctrine/jel/doddict/index.html

US Department of Defense (DOD) model and simulation

master plan. (1995). [Chapter 8] http://

www.msiac.dmso.mil/ms_needs_documents/

MS_Master_Plan.PDF
Annex A A-23

Van Creveld, M. (1985). Command in war. Cambridge,

MA: Harvard University Press. [Chapter 6]

Van Lente, H. (1993). Promising technology; The

dynamics of expectations in technological

developments. Unpublished doctoral

dissertation, Twente University Press, The

Netherlands. [Chapter 6] Available from

h.vanlente@geog.uu.nl

Wagner, H. M. (1975). Principles of management

science (2nd ed.). Upper Saddle River, New

Jersey: Prentice-Hall. [Chapter 4]

Wallace, G. J., & Moffat, J. (1997). The use of Petri

Nets to model satellite communications systems.

Journal of Defence Science, 2(1). [Chapter 8]

Available from ksglasgow@dstl.gov.uk

Wilson, J., & Corlett, E. (Eds.). (1990). Evaluation of

human work: A practical ergonomics

methodology (2nd ed.). London: Taylor &

Francis. [Chapter 10]

Windass, S. (2001). Just war and genocide—The Ottawa

Papers. Woodstock, United Kingdom: Foundation

for International Security. [Chapter 3]

Winer, B. J. (1962). Statistical principles in

experimental design. New York: McGraw-Hill.

[Chapter 10]

Witty, S., & Moffat, J. (2000). Catastrophe theory and military

decisionmaking. Defence Science and Technology

Laboratory (DSTL) [Unpublished Report]. [Chapter

3] Available from ksglasgow@dstl.gov.uk
A-24 NATO Code of Best Practice for C2 Assessment

The following additional documents are under

development:

NATO C3 System Baseline Architecture

NATO C3 System Overarching Architecture
B-1

ANNEX B

Acronyms

A

ACCES Army Command and Control Evaluation

System

ACE Resources Allied Command Europe Resources

(part of SHAPE)

AF(N) Regional Command

AHWG AC/243 Panel 7 Ad Hoc Working Group

AI Artificial Intelligence

AMF(L) ACE Mobile Force (Land)

ANT Actor Network Theory

ARRC ACE Rapid Reaction Corps

ATS Norman’s Activation Trigger Scheme

B

C

C2 Command and Control
B-2 NATO Code of Best Practice for C2 Assessment

C3 Command, Control, and Communications

or Consultation

C3I Command, Control, Communications or

Consultation, and Intelligence

C4 Command, Control, Communications or

Consultation, and Computers

C4I Command, Control, Communications or

Consultation, Computers, and Intelligence

C4ISR Command, Control, Communications or

Consultation, Computers, Intelligence,

and Surveillance and Reconnaissance

CIMIC Civil-Military Co-operation

CIS Command Information Systems

CISS Center for Information Systems Security

COBP Code of Best Practice

CPX Command Post Exercise

CTA Constructive Technology Assessment

D

DCEP Data Collection/Engineering Plan

DISA Defence Information Systems Agency

DOTMLP Doctrine, Organisation, Training, Material,

Leadership, Personnel

DOTMLPF Doctrine, Organisation, Training, Material,

Leadership, Personnel, Facilities/Forces
Annex B B-3

DP Dimensional Parameters

DSTL Defence Science and Technology

Laboratories (UK)

E

EEA Essential Elements of Analysis

F

FS Feasibility Study

FTX Field Post Exercise

G

GRR Generic Risk Register

H

HB(A) UK Historical Branch (Army)

HEAT Headquarters Effectiveness Assessment

System

HQ Headquarters

I

IO International Organisation

IRDS Information Resource Dictionary Systems
B-4 NATO Code of Best Practice for C2 Assessment

IRTF(L) Immediate Reaction Task Force (Land)

J

JCSC Joint Sub-Regional Command South

Centre

JCSE Joint Sub-Regional Command South

East

JFCOM Joint Forces Command

K, L

M

MAPEX Map Exercise

MAUT Multi-Attribute Utility Theory

MCES Modular Command and Control Evaluation

Structure

M-E-M Model-Exercise-Model

METT-TC Mission, Enemy, Troops, Terrain, Troops,

Time, and Civil considerations

MND(C) Multinational Division (Centre)

MoCE Measures of C2 Effectiveness

MoE Measures of Effectiveness

MoFE Measures of Force Effectiveness

MoM Measures of Merit
Annex B B-5

MoP Measures of Performance

MoPE Measures of Policy Effectiveness

MORS Military Operations Research Society

M&S Modelling and Simulation

M-T-M Model-Test-Model

N

NATO North Atlantic Treaty Organization

NC3A NATO C3 (Consultation, Command &

Control) Agency

NGO Non Governmental Organisations

NL MOD Netherlands Ministry of Defense

O

OA Operational Analysis

OOTW Operations Other Than War

OR Operations Research

P

PESTLE Political, Economic, Social, Technological,

Legal, and Environmental

PfP Partnership for Peace

PRL Policy Requirements Land
B-6 NATO Code of Best Practice for C2 Assessment

PVO Private Volunteer Organisations

Q

R

RIGOUR Repeatability, Independence, Grounding

in reality, Objectivity of process,

Uncertainty, and Robustness

ROE Rules of Engagement

S

SACLANT OA Supreme Allied Command Atlantic

Operational Analysis Cell

SCOT Social Construction of Technology

SFS Strike Force South

SHADE Shared Data Environment

SHAPE Supreme HQ Allied Powers Europe

SMP Study Management Plan

SOW Statements of Work

STA Surveillance Targeting and Acquisition

STAFFEX Staff Exercise

STS Science and Technology Studies

SWOT Strengths, Weaknesses, Opportunities,

Threats
Annex B B-7

T

TA Technology Assessment

TRAC TRADOC Analysis Center

TRADOC US Army Training & Doctrine Command

TTP Tactics, Techniques, and Procedures

U

V

VTC Video Teleconference

VV&A Verification, Validation, and Accreditation

VV&C Verification, Validation, and Certification

W

WBS Work Breakdown Structure

X, Y, Z
C-1

ANNEX C

The MORS Code

of Ethics

The MORS Code of Ethics and Responsibilities for

Practitioners of Military Operations Research

Military OR Professionals must aspire to be:

• Honest, open and trustworthy in all their relationships;

• Reliable and consistent in the conduct of

assignments and responsibilities, always doing

what is right rather then expedient;

• Objective, constructive and accurate in what they

say and write;

• Accountable for what they do and choose not to do;

• Respectful of the work of others, giving due

credit and refraining from criticism of them

unless warranted; and

• Free from affiliation with others or with activities

that would compromise them, their employers, or

the Society.
D-1

ANNEX D

Human And

Organisational

Issues Checklist

This checklist presents a summary of a particular set

of human and organisational issues that is specifically

relevant to behaviour and performance of humans in

command and control situations. Its purpose is to

sensitise the analyst and help him to assess whether

human or organisational issues are part of the problem

domain and should be addressed in the solution space.

Also, it should assist him in identifying human sciences

disciplines for consultation.

It should be pointed out, however, that this checklist is

preliminary and must not be considered to be

comprehensive. The field of human sciences is too

large and the set of human and organisational issues

to ill defined to provide the analyst with an exhaustive

checklist, at this time. Research in this area is ongoing

and we expect that later editions of this document will

provide the analyst revised and improved checklists.

1. Human Issues

• Physiological factors
D-2 NATO Code of Best Practice for C2 Assessment

 This refers to bio-medical and environmental

factors that influence behaviour and performance

• Stress

• Fatigue and lack of sleep

• Blood sugar

• Fitness conditions

• Weather conditions

• Geographical terrain conditions

• Ergonomic factors (e.g. performance

degradation due to working in protective suits)

 Ergonomics is a science discipline that studies

human work and work environment relationships.

• Behavioural factors (related to functioning in a

C2 group)

• Social Competence & Experience

Capability to interact with others

• Communication skills

• Language skills

• Empathy (Social Awareness/

understanding)

• Conflict handling style

• Frustration handling style

• Military Competence & Experience

• Handling Danger
Annex D D-3

• Defining mission objectives

• Task Competence & Experience

• Cognitive factors

 These factors are related to how humans

perceive their environment, how they give

meaning to what they see.

• Information processing style

• Information processing capacity

• Creating Situational Awareness

• Individual decision making

• Risk tolerance

• Pre-disposition!

• Receptivity to new information

(open / closed)

• Expertise

• Prior training and knowledge

(operational codes)

• Emotional factors

• Morale

• Attitude

• Separation from Family

• Fear

• Stress
D-4 NATO Code of Best Practice for C2 Assessment

• Resilience (ability to overcome

negative feedback)

• Leadership factors

 There exists no clear-cut concept of leadership

effectiveness, but we do know some of the

factors that affect the effectiveness of a leader

• Expectation and behaviour of superiors

• Expectations and behaviour of

subordinates

• Expectations and behaviour of colleagues

• Personality and experience

• Organisational Culture and policy – with

regard to leadership and command.

Allocation or responsibility and authority.

• Ability to Motivate and to Direct others

• Moral and Judicial responsibilities

• Coaching capabilities (towards

subordinates)

2. Organisational Issues

• Structure

• Number of echelons or layers

• Span of control for nodes

• Pattern of linkages between nodes (e.g.

hierarchical, multi-connected)

• Permanent versus transitory
Annex D D-5

• Formal versus Informal

• Function

• Distribution of responsibility

• Distribution of authority

• Distribution of Information

• Functional Specificity

• Ambiguity in command relationships

• Capacity

• Differences in communication systems /

Architectures

• Differences in information processing

systems / architectures

• Differences in field training and operational

experience

• Differences in personnel

• Experience

• Training

• Cognitive ability

• Roles

• Allocation of Responsibility / Authority

• Role Conflicts. The interference of multiple

Roles in one individual

• Sociological factors

• Understanding of environment
D-6 NATO Code of Best Practice for C2 Assessment

• Political

• Social

• Cultural

• Interoperability Issues

• National and cultural differences

• Organizational approaches and values

• Communication standards and technology

• Differences in perceptions

• Organisational command style

• Decentralised or Centralised

• Collaborative versus Authoritative

• Formal versus Informal

• Command products: orders, objectives,

missions

• Organisational Culture

 Shapes patterns of organisational behaviour and

reflects thought and activity patterns by

members of the organisation.

• Belief systems

• Organisational norms & how are they

expressed

• Organisational values & how are they

expressed

• Open to organisational learning
Annex D D-7

I.e. does the organisational culture embed

characteristics that enable and facilitate

organisational earning?

3. Interaction between Human and

Organisational Issues

• Group decision making

Groups can exert social pressure on members

that affect the decision-making capability of

individual members.

• Group Dynamics

The behaviour of individuals in a group will be

influenced by the fact that they are interacting in

a group. This interactive process contains a

number of dynamic dimensions and does effect

the result of interaction. A command post is a

group of people and its performance is affected

by it group dynamics.

• Social interaction & communication

• Communication

• Social Identity and social conflicts

• Individual dominance/leadership

• Cohesion

• Teambuilding, teamwork

• Trust in group member’s competence and

loyalty.

• Cultural factors
D-8 NATO Code of Best Practice for C2 Assessment

Culture is more or less the whole of beliefs and

assumptions, including norms and values, about

things and behaviour in a group. As such it

guides the behaviour of people, also in the

military and in command posts.

• Note! Culture itself is immaterial, intangible.

It can only be distilled form observed actions

by members of an organisation or societal

group

• Note! Culture is learned through interaction

with others.

• Especially in multi-national OOTW,

members of different cultural

background have to interact. Mutually

acceptable norms of behaviour will

have to be developed in the process.

• Socialisation process

• Culture lag

Changes in the environment or to the

organisation often require adaptations in

behaviour. Sometimes these changes are

required or introduced so fast that an

organisation hasn’t had the time to the whole

of its culture to adapt to the new situation,

i.e,. to embed the changes in its culture. This

may lead to friction.

• E.g.1: The introduction of new

technologies may change the way we

work. If this is done too fast, or without

thoughtful guidance it may have a
Annex D D-9

disruptive effect on behaviour and thus

on organisational effectiveness.

• E.g.2: Peace Keeping and War

Operations each require different

types of behaviours which is often

quite different from the type of

behaviour in the peaceful environment

at home. Western armies operate on

personnel rotation schedules whereby

individuals change frequently from one

situation to another. This may also

result in culture lag causing friction.

• Sub-culture

Within the larger set of culture, groups may

develop specialised cultures, such as, e.g.,

distinct unit cultures.

• Sometimes these sub-cultures may

conflict over some issues with the

dominant organisational or societal

culture. Understanding these

differences is often the first step in

resolving these situations.

• Social Control.

Individual (group, organisational) members

watch and correct each other if they accepted

procedures of behaviour aren’t followed.

• If this corrective action pattern is too

strict, the group becomes rigid and is

less able to adapt to new situations.
D-10 NATO Code of Best Practice for C2 Assessment

• If it is too loose, and no culture or

norms are enforced, chaos and

uncertainty may rule the group.

• Commanders Intent.

 A leader can effect the current (sub)

culture of his unit by setting the example

and by establishing and enforcing required

procedures and behaviour.

• Both must be synchronized, else

friction will occur.

• By action and word a leader sets the

level and direction of (social) control.

• Ethics

• Cooperability

• Effectiveness of communication

• Willingness to co-operate and collaborate

4. Environmental factors

These factors affect individual, organisational

and group issues.

• Noise

• Visibility

• Temperature / humidity

• Terrain type

• Infrastructure in area of operations. (E.g.

transport, communication, healthcare,
Annex D D-11

agriculture, food distribution, water, civil

administration -infrastructures)

• Military support infrastructure

• Social, economic, and political situation in area

of operations

• Rules of Engagement

• Ease of interaction

• Command Post layout

• Communication mechanisms (e.g. voice,

teletype, VTC)

• Geographic distribution of the command

Related Human Science Disciplines

This list provides an overview of the relevant scientific

disciplines with the expertise to answer questions

arising in the context of human and organizational

issues in command and control. The list implies a rough

taxonomy of disciplines only. Even though boundaries

between disciplines are frequently fuzzy, an attempt

was made to minimise the overlap.

1 Psychology

1.1 Individual Psychology

1.1.1 Cognitive Psychology

1.1.2 Learning Psychology

1.1.3 Action Theories
D-12 NATO Code of Best Practice for C2 Assessment

1.1.4 Differential Psychology

1.1.5 Psychometrics

1.2 Social Psychology

1.3 Clinical Psychology

2 Educational Sciences and

Pedagogics

3 Sociology

3.1 Social Morphology (e.g. Social Stratification,

Demography)

3.2 Sociology of Political Power (See Also Political

Sciences)

3.3 Mass Communication

3.4 Sociological Methodology (Empirical Social

Research)

4 Organisational Sciences

5 Economic Sciences

6 Political Sciences

6.1 Science of Domestic Politics

6.2 Science of International Relations

6.3 Political Disaster Research
Annex D D-13

7 Ergonomics, Human Factors

Research

7.1 Physiology and Anthropometrics

7.2 Technical Ergonomics (Man-Machine-Interface,

Workspace)

7.3 Cognitive Ergonomics (See Also Individual

Psychology)

7.4 Crew Ergonomics (See Also Social Psychology)

7.5 Safety and Health Hazard Prevention

8 International Law of War,

International Humanitarian Law

9 Ethics

9.1 Deontology (Theories of Ethical Norms and

Values)

9.2 Theories of Ethical Practise and Morality

10 Social and Legal Philosophy

11 Cultural Anthropology

Literature

Ali, M.I. et al. (2000). Enabling Learning: A Progress

Report On The Enterprise Social Learning

Architecture Task. A paper presented at CCRT

Symposium in 2000 in Canberra, Australia.
D-14 NATO Code of Best Practice for C2 Assessment

Capstick, M.D. (1998). Command and Leadership in

Other Peoples Wars. Land Force Western

Headquarters, Edmonton, Canada. Published

at the NATO RTO Workshop: The Human in

Command, Kingston, Canada, 1998.

Jager, H.de & Mok, A.K. grondbeginselen der

Sociologie, Gezichtspunten en begrippen. H.E.

Stenfert Kroese B.V.: Leiden, the Netherlands.

LD-1. (200). Leidraad Commandovoering. Doctrine

Commission Royal Netherlands Army, RNLA

Staff, The Hague.

Pigeau, R & MacCann, (1998). Re-Defining Command

and Control. Defence and Civil Institute of

Environmental Medicine. Ontario, Canada.

Published at the NATO RTO Workshop: The

Human in Command, Kingston, Canada, 1998.

Shattuck, L.G. (1998). Communication of Intent in

Military Command and Control Systems.

Department of behaviroal Sciences and

Leadership, United States Military Academy.

Westpoint, United States. Published at the

NATO RTO Workshop: The Human in

Command, Kingston, Canada, 1998.

Persson, P., James, M.N. & Eriksson, H. Command

and Control: A Biased Combination? National

Defence College, Sweden. Published at the

NATO RTO Workshop: The Human in

Command, Kingston, Canada, 1998.
Annex D D-15

Vogelaar, A.L.W., Muuse K.F. & Rovers, J.H. (eds)

(1998). The Commanders Responsibility in

Difficult Circumstances. NL Arms, Netherlands

annual review of military studies. Gianotten b.v.:

Tilburg.

Winslow, D (1998) Misplaced Loyalties. The Role of

Military Culture in the Breakdown of Discipline

in Peace Operations. Department of Sociology.

University of Ottawa. Published at the NATO

RTO Workshop: The Human in Command,

Kingston, Canada, 1998.

Zsambok, C. E., & G. Klein. (1997). Naturalistic

Decision Making. Mahwah, New Yersey:

Lawrence Erblaum Associates, Publischers
E-1

ANNEX E

Alternate

Definitions of Risk

and Uncertainty

A

 number of different definitions of risk and

uncertainty are in common use, and this can lead

to confusion and misunderstanding. This COBP adopts

specific working definitions of risk, uncertainty, and

sensitivity; however, this annex lists a range of others

for information.

Risk

Risk is defined in this COBP as the possibility of

suffering harm or loss. In other words, exposure to

harm or loss. This includes an opportunity loss.

Alternate definitions of risk in common

use include:

• not achieving your objective;

• the likelihood of not achieving your objective;

• a threat to successful outcome;
E-2 NATO Code of Best Practice for C2 Assessment

• an assessment of the probability of failure;

• an uncertain future scenario; and

• a perception of consequential pain.

The alternate definitions may be seen as subtle

variations of each other and are all equally valid.

However, this very subtlety can be a source of

confusion and doubt within a study unless a clear

working definition is adopted.

Uncertainty

Uncertainty is defined in this COBP as an inability to

determine a variable value or system state (or nature)

or to predict its future evolution.

Alternate definitions of uncertainty in

common use include:

• a lack of clarity in the definition of a system

or variable;

• a lack of confidence in an assumption or

result; and

• a lack of knowledge about a subject of interest.

All of these definitions have some validity, but this

should not be allowed to cloud the thinking during

C2 assessment.
Annex E E-3

Sensitivity

Sensitivity, as used in this COBP, describes a predisposition to respond strongly to a stimulus or

variations in an input factor.

Alternate definitions for sensitivity in

common use include:

• an indication of the importance or criticality of a

feature or variable of an analysis; and

• a measure of the political profile of the subject of

study.

Sensitivity, as it relates to the production of robust

analysis, must be clearly defined in terms of response

to stimulus rather than importance or criticality. In C2

systems many insensitive factors are critical and many

sensitive ones are relatively unimportant.
F-1

ANNEX F

SAS-026 History

T

his revised and expanded version of the COBP

for C2 assessment was developed by SAS-026

building upon the initial (1998) version of the COBP

produced by SAS-002. This edition of the COBP is a

synthesis of decades of expertise from various

countries and hundreds of analyses. It was developed

using a set of case studies and incorporates feedback

from users of the initial version. Lastly, SAS-039

provided a peer review of the final draft product.

Members of SAS-026 are listed below.

Dave Alberts (chair), United States

Tim Bailey, United States

Paul Choinard, Canada

Cornelius d’Huy, The Netherlands

Uwe Dompke, NC3A

Dean Hartley, United States

Richard Hayes, United States

Reiner Huber, Germany

Don Kroening, United States

Stef Kurstjens, The Netherlands
F-2 NATO Code of Best Practice for C2 Assessment

Nick Lambert, NC3A

Georges Lascar, France

Christian Manac’h, France

Graham Mathieson, United Kingdom

Jim Moffat, United Kingdom

Orhun Molyer, Turkey

Valdur Pille, Canada

David Signori, United States

Mark Sinclair, United States

Mink Spaans, The Netherlands

Stuart Starr, United States

Swen Stoop, The Netherlands

Hans Olav Sundfor, Norway

Klaus Titze, Germany

Andreas Tolk, United States

Corinne Wallshein, United States

Gary Wheatley, United States

John Wilder, United States