A DoD CCRP/NATO Collaboration

This major revision to the Code of Best Practice for

C2 Assessment is the product of a NATO Research

and Technology Organisation (RTO) sponsored

Research Group (SAS-026). It represents over a

decade of work by many of the best analysts from

the NATO countries. A symposium (SAS-039) was

hosted by the NATO Consultation Command Control

Agency (NC3A) and provided the venue for a

rigorous peer review of this code.

This publication is the latest in a series produced

by the Command and Control Research Program

(CCRP) under the auspices of the Assistant

Secretary of Defense (C3I). The CCRP has

demonstrated the importance of having a research

program focused on the national security

implications of the Information Age. The research

sponsored and encouraged by the CCRP contributes

to the development of the theoretical foundations

necessary to support the Information Age

transformation of the Department. Other CCRP

initiatives are designed to acquaint military and

civilian leaders with emerging issues related to

transformation. This CCRP Publication Series is a

key element of this effort.

Check our Web site for the latest CCRP activities and publications.

This code is also available on our Web site.

www.dodccrp.org
NATO Code

of

Best Practice

for

C2 Assessment

Analyst’s

Summary Guide
1

NATO CODE OF

BEST PRACTICE FOR

C2 ASSESSMENT

Analyst’s

Summary Guide

Introduction

Background and Aims

N

ATO has produced a Code of Best Practice

(COBP) 1 in order to facilitate high quality

assessment in the area of Command and Control

(C2). The COBP is the product of international

collaboration drawing together the operational and

analytical experience of leading military and civilian

defence experts from across the NATO nations. The

COBP enhances the understanding of best practice

and outlines a structured process for the conduct of

operational analysis for C2, which is the core

capability of Information Age defence and security.

The command and control aspects of military

capability are difficult to assess. Use of the COBP

will increase the likelihood of quality products that
2 NATO Code of Best Practice for C2 Assessment

are complete, relevant, transparent, credible, and

authoritative. In particular, the COBP will:

• Support broadening of the analysis process;

• Support effective use of analysis in direct support

of operations;

• Improve the quality and coherence of business cases;

• Reduce risk and cost of the planning,

preparation, analysis and presentation of

supporting documentation;

• Provide a methodology acceptable to both

analysts and decisionmakers.

The COBP facilitates effective structuring of the

analytic process. It describes a framework that can

be used to design, execute, review, and deliver high

quality C2 assessments that include all key factors

and stakeholders. Use of the COBP should be

regarded as a community standard for all levels and

scales of assessment studies.

Definition of C2

C2 has been defined by NATO as Military Function

01: “The Organisation, Process, Procedures, and

Systems necessary to allow timely political and military

decisionmaking and to enable military commanders

to direct and control military forces.”2 C2 systems are

further defined in NATO documents to include:

headquarters facilities, communications, information

systems, and sensors & warning installations.3 More

recently, the term C2 has referred to the collaborative
Analyst’s Summary Guide 3

and consultative processes that are an inherent part

of coalition operations.

For the purposes of the COBP, the term C2 is

intended to be an umbrella term that encompasses

the concepts, issues, organisations, activities,

processes, and systems associated with the NATO

definition of C2 as well as the other terms

enumerated above.

Why is C2 Special?

C2 is special because it explicitly involves

representation of the human component. The focus

of military research and analysis has predominantly

been on the physical domain. C2 deals with

distributed teams of humans operating under stress

and in a variety of other operating conditions. C2

problems are thus dominated by their information,

behavioral, and cognitive aspects, which have been

less well researched and understood. This focus

creates a multidimensional, complex analytic space

that involves multi-sided dynamics including

friendly, adversary, and other actors, actionreaction dynamics, and tightly coupled interactions

among elements such as doctrine, concepts of

operations, training, materiel, and personnel.

C2 issues are difficult to decompose and

recompose without committing errors of logic.

Moreover, the composition rules by which the

various factors inherent to C2 interact are poorly

understood except in arenas that have been

previously studied in detail. Finally, the C2 arena

is weakly bounded, with issues that, although on
4 NATO Code of Best Practice for C2 Assessment

initial examination appear quite finite, often prove

to be linked to very high-level factors.

The COBP is intended to assist the community in

dealing with, and overcoming, the barriers to effective

C2 assessment.

Structure of the Guide

The rest of this guide is structured to follow the chapters

of the COBP and abstracts definitions, principles, and

key points of guidance.

Preparing for Success:

Assessment Participants,

Relationships, and Dynamics

The issues associated with assessment

participants, relationships, and dynamics must be

addressed in order to establish a sound foundation

for a successful assessment. As an initial step, the

individuals who are involved in the study must be

identified, the relationships among the participants

must be understood, and a common understanding

of the study’s goals, objectives, scope, and

administrative aspects (e.g., schedule, budget)

must be established and documented. An

assessment team must be assembled that

manifests the necessary skills and experience, and

the broad plan of attack must be formulated.
Analyst’s Summary Guide 5

Assessment Participants

Early in the study, the assessment team should

identify which individuals and organisations should

perform key roles in the assessment. The roles of

interest include, inter alia, assessment team

members, decisionmakers or problem owners,

stakeholders, bill payers, existing and future study

teams, peer reviewers, data providers, assumption

providers, and data collectors. It is prudent for the

assessment team to map the roles onto the

individuals and organisations involved and to

understand their interrelationships.

Relationships Among Participants

and the Conduct of the Assessment

The relationship among the assessment team, the

key sponsor, and the stakeholders is of paramount

importance and will influence the course and

success of the effort. The assessment team

should be aware that the diverse participants may

have divergent perspectives and agendas.

The assessment team should undertake the

following actions:

• Maintain long-term relationships (including an

ongoing dialogue) with the sponsor and stakeholder

organisations. This will yield substantial dividends in

the form of easier communication, greater trust, and

stronger support.

• Generate a concise, agreed-to Terms of

Reference, covering goals, scope, products,

schedule, and resources.
6 NATO Code of Best Practice for C2 Assessment

• Find out at an early stage in the project what the

products of the study are to be used for by the

sponsor and stakeholder organisations.

• Early in the study, agree on a common language

and develop a study glossary. This should evolve

during the course of the study.

• Perform a rapid first pass of all the phases of the

project to help establish the approximate budget

requirements, especially in large C2 projects.

The assessment team should establish

strategies to address shortfalls in the event the

sponsor’s resources are limited to a level below

what is required to support a quality study.

Assessment Team

• The precise skills and experience required by

the assessment team must be established

following initial problem formulation. The

assessment team must be interdisciplinary.

• As an illustration, the following are representative

skills and experience needed for the core team

and consultants and part-time team members for

a recent, complex C2 assessment:

û Core Team: project management; OR/OA

skills; cross military experience; human science

and organistational theory; data collection.

û Consultants and Part-time Team Members:

military; training and exercise planning;

communications and information systems
Analyst’s Summary Guide 7

expertise; human computer interface

expertise; OOTW related issues.

• Sufficient time and a facilitating process should

be built into the project plan for the group of

individuals to coalesce into a team.

Assessment Process

• The assessment process is non-linear and iterative.

• The assessment team must realize that all of

the elements of the C2 assessment are

interrelated. Hence Problem Formulation,

Solution Strategy, Measures of Merit,

Scenarios, Human/Organisational Factors,

Models and Tools, Data, and products are all

interdependent (See Figure 1).

• Peer review is a necessity, not a luxury.

Problem Formulation

Effective problem formulation is fundamental to the

success of all assessments, but particularly in C2

assessment because the issues are often illdefined and complex, involving many dimensions

and a rich context. The assessment team must

perform problem formulation carefully and

understand its underlying principles. There are

principles of problem formulation that apply to C2

assessment in general, some of which are

particularly important to C2 assessment for OOTW,

in particular. Drawing on these principles, one can

characterize the problem formulation process and
8 NATO Code of Best Practice for C2 Assessment

the products that are to be produced in the problem

formulation phase.

Figure 1. C2 Assessment Process

Definition of Problem Formulation

• Problem formulation is an iterative process that

evolves over the course of the study. Iteration

is essential even for small studies or where

time is short—it will save time later and help

ensure quality.

• Problem Formulation is fundamentally a social

process of developing a shared understanding.

• The context of the study includes geopolitical

context; political, social, historical, economic,

geographic, technological environments; actors;

threats; aims and objectives of the assessment,
Analyst’s Summary Guide 9

including the decisions to be supported; generic

C2 issues; relevant previous studies; and

stakeholders and their organizational affiliations.

• The aspects of the problem include issues to be

addressed; assumptions; high-level Measures of

Merit (MoM); independent variables (controllable

and uncontrollable); constraints on the values of

the variables (domain and range); time

constraints on delivery of advice to the

decisionmaker; and whether this is a single

decision or (possibly one of) a chain of decisions

to be made over time.

• The problem is not formulated until the

assessment team has addressed each aspect of

the problem.

Principles of Problem Formulation

• Proper problem formulation takes substantial

time and effort!

• Explicit problem formulation must precede

construction of concepts for assessment or

method selection.

• The assessment team must have an

understanding of the decisions to be supported

by the assessment and the viewpoints of the

various stakeholders to clarify the study issues.

• Problem formulation must not only provide

problem segments amenable to analysis, but

also a clear and valid mechanism for meaningful
10 NATO Code of Best Practice for C2 Assessment

synthesis to provide coherent knowledge about

the original, larger problem.

• Problem formulation must be broad and iterative

in nature, accepting the minimum of a priori

constraints and using methods to encourage

creative and multi-disciplinary thinking.

• The problem formulation process should not

focus prematurely on subsets of the problem.

• Practical constraints such as data availability,

study resources (including time), and limitations

of tools should be treated as modifiers of the

problem formulation rather than initial drivers.

Such constraints may, in the end, drive the

feasible solutions, but it is important to recognise

this as a compromise rather than an ideal.

• Problem formulation should address risk from

multiple perspectives. Risk analysis techniques

should be used to directly explore options to

mitigate risk.

Principles Particularly Appropriate for

OOTW C2 Assessments

• Problem formulation must address the

geopolitical context of the OOTW problem and

seek to identify the “broad” C2 issues contained

within the Terms of Reference for the study.

• OOTW C2 assessments often involve policyrelated impacts outside the context of a particular

military operation. Therefore, MoM hierarchies

must contain Measures of Policy Effectiveness.
Analyst’s Summary Guide 11

• The assessment team must have an historical

perspective to understand OOTW issues

because social conflict and structures often have

roots far back in history.

• The assessment team must have access to

subject matter experts from a broad range of

disciplines (e.g., social scientists, historians, and

regional experts in OOTW assessment).

Problem Formulation Process

• The process begins with the sponsor presenting

the assessment team with a problem to assess

and an articulation of broad constraints.

• During the early stages of problem formulation the

assessment team should quickly cover the whole

assessment process and produce an initial formulation.

• The assessment team must identify what it

perceives as the real issues to address. The

team must engage in a dialogue with the key

sponsor and stakeholders to get “buy in” for

these issues.

• In dealing with fuzzy or uncertain boundaries, the

assessment team should explore and

understand the significance of each proposed

boundary. The assessment team should keep an

open mind during the early stages of problem

formulation about where the boundaries lie and

their dimensional nature.

• While clear definitions and hard conceptual

boundaries are ultimately necessary in order to
12 NATO Code of Best Practice for C2 Assessment

create a manageable problem space, the

assessment team should avoid coming to

closure prematurely.

• Identification of high-level MoM should start with

ideal measures of the desired benefits before

considering what can be practically generated by

analysis (the latter may force the use of

surrogate MoM, but these must be clearly related

to the desired measures).

• The assessment team should identify, develop (if

necessary), and apply appropriate tools to

support problem formulation. Representative

tools and techniques include: techniques for

supporting expert elicitation, influence diagrams,

causal maps, system dynamics models, and

agent-based models.

Problem Formulation Products

An iteration of the problem formulation process can

be said to be complete when the following is

accomplished and documented: the “real” question

to be answered is known; the assumptions have been

articulated; and the high level MoM, the independent

variables, and the constraints associated with the

variables have been identified.

Solution Strategies

The Problem Formulation phase clarified “what” is

to be achieved during the assessment. The Solution

Strategy phase must transform this understanding

into “how” these goals and objectives are to be

achieved. Even if the way ahead seems clear, the
Analyst’s Summary Guide 13

articulation of a formal solution strategy is

necessary. Thus, the assessment team must

understand the definition and principles of the

solution strategy, the process by which it is

developed, and the products that it produces.

Definitions and Principles

A solution strategy consists of the specification of a

set of sequential and parallel analytical steps

(documented in the Study Plan), often involving

several methodologies and tools. The solution

strategy is designed to begin with what is known, and

by execution of the specified steps, leads to what

one desires to know—an illumination of the issues.

• The Study Plan consists of two inter-related

parts—the formulated problem (the What) and

the solution strategy (the How).

• The solution strategy should not be designed

before an initial pass through the problem

formulation process and the problem formulation

products are available to the team.

• The assessment team should always remember

the inherently iterative nature of the process.

Developing a Solution Strategy

• The solution strategy should strike an artful

balance between what the team would like to do

and what is possible to do, given the state of the

art, available data, tools, schedule, and

resources available.
14 NATO Code of Best Practice for C2 Assessment

• The team should first elaborate on the measures

that are to be evaluated in the study. Using these

measures and consideration of human and

organizational factors, a conceptual model of the

analysis should be developed and refined as

data requirements, methods and tools, and

scenarios are selected for the analysis.

• Frequently, a solution strategy becomes complex,

thereby requiring the team to decompose the

problem into parts, each of which requires

assessment with its own set of tools.

• Taken together, the solution strategy must

include the MoMs, relevant human and

organizational factors, specification of scenarios,

data collection requirements, and methods and

tools to be used in the analysis.

Planning Documents

• The solution strategy is documented in a Study

Plan that links the problem formulation and

solution strategy together in one plan.

• The Study Plan should be developed in an

iterative fashion, applying guidance and

feedback received from the study sponsor and

the study stakeholders.

• The Study Plan is typically approved and signed

by the sponsor (decisionmaker). Often, the Study

Plan is supported by a Study Management Plan

to guide, manage, and coordinate the efforts of

the effort.
Analyst’s Summary Guide 15

• The Study Management Plan may have

subordinate plans, to include an Analysis

Plan(s), Modeling and Simulation Plan, Data

Collection Plan, Configuration Management

Plan, Quality Assurance Plan, Review Plan,

Deliverable Plan, Security Plan, a Study Risk

Register, and Glossary.

Measures of Merit

Definitions

MoM is a generic term to encompass different classes

of measures. The measures are defined in hierarchical

levels related to each other, each in terms of its own

boundary. An orchestrated set of MoMs is typically

required for C2 assessments. The COBP has adopted

the following hierarchical set MoMs:

• Measures of Policy Effectiveness (MoPE) that

focus on policy or societal outcomes (e.g.,

transition measures, which focus on the progress

in the transfer of responsibilities to a follow-on

military force or civil agency, and normality

indicators which measure the quality of life of the

civilian population).

• Measures of Force Effectiveness (MoFE) that focus

on how a force performs its mission or the degree

to which it meets its objectives (e.g. loss exchange

ratios, combat effectiveness, number of targets

destroyed and desirable adversary behaviour).

• Measures of C2 Effectiveness (MoCE) that focus

on the impact of C2 systems within the
16 NATO Code of Best Practice for C2 Assessment

operational context (e.g., time to develop a

Course of Action, ability to provide information in

required format, impact of information

operations, and planned quality).

• Measures of Performance (MoP) that focus on

internal system structure, characteristics, and

behaviour (e.g., time to recognize an event,

correctness of perception and system reliability).

• Dimensional Parameters (DP) that focus on the

properties or characteristics inherent in the C2

system (e.g., bandwidth, data access times,

cost, and size; characteristics of organization

forms, attributes of personnel).

Principles

• A multi-faceted and sometimes multi-phased

approach is recommended, as no single

measure or methodology exists to satisfactorily

assess the overall effectiveness of C2.

• Established objectives for the assessment must

directly link to the MoMs.

• Selection of MoMs should consider assumptions,

constraints, models, tools, scenarios, other elements

of the analytic plan, and assessment processes.

• Identify selected MoMs, their thresholds and

standards, their means of collection, their

relationship to the assumptions, and their

imposed constraints in the assessment.
Analyst’s Summary Guide 17

• A detailed review of reliability and validity for the

selected measures will determine a level of

confidence for each MoM.

• The analyst must establish and measure control

variables to correlate MoMs in a spectrum of

multiple scenarios.

• The analyst must pay particular attention to

measurements related to the human element,

since variations in measurements may well

cause unacceptable levels of uncertainty.

• Analysis of uncertainties and measures of

central tendency and dispersion are significant

for C2 issues.

• For C2 acquisition analyses, measures should be

generated in parallel with system development,

so they can be used as standards for system

tests and operations.
18 NATO Code of Best Practice for C2 Assessment

Figure 2. Relationships of Measures of Merit

Human and Organizational Factors

The human dimension largely distinguishes C2

analysis from other military operations analysis. C2

analysis must deal with distributed teams including

military, interagency, coalition and other non-state

actors operating under stress and their varying

decisionmaking behaviours. In OOTW, particular

attention must be paid to the behaviour of and

interaction with non-military organizations, political

groups, and amorphous groups such as crowds and

refugees. Thus, the formulation of the problem and

the development of solution strategies cannot be
Analyst’s Summary Guide 19

completed without explicit consideration of both

human and organization issues.

• Human and organizational factors must be

considered as part of structuring the problem,

selecting MoMs, defining scenarios, developing

solution strategies, and selecting methods and

tools, and they should be reviewed throughout

the entire analytical process.

• The assessment team must include, or have

access to, experts from organizational science and

the various human science disciplines (such as

cultural anthropology, demography, sociology,

social and individual psychology, political science).

Human Factors

Human factors of interest fall into three major categories:

• Human behaviour related to performance degradation

and as a consequence of social interaction among

individuals or members of a group;

• Decisionmaking behaviour (cognitive questions)

including the cognitive complexity of issues and

the capacities of the commanders and other

decisionmakers of interest;

• Command style.

Human performance depends on psycho-physiological

variables (e.g., stress, sleep deprivation, hunger, and

alertness) and on ergonomic and external factors. Individual

and group behaviour is the result of social processes and

factors (e.g., fear, morale, values), and the cultural,

educational, and religious background of individuals.
20 NATO Code of Best Practice for C2 Assessment

• Any time human performance and/or behaviour

are at issue, parameters and/or models are

needed to reflect those issues. Unless

specialists can provide valid parameters from

work in other contexts or from field experience in

OOTW, some experimentation may be

necessary and appropriate to develop them.

• Decisionmaking that is automatable, contingent,

rule, or algorithmically based can be modelled

directly, but time requirements and error

representation must be incorporated if humans

are involved in the actual process.

• Complex decisions (e.g., courses of action in

response to events in the operations space) are

best being modelled with “human in the loop”

techniques. Closed form techniques for

modelling complex decisions are still in the

experimental stage.

• Accounting for differences in command styles

and how they affect military decisionmaking is of

special importance in OOTW which tend to be

multinational coalition operations. They may be

reflected by attributes such as the background of

commanders, their field experience, risk

attitudes, and organizational and orders style.

Organizational Factors

Organizational design reflects the interaction among

the tasks to be done, the people to do them, and

the systems and tools that support those people.
Analyst’s Summary Guide 21

The principal differences between organizations are

related to structure, function and capacity.

Structural differences include:

• Number of echelons in the command structure;

• The span of control for nodes in the command

structure; and

• The pattern of (formal and informal, permanent

and transitory) linkages between nodes

(hierarchical tree, spokes of a wheel, multiconnected, networked).

Functional differences include:

• The distribution of responsibility (location of

functional activities such as, e.g., intelligence,

logistics, civil military cooperation (CIMIC);

• The distribution of authority (ideally co-located

with responsibility);

• Functional specificity (of combat, support, or

service capabilities) versus integrated

capabilities (mission tailored task forces); and

• Degree of ambiguity in command relationships.

Capacity differences are related to:

• Personnel (quality, training, experience);

• Information and Communications systems and

architectures; and

• C2 field training and operational experience.
22 NATO Code of Best Practice for C2 Assessment

A systematic approach using a hypothesis-testing

logic and aided by organisation theory expertise

should be used for addressing organizational issues

in C2 assessment because of the large number and

many indirect effects of organizational variables.

Scenarios

Scenarios provide the context for the conduct of the

operational analysis and bound the arena of the

analysis. Scenarios consist of several static

dimensions shown in Table 1 and include the dynamic

evolution of events in time. Operational scenarios detail

threats, orders of battle, tactics, rules of engagement

and courses of action, deployments, reserves,

adversary forces, and non-combatants.

Attributes

Several prerequisites are essential before using

scenarios for C2 analysis:

• Approval: the analyst should strive for the

creation of a family of scenarios consistent with

high-level guidance and policy;

• Breadth: scenarios should reflect those factors

that are hypothesised to have a significant

impact on C2 issues;

• Capability: scenarios should stress C2

capabilities, including human and organisational

factors (military and/or civilian) where appropriate;

• Credibility: scenarios should include logical

assumptions about the problem under analysis; and
Table 1. The Static Dimensions of the Scenario Framew
24 NATO Code of Best Practice for C2 Assessment

• Plausibility: scenarios should represent

plausible situations.

Principles

• Analysts need to use multiple scenarios and

vignettes to cover or sample the interesting problem

space since no single scenario is sufficient.

• Analysts should explicitly identify and describe

the scenarios prior to the execution of a study. It

is best practice to revisit the scenario definition

periodically during the conduct of the study.

• For coalition C2 assessments, scenarios should

be developed or adapted by teams with

representatives from all participating nations.

• Scenarios should reflect C2 organizations and

infrastructure (including human issues),

processes, and systems relevant to the analysis.

• Scenarios must consider mission scope,

levels of hierarchy, and data flow

aggregration/disaggregation.

• Key scenario assumptions and constraints

should be identified and documented.

• Boundaries of the scenario spaces should be

defined by the problem being analyzed.

• Scenarios should reflect the factors that have

significant impact on C2, stress C2 issues, are

credible to the military, and are credible in terms

of civil-military objectives.
Analyst’s Summary Guide 25

Methods and Tools

This section considers the best methods and tools

(quantitative or qualitative) for assessing C2

processes, performance and effectiveness. The

section covers methods and tools used for analysis,

training or operations, each of which has different

requirements. Available methods can be categorised

into several classes, namely: data collection/

generation, data organisation/relationship, solving and

support. Table 2 illustrates this categorisation for a

sample of methods.

A key challenge for C2 assessment methods is to

make properly quantified linkages between MoP,

MoCE, MoFE, and MoPE. All C2 assessments

require a high level of creative problem structuring

and solving in making these linkages.

Method and Tool Selection

The following issues must be considered when

selecting methods and tools for C2 assessments.

• The inherent complexity of C2 assessment

problems, which calls for an orchestrated set of

complementary tools to cover the wide range of

variables involved (including hierarchies or

federations of models rather than dealing with all

issues in a single model).

• The requirement to explore a wide range of

scenarios and still represent C2 processes and

performance effectively.
26 NATO Code of Best Practice for C2 Assessment

• The appropriate treatment of human issues in

models and the balance of realism against

possible sources of uncertainty in the use of

“human-in-the-loop” techniques.

• The relative advantages of stochastic and

deterministic models (depending on the nature of

the assessment problem and other elements of

the solution approach).

• The appropriate balance for the representation of

friendly, adversary and other forces.

Table 2. Examples of Methods and Tools Categorized by Use
Analyst’s Summary Guide 27

• The challenge of VV&A and the development

of trust in methods and tools, especially in

novel applications areas such as unfamiliar

OOTW situations.

Best practice for the application of analysis tools

for C2 assessment is still emerging, but the following

criteria should be used when selecting models.

• Functionality-based criteria (including resolution,

completeness/scope, functionality, explicitness,

the ability to generate required MoM, validity,

and accreditation).

• Performance-related criteria (including

responsiveness, simplicity, preparation/use time,

data availability, interoperability with other tools,

resource requirements and credibility).

Utility of Different Types of Methods

and Tools

• The use of model federations, particularly with an

object-oriented approach, encourages

development using holistic and evolutionary

principles, under which modellers should capture

a complete model of the process, including parts

whose representation is still unclear (ready for

improvement as understanding develops).

• Agent-oriented modelling favours the capture of

the cognitive nature of command tasks. This is

important in establishing the linkage between

MoP and MoFE, but is constrained by the difficulty

in representing human performance, command

styles and organisational relationships.
28 NATO Code of Best Practice for C2 Assessment

• Very fast-running models can be used as

screening tools during problem formulation to

scan the problem space, allowing identification of

areas of concern for deeper analysis.

• The model-test-model or model-exercise-model

processes should be considered where the

assessment problem requires generation of new

data to validate existing models of C2 factors.

Assessment teams should be aware of the latest

advances in fast, agile modelling and exploit them.

Directions for Future Development

The following ideas should be considered in new

model developments (taking account of the

appropriate level of model aggregation).

• Understanding of adversary intent can be

represented by having a set of prescribed options;

• HQs with a local “picture” in each should be

explicitly represented to allow different

perception-based behaviours to be represented,

facilitating the study of factors such as

deception, shock, and surprise;

• Information should be represented as a

commodity as follows:

û a realistic flow of information around the

operational environment;

û the collection of information from multiple

sources and the tasking of information

collection assets;
Analyst’s Summary Guide 29

û the processing of information;

û C2 systems as entities on the battlespace/

operational space;

û unit perceptions built, updated, and validated

from the information available to the unit

through its information systems;

û the commander’s decision based on the

unit’s perception of the operational space;

û information operations actions and impacts

across all sides represented in the model.

Data, Meta-data, and the

Common Data Infrastructure

While the value of data for an individual study effort is

well understood by the analytic community at large,

the aggregated worth of data is still undervalued by

many. Data can be described as the fundamental

elements of information and knowledge that comprises

the corporate whole - consequently its aggregated

value particularly when addressed in a context larger

than an individual study is significantly greater than

the sum of the parts. Explicitly dealing with this issue

resulted in data receiving its own chapter in the code.

Data are factual information organised for analysis.

• The Data Taxonomy in the COBP lays out a

number of types of data including broad

categories of sources that will be of interest to

the analyst. The ability to determine the needed
30 NATO Code of Best Practice for C2 Assessment

data and the ability to assemble or collect these

data will constrain the solution strategy.

• A major challenge is to find, organise, verify,

process, and convert source data into the data

needed for the study. The team needs to know

(a) what data are needed in which structure; (b)

who owns these data; (c) security issues; and (d)

costs to buy, collect, or generate data.

• If the data are not available and can neither be

aggregated nor derived from the available

sources, it is good practice to use the knowledge

of subject matter experts to generate the

necessary data.

Meta-data are “information about information,” which

are used to document the data and related issues.

• The source of the data, the reliability, and assorted

assumptions and constraints must be captured in

standardized meta-data sets assigned to the data

sets comprising the needed information.

• Data that are collected without adequate

documentation are frequently viewed as suspect

or unusable. To avoid having good data thrown

away due to the lack of documentation,

acceptable community standards for

documentation must be employed. This

information has to be captured in standardized

meta-data sets.

• The initial data available will often be vague,

uncertain, incomplete, and contradictory. On the

other hand analysts prefer data to be sharp,
Analyst’s Summary Guide 31

certain, complete, and consistent. The

assumptions inherent in accomplishing this

transformation must be captured in standardized

meta-data sets.

A Common Data Infrastructure facilitates reuse

of data. It is based on agreed standards for data

formats, meta-data, and documentation.

• Data are central to the conduct of good

assessments. Adhering to data engineering

principles contributes to data reuse. Archiving of

data in retrievable form using standardized metadata sets is one essential component to facilitate

data reuse.

• As the data being used today by the analysts will

be the data needed tomorrow by systems

engineers, decisionmakers, and commanders for

their operations, it is good practice to align the

standardization processes and respective tool

sets between the analytical and operational

communities in order to facilitate the transfer of

data between those communities.

Risk and Uncertainty

Definition

Risk is commonly defined as the possibility of

suffering harm or loss. Uncertainty can be defined

as an inability to determine a variable or system state

or predict its future evolution.
32 NATO Code of Best Practice for C2 Assessment

There are risks associated with the uncertainties in

decisionmaker’s situation that are the subject of the

assessment and there are risks related to the conduct

of the assessment itself. Using the COBP should help

minimise the risks involved in C2 studies, but the

following risk and uncertainty issues remain and are

intrinsic to any C2 study.

Scope of Risk and Uncertainty

• The explicit treatment of risk and uncertainty is

best practice in all studies and of particular

importance in C2 assessment. It is advisable not

to skip risk analysis, even when time and

resources are limited, and this should be

stressed to decisionmakers and study sponsors.

• Perceptions of risk and uncertainty can

substantially differ from objective assessments and

this should be explicitly considered. Analysts

should find out how study sponsors perceive risks.

• A thorough understanding of study variables is

essential for effective treatment of uncertainty.

OOTW studies typically have less well-formed

quantitative factors and more qualitative factors.

• The assessment team must be aware of sources

of uncertainty in all aspects of a study, including

those related to the assumptions and limitations

inherent in parameter values, models, scenarios,

data structures, and the boundaries used to

scope and focus the study. Humans involved in

the assessment process make assumptions that

should be identified, documented and analysed.
Analyst’s Summary Guide 33

• In C2 assessments, analysts need to be

particularly alert to the possibility of chaotic

behaviours arising from dynamic interactions of

human and organisational factors, and

discontinuous, non-linear divergence arising from

the multiple options for choice of course of action.

Treatment of Risk and Uncertainty

• Where uncertainties cannot be reduced by

acquiring more information, multi-factorial

sensitivity analysis should be used to establish the

regions for which the results are valid and to isolate

those factors that may introduce uncertainty.

• Sensitivity analysis should not only deal with

statistical variance but should also consider

qualitative variations in models, perspectives and

assumptions. A range of analytic tools relevant

to sensitivity analysis is identified in the full text

of the COBP.

• Checklists and structured appraisal help to

maintain study rigour, but neither is a substitute

for critical thinking in the specific study context.

• The team should expect a complex and partly

hidden set of risks to C2 studies and serious

efforts should be made to illuminate them in an

explicit risk-based analysis with portfolio-based

solutions. Such analysis needs metrics for risk

and failure as well as MoMs.

• The assessment team should make a complete

list of risks and treat them in appropriate detail in
34 NATO Code of Best Practice for C2 Assessment

a risk register for the study. A generic risk register

has been produced to support the COBP.

Communication of Risk and Uncertainty

Communication of uncertainty and risk is particularly

important for C2 assessment products. Such

communication must take account of the complexities

of the subject and human limitations in understanding

risk and uncertainty.

A continuing dialogue with stakeholders about

uncertainty will facilitate common understanding,

including the impact of uncertainty on the robustness

of conclusions and methods for mitigating them.

Products

Assessment products communicate results to

decisionmakers and stakeholders, establish the

credibility of the effort, and provide a lasting record of

the project. Typical C2 assessment products include

the Study Plan, Periodic Status Reports, the Project

Journal and a Final Report and Briefing, which include

a variety of supporting data and documents.

• The Study Plan, a living document kept current

throughout the assessment, explains the problem

under analysis, the solution strategy, the tasks

involved, and how they fit together. This will

include, as appropriate, plans necessary for the

assessment, such as data collection and data

analysis plans. The elements of a Study Plan are

enumerated in the Solution Strategy section.
Analyst’s Summary Guide 35

• Periodic Status Reports describe the state of the

effort over time and can be assembled to create

a record of the work accomplished, problems

encountered, results from peer reviews, and the

adjustments made by the team.

• The Project Journal records interactions with

professionals outside the core team and

captures the analytic assumptions and decisions

made by the assessment team (such as choice

of tools and selection of data or methods for

generating data) over time.

• The Final Report presents the results of the effort

(findings, recommendations, and lessons

learned) and incorporates, as necessary, items

from the other key documents. It will also

typically include appendices that provide

supporting detail such as participants in the

effort, references, glossary, list of acronyms and

abbreviations, as well as technical material such

as data collection instruments.

• The Final Briefing, which typically is circulated

more broadly than the Final Report, will provide

a summary of the most important aspects of the

project, including the problem statement,

solution strategy, research accomplished,

findings, recommendations, and lessons

learned. Best practice requires that this

document be richly annotated.

• Data, models, supporting scenario materials, and

other products necessary to make the results

credible and authoritative or that may be of value

to other researchers or assessment teams
36 NATO Code of Best Practice for C2 Assessment

should be archived and made available as

broadly as possible.

In general, all the products should be circulated as

widely as possible so that others can benefit from the

work accomplished and redundant projects avoided.

This requires that specific effort be made to downgrade

or sanitize products and to identify those who would

benefit from reviewing them.

1North Atlantic Treaty Organisation (NATO) NATO Code of Best

Practice for C2 Assessment, 2002.

2North Atlantic Treaty Organisation (NATO) Annex B to MC

Guidance for Defence Planning. MC-299/5. (1996)

3North Atlantic Treaty Organisation (NATO) Bi-MNC C2 plan

part 2—Command and control requirements. (1998).
37

ANNEX A

Decisionmaker’s

Temporal

Question List

A

ttached is a list of questions that have been

included in the decisionmakers’ guide. It would

be prudent if the analyst is prepared to respond to

these questions.

This annex contains key questions that a

decisionmaker should ask the C2 assessment team.

These questions are organized temporally according

to the following phases of a study (prior to the study,

at initial review, after first iteration, and at final report).

Prior to the Study:

• Do you understand what decisions(s) I have to

make, when I have to make them, and the context

within which the decision(s) will be made?

• Do you need any information or authorization

from me?

• Who will be on the study team?

û Are there adequate skills, experience present

in the team?
38 NATO Code of Best Practice for C2 Assessment

û For OOTW studies in particular, are there

adequate social scientific skills in the team?

• Who are the key organizations/individuals with

whom you plan to interact (e.g., stakeholders,

data providers, review team)?

û Have you coordinated the Terms of

Reference with them?

û How do you plan to interact with them?

• Particularly for OOTW studies, how will you

acquire the requisite knowledge of the culture/

historical context?

• How will you undertake problem formulation? e.g.,

û What products will you review/mine?

û What methods and tools are applicable

• When will key events occur (e.g., reviews,

production of interim products)?

At the Initial Review:

• What do you perceive the “real” issues to be?

• What assumptions do you plan to make to

scope the effort?

• What do you plan to use for

û High level MoMs?

û Scenarios of interest?
Analyst’s Summary Guide 39

• Have you identified any additional organizations/

individuals with whom you plan to coordinate?

• How do you plan to attack the problem?

û What methodology will you employ?

û How will you treat the diverse aspects of a

mission capability package (including concept

of operations, organization, doctrine, C2

approach, systems, personnel, facilities, in

other words everything needed to field a real

capability.)

û How do you plan to address organization/

human issues?

û What specific methods and tools will you

employ? Why do you think they are

appropriate?

• What data will you employ? Where will you get

them? Why do you think they are appropriate?

How do you plan to make the data accessible to

others? How do you plan to depict the results of

the study?

At First Iteration:

• What specific MoMs were selected? What

relationships were established among the

MoMs?

• What range of scenarios were selected? Why?

• What plans do you have to illuminate uncertainty/

sensitivity?
40 NATO Code of Best Practice for C2 Assessment

• What feedback did you receive from the

independent review team? What steps did you

take to respond to it?

• What do you plan to do on subsequent iterations?

û Use additional tools?

û Consider additional scenarios, assumptions?

û Modify assessment boundaries?

At Final Report:

• What are the major findings, recommendations?

• What are the key points of uncertainty/sensitivity?

• What issues were not addressed that should be

treated in subsequent assessments?

• What key lessons did you learn with respect to

methods, tools, and data?

• What steps are you going to take to disseminate

key insights, products to the broader community?

• Are there any voids/issues that warrant

further research?