A DoD CCRP/NATO Collaboration

This major revision to the Code of Best Practice for

C2 Assessment is the product of a NATO Research

and Technology Organisation (RTO) sponsored

Research Group (SAS-026). It represents over a

decade of work by many of the best analysts from

the NATO countries. A symposium (SAS-039) was

hosted by the NATO Consultation Command Control

Agency (NC3A) and provided the venue for a

rigorous peer review of this code.

This publication is the latest in a series produced

by the Command and Control Research Program

(CCRP) under the auspices of the Assistant

Secretary of Defense (C3I). The CCRP has

demonstrated the importance of having a research

program focused on the national security

implications of the Information Age. The research

sponsored and encouraged by the CCRP contributes

to the development of the theoretical foundations

necessary to support the Information Age

transformation of the Department. Other CCRP

initiatives are designed to acquaint military and

civilian leaders with emerging issues related to

transformation. This CCRP Publication Series is a

key element of this effort.

Check our Web site for the latest CCRP activities and publications.

This code is also available on our Web site.

www.dodccrp.org
NATO Code

of

Best Practice

for

C2 Assessment

Decisionmaker’s

Guide
1

NATO CODE OF

BEST PRACTICE FOR

C2 ASSESSMENT

Decisionmaker’s

Guide

Introduction

Purpose

N

ATO has produced a Code of Best Practice

(COBP) 1 in order to facilitate high quality

assessments supporting decisionmaking in the area of

Command and Control (C2). The COBP is the product

of international collaboration drawing together the

operational and analytical experience of leading military

and civilian defence experts from across the NATO

nations. The COBP enhances the understanding of best

practice and outlines a structured process for the

conduct of operational assessment for C2, which is the

core capability of Information Age defence and security.

The command and control aspects of military capability

are difficult to assess. Use of the COBP will:

• Increase the likelihood of quality products
2 NATO Code of Best Practice for C2 Assessment

û Complete

û Relevant

û Transparent (visible)

û Credible (believable)

û Authoritative (rigorous)

• Reduce risk/cost associated with producing the products

The COBP emphasizes the critical role of

decisionmakers in the conduct of good assessment

and describes a framework to help to structure this

participation. The COBP should be used by

decisionmakers to frame study requirements, provide

additional guidance, and review products. Use of the

COBP should be regarded as a community standard

for all C2 assessments.

This decisionmaker’s guide aims to help

decisionmakers who commission, fund, oversee, and

employ C2 assessments. It provides 1) an executive

summary of the NATO Code of Best Practice (COBP)

for C2 Assessment, 2) information on how

decisionmakers can best ensure that the COBP is

adhered to by those carrying out C2 assessments,

and 3) guidance on the limits of assessment and the

use of C2 assessment results.

Background

The initial version of the NATO COBP for C2

Assessment was published in 1998, as the culmination

of several efforts by various NATO study groups to

address C2 assessment challenges and practices.
Decisionmaker’s Guide 3

While this initial version focused on Article V related

operations, this version of the COBP addresses all

issues of C2 assessment across the full spectrum of

operations, to include Operations Other Than War

(OOTW). Experience with the initial version of the

COBP highlighted the need to produce a short,

executive-level version of the COBP aimed specifically

at decisionmakers.

Command and Control Assessment

C22 is recognized as a critical element of successful

military operations and a key aspect of Information

Age transformation. Until recently, however, physicsdominated issues of military operations, rather than

C2 ones, have been the primary, almost exclusive,

focus of military analysis and assessments. This,

coupled with the inherent complexity of C2 (which

involves both the information and cognitive domains),

has presented the assessment community with

challenges that are less well researched and

understood and with a tool kit that is clearly lacking.

The NATO COBP for C2 Assessment, therefore, has

been developed to help C2 analysts deal with these

new Information Age assessment challenges so that

they can improve their ability to take on analyses of

requirements, analyses of alternatives, research on

new C2 concepts and capabilities, and support real

world operations. This guide addresses how the

decisionmaker, as a customer of C2 assessments,

plays a key role in such assessments.
4 NATO Code of Best Practice for C2 Assessment

Role of the Decisionmaker

The decisionmaker has a critical role in any assessment.

The decisionmaker provides the initial articulation of the

problem or issue at hand and establishes the conditions

under which the effort takes place. In addition, it is the

decisionmaker who determines how the assessment

results are interpreted and whether they influence

decisions. When the decisionmaker interposes

subordinates between him and the assessment team

he must understand the resulting risks and take action

to mitigate them where possible.

The decisionmaker’s role in a C2 assessment is most

definitely a hands-on one. Experience shows that C2

assessments that have had active decisionmaker

participation are more likely to result in products that

satisfy both decisionmakers and the members of the

assessment team. Annex A provides some key

questions to encourage the necessary discussions

between the decisionmaker and the assessment team

at various points in the study effort. This guide will

provide, at a high level, information that will assist the

decisionmaker in assessing the answers. The full

COBP provides a more detailed treatment of these

items and the decisionmaker may want to consult the

full COBP as appropriate.

The interaction between the decisionmaker and the

assessment team is not only critical to getting the effort

off on the right foot but is essential if the decisionmaker

is to fully understand the results of the assessment

and the assumptions that underlie these results.

Furthermore the assessment team will, at various times

during the study, have important choices to make.
Decisionmaker’s Guide 5

Decisionmaker involvement in these choices can make

the difference in a study’s success or failure

Establish a reasonable set of

initial conditions

Stay involved throughout the study

Maintain consistent focus on the

real problem

Organization of the Guide

The remainder of this guide is organized into the

following four sections:

• Preparing for Success

• Overview of the Assessment Process

• Monitoring the Assessment

• Reviewing the Products

In Preparing for Success, the things that the

decisionmaker can do to enhance the likelihood that

the effort is successful are discussed. In the Overview

of the Assessment Process, each of the component

steps in the assessment and their relationships one to

another are discussed. In Monitoring the Assessment,

key considerations are identified, any one of which

could be a determining factor in the success or failure

of the effort. Reviewing the Products addresses the

nature of the products a C2 assessment should be

expected to produce.
6 NATO Code of Best Practice for C2 Assessment

Preparing for Success

The are two major prerequisites for successful C2

assessments. First, a clear and unambiguous

statement of the decision or issue to be addressed.

Second, a well qualified assessment team with

adequate time and resources. While this may seem

obvious, all too often either one or both of these two

conditions are not met. In these cases, it may be best

not to undertake an effort that will be doomed to failure

from the start. This extreme option can be avoided by

properly scoping a study to make the effort feasible.

To begin with, every effort needs to be made to ensure

that the assessment team understands the problem

or issue they are to address. Any statement of the

problem will contain a set of implicit assumptions and

constraints that may be known and understood by the

decisionmaker but not by members of the assessment

team. An effort should be made to make these explicit.

C2 assessments are often undertaken on very

“ambitious” schedules with inadequate resources.

Experience has shown that C2 assessments involve

great complexity, difficulty in getting appropriate data,

and the need for significant sensitivity analysis to deal

with uncertainty and risk. Therefore, plan

conservatively. This will avoid the situation where the

effort runs out of time and resources before significant

parts of the assessment can be completed or the

assessment team is forced to not consider or measure

or analyze key, often driving aspects of the problem.

An assessment team will rarely have all of the expertise

or experience needed to do the job right. The same is

true for empirical data and the results of other efforts.
Decisionmaker’s Guide 7

Access to the right people and information is every bit

as important as having a well articulated problem and

adequate time and resources. It is essential that these

needs are anticipated before the study begins so that

appropriate arrangements for access are arranged.

Failure to accomplish this at the start often results in

delays, cost overruns, and compromises the quality

of the products. Additionally, the routine collection of

data during exercises and military operations will

facilitate a standing collection of data available for

future C2 assessments.

The NATO Code of Best Practice for C2 Assessment

has proven itself to be useful to both highly experienced

analysts and those without much experience. It is

recommended that decisionmakers insist that the

assessment team review the COBP before they

develop their study plans and inform the decisionmaker

if, when, and why the advice and or processes

contained in the COBP will not be followed. Any

deviation from the best practices described in the

COBP carry risks and these risks need to be

understood by decisionmakers before they agree to

waive portions of the COBP for a particular effort.

Ensure that the problem is

understood

Allocate adequate time and

resources to the effort

Plan conservatively

Ensure study leadership has

appropriate breadth and experience

Ensure access to needed expertise

and data

Mandate the use of the COBP
8 NATO Code of Best Practice for C2 Assessment

Overview of the Assessment Process

Figure 1 depicts the C2 Assessment Process. This

process is iterative. It is applicable to any type of C2

assessment, regardless of the scope or focus of the

assessment. In fact, each of the steps in the

assessment process will be revisited several times

during the course of the effort.

Figure 1. C2 Assessment Process

In Problem Formulation, the Assessment Team

answers the question of what is to be addressed by

the effort. The Solution Strategy addresses how this

will be accomplished. The remainder of the process

carries out the solution strategy and produces the

assessment products. Problem Formulation and

development of the Solution Strategy for C2

assessments should be an iterative process.
Decisionmaker’s Guide 9

The decisionmaker should schedule an initial review with

the assessment team to ensure that they are on the right

track after the assessment team has gone through at

least one full iteration, specifying not only what they will

do but also thinking about each of the remaining steps in

the process in some detail. Prior to this review, the

assessment team should have their study plan (this

consists of both a statement of the formulated problem

and their solution approach) peer reviewed.

It should be expected that modifications will need to

be made in either the solution approach or in the

formulated problem as greater understanding is

gained. The decisionmaker must be informed when

significant changes are made. Such changes may for

example be the result of discovering that some data

needed to support a particular measure of merit (MoM)

is not available. In such cases the study plan needs to

be revised to develop a surrogate MoM, collect the

appropriate data, or conduct a sensitivity analysis.

Risks and uncertainties are an inherent part of C2

assessments. These risks and uncertainties can not

be completely eliminated, therefore they must be

managed. It should be noted that each iteration ends

with a consideration of residual study risk.

Decisionmakers should expect that any

recommendations to significantly change the study

plan should be accompanied by an explanation of the

risks to study objectives, schedule, and costs. Hence,

the continuous involvement of the decisionmaker

ensures that the study effort remains appropriately

focused and that the study team is kept aware of

decisions and developments that influence the study.
10 NATO Code of Best Practice for C2 Assessment

Involvement of decisionmaker is key

Process must be iterative

Initial review needed to ensure

proper start

Look for explicit treatment of risk

and uncertainty

Expect adoption of study plan as

insight is gained

Monitoring the Assessment

The first two steps in the assessment process are

Problem Formulation and Solution Strategy. The

product of these two steps is the study plan which

describes the what and how of the assessment effort.

Participating in Problem Formulation

Effective problem formulation is fundamental to the

success of any assessment, but particularly C2

assessment because C2 issues are often ill-defined

and complex. The problem formulation process

identifies the context of the study and aspects of the

problem-related issues for assessment. The context

of the study includes the geopolitical environment, aims

and objectives of the study, and the decisions to be

supported by the assessment. Problem-related

aspects include the issues to be addressed,

assumptions, high-level MoMs, independent variables

(both controlled and uncontrolled), and constraints on

variables. It must be recognized that problem

formulation takes time and must precede development

of the assessment concept or selection of the solution

approach (including methods and tools). The
Decisionmaker’s Guide 11

decisionmaker plays a critical role in problem

formulation by providing an understanding of the real

problem to be addressed and how study results will

support pending decisions.

The team should be expected to quickly cover the whole

problem and produce an initial problem formulation to

prevent premature narrowing of the assessment and

to allow a common understanding among all team

members. This will identify the key issues to be

addressed and define the context of the study.

As a result of this process, the assessment team may

discover important issues related to the problem that

need to be discussed with the decisionmaker.

All key variables should be included

in the assessment

Relationships should be known or

hypothesized

All controllable variables should be

identified

Key assumptions should be made

explicit

Monitoring Solution Strategy

A solution strategy consists of the specification of a set

of sequential and parallel analytical steps, often involving

several methodologies and tools. The solution strategy

should begin with what is known, and by its execution,

lead to what the decisionmaker desires to know—insight

into the issue(s). The solution strategy can be simple,

moderately complicated, or extremely complex.
12 NATO Code of Best Practice for C2 Assessment

The development of a solution strategy is an iterative

process that should strike an artful balance between

what the team would like to do and what is possible to

do, given the state of the art, available data, tools,

schedule, and resources available. The team first

elaborates on the measures that are to be evaluated

in the study. Using these measures and consideration

of human and organizational factors, a conceptual

model of the assessment is developed. The conceptual

model is based on the issues formulated in the problem

formulation and is not driven by the availability of tools

and data. The conceptual model is the embodiment of

our current understandings and may be changed by

the study findings.

Frequently a solution strategy becomes complex,

requiring the team to decompose the problem into

parts, each of which requires assessment with its own

set of tools. Taken together, the solution strategy

consists of the MoMs, relevant human and

organizational factors, specification of scenarios, data

collection requirements, and methods and tools to be

used in the assessment.

Solution strategy should be clearly

articulated

Potential problems should be

identified with contingency plans

Conceptual model should not be

constrained by available data or tools

Conceptual model initially

represents current understandings

and may evolve
Decisionmaker’s Guide 13

Approving the Study Plan

The solution strategy is documented in a study plan that

links the problem formulation and solution strategy

together in one plan. The study plan shoud be complete

and include: MoMs, Human Organizational Factors,

Scenarios, Methods and Tools, Data consideration, Peer

reviews, in-process reviews, risk and uncertainty, and

products. The Study Plan should be developed in an

iterative fashion, applying guidance and feedback

received from the decisionmaker and other stakeholders.

It is important that the Study Plan is peer reviewed.

The study plan, presented by the assessment team to

the decisionmaker, should be approved before significant

resources are expended. Often the study plan is

supported by a study management plan to guide,

manage, and coordinate the effort. The study

management plan may have subordinate plans, to

include an analysis plan, modeling and simulation plan,

data collection plan, configuration management plan,

quality assurance plan, review plan, deliverable plan,

security plan, as well as a study risk register, and glossary.

Study plan should be complete

Study plan should be realistic in

terms of schedule and resources

and allow for multiple iterations

Conducting In-Process Reviews

The in-process reviews are critical to the success of

the effort. They should be at key junctures including:

After initial problem formulation, After initial version of

study plan, Prior to data collection, Between collection
14 NATO Code of Best Practice for C2 Assessment

and assessment, Prior to draft products. These are

an opportunity for the decisionmaker to make sure that

the study, if completed as planned, will answer the

question of interest and also to make appropriate

changes in the study plan to correct its deficiencies.

Furthermore, reviews help the decisionmaker to

harmonize studies conducted in parallel under their

aegis dealing with interrelated issues.

During the initial review the decisionmaker needs to

make sure that all of the key assumptions related to

the assessment are fully discussed and are

appropriate. In addition, these reviews should include

a full discussion of study and decision risks.

A discussion of the results of the peer review and a

review of the products that the study will produce

should also be done at this time.

Conduct reviews at key junctures:

After initial problem formulation

After initial version of study plan

Prior to data collection

Between collection and

assessment

Prior to draft products

Accepting Measures of Merit (MoMs)

MoMs are central to meeting the objectives of all

assessments. Their development is particularly

challenging for C2 assessments, given the nature of

the problems addressed. It is recognized that no single

measure, or class of measures, is sufficient. An

orchestrated set of MoMs is typically required for C2
Decisionmaker’s Guide 15

assessments. The COBP has adopted the following

hierarchical set of MoMs:

• Measures of Policy Effectiveness (MoPE) that

focus on policy or societal outcomes

• Measures of Force Effectiveness (MoFE) that

focus on how a force performs its mission or the

degree to which it meets its objectives

• Measures of C2 Effectiveness (MoCE) that focus

on the impact of C2 systems within the

operational context

• Measures of Performance (MoP) that focus on internal

system structure, characteristics and behaviour

• Dimensional Parameters (DP) that focus on the

properties or characteristics inherent in the C2 system

A good C2 assessment will establish relationships

among the measures used in the assessment.

Establishing these relationships between types of

measures, however, is among the most challenging

parts of any assessment. It is important to recognize

that even approximate relationships among the right

MoMs are far preferable to “precise” relationships

between MoMs that do not adequately reflect the key

issues at hand. Criteria have been developed that help

to ensure that the measures that are selected are both

valid and reliable. For a measure to be considered

valid, it must be mission oriented, realistic, appropriate,

inclusive, discriminatory, meaningful, simple, relevant,

and generalisable. For a measure to be considered

reliable, it must be discriminatory, measurable/

observable, quantitative, objective, sensitive, and

consistent. A proposed measure may be reliable but
16 NATO Code of Best Practice for C2 Assessment

not valid, or it may be valid but not reliable. The

decisionmaker should insist that, to the extent possible,

the measures to be used are valid, reliable, and, if

calculated, will provide desired information. The risk

assessment should characterize the degree to which

the MoMs may not be valid and reliable and describe

the mitigating actions taken.

Ensure MoMs are valid, measured

reliably, credible, and complete

Considering Human and

Organizational Factors

The human dimension largely distinguishes C2

assessment from other military operations

assessment. C2 assessment must deal with distributed

teams including military, interagency, coalition and

other non-state actors operating under stress and their

varying decisionmaking behaviours. In operations

other than war, particular attention must be paid to

behaviour of and interaction with non-military

organizations, political groups, and amorphous groups

such as crowds and refugees. Thus, the formulation

of the problem and the development of solution

strategies cannot be completed without explicit

consideration of both human and organizational issues.

Human performance affects behaviour and vice versa.

Human performance depends on psycho-physiological,

and cognitive variables such as stress, fatigue, sleep

deprivation, hunger, alertness, personality, and

predisposition, as well as on ergonomic and external

factors. Individual and group behaviors are also the

result of social interaction impacted by fear, morale,
Decisionmaker’s Guide 17

value systems, culture, education, and religious

backgrounds of individuals. When human performance

and/or behavior is at issue, parameters and/or models

will need to reflect those issues. Particularly critical is

how human decisionmaking is addressed in the

assessment, and/or in models used in the assessment.

The human factors mentioned above, as well as the

command style, risk style, and other command attributes

that impact human decisionmaking, must be accounted

for in the modeling and assessment. The decisionmaker

can be invaluable in this regard if he/she is also the

subject of assessment by making sure the assessment

team understands their approach to decisionmaking.

There is a strong direct link between human and

organizational issues. Properly done, organizational

design reflects the interaction of tasks to be done, the

people available to perform them, and the systems or

tools that support them. Because of the need for coevolution organizational design is often a key

independent variable. Organizations, then, depend on

the capabilities, training, and experience of the people

in the C2 system. Organizational factors must be

addressed as part of most C2 analyses. Organizational

factors include structural (e.g. number of echelons, span

of control), functional (e.g. distribution of functions,

information, and authority), and capacity (e.g. personnel,

communications) factors. The large number of

organizational variables that may be relevant to C2

assessment must be approached carefully and

systematically. When possible, organization theory

expertise should be brought into the team. The

assessment team will be challenged to identify factors

that are particularly relevant to their C2 assessment,

and to identify and employ appropriate tools.
18 NATO Code of Best Practice for C2 Assessment

Explicit consideration of human/

organization performance

Inclusion of appropriate expertise

on the team

Approving Scenarios

The selection of a proper set of scenarios is critical to

the assessment. Scenarios consist of four elements—

a context (e.g., a characterization of a geopolitical

situation), the participants (e.g., intentions, capabilities

of friendly, hostile, neutral), the environment (e.g.,

natural—weather and man made—mines), and the

evolution of events in time. Scenarios can be approved

or unapproved. Some are operational scenarios,

meaning they contain additional details and may exist

in a model. Sometimes smaller scenarios, called

vignettes are developed for analytic use.

In C2 assessments, the purpose of scenarios is to

ensure that the assessment is informed by

decisionmaker planning assumptions and the

appropriate range of opportunities to observe the

relevant variables and their interrelationships. Although

the ideal would be for the assessment to be scenario

independent, rarely does this happen due to the

breadth and complexity of C2. Therefore, scenarios

must be considered throughout the assessment

process, especially during problem formulation. In

essence, the role of scenarios is to define a set of

conditions and restrictions to enable credible

assessment as well as to create a structure within

which the results of the assessment can be understood

and interpreted.
Decisionmaker’s Guide 19

Scenarios must be designed or selected to address

C2 under a broad range of circumstances. C2

assessments need to use multiple scenarios, no single

scenario is sufficient. For C2 assessment, scenarios

should reflect the C2 organization, processes, and

systems relevant to the assessment. In selecting

scenarios for C2 assessment, the analyst should

ensure that the scenarios reflect the factors that have

significant impact on C2, stress C2 issues, are credible

to the military, are credible in terms of civil-military

objectives, and will facilitate the study design process.

Note that this implies a broader selection of scenarios

than is normally contained within current sets of

“approved” scenarios. Due to their critical importance

to the study, the decisionmaker must pay particular

attention to the design and selection of scenarios.

Ensure:

Multiple scenarios

The set of scenarios cover the

range of relevant situations

Understanding the Methods and Tools

There is a broad range of methods and tools available

to be applied to C2 assessment. The methods and tools

fall into one or more of four categories—data collection/

generation tools (e.g., simulations, exercises,

experiments, expert elicitation, real world operations),

data organization tools (e.g., causal mapping, multicriteria decision analysis, neural nets), “solving” tools

(e.g., mathematical analysis, linear programming, goal

programming), and support tools (e.g., data analysis,

databases, checklists, spreadsheets). Although the
20 NATO Code of Best Practice for C2 Assessment

focus of recent past research has been on the

development of computer simulations, virtually any of

the analytic tools in the analyst’s inventory could

potentially be applied to C2 assessment.

The development or selection of tools for C2

assessment should be based on evaluation of

candidates against functionality-based and

performance-based selection criteria. Functionalitybased selection include resolution/detail, completeness/

scope, the functionality coverage provided, the

explicitness of entity representation, the ability to

generate appropriate MoMs, and whether the tool has

been verified, validated, and accredited (VV&A) for the

intended use. Performance-based selection criteria

include responsiveness, simplicity, time to prepare/use,

data availability, interoperability with other tools,

resources required, and credibility with customers and

users. Decisionmakers should refer issues regarding

the selection of methods and tools to peer review.

Although the assessment team will employ any tool that

assists in addressing study issues, credible computer

simulations are what most analysts seek. The

development of C2 simulations, especially those that

link C2 to force effectiveness, has been the subject of

much research recently. Decisionmakers should be

aware that, although significant progress has been

made, many challenges still exist in the modeling of

C2. Among these challenges are representation of

human behaviour, linking/federating models,

representing adversarial entities, dealing with

uncertainty in model representations, conduct of

sensitivity analysis with models, and the VV&A process.
Decisionmaker’s Guide 21

The assessment team should seek approval from the

decisionmaker for the models that will be employed

as part of the assessment. Decisionmakers should ask

how C2 is represented in proposed models and how

model outputs will provide the data necessary to

address the C2 issues under study.

Study needs should drive tool

selection, not vice versa

Seek clear rationale for tool selection

Multiple tools are normally needed

Demand explicit linkage from

methods and tools to study issues

Comprehending the Data

The role and importance of data in C2 assessments is

underestimated by many. The ability to determine what

data are needed and the ability to collect these data in

fact determine in large part the solution strategy (an

example of the importance of thinking of the assessment

process as iterative in nature). There may be many types

of data required, to include scenario data, human

performance data, and systems performance data.

Decisionmakers and analysts prefer data to be sharp,

certain, complete, and consistent, but recognize this

will not always be the case. For C2 assessment, it is

particularly difficult to meet these criteria.

Hence, not all C2 related data required/desired by the

analysts will be readily available. Some of it will have

to be aggregated or derived from other sources, or

perhaps generated from original sources. C2 related

data can be obtained from a variety of sources to

include official sources (e.g., military databases), open
22 NATO Code of Best Practice for C2 Assessment

sources (e.g., Internet), legacy studies, and subject

matter experts. The assessment team must

understand what data are needed and in what form,

who owns the data if it already exists, and the costs

involved in buying, collecting, or generating required

data. These costs can be significant. The

decisionmaker can help the assessment team by

assisting in identifying available or potential data

sources. Furthermore, the quality of the study results

is influenced by the quality of data used as much as

by the selection of respective methods and tools.

Articulate the assumptions related

to collection

Understand data anomalies and

adjustments made to data

Understand data sources

Look for data distributions, not

just averages

Help the assessment team get the

data they need

Understanding Risks and Uncertainties

There are risks associated with the decision at hand

and there are risks related to the assessment process.

Failure to deal effectively with both of these types of

risk will jeopardize study goals.

Risk is commonly defined as the possibility of suffering

harm or loss, to include opportunity loss. Risk often

has a negative connotation, yet “taking risks” can also

be positive. Uncertainty can be defined as an inability

to determine a variable value or system state or predict

its future evolution. Uncertainty is inherent in risk. Risk
Decisionmaker’s Guide 23

and uncertainty are especially prevalent in C2 and in

C2 assessment. C2 issues are inherently complex and

have many interacting factors. Additionally, C2

assessment is effected by uncertainties in scenarios,

data, and models.

Risk can be dealt with in C2 assessments by either

reducing the uncertainty that underlies the risk, by

embracing and accounting for residual uncertainties,

or by communicating the risk involved and adopting a

decision strategy that mitigates the risks. Uncertainty

can be reduced in C2 assessment in a number of ways,

to include conducting certification of data, conducting

proper VV&A of models, and in conducting sensitivity

analysis. The assessment team should make a serious

effort to illuminate the risks and to conduct an explicit

risk-based assessment (consideration of multiple

potential solutions). With problems involving human

decisionmaking the analyst must be aware of the

diversity of courses of action that are possible as a

scenario evolves. The analyst should also attempt not

to unduly bound the problem during problem formulation

to reduce risk. The decisionmaker can help the analyst

with these efforts by helping to identify risks and

uncertainties and by discussing his or her own approach

to risk and uncertainty regarding the study issues and

the decisions to be made based on the assessment.

Failure to adequately deal with risk

jeopardizes study goals

Sensitivity analyses should be

provided and explained
24 NATO Code of Best Practice for C2 Assessment

Reviewing the Products

Assessment products include the study plan, periodic

status/progress reviews, and the final report. The

decisionmaker has an important role in each. The study

plan should be presented at the initial review and the

decisionmaker should approve it. The study plan

should serve as a shared understanding between the

decisionmaker and among the study participants.

However the iterative nature of good C2 assessments

means the study plan should be flexible.

Periodic reviews should be chaired by the

decisionmaker. The study plan should go through a

peer review process prior to finalization. In addition to

these products, there are a number of other products

that best practice demands be produced and maintained

during the course of the study. These include a project

journal, a study design and implementation plan, a data

collection plan, a data analysis plan, and a risk register.

All products should be archiveable and readily

accessible by the community, within the constraints of

security. It is recommended practice that a version be

prepared at the lowest classification possible to allow

for the widest distribution possible and reuse in the

assessment community.
Decisionmaker’s Guide 25

The decisionmaker should be able to

brief the study and explain its results

Data should be saved in a form

amenable to re-use

Study products should be peer

reviewed

A plan should be developed for

archiving and dissemination

1North Atlantic Treaty Organisation (NATO) NATO Code of Best

Practice for C2 Assessment, 2002.

2NATO defines C2 as “The Organization, Process, Procedures,

and Systems necessary to allow timely political and military

decisionmaking and to enable military commanders to direct

and control military forces.”
27

ANNEX A

Decisionmaker’s

Temporal

Question List

T

his annex contains key questions that a

decisionmaker should ask the C2 assessment

team. These questions are organized temporally

according to the following phases of a study (prior to

the study, at initial review, after first iteration, and at

final report).

Prior to the Study:

• Do you understand what decisions(s) I have to

make, when I have to make them, and the

context within which the decision(s) will be

made?

• Do you need any information or authorization

from me?

• Who will be on the study team?

û Are there adequate skills, experience present

in the team?
28 NATO Code of Best Practice for C2 Assessment

û For OOTW studies, in particular, are there

adequate social scientific skills in the team?

• Who are the key organizations/individuals with

whom you plan to interact (e.g., stakeholders,

data providers, review team)?

û Have you coordinated the Terms of

Reference with them?

û How do you plan to interact with them?

• Particularly for OOTW studies, how will you

acquire the requisite knowledge of the culture/

historical context?

• How will you undertake problem formulation? e.g.,

û What products will you review/mine?

û What methods and tools are applicable?

• When will key events occur? (e.g., reviews,

production of interim products)

At the Initial Review:

• What do you perceive the “real” issues to be?

• What assumptions do you plan to make to scope

the effort?

• What do you plan to use for:

û High level MoMs?

û Scenarios of interest?

• Have you identified any additional organizations/

individuals with whom you plan to coordinate?
Decisionmaker’s Guide 29

• How do you plan to attack the problem?

û What methodology will you employ?

û How will you treat the diverse aspects of a

mission capability package (including concept

of operations, organization, doctrine, C2

approach, systems, personnel, facilities, in

other words everything needed to field a real

capability.)

û How do you plan to address organization/

human issues?

û What specific methods and tools will you

employ? Why do you think they are

appropriate?

û What data will you employ? Where will you

get them? Why do you think they are

appropriate? How do you plan to make the

data accessible to others? How do you plan

to depict the results of the study?

At First Iteration:

• What specific MoMs were selected? What

relationships were established among the

MoMs?

• What range of scenarios were selected? Why?

• What plans do you have to illuminate uncertainty/

sensitivity?
30 NATO Code of Best Practice for C2 Assessment

• What feedback did you receive from the

independent review team? What steps did you

take to respond to it?

• What do you plan to do on subsequent iterations?

û Use additional tools?

û Consider additional scenarios, assumptions?

û Modify assessment boundaries?

At Final Report:

• What are the major findings, recommendations?

• What are the key points of uncertainty/sensitivity?

• What issues were not addressed that should be

treated in subsequent assessments?

• What key lessons did you learn with respect to

methods, tools, and data?

• What steps are you going to take to disseminate

key insights, products to the broader community?

• Are there any voids/issues that warrant

further research?