7/31/03 DRAFT 

OFTOASD/ NII

NCO Conceptual Framew

Case Study* Template 

Illustrative Example

* Adapted from the RAND Air-to-Air Case S
7/31/03 DRAFT 

OFTOASD/ NII

Agenda

• Case Study Background• The Bottom Line Result• Problem Formulation• Solution Strategy• The Way Ahead
7/31/03 DRAFT 

OFTOASD/ NII

Case Study Backgroun

• Operational Environment 

– What: Small Air-to-Air Engagements*– Where: On Test Ranges– When: 1990’s (12,000 Sorties/ 19,000 Fl– Organization(s) Involved: USAF

• Key Difference

– Baseline: Voice Only Communications– Treatment: Voice Supplemented with Lin

• Dependent Variable: Kill Ratio

*Orchestrated and Analyzed by Joint Tactical Information Distributio

Operational Special Project
7/31/03 DRAFT 

OFTOASD/ NII

Air-to-Air Combat: 4 vs. 4 Eng

Blue12

Blue11

Blue 13

AWACS

Blue 14
7/31/03 DRAFT 

OFTOASD/ NII

Agenda

• Case Study Background• The Bottom Line Result• Problem Formulation• Solution Strategy• The Way Ahead


7/31/03 DRAFT 

OFTOASD/ NII

The Bottom Line Resul

• Sharing of Information Across Platforms Inv

Operation Resulted in a Kill Ratio ImprovemThan 2 to 1 (2.61 x Increase- Day; 2.59 x Inc

Night 3.62 9.4Day 3.10 8.1

Voice Only Voice + Li

Kill Ratio

• Under the Following Conditions:

– Blue Mission Capability Packages 

were Co-evolved

– No Red Learning 
7/31/03 DRAFT 

OFTOASD/ NII

Agenda

• Case Study Background• The Bottom Line Result• Problem Formulation• Solution Strategy• The Way Ahead


7/31/03 DRAFT 

OFTOASD/ NII

Problem Formulation

• Identify the Research Question• Identify the Relevant Variables• Identify the Assumptions and Constraints
7/31/03 DRAFT 

OFTOASD/ NII

Identify the Research Ques

Identify the Research Question:

What Explains the Difference in Kill RatiFighter Planes Equipped with Voice Only Equipped with Link 16 Data Communicat
7/31/03 DRAFT 

OFTOASD/ NII

Identify the Relevant Variab

• Dependent Variable: Degree of Effectivenes

Ratios)

• Independent Variable: Co-Evolved Mission 

Package (MCP) that Results from the IntroduData Communication Capability

– MCP Elements Include:

• CONOPS/Doctrine• Organization• Command Arrangements• Training/Education• Personnel• Others ...

• Treatment: Introduction of Link 16 Data Com

Capability
7/31/03 DRAFT 

OFTOASD/ NII

Identify the Relevant Variab

• Other Independent Variables

– Controllable: They Can be Manipulated (P

Information Domains)

• Quality of Organic Information• Quality of Networking• Degree of Information “Share-ability”

– Uncontrollable: It is Possible to Alter them 

on Human Decisions (Social and Cognitiv

• Quality of Individual Information• Degree of Shared Information• Degree of Shared Sensemaking• Degree of Actions/ Entities Synchroniz
7/31/03 DRAFT 

OFTOASD/ NII

Identify the Assumptions and C

• Assumptions:

– No Red Learning

• Constraints on Variables: “Acceptable” Thres

the Variables 

– Red Behavior Did Not Vary

• Study Limitations:

– Data Available Only for a Limited Numbe

Number of Sorties, Kill Ratios for Missionand Voice + Link 16 Across Day and Nigh

• No Data on Many Conceptual Framewo• Limited “Test” of the Conceptual Fram

– Findings May be Limited to Tactical Comb
7/31/03 DRAFT 

OFTOASD/ NII

Agenda

• Case Study Background• The Bottom Line Result• Problem Formulation• Solution Strategy• The Way Ahead


7/31/03 DRAFT 

OFTOASD/ NII

Solution Strategy 

• Postulate a “Conceptual Model” (NCO “S• Identify What We Know • Identify What We Need to Know
7/31/03 DRAFT 

OFTOASD/ NII

The Story Line (As it Unfo

Change in Communications Capability (Voice + Link 16) 

Commun

Capab

Quality of Individual Sensemaking

Degree of Decision/ Synchronization

Degree of Effectiveness

Degree of Information “Share -ability”

Quality of Networking

Force

Quality of Individual Information Degree of Shared Information

Quality of Organic 

Information

C2 Effectors

Value Added

Services

Quality

of

Interactions

Information

Sources

Degree of Actions/ Entities Synchronized

C2 Agility

Force Agility

Degree of Networking Net Readiness of Nodes

Individual Understanding

Degree of Shared Sensemaking

Shared Awareness

Individual Decisions Collaborative Decisions

Shared Understanding

Individual Awareness

C2 
7/31/03 DRAFT 

OFTOASD/ NII

The Story Line (As it Unfo

Change in Communications Capability (Voice + Link 16) Results in impr

Quality of the N

Quality of Individual Sensemaking

Degree of Decision/ Synchronization

Degree of Effectiveness

Degree of Information “Share -ability”

Quality of Networking

Force

Quality of Individual Information Degree of Shared Information

Quality of Organic 

Information

C2 Effectors

Value Added

Services

Quality

of

Interactions

Information

Sources

Degree of Actions/ Entities Synchronized

C2 Agility

Force Agility

Degree of Networking Net Readiness of Nodes

Individual Understanding

Degree of Shared Sensemaking

Shared Awareness

Individual Decisions Collaborative Decisions

Shared Understanding

Individual Awareness

C2 

Communication

Capability
7/31/03 DRAFT 

OFTOASD/ NII

The Story Line (As it Unfo

Change in Communications Capability (Voice + Link 16) 

Results in improvemQuality of the Netw

ImprovemenShare-abilit

Quality of Individual Sensemaking

Degree of Decision/ Synchronization

Degree of Effectiveness

Degree of Information “Share -ability”

Quality of Networking

Force

Quality of Individual Information Degree of Shared Information

Quality of Organic 

Information

C2 Effectors

Value Added

Services

Quality

of

Interactions

Information

Sources

Degree of Actions/ Entities Synchronized

C2 Agility

Force Agility

Degree of Networking Net Readiness of Nodes

Individual Understanding

Degree of Shared Sensemaking

Shared Awareness

Individual Decisions Collaborative Decisions

Shared Understanding

Individual Awareness

C2 

Communication

Capability
7/31/03 DRAFT 

OFTOASD/ NII

Change in Communications Capability (Voice + Link 16) 

Results in improvemQuality of the Netw

That lead tinformation individuals 

Quality of Individual Sensemaking

Degree of Decision/ Synchronization

Degree of Effectiveness

Degree of Information “Share -ability”

Quality of Networking

Force

Quality of Individual Information Degree of Shared Information

Quality of Organic 

Information

C2 Effectors

Value Added

Services

Quality

of

Interactions

Information

Sources

Degree of Actions/ Entities Synchronized

C2 Agility

Force Agility

Degree of Networking Net Readiness of Nodes

Individual Understanding

Degree of Shared Sensemaking

Shared Awareness

Individual Decisions Collaborative Decisions

Shared Understanding

Individual Awareness

ImprovemenShare-abilit

C2 

The Story Line (As it Unfo

Communication

Capability
7/31/03 DRAFT 

OFTOASD/ NII

Change in Communications Capability (Voice + Link 16) Results in improvem

Quality of the Netw

That lead tinformation individuals 

Quality of Individual Sensemaking

Degree of Decision/ Synchronization

Degree of Effectiveness

Degree of Information “Share -ability”

Quality of Networking

Force

Quality of Individual Information Degree of Shared Information

Quality of Organic 

Information

C2 Effectors

Value Added

Services

Quality

of

Interactions

Information

Sources

Degree of Actions/ Entities Synchronized

C2 Agility

Force Agility

Degree of Networking Net Readiness of Nodes

Individual Understanding

Degree of Shared Sensemaking

Shared Awareness

Individual Decisions Collaborative Decisions

Shared Understanding

Individual Awareness

ImprovemenShare-abilit

C2 

The Story Line (As it Unfo

Communication

Capability

That lead Sensemak
7/31/03 DRAFT 

OFTOASD/ NII

Change in Communications Capability (Voice + Link 16) 

Results in improvemQuality of the Netw

That lead tinformation individuals 

Quality of Individual Sensemaking

Degree of Decision/ Synchronization

Degree of Effectiveness

Degree of Information “Share -ability”

Quality of Networking

Force

Quality of Individual Information Degree of Shared Information

Quality of Organic 

Information

C2 Effectors

Value Added

Services

Quality

of

Interactions

Information

Sources

Degree of Actions/ Entities Synchronized

C2 Agility

Force Agility

Degree of Networking Net Readiness of Nodes

Individual Understanding

Degree of Shared Sensemaking

Shared Awareness

Individual Decisions Collaborative Decisions

Shared Understanding

Individual Awareness

ImprovemenShare-abilit

C2 

The Story Line (As it Unfo

Communication

Capability

That lead Sensemak

That contribuAction/Entity 
7/31/03 DRAFT 

OFTOASD/ NII

Change in Communications Capability (Voice + Link 16) 

Results in improvemQuality of the Netw

The Story Line (As it Unfo

That lead tinformation individuals 

That lead Sensemak

ImprovemenShare-abilit

Quality of Individual Sensemaking

Degree of Decision/ Synchronization

Degree of Effectiveness

Degree of Information “Share -ability”

Quality of Networking

Force

Quality of Individual Information Degree of Shared Information

Quality of Organic 

Information

C2 Effectors

Value Added

Services

Quality

of

Interactions

Information

Sources

Degree of Actions/ Entities Synchronized

C2 Agility

Force Agility

Degree of Networking Net Readiness of Nodes

Individual Understanding

Degree of Shared Sensemaking

Shared Awareness

Individual Decisions Collaborative Decisions

Shared Understanding

Individual Awareness

C2 

Communication

Capability

That ultimately results in dramatically improved Effectiveness

That contribuAction/Entity 

NCO Value Chain
7/31/03 DRAFT 

OFTOASD/ NII

Degree of Effectiveness

D

Quality of Individual 

Information

Quality of 

Networking

Degree of 

Information 

“Share-ability”

1.0

0.28

1.0 0.5 0.08

1.0

0.4

0.91

3.10:1

8.11:1

1.0

0.22

0.45

0.91

Deg

Actions

Synch

0.34

0.68

Quality of 

Organic Info

Overall average dimensions and p

Kill Ratio

(Effectiveness)

Voice

Voice + Link 16
7/31/03 DRAFT 

OFTOASD/ NII

Solution Strategy 

• Postulate a “Conceptual Model” (NCO “S• Identify What We Know • Identify What We Need to Know


7/31/03 DRAFT 

OFTOASD/ NII

Identify What We Kno

• Mission Capability Package (MCP) that Repr

Baseline (with Voice Only Comms) and MCP the Co-Evolved Impact of the Treatment (wit16 Comms) 

• The Differences in Quality of Organic Inform

Networking, Degree of Information Share-abIndividual Information, and Degree of Shared Resulting from Voice + Link 16

• Hypothesized Differences in Degree of Share

and Degree of Actions/ Entities Synchronized
7/31/03 DRAFT 

OFTOASD/ NII

MCP: Base Line vs. Treatm

Force

Mission Capability PackagesRolesRelevantAttributesElements(Network, Nodes)

Value added 

Services C2 Information Sources 

• Coverage• Persistence• Performance

• Capability• Capacity• Quality of Service

Embedded in

the NCW 

Conceptual

Framework

Voice Only

(MCP #1)

AWACS Voice Network 

Functions • Detect/ 

ID Targets

•Fuse Data•ID Info•Distribute Info.

•Assign Aircraft to Targets•CoordinateEngagements

Voice + Link 

(MCP #2)

Data +Network
7/31/03 DRAFT 

OFTOASD/ NII Explanation of Numeric 

for NCO CF Variabl

• The Numeric Values for the NCO CF Variab

by the RAND Case Study Team Using an An

• Model Functions Based on Quantitative and Q

– Quantitative Inputs:

• Known Performance of Voice and Link 

Systems

• Known Performance of Other Related 

etc.)

• Known Exogenous Factors (Training, 

Doctrine, etc.)

– Qualitative Inputs:

• Interviews with Pilots and Other SMEs• Research and Analysis of Existing Doc
7/31/03 DRAFT 

OFTOASD/ NII Example Computational M

Quality of Organic Inform

Sensor coverage

- Field of regard- Instantaneous field of view- Area coverage rate- Revisit rate (e.g. AWACS 10 sec scan)- Range

Probability of detection

-Fn of RCS-Fn of doppler

Probability of false alarmProbability of classificationSighting location errorSighting velocity errorRadar processor track precision

Information Sources 

Quality 

Consistency

PrecisionCurrency

Correctness

Objective Measures 

AWACs

AMTI radar

IFFN

NCTR#1

Etc. 

Blue11 

A

Exogenous variables• Environmental conditions • Doctrine

AMTI radar

IFFN

NCTR#1

Etc. 

f(...)

AMTI: Airborne Moving TIFFN: Identification, FriendNCTR: Non Cooperative Ta

AWACs: Airborne Warning & Control System
7/31/03 DRAFT 

OFTOASD/ NII

Quality of Individual Sensemaking

Degree of Decision/ Synchronization

Degree of Effectiveness

Degree of Informa

Quality of 

Quality of Individual Information Degre

Quality of Organic 

Information

Quality

of

Interactions

Degree of Actions/ Entities Synchronized

C2 A

Force A

Degree of Networking 

Individual Understanding

Degre

Individual Decisions 

Individual Awareness

for 

(i.e

h

th

-

Force C2 

Value Added Services Information Sources 

Degree of 

Actions/ Entities 

Synchronized

Degree of 

Sensemaking

Degree of Shared 

Information

Quality of Individual 

Information

Quality of 

Networking

Degree of 

“Share- ability”

Kill Ratio

(Effectiveness)

Quality of 

Organic Info

Shared

0.28

1.0

Voice + Link 16

Voice Overall average over information quality 

dimensions and package members

Information 

What We Know: Quality of Org
7/31/03 DRAFT 

OFTOASD/ NII Explanation of Quality of Or

Information Values

Blue11, 12

Blue13, 14

0

0.25

0.5

0.75

1

B B B B Q1 Q2 Q3 Q4

Q1B: Completeness: Detection

Q2B: Correctness: ID

Q3B: Correctness: Location

Q4B: Correctness: Velocity

0

0.25

0.5

0.75

1

B B B B Q1 Q2 Q3 Q4

0

0.25

0.5

0.75

1

B B B B Q1 Q2 Q3 Q4

Red 1, 2

Red 3, 4

Blue11, 12

Blue13, 14

0

0.25

0.5

0.75

1

B B B B Q1 Q2 Q3 Q4

Q1B: Completeness: Detection

Q2B: Correctness: ID

Q3B: Correctness: Location

Q4B: Correctness: Velocity

0

0.25

0.5

0.75

1

B B B B Q1 Q2 Q3 Q4

0

0.25

0.5

0.75

1

B B B B Q1 Q2 Q3 Q4

Blue11, 12

Blue13, 14

0

0.25

0.5

0.75

1

B B B B Q1 Q2 Q3 Q4

Q1B: Completeness: Detection

Q2B: Correctness: ID

Q3B: Correctness: Location

Q4B: Correctness: Velocity

0

0.25

0.5

0.75

1

B B B B Q1 Q2 Q3 Q4

0

0.25

0.5

0.75

1

B B B B Q1 Q2 Q3 Q4

Red 1, 2

Red 3, 4

What Blue Knows About Blue

T T T T Q1 Q 2 Q3 Q 4

Blue11, 12

Blue13, 14

0

0.25

0.5

0.75

1

T T T T Q1 Q 2 Q3 Q 4

T Q 1

T Q 2

Completeness: DetectionCorrectness: ID

0

0.25

0.5

0.75

1

T T T T Q1 Q 2 Q3 Q 4

Blue11, 12

Blue13, 14

0

0.25

0.5

0.75

1

T T T T Q1 Q 2 Q3 Q 4

T Q 1

T Q 2

Completeness: DetectionCorrectness: ID

T Q 1

T Q 2

T Q 1

T Q 2

Completeness: DetectionCorrectness: ID

0

0.25

0.5

0.75

1

What Blu

Note: It is assumed that the Quality of Organic Information is the same foand Voice + Link 16 (i.e., communication capability has no effect on Qual

Quality of Organic Information = 0.28 (Based on completene

known sensor performance; see slide 27 for details on com
7/31/03 DRAFT 

OFTOASD/ NII

Quality of Organic Informa

Information Gathered by Individual Sensors that is Not Shared and is U

No DifferenNo DifferenNo DifferenNo DifferenNo DifferenNo DifferenNo DifferenNo Differen

Link 

No DifferenceNo DifferenceNo DifferenceNo DifferenceNo DifferenceNo DifferenceNo DifferenceNo Difference

Voice Only (.28)

Completeness

Relevance 

Accuracy 

Timeliness 

Precision

Consistency

Currency

Correctness

Attribute

The Quality of Organic Information is the Same for Op

Only and with Voice + Link 16

Note: The Values of the NCO CF Variables were Calculated Using an AnValues for Each Attribute Are Estimates that Reflect the Calculations Perf
7/31/03 DRAFT 

OFTOASD/ NII

Quality of Organic Informa

Quality of Individual 

Information Deg

I

*

0.28

Kill Ratio

(Effectiveness)

Deg

Actions

Synch

Degree of 

Information 

“Share-ability”

S

Quality of 

Networking

Quality of 

Organic Info

Overall average o

Voice dimensions and pa

Voice + Link 16
7/31/03 DRAFT 

OFTOASD/ NII

Quality of Individual Sensemaking

Degree of Decision/ Synchronization

Degree of Effectiveness

Degree of Informa

Quality of 

Force

Quality of Individual Information Degre

Quality of Organic 

Information

C2 

Value Added 

Services

Quality

of

Interactions

Information

Sources

Degree of Actions/ Entities Synchronized

C2 A

Force A

Degree of Networking 

Individual Understanding

Degre

Individual Decisions 

Individual Awareness

C2 

Degre

Info

0.28

Overall average over indimensions and packag

Quality of Individual 

Information

Quality of 

Networking

Degree of 

Information 

“Share-ability”

Kill Ratio

(Effectiveness)

Quality of 

Organic Info

De

S

Sen

Voice

Voice + Link 16

Degree o

Actions/ En

Synchroniz

1.0 0.5

What We Know: Quality of Net

OFT OASD/ NII
7/31/03 DRAFT 

OFTOASD/ NII

Explanation of Quality of Netw

• Based on Quality of Service Metrics, There w

Improvement in Network Quality Between thTreatment

• Calculations Based on Known Performance S

Voice and Link 16 Communication Systems for Details)
7/31/03 DRAFT 

OFTOASD/ NII

Quality of Networking

Node Assurance Not Calculated 

Collaboration Support Not Calculated 

P & R Capability Support Not Calculated 

Connectivity Not Calculated 

Capacity Not Calculated 

Net Readiness of NodesDegree of Networking

Assumed Static

Assumed 100%

One Military Vocoder Channel 

(Limited Capacity)

AWACS over Radio

Voice Only (0.5)

Network Agility

Network Assurance

Quality of Service

Reach

Attribute

Note: The Values of the NCO CF Variables were Calculated Using an Analytica MAttribute Are Estimates that Reflect the Calculations Performed in the Model.
7/31/03 DRAFT 

OFTOASD/ NII

Quality of Networking

Quality of Individual 

Information D

0.28

Kill Ratio

(Effectiveness)

Deg

Actions

Synch

1.0 0.5

Degree of 

Information 

“Share-ability”

Quality of 

Networking

Quality of 

Organic Info

Overall average o

Voice dimensions and pa

Voice + Link 16
7/31/03 DRAFT 

OFTOASD/ NII What We Know: Degree of Informati

Quality of Individual Sensemaking

Degree of Decision/ Synchronization

Degree of Effectiveness

Quality of 

Force

Quality of Individual Information Degre

Quality of Organic 

Information

C2 

Value Added 

Services

Quality

of

Interactions

Information

Sources

Degree of Actions/ Entities Synchronized

C2 A

Force A

Degree of Networking 

Individual Understanding

Degre

Individual Decisions 

Individual Awareness

C2 

Deg

I

0.28

Quality of Individual 

Information

Quality of 

Networking

Degree of 

Information 

“Share -ability”

Kill Ratio

(Effectiveness)

Quality of 

Organic Info

S

Voice

Voice + Link 16

Degre

Actions/ 

Synchro

1.0 0.5 0.08

1.0

Overall average over infordimensions and package m

Degree of Informat
7/31/03 DRAFT 

OFTOASD/ NII Explanation of Degree o

Information “Share-ability” V

Degree of Infor

“Share-abil

Quantity of Retriev

Quantity of Poste

Nod

Collabo

P&R Ca

Co

C

Network Agility

Network Assurance

Quality of Service

Reach

Degree of Networking

Source/Sensors Value Added Info Processors C2 

Force

Ease of Use

Network

Net R

Exogenous variables:•Number of nodes•File sizes and number offiles•Variables impacting howquickly nodes can transmit pieces of information (CONOPS, codingschemes, etc.)•Policies determiningpriority for posting•Expiration age for each type of info objects•Maximum queue lengths

•Whether nodes can transmit to network•Posting channel numbers, types, and bandwidth (for data links only)•Prob of correct transmission•Adjustments to probability that QoS will be delivered 

f(...): Vector for number of info objects that can be posted, by object typePhase 1. For each type of info object do:1. If nodes can post object, do 2. Else, Num(type) = 02. Use QoS parameters, network agility parameters, and exo variables to

determine rate at which nodes can post info items of that type. Multiply thisrate by probability of correct transmission, yielding theoretical transmissionrate.

Phase 2.1. Use theoretical rates for each info type plus priority policies to determine what

fractions of postings will be of each info type.

2. Multiply fractions of postings times theoretical rates times expiration age to

get Num(type) for each info type.

Using Link 1

standarnumber All tracsecondInfo on secondposted 

Using Voice AF CONOPS and coding 

standards, aircraft can transmit three tracks every ten seconds on a military coding voice channel.Value = .08

Note: Calculations based on a single attribute (Quantity of Pogeneral, calculations should be based on as many attributes as 
7/31/03 DRAFT 

OFTOASD/ NII

Degree of Information “Share-

The Degree to Which Information Could be Shared Among 

NoNo

Lin

Not CalculatedNot Calculated

Low 

Voice Only (.08)

Ease of Use

Quantity of Retrievable 

Information

Quantity of Posted 

Information 

Attribute

Note: The Values of the NCO CF Variables were Calculated Using an AnValues for Each Attribute Are Estimates that Reflect the Calculations Perf
7/31/03 DRAFT 

OFTOASD/ NII

Degree of Information “Share-

D

0.28

Quality of Individual 

Information

Quality of 

Networking

Degree of 

Information 

“Share-ability”

Kill Ratio

(Effectiveness)

Deg

Actions

Synch

1.0 0.5 0.08

1.0

Quality of 

Organic Info

Overall average odimensions and pa

Voice Voice + Link 16
7/31/03 DRAFT 

OFTOASD/ NII What We Know: Quality of Individu

Quality of 

Force

Quality of Organic 

Information

C2 

Value Added 

Services

Information

Sources

Degree of Networking 

C2 

Degree of Informa

Quality of Individual Sensemaking

Degree of Decision/ Synchronization

Degree of Effectiveness

Quality of Individual Information Degre

Quality

of

Interactions

Degree of Actions/ Entities Synchronized

C2 A

Force A

Individual Understanding

Degre

Individual Decisions 

Individual Awareness Degree of Shared 

Information 

0.28

Overall average over information quality dimensions and package members

Quality of Individual

Information

Quality of 

Networking

Degree of 

Information 

“Share-ability”

Kill Ratio

(Effectiveness)

Quality of 

Organic Info

Degree of 

Shared 

Sensemaking

Voice

Voice + Link 16

Degree of 

Actions/ Entities 

Synchronized

1.0 0.5 0.08

1.0

0.4

0.91
7/31/03 DRAFT 

OFTOASD/ NII Explanation of Qualit

Individual Information V

“

Quantity of Retrievable Info

Quantity of Posted Info

Ease of Use

Consistency

Precision Timeliness

Precision TimelinessCurrency Relevance

Consistency Accuracy

Correctness Completeness

Currency

Accuracy

Correctness Completeness

Relevance

Objective Measures Fitness for Use

Objective Measures Fitness for Use

•Organic info assumed to be correct, within knownmargins of error Exogenous variables:

•Track, capability, intent information, all instandard formats•No fusion performed•Pilots will use organic information in preferenceto radio-reported information •Pilots have sufficient training to use radio

•Assuming voice signalreaches blue AC with no infodegradation, but has only70% chance of being audible•Military vocoder repeatsauditory errors

F(...): Correctness of information object1. Assumed correctness for each of blue AC’s own tracks2. If info received from vocoder, approximately a 70% 

chance that message will have been heard correctlyFusion does not apply in this case. 

Total: 100% of organic info objects are correct; only 70% of

voice-reported info objects are correct (others are garbled to point of unusability)

Using Voice: Assumed correctness for each of blue 

AC’s own tracks. If info received fromvocoder, approximately a 70% chance that message will have been heard correctly Fusion does not apply in this case. If 100% of organic info objects are correct; only 70% of voicereported info objects are correct (others are garbled to point of un-usability)Value = 0.4

Using Link 16: Assum

AC’s own track16 or info displato use display). whether organic Value = 0.91

Note: Calculations based on a single attribute (Correctness). In should be based on as many attributes as possible.
7/31/03 DRAFT 

OFTOASD/ NII

Quality of Individual Inform

Information Gathered by Individuals from the Network and O

Timeliness Not Calculated N

NNNNNN

Not CalculatedNot CalculatedNot CalculatedNot CalculatedNot CalculatedNot Calculated

Medium

Voice Only 

(0.4)

Completeness

Relevance 

Accuracy 

Precision 

Consistency 

Currency 

Correctness

Attribute

Note: The Values of the NCO CF Variables were Calculated Using an AnValues for Each Attribute Are Estimates that Reflect the Calculations Perf
7/31/03 DRAFT 

OFTOASD/ NII Comparing Quality of In

Information (1): Baseline v

Voice Only Heads-up Display Link 16 H

X FRIENDLY

TARGET

400450

500 500

10001500

5 5

10 10

5 5

08 09 10

Heads-up Display with PlatformCentric Operations Heads-up D

Centr

X 

400450

500 

5 

10 

5 

08 

Blue Pilot 11 or 12 View

Blue Pilots 13 and 14 See Nothing

Commo
7/31/03 DRAFT 

OFTOASD/ NII Comparing Quality of In

Information (2): Baseline v

Voice Only 

0

0.25

0.5

0.75

1

T T 

Q1 Q 2 

T 

Q1 Q 2 

T 

Q1 Q 2 

0

0.25

0.5

0.75

1

0

0.25

0.5

0.75

1

Threat 

AWACS

Blue 13 & 14

B B B B Q1 Q2 Q3 Q4

B B B B Q1 Q2 Q3 Q4

B B B B Q1 Q2 Q3 Q4

T T T T Q1 Q 2 Q3 Q 4

T T T T Q1 Q 2 Q3 Q 4T T T T Q1 Q 2 Q3 Q 4

Threat Tracks Blue Tracks

0

0.25

0.5

0.75

1

0

0.25

0.5

0.75

1

0

0.25

0.5

0.75

1

0

0.25

0.5

0.75

1

0

0.25

0.5

0.75

1

0

0.25

0.5

0.75

1

Blue 11 & 12

Q 1Q 2

Q 3

Q 4

Completeness: DetectionCorrectness: ID

Correctness: LoCorrectness: Ve
7/31/03 DRAFT 

OFTOASD/ NII

Quality of Individual Inform

D

0.28

Overall average odimensions and p

Quality of Individual 

Information

Quality of 

Networking

Degree of 

Information 

“Share-ability”

Kill Ratio

(Effectiveness)

S

Voice

Voice + Link 16

Deg

Actions

Synch

1.0 0.5 0.08

1.0

0.4

0.91

Quality of 

Organic Info
7/31/03 DRAFT 

OFTOASD/ NII What We Know: Degree of Shared I

Quality of Individual Sensemaking

Degree of Decision/ Synchronization

Degree of Effectiveness

Quality of 

Force

Quality of Individual Information Degre

Quality of Organic 

Information

C2 

Value Added 

Services

Quality

of

Interactions

Information

Sources

Degree of Actions/ Entities Synchronized

C2 A

Force A

Degree of Networking 

Individual Understanding

Degre

Individual Decisions 

Individual Awareness

C2 

Degree of Shared 

Information 

0.28

Overall average over information quality dimensions and package members

Quality of Individual 

Information

Quality of 

Networking

Degree of 

Information 

“Share-ability”

Kill Ratio

(Effectiveness)

Quality of 

Organic Info

Degree of 

Shared 

Sensemaking

Voice

Voice + Link 16

Degree of 

Actions/ Entities 

Synchronized

1.0 0.5 0.08

1.0

0.4

0.91

1.0

0.22

Degree of Informa
7/31/03 DRAFT 

OFTOASD/ NII Explanation of Degree o

Shared Information Valu

f(...)

Whether sender and receiver arepart of the samecollaborative group

Degree of Information “Shar

Ease of Use

Quantity of Posted In

Quantity of Retrievable 

Probabilit

y that sender

will attempt to shareinformation with receivers

Number of communications “hops” between sender and receiver

Currency

Consistency

Correctness

Degree of Shared In

Precision

Quality

Objective Mea

Currency

Consistency

Correctness

Degree of Shared In

Precision

Quality

Objective Mea

Currency

Consistency

Extent

Correctness

Degree of Shared Sense

maki

Precision

Quality

Objective Me

Currency

Consistency

Extent

Correctness

Degree of Shared Sense

maki

Precision

Quality

Objective Me

Extent: Proportion of f

that share inform

Individual Characteristics

Organizational Characteristics

Organizational & Ind. Behavior

Quality of Interactions

Quality

Depth

Breadth

Agility

Intensity

Response

Trust

Interdepend

Syn

Adaptability

Flexible

Selectivity

Mode

Latency

Confidence

CompetenceCompetence

Confidence

Hardness

Permanence

Structure

Efficiency

T vs. T

Quantity

Robustness

Innovative

Reach

Continuity

Synchronicity

Risk Prop

Org. Ident

Risk Prop

Diversity

Trust

Size

Autonomy

Cooperation

Deg. Of Eng

Whethersender can communicateinfo with receiver

Whetherinformation can be ph

ysicall

y

shared across network

Matrix showing probabilities that pa

elements have been share

d with p

For each element of the matrix, Pr(sh• the probability that the info is ret• the probability the sender and rec

collaborative group;

• the probability the sender and rec

within the collaborative group;

• the probability the sender attemp

with t

he receiver; and

• the probability the information is 

of the number of “hops” between 

Using Voi

ce: Extent of 

Sharing li

mited by 

physical constraints of communication cap

ability

Value = 0.22 Using Link 16

posted all p

arti

Value = 

Note: Calculations based on a single attribute (Extent). In genebe based on as many attributes as possible.
7/31/03 DRAFT 

OFTOASD/ NII

Degree of Shared Informa

NNNNNNNNN

L

Not CalculatedNot CalculatedNot CalculatedNot CalculatedNot CalculatedNot CalculatedNot CalculatedNot CalculatedNot Calculated

Low

Voice Only (0.22)

Correctness

Consistency 

Currency 

Precision 

Quality

Completeness

Accuracy 

Relevance 

Timeliness 

Extent

Attribute

Note: The Values of the NCO CF Variables were Calculated Using an AnValues for Each Attribute Are Estimates that Reflect the Calculations Perf
7/31/03 DRAFT 

OFTOASD/ NII

Comparing Degree of Shared Inform

Base Line vs. Treatme

Voice Only Voice + L

0

0.25

0.5

0.75

1

T T T T Q1 Q 2 Q3 Q4

Threat Tracks

0

0.25

0.5

0.75

1

T T 

Q1 Q 2 

Threat 

All blue AC have the same shared informationin this example(all listen to the same voice channel or receive the same Link 16 broadcasts)

Blue Tracks

T T T T Q1 Q 2 Q3 Q4 0

0.25

0.5

0.75

1

Blue 

T T 

Q1 Q 2 

0

0.25

0.5

0.75

1

Q 1Q 2

Q 3

Q 4

Completeness: DetectionCorrectness: ID

Correctness: LoCorrectness: Ve
7/31/03 DRAFT 

OFTOASD/ NII

Degree of Shared Informa

De

0.28

Overall average odimensions and p

Quality of Individual 

Information

Quality of 

Networking

Degree of 

Information 

“Share-ability”

Kill Ratio

(Effectiveness)

S

Voice

Voice + Link 16

Deg

Actions

Synch

1.0 0.5 0.08

1.0

0.4

0.91

1.0

0.22

Quality of 

Organic Info
7/31/03 DRAFT 

OFTOASD/ NII

Quality of Individual Sensemaking

Degree of Decision/ Synchronization

Quality of 

Force

Quality of Organic 

Information

C2 

Value Added 

Services

Quality

of

Interactions

Information

Sources

Degree of Actions/ Entities Synchronized

C2 A

rce A

Degree of Networking 

Individual Understanding

Degre

Individual Decisions 

Individual Awareness

C2 

Quality of Individual Information Degre

Degre

C

Overall average over information quality dimensions and package members

Voice

Voice + Link 16

Degree of Shared 

Information

Quality of Individual 

Information

Quality of 

Networking

Degree of 

Information 

“Share-ability”

Degree of 

Actions/ Entities 

Kill Ratio Synchronized

(Effectiveness)

Quality of 

Organic Info

Degree of 

Shared 

Sensemaking

1.0

0.28

1.0 0.5 0.08

1.0

0.4

0.91

1.0

0.22

0.45

0.91

What We Know (Inferred): Degree of Share

Degree of Informa

F Degree of Effectiveness o
7/31/03 DRAFT 

OFTOASD/ NII Explanation of Degree o

Shared Sensemaking Valu

Quality of Individual Sensemaking: Awareness

Uncertainty

Timeliness

Relevance

Accuracy

Completeness

Fitness for Use

Currency

Consistency

Correctness

Precision

Objective Measures

Quality of Individual Sensemaking: Understanding

Uncertainty

Timeliness

Relevance

Accuracy

Completeness

Fitness for Use

Currency

Consistency

Correctness

Precision

Objective Measures

Currency

Consistency

Correctness

Precision

Objective Measures

Quality of Individual Decisions

Fitness for Use

Innovativeness

Agility

Timeliness

Relevance

Accuracy

Completeness

Appropriateness

Uncertainty

Responsiveness

Flexibility

Robustness

Adaptability

Objective Measures

Mode of D. M.

Precision

Currency

Consistency

Degree of Info

Quantit

Quan

•Whether tinteractiongathering asensemakin•The numb•Whether tinteractionrequiremen

•Whether thfor flexible making

Exogtraini

Individual Characteristics

Organizational Characteristics

Organizational & Ind. Behavior

Quality of Interactions

Quality

Depth

Breadth

Agility

Intensity

Response

Trust

Interdepend

Synchronization

Adaptability

Flexible

Selectivity

Mode

Latency

Confidence

CompetenceCompetence

Confidence

Hardness

Permanence

Structure

Efficiency

T vs. T

Quantity

Robustness

Innovative

Reach

Continuity

Synchronicity

Risk Prop

Org. Ident

Risk Prop

Diversity

Trust

Size

Autonomy

Cooperation

Engagement

Risk Propensity

Voice Only Link 16

Using Voice Only: Currency of Shared 

Sensemaking 

• Participants spend most of their time gathering 

and validating information from AWACS and other blue AC radars

• Voice communications adds delay over visual 

communications

• Inflexible command structures require a variety of 

explicit checks and permissions before engagement decisions can be madeValue = 0.45

Using Link 16: Currenc• Participants automatica

information available frradars, so pilots incur ninformation verbally

• Near-real time visual in

faster than voice transm

• Flexible command struc

targets, and support eng(Commanders only inteValue = 0.91
7/31/03 DRAFT 

OFTOASD/ NII Degree of Shared Sensem

(Currency)

Hypotheses:I. Information sharing via Voice + Link 16 leads to less time nec

information, which results in more time available for flight lea

II. Information sharing via Voice + Link 16 leads to less time nec

gather and monitor critical information, which results in oppospend time sensemaking

B11 (Flight lead)

Time

Voice Only Information Awareness Link-16 +Voice Info Awareness Understanding DecisionsB12 (Wingman)Voice Only Information Link-16 +Voice Info Awareness Understanding Decisions
7/31/03 DRAFT 

OFTOASD/ NII Degree of Shared Sensem

(Quality: Extent, Correct

B11 (Flight lead)Voice OnlyLink-16 +Voice

Information

Information

Info Awareness

Awareness Un

Understanding Decisions

Voice OnlyLink-16 +VoiceB12 (Wingman)

Info Awareness Understanding Decisions

Hypothesis: Information sharing via Voice + Link 16 leads to less information gathering, which results in more time available for flshare awareness, understandings and make decisions, which resuextent and correctness of sensemakingEvidence: Interviews with pilots reveal that with Link 16 +Voice, traffic focused on sharing information decreased dramatically leaand share high quality sensemaking 

Time
7/31/03 DRAFT 

OFTOASD/ NII

Degree of Shared Sensema

Extent Medium H

Not CaNot CaNot CaNot CaNot CaNot Ca

H

Not Ca

H

Lin

(0

Not CalculatedNot CalculatedNot CalculatedNot CalculatedNot CalculatedNot Calculated

Medium

Not Calculated

Medium

Voice Only 

(0.45)

Uncertainty

Completeness

Relevance 

Accuracy 

Timeliness 

Precision 

Consistency 

Currency 

Correctness

Attribute

Note: The Values of NCO CF Variables and Attribute
7/31/03 DRAFT 

OFTOASD/ NII

Degree of Shared Sensema

D

Overall average dimensions and p

Quality of Individual 

Information

Quality of 

Networking

Degree of 

Information 

“Share-ability”

Deg

Actions

Kill Ratio Synch

(Effectiveness)

Quality of 

Organic Info

1.0

0.28

1.0 0.5 0.08

1.0

0.4

0.91

1.0

0.22

0.45

0.91

Voice

Voice + Link 16
7/31/03 DRAFT 

OFTOASD/ NII

Quality of Individual Sensemaking

Degree of Decision/ Synchronization

Degree of Informa

Quality of 

Force

Quality of Organic 

Information

C2 

Value Added 

Services

Quality

of

Interactions

Information

Sources

C2 A

Degree of Networking 

Individual Understanding

Degre

Individual Decisions 

Individual Awareness

C2 

Quality of Individual Information Degre

Overall average over information quality dimensions and package members

Voice

Voice + Link 16

Degree of Shared 

Information

Quality of Individual 

Information

Quality of 

Networking

Degree of 

Information 

“Share-ability”

Degree of 

Actions/ Entities 

Kill Ratio Synchronized

(Effectiveness)

Quality of 

Organic Info

Degree of 

Shared 

Sensemaking

1.0

0.28

1.0 0.5 0.08

1.0

0.4

0.91

1.0

0.22

0.45

0.91

0.34

0.68

What We Know (Inferred): Degree of Actions/ 

Degree of Actions/ Entities Synchronize

Force A

Degree of Effectiveness
7/31/03 DRAFT 

OFTOASD/ NII Explanation of Degree o

Entities Synchronized 

• Based on Interviews with Pilots and SMEs, it 

that Pilots with Link 16 Utilized Different Ta

– Four New Tactics Emerged (See Next Slid– These New Tactics Were Employed Appr

the Time (Interview Results)

– Value of Degree of Action/ Entities Synch

• Voice Only: 0.34• Voice + Link 16: 0.68
7/31/03 DRAFT 

OFTOASD/ NII Degree of Actions/ Entities Sy

Reported Tactical Improvements Enabled b

Greater understanding and more time available allow“high-awareness” tactics that lead to major increases 

1. Increased numbers of engagements in 

the same time period

2. Employme

combatant rather 

Flight lead

Wingman Time

3. Advance vectoring to engage red A/Cs 

from position of maximum advantage 

4. Employment of 

trap and d
7/31/03 DRAFT 

OFTOASD/ NII

Degree of Actions/ Entities Sync

Synchronized Actions Medium 

Medium

Voice Only 

(0.34)

Synchronized Entities

Attribute

Note: The Values of NCO CF Variables and Attribute
7/31/03 DRAFT 

OFTOASD/ NII

Degree of Decision/ Synchron

D

Overall average dimensions and p

Quality of Individual 

Information

Quality of 

Networking

Degree of 

Information 

“Share-ability”

Deg

Actions

Kill Ratio Synch

(Effectiveness)

Quality of 

Organic Info

1.0

0.28

1.0 0.5 0.08

1.0

0.4

0.91

1.0

0.22

0.45

0.91

0.34

0.68

Voice

Voice + Link 16
7/31/03 DRAFT 

OFTOASD/ NII

Quality of Individual Sensemaking

Degree of Decision/ Synchronization

Degree of Informa

Quality of 

Force

Quality of Organic 

Information

C2 

Value Added 

Services

Quality

of

Interactions

Information

Sources

C2 A

Degree of Networking 

Individual Understanding

Degre

Individual Decisions 

Individual Awareness

C2 

Quality of Individual Information Degre

Overall average over information quality 

dimensions and package members

Voice

Voice + Link 16

Degree of Shared 

Information

Quality of Individual 

Information

Quality of 

Networking

Degree of 

Information 

“Share-ability”

Kill Ratio

(Effectiveness )

Quality of 

Organic Info

Degree of 

Shared

Sensemaking

1.0

0.28

1.0 0.5 0.08

1.0

0.4

0.91

3.10:1

8.11:1

1.0

0.22

0.45

0.91

0.34

0.68

Degree of 

Actions/ Entities 

Synchronized

What We Know (Inferred): Degree of Decision

Degree of Actions/ Entities Synchronized

Force A Degree of Effectiveness
7/31/03 DRAFT 

OFTOASD/ NII

Degree of Effectivenes

Not CaNot CaNot Ca

Kill R

8.11 

9.40 

Lin

Not CalculatedNot CalculatedNot Calculated

Kill Ratio:

3.10 Day

3.62 Night

Voice Only

EfficiencyAgilityTimelinessAchievement of Objectives

Attribute

Note: The Indicator of Achievement of Objectives is highly likely that other case studies will have multiple success. Also, not all attributes will be applicable for The values of the variable were provided in the originwere not computed or inferred.
7/31/03 DRAFT 

OFTOASD/ NII

Degree of Effectiveness

D

Quality of Individual 

Information

Quality of 

Networking

Degree of 

Information 

“Share-ability”

1.0

0.28

1.0 0.5 0.08

1.0

0.4

0.91

3.10:1

8.11:1

1.0

0.22

0.45

0.91

Deg

Actions

Synch

0.34

0.68

Quality of 

Organic Info

Overall average dimensions and p

Kill Ratio

(Effectiveness)

Voice

Voice + Link 16
7/31/03 DRAFT 

OFTOASD/ NII

Quality of 

Force

Quality of Organic 

Information

C2 

Value Added 

Services

Information

Sources

Degree of Networking 

C2 

Data Currently Available for Air-to-

Degree of Informa

Degre

Quality of Individual Sensemaking

Degree of Decision/ Synchronization

C2 A

rce A

Degree of Effectiveness

Quality of Individual Information Degre

Quality

of

Interactions

Individual Awareness 

Individual Understanding

Individual Decisions 

Degree of Actions/ Entities Synchronized

Fo
7/31/03 DRAFT 

OFTOASD/ NII

Solution Strategy (3)

• Postulate a “Conceptual Model” (NCO “S• Identify What We Know • Identify What We Need to Know




7/31/03 DRAFT 

OFTOASD/ NII

Identify What We Need to 

• What is the Resulting Impact on the Quality a

Sensemaking, Quality of Interactions, and Sy

• How Does the Addition of Link 16 Effect the 

Capability Package? (Are There Associated and Tactics?)
7/31/03 DRAFT 

OFTOASD/ NII

Agenda

• Case Study Background• The Bottom Line Result• Problem Formulation• Solution Strategy• The Way Ahead


7/31/03 DRAFT 

OFTOASD/ NII The Way Ahead:

Further Development of the Air-to-A

• Research Plan to Complete Case Study

– Additional Data Collection/Modeling/Analy

• Sensemaking• Quality of Interactions• Synchronization

– Sources of Data: Interviews, Voice Tapes, R– Analysis Tools: Explore Use of Analytica, O

• Execute and Update

– Phase I (Air-to-Air Case Study Completed s

Data Sharing 

– Phase II (Hypothetical Continuation of Air-

Sensemaking/ Synchronization (To be Comp