## Change Request Number: 02-74 Date Received: 8/13/02

**Title: "Guidelines for the IAT's Alternatives Risk Assessment"**

# **Name:** Art Politano

**Routing Symbol:** ASD-400

**Phone:** (202) 358-5221

**Policy OR Guidance:** Guidance

**Summary of Change:** In Investment Analysis Tool Set of FAST, create
Investment Analysis Standard Guideline Section. Under this section, add
the attached document: Guidelines for the Investment Analysis Team's
Alternatives Risk Assessment

The Guidelines examines a broad range of issue areas which may impact
the investment merit of alternative investments. For the most part it
relies of available data, expert judgment, and double blind assessment
by a panel of stakeholders to reach consensus on the degree of risk to
be assigned. The Guidelines also represents a harmonization of existing
risk processes from the Mission Need Area (such as Portfolio
Management), and subject area organizations (such as Human Factors,
Safety, and Security Organizations). The application of the Guidelines
will result in the identification of mitigation actions for each medium
or higher risk, and the costing of those actions. The cost and benefit
impacts of mitigating risk is then included in the development of the
acquisition program baseline (APB), so that the program can have an
increasing likelihood of success.

**Reason for Change:** To ensure consistency and accuracy in the conduct
of Agency Investment Analysis that complies with governmental standards
and guidelines as well as industry best practices.

**Development, Review, and/or Concurrence:** Concurrence obtained from:
ASD, AUA, AND, APO, ARQ, AFZ, and AAR.

**Target Audience:** All Investment Analysis Teams pursuing risk
analysis for JRC 2(a) and 2(b), e.g. ASD, AUA, AND, etc.

# **Potential Links within FAST for the Change:** N/A

**Attachments:** Language of change.

### Briefing Planned: YES, 8/26/02

#### ASAG Responsibilities: Review and Comment

**DRAFT**

![](media/image1.png){width="2.3125in" height="2.21875in"}

**Guidelines for the**

**Investment Analysis Team's**

**Alternatives Risk Assessment**

July 2002

Revised by:

Investment Analysis and Operations Research, ASD-400

Federal Aviation Administration

800 Independence Avenue, SW

Washington, DC 20591

**Page Intentionally Left Blank**

This document presents the guidelines by which the Investment Analysis
(IA) Team conducts Alternatives Risk Assessments as part of the Federal
Aviation Administration\'s (FAA\'s) IA process as proscribed in the
Acquisition Management System (AMS). The guidelines were developed to
assist IA Teams in their analysis of candidate alternatives/solutions.

Risk assessment is a dynamic enterprise. As the National Airspace System
(NAS) operations and environment change, we expect that new issues and
risks affecting investment analysis will surface. Since the first
publication of these Guidelines in July 1997, information security,
human factors, and safety issues have gained visibility and prominence
as additional risk to be considered. Accordingly, in July 1999, Art
Politano, Investment Analysis and Operations Research (ASD-400), and Don
Weitzman, Systems Engineering Technical Assistance (SETA), updated the
set of risk facets to include their assessment.

In April 2002, Art Politano and David Chin (ASD-400), and Christopher
Hunt and Darren Wilson (SETA-II), updated the guidelines again. The
primary reasons for this update is to: (1) harmonize the IA Team
Alternatives Risk Assessment and the other risk analyses in the
Acquisition phase, and (2) update the methodology based on what we have
learned from numerous applications. This document represents the state
of the practice on the IA Team's Alternatives Risk Assessment.

**Page Intentionally Left Blank**

[1.0 INTRODUCTION 1](#introduction)

[1.1 Acquisition Management System 1](#acquisition-management-system)

[1.2 Investment Analysis Phase 1](#investment-analysis-phase)

[1.3 Risk Analysis throughout the AMS
2](#risk-analysis-throughout-the-ams)

[2.0 Risk Assessments During the Acquisition Phases
3](#risk-assessments-during-the-acquisition-phases)

[2.1 General 3](#general)

[2.1.1 Risk at Each Phase of the AMS 3](#risk-at-each-phase-of-the-ams)

[2.1.2 Extending Across the Phases of the AMS
3](#extending-across-the-phases-of-the-ams)

[2.2 Risk Assessment during Mission Analysis -- Portfolio Management
5](#risk-assessment-during-mission-analysis-portfolio-management)

[2.3 Risk Assessment during Investment Analysis
6](#risk-assessment-during-investment-analysis)

[2.3.1 The IA Team Alternatives Risk Analysis -- JRC-2a (Initial IA
Decision)
6](#the-ia-team-alternatives-risk-analysis-jrc-2a-initial-ia-decision)

[2.3.2 The IA Team Life Cycle Risk Analysis - JRC-2b (Final IA Decision)
7](#the-ia-team-life-cycle-risk-analysis---jrc-2b-final-ia-decision)

[2.4 Risk Assessment during Solution Implementation - IPT Programmatic
Risk Management Process
7](#risk-assessment-during-solution-implementation---ipt-programmatic-risk-management-process)

[2.5 Risk Assessment during In-Service Management
8](#risk-assessment-during-in-service-management)

[3.0 Integration of IA Team Alternatives Risk Assessment with other
Assessments
9](#integration-of-ia-team-alternatives-risk-assessment-with-other-assessments)

[3.1 Connections with Portfolio Risk Assessment
9](#connections-with-portfolio-risk-assessment)

[3.2 Connection with Human Factors, Safety, and Security Assessments
9](#connection-with-human-factors-safety-and-security-assessments)

[3.2.1 Human Factors Assessment 9](#human-factors-assessment)

[3.2.2 Safety Assessment 10](#safety-assessment)

[3.2.3 Security Assessment 10](#security-assessment)

[3.3 Connections with IPT Programmatic Risk Assessment
12](#connections-with-ipt-programmatic-risk-assessment)

[3.3.1 Initial Investment Decision at JRC-2a
12](#initial-investment-decision-at-jrc-2a)

[3.3.2 Final Investment Decision at JRC-2b
12](#final-investment-decision-at-jrc-2b)

[3.4 Connections during Solution Implementation
12](#connections-during-solution-implementation)

[4.0 IA Team Alternatives Risk Assessment
14](#ia-team-alternatives-risk-assessment)

[4.1 Risk Facets 14](#risk-facets)

[4.2 Risk Facet Inclusion and Flexibility
15](#risk-facet-inclusion-and-flexibility)

[4.3 Clarification of Risk Facets for Benefits and Costs Estimates
15](#clarification-of-risk-facets-for-benefits-and-costs-estimates)

[4.3.1 Symmetrical Risks Associated with Uncertainty
15](#symmetrical-risks-associated-with-uncertainty)

[4.3.2 Uncertainty of Cost and Benefit Assumptions, and Qualitative
Benefits
16](#uncertainty-of-cost-and-benefit-assumptions-and-qualitative-benefits)

[4.3.3 Cost and Benefit Facet Risk Uses
16](#cost-and-benefit-facet-risk-uses)

[4.4 IA Team Alternatives Risk Assessment Process
16](#ia-team-alternatives-risk-assessment-process)

[4.4.1 Matrix Table Results of Probability and Severity
16](#matrix-table-results-of-probability-and-severity)

[4.4.2 Principles to Ensure Objectivity
17](#principles-to-ensure-objectivity)

[5.0 Tailoring the Process 18](#tailoring-the-process)

[5.1 Criteria for Choosing the Level of Effort
18](#criteria-for-choosing-the-level-of-effort)

[5.2 Approaches 18](#approaches)

[5.2.1 Staff Reviews 18](#staff-reviews)

[5.2.2 One-Day Meeting 18](#one-day-meeting)

[5.2.3 Sub Teams 19](#sub-teams)

[5.2.4 Full IA Team Meetings 19](#full-ia-team-meetings)

[5.2.5 Team Membership 19](#team-membership)

[5.2.6 How to Select and Where to Find Team Members
19](#how-to-select-and-where-to-find-team-members)

[6.0 STEPS IN THE RISK ASSESSMENT PROCESS
20](#steps-in-the-risk-assessment-process)

[6.1 Step 1 - Identify Risks 20](#step-1---identify-risks)

[6.2 Step 2 - Determine Facet Weights
37](#step-2---determine-facet-weights)

[6.3 Step 3 - Estimate Severity of Impact of an Adverse Event
37](#step-3---estimate-severity-of-impact-of-an-adverse-event)

[6.4 Step 4 - Estimate the Probability of an Adverse Event
38](#step-4---estimate-the-probability-of-an-adverse-event)

[6.5 Step 5 - Assign Risk Ratings 48](#step-5---assign-risk-ratings)

[6.6 Step 6 - Calculate Overall Alternative Risk Rating
48](#step-6---calculate-overall-alternative-risk-rating)

[6.7 Step 7 - Compare Risks among Alternatives
50](#step-7---compare-risks-among-alternatives)

[6.8 Step 8 - Transition to Joint Responsibility with the IPT for JRC-2b
51](#step-8---transition-to-joint-responsibility-with-the-ipt-for-jrc-2b)

[7.0 Risk assessment products and Mitigation Strategies
54](#risk-assessment-products-and-mitigation-strategies)

[7.1 Products of IA Team Alternatives Risk Assessment
54](#products-of-ia-team-alternatives-risk-assessment)

[7.1.1 Investment Analysis Report 54](#investment-analysis-report)

[7.1.2 Acquisition Program Baseline 55](#acquisition-program-baseline)

[7.1.3 Risk Issues Database 55](#risk-issues-database)

[7.2 Mitigation Strategies 55](#mitigation-strategies)

[7.2.1 Mitigation Actions and Cost Range Estimation
55](#mitigation-actions-and-cost-range-estimation)

[7.2.2 Mitigation Actions and Benefits Range Estimation
56](#mitigation-actions-and-benefits-range-estimation)

[7.2.3 Tracking and Evaluation of Mitigations
56](#tracking-and-evaluation-of-mitigations)

[8.0 Evolution of Risk Assessment 58](#evolution-of-risk-assessment)

[8.1 Aviation Environment Changes 58](#aviation-environment-changes)

[8.2 Methodological Changes 58](#methodological-changes)

[Appendix A: Team Participants 1](#appendix-a-team-participants)

[Appendix B: Risk Issue Worksheet 1](#appendix-b-risk-issue-worksheet)

# INTRODUCTION

## Acquisition Management System {#acquisition-management-system .unnumbered}

The Federal Aviation Administration (FAA) Acquisition Management System
(AMS) establishes policy and guidance for all aspects of the acquisition
life cycle. The AMS simplifies, integrates, and unifies the elements of
life cycle acquisition management. It increases the quality, reduces the
time, and decreases the cost of delivering needed services to its
customers.

The AMS has several phases including:

-   Mission analysis,

-   Investment analysis,

-   Solution implementation,

-   In-service management, and

-   Service life extension.

The Investment Analysis (IA) Team Alternatives Risk Assessment is
conducted during the Investment analysis phase.

## Investment Analysis Phase {#investment-analysis-phase .unnumbered}

The IA phase begins when the FAA determines a potential need to expend
funds to meet a mission capability shortfall or to take advantage of a
technological opportunity. During the IA phase, the sponsoring and
acquiring organizations are partners in ensuring the critical needs of
the users are satisfied by an affordable solution. The IA phase ends
with a decision on how the requested funds are allocated.

Major activities in the IA phase are:

-   Identifying and analyzing candidate solutions,

-   Developing an Investment Analysis Plan (IAP),

-   Assessing affordability,

-   Developing an Acquisition Program Baseline (APB),

-   Writing an Investment Analysis Report (IAR), and

-   Recommending the best investment alternative.

In the IAR, the IA Team analyzes alternatives and candidate solutions.
Of the alternatives identified, those determined viable become
candidates for detailed analysis. The IA Team evaluates candidate
solutions by compiling and analyzing such factors as cost, benefit,
schedule, and *risk*. The IA Team documents the results in an IAR that
contains comprehensive, quantitative data for each candidate solution.

During the IA phase, the IA Team conducts the Alternatives Risk
Assessment, as referenced in AMS policy sections 2.9.13, 2.4.4.1 and
2.4.4.2, which identifies risks and helps to establish realistic
baselines that include funding for the risk mitigation.

## Risk Analysis throughout the AMS {#risk-analysis-throughout-the-ams .unnumbered}

The following describe key elements of risk management at phases in the
AMS:

**Mission Analysis:** Identify and characterize risks to the agency's
ability to execute its legislated responsibilities and satisfy customer
demands for service. Typically, these risks arise from changes in the
operational environment and shortfalls in operational capability.

**Investment Analysis:** Ensure that primary risks associated with
candidate solutions are fully identified and evaluated. Sufficient time
and money must be included for each candidate solution in the APB to
mitigate risk and achieve program success.

**Program Implementation:** Detect and reduce risks early to avoid
greater cost of risk consequences later in the life cycle. The
Integrated Product Teams (IPTs) manage risks throughout the
implementation phase of their products and services. A Risk Management
Plan or risk planning section is developed for the Acquisition Strategy
Paper (ASP), and risk-mitigations are documented in the Integrated
Program Plan (IPP). Risk management requirements and activities are also
included in any prime contract for products or services.

**In-Service Management:** Assess and mitigate the risks continually and
adjust mitigation plans. The Risk Mitigation Plan contains actions to
scan the program's operational environment to understand any changes in
trends that impact the following:

-   The operation of the program in the NAS,

-   The timing and progress of implementing the committed mitigations,
    and

-   The risk status of the program.

If the risk status of the program worsens, the In-Service Management
organization will develop additional mitigations.

# Risk Assessments During the Acquisition Phases

## General {#general .unnumbered}

Risk analysis anticipates uncertainties in a potential capital
investment, assesses the degree of the risk, and identifies mitigations.
Risk mitigation is critical to an investment because it increases the
odds of a program being successful.

Paragraph 2.9.13 of the Life Cycle AMS states:

> "Risk management is applied throughout the acquisition management life
> cycle to identify and mitigate risks associated with achieving agency
> goals and objectives. Each line of business shall institute risk
> management processes that: (1) identify and assess risk areas; (2)
> develop and execute risk mitigation or elimination strategies; (3)
> track and evaluate mitigations; and (4) continue mitigations until
> risk is eliminated or its consequences reduced acceptably."

### Risk at Each Phase of the AMS {#risk-at-each-phase-of-the-ams .unnumbered}

Risk considerations vary in specificity and focus at each phase of the
AMS as knowledge of the investment varies. These phases include:

-   Mission Analysis - ATS and AIO have begun to explore the use of a
    **portfolio management approach**.

-   Investment Analysis - ARA's ASD has adopted an **investment risk
    analysis approach.**

-   Solution Implementation - ARA's IPT's have adopted a **programmatic
    risk assessment approach**.

### Extending Across the Phases of the AMS {#extending-across-the-phases-of-the-ams .unnumbered}

Extending across all the phases of the AMS are areas that warrant
separate assessments. These areas include:

-   Human factors (AAR),

-   Safety (ASD and ASY), and

-   Security (ASD, AIO, AIS, and ACS).

These areas call for a more detailed examination of risks in each phase
of a potential investments acquisition.

This report recognizes that the AMS has many approaches to risk
assessments, and seeks to harmonize and integrate them. Figure 2-1 on
the next page illustrates an approach to harmonization.

.

![](media/image2.wmf){width="9.720833333333333in"
height="6.040277777777778in"}**Figure 2-1: Integrated Risk Assessment
and Management Approach in IA**

## Risk Assessment during Mission Analysis -- Portfolio Management {#risk-assessment-during-mission-analysis-portfolio-management .unnumbered}

Risk assessment during the mission analysis is a component of portfolio
management. The portfolio view examines the overall range of programs,
i.e., the big picture. It allows the managers to pick the optimum set of
programs by taking into account the organization's goals and how the
organization can meet those goals.

Early in the AMS, at the Joint Resources Council (JRC-1) mission need
decision; the JRC approves a project for further development if it is
consistent with the FAA mission, and if there is a need for that
project. The decision is made with a full awareness of other projects
being implemented, and it implies that the project is consistent with
the package of programs already underway.

Portfolio management identifies the possible benefits of a program or a
package of programs and facilitates tracking the delivery of the
benefits.

A questionnaire, which addresses the following questions, is used to
help determine program risk.

-   Are we doing the right things?

-   Are we doing them the right way?

-   Are we getting them done well?

-   Are we getting the benefits?

**Figure 2-2: Typical Value Plot**

Answers to the questions are weighted and transferred to a spreadsheet.
Each question receives a risk score to produce an overall risk score.
![](media/image3.wmf){width="3.323611111111111in"
height="3.5819444444444444in"}

Analysts then combine the risk scores with the monetary component and an
alignment component, and prepare a value plot (Figure 2-2).[^1] This
gives the organization a graphical representation of how programs
compare to each other and their relative value.

A typical value plot shows small squares plotted on financial worth
versus risk axes (each square is divided into four quadrants depicting
the risk associated with each of the four questions). The border of a
square represents a program's alignment with company goals and values.

## Risk Assessment during Investment Analysis {#risk-assessment-during-investment-analysis .unnumbered}

For the Initial IA Decision at JRC-2a, the critical issue is determining
the most viable acquisition alternatives worthy of further analysis.
Consequently, the risk assessment for the Initial IA Decision is
macroscopic and comprehensive. The process includes the following:

-   Identifying risk issues,

-   Identifying mitigation strategies,

-   Assessing investment alternatives, and

-   Affirming mitigation strategies that will affect cost or benefit
    estimates.

### The IA Team Alternatives Risk Analysis -- JRC-2a (Initial IA Decision) {#the-ia-team-alternatives-risk-analysis-jrc-2a-initial-ia-decision .unnumbered}

In general, within the IA Team Alternatives Risk Assessment, there are
thirteen facets that constitute an exhaustive range of potential issues,
which include: technical, operability, producibility, supportability,
benefits estimate, cost estimate, schedule, management, funding,
stakeholder, security, human factors, and safety process. Discussion
with program staff, users, and stakeholders can help identify risk
issues. In the course of identifying issues, the IA Team often
formulates and records preliminary mitigations, and estimates the
likelihood of the issue surfacing and the potential severity of the
issue.

The next step may be to convene a team of stakeholders, investment and
program analysts, union representatives, and other interested
colleagues. A team comprised of a broad skill-set helps to form an
objective assessment. The team reviews the material and arrives at
consensus of each facet's rating for each alternative by double blind
voting (independent votes not visible to others) and justification
discussions. The team also affirms or complements issue mitigations. At
the end of these discussions, the least-risk alternative emerges.
Depending on the complexity of the investment, a full team analysis may
not be necessary (see Section 5.0, Tailoring the Process).

The analyst then coordinates with the cost, benefit, and other analysts
to link the impact of the risks and their mitigation on the cost and/or
the benefit estimate of the recommended alternative. This impact
contributes to developing ranges around the most likely cost or benefit
element in the respective Work Breakdown Structure (WBS).

The products of this effort may be a technical memorandum, provided by
the Risk Assessment Team Lead, but most often are:

-   A standalone summary section in the IAR,

-   Solid input for creating cost and benefit high confidence ranges
    around the most likely numbers,

-   Direct input into the Risk Management Matrix in the Initial APB, and

-   Risk Issues Database.

Lastly, the IA Team summarizes the issue ratings in a matrix for the
JRC, and the risk issues database forms the foundation for continued
analysis in the Solution Implementation phase.

### The IA Team Life Cycle Risk Analysis - JRC-2b (Final IA Decision) {#the-ia-team-life-cycle-risk-analysis---jrc-2b-final-ia-decision .unnumbered}

For the Final IA Decision at JRC-2b, the risks for the preferred
alternative are identified and costed in detail to ensure successful
implementation of the program. The detailed examination includes: (1)
providing a technical description of the risk issues, (2) quantifying
the impact on cost, schedule, benefit, and performance baselines, and
(3) describing the reason for the expected change. The detailed analysis
is based on either past performance history or estimates from a
simulation or scenario analysis. The Final APB includes the funding
needed to mitigate risks threatening the program's success.
Consequently, the risk analysis for the Final Investment Decision is
more focused. At this stage, the IPT Programmatic Risk Assessment is
complemented with the remaining IA risk issues.

## Risk Assessment during Solution Implementation - IPT Programmatic Risk Management Process {#risk-assessment-during-solution-implementation---ipt-programmatic-risk-management-process .unnumbered}

The IPT Programmatic Risk Assessment reviews the JRC-2a risk issues and
incorporates the relevant issues. In addition, since the APB includes
benefit and performance elements, the IPT working with the IA Staff,
complements the focus on cost, schedule, and technical elements by
adding the benefits and the performance elements.

As envisioned, the IPT Programmatic Risk Assessment begins with the
information gained for the Initial IA Decision at JRC-2a and examines
all issues from the perspective of the Final IA Baseline Metrics. The
metrics are cost, schedule, benefit, and performance. For the Final IA
at JRC-2b, the focus is on the issues implications to cost, schedule,
benefits, and performance. In the Final IA phase, any interdependency
risks (i.e., risks linked to the implementation of one program on the
success of other programs) will also be examined and mitigated.

The overall IPT Programmatic Risk Management Process entails the
following steps:

-   Identifying risks,

-   Assessing risks,

-   Selecting risk mitigation options,

-   Developing a Risk Mitigation Plan, and

-   Establishing the groundwork for monitoring and tracking risk.

Identifying risks focuses on staffing problems, design changes,
operational issues, test changes, negative trends, etc. Assessing risks
focuses on understanding the size and impact of the risk. Risk
mitigation focuses on what to do with the risk (avoid, transfer, assume,
or mitigate). The Risk Mitigation Plan includes necessary changes in
budgets, mitigations, and communication with stakeholders to get support
for mitigations.

If there are insufficient resources to mitigate the risk issues, the
program will probably rebaseline and review all the risk issues.

Lastly, the groundwork for monitoring and tracking includes expected
timing for the mitigation action and expected impact for reducing risk.
This allows tracking of mitigation actions, assessment of their impacts,
and the additional follow-up remedial actions. The IPT Programmatic Risk
Assessment focuses on cost, schedule, performance, and technical aspects
of the preferred alternative from the JRC-2a decision.

## Risk Assessment during In-Service Management {#risk-assessment-during-in-service-management .unnumbered}

The Risk Mitigation Plan is carried out during the Solution
Implementation and In-Service Management Phase. It is a product of
JRC-2b and contains actions to scan the program's operational
environment to understand any changes in trends that impact the
following:

-   The operation of the program in the NAS,

-   The timing and progress of implementing the committed mitigations,
    and

-   The risk status of the program.

# Integration of IA Team Alternatives Risk Assessment with other Assessments

The fact that each phase of the AMS has a risk assessment process
underscores the importance of risk assessment in the AMS. While each
phase's risk assessment fulfills functional needs, there remains a
compelling need to ensure an integrated framework with a smooth,
continuous flow of risk and mitigation information.

## Connections with Portfolio Risk Assessment {#connections-with-portfolio-risk-assessment .unnumbered}

Portfolio management, as of this writing, is only beginning to be
applied. Typically, a questionnaire is used to interview project
knowledgeable experts about potential risks. With little effort, the
survey can be expanded to cover all of the thirteen facets of the IA
phase, e.g., operability, human factors, benefit estimate,
supportability, and producibility.

The IA Staff may use the output of portfolio management to determine the
more significant risks of a program and focus on the mitigation of these
risks.

## Connection with Human Factors, Safety, and Security Assessments {#connection-with-human-factors-safety-and-security-assessments .unnumbered}

The IA Team Alternatives Risk Assessment considers all risk facets,
including human factors, safety process, and security risks.
Nevertheless, human factors, safety, and security may conduct separate
assessments. The consideration of issues in each of those separate
assessments should involve those organizations responsible for
conducting them, either directly through a special meeting, and/or by
their designated representatives participation in the IA Team
Alternatives Risk Assessment, or indirectly through guidance materials
from their offices.

The Risk Team Lead for the IA is responsible for alerting the
appropriate human factors, safety, and security offices in the FAA to
ensure that their assessments or inputs can be adequately integrated
into the IA Team Alternatives Risk Assessment.

### Human Factors Assessment {#human-factors-assessment .unnumbered}

The Human Factors Assessment for Investment Analysis entails a process
that provides essential and inter-related components to the products of
the Investment Analysis Team, including: a) the human-system performance
contribution to program benefits, b) an assessment of the human-system
performance risks, and c) the estimated costs associated with mitigating
human factors risks and with conducting the engineering program support.
The human factors components related to benefits, risks, and costs are
integrated with other program components in the IA products and
documentation to ensure that

:

-   Human-system performance capabilities and limitations are properly
    reflected in the system requirements

-   Human-system performance characteristics and their associated cost,
    benefits, and risks assist in deciding among solution alternatives

-   Human-system performance risks and their mitigation are
    appropriately addressed in program baselines and plans

In support of these activities and objectives, the FAA Office of the
Chief Scientific and Technical Advisor for Human Factors has
considerable expertise in understanding the human factors risks of a
potential acquisition, and their input is a critical driver. They
published a Human Factors Job Aid to guide the consideration of human
factors in all phases of the AMS.[^2]

In an effort to capture human factors risk, the Office of the Chief
Scientific and Technical Advisor for Human Factors may:

-   Choose to conduct their own assessment of the human factors impact
    of an acquisition in support of a particular IA;

-   Participate in IA Team deliberations and establish a human factors
    risk sub team, and/or assign responsibility to an IPT Human Factors
    Specialist; or

-   Direct the IA Team to rely on other designated expertise for the
    human factors input.

### Safety Assessment {#safety-assessment .unnumbered}

The FAA Office of Architecture and System Engineering is responsible for
providing input on the Safety process Risk facet of the IA Team
Alternatives Risk Assessment.[^3]

The FAA System Safety Management Program (SSMP) stipulates a safety risk
management process. For the earliest IA phase (JRC-2a), it requires a
Comparative Safety Assessment (CSA) to support alternative selection.
For JRC-2b, a Preliminary Hazard Analysis (PHA), which is a risk
assessment of hazards at the system level, should be used. This is
consistent with the more detailed focus on one alternative at JRC-2b.

Alerted early enough in the IA, the FAA Office of Architecture and
System Engineering, working in cooperation with the Office of System
Safety, may be able to prepare a CSA in time for the IA Team
Alternatives Risk Assessment.

### Security Assessment {#security-assessment .unnumbered}

The Office of Information Services is responsible for Information
Security, and with the FAA Office of Architecture and System
Engineering, sets uniform methods for anticipating and mitigating
information security risks.[^4] \[Protocols for assessing and mitigating
risks are being developed as of this writing.\] As with human factors
and operational safety, the cognizant security organization provides
needed input to assessing the information security risk of a particular
investment.

The Office of NAS Operations and the Office of Civil Aviation Security
Policy and Planning are responsible for the physical security of FAA
facilities.

In the ideal case, the Security Risk Assessment will be handled
simultaneously with the Information Security Assessment. There may be
different protocols making simultaneous considerations difficult, still
the joint consideration appears reasonable.

## Connections with IPT Programmatic Risk Assessment {#connections-with-ipt-programmatic-risk-assessment .unnumbered}

### Initial Investment Decision at JRC-2a {#initial-investment-decision-at-jrc-2a .unnumbered}

The IA Team Alternatives Risk Assessment contains a high-level
comparative assessment between all alternatives and across all risk
facets. The products of the assessment are a risk assessment briefing,
technical memorandum, and/or a risk management matrix. The risk
management matrix is required for the Initial APB.

These products should be comparable in detail with the IPT Programmatic
Risk Assessment. The list of major risks and mitigation strategies
identified for the preferred alternative are examined in greater detail
at the Final IA Decision to form an APB that is stable throughout the
Solution Implementation phase of AMS.

### Final Investment Decision at JRC-2b {#final-investment-decision-at-jrc-2b .unnumbered}

The products of the IA Team Alternatives Risk Assessment feed into the
IA Team Life Cycle Risk Assessment and gives the IPT a starting point
for the IPT Programmatic Risk Assessment.

The Office of Architecture and System Engineering and the Office of
Investment Analysis and Operations Research, in collaboration with the
IA Team, will provide input on the assessment of cost, benefit,
schedule, performance, and interdependency risks.

The product of the IA Team Life Cycle Risk Assessment is the
identification of detailed cost, benefit, schedule, technical, and
performance risks, and respective mitigation strategies. These
mitigation strategies are discussed with subject matter experts to
estimate their impacts on cost, schedule, benefit, and performance
baseline that the JRC is to approve in the Final IA Decision.

Another product of the IA Team Life Cycle Risk Assessment is the Risk
Mitigation or Management Plan, which guides the understanding and
mitigation of risks throughout the Solution Implementation and
subsequent phases of the AMS.[^5]

## Connections during Solution Implementation {#connections-during-solution-implementation .unnumbered}

The Risk Management Plan proactively guides risk monitoring and
mitigation during the investment's implementation. External and internal
programmatic events may affect the implementation. Accordingly, it will
be useful to anticipate changes and develop contingencies for mitigating
them.

In some cases, there are such significant changes that the program
cannot mitigate the risk consequences because either the impacts exceed
expectations or a new unanticipated risk materializes, and must undergo
a rebaselining to get the program back on track. In undergoing a
rebaselining, the program typically revises the baseline metrics of
costs, schedule, performance, and benefits to reflect the current
circumstances. When this is necessary, the Program Office, together with
the IA Staff, updates the risk assessment reported in the original APB
with a traceable record of how and why the risks changed, and expected
mitigations for the risks.

Reporting of risks for rebaselining follows the same format and
documentation as reporting for the original program baseline. However,
an in-depth review should be necessary only for those risk facets whose
risks have worsened. Any rebaselining of a program must accompany the
record of risk changes for original and past rebaselines.

# IA Team Alternatives Risk Assessment

## Risk Facets {#risk-facets .unnumbered}

The life cycle risks are broken down into thirteen facets of risk. These
risk facets have been selected to facilitate risk identification and
quantification. The thirteen risk facets are defined as follows:

-   **RiskTechnical** is the risk associated with (1) developing a new
    or extending an existing technology to provide greater performance
    than previously demonstrated, or (2) achieving a level of
    performance. It also refers to how well the system operates to
    design or safety specifications.

-   **RiskOperability** is the risk associated with how well the system
    to be produced will operate within the NAS and interact with other
    systems. It addresses NAS or other system interfaces, the degree to
    which they are known and complete, and the degree to which the
    operational concept has been demonstrated and evolved to the point
    of a design baseline.

-   **RiskProducibility** is the risk associated with the capabilities
    to manufacture and produce the desired system.

-   **RiskSupportability** is the risk associated with fielding and
    maintaining the resulting systems.

-   **RiskBenefit Estimate** considers the difficulty in estimating the
    benefits and in realizing benefits. This risk facet addresses the
    accuracy and uncertainty of the benefit estimate, including such
    issues as inadequate methods to estimate the benefits, lack of data
    to estimate the benefits, uncertainty of assumptions, and whether
    the alternative is defined enough to estimate the benefits.

-   **RiskCost Estimate** considers the difficulty in estimating the
    cost and in adhering to the cost. This risk facet addresses the
    accuracy of the cost estimate, including such issues as inadequate
    methods to estimate the cost, lack of data to estimate the cost,
    uncertainty of assumptions, and whether the alternative is defined
    enough to estimate the cost.

-   **RiskSchedule** considers the likelihood that the alternative will
    be completed within the specified schedule.

-   **RiskManagement** refers to complexity of the alternative to manage
    (e.g., number of sub-tasks and/or number of performing
    organizations) and considers the risks of obtaining and using
    applicable resources and activities that may be outside of the
    alternative\'s control but can affect the alternative\'s outcome.

-   **RiskFunding** addresses the availability of funds when they are
    needed and a confidence in management and Congress that those funds
    will continue to be provided.

-   **RiskStakeholder** is the risk associated with various stakeholders
    supporting the development and operation of the alternative, such as
    internal FAA organizational users, Congress, airline and general
    Aviation users, and potential equipment and aircraft manufacturers.

-   **RiskSecurity** addresses a system\'s vulnerability to external
    threats and the risks likely to occur in employing countermeasures.
    This pertains to both information and physical issues.

-   **RiskHuman Factors** focuses on the effectiveness and suitability
    of the human-in-the-loop aspects of the system with respect to both
    operations and maintenance.

-   **RiskSafety** considers the risk associated with the performance
    (or lack thereof) of appropriate safety risk management activities
    and in implementing the identified safety requirements. This risk
    facet shall not be confused with the operational risk of system
    hazards. The operational risk of system hazards are documented in
    other analyses.

## Risk Facet Inclusion and Flexibility {#risk-facet-inclusion-and-flexibility .unnumbered}

If the risk assessment team determines a facet to be inconsequential,
the facet can be eliminated, but only if no perceptible risks for the
facet can be determined. A risk checklist (see Table 6-2) can be used to
assist the analysis in determining whether a facet should be excluded.
The rationale for excluding any facet should be documented.

The risks associated with each facet should consider the whole life
cycle of the investment. Therefore, the thirteen risk facet definitions
must be flexible enough to include the development of the investment
through the full life cycle. Facet definitions need to be flexible
because they must cover all potential risk situations or program
evolutionary stages. Some programs may be at such a preliminary stage
that the risk facet definitions may be difficult to apply. The risk team
may need to adjust and agree to definitions to handle more mature
programmatic risks and preliminary investment risks.

## Clarification of Risk Facets for Benefits and Costs Estimates {#clarification-of-risk-facets-for-benefits-and-costs-estimates .unnumbered}

### Symmetrical Risks Associated with Uncertainty {#symmetrical-risks-associated-with-uncertainty .unnumbered}

Risk may increase or decrease an estimate for benefit performance and
cost values because they may either ameliorate or worsen the behavior of
a system. Therefore, risk considerations should include both sides of
the uncertainties associated with the estimates. One way to capture this
in the IA is to include high and low estimates or a probabilistic
distribution around both costs and benefits. If the benefits are
underestimated, the risk lies in the denial for requested funding of the
program. Similarly, if costs are overestimated, it is possible that the
program will not be funded or another alternative will be chosen with
fewer capabilities. If both cases occurred simultaneously, benefits were
underestimated and costs overestimated, either the alternative will be
different (as well as the accompanying capabilities) and/or the project
may not be funded at all.

### Uncertainty of Cost and Benefit Assumptions, and Qualitative Benefits {#uncertainty-of-cost-and-benefit-assumptions-and-qualitative-benefits .unnumbered}

Risk benefit estimate considers the difficulty in estimating the
benefits and in realizing benefits. This risk facet addresses the
accuracy of the benefit estimate and issues such as:

-   Inadequate methods to estimate the benefits,

-   Lack of data to estimate the benefits,

-   Whether the link of the alternative to projected benefits is
    tenuous,

-   Whether the alternative is defined enough to estimate the avoided
    cost,

-   Uncertainty of the major assumptions used in both cost and benefit
    estimates, and

-   Realization of benefits.

Qualitative benefits should be characterized in as much detail as
possible, and their uncertainty projected. Often qualitative benefits
cannot be measured in financial terms, or in physical numerical terms.
This becomes important when there are potentially large unquantifiable
qualitative benefits, such as enabling benefits from advanced
technology. Therefore, the risks to the accuracy of the benefits
calculations should be included in the risk assessment.

### Cost and Benefit Facet Risk Uses  {#cost-and-benefit-facet-risk-uses .unnumbered}

The IA Team Alternatives Risk Assessment provides more information about
the uncertainty in the cost and benefit estimates. This information
helps to establish the cost and benefits ranges, which in turn are used
to develop the high confidence cost and benefits.

The risk mitigation costs can be included in the total cost figures and
for traceability and validation should be closely linked to the WBS for
costs[^6]. Similarly, risk ranges can be developed for the benefits
calculations by estimating uncertainties for all benefits assumptions.

## IA Team Alternatives Risk Assessment Process {#ia-team-alternatives-risk-assessment-process .unnumbered}

### Matrix Table Results of Probability and Severity  {#matrix-table-results-of-probability-and-severity .unnumbered}

A frequency analysis using a matrix risk table with probabilities and
severities on the X and Y coordinates should be constructed. With three
levels of probabilities and three levels of severity, development of the
risk matrix will yield a 3-row by 3-column table. According to the Risk
Facet Weighting Score Assignment in Section 6, those risks that are high
correspond to (high probability with substantial severity); (medium
probability with substantial severity); and (high probability with
moderate severity). The definitions of medium and low overall ratings
are in Section 6. This can be done for individual risks, as well as by
risk facet. By applying this technique to all of the individual risks
and placing their tallied number in each box for probability and
associated severity, the decision-maker can see the total number of
individual

risks and their distribution through various combinations of both
probability and severity. For example, in Figure 4-1, the number 6 in
the cell corresponding to high probability and substantial severity
denotes that there were 6 total risks with this probability and
severity. The sum of all of the numbers in the table represents the
total number of risks evaluated. Please see Section 6 for more detail on
the definitions of probability and severity associated with each risk
facet.

**Figure 4-1: Example of Risk Facet Rating Matrix Severity of Impact**

Alternatively, the matrix could be applied to risk facets, and by
tallying the facets for each box in the matrix table, a distribution of
the thirteen facets can be clearly evaluated. Furthermore, the risk
facets can be given an initial corresponding to the risk facet placed in
each box in the table. Additional information will allow the
decision-maker to view the highest-ranked risk facets.

### Principles to Ensure Objectivity {#principles-to-ensure-objectivity .unnumbered}

Objectivity is essential to a successful risk assessment. To ensure
objectivity, the risk team evaluators must have an open mind and have
appropriate knowledge and experience. Moreover, the IA Team must be
balanced by a broad perspective of members (i.e., users, airline
representatives, unions, sponsor(s), implementers, and any other
stakeholder).[^7]

Group review of individual risks is important to ensure objectivity as
it allows those with broader experience to add a different perspective
and provide specific information regarding risk mitigation strategies.
Part of the review may include briefing other interested parties or
stakeholders throughout the agency on the results.

All risks should be verified by either data and/or supporting
documentation via risk databases, studies, reviews, or publications.
Increasing the credibility, validity, and traceability of the risk
assessment will enhance the objectivity of the assessment.

# Tailoring the Process

The IA Team Risk Lead is responsible for tailoring the Risk Assessment
to fit a respective acquisition. This tailoring involves evaluating the
acquisition to determine the effort required, choosing an approach for
the Risk Assessment, and selecting the members of the Risk Assessment
Team, if necessary.

## Criteria for Choosing the Level of Effort {#criteria-for-choosing-the-level-of-effort .unnumbered}

Determining the level of effort required for a Risk Assessment involves
a preliminary evaluation of the information currently available on a
certain program or project. This evaluation involves determining the
scope, magnitude, complexity, and schedule of the proposed acquisition,
as well as the impact it will have on the NAS. Other factors to consider
are FAA Advisory Board (FAB) waivers and any congressional mandates that
may have been issued. Information in these areas can be obtained from
documentation on the proposed technology, or documentation on the
proposed program such as an Initial Requirements Document (IRD) or
Concept of Operations document. Subject matter experts and program
office personnel are also valuable sources of information in this
preliminary evaluation.

## Approaches {#approaches .unnumbered}

The IA Team Risk Lead determines the effort required and selects a
corresponding approach.

### Staff Reviews {#staff-reviews .unnumbered}

When an acquisition is on the low end of scope, cost, and complexity,
the IA Team uses staff reviews. This approach involves an internal
ASD-400 review of documentation and comments on facet weights in Risk
Issue Worksheets (see Appendix B) for any risks they have identified.
The IA Team Risk Lead then integrates all risks into a Risk Issue
Briefing and an assessment for the IAR.

### One-Day Meeting {#one-day-meeting .unnumbered}

When an acquisition is thought to be low to mid-range in scope, cost,
and complexity, the IA Team uses the one-day meeting approach. The IA
Team Risk Lead facilitates this meeting involving the IA Team and
representatives from contributing organizations. The IA Team Risk Lead
briefs the Risk Assessment Guidelines for the Investment Analysis
Process and distributes documentation. The IA Team and representatives
from contributing organizations together complete Risk Issue Worksheets
for any risks they have identified and come to consensus on facet
weights. Risk assessment, based on the risk issues identified, is
accomplished as part of this one-day meeting. Results may be available
at the end of the meeting when the assessment is computerized. The IA
Team Risk Lead then integrates all risks into a Risk Issue Briefing and
an assessment for the IAR.

### Sub Teams {#sub-teams .unnumbered}

When an acquisition needs more in-depth expertise to determine
individual risks, mitigation strategies, and accompanying probabilities
and severities, the IA Team uses the sub team's approach. Sub teams and
sub team leads are chosen for each facet from the IA Team, the
sponsoring program office, and representatives from contributing
organizations.

The IA Team Risk Lead facilitates an initial risk meeting involving the
IA Team, the sponsoring program office, and representatives from
contributing organizations. The IA Team Risk Lead briefs the Risk
Assessment Guidelines for the Investment Analysis Process and
distributes documentation.

The entire group comes to consensus on facet weights. Sub team leads are
then chosen and each sub team lead completes a Risk Issue Worksheet for
any individual risks that have been identified. The sub team leads make
any necessary revisions to the worksheets and submit them to the IA Team
Risk Lead. The IA Team Risk Lead then integrates all risks into a Risk
Issue Worksheet and an assessment for the IAR.

### Full IA Team Meetings {#full-ia-team-meetings .unnumbered}

When an acquisition has a large scope, cost, and/or is of high
complexity, the IA Team convenes a full IA Team meeting. The IA Team
Risk Lead facilitates an initial risk meeting with the IA Team, the
program office, representatives from contributing organizations, and
subject matter experts. The IA Team Risk Lead briefs the Risk Assessment
Guidelines for the Investment Analysis Process and distributes
documentation. Sub teams and sub team leads are chosen from contributing
organizations, the sponsoring program office, and the IA Team. Each sub
team member then completes a Risk Issue Worksheet for any risks they
have identified and submits them to the sub team lead. The sub team
leads then makes any necessary revisions to the worksheets and submits
them to the IA Team Risk Lead. The IA Team Risk Lead integrates all
risks into a briefing for the second meeting where all risks, mitigation
strategies and accompanying probabilities and severities are agreed to.
The IA Team Risk Lead then integrates all risks into a Risk Issue
Worksheet and an assessment for the IAR.

### Team Membership {#team-membership .unnumbered}

The selection of team members for a Risk Assessment should start with
the IA Team members, then expand to include the sponsoring program
office, contributing organizations, subject matter experts, and any
other stakeholder. As previously discussed, the level of effort and
approach will help determine the Risk Assessment Team membership.

### How to Select and Where to Find Team Members {#how-to-select-and-where-to-find-team-members .unnumbered}

Risk Assessment Team Members can be selected by contacting major
stakeholders in IA and the various divisions responsible for
contributing different information and assessments to the IAR. Appendix
A is provided for guidance as of April 2002 and is subject to change.

# STEPS IN THE RISK ASSESSMENT PROCESS

The risk assessment process is comprised of the following eight steps:

1)  **Step 1** - Identify risks

2)  **Step 2** - Determine facet weights

3)  **Step 3** - Estimate severity of impact of an adverse event

4)  **Step 4** - Estimate probability of an adverse event

5)  **Step 5** - Assign risk ratings

6)  **Step 6** - Calculate overall alternative risk rating

7)  **Step 7** - Compare risks among alternatives

8)  **Step 8** - Transition to joint responsibility with the IPT for
    JRC-2b

## Step 1 - Identify Risks  {#step-1---identify-risks .unnumbered}

Risk identification is an organized and thorough approach to seek out
risks. It is not a process of trying to invent highly improbable
scenarios of unlikely events to cover every conceivable future
possibility.

A **Top-Level Risk Matrix** (Table 6-1 shows a sample) may be used for
each alternative to provide a structured and consistent risk
identification process for the thirteen risk facets and to document the
results. A Top-Level Matrix is useful for helping frame the development
of risks, as the goals, plans, and risks are defined. This step is
optional, if a framing context is unnecessary.

-   [Define Goals Relating to Risk Facets]{.underline} - Goals (for the
    alternative being assessed) that address potential risks in each
    risk facet are defined in the Top-Level Risk Matrix. By defining
    goals, as they relate to mitigating the potential risks in each risk
    facet, the specific risks that will be important to the alternative
    can be more easily identified. This information will also aid the
    process in Steps 2, 3, and 4 to quantify the risks.

> Requirements specified in an IRD should be considered in defining
> goals. If the requirements are not explicit enough to yield goals
> related to the risk facets, this is an alert that the requirements
> might need to be more explicit.

-   [Define Plans Relating to Risk Facets]{.underline} - Plan(s) for
    achieving the goals related to each risk facet, and hence mitigating
    risks, are listed in the Top-Level Risk Matrix. The Top-Level Risk
    Matrix serves as a forcing function to insure there are plans to
    address all goals.

-   [Identify Risks]{.underline} - Risk identification involves
    identifying the risks pertaining to each risk facet for successfully
    completing and implementing the alternative. The goals and plans
    related to each risk facet will aid in identifying the important
    risks.

-   See Appendix B, Risk Issue Worksheet, for a suggested template for
    recording risk issues. This worksheet contains: (1) risk issue, one
    per worksheet, (2) estimates of the probability and severity of the
    risk for various alternatives, (3) mitigations actions for the
    risk, (4) estimates for probability and severity for the risk after
    the mitigations have been carried out, (5) timing of the mitigation,
    and (6) elements of the work breakdown structure that the mitigation
    will affect.

The above three steps do not have to be sequentially completed.
Iterating among the steps may be helpful. As more risks are listed, the
goals and plans may be revised. The risks listed under each risk facet
in Table 6-1 provide the basis for the risk quantification in Section 7.
The statement of program goals relating to the risk categories and the
plans for mitigating the risks will help quantify the risks. Risks
cannot be assessed or managed until they are identified and described in
an understandable way.

Table 6-1 is presented with sample entries in each box to clarify how
the table is used. The sample entries are constructed for a possible
alternative related to satellite surveillance.

Table 6-1: Top-Level Risk Matrix (Sample)

+-----------+----+-----------------------------------------------------+
| **RISK    | ** |                                                     |
| FACET**   | RI |                                                     |
|           | SK |                                                     |
|           | ID |                                                     |
|           | EN |                                                     |
|           | TI |                                                     |
|           | FI |                                                     |
|           | CA |                                                     |
|           | TI |                                                     |
|           | ON |                                                     |
|           | ** |                                                     |
+-----------+----+-----------------------------------------------------+
| **Te      | ** | To transition from ground-based radar surveillance  |
| chnical** | Go | to a joint satellite and ground-based surveillance  |
|           | al | system.                                             |
|           | s: |                                                     |
|           | ** | Formulate requirements for, develop, and implement  |
|           |    | new technology to provide joint satellite and       |
|           | ** | ground-based surveillance.                          |
|           | Pl |                                                     |
|           | an | -   Undue reliance on currently unavailable or      |
|           | s: |     unproved technology.                            |
|           | ** |                                                     |
|           |    | -   No or minimal prototype testing.                |
|           | ** |                                                     |
|           | Ri | -   Inaccurate/simplistic modeling.                 |
|           | sk |                                                     |
|           | s: |                                                     |
|           | ** |                                                     |
+-----------+----+-----------------------------------------------------+
| **Oper    | ** | To provide users and the FAA with operational       |
| ability** | Go | benefits, such as the implementation of free        |
|           | al | flight.                                             |
|           | s: |                                                     |
|           | ** | Determine the surveillance requirements of free     |
|           |    | flight and other advanced automation programs in    |
|           | ** | order to provide a design that fully satisfies      |
|           | Pl | these requirements.                                 |
|           | an |                                                     |
|           | s: | -   Incompatibilities with future NAS systems.      |
|           | ** |                                                     |
|           |    | -   Incompatible or inconsistent operations with    |
|           | ** |     existing systems or regulations.                |
|           | Ri |                                                     |
|           | sk | -   Uncertain operational requirements of the other |
|           | s: |     programs.                                       |
|           | ** |                                                     |
+-----------+----+-----------------------------------------------------+
| **Produc  | ** | To develop and manufacture ground-based and         |
| ibility** | Go | aircraft-based system components to meet            |
|           | al | requirements and be within the cost estimates.      |
|           | s: |                                                     |
|           | ** | Use non-developmental items (NDI) and commercial    |
|           |    | off-the-shelf (COTS) items, and integrated NDI/COTS |
|           | ** | to the extent possible.                             |
|           | Pl |                                                     |
|           | an | -   Custom design and manufacture required.         |
|           | s: |                                                     |
|           | ** |                                                     |
|           |    |                                                     |
|           | ** |                                                     |
|           | Ri |                                                     |
|           | sk |                                                     |
|           | s: |                                                     |
|           | ** |                                                     |
+-----------+----+-----------------------------------------------------+
| **Support | ** | To provide support for both existing and new        |
| ability** | Go | surveillance systems during transition to the new   |
|           | al | system.                                             |
|           | s: |                                                     |
|           | ** | Closely coordinate with Airway Facilities (AF),     |
|           |    | including the field, and establish the Project      |
|           | ** | Office within the appropriate Integrated Product    |
|           | Pl | Team.                                               |
|           | an |                                                     |
|           | s: | -   Satellite support not under FAA control.        |
|           | ** |                                                     |
|           |    | -   Unclear Logistics Center responsibilities.      |
|           | ** |                                                     |
|           | Ri | -   Existing system may not be maintainable over    |
|           | sk |     the implementation period required for new      |
|           | s: |     system.                                         |
|           | ** |                                                     |
+-----------+----+-----------------------------------------------------+

Table 6-1: Top-Level Risk Matrix (Sample), Cont'd

+------------+---+----------------------------------------------------+
| **RISK     | * |                                                    |
| FACET**    | * |                                                    |
|            | R |                                                    |
|            | I |                                                    |
|            | S |                                                    |
|            | K |                                                    |
|            | I |                                                    |
|            | D |                                                    |
|            | E |                                                    |
|            | N |                                                    |
|            | T |                                                    |
|            | I |                                                    |
|            | F |                                                    |
|            | I |                                                    |
|            | C |                                                    |
|            | A |                                                    |
|            | T |                                                    |
|            | I |                                                    |
|            | O |                                                    |
|            | N |                                                    |
|            | * |                                                    |
|            | * |                                                    |
+------------+---+----------------------------------------------------+
| **Cost     | * | To provide users and FAA with benefits, such as    |
| Estimate** | * | free flight, within estimated program cost.        |
|            | G |                                                    |
|            | o | Implement cost control tools that will be used by  |
|            | a | the program office.                                |
|            | l |                                                    |
|            | s | -   Speculative life cycle costs.                  |
|            | : |                                                    |
|            | * | -   User avionics costs difficult to estimate.     |
|            | * |                                                    |
|            |   |                                                    |
|            | * |                                                    |
|            | * |                                                    |
|            | P |                                                    |
|            | l |                                                    |
|            | a |                                                    |
|            | n |                                                    |
|            | s |                                                    |
|            | : |                                                    |
|            | * |                                                    |
|            | * |                                                    |
|            |   |                                                    |
|            | * |                                                    |
|            | * |                                                    |
|            | R |                                                    |
|            | i |                                                    |
|            | s |                                                    |
|            | k |                                                    |
|            | s |                                                    |
|            | : |                                                    |
|            | * |                                                    |
|            | * |                                                    |
+------------+---+----------------------------------------------------+
| **Benefit  | * | To provide users and FAA with benefits, such as    |
| Estimate** | * | free flight, within estimated program cost.        |
|            | G |                                                    |
|            | o | Implement benefit identification, estimation, and  |
|            | a | tracking tools that will be used by the program    |
|            | l | office.                                            |
|            | s |                                                    |
|            | : | -   Difficult to identify benefits.                |
|            | * |                                                    |
|            | * | -   Difficult to estimate benefits.                |
|            |   |                                                    |
|            | * |                                                    |
|            | * |                                                    |
|            | P |                                                    |
|            | l |                                                    |
|            | a |                                                    |
|            | n |                                                    |
|            | s |                                                    |
|            | : |                                                    |
|            | * |                                                    |
|            | * |                                                    |
|            |   |                                                    |
|            | * |                                                    |
|            | * |                                                    |
|            | R |                                                    |
|            | i |                                                    |
|            | s |                                                    |
|            | k |                                                    |
|            | s |                                                    |
|            | : |                                                    |
|            | * |                                                    |
|            | * |                                                    |
+------------+---+----------------------------------------------------+
| **         | * | To fully implement the new system by the year 20XX |
| Schedule** | * | according to the schedule for the acquisition.     |
|            | G |                                                    |
|            | o | Initiate the acquisition program at the earliest   |
|            | a | possible time. Implement and maintain a program    |
|            | l | office with separate staff and budget, and with    |
|            | s | the authority and responsibility for implementing  |
|            | : | new system.                                        |
|            | * |                                                    |
|            | * | -   Insufficient schedule margin.                  |
|            |   |                                                    |
|            | * | -   Schedule sensitive to technical complexity.    |
|            | * |                                                    |
|            | P | -   Uncertainties in contracting process.          |
|            | l |                                                    |
|            | a | -   Excessive task concurrency.                    |
|            | n |                                                    |
|            | s |                                                    |
|            | : |                                                    |
|            | * |                                                    |
|            | * |                                                    |
|            |   |                                                    |
|            | * |                                                    |
|            | * |                                                    |
|            | R |                                                    |
|            | i |                                                    |
|            | s |                                                    |
|            | k |                                                    |
|            | s |                                                    |
|            | : |                                                    |
|            | * |                                                    |
|            | * |                                                    |
+------------+---+----------------------------------------------------+
| **Ma       | * | To provide the implementation planning, resources, |
| nagement** | * | and controls needed to accomplish the development  |
|            | G | and implementation, while meeting the              |
|            | o | requirements, cost, and schedule estimates         |
|            | a | identified in the program plan.                    |
|            | l |                                                    |
|            | s | Implement and maintain a program office with       |
|            | : | separate staff and budget, and with the authority  |
|            | * | and responsibility for implementing new system.    |
|            | * |                                                    |
|            |   | -   Inadequate program office staffing.            |
|            | * |                                                    |
|            | * | -   Inadequate resource allocation.                |
|            | P |                                                    |
|            | l | -   Inadequate authority.                          |
|            | a |                                                    |
|            | n | -   Undefined integration responsibilities.        |
|            | s |                                                    |
|            | : | -   Unplanned slips in other programs.             |
|            | * |                                                    |
|            | * | -   Excessive span of control.                     |
|            |   |                                                    |
|            | * | -   Uncontrolled requirements changes.             |
|            | * |                                                    |
|            | R | -   Requirements freeze not enforced.              |
|            | i |                                                    |
|            | s |                                                    |
|            | k |                                                    |
|            | s |                                                    |
|            | : |                                                    |
|            | * |                                                    |
|            | * |                                                    |
+------------+---+----------------------------------------------------+

Table 6-1: Top-Level Risk Matrix (Sample), Cont'd

+------------+---+----------------------------------------------------+
| **RISK     | * |                                                    |
| FACET**    | * |                                                    |
|            | R |                                                    |
|            | I |                                                    |
|            | S |                                                    |
|            | K |                                                    |
|            | I |                                                    |
|            | D |                                                    |
|            | E |                                                    |
|            | N |                                                    |
|            | T |                                                    |
|            | I |                                                    |
|            | F |                                                    |
|            | I |                                                    |
|            | C |                                                    |
|            | A |                                                    |
|            | T |                                                    |
|            | I |                                                    |
|            | O |                                                    |
|            | N |                                                    |
|            | * |                                                    |
|            | * |                                                    |
+------------+---+----------------------------------------------------+
| *          | * | To obtain the required development and             |
| *Funding** | * | implementation funding identified in the program   |
|            | G | plan in a timely manner.                           |
|            | o |                                                    |
|            | a | Obtain top-management support; reprogram available |
|            | l | funding to get an early start on the acquisition   |
|            | s | alternative.                                       |
|            | : |                                                    |
|            | * | -   Unfavorable agency priorities.                 |
|            | * |                                                    |
|            |   | -   Inadequate funding.                            |
|            | * |                                                    |
|            | * |                                                    |
|            | P |                                                    |
|            | l |                                                    |
|            | a |                                                    |
|            | n |                                                    |
|            | s |                                                    |
|            | : |                                                    |
|            | * |                                                    |
|            | * |                                                    |
|            |   |                                                    |
|            | * |                                                    |
|            | * |                                                    |
|            | R |                                                    |
|            | i |                                                    |
|            | s |                                                    |
|            | k |                                                    |
|            | s |                                                    |
|            | : |                                                    |
|            | * |                                                    |
|            | * |                                                    |
+------------+---+----------------------------------------------------+
| **Safety** | * | To ensure that the appropriate safety risk         |
|            | * | management effort is implemented in the program.   |
|            | G |                                                    |
|            | o | Identify safety objectives/ requirements/hazards   |
|            | a | and the criteria for acceptable risk for all ATM   |
|            | l | programs.                                          |
|            | s |                                                    |
|            | : | -   Safety hazard and the associated safety        |
|            | * |     requirements ambiguous, not fully              |
|            | * |     characterized                                  |
|            |   |                                                    |
|            | * | -   Interdependent relationships contributing to   |
|            | * |     system failure not fully considered            |
|            | P |                                                    |
|            | l | -   Acceptability criteria not fully known or      |
|            | a |     understood for future NAS environments (e.g.,  |
|            | n |     free flight)                                   |
|            | s |                                                    |
|            | : | -   Safety requirements not validated              |
|            | * |                                                    |
|            | * |                                                    |
|            |   |                                                    |
|            | * |                                                    |
|            | * |                                                    |
|            | R |                                                    |
|            | i |                                                    |
|            | s |                                                    |
|            | k |                                                    |
|            | s |                                                    |
|            | : |                                                    |
|            | * |                                                    |
|            | * |                                                    |
+------------+---+----------------------------------------------------+
| **         | * | To provide a security infrastructure to protect    |
| Security** | * | NAS programs.                                      |
|            | G |                                                    |
|            | o | Formulate plans for the design, procurement,       |
|            | a | configuration, and maintenance of the security     |
|            | l | infrastructure.                                    |
|            | s |                                                    |
|            | : | -   Severity of system vulnerability ambiguously   |
|            | * |     understood.                                    |
|            | * |                                                    |
|            |   | -   Difficulty of threats to system not clearly    |
|            | * |     understood.                                    |
|            | * |                                                    |
|            | P | -   Countermeasures have uncertain operational     |
|            | l |     effectiveness.                                 |
|            | a |                                                    |
|            | n |                                                    |
|            | s |                                                    |
|            | : |                                                    |
|            | * |                                                    |
|            | * |                                                    |
|            |   |                                                    |
|            | * |                                                    |
|            | * |                                                    |
|            | R |                                                    |
|            | i |                                                    |
|            | s |                                                    |
|            | k |                                                    |
|            | s |                                                    |
|            | : |                                                    |
|            | * |                                                    |
|            | * |                                                    |
+------------+---+----------------------------------------------------+
| **Human    | * | To ensure that human-in-the-loop system            |
| Factors**  | * | performance and effectiveness maintain or enhance  |
|            | G | the capability, safety, and capacity of the NAS.   |
|            | o |                                                    |
|            | a | To identify human-in-the-loop requirements and     |
|            | l | objectives to meet benefits goals through          |
|            | s | effective human-system integration to provide a    |
|            | : | system design that is usable, operationally        |
|            | * | suitable, safe, and acceptable to the user         |
|            | * | community.                                         |
|            |   |                                                    |
|            | * | -   Human-in-the-loop benefits or requirements are |
|            | * |     not fully or adequately defined to ensure      |
|            | P |     usability.                                     |
|            | l |                                                    |
|            | a | -   Human-system interfaces fail to provide        |
|            | n |     operational suitability.                       |
|            | s |                                                    |
|            | : | -   Human-in-the-loop interactions with system or  |
|            | * |     other systems contribute to human error or     |
|            | * |     inability to meet task or system performance   |
|            |   |     goals or benefits.                             |
|            | * |                                                    |
|            | * |                                                    |
|            | R |                                                    |
|            | i |                                                    |
|            | s |                                                    |
|            | k |                                                    |
|            | s |                                                    |
|            | : |                                                    |
|            | * |                                                    |
|            | * |                                                    |
+------------+---+----------------------------------------------------+
| **Sta      | * | To meet the user demands for more flexibility in   |
| keholder** | * | flight paths.                                      |
|            | G |                                                    |
|            | o | Involve the user/international community in the    |
|            | a | system design and evaluation process.              |
|            | l |                                                    |
|            | s | -   Resistance to avionics equipage requirements.  |
|            | : |                                                    |
|            | * | -   Diverse user community.                        |
|            | * |                                                    |
|            |   | -   Conflicting user demands.                      |
|            | * |                                                    |
|            | * | -   Conflicting user opinions.                     |
|            | P |                                                    |
|            | l |                                                    |
|            | a |                                                    |
|            | n |                                                    |
|            | s |                                                    |
|            | : |                                                    |
|            | * |                                                    |
|            | * |                                                    |
|            |   |                                                    |
|            | * |                                                    |
|            | * |                                                    |
|            | R |                                                    |
|            | i |                                                    |
|            | s |                                                    |
|            | k |                                                    |
|            | s |                                                    |
|            | : |                                                    |
|            | * |                                                    |
|            | * |                                                    |
+------------+---+----------------------------------------------------+

**Page Intentionally Left Blank**

As an aid in completing the risk lists in Table 6-1, use Table 6-2, Risk
Checklist by Risk Facet. Use this table as a starting point for listing
risks for any alternative. Table 6-2 is comprehensive and addresses all
program stages; hence, the table may contain risk elements not
appropriate to the IA phase or to a particular alternative being
assessed.

The relevant items in the checklist should be evaluated to determine
whether they apply to the particular alternative. Other potential risks
not listed in Table 6-2 should be added to the risk checklist for the
particular alternative. The alternative\'s risk checklist should contain
all possible risks that might be related to the alternative. After
listing all possible risks, those which are extremely unlikely or where
the outcome is irrelevant to program goals should be eliminated from the
list. The checklist should be directed towards those that will have a
meaningful impact on the program, such as impacts on milestones on the
critical path. All meaningful risks should be listed in Table 6-1.

**Page Intentionally Left Blank**

Table 6-2: Risk Checklist by Risk Facet

+------------------------+---------------------+-----------------------+
| **Technical Risks**    | **Operability       | **Producibility       |
|                        | Risks**             | Risks**               |
+========================+=====================+=======================+
| **Technology**         | **System            | **Design Production** |
|                        | Operation**         |                       |
| -   Undue reliance on  |                     | -   Highly complex    |
|     currently          | -   Undefined       |     design            |
|     unavailable or     |     external        |                       |
|     unproved           |     interfaces      | -   Undeveloped       |
|     technology         |                     |     production        |
|                        | -   Marginal        |     requirements      |
| -   Possible better    |     availability    |                       |
|     new technology may |                     | -   Inadequate        |
|     be available by    | -   Insufficient    |     built-in test     |
|     time alternative   |     reliability     |     equipment         |
|     is implemented     |                     |                       |
|                        | -   Inadequate      | -   Non-standard      |
| **System Engineering** |     performance     |     remote            |
|                        |                     |     maintenance       |
| -   Technically        | -   Unsatisfactory  |     monitoring        |
|     incompatible with  |     OT&E results    |                       |
|     NAS Architecture   |                     | -   Novel/unproved    |
|                        | **Systems           |     technologies      |
| -   Inadequate         | Inter-operability** |                       |
|     functional         |                     | **Manufacturing**     |
|     analysis           | -   Operationally   |                       |
|                        |     incompatible    | -   Deficient         |
| -   Deficient          |     with NAS        |     manufacturing     |
|     functional         |     Architecture    |     plan              |
|     allocation         |                     |                       |
|                        | -                   | -   Novel/unproved    |
| -   Incomplete         |   Incompatibilities |     manufacturing     |
|     integration        |     with Concept of |     technologies      |
|                        |     Operations      |                       |
| -   Undefined internal |                     | -   Speculative       |
|     interfaces         | -                   |     manufacturing     |
|                        |   Incompatibilities |     strategy          |
| -   Vague operational  |     with future NAS |                       |
|     environment        |     systems         | -   Custom design and |
|                        |                     |     manufacture       |
| -   Insufficient       | -   Places undue    |     required          |
|     requirements       |     loads on other  |                       |
|     analysis           |     systems         | -   Significant       |
|                        |                     |     special tooling   |
| -   Unstable           | -   Incompatible or |                       |
|     requirements       |     inconsistent    | -   Undefined tooling |
|                        |     operation with  |     requirements      |
| -   Non-compliant or   |     existing        |                       |
|     unvalidated        |     systems or      | -   Unclear           |
|     requirements       |     regulations     |     production        |
|                        |                     |     requirements      |
| -   Weak or            | -   Unspecified     |                       |
|     non-existent       |     operational     | -   Premature         |
|     failure modes      |     interfaces      |     initiation of     |
|     analysis           |                     |     manufacturing     |
|                        | -   Marginal        |                       |
| -   Requirements       |                     | -   Unavailable or    |
|     difficult to trace |   inter-operability |     limited           |
|                        |                     |     manufacturing     |
| -   Unidentified       |                     |     facilities        |
|     safety/security    |                     |                       |
|     considerations     |                     | -   Inadequate        |
|                        |                     |     quality assurance |
| **System Design**      |                     |     program           |
|                        |                     |                       |
| -   Inadequate         |                     | -   Excessive         |
|     capacity           |                     |     standards         |
|                        |                     |                       |
| -   Highly complex     |                     | -   Unavailable       |
|                        |                     |     equipment         |
| -   Lack of design     |                     |                       |
|     details            |                     | -   Inexperienced     |
|                        |                     |     contractor        |
| -   Insufficient       |                     |                       |
|     design margins     |                     | -   Inadequate        |
|                        |                     |     configuration     |
| -   Immature design    |                     |     management        |
|                        |                     |     process           |
| -   Unsatisfactory     |                     |                       |
|     growth potential   |                     | -   Insufficient      |
|                        |                     |     skilled labor     |
| -   Undefined physical |                     |                       |
|     properties         |                     | -   Shallow           |
|                        |                     |     industrial base   |
| -   Incomplete         |                     |                       |
|     hardware design    |                     | **Parts and           |
|                        |                     | Materials**           |
| -   Incomplete         |                     |                       |
|     software design    |                     | -   Undefined long    |
|                        |                     |     lead items        |
| -   Inadequate         |                     |                       |
|     software tools     |                     | -   Unavailable       |
|                        |                     |     government        |
| -   Difficulty of      |                     |     furnished         |
|     developing         |                     |     equipment         |
|     real-time, safety  |                     |                       |
|     critical software  |                     | -   Ineffective       |
|                        |                     |     incoming          |
| -   Immature software  |                     |     materials         |
|     language           |                     |     handling          |
|                        |                     |                       |
| -   Ineffective fault  |                     | -   Unidentified      |
|     detection          |                     |     hazardous         |
|                        |                     |     materials         |
| -   Inordinate use of  |                     |                       |
|     unique resources   |                     | -   Unavailable parts |
|                        |                     |                       |
| -   Complex/incomplete |                     | > **Testing and       |
|     man/machine design |                     | > Documentation**     |
|                        |                     |                       |
| -   Undefined          |                     | -   Inadequate        |
|     technical approach |                     |     consideration of  |
|                        |                     |     special test      |
| **System Test**        |                     |     equipment         |
|                        |                     |                       |
| -                      |                     | -   Insufficient      |
|  Inaccurate/simplistic |                     |     qualification     |
|     modeling           |                     |     testing           |
|                        |                     |                       |
| -   Insufficient       |                     | -   Deficient         |
|     simulation         |                     |     technical data    |
|                        |                     |     package           |
| -   No or minimal      |                     |                       |
|     prototype testing  |                     | -   Ineffective       |
|                        |                     |     factory           |
| -                      |                     |     acceptance test   |
|  Incomplete/inadequate |                     |     program           |
|     test planning      |                     |                       |
|                        |                     | -   Untested design   |
| -   Unsatisfactory     |                     |     changes           |
|     OT&E results       |                     |                       |
|                        |                     |                       |
| **Technical            |                     |                       |
| Documentation**        |                     |                       |
|                        |                     |                       |
| -   Inadequate design  |                     |                       |
|     documentation      |                     |                       |
|                        |                     |                       |
| -   Insufficient test  |                     |                       |
|     documentation      |                     |                       |
|                        |                     |                       |
| -                      |                     |                       |
|   Ambiguous/incomplete |                     |                       |
|     requirements       |                     |                       |
|     documentation      |                     |                       |
|                        |                     |                       |
| -   Undocumented       |                     |                       |
|     technical details  |                     |                       |
+------------------------+---------------------+-----------------------+

Table 6-2: Risk Checklist by Risk Facet, Cont'd

+-------------------+---------------+--------------+------------------+
| **Supportability  | **Cost        | **Benefit    | **Schedule       |
| Risks**           | Estimate      | Estimate     | Risks**          |
|                   | Risks**       | Risks**      |                  |
+===================+===============+==============+==================+
| **O&M**           | **Cost        | **Benefit    | **Schedule       |
|                   | Estimation**  | Iden         | Estimation**     |
| -   Inadequate    |               | tification** |                  |
|     O&M concept   | -             |              | -   Inadequate   |
|                   |    Inadequate | -   Same     |     schedule     |
| -   Undeveloped   |     cost      |     benefits |     estimating   |
|     O&M strategy  |               |     claimed  |     tools        |
|                   |    estimating |     by other |                  |
| -   Specialized   |     tools     |     programs | -   Erroneous    |
|     O&M equipment |               |              |     estimations  |
|                   | -             | -            |                  |
| -   Insufficient  |    Estimation | Unidentified | -   Faulty BOEs  |
|                   |     errors    |     major    |                  |
|   maintainability |               |     benefits | -   Insufficient |
|                   | -             |              |     schedule     |
| -                 |    Inaccurate | -            |     margin       |
|    Unsatisfactory |     discount  |  Unrealistic |                  |
|     maintenance   |     rate      |              | -   Optimistic   |
|     interfaces    |               |   identified |     schedule     |
|                   | -   Faulty    |     benefits |     duration     |
| -   Inadequate    |     BOEs\*\*  |              |                  |
|     maintenance   |               | -            | -                |
|     procedures    | -             |    Difficult |    Inappropriate |
|                   |  Insufficient |     to       |     program      |
| -   Undeveloped   |     cost      |     identify |     schedule     |
|     maintenance   |     margin    |     benefits |                  |
|     plan          |               |              | **Schedule       |
|                   | -             | **Benefit    | Dependency**     |
| -   Configuration |   Unrealistic | Estimation** |                  |
|     management    |     overhead  |              | -                |
|     not enforced  |     and G&A   | -   Benefits |    Unpredictable |
|                   |     rates     |     not      |     labor        |
| -   Deficient     |               |              |     strikes      |
|     change        | -   Relies on | quantifiable |                  |
|     process       |     scarce    |              | -   Improper     |
|                   |     resources | -            |     test         |
| -   Logistics     |               |    Difficult |     scheduling   |
|                   | -             |     to       |                  |
| -   Insufficient  |   Speculative |     estimate | -   Excessive    |
|     spares        |     life      |     benefits |     task         |
|     planning      |     cycle     |              |     concurrency  |
|                   |     costs     | -   Tenuous  |                  |
| -   Spares        |               |              | -   Unidentified |
|                   | -             | relationship |     need for     |
|    unavailability |   Sensitivity |     to       |     procedures   |
|                   |     analysis  |              |     development  |
| -   Inaccessible  |     to cost   |    projected |                  |
|     site location |     drivers   |     benefits | -   Unidentified |
|                   |     not       |              |     need for     |
| -   Inadequate    |               | -   External |     regulations  |
|     training      |    undertaken |     forces   |     development  |
|                   |               |     may      |                  |
| -   Unclear       | **Cost        |     affect   | -   Inordinate   |
|     Logistics     | Management**  |              |     number of    |
|     Center        |               |    achieving |     critical     |
|                   | -   U         |     benefits |     path items   |
|  responsibilities | nsatisfactory |              |                  |
|                   |     cost      | -            | -   Unidentified |
| **Testing and     |     controls  |    Erroneous |     need for     |
| Support**         |               |     benefits |     standards    |
|                   | -             |              |     development  |
| -   Insufficient  |  Insufficient |  estimations |                  |
|     support       |     cost      |              | -                |
|     equipment     |               | -            |    Uncertainties |
|                   |    monitoring |   Inaccurate |     in           |
| -   COTS/NDI -    |               |     inflat   |     contractor   |
|     Industry      | **Product     | ion/discount |     process      |
|     refresh rate  | Cost**        |     rates    |                  |
|     not likely to |               |              | -                |
|     be consistent | -   Undefined | -            |    Uncertainties |
|     with FAA      |               |  Speculative |     in           |
|     needs         |    government |     cost     |     contractor   |
|                   |     furnished |              |     stability    |
| -   Undeveloped   |     equipment |    avoidance |                  |
|     support       |               |              | -   Schedule too |
|     requirements  | -   Reliance  | -   Faulty   |     ambitious    |
|                   |     on        |     BOEs.    |     for degree   |
| -   Inadequate    |               |              |     of technical |
|     automated     |   unavailable |   Inadequate |     complexity   |
|     test          |     NDI/COTS  |              |                  |
|     equipment     |               |   estimating | -   Unavailable  |
|     (ATE)         | -             |     tools    |     materials    |
|                   |   Unavailable |              |                  |
| -   Unidentified  |               |              | -   Unavailable  |
|     field support |    government |              |     parts        |
|     requirements  |               |              |                  |
|                   |    facilities |              | -   Unavailable  |
| -   Poor          |               |              |     government   |
|     diagnostics   | -             |              |     furnished    |
|                   |   Unavailable |              |     information  |
| -   Insufficient  |               |              |                  |
|     testing and   |    contractor |              | -   Unavailable  |
|     support       |               |              |     facilities   |
|     facilities    |    facilities |              |                  |
|                   |               |              | -   Unavailable  |
| -   Unski         | -             |              |     personnel    |
| lled/insufficient |    Inadequate |              |                  |
|     manpower      |     budget    |              | -   Unavailable  |
|                   |     for tests |              |     tools        |
| **Support         |               |              |                  |
| Documentation**   | -   Undefined |              | -   Unavailable  |
|                   |     hardware  |              |     contractor   |
| -   Deficient     |     costs     |              |                  |
|     technical     |               |              | **Schedule       |
|     data          | -   Hidden    |              | Management**     |
|                   |     software  |              |                  |
| -   Faulty        |     costs     |              | -                |
|     maintenance   |               |              |   Unsatisfactory |
|     plan          | -             |              |     schedule     |
|                   |  Unidentified |              |     controls     |
| -   Undefined     |     parts and |              |                  |
|     data rights   |     materials |              | -   Insufficient |
|                   |               |              |     program      |
| -   Inappropriate |               |              |     schedule     |
|     release cycle |               |              |     monitoring   |
|                   |               |              |                  |
| **System          |               |              | -   Improper     |
| Implementation**  |               |              |     contract     |
|                   |               |              | or/subcontractor |
| -   Deficient     |               |              |     schedule     |
|                   |               |              |     monitoring   |
|    implementation |               |              |                  |
|     approach      |               |              |                  |
|                   |               |              |                  |
| -   Uncertain     |               |              |                  |
|     transition    |               |              |                  |
|     strategy      |               |              |                  |
|                   |               |              |                  |
| -   Unclear rules |               |              |                  |
|     and           |               |              |                  |
|     procedures    |               |              |                  |
|                   |               |              |                  |
| -   Insufficient  |               |              |                  |
|     p             |               |              |                  |
| ersonnel/staffing |               |              |                  |
|                   |               |              |                  |
| -   Unspecif      |               |              |                  |
| ied/inappropriate |               |              |                  |
|     standards     |               |              |                  |
+-------------------+---------------+--------------+------------------+

Table 6-2: Risk Checklist by Risk Facet, Cont'd

+-------------------------+------------------+------------------------+
| **Management Risks**    | **Funding        | **Stakeholder Risks**  |
|                         | Risks**          |                        |
+=========================+==================+========================+
| **Planning**            | **Funding        | **Congressional        |
|                         | Constraint**     | Based**                |
| -   Inadequate program  |                  |                        |
|     plans               | -   Unfavorable  | -   Impact of          |
|                         |     agency       |     congressional      |
| -   Incomplete          |     priorities   |     mandates           |
|     contingency plans   |                  |                        |
|                         | -   Inadequate   | -   Unfavorable        |
| -   Deficient risk      |     funding      |     congressional      |
|     management plans    |                  |     hearings on        |
|                         | -   Unavailable  |     program            |
| -   Inadequate          |     funding      |                        |
|     management approach |                  | -   Critical GAO       |
|                         | -   Lengthy      |     report             |
| -   Unplanned slips in  |     budget cycle |                        |
|     other programs      |                  | **Administration       |
|                         | -   Inadequate   | Based**                |
| -   Adverse             |     OMB marks    |                        |
|     environmental       |                  | -   Conflicting FAA    |
|     impacts             | -   Constraining |     priorities         |
|                         |     unique       |                        |
| -   Unsubstantiated     |     budget       | -   Conflicting DOT    |
|     funding profile     |     scoring      |     priorities         |
|                         |     rules for    |                        |
| -   Unsubstantiated     |                  | **Aviation Community** |
|     manpower            |  lease-purchases |                        |
|     requirements        |     and leases   | -   Many different     |
|                         |     per OMB A-11 |     stakeholders       |
| -   Unidentified        |                  |                        |
|     personnel skills    | **Funding        | -   Diverse user       |
|                         | Support**        |     community          |
| -   Minimal resource    |                  |                        |
|     alternatives        | -   Inadequate   | -   Conflicting user   |
|                         |     user support |     demands            |
| -   Excessive           |                  |                        |
|     dependencies on     | -   Ambiguous    | -   Conflicting user   |
|     other system        |     operator     |     opinions           |
|                         |     support      |                        |
| -   Unexpected          |                  | -   Conflicting user   |
|     acquisition         | -   Unclear      |     priorities         |
|     regulation changes  |     political    |                        |
|                         |     support      | -   Inordinate         |
| **Organizing**          |                  |     pressure from user |
|                         | -   Marginal     |     groups             |
| -   Excessive span of   |                  |                        |
|     control             |    cost/benefits | -   Marginal user      |
|                         |                  |     support            |
| -   Inadequate          | -   Inconsistent |                        |
|     authority           |     FAA plans    | -   Strained           |
|                         |                  |     relationships with |
| -   Undefined           | -   Lack of      |     users              |
|     responsibilities    |     alignment of |                        |
|                         |     necessary    | -   Resistance to      |
| -   Unclear             |     funding      |     avionics equipage  |
|     communications      |     profile with |     requirements       |
|                         |     agency       |                        |
| -   Undefined           |                  | -   Inordinate media   |
|     integration         |    affordability |     attention          |
|     responsibilities    |     profile      |                        |
|                         |                  |                        |
| -   Ambiguous           | **Fiscal         |                        |
|     organizational      | Management**     |                        |
|     interfaces          |                  |                        |
|                         | -   Insufficient |                        |
| -   Inadequate          |     funding      |                        |
|     contractor          |     requirements |                        |
|     organization        |                  |                        |
|                         | -   Insufficient |                        |
| **Implementing**        |     fiscal       |                        |
|                         |     controls     |                        |
| -   Insufficient        |                  |                        |
|     management tools    | -   Insufficient |                        |
|                         |     fiscal tools |                        |
| -   Inadequate program  |                  |                        |
|     office staffing     | -   Insufficient |                        |
|                         |     funding      |                        |
| -   Inadequate resource |     plans        |                        |
|     allocation          |                  |                        |
|                         |                  |                        |
| -   Deficient personnel |                  |                        |
|     management          |                  |                        |
|                         |                  |                        |
| -   Lack of             |                  |                        |
|     coordination        |                  |                        |
|                         |                  |                        |
| -   Tenuous top         |                  |                        |
|     management support  |                  |                        |
|                         |                  |                        |
| -   Cumbersome FAA      |                  |                        |
|     contracting process |                  |                        |
|                         |                  |                        |
| -   Instability of      |                  |                        |
|     contractor          |                  |                        |
|                         |                  |                        |
| -   Uncertainties in    |                  |                        |
|     procurement         |                  |                        |
|                         |                  |                        |
| -   Unavailable         |                  |                        |
|     personnel           |                  |                        |
|                         |                  |                        |
| -   Deficient change    |                  |                        |
|     implementation      |                  |                        |
|                         |                  |                        |
| **Control**             |                  |                        |
|                         |                  |                        |
| -   Undefined or        |                  |                        |
|     ineffective change  |                  |                        |
|     management          |                  |                        |
|                         |                  |                        |
| -   Unsatisfactory      |                  |                        |
|     configuration       |                  |                        |
|     management          |                  |                        |
|                         |                  |                        |
| -   Insufficient        |                  |                        |
|     contract evaluation |                  |                        |
|                         |                  |                        |
| -   Inadequate planning |                  |                        |
|     for contractor      |                  |                        |
|     monitoring          |                  |                        |
|                         |                  |                        |
| -   Insufficient        |                  |                        |
|     financial           |                  |                        |
|     management          |                  |                        |
|                         |                  |                        |
| -                       |                  |                        |
|   Irregular/unscheduled |                  |                        |
|     program reviews     |                  |                        |
|                         |                  |                        |
| -   Insufficient        |                  |                        |
|     history/records     |                  |                        |
|                         |                  |                        |
| -   Undefined key       |                  |                        |
|     metrics             |                  |                        |
|                         |                  |                        |
| -   Uncontrolled        |                  |                        |
|     requirements        |                  |                        |
|     changes             |                  |                        |
|                         |                  |                        |
| -   Requirements freeze |                  |                        |
|     not enforced        |                  |                        |
|                         |                  |                        |
| -   Inadequate tracking |                  |                        |
|     systems             |                  |                        |
+-------------------------+------------------+------------------------+

**Table 6-2: Risk Checklist by Risk Facet, Cont'd**

+---------------------+------------------------+-----------------------+
| **Security Risks**  | **Human Factors        | **Safety Risks**      |
|                     | Risks**                |                       |
+---------------------+------------------------+-----------------------+
| **Vulnerability**   | **Human-in-the-loop    | **Hazards**           |
|                     | Effectiveness**        |                       |
| -   Incomplete      |                        | -   Hazards and       |
|     vulnerability   | -   Inadequate         |     service-level     |
|     assessment      |     definition of      |     effects not fully |
|                     |     human-in-the-loop  |     identified        |
| -   Security policy |     operational        |                       |
|     and procedures  |     objectives         | -                     |
|     not in place    |                        |    Inter-relationship |
|                     | -   Inadequate         |     of hazard effects |
| -   Easy access to  |     specification of   |     not established   |
|     communication   |     human-in-the-loop  |                       |
|                     |     benefits           | -   Hazards not       |
| -   No provision    |                        |     classified per    |
|     for firewalls   | -   Inadequate         |     common scheme     |
|     between shared  |     analysis of        |                       |
|     networks or     |     human-in-the-loop  | -   Hazard class not  |
|     Virtual Private |     system capability  |     based on          |
|     Networks        |     to deliver         |     operational       |
|                     |     expected benefits  |     environment       |
| **Threat**          |     or enhancement     |     definition        |
|                     |                        |                       |
| -   Incomplete      | -   Human error        | **System Safety       |
|     threat          |     mechanisms or      | Interdependence**     |
|     assessment on   |     metrics not fully  |                       |
|     intent and      |     identified         | -   Hazard            |
|     capability to   |                        |     interdependence   |
|     exploit         | -   Time required to   |     poorly understood |
|     vulnerability   |     perform tasks is   |                       |
|                     |     unknown            | -   Interoperability  |
| -   No              |                        |     of components on  |
|     prioritization  | -   Automation does    |     system safety not |
|     of threat       |     not provide the    |     investigated      |
|     severity        |     necessary          |                       |
|                     |     functionality or   | -   Systemic approach |
| -   No provision    |     information to     |     to safety is      |
|     for penetration |     support effective  |     lacking one or    |
|     testing         |     decision-          |     more components   |
|                     | making/problem-solving |     (planning,        |
| -   Threat          |                        |     requirements,     |
|     difficulty not  | **Human-in-the-loop    |     procedures,       |
|     considered      | Suitability**          |     operation,        |
|                     |                        |     aircraft          |
| **Countermeasures** | -   Lack of            |     certification, or |
|                     |     consistency,       |     user approval)    |
| -   Few             |     compatibility, or  |                       |
|     countermeasures |     congruity with     | **Mitigation          |
|     defined         |     operational        | Strategies**          |
|                     |     environment or     |                       |
| -   Effectiveness   |     legacy systems     | -   Mitigation        |
|     of              |                        |     strategies not    |
|     countermeasures | -   Human-system       |     shared            |
|     on              |     design/interface   |                       |
|     infrastructure  |     induces            | -   Operational and   |
|     not testing     |     new/additional     |     safety objective  |
|                     |     human error        |     not established   |
| -   Inadequate      |     potential          |                       |
|     configuration   |                        | -   Lack of           |
|     audit           | -   Inadequate         |     critical/valid    |
|                     |     incorporation of   |     safety            |
| -   Lack of         |     functional         |     information       |
|     monitoring and  |     requirements to    |                       |
|     enforcement     |     support            | -   Mitigation        |
|                     |     user-system        |     strategies not    |
| -   Insufficient    |     performance goals  |     tied to hazards   |
|     funding         |                        |     or safety         |
|     tools/controls  | **User Acceptability** |     requirements      |
|                     |                        |                       |
| -   Ambiguous       | -   New tasks impose   | -   Plan for          |
|     funding support |     excessive          |     development and   |
|                     |     attention, memory, |     operational       |
|                     |     or workload        |     assurance not in  |
|                     |     demands            |     place             |
|                     |                        |                       |
|                     | -   Requires new       |                       |
|                     |     teaming and        |                       |
|                     |     communication      |                       |
|                     |     links              |                       |
|                     |                        |                       |
|                     | -   Operations         |                       |
|                     |     interface is       |                       |
|                     |     unacceptable to    |                       |
|                     |     user               |                       |
|                     |                        |                       |
|                     | -   Maintenance        |                       |
|                     |     interface is       |                       |
|                     |     unacceptable to    |                       |
|                     |     user               |                       |
+---------------------+------------------------+-----------------------+

**Page Intentionally Left Blank**

## Step 2 - Determine Facet Weights {#step-2---determine-facet-weights .unnumbered}

The IA Team assigns a numerical weight to each risk facet reflecting its
importance in the successful implementation and operation of the
program. Obviously, a program is successful when it fulfills its mission
need. The following are important points to note in assigning facets
weights:

-   The importance is assigned relative to other facets within the
    rankings (i.e., a facet\'s importance to an acquisition completing
    its life cycle, compared to the importance of the facets directly
    preceding and following it in the rankings).

-   The weights should then be \"normalized". To "normalize" a sequence
    of numbers (1) add the numbers together to obtain a total and (2)
    divide each number of the original sequence by the total. This will
    result in a new sequence of "normalized" numbers; each number in the
    new sequence will be between 0 and 1 and the total of this new
    sequence will be 1.

-   (Note: A weight of zero means that the risk facet does not apply
    (i.e., it falls below the threshold of what is important compared to
    the other facets)

-   A weight of 1.00 means that [only]{.underline} that risk facet
    applies (i.e., it far exceeds the threshold of what is important
    compared to the other facets).

-   The ranking and assignment of facet weights may be the same for all
    alternatives and based on a team consensus (i.e., ASD and the
    sponsoring program office) before other steps of the risk assessment
    process are conducted.

-   The facet weights may remain the same for all alternatives, with the
    exception of the reference case, where the success of the existing
    program may have a different weighting of facets.

-   Do not exclude a facet from the analysis solely because it has a low
    facet weight.

Assigning risk facet weights is a way of conveying the importance of
each facet in bringing about the success of a program. To achieve
objectivity in assigning risk facet weights, the risk team members
should determine facet weights in a full team meeting before the
evaluation of the individual risks. If the determination of the facet
weights were to occur during the risk evaluation or before evaluation of
the alternatives, risk team members might be biased. However, team
members may want to make small adjustments to the weights after the
assessment for sensitivity purposes. Any change in weights at this point
should be well documented and explained.

## Step 3 - Estimate Severity of Impact of an Adverse Event  {#step-3---estimate-severity-of-impact-of-an-adverse-event .unnumbered}

For each risk, the severity of the impact of the adverse event on the
alternative (expressed as substantial, moderate, or minor) is determined
using Table 6-3, Estimating the Severity of Impact of an Adverse Event,
as guidance. The severity from the highest rated risk within each facet
is entered in the third column of Table 6-5.

To estimate the severity of impact the IA Team can use expert
interviews, analogy comparisons, evaluation of program plans, or the
Delphi method. The IA Team can also use multiple methods and approaches
other than those listed.

## Step 4 - Estimate the Probability of an Adverse Event {#step-4---estimate-the-probability-of-an-adverse-event .unnumbered}

In spite of attempts to be analytic about quantifying risks,
considerable subjectivity remains. The degree of risk perceived in a
situation is partially a reflection of the personality of the risk
assessor(s). A risk-rating scheme built against a set of definitions
eliminates some of the ambiguity. Further, the rating scheme should be
simple. The following risk rating scheme involves determining a high,
medium, or low overall risk rating using the notion that the degree of
risk is a judgment reflecting the probability of occurrence of an
adverse event and the severity of impact on the alternative should the
adverse event occur.

If a particular risk facet does not apply to the alternatives being
assessed, then the probability of an adverse event and the severity of
the impact of the adverse event do not need to be estimated for that
risk facet.

For each risk, the probability of occurrence of an adverse event
(expressed as high, medium, or low) is determined using Table 6-4,
Estimating the Probability of an Adverse Event, as guidance. The
probability from the highest rated risk within each facet is entered in
the second column of Table 6-5, Determining Overall Weighted Alternative
Risk Score.

Below are four possible methods to estimate the probability of
occurrence and severity of impact. More than one method, as well as
approaches other than those listed below, can be used.

-   **[Expert Interviews]{.underline}** - Identify expert(s) and
    methodically question them about the risks in their area of
    expertise as related to the alternative. The questioning focuses on
    extracting information about what the program risks are and their
    relative magnitude. Data collection sheets can be used to facilitate
    this process.

-   **[Analogy Comparisons]{.underline}** - Assess risks by using data
    from similar prior programs. The analogy comparisons are based on
    the idea that no new program, no matter how advanced or unique,
    represents a new concept or system.

-   **[Evaluation of Program Plans]{.underline}** - This technique
    highlights and isolates risks caused by insufficiencies and
    disparities in planning. It evaluates program plans for
    contradictions and voids. The plans do not need to be formal plans,
    but can include program management plans, acquisition plans,
    specifications, statements of work, or work breakdown structures.
    The process assesses the plans for correctness, completeness,
    currency, and consistency.

-   **[Delphi Technique]{.underline}** - The Delphi technique is a
    structured method to elicit intuitive thinking by a group and
    produce technological forecasts. It can be used for the systematic
    collection and collation of informed judgments obtained from a group
    of experts and for the refinement of these judgments by an iterative
    process to arrive at a joint judgment or decision. Typically,
    judgments of the individuals in a group are collected, perhaps
    integrated as a group response, and fed back to the individuals.
    Each individual then considers whether to contribute more
    information or to modify earlier views. This iterative process is
    continued until a reasonable consensus is obtained. The responses
    can be fed back anonymously if desired.

Table 6-3: Estimating the Severity of an Adverse Event Impacting the
Program

  -------------------- ------------------- ------------------- ---------------------
  **Facet**            **Substantial       **Moderate Severity **Minor Severity of
                       Severity of         of Impact**         Impact**
                       Impact**                                

  **Technical**        Performance or      Performance or      Performance and
                       problem data        problem data        problem data indicate
                       indicate that with  indicate that with  that only minor
                       current alternative current alternative hardware/software
                       design margins,     design margins,     design changes will
                       full performance    full performance    be needed to meet
                       would not be met    objectives will     full performance
                       and alternate       only be met by: (1) objectives.
                       systems are not     significant         
                       available.          modification to a   
                                           design of a         
                                           component or        
                                           subsystem; or (2)   
                                           reallocation of     
                                           design margins      
                                           among subsystems.   

  **Operability**      No operationally    Technical,          Technical
                       suitable solutions  operationally       operationally
                       available without   suitable solutions  suitable solution is
                       major impacts on    partially           identified and
                       the overall system  identified. The     readily available.
                       performance. Will   solution is not     Will affect a few
                       cause significant   readily available   procedures but
                       impact to existing  or will have        operational
                       procedures, causing significant impacts implementation is
                       operational         on the overall      successful.
                       implementation to   system performance. 
                       be unsuccessful.    Will affect several 
                                           procedures and      
                                           cause operational   
                                           implementation to   
                                           be only partially   
                                           successful.         

  **Producibility**    Manufacturing and   Manufacturing or    Manufacturing and
                       production          production          production
                       capabilities        capabilities in     capabilities known
                       unavailable.        state of change,    and available.
                                           and some            
                                           capabilities will   
                                           be unavailable.     

  **Supportability**   System design       System design       System design
                       characteristics and characteristics and characteristics and
                       planned logistics   planned logistics   planned logistics and
                       and software        and software        software support
                       support resources   support resources   resources meet nearly
                       do not meet system  meet some but not   all system
                       utilization         all system          utilization
                       requirements.       utilization         requirements. Only
                       Support procedures  requirements. Some  minor support
                       or technologies     support procedures  procedures or
                       will be             or technologies     technologies will be
                       significantly       will be impacted    impacted, and
                       impacted and        and transition of   transition of support
                       prevent suitable    support to AF will  to AF to be
                       transition of       be difficult.       successful.
                       support to AF.                          

  **Cost Estimate**    Estimated costs are Estimated costs are Estimated costs are
                       likely to be        likely to be        likely to be exceeded
                       exceeded by more    exceeded by 10 -    by less than 10%.
                       than 25%.           25%.                

  **Benefit Estimate** Less than 75% of    75 - 90% of the     At least 90% of
                       the estimated       estimated benefits  estimated benefits
                       benefits achieved.  achieved.           achieved.
  -------------------- ------------------- ------------------- ---------------------

**Table 6-3: Estimating the Severity of an Adverse Event Impacting the
Program, Cont'd**

+--------+-------------------+-------------------+---------------------+
| **F    | **Substantial     | **Moderate        | **Minor Severity of |
| acet** | Severity of       | Severity of       | Impact**            |
|        | Impact**          | Impact**          |                     |
+========+===================+===================+=====================+
| **Sche | A schedule slip   | A schedule slip   | A schedule slip of  |
| dule** | of more than 25%. | of 10% --- 25%.   | less than 10%.      |
+--------+-------------------+-------------------+---------------------+
| **M    | Management,       | Management,       | Management, funding |
| anagem | funding and       | funding and       | and stakeholder     |
| ent,** | stakeholder       | stakeholder       | facets, and         |
|        | aspects, and      | facets, and       | environments known  |
| **Fu   | environments not  | environments in   | and stable, and     |
| nding, | known,            | state of change   | program or project  |
| and**  | understood, or    | but somewhat      | can be implemented  |
|        | integrated.       | known. Program or | as planned.         |
| **S    | Program or        | project can       |                     |
| takeho | project cannot be | partially be      |                     |
| lder** | implemented as    | implemented as    |                     |
|        | planned.          | planned.          |                     |
+--------+-------------------+-------------------+---------------------+
| **Sa   | Safety            | Safety hazards    | Safety              |
| fety** | requirements      | are identified    | r                   |
|        | /hazards not      | but requirements  | equirements/hazards |
|        | identified for    | for future NAS    | are identified, as  |
|        | future NAS        | systems remain to | well as the risk    |
|        | systems.          | be defined.       | acceptability       |
|        |                   |                   | criteria.           |
+--------+-------------------+-------------------+---------------------+
| **Secu | Security          | Some security     | Complete security   |
| rity** | protection at the | infrastructure is | infrastructure is   |
|        | system perimeter, | available for     | available for every |
|        | workstations, and | some NAS          | NAS                 |
|        | servers are not   | s                 | system/subsystem    |
|        | provided.         | ystems/subsystems | including intrusion |
|        | Security          | including dial-up | detection hardware  |
|        | infrastructure    | protection for    | and software.       |
|        | and intrusion     | remote users.     |                     |
|        | detection         | Some intrusion    |                     |
|        | hardware and      | detection         |                     |
|        | software is not   | hardware and      |                     |
|        | available or not  | software are      |                     |
|        | acceptable.       | available.        |                     |
+--------+-------------------+-------------------+---------------------+
| *      | If one or more of | If one or more of | If all of the       |
| *Human | the following     | the following     | following           |
| Fac    | conditions are    | conditions are    | conditions are      |
| tors** | present:          | present:          | present:            |
|        |                   |                   |                     |
|        | 1)  Size of the   | 1)  Size of the   | 1)  Workload        |
|        |     workforce     |     workforce     |     analyses assure |
|        |     affected by   |     affected by   |     that staffing   |
|        |     system        |     system        |     levels support  |
|        |     changes is    |     changes is    |     system          |
|        |     large and     |     small and     |     performance     |
|        |     staffing      |     staffing      |     goals.          |
|        |     levels and    |     levels and    |                     |
|        |     system        |     system        | ```{=html}          |
|        |     performance   |     performance   | <!-- -->            |
|        |     goals are not |     goals are     | ```                 |
|        |     supported by  |     partially     | 9)  Analyses        |
|        |     workload      |     supported by  |     indicate        |
|        |     analyses.     |     workload      |     personnel skill |
|        |                   |     analyses or   |     and ability     |
|        | ```{=html}        |     by current    |     requirements    |
|        | <!-- -->          |     staffing.     |     are met by      |
|        | ```               |                   |     current         |
|        | 1)  Analyses      | ```{=html}        |     workforce.      |
|        |     indicate      | <!-- -->          |                     |
|        |     personnel     | ```               | 10) Early training  |
|        |     skill and     | 5)  Analyses      |     analyses        |
|        |     ability       |     indicate      |     influenced      |
|        |     requirements  |     personnel     |     alternative     |
|        |     are changing  |     skill and     |     analysis and    |
|        |     or unmet by   |     ability       |     design to       |
|        |     current       |     requirements  |     ensure ease in  |
|        |     workforce.    |     are partially |     performing all  |
|        |                   |     met by        |     critical tasks. |
|        | 2)  Early         |     current       |                     |
|        |     training      |     workforce.    | 11) A               |
|        |     analyses are  |                   |     human-centered  |
|        |     lacking or    | 6)  Early         |     design approach |
|        |     fail to       |     training      |     has been used   |
|        |     influence     |     analyses      |     to design the   |
|        |     selection of  |     partially     |     physical and    |
|        |     design        |     identify      |     cognitive       |
|        |     alternatives  |     factors       |     human-system    |
|        |     for critical  |     affecting     |     integration     |
|        |     tasks such as |     design        |     elements, and   |
|        |     problem       |     alternatives  |     the integration |
|        |     solving and   |     for critical  |     of the system   |
|        |                   |     tasks.        |     and its         |
|        |  decision-making. |                   |     components into |
|        |                   | 7)  Physical and  |     the user work   |
|        | 3)  Physical and  |     cognitive     |     environment.    |
|        |     cognitive     |     human-system  |                     |
|        |     human-system  |     integration   | 12) System changes  |
|        |     integration   |     design        |     affect no       |
|        |     design        |     elements and  |     safety critical |
|        |     elements and  |     integration   |     components and  |
|        |     integration   |     of the system |     analyses have   |
|        |     of the system |     and its       |     proven system   |
|        |     and its       |     components    |     safety and      |
|        |     components    |     into the user |     workforce       |
|        |     into the user |     work          |     health are      |
|        |     work          |     environment   |     assured.        |
|        |     environment   |     have been     |                     |
|        |     have not been |     partially     |                     |
|        |     fully         |     analyzed or   |                     |
|        |     analyzed or   |     partially     |                     |
|        |     do not comply |     comply with   |                     |
|        |     with human    |     human factors |                     |
|        |     factors       |     engineering   |                     |
|        |     engineering   |     best          |                     |
|        |     best          |     practices.    |                     |
|        |     practices.    |                   |                     |
|        |                   | 8)  System        |                     |
|        | 4)  System        |     changes       |                     |
|        |     changes       |     affect minor  |                     |
|        |     affect safety |     safety        |                     |
|        |     critical      |     components or |                     |
|        |     components    |     analyses show |                     |
|        |     and analyses  |     limited       |                     |
|        |     have not yet  |     impact on     |                     |
|        |     proven system |     system safety |                     |
|        |     safety and    |     and workforce |                     |
|        |     workforce     |     health.       |                     |
|        |     health are    |                   |                     |
|        |     assured.      |                   |                     |
+--------+-------------------+-------------------+---------------------+

Table 6-4: Estimating the Probability of an Adverse Event Impacting the
Program

  ------------------------------------------------------------------------------------
  **Facet**            **High Probability of  **Medium Probability  **Low Probability
                       an Adverse Event**     of an Adverse Event** of an Adverse
                                                                    Event**
  -------------------- ---------------------- --------------------- ------------------
  **Technical**        Design unknown.        Design is in          Design is mature.
                       Approach to meet       development or        Technology within
                       requirements carried   prototype phase.      state-of-the-art
                       only through           Technology prototype  or off the shelf.
                       conceptual design and  or engineering model  Performance
                       analysis. Technology   tested in relevant    specifications are
                       is only concept or     environment but not   known.
                       experimental.          operated in fielded   
                                              environment.          

  **Operability**      NAS or other           NAS or other          NAS or other
                       interfaces not fully   interfaces somewhat   interfaces are
                       known or documented.   known and partially   known and
                       Operational concept or documented.           documented. Design
                       implementation of      Operational concept   approaches for the
                       concept has yet to be  has evolved to the    operational
                       established.           point of a design     concept have been
                       Significant changes to baseline. Changes to  demonstrated or
                       procedures are highly  procedures are        implemented. Few
                       likely.                likely.               changes to
                                                                    procedures are
                                                                    likely to.

  **Producibility**    Manufacturing and      Manufacturing or      Manufacturing and
                       production             production            production
                       capabilities are       capabilities are      capabilities are
                       likely to be           likely to change.     not likely to
                       unavailable                                  change.

  **Supportability**   New support            Items similar in      Similar items have
                       technologies and       concept have been     been fielded and
                       procedures or          supported as fielded  are currently
                       substantial            systems or supported  being supported,
                       modifications to       during test.          or similar items
                       existing support       Substantial           have been
                       technologies or        modifications are     demonstrated to be
                       procedures are likely  likely to be required supportable during
                       to be required and     to support            field-testing.
                       could prevent suitable technologies or       Only minor changes
                       transition of support  procedures of AF.     to existing
                       to AF.                                       support
                                                                    technologies or
                                                                    procedures are
                                                                    likely to be
                                                                    required.
                                                                    Transition of
                                                                    support to AF
                                                                    expected to be
                                                                    straightforward.

  **Cost Estimate**    Basis for cost         Cost factors not      Cost factors
                       estimation is          certain, but          understood and
                       inadequate, or major   scope/definition      based on or
                       uncertainties exist    required for          extrapolated from
                       related to the         estimation is         similar items in
                       scope/definition       adequate. Costs are   production.
                       required for           likely to exceed      Definition
                       estimation. Costs are  estimation.           required for
                       likely to                                    estimation is
                       significantly exceed                         adequate.
                       estimation.                                  

  **Benefit Estimate** Major uncertainties    Benefits not certain, Benefits
                       exist related to       but scope/definition  understood and
                       benefit estimation;    required for          based on or
                       extremely tenuous      estimation is         extrapolated from
                       relationship of        adequate; slightly    similar items in
                       alternative to         tenuous relationship  operation.
                       projected benefits; or of alternative to     Definition
                       very likely external   projected benefits;   required for
                       forces will affect     or possible external  estimation is
                       achieving benefits.    forces may have some  adequate. Direct
                                              affect on achieving   relationship of
                                              benefits.             alternative to
                                                                    benefits. Little
                                                                    likelihood of
                                                                    external forces
                                                                    affecting the
                                                                    achievement of the
                                                                    benefits.
  ------------------------------------------------------------------------------------

Table 6-4: Estimating the Probability of an Adverse Event Impacting the
Program

+---------+-------------------+--------------------+-------------------+
| **      | **High            | **Medium           | **Low Probability |
| Facet** | Probability of an | Probability of an  | of an Adverse     |
|         | Adverse Event**   | Adverse Event**    | Event**           |
+=========+===================+====================+===================+
| **Sch   | Many schedule     | Some schedule      | Adequate schedule |
| edule** | interdependencies | interdependencies  | with substantial  |
|         | are likely to     | with little        | margins and       |
|         | develop, for      | schedule margin    | achievable plans  |
|         | which there is    | are likely to      | to minimize       |
|         | little or no      | develop. Plans to  | unknowns. High    |
|         | flexibility to    | minimize unknowns  | knowledge and     |
|         | absorb delays.    | are generally      | experience base.  |
|         | Few or no plans   | complete; some     | There are no      |
|         | to minimize       | uncertainties      | schedule          |
|         | unknowns;         | exist. Little      | dependencies      |
|         | difficult or      | knowledge and      | beyond the        |
|         | complex system to | experience in some | control of the    |
|         | develop.          | areas.             | alternative.      |
|         | Knowledge and     |                    |                   |
|         | experience base   |                    |                   |
|         | very limited.     |                    |                   |
+---------+-------------------+--------------------+-------------------+
| *       | Management,       | Management,        | Management,       |
| *Manage | funding and       | funding and        | funding and       |
| ment,** | stakeholder       | stakeholder        | stakeholder       |
|         | facets, and       | facets, and        | facets, and       |
| **F     | environments      | environments in    | environments      |
| unding, | likely to be      | state of change    | known and stable. |
| and**   | unknown or        | but somewhat       |                   |
|         | unstable.         | known.             |                   |
| *       |                   |                    |                   |
| *Stakeh |                   |                    |                   |
| older** |                   |                    |                   |
+---------+-------------------+--------------------+-------------------+
| **S     | Hazards and their | Process for        | Mitigation        |
| afety** | impact on NAS     | assessing safety   | strategies are    |
|         | services likely   | likely to be       | funded and        |
|         | to be             | developed and      | applied.          |
|         | inadequately      | applied.           |                   |
|         | defined.          | Mitigation         |                   |
|         | Interdependency   | measured are       |                   |
|         | of system         | identified and     |                   |
|         | components in     | related to         |                   |
|         | contributing to   | hazards.           |                   |
|         | system failure    | Mitigation         |                   |
|         | likely to be      | strategies are     |                   |
|         | poorly            | palatable.         |                   |
|         | considered.       |                    |                   |
|         | Mitigation        |                    |                   |
|         | strategies not    |                    |                   |
|         | likely to be      |                    |                   |
|         | directly tied to  |                    |                   |
|         | hazards.          |                    |                   |
|         | Mitigation        |                    |                   |
|         | measures likely   |                    |                   |
|         | to be             |                    |                   |
|         | unpalatable.      |                    |                   |
+---------+-------------------+--------------------+-------------------+
| **Sec   | Vulnerability and | Vulnerability and  | Vulnerability and |
| urity** | threat            | threat assessment  | threat            |
|         | assessments       | planned but not    | assessments       |
|         | likely to be un   | likely to be       | conducted.        |
|         | planned or        | completed or       | Countermeasures   |
|         | conducted.        | conducted on time. | developed for     |
|         | Countermeasures   | Theoretical        | each threat, and  |
|         | notare likely to  | countermeasures    | their ability to  |
|         | be identified or  | are not likely to  | withstand threats |
|         | tested.           | be well            | proved.           |
|         |                   | identified.        |                   |
+---------+-------------------+--------------------+-------------------+

Table 6-4: Estimating the Probability of an Adverse Event Impacting the
Program

+--------+--------------------+--------------------+------------------+
| **F    | **High Probability | **Medium           | **Low            |
| acet** | of an Adverse      | Probability of an  | Probability of   |
|        | Event**            | Adverse Event**    | an Adverse       |
|        |                    |                    | Event**          |
+========+====================+====================+==================+
| *      | If one or more of  | If one or more of  | If all of the    |
| *Human | the following      | the following      | following        |
| Fac    | conditions are     | conditions are     | conditions are   |
| tors** | present:           | present:           | present:         |
|        |                    |                    |                  |
|        | 1)  System         | 1)  System         | 1)  System       |
|        |     requirements   |     requirements   |     requirements |
|        |     or designs     |     or designs     |     and designs  |
|        |     lack           |     include        |     include      |
|        |     human-system   |     incomplete     |     human-system |
|        |     performance    |     human-system   |     performance  |
|        |     objectives or  |     performance    |     objectives   |
|        |     are derived    |     objectives, or |     derived from |
|        |     without        |     are derived    |                  |
|        |     comprehensive  |     with limited   |    comprehensive |
|        |                    |                    |     h            |
|        |  human-in-the-loop |  human-in-the-loop | uman-in-the-loop |
|        |     performance    |     performance    |     performance  |
|        |     research,      |     research,      |     research,    |
|        |     studies, or    |     studies, or    |     studies, and |
|        |     analyses.      |     analyses.      |     analyses.    |
|        |                    |                    |                  |
|        | ```{=html}         | ```{=html}         | ```{=html}       |
|        | <!-- -->           | <!-- -->           | <!-- -->         |
|        | ```                | ```                | ```              |
|        | 13)                | 22)                | 31) Analysis     |
|        |  Human-in-the-loop |  Human-in-the-loop |     indicates    |
|        |     performance    |     performance    |     that         |
|        |     goals are      |     goals are      |     h            |
|        |     unstated or    |     partially      | uman-in-the-loop |
|        |     not achievable |     stated or      |     performance  |
|        |     within the     |     partially      |     goals are    |
|        |     proposed       |     achievable     |     achievable   |
|        |     operational    |     within the     |     within the   |
|        |     and            |     proposed       |     proposed     |
|        |     maintenance    |     operational    |     operational  |
|        |     concepts or    |     and            |     and          |
|        |     using the      |     maintenance    |     maintenance  |
|        |     proposed       |     concepts or    |     concepts and |
|        |     design         |     using the      |     using the    |
|        |     approach.      |     proposed       |     proposed     |
|        |                    |     design         |     design       |
|        | 14) Human          |     approach.      |     approach.    |
|        |     interface      |                    |                  |
|        |     issues and     | 23) Human          | 32) Human        |
|        |     risk           |     interface      |     interface    |
|        |     mitigation     |     issues and     |     issues and   |
|        |     strategies are |     risk           |     risk         |
|        |     not adequately |     mitigation     |     mitigation   |
|        |     supported by   |     strategies are |     strategies   |
|        |     research,      |     partially      |     are          |
|        |     funding,       |     supported by   |     adequately   |
|        |     technical      |     research,      |     supported by |
|        |     expertise, or  |     funding,       |     research,    |
|        |     other          |     technical      |     funding,     |
|        |     resources.     |     expertise, or  |     technical    |
|        |                    |     other          |     expertise,   |
|        | 15) Proposed       |     resources.     |     and other    |
|        |     automation     |                    |     resources    |
|        |     lacks analyses | 24) Analyses show  |     needed to    |
|        |     to ensure full |     proposed       |     complete the |
|        |     functionality  |     automation     |     design       |
|        |     or information |     supports       |     within       |
|        |     to support     |     partial        |     program      |
|        |     user tasks.    |     functionality  |     constraints. |
|        |                    |     and            |                  |
|        | 16) User tasks and |     information    | 33) Automation   |
|        |     skills are not |     needed to      |     provides     |
|        |     well defined   |     support user   |     full         |
|        |     or do not      |     tasks.         |                  |
|        |     conform to     |                    |    functionality |
|        |     current skill  | 25) User tasks and |     to support   |
|        |     levels.        |     skills are     |     user         |
|        |                    |     defined but    |                  |
|        | 17) Human-system   |     changing user  | decision-making. |
|        |     task           |     roles require  |                  |
|        |     performance    |     reevaluation   | 34) User tasks   |
|        |     times are      |     of skills and  |     and skills   |
|        |     unknown or not |     training.      |     are well     |
|        |     quantified.    |                    |     defined or   |
|        |                    | 26) Human-system   |     remain       |
|        | 18) Potential for  |     task           |     essentially  |
|        |     human error    |     performance    |     unchanged.   |
|        |     has not been   |     times are      |                  |
|        |     quantitatively |     partially      | 35) Human-system |
|        |     analyzed or    |     known or       |     task         |
|        |     the impact on  |     partially      |     performance  |
|        |                    |     quantified.    |     times are    |
|        |  human-in-the-loop |                    |     known and    |
|        |     system         | 27) Potential for  |     acceptable.  |
|        |     capabilities   |     human error    |                  |
|        |     is unknown or  |     has been       | 36) Potential    |
|        |     changing.      |     partially      |     for human    |
|        |                    |     analyzed or    |     error has    |
|        | 19) Physical or    |     impact on      |     been         |
|        |     cognitive      |                    |                  |
|        |     human-system   |  human-in-the-loop |   quantitatively |
|        |     integration    |     system         |     analyzed and |
|        |     design         |     capabilities   |     impact on    |
|        |     elements are,  |     is partially   |     h            |
|        |     individually   |     known.         | uman-in-the-loop |
|        |     or in the      |                    |     system       |
|        |     aggregate,     | 28) Physical or    |     capabilities |
|        |     unknown or     |     cognitive      |     is known.    |
|        |     sufficiently   |     human-system   |                  |
|        |     deficient to   |     integration    | 37) Physical or  |
|        |     detract from   |     design         |     cognitive    |
|        |     efficient or   |     elements are,  |     human-system |
|        |     effective task |     individually   |     integration  |
|        |     performance.   |     or taken       |     design       |
|        |                    |     together,      |     elements are |
|        | 20) Requirements   |     partially      |     individually |
|        |     for            |     known.         |     and taken    |
|        |     integration of |                    |     together     |
|        |     the system or  | 29) Some elements  |     sufficiently |
|        |     its components |     for            |     mature to    |
|        |     into the user  |     integration of |     assure       |
|        |     work           |     the system or  |     efficient or |
|        |     environment    |     its components |     effective    |
|        |     are            |     into the user  |     task         |
|        |     undetermined   |     work           |     performance. |
|        |     or changing.   |     environment    |                  |
|        |                    |     are new or     | 38) Integration  |
|        | 21) User groups do |     changing.      |     of the       |
|        |     not contribute |                    |     system or    |
|        |     to             | 30) User groups    |     its          |
|        |     requirements   |     partially      |     components   |
|        |     development,   |     contribute to  |     into the     |
|        |     design, or     |     requirements   |     user work    |
|        |     analysis.      |     development,   |     environment  |
|        |                    |     analysis, and  |     is fully     |
|        |                    |     design.        |     compatible   |
|        |                    |                    |     with the     |
|        |                    |                    |     larger       |
|        |                    |                    |     system and   |
|        |                    |                    |     operations.  |
|        |                    |                    |                  |
|        |                    |                    | 39) User group   |
|        |                    |                    |     input is an  |
|        |                    |                    |     integral     |
|        |                    |                    |     part of      |
|        |                    |                    |     requirements |
|        |                    |                    |     development, |
|        |                    |                    |     design, and  |
|        |                    |                    |     analysis.    |
+--------+--------------------+--------------------+------------------+

Table 6-5: Determining Overall Weighted Alternative Risk Score

+------------+-----------+------------+------------+----------+------+
| **Facet**  | *         | **P        | **Facet    | **Facet  | **   |
|            | *Severity | robability | Risk       | Weight** | Weig |
|            | of Impact | of an      | Rating**   |          | hted |
|            | (Sub      | Adverse    |            | *        | F    |
|            | stantial, | Event      | **Score    | *(0-1)** | acet |
|            | Moderate, | (High,     | (0-10)**   |          | Sco  |
|            | Minor)**  | Medium,    |            |          | re** |
|            |           | Low)**     | **(from    |          |      |
|            |           |            | Table      |          | **   |
|            |           |            | 6-7)**     |          | (0-1 |
|            |           |            |            |          | 0)** |
+------------+-----------+------------+------------+----------+------+
| Technical  |           |            |            |          |      |
+------------+-----------+------------+------------+----------+------+
| O          |           |            |            |          |      |
| perability |           |            |            |          |      |
+------------+-----------+------------+------------+----------+------+
| Pro        |           |            |            |          |      |
| ducibility |           |            |            |          |      |
+------------+-----------+------------+------------+----------+------+
| Supp       |           |            |            |          |      |
| ortability |           |            |            |          |      |
+------------+-----------+------------+------------+----------+------+
| Cost       |           |            |            |          |      |
| Estimate   |           |            |            |          |      |
+------------+-----------+------------+------------+----------+------+
| Benefit    |           |            |            |          |      |
| Estimate   |           |            |            |          |      |
+------------+-----------+------------+------------+----------+------+
| Schedule   |           |            |            |          |      |
+------------+-----------+------------+------------+----------+------+
| Management |           |            |            |          |      |
+------------+-----------+------------+------------+----------+------+
| Funding    |           |            |            |          |      |
+------------+-----------+------------+------------+----------+------+
| Security   |           |            |            |          |      |
+------------+-----------+------------+------------+----------+------+
| Human      |           |            |            |          |      |
| Factors    |           |            |            |          |      |
+------------+-----------+------------+------------+----------+------+
| Safety     |           |            |            |          |      |
+------------+-----------+------------+------------+----------+------+
| S          |           |            |            |          |      |
| takeholder |           |            |            |          |      |
+------------+-----------+------------+------------+----------+------+
| Overall    |           |            |            |          |      |
| Weighted   |           |            |            |          |      |
| A          |           |            |            |          |      |
| lternative |           |            |            |          |      |
| Risk Score |           |            |            |          |      |
+------------+-----------+------------+------------+----------+------+

## Step 5 - Assign Risk Ratings {#step-5---assign-risk-ratings .unnumbered}

Once facet weights are determined, and individual risks are identified
for each alternative, a risk rating score is determined for each risk
and each facet. The highest rated risk in each facet is then used to
represent the facet according to the assignment scheme shown in Table
6-6, Risk Rating Score Assignments.

Table 6-6: Risk Rating Score Assignments

  ------------------- ----------------- ----------------- -----------------
                      **Severity of                       
                      Impact**                            

  **Probability of an **Substantial**   **Moderate**      **Minor**
  Adverse Event**                                         

  **High**            10                8                 5

  **Medium**          8                 5                 2

  **Low**             5                 2                 0
  ------------------- ----------------- ----------------- -----------------

## Step 6 - Calculate Overall Alternative Risk Rating {#step-6---calculate-overall-alternative-risk-rating .unnumbered}

The final step in assessing the risk for any alternative is to calculate
an overall risk rating for the alternative.

The weights, from Step 2, are entered into the fifth column of Table
6-5. Multiplying the entries in the fourth and fifth columns, and
entering the results into the last column of Table 6-5 calculate the
weighted facet score for each risk facet. The overall weighted average
alternative risk score is entered in the bottom row of Table 6-5 by
adding the individual weighted risk facet scores in the last column.

Once the overall weighted alternative risk score is calculated for each
alternative (refer to Table 6-5), a descriptive overall alternative risk
rating (i.e., high, medium, or low) is determined using Table 6-7. This
rating can also be entered into a common table to permit comparison of
risk assessment results across alternatives (refer to Table 6-8).

There are metrics in addition to the overall risk level, which
represents the risk levels weighted by importance of each facet to the
success of the program. Some of these alternative metrics are listed
below.

-   Analyze the horizontal facets (i.e., analyze one facet across
    alternatives) between alternatives. Do this after analyzing the
    vertical facets to evaluate the total overall weighted risk score
    for each alternative.

-   Apply a frequency analysis using a histogram approach by listing the
    distribution of the facet risk ratings (either numerically by risk
    rating score and/or by qualitative group such as high, medium, or
    low) across all facets. This can also be done at the individual risk
    level for the whole program by citing the number of total individual
    risks that are high, medium, or low, and/or the number of risks
    listed by risk rating score.

-   A more traditional method is to present a risk matrix 3 x 3 with
    each cell referring to the number of risks associated with that
    cell's risk. This provides information on the total number of risks
    and their distribution.

-   Interdependencies (see Section 6.7) can be shown in an
    interdependencies matrix 13 x 13.

Table 6-7: Overall Alternative Risk Rating

+------------+---------------------------------------------------------+
| **Overall  | **Description**                                         |
| Rating**   |                                                         |
|            |                                                         |
| *          |                                                         |
| *(Score)** |                                                         |
+------------+---------------------------------------------------------+
| **High**   | Alternatives with high overall risk rating should       |
|            | receive close attention. Risk facets with high-risk     |
| **(7.0 -   | ratings should be considered principal risks. Each high |
| 10)**      | risk should have strategies, metrics, a plan of action, |
|            | and milestones developed by the risk owner. They should |
|            | be aggressively managed and monitored on a continuous   |
|            | basis until the risk is mitigated to an acceptable      |
|            | level.                                                  |
+------------+---------------------------------------------------------+
| **Medium** | Alternatives with a medium overall risk rating require  |
|            | attention. Risk facets should be examined to see if any |
| **(3.0 -   | are rated high and should be placed on the principal    |
| 6.99)**    | risk list and managed as described above. Each medium   |
|            | risk should have candidate strategies, metrics, a plan  |
|            | of action, and milestones developed by the risk owner.  |
|            | They should be frequently managed and reviewed. Any     |
|            | risks on the principal risk list should be aggressively |
|            | managed and monitored on a continuous basis until the   |
|            | risk is mitigated to an acceptable level.               |
+------------+---------------------------------------------------------+
| **Low**    | Alternatives with a low overall risk rating do not      |
|            | normally require attention for risk. However, the risk  |
| **(0 -     | owner should periodically review status. Any high or    |
| 2.99)**    | medium risk facets should receive attention as          |
|            | described above.                                        |
+------------+---------------------------------------------------------+

A risk rating for a subset of risk facets can be determined by summing
the weighted facet scores (last column of Table 6-5) for risk facets in
the subset. For example, the sum of the weighted facet scores for
Technical, Operability, and Supportability will yield a score for the
risk of the alternative output performing as designed.

These risk ratings reflect a comprehensive review of a set of risk
facets relevant to every initial IA. The comprehensive review is
conducted at the macro level to be consistent with the level of analysis
warranted in the initial investment decision. Often the safety risk
management, human factors, or the security organizations of FAA develop
more detailed, microscopic assessments. When this information is
available, the risk information from these specific assessments should
be used as a basis for assigning a risk level for the macro level
assessment. When specific information is not available, every effort
should be made to use the direct input from the respective offices to
ensure that the risk level has been reasonably assigned. In all cases,
the macro level assessments and the micro level assessments should be
consistent, complementary, and mutually supportive.

## Step 7 - Compare Risks among Alternatives  {#step-7---compare-risks-among-alternatives .unnumbered}

The risk assessment process is repeated to determine an overall
alternative risk rating for each alternative. The individual risk facet
ratings and the overall risk rating for all alternatives can be entered
into a table, such as Table 6-8, Risk Ratings of Alternatives, to permit
comparison of risk assessment results across alternatives.

Table 6-8: Risk Ratings of Alternatives

+----------------------------+---------+---------+---------+---------+
| **Risk Assessment          | *       | *       | *       | *       |
| Ratings**                  | *Altern | *Altern | *Altern | *Altern |
|                            | ative** | ative** | ative** | ative** |
|                            |         |         |         |         |
|                            | **1**   | **2**   | **3**   | **. .   |
|                            |         |         |         | .**     |
+----------------------------+---------+---------+---------+---------+
| Technical                  |         |         |         |         |
+----------------------------+---------+---------+---------+---------+
| Operability                |         |         |         |         |
+----------------------------+---------+---------+---------+---------+
| Producibility              |         |         |         |         |
+----------------------------+---------+---------+---------+---------+
| Supportability             |         |         |         |         |
+----------------------------+---------+---------+---------+---------+
| Cost Estimate              |         |         |         |         |
+----------------------------+---------+---------+---------+---------+
| Benefit Estimate           |         |         |         |         |
+----------------------------+---------+---------+---------+---------+
| Schedule                   |         |         |         |         |
+----------------------------+---------+---------+---------+---------+
| Management                 |         |         |         |         |
+----------------------------+---------+---------+---------+---------+
| Funding                    |         |         |         |         |
+----------------------------+---------+---------+---------+---------+
| Security                   |         |         |         |         |
+----------------------------+---------+---------+---------+---------+
| Human Factors              |         |         |         |         |
+----------------------------+---------+---------+---------+---------+
| Safety Process\*           |         |         |         |         |
+----------------------------+---------+---------+---------+---------+
| Stakeholder                |         |         |         |         |
+----------------------------+---------+---------+---------+---------+
| Overall Weighted           |         |         |         |         |
| Alternative Risk Score     |         |         |         |         |
+----------------------------+---------+---------+---------+---------+
| Overall Alternative Risk   |         |         |         |         |
| Rating (H, M, L)           |         |         |         |         |
+----------------------------+---------+---------+---------+---------+

\* The risk shown for the "Safety Process" facet is the risk of not
performing adequate safety risk management. The risk of this facet must
not be confused with the risk associated with operation of the system.
The evaluation of operational risk is documented in system safety design
analysis reports.

To facilitate cross comparison of alternatives, Table 6-8 cells may be
filled with green, yellow, and red shading to represent the
alternatives\' relative risk rating. Any risk facet receiving a score of
10, using Table 6, shall be shaded red for high risk. The same high-risk
designation is reasonable for any risk facet receiving a score of 8.
This is consistent with Table 6-7 score ratings. Similarly, any facet
receiving a score of five shall be shaded yellow or white to represent
medium risk. Lastly, any facet receiving a score of two or zero shall be
shaded green for low risk. In this way, the individual risk of each
facet can instantly be identified and compared.

The risk assessment results contained in Table 6-8 should be used with
the other evaluation factors (i.e., life cycle costs, benefits,
schedule, and performance) to narrow the set of alternatives to the most
promising one(s) to present to the JRC and to justify those in the
subset. The JRC can also use Table 6-8 as part of their decision
information.

**[Interdependencies of Risks]{.underline}**

When interdependencies between risks are expected to be significant, the
risk analyst may complement the comparison of risks among alternatives
by considering the interdependencies of risks for the preferred
alternative. These interdependencies affect the different risk
categories or facets.

Risks most frequently affect the costs, schedule, technical,
benefits/performance, funding, and/or schedule. Entering the risk more
than once is double counting, so an option is to place the risk in the
category in which it has the highest risk rating. Include the risk in
only one category, but describe it in the other risk facets in
qualitative text. An interdependency risk matrix as illustrated in Table
6-9 may be used.

![](media/image4.wmf){width="7.6097222222222225in"
height="2.33125in"}**Table 6-9: Interdependency Risk Matrix**

An interdependency risk matrix consists of a 13 x 13 matrix, 1 row, or
column for each risk facet. Each cell represents the interdependencies
between each row and each column. Within each cell, are two numbers,
with the first number representing the number of interdependent risks
between the row and column risk facet. The second number represents the
highest-ranking risk level (high, medium, or low) between the two risk
facets among all of the interdependent risks. Alternatively, there can
be three numbers, which distributes the number of interdependent risks
among high, medium, or low, with a number to represent the total number
of each risk level. In Table 6-9, the numbers 6/10 would refer to 6
total technical risks that are interdependent with the cost facet, of
which the highest rated risk is rated a risk score of 10, corresponding
to a high risk level.

## Step 8 - Transition to Joint Responsibility with the IPT for JRC-2b {#step-8---transition-to-joint-responsibility-with-the-ipt-for-jrc-2b .unnumbered}

As Figure 2-1 (in Section 2 above) shows, the product of the IA Team
Alternatives Risk Assessment is used to: (1) identify cost and benefit
implications of risk and mitigation strategies, and (2) form the joint
basis for a detailed IA Team Life Cycle Risk Assessment. In the IA Team
Life Cycle Risk Assessment, responsibility is shared between the
Integrated Product Team (IPT) and the IA Staff to develop a detailed
risk assessment of the four APB metrics and an interdependency metric
for the preferred alternative. The final step at the conclusion of
JRC-2a, then, is beginning the transition of risk assessment and
mitigation from the IA Staff leadership to a joint responsibility with
the designated IPT.

The transition is aided by the risk issue database (i.e., the collection
of risk issues) and the assessments from the risk sections of the IAR
and the APB. The risk issue database contains the following:

-   Issue-by-issue description,

-   Risk probability,

-   Risk severity, and

-   Listing of planned mitigation actions.

The risk assessment team compiles the risk issue database as part of the
risk assessment process. The database is available to the IPT, and is a
basis for further risk assessments.

The risk issue database and the IPT's own risk identification protocol
complement each other to synthesize a detailed IA Team Life Cycle Risk
Assessment.

A key aspect for synthesizing the IA Team Alternatives Risk Assessment
is the conversion of risk issue ratings from the IA Team Alternatives
Risk Assessment into the IPT protocol. Having reviewed the IPT
Programmatic Risk ratings criteria for risk likelihood and risk
consequences, a conversion can be readily constructed. This suggested
conversion is mapped in Table 6-10.

Table 6-10: Alternatives Risk Assessment Mapping to IPT Protocol

+-----------------+-----------------+-----------------+-----------------+
| **IA Rating     | **IPT           | **IA Rating     | **IPT           |
| \               | Programmatic**  | --Severity**    | Programmatic**  |
| --Probability** |                 |                 |                 |
|                 | **Rating \--    |                 | **Rating        |
|                 | Likelihood**    |                 | \-              |
|                 |                 |                 | -Consequences** |
+-----------------+-----------------+-----------------+-----------------+
| Low             | 1               | Minor           | 1               |
+-----------------+-----------------+-----------------+-----------------+
| Low             | 2               | Minor           | 2               |
+-----------------+-----------------+-----------------+-----------------+
| Medium          | 3               | Moderate        | 3               |
+-----------------+-----------------+-----------------+-----------------+
| High            | 4               | Substantial     | 4               |
+-----------------+-----------------+-----------------+-----------------+
| High            | 5               | Substantial     | 5               |
+-----------------+-----------------+-----------------+-----------------+

Figure 6-1 shows the coverage of mapping the risk levels. The IPT matrix
is the color matrix shown: red for high risk; yellow for medium; and red
for high-risk levels. The IA Team Risk Assessment matrix ratings result
in the lettered levels: H for high, and so on. As the figure shows the
IA ratings, only cover a portion of the 5 x 5 IPT matrix. In fact, the
IA Team Risk Assessment matrix covers a 3 x 3 matrix. Partial coverage
is understandable, when one considers that the level of detail available
from information in the Initial IA phase of AMS is not very deep. It is
natural, then, to work jointly in the Final IA phase of AMS to examine
the risks further and develop the detail that allows a finer mapping,
and a risk assessment and mitigation that endure through the life cycle
of the investment.

Figure 6-1: Conversion Mapping Coverage

  ------------------------- -------- --------------- ---------- -------- ------- ------
                            5                                                     

                            4                        L          M        H        

  Likelihood/               3                        L          M        M        

  Probability               2                        L          L        M        

                            1                                                     

                                     1               2          3        4       5

                                                                                 

                                     Consequences/                               

                                                     Severity                    
  ------------------------- -------- --------------- ---------- -------- ------- ------

Moreover, there are thirteen facets and four baseline metrics. Each of
the facet issues may affect cost, schedule, performance, or the benefit
metric. Which metric is impacted may turn out to be a judgment call, and
this necessarily requires a joint review and collaborative discussions,
hence the need for joint responsibility.

# Risk assessment products and Mitigation Strategies

Each risk identified as part of the IA Risk Assessment must have an
accompanying mitigation strategy. A mitigation strategy should
adequately describe actions required to mitigate that risk to an
acceptable level. Mitigation strategies should also include what
schedule, benefit, and cost implications implementing the mitigation
strategy will have.

## Products of IA Team Alternatives Risk Assessment  {#products-of-ia-team-alternatives-risk-assessment .unnumbered}

There are generally three essential products that the IA Risk Assessment
yields: 1) a section of the IAR, 2) input to the APB, and 3) the Risk
Issues Database.

### Investment Analysis Report {#investment-analysis-report .unnumbered}

The IAR contains a separate section for the risk assessment results and
findings that will contain the following:

-   Risk summary table,

-   Risk summary matrix,

-   Small paragraph on how the assessment was conducted, and

-   Listing of all the high and medium level risks with mitigation.

**Risk Summary Table** (see Table 6-8) consists of listing of each risk
facet with its risk level of high, medium, or low. At the bottom of the
table should be the Overall Risk Level for the program as determined by
applying the facet weights to the numerical value for each facet's risk
level.

**Risk Matrix** (similar to Figure 4-1) records the number of individual
risks contained in each 3 x 3 cell corresponding to three rows/levels of
probability (high, medium, and low) and three columns/levels of severity
(substantial, moderate, and minor). The high-risk cells in the table are
colored red, the medium risks cells are colored yellow, and the low
risks cells are colored green.

**Describe** the risk assessment process. Topics can include the use of
sub teams, experts, or panels, and the overall processes used to list
the risks, score the risks (probability and severity), and develop
mitigation strategies. Describe how the facet weights were derived;
describe, in detail, why the risk was assigned at a particular risk
level; and describe mitigation in as much detail as possible.

The cost analysis team will use these risk inputs to develop their high
confidence costs. Eventually, the risks and their mitigations will be
transferred to the IPT.

Listing the medium and high risks for each alternative is essential to
the IAR. If there appears to be a large number of medium and high risks,
the author may prefer to list the high risks in the body of the IAR and
list the medium risks in an appendix.

### Acquisition Program Baseline {#acquisition-program-baseline .unnumbered}

The APB is a summary review of the program's baseline costs, benefits,
and risks and is much smaller than the IAR. Therefore, the risk
assessment section of the APB will contain only the following:

-   Risk summary table,

-   Risk matrix, and

-   Summary discussion of the overall program risk level.

If the APB needs more detail, describe the highest risks.

### Risk Issues Database {#risk-issues-database .unnumbered}

The appendixes contain a template for compiling risks into a risk issues
database. The risk issues database is the collection of the risk issues
identified during the assessment. Eventually, the IA Team will hand over
this risk database to the IPT who will incorporate it into their program
risk plan and database. The IPT will then become the owner of the risks
and be responsible for mitigating them.

## Mitigation Strategies {#mitigation-strategies .unnumbered}

### Mitigation Actions and Cost Range Estimation {#mitigation-actions-and-cost-range-estimation .unnumbered}

After the Risk Assessment Team has identified all risks, the IA Teams
Risk Lead will compile all high and medium risk worksheets into a
briefing. This briefing will then be used in a meeting with the Cost
Estimating Team, the Risk Assessment Team, and with any other major
stakeholders to ensure accurate ranges are attached around the elements
of the WBS affected by the mitigation strategies for high and medium
risks. The objective of this meeting is to confirm that the cost of each
mitigation strategy for a high or medium risk is covered in the cost
estimate and can be implemented by the program with the money allotted
to the different elements of the WBS.

Information gathered during the risk assessment can also be used to
attach risk ranges around the different elements of the WBS before Monte
Carlo simulations are run to generate cumulative distributions of total
possible program costs, allowing areas of identified uncertainty to be
accounted for. Using this information at this stage will improve the
accuracy of forecasted cost distributions.

It is not always obvious whether an activity is part of an alternative
or whether it is a mitigation activity. It is important that the cost of
the activity be included in the cost estimate and that there be
consistency among the alternatives that the activity be judged part of
the alternatives themselves or part of the mitigations to risks.

### Mitigation Actions and Benefits Range Estimation {#mitigation-actions-and-benefits-range-estimation .unnumbered}

After the Risk Assessment Team has completed identifying all risks, the
IA Team Risk Lead will compile all high and medium risk worksheets into
a briefing. This briefing will then be used in a meeting with the
Benefits Estimating Team, the Risk Assessment Team, and any other major
stakeholders to ensure accurate ranges are attached around benefits
estimates affected by mitigation strategies for high and medium risks.
This meeting is to confirm that the cost and impact of each mitigation
strategy for a high or medium risk is covered in the benefits estimates.

### Tracking and Evaluation of Mitigations {#tracking-and-evaluation-of-mitigations .unnumbered}

The program office is responsible for maintaining a risk database to see
that all risk mitigation strategies are carried out and all high and
medium risks are acceptably mitigated. The program office will use
Programmatic Risk Mitigation Plan and schedules ending with Risk
Resolution Date.

**Page Intentionally Left Blank**

# Evolution of Risk Assessment

## Aviation Environment Changes {#aviation-environment-changes .unnumbered}

The foregoing Risk Assessment Guidelines describe a standard operating
procedure, which can be replicated, traced, and verified. Even though
implementing a procedure adds consistency, risks and the risk assessment
process still evolve. Not only will risks change throughout the life
cycle of a program as more information is known, but mitigations will
eliminate some risks. Additionally, the focus of risks may change over
time. Risks may be added to the risk facet list and/or combined into
others. With the advent of new technologies and procedures, the number
of risk facets may grow to incorporate potential risk areas not included
in the current set of risk facets.

## Methodological Changes {#methodological-changes .unnumbered}

As risks and the risk assessment processes change, the methods of
assessing risk may change. Additional metrics may help to clarify the
description, level, and distribution of risks.

The following are examples of changes in methodology that may be applied
to the risk assessment:

-   Cross tabulating a risk facet across alternatives,

-   Adding a risk matrix to the original Guidelines for Risk Assessment,

-   Expanding the risk matrix or risk evaluation categories for
    probability and severity, and

-   Changing risk facet definitions to encompass a wider range of risks.

**Page Intentionally Left Blank**

**APPENDIX A**

**TEAM PARTICIPANTS**

**Page Intentionally Left Blank**

####   {#section .unnumbered}

#### Appendix A: Team Participants

  -------------------- ----------------------- ------------------- --------------
  **FACET**            **Contributing          **Possible FAA      **Contact\'s
                       Organizations**         Contacts**          \#**

  **Technical**        AUA/AND representative                       

                       ARU-100                 Steve Anderson      202-385-7651

                       ASD-100                 Phil Decara         202-358-5323

                       ASD-100                 Betty Falato        202-358-5377

  **Operability**      ARU-100                 Steve Anderson      202-385-7651

                       Air Traffic Contact                          

  **Producibility**    ASD-100                 Phil Decara         202-358-5323

                       AUA/AND representative                       

                       Market Survey Author                         

  **Supportability**   ARU-100                 Ralph Taylor        202-385-7661

                       AOS-320                 Ana Mack            609-485-6629

                       AUA/AND representative                       

  **Benefits**         ASD-430                                      

  **Cost**             ASD-410                                      

  **Schedule**         AUA/AND representative                       

  **Management**       AUA/AND representative                       

  **Funding**          ASD-300                 Greg Street         202-358-5287

                       ARQ-100                 Debra Griffith      202-358-7509

                       AFZ-400                 Cynthia Beck        202-267-3588

  **Stakeholder**      ARU-100                 Ralph Taylor        202-385-7661

                       ARQ-300                 Bob Fitzpatrick     202-358-7608

  **Information        ASD-100                 Mary Beth Dormuth   202-358-5381
  Security**                                                       

                       ASD-4                   Feisel Keblawi      202-358-5317

                       AUA/AND representative                       

                       AIS                                          

                       ACS                                          

  **Human Factors**    AAR-100                 Dino Piccione       202-493-5305

                       AAR-100                 Glen Hewitt         202-267-7163

  **Safety**           ASD-110                 Scott Van Buren     202-358-5326
  -------------------- ----------------------- ------------------- --------------

**Page Intentionally Left Blank**

**APPENDIX B**

**RISK ISSUE WORKSHEET**

**Page Intentionally Left Blank**

#### Appendix B: Risk Issue Worksheet

![](media/image5.wmf)

[^1]: "Think Value", page 2-15, May 1999, DMR Associates.

[^2]: An example of the Job Aid can be found at the following URL for
    FAA Acquisition System Toolset:
    <http://fast.faa.gov/human/htm/ccontent.htm>.

[^3]: More information on System Safety as it applies to all AMS phases
    can be found at the following URL for FAA Acquisition System
    Toolset: http://fast.faa.gov/toolsets/safmgmt/index.htm.

[^4]: More information on Security Risk as it applies to all AMS phases
    can be found at the following URL for FAA Acquisition System
    Toolset:
    http://fast.faa.gov/Riskmgmt/Secriskmgmt/secriskmgmt.htm#11.1.

[^5]: For more information on Risk Mitigation Plans, please see the
    course notes for "FAA Programmatic Risk Management."

[^6]: Please see the following URL for further information on the Work
    Breakdown Structure: http://fast.faa.gov/wbs/wbssec.htm.

[^7]: Prior risk team lists have been generated for previous IAs and can
    be found in ASD-400 Corporate History database.