![](media/image1.png){width="2.3125in" height="2.21875in"}

Investment Analysis Risk Guidelines for

Final Investment Decision

#### April 2003 {#april-2003 .unnumbered}

Art Politano

Investment Analysis and Operations Research, ASD-400

Federal Aviation Administration

800 Independence Avenue, SW

Washington, DC 20591

**Page Intentionally Left Blank**

This document presents the guidelines by which the Investment Analysis
(IA) team may conduct Risk Assessments for Final Investment Decisions,
as part of the Federal Aviation Administration's (FAA's) Acquisition
Management System (AMS). The guidelines were developed to assist IA
Teams in their analysis of a preferred alternative.

This risk methodology builds on the IAT Alternatives Risk Assessment for
the Initial Investment Decision. Using these risks as a starting point,
the methodology in these guidelines updates the risks and funnels them
to the four baseline parameters. The baseline parameters are cost,
benefits, schedule, and performance. For each of these parameters then,
the risk information is used to support risk-adjusted baselines.

We are learning that environmental issues are playing an increasing role
in Investment Analysis. In addition, we recognize that the methodology
covered in this report fits acquisition of systems quite well, but
facilities not so well. Accordingly, this methodology should be viewed
as an evolving one, and needs to be constantly updated to remain
relevant to the context and nature of all Investment Analyses.

**Page Intentionally Left Blank**

**Table of Contents**

# [1.0 Background 1](#__RefHeading___Toc30474863) {#background-1 .TOC-1}

[1.1 Consistency with Revised Policy
1](#consistency-with-revised-policy)

[1.2 Risk Issues are More Critical 1](#risk-issues-are-more-critical)

# [2.0 Risk for the Initial Investment Decision 2](#risk-for-the-initial-investment-decision) {#risk-for-the-initial-investment-decision-2 .TOC-1}

[2.1 History of Risk in Initial Investment Decision Analysis
2](#history-of-risk-in-initial-investment-decision-analysis)

[2.2 Focus of Guidelines 2](#focus-of-guidelines)

[2.3 Risk Process and Products for Initial Investment Decision
2](#risk-process-and-products-for-initial-investment-decision)

# [3.0 Final Investment Decision Framework 7](#final-investment-decision-framework) {#final-investment-decision-framework-7 .TOC-1}

[3.1 Contracting Implications 7](#contracting-implications)

[3.2 Trackable Baselines and Congressional Sanctions
7](#trackable-baselines-and-congressional-sanctions)

# [4.0 Risk Information Flow Between Initial and Final Investment Decisions 8](#risk-information-flow-between-initial-and-final-investment-decisions) {#risk-information-flow-between-initial-and-final-investment-decisions-8 .TOC-1}

[4.1 Detailed Risks, Consequences and Mitigation from Initial Investment
Decision
9](#detailed-risks-consequences-and-mitigation-from-initial-investment-decision)

[4.2 Update of Risk Issues 9](#update-of-risk-issues)

[4.3 Funneling of All Risks to Baseline Parameters
10](#funneling-of-all-risks-to-baseline-parameters)

[4.4 Assessment for Each Parameter 10](#assessment-for-each-parameter)

[4.5 Integration of Human Factors, Information Security, and Safety Risk
Issues
13](#integration-of-human-factors-information-security-and-safety-risk-issues)

# [5.0 Calculation of Cost, Schedule, Benefit and Performance Risk 14](#calculation-of-cost-schedule-benefit-and-performance-risk) {#calculation-of-cost-schedule-benefit-and-performance-risk-14 .TOC-1}

# [6.0 Process Integration 16](#process-integration) {#process-integration-16 .TOC-1}

# [7.0 Basis of Risk Estimate 16](#basis-of-risk-estimate) {#basis-of-risk-estimate-16 .TOC-1}

# [8.0 Products 16](#products) {#products-16 .TOC-1}

# [9.0 Planning for Life Cycle Risk Mitigation 17](#planning-for-life-cycle-risk-mitigation) {#planning-for-life-cycle-risk-mitigation-17 .TOC-1}

# [10.0 Closure 17](#closure) {#closure-17 .TOC-1}

[References 18](#references)

[Appendix: Standard Basis of Estimate for Risk Analyses
19](#appendix-standard-basis-of-estimate-for-risk-analyses)

# Background 

## Consistency with Revised Policy

The Acquisition Management System Policy for Investment Analysis was
modified in February 2002. Investment Analysis was divided into two
distinct decision points, the Initial Investment Decision, and the Final
Investment Decision. The Initial Investment Analysis rigorously
evaluates alternative solutions to mission need and determines which
offers the best value and most benefit to the agency and its customers
within acceptable cost and risk. The Final Investment Analysis develops
detailed plan and final requirements for the proposed investment
program, including a life cycle program baseline that establishes cost,
schedule, performance, benefits parameters, and defines risk-management
boundaries for program execution. While guidelines have been developed
and adopted into AMS (September 2002) for the Initial Investment
Decision, no guidelines exist for the Final Investment Decision. Indeed,
there is a great misunderstanding as to how best to assess the risks for
a Final Investment Decision. This paper, then, is designed to provide
guidance on how best to conduct a risk assessment for the Final
Investment Decision.

The approach builds on the risk information gleaned as part of the
Initial Investment Decision. The approach re-examines the importance of
the risk issues identified, updates the risk identification of all
issues, frames the assessment to be compatible with human factors,
information security, and safety considerations, and, produces a product
which strengthens the investment program by facilitating the risk
assessment and mitigation in the Solution Implementation and the
In-Service Management Phases of AMS.

## Risk Issues are More Critical

Currently, risk assessment is not defined well enough for the Final
Investment Analysis. In the Initial Investment Analysis, the risks and
attendant mitigation actions are defined at an aggregate level, e.g., a
level specific enough to assist in distinguishing the merits of
investment alternatives. The Acquisition Program Baseline (APB), which
is produced from the Initial Investment Decision, is essentially used as
a placeholder to develop a funding wedge for the Acquisition Program.
For the Final Investment Decision, the resulting APB is the basis of a
contracted agreement for the FAA to assure the availability of a
specific amount of funding for implementing the acquisition. On the
basis of the APB available from the Final Investment Decision, the
Integrated Product Team (IPT) is to be able to enter into a contract
with a vendor and procure a NAS System. Should the APB not reflect
realistic risk-adjusted costs, then there is a significant risk that the
program baselines will be breached.

# Risk for the Initial Investment Decision

## History of Risk in Initial Investment Decision Analysis

The use of risk analysis for FAA Investment Analysis began with the
publication of *Risk Assessment Guidelines for the Investment Analysis
Process* in March of 1997. The *Guidelines*, in turn, drew heavily from
the risk concepts of *Acquisition and Program Risk Management Guide*,
Revision 1, FAA-P-1810 of September 29, 1995. The *Guidelines* were
revised in 1999, to add human factors, information security, and safety
process considerations, and in 2002 to focus their relevance to the
Initial Investment Analysis. The title also changed to *Guidelines for
Investment Analysis Team's Alternatives Risk Assessment* to reflect the
purpose of the effort: to comprehensively examine the range of
alternatives under investment consideration.

In September 2002, the 2002 *Guidelines* were published in the FAA
Acquisition Management System Toolset. They appear in the Investment
Analysis Section, under the Standard Guidelines:
[**http://fast.faa.gov/ams/non_index/investab.htm**](http://fast.faa.gov/ams/non_index/investab.htm).
The section also includes: Risk Metrics for the Initial Investment
Analysis and the Basis of Estimate for Risk Analysis.

## Focus of Guidelines

In general, within the IA Team's Alternatives Risk Assessment, there are
thirteen facets that constitute an exhaustive range of potential issues,
which include: technical, operability, producibility, supportability,
benefits estimate, cost estimate, schedule, management, funding,
stakeholder, information security, human factors, and safety process.

The *Guidelines,* given the changing nature of the NAS (National
Airspace System) environment, should be taken as dynamic and expected to
change in order to keep them relevant to evolving NAS issues. Most
recently, for example, we have learned of the need to add (a) a risk
facet focusing on environmental issues, and (b) risk items for facility
buildings. This way, the risk facets and risk checklist would expand the
relevance of the methodology to be applicable to a larger range of risk
issues.

## Risk Process and Products for Initial Investment Decision 

Application of the *2002 Guidelines* entails following a deliberate,
objective process for identifying and planning for risks.

The initial step is to identify potential risk issues, using a standard
risk issues checklist. In the course of identifying issues, either
through research of available written materials or interviews with
knowledgeable stakeholders, the IA team often formulates and records
preliminary mitigations, and estimates the likelihood of the issue
surfacing and the potential severity of the issue. Stakeholders may
include manufactures, operational users, airlines, etc.

The next step may be to convene a team of stakeholders, investment and
program analysts, union representatives, and other interested
colleagues. A team comprised of a broad functional-set helps to form an
objective assessment. The team reviews the material and arrives at
consensus of each facet's rating for each alternative by double blind
voting (independent votes not visible to others) and justification
discussions. The team also affirms or complements issue mitigations. At
the end of these discussions, the least-risk alternative emerges.

The analyst then coordinates with the cost, benefit, and other analysts
to link the impact of the risks and their mitigation on the cost and/or
the benefit estimate of the recommended alternative. This impact
contributes to developing ranges around the most likely cost or benefit
element in the respective Work Breakdown Structure (WBS).

Discussion with program staff, users, and stakeholders can help identify
risk issues.

The products of this effort may be a technical memorandum, provided by
the risk assessment team lead, but most often are:

-   A stand-alone summary section in the Investment Analysis Report
    (IAR)

-   Solid input for creating cost and benefit high-confidence ranges
    around the most likely numbers

-   Direct input into the Risk Management Matrix in the APB

-   Risk Issues Database

The following hypothetical example illustrates the risk assessment
products used for program XXX. The IA team identified 120 individual
risks in the XXX Risk Assessment. Tables 1 and 2 summarize the
criticality of these risks. Table 1 shows only 68 risks were medium or
higher, and of these 7 were high risks. AMS requires mitigation of all
medium or higher risks. Table 2 identifies risk facets by level of risk.
Funding and information security were assessed as the highest risk.

**Table 1: XXX** **Risk Facet Rating Matrix**

> **Severity of Impact**

**Table 2:** Risk Score Summary

  ------------------------------------ ----------------------------------
  **FACET**                            **OVERALL RISK RATING**

  Funding                              High

  Schedule                             Medium

  Human Factors                        Medium

  \*Safety                             Medium

  Technical                            Low

  Benefits                             Low

  Info Sec                             High

  Management                           Medium

  Operability                          Medium

  Stakeholder                          Low

  Producibility                        Low

  Supportability                       Low

  Cost                                 Low

  \*\*Overall Weighted Alternative     \*\*Medium
  Risk Score                           
  ------------------------------------ ----------------------------------

The Investment Analysis Report for XXX reported:

-   Funding-related issues, including inadequate funding and
    affordability, represent high risks to the XXX program. Comparison
    of the XXX program's life-cycle costs and the Capital Investment
    Plan (CIP) funding line suggested that XXX could face funding
    shortfalls in future years. Any failure to sustain XXX funding could
    have a substantial impact upon the NAS because XXX is a critical
    path to modernization.

-   Definition of an XXX Build 0 is itself a mitigation strategy to
    ensure that only the most critical functions are implemented first.
    Keeping FAA's management informed on the criticality and status of
    the XXX program can also mitigate funding risks. Other strategies
    for mitigating funding risk focus on the program office's management
    processes: development of a high-confidence life-cycle cost estimate
    that takes into account the cost of mitigation strategies provides
    the program confidence in its baseline. Establishing financial
    management systems will provide program management timely
    information on the status of funds.

-   Information security is rated as high risk because of uncertainty
    associated with unknown potential security requirements changes as a
    result of the need for increased security. Other information
    security issues center around the need for security testing using
    scanning tools to identify residual risks, and system failure
    recovery testing.

The risk assessment addressed such high-risk areas as information
security and human factors. Cost ranges were assigned to CLINS such as
production, procurement, and deployment to address the additional
dollars for software and hardware that would be necessary to increase
security and provide redundancy to the XXX system. Additional risk
dollars were added to testing, system engineering, and program
management to mitigate risks associated with human factors.

Based on the risks identified, low and high values of variables were
entered into Crystal Ball, an analysis software package, to calculate a
high-confidence cost estimate. Crystal Ball uses Monte Carlo simulation,
a statistical sampling technique, to incorporate the risk ranges and
arrive at a cost estimate considered a high-confidence value. The
high-confidence percentile value was recorded and was used to develop
the budget and the APB.

The final product to point out is the risk issue database. A
hypothetical example of a relevant risk issue worksheet is on the
following page, as Figure 1. As Figure 1 shows, the Risk Issue Worksheet
summarizes key information, including background, impact, and mitigation
aspects. Background information includes: (a) the risk facet that the
issue belongs to, (b) the risk number, (c) the name of identifier of the
issue, and (d) a brief description of the issue. Impact information
includes: (a) the probability level of the issue materializing, and (b)
the severity of the consequences if the issue materializes. Mitigation
information includes: (a) suggested mitigation strategy, and (b) timing
for the mitigation strategy. Impact information is provided for the risk
issue before and after expected implementation of mitigation strategy.

**Figure 1: Risk Issues Worksheet**

# Final Investment Decision Framework 

## Contracting Implications 

With the completion of the Initial Investment Decision, the IPT can
issue a Request for Offer (RFO) to obtain the market place's bids for
the selected alternative of the potential acquisition. The Initial APB
cannot be used as a basis of awarding a contract because the cost
estimate is still at a high aggregate level, lacking detailed specific
assumptions. An RFO, issued after the Initial Investment Analysis is
complete, may provide the marketplace's perspective regarding how
reasonable is the costing of the acquisition.

With the completion of the Final Investment Decision, the IPT obtains an
FAA commitment to make the funding level identified in the APB available
for contract award. The Final APB becomes the level of funding under
which the IPT can enter into a contract. Without a Final Investment
Decision, there is no FAA commitment to fund and a contract cannot be
let.

## Trackable Baselines and Congressional Sanctions

With the commitment to fund that comes with a Final Investment Decision,
the APB assumes a more important role --- that of a yardstick against
which the progress is measured. Baseline breaches of 10% and 50% must be
reported to Congress. Moreover, Congress requires the FAA to consider
termination of a program where the breach exceeds 10%, and to terminate
programs where the breach exceeds 50%.

To proactively ensure that the APB is as stable as possible, and the
breach levels are not exceeded, conservative risk-adjusted estimates are
developed for the baselines. The typical cost baseline developed has a
20% chance of being exceeded. The typical benefit baseline developed has
an 80% chance of being exceeded.

The conservative, risk-adjusted estimates are developed based on the
risk issues identified and appraised for cost, benefit, schedule, and
performance implications. The more thorough and solid the risk analysis
is, the more likely that the APB will accommodate any uncertainty that
develops. Accordingly, the more likely that the risk issues are
anticipated and mitigated, the more likely that the program will
succeed. The role of risk analysis for JRC 2(b) then, is to concentrate
the risk focus at a finer level of detail. This is to ensure that the
cost, schedule, benefit, and performance baselines are stable through
the anticipation and mitigation of risks.

# Risk Information Flow Between Initial and Final Investment Decisions 

The flow of risk information from the Initial Investment Analysis (2(a))
to Final Investment Analysis (2(b)) is best illustrated in the
following, Figure 2. It simply shows that risk information from the
Initial Investment Analysis is the basis of the risk assessment for the
Final Investment Analysis, along with key inputs from the IPT
Programmatic Risk Assessment to arrive at risk for four parameter
metrics and interdependency risks.

**Figure 2: Integrated Risk Assessment and Management Approach in
Investment Analysis**

![](media/image2.wmf){width="7.2659722222222225in"
height="5.965277777777778in"}

## Detailed Risks, Consequences and Mitigation from Initial Investment Decision

Out of the Initial Investment Decision (JRC 2(a)), one of the key
outputs is the risk issue database for the preferred alternative. An
example can be found in Figure 1. For each issue identified, there is
sufficient background information to understand the likelihood that the
issue may materialize, its potential impact, and opportunities for its
mitigation. At the time of the Initial Investment Decision, the
technical information available on the preferred alternative is often
sketchy so that the risk analysis is, necessarily, at a high level. The
risk assessment in the Initial Investment Analysis is summarized in a 3
by 3 matrix, such as illustrated in Table 1.

## Update of Risk Issues 

Following the Initial Investment Decision, the preferred alternative is
identified and the work for the Final Investment Decision begins. The
Final Investment Decision affords the IA team more time to examine the
preferred alternative. Correspondingly, it is during this time that the
risks are also revisited with respect to the preferred alternative.
Risks are examined more comprehensively and more in depth.

The risk team lead (RTL) for Investment Analysis is responsible for
revisiting the risk issues identified in the Initial Investment Analysis
to determine if the issues are still relevant, whether the issues have
been addressed, or whether other risk issues have come to light on the
basis of new information either from secondary sources, users,
implementers, the political process, etc. Ideally, the RTL will involve
other IA team members from the IPT, sponsors, and users to obtain as
objective information as possible regarding the relative importance of
the risk issues.

At this stage, issues that are no longer relevant are dropped from
further consideration, with appropriate documentation as to why they
have been dropped. In addition, new risk issues that are newly
identified now enter the pool for active consideration. For each new
issue identified by the RTL and team, a risk issue worksheet is
prepared.

With the update of the risk issues, and the continued advancement of the
investment analysis, there is an opportunity for the RTL and the risk
team to identify the candidate organizations which may need to be
responsible for undertaking mitigation actions. Assigning responsibility
may be a controversial endeavor, and consensus is essential though not
critical. The most prudent basis for assigning responsibility may be by
matching the type of risk that is to be faced with the functional
responsibility of the organization within FAA. In all cases, an
organization should agree to the responsibility and the IA team would
also need to endorse such assignment. Endorsement may be indicated by
simply listing the responsible organization along side the risk issues,
and schedule for mitigation action in the Risk Management Matrix, which
the AMS already requires in the APB.

## Funneling of All Risks to Baseline Parameters

Because of this step, the risk team now has an inventory of risk issues
and corresponding worksheet reflecting the most recently available
information. Each risk issue has been uniquely numbered.

During this step, the risk issues are placed in a baseline category that
they most influence and are assigned a risk level.

Per the AMS, there are four baseline categories: cost, benefit,
schedule, and performance. Each of the risk issues identified in the
last step should now be assigned to a baseline category. This assignment
could be done by the RTL or the risk team. In either case, the IA team's
cost, benefit, schedule, product team liaison lead, should affirm
placement. Should a risk issue span multiple categories, the issue would
be carried in each category.

Moreover, it is also important at this point to consider whether there
are any risks of interdependency with other acquisitions in the NAS
domain that a proposed acquisition may be in. For example, a
high-capacity surveillance tool may be proposed, but its implementation
might depend on the existence of an infrastructure to enable that tool.
The risk of successfully developing an infrastructure would impact the
successful development of the surveillance tool. Interdependency risks,
informed by examining the NAS Architecture development waterfall, would
also be translated to one of the four parameter baseline categories, if
possible.

## Assessment for Each Parameter

Each parameter lead of cost, benefit, schedule, and performance now
takes all of the risk issues placed in his/her respective category and,
with the help of subject matter experts, assesses the risk level by
likelihood of occurrence and by consequence of occurrence according to
the criteria presented for each parameter below.

[Likelihood of Occurrence Evaluation Criteria]{.underline}

The *likelihood of occurrence* is defined to be the perceived chance
that an identified risk will actually happen. Five levels of perceived
likelihood are proposed for use in risk assessment. These are identified
in the table below.

![](media/image3.wmf){width="5.259722222222222in"
height="1.8958333333333333in"}

[Consequence of Occurrence Evaluation Criteria]{.underline}

The *consequence of occurrence* is defined to be the perceived impact on
the overall project attribute if an identified risk were actually to
occur. Five levels of perceived consequence of occurrence are proposed
for use in risk assessment. Consequences are rated according to impact
on cost, schedule, benefits, and/or technical performance.

> ![](media/image4.wmf){width="7.152083333333334in" height="1.875in"}

For each risk category of cost, schedule, benefit, and performance, all
the risk issues are then plotted by their unique number on figure 3.

**Figure 3: Assessment of Risk Issues**

![](media/image5.wmf)

All of those issues that are plotted in the "medium" or "high" cells
have to be reported on. The cost, schedule, benefit, and performance
leads or experts work with the risk lead and the IPT liaison staff to
identify actions that would be necessary to reduce or mitigate the
risks.

The cost lead estimates the cost of the mitigation actions and includes
the costs in establishing the high-confidence range for the cost
baseline. The benefit lead, the schedule lead, and the performance lead
do the same for their part.

If some risks are beyond the control of FAA, then these risk issues may
simply have to be acknowledged and accepted, without mitigation.

## 

## 

## Integration of Human Factors, Information Security, and Safety Risk Issues

Risk issues related to key focus areas of concern, e.g., human factors,
security (physical and information), and safety management should be
worked in co-ordination with the offices responsible for planning in
these fields. The Human Factors Division (AAR-100) of FAA has the
expertise to develop human factors risk issues in more detail, and
re-assess their impact on acquisitions. Similarly, the NAS Architecture
Branch (ASD-110) has the expertise to assess operational safety risk
issues; the Security Systems Engineering Branch (ASD-130) has the
expertise to assess security issues. The RTL and a representative from
each of the offices should develop the details for each of the
identified risk issues and assess the level of risk (low, medium, and
high). The protocol to be followed in identifying new risk issues, and
assessing the impact of risk issues on the success of the acquisition
should give preference to the methodologies in place for each Office's
specialty area. As a default, the risk checklist used to identify risk
issues for the Initial Investment Analysis (see [Investment Analysis
Teams Alternatives Risk Assessment Guide) should be reviewed to ensure
that]{.underline} [human]{.underline} [factors,]{.underline}
[information]{.underline} [security, and]{.underline} [safety are
comprehensively reviewed for possible changes in status and
relevance.]{.underline}

When the risk assessment comes from the offices responsible for human
factors, security, or operational safety, the risk issues that have
medium or higher risk level should be identified, as should actions
which must be implemented to mitigate the risk issues. This information
is provided to the RTL and, in turn, the RTL provides the information to
the cost, benefit, schedule, and performance leads.

With the identified mitigation actions for human factors, security, and
operational safety risks, the cost lead estimates the cost of the
mitigation actions and includes the costs in establishing the
high-confidence range for the cost baseline. The benefit lead, the
schedule lead, and the performance lead do the same for their part of
the Investment Analysis.

# Calculation of Cost, Schedule, Benefit and Performance Risk

Figure 4, below, illustrates the general approach in calculation of
risk-adjusted numbers for the cost, schedule, benefit, and performance
baselines.

**Figure 4: Risk Adjustments**

![](media/image6.wmf){width="6.111805555555556in"
height="4.351388888888889in"}

Typically, initial calculations are done for each of the baselines on
the basis of a model which includes mathematical relationships between
variables.

This initial calculations result in a point estimate of the cost, or
benefit, or schedule, or performance.

In the course of the risk analysis, mitigation actions are identified or
some variable is expected to behave differently than expected. The
change may incrementally add or decrease the value of a variable in the
model. For example:

Cost The cost for using a new automation tool may have been identified
and considered in the overall cost of an acquisition. However, a human
factors risk may have been identified which concerns the need to train
controllers to handle the probable workload adjustment in jointly using
the technology (R-side and D-side controllers). The cost then becomes:

> Technical Training Cost + probable increment training cost for
> effective teaming.

Benefit The delay benefit of an automation tool may have been
calculated, using national values for growth of airport operations.
However, the local growth rate of operations at an airport are greater
than the national rate forecast by FAA. The benefit then becomes:

> Delay reduction based on national growth value + probable incremental
> delay reduction due to additional reported growth

Schedule The schedule of a project is calculated to be complete in 5
years, however consultation with software experts has surfaced a concern
that given the number of lines of code that are to be developed, a
critical software module will take an additional year to complete. The
schedule then becomes:

> Schedule as expected + probable additional software development time

Performance A modernization project is expected to integrate and process
xxx surveillance sources, yet because of unexpected development problems
only yy surveillance sources can be integrated, and this in turn, will
impact the amount of NAS coverage. Performance then becomes

> Coverage with xxx sensors - probable decrease coverage due to
> availability

These probable changes in variable values may be many and are all
incorporated in establishing the range of estimates around the most
likely or the point value of the initial calculations. Using @ Risk, or
Crystal Ball Software tools, the analyst can then create triangular
distributions around each independent variable and calculate the
dependent variable (overall cost, benefit, schedule or performance).

The tool calculates the dependent variable by randomly drawing a value
from the probability distribution of each of the independent variables.
This is done on the basis of a Monte Carlo simulation. The overall
result is itself a calculation with its own distribution. From this
forecast distribution, a value is chosen which corresponds to a
high-confidence estimate. The confidence value will vary by baseline
metric:

Cost Value -- The value that has an 80 percent certainty of being
underrun.

Benefit Value -- The value that has an 80 percent certainty of being
exceeded.

Schedule Value -- The value that has an 80 percent certainty of being
underrun.

Performance Value --The value that has an 80 percent certainty of being
exceeded.

# Process Integration

Risk assessment for the Final Investment Decision, JRC 2(b), is
necessarily an iterative process and reflects a dynamic interplay of IA
team members, IPT colleagues for any adjustments to delivery timing and
technical content, subject matter experts in specialty engineering to
understand the intricacies of the risks and possible mitigation efforts,
and cost, benefit, schedule, and performance analysts to conduct risk
and sensitivity analysis.

Other colleagues that will have to play a role, of course are users,
sponsors, and stakeholders. Who else might be able to say whether the
performance risk is too high or what might be reasonable performance
trade-offs.

In applying the risk assessment process for JRC 2(b), it is important to
point out that often Investment Analyses come in different levels of
complexity, and importance. Accordingly, it is important to think of the
risk assessment process as flexible. The goal is to always add value,
rather than to be a stumbling block or a hurdle. Larger, more complex
projects should certainly get a more detailed analysis than smaller
simpler projects.

# Basis of Risk Estimate

In conducting a risk assessment for the final investment decision, it
will be critically important to document the basis of estimate for the
risk analysis. A template for the basis of estimate in included in the
appendix of this document.

# Products

The APB normally includes risk-adjusted cost, schedule, benefit and
performance baselines, on a yearly basis, over the program's life cycle.

In addition to the risk-adjusted APB, another product which is important
to develop is a Risk Management Plan. The Risk Plan can be part of an
element of the Integrated Program Plan or it can be a stand-alone
document.

The Risk Plan would identify the steps that would have to be undertaken,
the timing of milestones, and the responsible agents, for all risks
meeting and exceeding the medium level. The goal is to demonstrate a
commitment to continuously mitigate risks, as identified in Investment
Analysis.

Correspondingly, the Risk Plan would also include an element to monitor
and track progress in mitigating risk. The key to risk mitigation status
is to show the risk resolution status in comparison with established
milestone dates. This step would be designed to communicate progress to
all stakeholders of the program.

Lastly, the Risk Plan would also identify a process for scanning the
program's environment and identifying emerging risks. This process would
be carried out in those phases of AMS following Investment Analysis.

The IPT member of the IA team is the ideal actor in developing the Risk
Plan, since the IPT is already responsible for development of the
Integrated Program Plan, and the IPT would be responsible for carrying
out the next phase of the program's life cycle, the Solution
Implementation phase.

The *FAA Programmatic Risk Management Course and Materials* provide
solid information on risk management and tracking.

# Planning for Life Cycle Risk Mitigation

The phases of AMS following the Investment Analysis phase include
Solution Implementation, In-Service Management, and Service Life
Extension. The life cycle of a program will vary, but may be as long as
20 years. A program will likely always have risks, although the
comprehensiveness of the risks and the focus of the risks will vary.
Early in the life cycle, we can expect the risks to cover a broad range
of issues, the issues to be treated at the macro level, and the level of
uncertainty to be high. As a program advances through its life cycle, we
would expect the range of risk issues to narrow, perhaps focusing on AMS
phase-specific issues. Also, the treatment of those issues would be more
focused at a more micro level.

The cost to address program problems increases as the program advances
in the life cycle \-- a design error is less costly to fix in the design
step of a program, rather than during implementation. Anticipating,
mitigating and tracking risk issues gives programs managers an
opportunity to reduce unnecessary expenses, and have more control in the
success of their programs. Accordingly, planning and carrying out a risk
management plan should be a life cycle long enterprise, because it
economically addresses potential problems and keeps a program on the
path to success.

# Closure

This report provides guidelines for addressing risk for a program
undergoing a Final Investment Decision or JRC 2(b). The intent of the
report has been to advance the treatment of risk with respect to the
development of a preferred alternative, and to integrate treatment of
risk with the subsequent phases of the AMS following Investment
Analysis.

Risk is necessarily a subjective exercise because it deals with
uncertainty. The degree of uncertainty regarding an issue, or the
specifics of necessary steps for its mitigation will vary from one
colleague to another. The best way to accommodate the subjective nature
of the exercise is to seek a consensus within a broad range of
Investment Analysis colleagues, including sponsors, users, implementers,
facet area experts, and other stakeholders. A broad risk analysis team
can be expected to self-manage around any tendency toward bias by some
individuals.

### References {#references .unnumbered}

1.  Politano, Arturo, [Investment Analysis Teams Alternatives Risk
    Assessment]{.underline} [Guide]{.underline}, Federal Aviation
    Administration, Operations Research and Analysis Branch, August
    2002.

2.  ICIP Course Notes: FAA Programmatic Risk Management, March 2001

#### Appendix: Standard Basis of Estimate for Risk Analyses {#appendix-standard-basis-of-estimate-for-risk-analyses .unnumbered}

Investment Analysis Standardization Working Group

**Usage**

In order to maintain a high level of quality for risk analysis conducted
in the Initial (JRC 2(a)) or Final (JRC 2(b)), a number of principles
must be adhered to in conducting a risk analysis. These principles are
listed below, by Investment Decision. The risk analyst is encouraged to
review these principles before the conduct of the risk analysis, and
affirm adherence to them after the completion of the analysis.

General Principles for both Initial (JRC 2 (a)) and Final (JRC 2 (b))
Investment Analyses:

1.  **Risk assessment is done by a risk team whose members are objective
    and can represent multiple viewpoints and stakeholders.**

2.  **Risk mitigation actions are integrated with appropriate cost,
    benefit, schedule, and other relevant Work Breakdown Structures to
    facilitate their adoption in the Solution Implementation, and
    In-Service Management Phases of their life cycle.**

**Principles for Initial Investment Analysis (JRC 2(a):)**

1.  Findings of the Portfolio Risk Assessment in the Mission Need
    Analysis phase of AMS are used to inform the risk analysis for the
    Initial Investment Analysis.

2.  A common agreed to description of viable alternatives exists.

3.  The description relates to the given requirements or mission need
    statement.

4.  The Risk Assessment team consists of individuals that are very
    knowledgeable with each risk facet and all alternatives.

5.  Facet weights are determined prior to evaluation of any alternative
    to avoid structural bias. Weights for the reference case may be
    different than weights for all alternatives. The rationale for
    relative weights selected is explained.

6.  Issues and attendant risks have been identified and commonly
    acknowledged for each facet.

7.  The risk score for each investment alternative has been developed
    independently of other investment alternatives.

8.  Each team member determines the risk score for each alternative
    independently of other team members (blind scoring) and writes an
    explanation for the qualitative or quantitative basis for such
    scoring.

9.  The risk team is brought together as a group to discuss and
    reconcile the team members' scoring for each alternative.

10. Each team member is given an opportunity to consider input from the
    team and independently rescore risk for each alternative.

11. The risk team lead documents the strategy taken to synthesize the
    scoring into a final rating for each alternative.

12. The risk team lead writes a technical memorandum or report which
    documents the reasons for the risk scores for the most viable
    candidate, and the actions and strategies which must be employed
    (including implementation schedule, cost, and performance impacts)
    to mitigate all medium and high risks.

13. The risk assessment results are incorporated in the APB and the
    Investment Analysis Report consistent with AMS guidelines.

14. The risk team lead coordinates with the selected IPT after JRC 2 (a)
    to facilitate transition of risk issues and information for a more
    detailed assessment as part of the Final Investment Analysis
    Decision (JRC 2(b)).

15. The results of the risk analysis are made usable in maintaining and
    updating a NAS-wide portfolio management framework of NAS
    investments.

**Principles for Final Investment Analysis (JRC 2(b)):**

1.  The risk team considers the risk issues identified in JRC 2 (a) as a
    starting point for the JRC 2 (b) decision and complements this with
    other available information from the Integrated Product Team.

2.  The risk team ensures an independent validation of how the risk
    issue will affect the program's baseline elements: cost, schedule,
    benefit, technical performance, and interdependency (with other NAS
    elements) risks.

3.  The risk team checks estimates against historical data, such as
    trade studies, whenever possible.

4.  Detailed analysis of risk has the following information:

-   Technical description

-   Related baseline element changes

-   Detailed rationale element changes

5.  Detailed analysis is based on:

-   Past performance (actual baseline element changes, e.g.,
    cost/schedule/benefit/performance history) when available

-   Discrete estimate (Simulated, scenario analysis, or other
    approach)

6.  The risk team determines the tasks on the work breakdown structure
    (WBS) that could be affected by the risk, and whether the
    consequences of the risk would require adding tasks that are not on
    the WBS.

7.  The Basis of Estimate for Risk Analysis is reviewed for incomplete
    and or bias implementation.

8.  The model for quantifying risk is reviewed by subject matter experts
    (SMEs) to affirm that it is based on the appropriate quantitative
    analysis techniques based on available data.

9.  The risk analysis is a probabilistic analysis (whether quantitative
    or scenario analysis) for the four key APB metrics of cost,
    schedules, benefits, and performance estimates.

10. Correlation between sub components of an APB metric is noted,
    isolated, and accounted for, by coordination with Cost and Benefit
    Analysis Teams.

11. A risk management plan is prepared to guide risk mitigation and risk
    monitoring in the Solution Implementation and In-Service Management
    phases of the AMS.

12. The results of the risk analysis are made usable in maintaining and
    updating a NAS-wide portfolio management framework of NAS
    investments.