**Investment Analysis Standards and Guidelines**

**Title:** Risk Metrics for Initial Investment Analysis (JRC 2(a))

**Type (Standard or Guideline):** Standard

**Approval Date:** Approval July 2002

**Purpose:** The Standard Risk Metrics form the investment risk
standards. They should be used to ensure quality and uniformity of
practice, unless there is a compelling documented case to do otherwise.

**Applicability:** The Standard Risk Metrics are required to be used by
all FAA entities and organizations pursuing an Initial Investment
Decision (JRC 2(a)) for any system acquisition. For a facility
acquisition, these standard metrics would need to be tailored and
modified prior to beginning the risk analysis. Work continues on
modifying this standard to address facility, security, and environmental
issues. Accordingly, this standard should be viewed as an evolving one,
and constantly updated to remain more relevant to the context and nature
of all IAs. For the Initial Investment Decision, the Joint Resource
Council considers the merits of all investment alternatives which meet
the requirements and down-selects to the best candidate solution.

**Description:** The standard metrics are the risk facets of technical,
operability, producibility, supportability, cost estimating, benefit
estimating, schedule, management, funding, stakeholder, information
security, human factors, and safety. Within each facet is a checklist of
potential issues to be considered for each alternative. The number of
issues identified as affecting the alternative under consideration is an
indicator of magnitude of risk. This measure of risk forms the basis for
the application of a methodology to assess the level, consequence, and
mitigation of risk.

**Content:** The Standard Risk Facets, and Risk Checklist are defined
and presented below.

**Configuration Management:** This Standard will be maintained by the
Investment Analysis and Operations Research Directorate (ASD-400) in the
FAA Acquisition System Toolset (FAST). It will be reviewed annually by
ASD-400 for currency with acquisition environment. If any changes are
necessary, they will be proposed by

ASD-400 through the normal ASAG/ FAST change process.

## Risk Metrics for Initial Investment Analysis (JRC 2a)

## Terminology: Risk

**Risk** is defined as an event whose outcome is unknown in advance. In
this context, risk implies some probability that an acquisition
alternative will fail to deliver the benefits projected, either in whole
or in part, and that failure may have consequences to its success. The
risk can derive from uncertainties in an alternative\'s concept or
problems encountered during design, development, implementation, or
operation.

## Risk Consideration During Initial Investment Analysis 

For the Initial Investment Analysis Decision (JRC-2a), the critical
issue to be answered is which of the viable acquisition alternatives is
worth examining in more detail for implementation. Consequently, the
risk assessment for the Initial Investment Decision is necessarily done
at the macroscopic, and comprehensive levels.

The process includes

-   Identification of risk issues,

-   Development of potential mitigation strategies,

-   Assessment of investment alternatives, and

-   Affirmation of mitigation strategies to be examined for cost or
    benefit estimation impact.

### The Investment Analysis Team (IAT) Alternatives Risk Assessment

In general, within the IAT Alternatives Risk Assessment there are
thirteen facets that constitute an exhaustive range of potential risk
areas, including: technical, operability, producibility, supportability,
benefit estimate, cost estimate, schedule, management, funding,
stakeholder, information security, human factors, and safety. Risk
issues may be identified by discussion with program staff, users,
stakeholders, or derived from secondary written sources. Usually, in the
course of identifying risk issues, mitigation actions come to mind and
are written down as preliminary mitigating actions. Also at this point,
the discussions or secondary information suggest the likelihood of the
issue surfacing and the potential severity of impact of the issue, if
unmitigated.

The next step is to convene a broad team of stakeholders, programmatic,
investment analysis, union representatives, and other interested
colleagues. A broad team is important because information is needed from
multiple perspectives. Only in so doing is it possible to get an
objective assessment. Together the team reviews the material and by
double blind voting and justifying discussions arrives at consensus of
each facet's rating for each alternative, and affirms or complements
issue mitigation strategies or actions. In a double blind voting
procedure, each group member votes before and after group discussions
without knowing how other group members are voting. At the end of these
discussions, the relative risk levels of alternative s emerge.

This information is provided to the IAT. In turn, the IAT uses this
information and other inputs from cost, benefit, and other analyses, to
put together its recommendation for down-selection.

The risk analyst then identifies whether the mitigation strategies are
likely to impact the cost estimation or the benefit estimation of the
down-selected alternative, and works with appropriate staff to estimate
the impact within the appropriate work breakdown structure categories.

The product of this effort may be a technical memorandum, but most often
is:

A stand-alone summary section in the Investment Analysis Report;

Solid input for creating cost and benefit high confidence ranges around
the most likely numbers, and

Direct input into the Risk Management Matrix in the Initial Acquisition
Program Baseline. The Risk Management Matrix may be found in the FAA
Acquisition System Toolset ([www.fast.faa.gov](http://www.fast.faa.gov/)
, APB Template, Section 6, Table 6.1).

Lastly, the issue ratings are summarized into a matrix for the JRC, and
for continued analysis in the Solution Implementation Phase.

## Risk Facets

The life cycle risks are broken down into thirteen components, or
facets, of risk, which are used to assess the overall risk. These risk
facets have been selected to reflect the comprehensive risks associated
with an acquisition alternative's development, implementation,
operation, in such a way as to achieve the projected benefits. The
facets are intended to facilitate the identification of risk and its
quantification. The thirteen risk facets are defined as follows:

-   **RiskTechnical** is the risk associated with (1) developing a new
    or extending an existing technology to provide greater performance
    than previously demonstrated, or (2) achieving a level of
    performance. It also refers to how well the system operates to
    design or safety specifications.

-   **RiskOperability** is the risk associated with how well the system
    to be produced will operate within the NAS and interact with other
    systems. It addresses NAS or other system interfaces, the degree to
    which they are known and complete, and the degree to which the
    operational concept has been demonstrated and evolved to the point
    of a design baseline.

-   **RiskProducibility** is the risk associated with the capabilities
    to manufacture and produce the desired system.

-   **RiskSupportability** is the risk associated with fielding and
    maintaining the resulting systems.

-   **RiskBenefit Estimate** considers the difficulty in estimating the
    benefits and in realizing benefits. This risk facet addresses the
    accuracy and uncertainty of the benefit estimate, including such
    issues as inadequate methods to estimate the benefits, lack of data
    to estimate the benefits, uncertainty of assumptions, and whether
    the alternative is defined enough to estimate the benefits.

-   **RiskCost Estimate** considers the difficulty in estimating the
    cost and in adhering to the cost. This risk facet addresses the
    accuracy of the cost estimate, including such issues as inadequate
    methods to estimate the cost, lack of data to estimate the cost,
    uncertainty of assumptions, and whether the alternative is defined
    enough to estimate the cost.

-   **RiskSchedule** considers the likelihood that the alternative will
    be completed within the specified schedule.

-   **RiskManagement** refers to complexity of the alternative to manage
    (e.g., number of sub-tasks and/or number of performing
    organizations) and considers the risks of obtaining and using
    applicable resources and activities that may be outside of the
    alternative\'s control but can affect the alternative\'s outcome.

-   **RiskFunding** addresses the availability of funds when they are
    needed and a confidence in management and Congress that those funds
    will continue to be provided.

-   **RiskStakeholder** is the risk associated with various stakeholders
    supporting the development and operation of the alternative, such as
    internal FAA organizational users, Congress, airline and general
    Aviation users, and potential equipment and aircraft manufacturers.

-   **RiskSecurity** addresses a system\'s vulnerability to external
    threats and the risks likely to occur in employing countermeasures.
    This pertains to both information and physical issues.

-   **RiskHuman Factors** focuses on the effectiveness and suitability
    of the human-in-the-loop aspects of the system with respect to both
    operations and maintenance.

-   **RiskSafety** considers the risk associated with the performance
    (or lack thereof) of appropriate safety risk management activities
    and in implementing the identified safety requirements. This risk
    facet shall not be confused with the operational risk of system
    hazards. The operational risk of system hazards are documented in
    other analyses.

## Risks Metrics

For the Initial Investment Analysis, measuring risk begins with the
identification of the number of specific risk issues for each risk area.
The identification of risks is conducted in an organized and
comprehensive manner.

Risk identification involves enumerating and describing the risks
pertaining to each of risk facet for successfully carrying out each of
the alternatives being considered. The number of risks identified
constitutes the measure, or quantification of risk. The measure of risk
is not to be confused with the assessment of risk. The assessment of
risk for each alternative follows a set methodology which considers the
probability of occurrence of risk and the severity of impact should that
risk materialize. Assessment of risk will be covered in a subsequent
product of the risk standardization effort, called Risk Methodology.

For purpose of risk enumeration, Table 1: Facet Risk Checklist, is
included in the following several pages. For each of the thirteen (13)
risk facets already identified as being relevant to the Initial
Investment Analysis Decision, a comprehensive set of risk issues are
listed. The analyst should examine each alternative to determine whether
each of the bulleted issues is present. The number of risk issues
present and their detailed description determine the metric for risk.
The risk issues that are found to be present are then the basis of
further research. Further research would include a risk assessment to
determine the level of risk, usually by understanding the probability of
the occurrence of risk, and the severity of their impact.

Table 1: Facet Risk Checklist

+------------------------+---------------------+-----------------------+
| **Technical Risks**    | **Operability       | **Producibility       |
|                        | Risks**             | Risks**               |
+========================+=====================+=======================+
| **Technology**         | **System            | **Design Production** |
|                        | Operation**         |                       |
| -   Undue reliance on  |                     | -   Highly complex    |
|     currently          | -   Undefined       |     design            |
|     unavailable or     |     external        |                       |
|     unproved           |     interfaces      | -   Undeveloped       |
|     technology         |                     |     production        |
|                        | -   Marginal        |     requirements      |
| -   Possible better    |     availability    |                       |
|     new technology may |                     | -   Inadequate        |
|     be available by    | -   Insufficient    |     built-in test     |
|     time alternative   |     reliability     |     equipment         |
|     is implemented     |                     |                       |
|                        | -   Inadequate      | -   Non-standard      |
| **System Engineering** |     performance     |     remote            |
|                        |                     |     maintenance       |
| -   Technically        | -   Unsatisfactory  |     monitoring        |
|     incompatible with  |     OT&E results    |                       |
|     NAS Architecture   |                     | -   Novel/unproved    |
|                        | **Systems           |     technologies      |
| -   Inadequate         | Inter-operability** |                       |
|     functional         |                     | **Manufacturing**     |
|     analysis           | -   Operationally   |                       |
|                        |     incompatible    | -   Deficient         |
| -   Deficient          |     with NAS        |     manufacturing     |
|     functional         |     Architecture    |     plan              |
|     allocation         |                     |                       |
|                        | -                   | -   Novel/unproved    |
| -   Incomplete         |   Incompatibilities |     manufacturing     |
|     integration        |     with Concept of |     technologies      |
|                        |     Operations      |                       |
| -   Undefined internal |                     | -   Speculative       |
|     interfaces         | -                   |     manufacturing     |
|                        |   Incompatibilities |     strategy          |
| -   Vague operational  |     with future NAS |                       |
|     environment        |     systems         | -   Custom design and |
|                        |                     |     manufacture       |
| -   Insufficient       | -   Places undue    |     required          |
|     requirements       |     loads on other  |                       |
|     analysis           |     systems         | -   Significant       |
|                        |                     |     special tooling   |
| -   Unstable           | -   Incompatible or |                       |
|     requirements       |     inconsistent    | -   Undefined tooling |
|                        |     operation with  |     requirements      |
| -   Non-compliant or   |     existing        |                       |
|     unvalidated        |     systems or      | -   Unclear           |
|     requirements       |     regulations     |     production        |
|                        |                     |     requirements      |
| -   Weak or            | -   Unspecified     |                       |
|     non-existent       |     operational     | -   Premature         |
|     failure modes      |     interfaces      |     initiation of     |
|     analysis           |                     |     manufacturing     |
|                        | -   Marginal        |                       |
| -   Requirements       |                     | -   Unavailable or    |
|     difficult to trace |   inter-operability |     limited           |
|                        |                     |     manufacturing     |
| -   Unidentified       |                     |     facilities        |
|     safety/security    |                     |                       |
|     considerations     |                     | -   Inadequate        |
|                        |                     |     quality assurance |
| **System Design**      |                     |     program           |
|                        |                     |                       |
| -   Inadequate         |                     | -   Excessive         |
|     capacity           |                     |     standards         |
|                        |                     |                       |
| -   Highly complex     |                     | -   Unavailable       |
|                        |                     |     equipment         |
| -   Lack of design     |                     |                       |
|     details            |                     | -   Inexperienced     |
|                        |                     |     contractor        |
| -   Insufficient       |                     |                       |
|     design margins     |                     | -   Inadequate        |
|                        |                     |     configuration     |
| -   Immature design    |                     |     management        |
|                        |                     |     process           |
| -   Unsatisfactory     |                     |                       |
|     growth potential   |                     | -   Insufficient      |
|                        |                     |     skilled labor     |
| -   Undefined physical |                     |                       |
|     properties         |                     | -   Shallow           |
|                        |                     |     industrial base   |
| -   Incomplete         |                     |                       |
|     hardware design    |                     | **Parts and           |
|                        |                     | Materials**           |
| -   Incomplete         |                     |                       |
|     software design    |                     | -   Undefined long    |
|                        |                     |     lead items        |
| -   Inadequate         |                     |                       |
|     software tools     |                     | -   Unavailable       |
|                        |                     |     government        |
| -   Difficulty of      |                     |     furnished         |
|     developing         |                     |     equipment         |
|     real-time, safety  |                     |                       |
|     critical software  |                     | -   Ineffective       |
|                        |                     |     incoming          |
| -   Immature software  |                     |     materials         |
|     language           |                     |     handling          |
|                        |                     |                       |
| -   Ineffective fault  |                     | -   Unidentified      |
|     detection          |                     |     hazardous         |
|                        |                     |     materials         |
| -   Inordinate use of  |                     |                       |
|     unique resources   |                     | -   Unavailable parts |
|                        |                     |                       |
| -   Complex/incomplete |                     | > **Testing and       |
|     man/machine design |                     | > Documentation**     |
|                        |                     |                       |
| -   Undefined          |                     | -   Inadequate        |
|     technical approach |                     |     consideration of  |
|                        |                     |     special test      |
| **System Test**        |                     |     equipment         |
|                        |                     |                       |
| -                      |                     | -   Insufficient      |
|  Inaccurate/simplistic |                     |     qualification     |
|     modeling           |                     |     testing           |
|                        |                     |                       |
| -   Insufficient       |                     | -   Deficient         |
|     simulation         |                     |     technical data    |
|                        |                     |     package           |
| -   No or minimal      |                     |                       |
|     prototype testing  |                     | -   Ineffective       |
|                        |                     |     factory           |
| -                      |                     |     acceptance test   |
|  Incomplete/inadequate |                     |     program           |
|     test planning      |                     |                       |
|                        |                     | -   Untested design   |
| -   Unsatisfactory     |                     |     changes           |
|     OT&E results       |                     |                       |
|                        |                     |                       |
| **Technical            |                     |                       |
| Documentation**        |                     |                       |
|                        |                     |                       |
| -   Inadequate design  |                     |                       |
|     documentation      |                     |                       |
|                        |                     |                       |
| -   Insufficient test  |                     |                       |
|     documentation      |                     |                       |
|                        |                     |                       |
| -                      |                     |                       |
|   Ambiguous/incomplete |                     |                       |
|     requirements       |                     |                       |
|     documentation      |                     |                       |
|                        |                     |                       |
| -   Undocumented       |                     |                       |
|     technical details  |                     |                       |
+------------------------+---------------------+-----------------------+

Table 1: Fecet Risk Checklist, Cont'd

+-------------------+---------------+--------------+------------------+
| **Supportability  | **Cost        | **Benefit    | **Schedule       |
| Risks**           | Estimate      | Estimate     | Risks**          |
|                   | Risks**       | Risks**      |                  |
+===================+===============+==============+==================+
| **O&M**           | **Cost        | **Benefit    | **Schedule       |
|                   | Estimation**  | Iden         | Estimation**     |
| -   Inadequate    |               | tification** |                  |
|     O&M concept   | -             |              | -   Inadequate   |
|                   |    Inadequate | -   Same     |     schedule     |
| -   Undeveloped   |     cost      |     benefits |     estimating   |
|     O&M strategy  |               |     claimed  |     tools        |
|                   |    estimating |     by other |                  |
| -   Specialized   |     tools     |     programs | -   Erroneous    |
|     O&M equipment |               |              |     estimations  |
|                   | -             | -            |                  |
| -   Insufficient  |    Estimation | Unidentified | -   Faulty BOEs  |
|                   |     errors    |     major    |                  |
|   maintainability |               |     benefits | -   Insufficient |
|                   | -             |              |     schedule     |
| -                 |    Inaccurate | -            |     margin       |
|    Unsatisfactory |     discount  |  Unrealistic |                  |
|     maintenance   |     rate      |              | -   Optimistic   |
|     interfaces    |               |   identified |     schedule     |
|                   | -   Faulty    |     benefits |     duration     |
| -   Inadequate    |     BOEs\*\*  |              |                  |
|     maintenance   |               | -            | -                |
|     procedures    | -             |    Difficult |    Inappropriate |
|                   |  Insufficient |     to       |     program      |
| -   Undeveloped   |     cost      |     identify |     schedule     |
|     maintenance   |     margin    |     benefits |                  |
|     plan          |               |              | **Schedule       |
|                   | -             | **Benefit    | Dependency**     |
| -   Configuration |   Unrealistic | Estimation** |                  |
|     management    |     overhead  |              | -                |
|     not enforced  |     and G&A   | -   Benefits |    Unpredictable |
|                   |     rates     |     not      |     labor        |
| -   Deficient     |               |              |     strikes      |
|     change        | -   Relies on | quantifiable |                  |
|     process       |     scarce    |              | -   Improper     |
|                   |     resources | -            |     test         |
| -   Logistics     |               |    Difficult |     scheduling   |
|                   | -             |     to       |                  |
| -   Insufficient  |   Speculative |     estimate | -   Excessive    |
|     spares        |     life      |     benefits |     task         |
|     planning      |     cycle     |              |     concurrency  |
|                   |     costs     | -   Tenuous  |                  |
| -   Spares        |               |              | -   Unidentified |
|                   | -             | relationship |     need for     |
|    unavailability |   Sensitivity |     to       |     procedures   |
|                   |     analysis  |              |     development  |
| -   Inaccessible  |     to cost   |    projected |                  |
|     site location |     drivers   |     benefits | -   Unidentified |
|                   |     not       |              |     need for     |
| -   Inadequate    |               | -   External |     regulations  |
|     training      |    undertaken |     forces   |     development  |
|                   |               |     may      |                  |
| -   Unclear       | **Cost        |     affect   | -   Inordinate   |
|     Logistics     | Management**  |              |     number of    |
|     Center        |               |    achieving |     critical     |
|                   | -   U         |     benefits |     path items   |
|  responsibilities | nsatisfactory |              |                  |
|                   |     cost      | -            | -   Unidentified |
| **Testing and     |     controls  |    Erroneous |     need for     |
| Support**         |               |     benefits |     standards    |
|                   | -             |              |     development  |
| -   Insufficient  |  Insufficient |  estimations |                  |
|     support       |     cost      |              | -                |
|     equipment     |               | -            |    Uncertainties |
|                   |    monitoring |   Inaccurate |     in           |
| -   COTS/NDI -    |               |     inflat   |     contractor   |
|     Industry      | **Product     | ion/discount |     process      |
|     refresh rate  | Cost**        |     rates    |                  |
|     not likely to |               |              | -                |
|     be consistent | -   Undefined | -            |    Uncertainties |
|     with FAA      |               |  Speculative |     in           |
|     needs         |    government |     cost     |     contractor   |
|                   |     furnished |              |     stability    |
| -   Undeveloped   |     equipment |    avoidance |                  |
|     support       |               |              | -   Schedule too |
|     requirements  | -   Reliance  | -   Faulty   |     ambitious    |
|                   |     on        |     BOEs.    |     for degree   |
| -   Inadequate    |               |              |     of technical |
|     automated     |   unavailable |   Inadequate |     complexity   |
|     test          |     NDI/COTS  |              |                  |
|     equipment     |               |   estimating | -   Unavailable  |
|     (ATE)         | -             |     tools    |     materials    |
|                   |   Unavailable |              |                  |
| -   Unidentified  |               |              | -   Unavailable  |
|     field support |    government |              |     parts        |
|     requirements  |               |              |                  |
|                   |    facilities |              | -   Unavailable  |
| -   Poor          |               |              |     government   |
|     diagnostics   | -             |              |     furnished    |
|                   |   Unavailable |              |     information  |
| -   Insufficient  |               |              |                  |
|     testing and   |    contractor |              | -   Unavailable  |
|     support       |               |              |     facilities   |
|     facilities    |    facilities |              |                  |
|                   |               |              | -   Unavailable  |
| -   Unski         | -             |              |     personnel    |
| lled/insufficient |    Inadequate |              |                  |
|     manpower      |     budget    |              | -   Unavailable  |
|                   |     for tests |              |     tools        |
| **Support         |               |              |                  |
| Documentation**   | -   Undefined |              | -   Unavailable  |
|                   |     hardware  |              |     contractor   |
| -   Deficient     |     costs     |              |                  |
|     technical     |               |              | **Schedule       |
|     data          | -   Hidden    |              | Management**     |
|                   |     software  |              |                  |
| -   Faulty        |     costs     |              | -                |
|     maintenance   |               |              |   Unsatisfactory |
|     plan          | -             |              |     schedule     |
|                   |  Unidentified |              |     controls     |
| -   Undefined     |     parts and |              |                  |
|     data rights   |     materials |              | -   Insufficient |
|                   |               |              |     program      |
| -   Inappropriate |               |              |     schedule     |
|     release cycle |               |              |     monitoring   |
|                   |               |              |                  |
| **System          |               |              | -   Improper     |
| Implementation**  |               |              |     contract     |
|                   |               |              | or/subcontractor |
| -   Deficient     |               |              |     schedule     |
|                   |               |              |     monitoring   |
|    implementation |               |              |                  |
|     approach      |               |              |                  |
|                   |               |              |                  |
| -   Uncertain     |               |              |                  |
|     transition    |               |              |                  |
|     strategy      |               |              |                  |
|                   |               |              |                  |
| -   Unclear rules |               |              |                  |
|     and           |               |              |                  |
|     procedures    |               |              |                  |
|                   |               |              |                  |
| -   Insufficient  |               |              |                  |
|     p             |               |              |                  |
| ersonnel/staffing |               |              |                  |
|                   |               |              |                  |
| -   Unspecif      |               |              |                  |
| ied/inappropriate |               |              |                  |
|     standards     |               |              |                  |
+-------------------+---------------+--------------+------------------+

Table 1: Facet Risk Checklist, Cont'd

+-------------------------+------------------+------------------------+
| **Management Risks**    | **Funding        | **Stakeholder Risks**  |
|                         | Risks**          |                        |
+=========================+==================+========================+
| **Planning**            | **Funding        | **Congressional        |
|                         | Constraint**     | Based**                |
| -   Inadequate program  |                  |                        |
|     plans               | -   Unfavorable  | -   Impact of          |
|                         |     agency       |     congressional      |
| -   Incomplete          |     priorities   |     mandates           |
|     contingency plans   |                  |                        |
|                         | -   Inadequate   | -   Unfavorable        |
| -   Deficient risk      |     funding      |     congressional      |
|     management plans    |                  |     hearings on        |
|                         | -   Unavailable  |     program            |
| -   Inadequate          |     funding      |                        |
|     management approach |                  | -   Critical GAO       |
|                         | -   Lengthy      |     report             |
| -   Unplanned slips in  |     budget cycle |                        |
|     other programs      |                  | **Administration       |
|                         | -   Inadequate   | Based**                |
| -   Adverse             |     OMB marks    |                        |
|     environmental       |                  | -   Conflicting FAA    |
|     impacts             | -   Constraining |     priorities         |
|                         |     unique       |                        |
| -   Unsubstantiated     |     budget       | -   Conflicting DOT    |
|     funding profile     |     scoring      |     priorities         |
|                         |     rules for    |                        |
| -   Unsubstantiated     |                  | **Aviation Community** |
|     manpower            |  lease-purchases |                        |
|     requirements        |     and leases   | -   Many different     |
|                         |     per OMB A-11 |     stakeholders       |
| -   Unidentified        |                  |                        |
|     personnel skills    | **Funding        | -   Diverse user       |
|                         | Support**        |     community          |
| -   Minimal resource    |                  |                        |
|     alternatives        | -   Inadequate   | -   Conflicting user   |
|                         |     user support |     demands            |
| -   Excessive           |                  |                        |
|     dependencies on     | -   Ambiguous    | -   Conflicting user   |
|     other system        |     operator     |     opinions           |
|                         |     support      |                        |
| -   Unexpected          |                  | -   Conflicting user   |
|     acquisition         | -   Unclear      |     priorities         |
|     regulation changes  |     political    |                        |
|                         |     support      | -   Inordinate         |
| **Organizing**          |                  |     pressure from user |
|                         | -   Marginal     |     groups             |
| -   Excessive span of   |                  |                        |
|     control             |    cost/benefits | -   Marginal user      |
|                         |                  |     support            |
| -   Inadequate          | -   Inconsistent |                        |
|     authority           |     FAA plans    | -   Strained           |
|                         |                  |     relationships with |
| -   Undefined           | -   Lack of      |     users              |
|     responsibilities    |     alignment of |                        |
|                         |     necessary    | -   Resistance to      |
| -   Unclear             |     funding      |     avionics equipage  |
|     communications      |     profile with |     requirements       |
|                         |     agency       |                        |
| -   Undefined           |                  | -   Inordinate media   |
|     integration         |    affordability |     attention          |
|     responsibilities    |     profile      |                        |
|                         |                  |                        |
| -   Ambiguous           | **Fiscal         |                        |
|     organizational      | Management**     |                        |
|     interfaces          |                  |                        |
|                         | -   Insufficient |                        |
| -   Inadequate          |     funding      |                        |
|     contractor          |     requirements |                        |
|     organization        |                  |                        |
|                         | -   Insufficient |                        |
| **Implementing**        |     fiscal       |                        |
|                         |     controls     |                        |
| -   Insufficient        |                  |                        |
|     management tools    | -   Insufficient |                        |
|                         |     fiscal tools |                        |
| -   Inadequate program  |                  |                        |
|     office staffing     | -   Insufficient |                        |
|                         |     funding      |                        |
| -   Inadequate resource |     plans        |                        |
|     allocation          |                  |                        |
|                         |                  |                        |
| -   Deficient personnel |                  |                        |
|     management          |                  |                        |
|                         |                  |                        |
| -   Lack of             |                  |                        |
|     coordination        |                  |                        |
|                         |                  |                        |
| -   Tenuous top         |                  |                        |
|     management support  |                  |                        |
|                         |                  |                        |
| -   Cumbersome FAA      |                  |                        |
|     contracting process |                  |                        |
|                         |                  |                        |
| -   Instability of      |                  |                        |
|     contractor          |                  |                        |
|                         |                  |                        |
| -   Uncertainties in    |                  |                        |
|     procurement         |                  |                        |
|                         |                  |                        |
| -   Unavailable         |                  |                        |
|     personnel           |                  |                        |
|                         |                  |                        |
| -   Deficient change    |                  |                        |
|     implementation      |                  |                        |
|                         |                  |                        |
| **Control**             |                  |                        |
|                         |                  |                        |
| -   Undefined or        |                  |                        |
|     ineffective change  |                  |                        |
|     management          |                  |                        |
|                         |                  |                        |
| -   Unsatisfactory      |                  |                        |
|     configuration       |                  |                        |
|     management          |                  |                        |
|                         |                  |                        |
| -   Insufficient        |                  |                        |
|     contract evaluation |                  |                        |
|                         |                  |                        |
| -   Inadequate planning |                  |                        |
|     for contractor      |                  |                        |
|     monitoring          |                  |                        |
|                         |                  |                        |
| -   Insufficient        |                  |                        |
|     financial           |                  |                        |
|     management          |                  |                        |
|                         |                  |                        |
| -                       |                  |                        |
|   Irregular/unscheduled |                  |                        |
|     program reviews     |                  |                        |
|                         |                  |                        |
| -   Insufficient        |                  |                        |
|     history/records     |                  |                        |
|                         |                  |                        |
| -   Undefined key       |                  |                        |
|     metrics             |                  |                        |
|                         |                  |                        |
| -   Uncontrolled        |                  |                        |
|     requirements        |                  |                        |
|     changes             |                  |                        |
|                         |                  |                        |
| -   Requirements freeze |                  |                        |
|     not enforced        |                  |                        |
|                         |                  |                        |
| -   Inadequate tracking |                  |                        |
|     systems             |                  |                        |
+-------------------------+------------------+------------------------+

**Table 1: Facet Risk Checklist, Cont'd**

+---------------------+------------------------+-----------------------+
| **Security Risks**  | **Human Factors        | **Safety Risks**      |
|                     | Risks**                |                       |
+---------------------+------------------------+-----------------------+
| **Vulnerability**   | **Human-in-the-loop    | **Hazards**           |
|                     | Effectiveness**        |                       |
| -   Incomplete      |                        | -   Hazards and       |
|     vulnerability   | -   Inadequate         |     service-level     |
|     assessment      |     definition of      |     effects not fully |
|                     |     human-in-the-loop  |     identified        |
| -   Security policy |     operational        |                       |
|     and procedures  |     objectives         | -                     |
|     not in place    |                        |    Inter-relationship |
|                     | -   Inadequate         |     of hazard effects |
| -   Easy access to  |     specification of   |     not established   |
|     communication   |     human-in-the-loop  |                       |
|                     |     benefits           | -   Hazards not       |
| -   No provision    |                        |     classified per    |
|     for firewalls   | -   Inadequate         |     common scheme     |
|     between shared  |     analysis of        |                       |
|     networks or     |     human-in-the-loop  | -   Hazard class not  |
|     Virtual Private |     system capability  |     based on          |
|     Networks        |     to deliver         |     operational       |
|                     |     expected benefits  |     environment       |
| **Threat**          |     or enhancement     |     definition        |
|                     |                        |                       |
| -   Incomplete      | -   Human error        | **System Safety       |
|     threat          |     mechanisms or      | Interdependence**     |
|     assessment on   |     metrics not fully  |                       |
|     intent and      |     identified         | -   Hazard            |
|     capability to   |                        |     interdependence   |
|     exploit         | -   Time required to   |     poorly understood |
|     vulnerability   |     perform tasks is   |                       |
|                     |     unknown            | -   Interoperability  |
| -   No              |                        |     of components on  |
|     prioritization  | -   Automation does    |     system safety not |
|     of threat       |     not provide the    |     investigated      |
|     severity        |     necessary          |                       |
|                     |     functionality or   | -   Systemic approach |
| -   No provision    |     information to     |     to safety is      |
|     for penetration |     support effective  |     lacking one or    |
|     testing         |     decision-          |     more components   |
|                     | making/problem-solving |     (planning,        |
| -   Threat          |                        |     requirements,     |
|     difficulty not  | **Human-in-the-loop    |     procedures,       |
|     considered      | Suitability**          |     operation,        |
|                     |                        |     aircraft          |
| **Countermeasures** | -   Lack of            |     certification, or |
|                     |     consistency,       |     user approval)    |
| -   Few             |     compatibility, or  |                       |
|     countermeasures |     congruity with     | **Mitigation          |
|     defined         |     operational        | Strategies**          |
|                     |     environment or     |                       |
| -   Effectiveness   |     legacy systems     | -   Mitigation        |
|     of              |                        |     strategies not    |
|     countermeasures | -   Human-system       |     shared            |
|     on              |     design/interface   |                       |
|     infrastructure  |     induces            | -   Operational and   |
|     not testing     |     new/additional     |     safety objective  |
|                     |     human error        |     not established   |
| -   Inadequate      |     potential          |                       |
|     configuration   |                        | -   Lack of           |
|     audit           | -   Inadequate         |     critical/valid    |
|                     |     incorporation of   |     safety            |
| -   Lack of         |     functional         |     information       |
|     monitoring and  |     requirements to    |                       |
|     enforcement     |     support            | -   Mitigation        |
|                     |     user-system        |     strategies not    |
| -   Insufficient    |     performance goals  |     tied to hazards   |
|     funding         |                        |     or safety         |
|     tools/controls  | **User Acceptability** |     requirements      |
|                     |                        |                       |
| -   Ambiguous       | -   New tasks impose   | -   Plan for          |
|     funding support |     excessive          |     development and   |
|                     |     attention, memory, |     operational       |
|                     |     or workload        |     assurance not in  |
|                     |     demands            |     place             |
|                     |                        |                       |
|                     | -   Requires new       |                       |
|                     |     teaming and        |                       |
|                     |     communication      |                       |
|                     |     links              |                       |
|                     |                        |                       |
|                     | -   Operations         |                       |
|                     |     interface is       |                       |
|                     |     unacceptable to    |                       |
|                     |     user               |                       |
|                     |                        |                       |
|                     | -   Maintenance        |                       |
|                     |     interface is       |                       |
|                     |     unacceptable to    |                       |
|                     |     user               |                       |
+---------------------+------------------------+-----------------------+