![](media/image1.wmf){width="1.6493055555555556in"
height="1.6354166666666667in"}

**FAA Acquisition Program Baseline Instability:**

**Causes and Recommended Corrective Actions**

**NAS Configuration Management and Evaluation Staff**

**Program Evaluation Division (ACM-10)**

**Report #2000-10**

**February 4, 2000**

# Executive Summary

Acquisition program baselines are management tools that are useful to
the FAA in several ways. First, they permit managers to measure progress
internal to the agency. Second, baselines allow upper management to plan
and allocate scarce resources among program areas. Finally, baselines
permit the outside world to measure agency progress. This evaluation
examined baselined programs to determine the causes for baseline
instability in FAA acquisition programs and to recommend practical
solutions to address the identified causes. The report also addresses an
Office of Research and Acquisitions (ARA-1) inquiry concerning the
conditions under which the FAA should rebaseline acquisition programs.

Overall, the evaluation team found that acquisition program baseline
instability was caused by nine documented and three underlying factors.
Many of these causes were interrelated, and complete separation of the
issues was not possible. The nine documented causes were grouped into
three broad areas as listed below:

a)  Problems during the planning phase

```{=html}
<!-- -->
```
1.  Poorly defined or new requirements

2.  Human factors issues

3.  Unsuitable acquisition strategy

4.  Unrealistic baseline

5.  Inadequate risk assessment

```{=html}
<!-- -->
```
b)  Problems during the solution implementation phase

```{=html}
<!-- -->
```
6.  Software development difficulties

7.  Poor contractor performance

```{=html}
<!-- -->
```
c)  Problems with multiple issues outside the Integrated Product Team's
    control

```{=html}
<!-- -->
```
8.  Unstable funding

9.  Interdependencies

The three underlying causes, all of which were derived from interview
opinions, included lack of trust, lack of accountability, and lack of
empowerment. Although these causes have been reported in the past by
both internal and external sources, analysis of program documentation
revealed relatively little supporting data to verify these interview
opinions. Nevertheless, the team received widespread reiteration of
these root causes in a majority of programs.

The team also found that acquisition documentation was not always
consistent and needs to be better maintained. During the evaluation, the
team attempted to review program documentation and found that some of
the required documents for the programs were missing. Further
investigation with the responsible FAA office, Integrated Product Team
and other sources determined that, in several cases, the required
documentation could not be located at all.

The team also found that, while the agency has undertaken a significant
number of program rebaseline actions in the past two years, the process
for modifying baselines has not been consistently applied. Agency
guidance related to the rebaseline process, including the documentation
required for these actions, is vague and incomplete. We believe the FAA
should rebaseline programs for two purposes. First, to restructure a
program when the program goals, methods, or strategy have changed to
such an extent that the baseline no longer provides a realistic measure
of program progress. Second, to reestablish a manageable measure of
progress when one or more baseline parameters have been breached to such
a significant extent that those measures are no longer applicable.

Finally, the evaluation team discovered multiple initiatives currently
underway within the FAA that relate to baseline stability issues. Some
of these initiatives were designed to increase baseline stability;
others were not necessarily so designed, but nevertheless may produce
that effect. All such initiatives discovered during the evaluation are
presented in the body of the report for information purposes. Some were
referred to in the recommendations section.

The team also noted several trends and made other observations,
including the following:

-   Distinct patterns among the variance causes indicate that, without
    intervention, variances can be expected to continue, particularly in
    pre-Acquisition Management System programs.

-   Most problems appeared to occur during the planning phase,
    indicating its importance in baseline stability.

-   The agency lacks a process for dealing with issues arising from the
    use of commercial off-the-shelf/non-developmental items (COTS/NDI)
    as an acquisition strategy.

-   Less complex, smaller programs appear to have a better chance of
    maintaining baseline stability, indicating the advisability of such
    strategies as evolutionary acquisition, spiral development, and
    baselining incremental segments of programs.

```{=html}
<!-- -->
```
-   In the past two years, funding fluctuations have increased in
    significance as a cause of baseline instability.

**Summary of Recommendations**

The team identified eleven specific recommendations to improve baseline
stability within the FAA. These recommendations are summarized below.
Specific steps to implement the recommendations, as well as potential
points of contact, are identified in the body of the report.

1.  The FAA should continue to develop an integrated approach to the
    agency's entire portfolio and implement that approach as soon as
    possible. The portfolio management approach recently initiated
    within the agency may be a viable solution.

2.  When it employs a COTS/NDI acquisition strategy, the FAA should take
    specific steps to align the acquisition planning, implementation,
    sustainment, and budgeting processes with COTS/NDI. These steps must
    be undertaken across all lines of business.

3.  The FAA should continue its efforts to improve customer service in
    the development and delivery of equipment and services.

4.  The FAA should continue its efforts to improve integration of human
    factors requirements into all stages of program planning and
    solution implementation.

5.  The FAA should take aggressive measures to implement its new policy
    on risk assessment and mitigation.

6.  FAA Integrated Product Teams should consider innovative ways of
    structuring and conducting acquisitions to reduce cycle time,
    minimize risks, and maximize product utility.

7.  The FAA should assess past performance for all contractors and use
    it as a factor in contract award.

8.  FAA Product Teams should share information among themselves, both
    formally and informally, on lessons learned throughout a program's
    lifecycle.

9.  The FAA should improve its management of program documentation.

10. The FAA should develop and implement a consistent policy on
    rebaselining programs.

11. The FAA should appoint a central mechanism within the agency to
    coordinate all groups, initiatives, and efforts. This mechanism
    would ensure that groups are properly chartered, are serving a
    useful purpose, and do not duplicate or overlap with established
    groups.

Table of Contents

Executive Summary [i](#executive-summary)

Introduction [1](#introduction)

Background [1](#background)

Purpose of Acquisition Program Baselines
[1](#purpose-of-acquisition-program-baselines)

Prior Reviews [1](#prior-reviews)

Additional Request [2](#additional-request)

Objectives [2](#objectives)

Scope [2](#scope)

Methodology [2](#methodology)

Evaluation Results [5](#evaluation-results)

Findings [7](#findings)

Finding #1. Acquisition program baseline instability was caused by nine
documented and three underlying factors.
[7](#finding-1.-acquisition-program-baseline-instability-was-caused-by-nine-documented-and-three-underlying-factors.)

Requirements for Baseline Stability
[7](#requirements-for-baseline-stability)

Baseline Instability Continues to be a Problem
[7](#baseline-instability-continues-to-be-a-problem)

The Effects of Baseline Instability
[8](#the-effects-of-baseline-instability)

The Causes of Baseline Instability
[10](#the-causes-of-baseline-instability)

Finding #2: Acquisition documentation was not always consistent and
needs to be better maintained.
[19](#finding-2-acquisition-documentation-was-not-always-consistent-and-needs-to-be-better-maintained.)

Finding #3. Agency guidance on rebaselining programs needs improvement.
[20](#finding-3.-agency-guidance-on-rebaselining-programs-needs-improvement.)

Evaluation of Rebaseline Actions in Our Sample
[20](#evaluation-of-rebaseline-actions-in-our-sample)

Justification for Rebaseline Actions
[21](#justification-for-rebaseline-actions)

Other Matter---Multiple Agency Initiatives Addressing Baseline
Instability Issues
[23](#other-mattermultiple-agency-initiatives-addressing-baseline-instability-issues)

Conclusion [28](#conclusion)

Trends and Observations [29](#trends-and-observations)

Recommendations [31](#recommendations)

Appendix A---Details for Causes of Baseline Instability
[36](#appendix-adetails-for-causes-of-baseline-instability)

Appendix B---Reports Addressing Baseline Issues
[42](#appendix-breports-addressing-baseline-issues)

Appendix C---Acronyms [44](#appendix-cacronyms)

# Introduction

## Background

#### Purpose of Acquisition Program Baselines

Acquisition program baselines are management tools that are useful to
the FAA in several ways. First, baselines allow program managers to
measure progress internal to the agency. Milestones provide effective
points at which managers can determine if goals are being reached. They
also provide decision points for the agency to determine if additional
investment in a particular product is warranted. In addition, baselines
are useful tools to hold managers and employees accountable for meeting
performance objectives.

Second, baselines allow upper management to plan and allocate scarce
resources among program areas. Accurate and detailed predictions of
program timelines and costs allow the agency to prepare for future
upgrades to the National Airspace System (NAS) and to make trade-offs
when scarce resources must be prioritized. Such tools are necessary if
the agency is to have an effective and integrated management process at
an overall, strategic planning level.

Finally, baselines allow the outside world to measure agency progress.
Baselines are communicated to Congress and the entire aviation community
and constitute a statement of intent on the part of the agency. Congress
and the Administration use baselines to measure FAA effectiveness in its
modernization effort. The aviation community, both domestic and
international, makes plans for such diverse issues as aircraft equipage
and facility upgrades based on these statements of intent.

#### Prior Reviews

Since the FAA was granted acquisition reform in April 1996, the Program
Evaluation Division has reported three times on the status and success
of acquisition reform. These evaluations targeted different aspects of
acquisition reform for review. In conjunction with these evaluations,
the Program Evaluation Division undertook a review of program baseline
stability.

The objective of that review was to determine if FAA acquisition
programs were maintaining better cost and schedule baseline stability
under acquisition reform than was achieved before acquisition reform was
implemented. Originally, it was intended that these results would be
published as part of the report entitled *Evaluation of Acquisition
Reform: The First Three Years: April 1996-March 1999*. However, because
this evaluation measured baseline stability rather than true cost and
schedule reductions, and because our sample differed from
pre-acquisition reform and acquisition reform programs previously
announced by the FAA, it was decided that the results would appear in a
separate document.

The Program Evaluation Division's Report #1999-05, entitled *FAA
Acquisition Program Baseline Stability: Comparison between growth rates
for programs conducted prior to acquisition reform and those conducted
under acquisition reform* (Baseline Stability Report), was published in
September 1999. The report concluded that the FAA was still experiencing
significant cost and schedule growth in the period since acquisition
reform was implemented. That evaluation did not address the causes for
continued growth in acquisition program baselines. However, the report
recommended that "The FAA Associate Administrator for Research and
Acquisitions (ARA-1) should review the procedures and timing for
establishing program baselines to determine if the process could be
improved."

#### Additional Request

At about the same time the Baseline Stability Report was published,
ARA-1 requested an evaluation to answer the question "Under what
circumstances should the FAA rebaseline programs, and why?" The Program
Evaluation Division undertook that evaluation and surmised that
rebaselining might be a possible solution for addressing program
instability. Therefore, to respond to the ARA-1 query, the team first
answered the question "Why do FAA Acquisition programs continue to
experience baseline instability?" The Division decided to combine the
follow-on evaluation recommended in the Baseline Stability Report with
the evaluation requested by ARA-1.

## Objectives

The objectives for this evaluation were to determine the causes for
baseline instability in FAA acquisition programs and to recommend
practical solutions for addressing the identified causes.

## Scope

Of the more than 100 programs in the FAA's Capital Investment Plan
(CIP), FAA considered 42 programs "baselined." The agency used this term
to denote that the program had cost, schedule, performance, and/or
benefits data documented in either an Acquisition Program Baseline (APB)
or in a Parameter Sheet. Data submitted by the 42 programs is tracked in
the Simplified Program Information Reporting and Evaluation (SPIRE)
system. While all 42 programs had submitted baseline data into the
tracking system at the time of our evaluation, eight of these programs
were not reporting in SPIRE on a monthly basis. We limited our review to
the cost and schedule data being tracked in SPIRE on the 42 programs.
Benefits and performance data were not reviewed.

For this evaluation, we defined the "baseline process" as beginning with
a program's investment analysis process and ending with its last
operational readiness demonstration (ORD) or November 1999 if the last
ORD had not been reached.

## Methodology

To determine the causes for baseline instability and recommend solutions
for addressing those causes, the team performed the following:

-   Identified the evaluation universe as the 42 acquisition programs
    the agency was tracking in SPIRE. Reviewed all programs reporting a
    variance of more than 10 percent as of June 1999 and one additional
    program that reported a variance of more than 10 percent in
    July 1999. The programs reviewed were: the Airport Movement Area
    Safety System (AMASS), Acquire, the Backup Emergency Communications
    system (BUEC), the Common Automated Radar Terminal System (Common
    ARTS), the NAS Infrastructure Management System Phase 1 (NIMS), the
    Operational and Supportability Implementation System (OASIS), the
    Oceanic Automation Program Build 1, the Standard Terminal Automation
    Replacement System (STARS), the Wide Area Augmentation System
    (WAAS), and the Weather and Radar Processor (WARP). They are listed
    in Table 1 with the most recent variance information from the
    November 1999 Baseline Management Variance Report.

> Table 1. Reviewed programs with variances

  ----- ------------------ ------------ ----------------- -----------------
        **Program**        **Date of    **Type and Amount 
                           Variance**   of Variance**     

                                        **Cost**          **Schedule**

  1     **AMASS**          11/99        105%              44%

  2     **Acquire**        2/98         ---               14%

  3     **BUEC**           6/99         ---               48%

  4     **Common ARTS**    6/99         3%                22%

  5     **NIMS Phase 1**   6/99         58%               123%

  6     **OASIS**          6/99         27%               23%

  7     **Oceanic Build    6/99         ---               16%
        1**                                               

  8     **STARS**          11/99        49%               43%

  9     **WAAS**           11/99        TBD[^1]           100%

  10    **WARP**           11/99        14%               20%
  ----- ------------------ ------------ ----------------- -----------------

-   Divided the programs into two categories: those initiated prior to
    acquisition reform, and those initiated under acquisition reform.
    The team defined pre-acquisition reform programs as those initiated
    prior to October 1996. This is consistent with the agency's
    definition and conforms to the reporting requirements in Public Law
    104-246.

-   Examined the documentation from each of the 10 programs, APBs,
    Parameter Sheets, Baseline Management Notices (BMNs), and Baseline
    Management Variance Reports. Documentation was collected from at
    least three months prior to variance reporting to January 2000 and
    from earlier timeframes when needed to track root causes.

-   For the cost segment of baselines, primarily examined the Facilities
    and Equipment (F&E) portions of programs. Research, Engineering and
    Development (R,E&D) funds were expended mostly in the portion of
    programs prior to baseline establishment, and Operations and
    Maintenance (O&M) funds were expended in the in-service portion of
    the program lifecycle, into which most programs reporting variances
    had not yet transitioned. Within the scope of the evaluation,
    however, any O&M or R,E&D funds that contributed to baseline
    instability were examined.

-   Reviewed numerous internal and external reports on baseline issues,
    including documents issued by the General Accounting Office (GAO),
    the Department of Transportation Office of the Inspector General
    (DOT OIG), and internal FAA reports. See Appendix B for a list of
    these reports.

-   Monitored program variances reported subsequent to July 1999. As of
    November 1999, five additional pre-acquisition reform programs had
    reported variances of more than ten percent.[^2] Although these
    programs were not formally examined as part of the evaluation, the
    team noted the documented causes of these variances.

-   Reviewed documentation for several programs that had not experienced
    variances. The team examined the process used to establish
    baselines, the status of these programs, and the point within the
    lifecycle the programs were traversing. The team focused on any
    commonality of experience to account for the baseline stability
    these programs had achieved to date.

-   Collected information from other agencies, including the Department
    of Defense and the National Aeronautics and Space Administration
    (NASA), to identify best-practice models. The team also identified
    and collected data on internal FAA lessons learned and on the many
    initiatives underway within the FAA to address segments of the
    baseline process. These data were analyzed to generate ideas for
    improvements to baseline processes that could help mitigate causes
    of baseline instability.

-   Interviewed numerous parties inside and outside the agency,
    including Integrated Product Team (IPT) Leads, Product Team (PT)
    Leads, business managers, cost and schedule estimators, operational
    representatives, Investment Analysis (IA) Team members, Systems
    Engineering/Operational Analysis Team (SEOAT) members, and
    participants in initiatives designed to improve the baseline
    process.

# Evaluation Results

Baselines are required for all FAA programs. Because of the complex
nature and significant size of FAA programs, developing and maintaining
stable baselines is a complex and difficult task to accomplish. As
documented in numerous reports, the FAA has constantly experienced
problems maintaining baseline stability. When baselines are not stable,
the agency may lose its ability to plan and manage programs; may lose
its ability to meet its customer and user needs, which affects its
overall mission; may lose its credibility with Congress and other
entities inside and outside of the FAA; and may lose the commitment of
its employees due to a loss in employee morale. We assessed ten
baselined programs with variances of more than ten percent and found
nine documented causes and three underlying causes for baseline
instability.

We also found acquisition documentation was not always consistent and
needs to be better maintained. Six documents are mandated for programs
under acquisition reform. The team attempted to obtain some of this
documentation, and, in some cases, the documents could not be located.
In addition, information being reported was not always accurate or
consistent. This is significant, because the agency needs complete and
accurate documentation in order to track and measure programs' success.

We also found that the process for modifying baselines has not been
applied consistently, and agency guidance related to the rebaselining
process, including the documentation required for these actions, is
vague and incomplete. In light of this, and in direct response to
ARA-1's question about when to rebaseline, we noted two situations in
which the agency should consider rebaselining.

We also found numerous ongoing groups and initiatives in the agency.
Some appear to be addressing the causes identified in this report for
baseline instability; others do not.

The team also noted several trends and made other observations,
including the following:

-   Distinct patterns among the variance causes indicate that, without
    intervention, variances can be expected to continue, particularly in
    pre-AMS programs.

-   Most problems appeared to occur during the planning phase,
    indicating its importance in baseline stability.

-   The agency lacks a process for dealing with issues arising from the
    use of commercial off-the-shelf/non-developmental items (COTS/NDI)
    as an acquisition strategy.

-   Less complex, smaller programs appear to have a better chance of
    maintaining baseline stability, indicating the advisability of such
    strategies as evolutionary acquisition, spiral development, and
    baselining incremental segments of programs.

```{=html}
<!-- -->
```
-   In the past two years, funding fluctuations have increased in
    significance as a cause of baseline instability.

We worked collaboratively with numerous parties within and outside of
the agency to address the issues identified in this report, and we have
noted 11 recommendations for corrective action.

# Findings

## Finding #1. Acquisition program baseline instability was caused by nine documented and three underlying factors.

FAA policy states that acquisition program baselines provide
"performance and benefit objectives that are to be achieved within
strict cost and schedule boundaries." Based on findings in our previous
Baseline Stability Report, APBs have continued to experience significant
growth in the years since acquisition reform was implemented. Possible
effects of instability include reduced ability to manage programs,
effects on agency ability to meet customer needs, potential loss of
credibility, and the potential for lowered employee morale. The
evaluation team identified causes of instability that occurred in the
planning and solution implementation phases of a program lifecycle,
causes that were outside the control of the implementing IPT, and some
underlying causes of baseline instability.

#### Requirements for Baseline Stability

Section 2.11.3 of the Acquisition Management System (AMS) requires that
every acquisition have an acquisition program baseline. Section 2.11.3,
"Stable Acquisition Program Baselines," also states that baselines "are
the control element in the AMS that enables the agency to plan
realistically and commit to full funding of new programs." As such,
baselines establish "performance and benefit objectives that are to be
achieved within strict cost and schedule boundaries." Public Law
104--264 requires that, for programs initiated after October 1996, the
agency must consider termination of any program having a variance of
more than 10 percent. Program baseline variances of more than 50 percent
require the FAA Administrator to either terminate the program or notify
Congress of the circumstances which necessitate its continuation.
General business practices and common sense dictate that baselines must
be stable to properly manage programs.

#### Baseline Instability Continues to be a Problem

Our previous baseline stability evaluation found, and this evaluation
confirmed, that baseline instability continued to be a concern in the
FAA under acquisition reform. Of the 42 programs with baselines
currently being tracked, 18 reported variances between 1996 and June
1999, including 13 of 28 programs initiated prior to acquisition reform
and 5 of 14 programs initiated under acquisition reform (i.e., after
October 1996). Table 2 contains a summary of these statistics.

Table 2. Baselined FAA Programs and Their Variances

  ----------------- ----------- ---------------- ---------- --------- ---------
                                **Baseline                            
                                Variances as of                       
                                June 1999**                           

                    Total       Total programs   \< 10%     \> 10%    Percent
                    programs    with variances                        

  Pre-Acquisition   28          13               7          6[^3]     46%
  Reform                                                              

  Acquisition       14          5                2          3         36%
  Reform                                                              

  Total             42          18               9          9[^4]     43%[^5]
  ----------------- ----------- ---------------- ---------- --------- ---------

#### The Effects of Baseline Instability

The results of baseline instability are significant, and the agency is
affected in at least four ways. First, baseline instability may affect
the agency's ability to plan and manage programs. Baselines provide a
guideline for anticipating new capabilities and decommissioning old
ones. Further, they allow the FAA to predict resource requirements on
macro and micro levels. When baselines are not met, the agency cannot
accurately anticipate either resource requirements or capabilities in
the future. Thus, the agency is forced to plan and budget on a
short-term basis, creating a reactive environment in which crisis
management becomes normal and expected. Strategic planning becomes
ineffective. Few programs are allowed to remain stable, since funds must
constantly be moved to "rescue" programs with significant variances.
Measurements of modernization progress become more difficult because
baseline changes for funding reasons are so frequent.

Second, baseline instability can affect the agency's mission when the
FAA is unable to meet customer and user needs in a timely manner. If
programs required to modernize and sustain NAS capabilities cannot be
delivered on schedule, potentially critical NAS capabilities will be
missing. The loss of benefits from unfielded systems also creates
inefficiencies in the NAS. The programs in our review contributed to
safety, security, and system efficiency, as well as other agency mission
functions.

Third, significant and sustained baseline instability may result in a
loss of credibility with Congress, the flying public, the Office of the
Secretary of Transportation, the Office of Management and Budget (OMB),
and the domestic and international aviation communities. In an
environment of slipped program milestones, these important groups begin
to perceive that the FAA cannot be trusted to finish planned tasks in a
timely and cost-effective manner. Such erosion in agency credibility
creates lessened agency support and funding from the Administration and
from Congress. Such loss of funding exacerbates the effects listed
above. Loss of credibility may also create negative publicity for the
FAA and may potentially affect the aviation industry. Eventually, loss
of credibility could lead to a lessening of the FAA's position of
leadership in the global aviation community.

Finally, baseline instability can negatively affect employee morale.
Employees may become cynical about establishing baselines since the
chances are quite high that any established baseline will change almost
immediately. The process begins to be viewed as superficial and
arbitrary, and employees may begin to manipulate the system. Eventually,
as negative feedback and the sense of frustration increase, employees
may become less committed to the failed processes and may become
discouraged enough to leave the agency. Attracting qualified
replacements in an agency perceived as ineffective may also become more
difficult.

![](media/image2.wmf){width="4.996527777777778in"
height="2.9972222222222222in"}The relationship among listed effects is
represented pictorially in Figure 1.

> **Figure 1. The effects of FAA Acquisition Program Baseline
> Instability**

#### The Causes of Baseline Instability

##### Documented Causes

The nine documented causes were grouped into three broad areas as listed
below:

a)  Problems during the planning phase

```{=html}
<!-- -->
```
1.  Poorly defined or new requirements

2.  Human factors issues

3.  Unsuitable acquisition strategy

4.  Unrealistic baseline

5.  Inadequate risk assessment

```{=html}
<!-- -->
```
b)  Problems during the solution implementation phase

```{=html}
<!-- -->
```
6.  Software development difficulties

7.  Poor contractor performance

```{=html}
<!-- -->
```
c)  Problems with multiple issues outside the Integrated Product Team's
    control

```{=html}
<!-- -->
```
8.  Unstable funding

9.  Interdependencies

Many of these causes were interrelated, and complete separation of the
issues was not possible. However, the team concluded that each of these
constituted a separate problem that deserved individual discussion.

A summary of the causes noted in each reviewed program is shown in Table
3 below, and a discussion is provided in the narrative that follows.

Table 3. Summary of Baseline Instability Causes

  ------------- -------- -------- ---------------- ---------- -------------- --------------- -------------- ---------------- --------------- -------------- -------------- ------------------------
                                  **Problems                                                                **Problems                       **Problems                    
                                  During the                                                                During Solution                  Outside                       
                                  Planning Phase**                                                          Implementation                   Integrated                    
                                                                                                            Phase**                          Product Team's                
                                                                                                                                             Control**                     

  **Program**   **Pre-   **Acq.   **Poorly Defined **Human    **Unsuitable   **Unrealistic   **Inadequate   **Software       **Contractor    **Unstable                    **Inter-dependencies**
                Acq.     Ref.**   or New           Factors    Acquisition    Baseline**      Risk           Development**    Performance**   Funding**                     
                Ref.**            Requirements**   Issues**   Strategy**                     Assessment**                                                                  

                                                                                                                                             **Internal**   **External**   

  **AMASS**     X                 X                X          X              X                              X                X                                             

  **Acquire**            X                                                                                                                                                 

  **BUEC**      X                                                                                                                            X              X              

  **ARTS**      X                                  X                                                                         X                                             

  **NIMS**               X        X                           X              X               X                                               X              X              

  **OASIS**              X        X                X          X                                                                              X              X              

  **Oceanic**   X                 X                                                                                          X                                             X

  **STARS**     X                 X                X          X              X                              X                X               X                             

  **WAAS**      X                 X                                          X               X              X                X                              X              

  **WARP**      X                 X                                                                         X                X               X                             X

  **Totals**    **7**    **3**    7                **4**      **4**          **4**           **2**          **4**            **6**           **5**          **4**          **2**
  ------------- -------- -------- ---------------- ---------- -------------- --------------- -------------- ---------------- --------------- -------------- -------------- ------------------------

###### Problems during the planning phase

Causes within this group resulted from deficiencies during activities
prior to the solution implementation phase. This grouping included five
of the nine documented causes; eight of the ten programs reviewed
experienced problems during the planning phase. This finding indicates
that the planning process is an area that requires specific
improvements. A summary of the finding is contained in the discussion
below. For more details of problems experienced by each affected
program, see Appendix A.

***1. Poorly Defined or New Requirements***. In seven of the ten
reviewed programs, user or system requirements were not adequately
defined and documented to allow implementation of the program within the
established baseline, or new requirements were added after the baseline
was established.[^6] These programs experienced both schedule delays and
cost increases resulting at least partially from changes in
requirements.

The requirements changes resulted from various planning shortfalls. Some
PTs did not understand the user requirements before beginning solution
implementation. Others experienced human factors issues that added
program requirements. Other programs had new requirements added after
the baseline was established.

***2. Human Factors Issues***. For purposes of this evaluation, human
factors issues included both human engineering and user preference
issues. Four of the ten programs incurred variances as a result of human
factors issues identified after the program baseline was established,
although in most cases there were indications that the acquisition
approach involved some significant human factors risks. The variances
were the result of several different sets of circumstances. For some
programs, the issues were not anticipated; for others, they were
ignored. For example, in at least one instance, when the FAA decided to
adopt a policy of acquiring commercial off-the-shelf/ non­developmental
item (COTS/NDI) systems whenever possible, the agency erroneously
believed that market place forces would provide an engineering solution
that was sufficient to meet actual NAS requirements. This lack of
attention to detailed requirements led to a false assumption that any
design difficulties could be easily, quickly, and inexpensively
resolved. Deficiencies in the design led to a complex relationship
between redesign and engineering activities, union negotiations, and
contractor--government deliberations on the program's future cost and
schedule. Completion of the design changes resulted in increases to both
cost and schedule for this program.

Other programs encountered human factors objections from field personnel
during the solution implementation phase of the program. The PTs
negotiated a set of required changes with union representatives, and
implementation of these changes resulted in increases to both cost and
schedule.

***3. Unsuitable Acquisition Strategy***. Four of the ten programs
selected an acquisition strategy that, in hindsight, was not
commensurate with the requirements or with the commercial availability
of the required product. Some programs chose strategies that, in the
interest of saving time, attempted to compress the acquisition cycle.
The results were longer development and deployment sequences and
increased costs. Other programs chose a COTS/NDI strategy without fully
understanding the implications, resulting in increased costs and
development delays.

***4. Unrealistic Baseline***. Four of the programs established
unrealistic baselines from the beginning of the program. Most of these
baselines appeared to be driven by unrealistic expectations from both
internal and external stakeholders. For example, some programs
promulgated cost and/or schedule baselines based on high-level
management decisions that the PT knew to be unrealistic. These decisions
were apparently the result of political expectations. Another program
baseline was established with unrealistic cost and performance baselines
due to an affordability assessment that forced program costs to fit
available funding.

***5. Inadequate Risk Assessment***. At least two of the programs did
not understand the risks involved at the beginning the program. The
original baseline was not based on a full understanding of the
technical, institutional, and programmatic risks. One of these programs
was initiated prior to AMS, and a formal risk analysis was not
performed. Although these were the only documented cases of inadequate
risk analysis per se, the two other programs that used a COTS/NDI
strategy could also be placed in this category. Neither program
understood the risks of electing to acquire a COTS/NDI system to satisfy
the requirements.

In the program planning area, two issues emerged that merit further
discussion. First, the evaluation team determined that acquisition
reform planning process changes did not have an impact on baseline
stability. Seven of the ten programs were initiated prior to acquisition
reform and, as a result, lacked the benefit of AMS-required planning
functions. For example, none of the seven programs completed an
investment analysis report before solution implementation began, [^7]
although two of the programs (STARS and WAAS) have since completed
mini-investment analysis reports. The baseline establishment process
used in these programs lacked the rigor required under AMS. Three of the
ten reviewed programs were initiated under acquisition reform;
investment analysis reports were completed for two of them (Acquire and
OASIS), both of which were initiated near the beginning of the AMS time
period. One of these two programs, however, seemed to experience the
same planning problems as those programs initiated prior to acquisition
reform. The third program initiated under acquisition reform (NIMS) was
the first program to be conducted under AMS, and its investment analysis
report was never officially published. The NIMS affordability assessment
was forced to fit a predetermined funding level. These three programs
did not have full success in applying AMS processes, but they all were
initiated near the beginning of the AMS time period.

Second, AMS's preference for COTS/NDI solutions has had a significant
impact on the baseline process. PTs experienced problems resulting from
the selection and implementation of COTS/NDI equipment to meet user
needs. Several programs elected to use a COTS/NDI strategy, and each
experienced similar planning problems associated with that choice. Some
of these problems seemed to arise from a lack of understanding about the
consequences of choosing a COTS/NDI strategy in terms of lifecycle
planning and budgeting, deployment, and sustainment. Successfully
adopting COTS/NDI as a procurement strategy entails recognition of a
different set of risks and requires proactive management to mitigate
those risks.[^8] However, the FAA did not have policy or guidance
concerning COTS/NDI strategy and its implications, and the affected PTs
did not adapt their development, deployment, and sustainment strategies
to plan for COTS.

The evaluation team did not attempt to determine if the agency policy on
COTS/NDI was the "right" approach to acquisitions. Other reports
addressed that issue. An internal report prepared by the FAA Office of
Acquisitions entitled *Effectiveness of COTS Solutions* found that COTS
programs tended to save the agency money in the procurement and
deployment stages. The *Independent Assessment of the Federal Aviation
Administration's Acquisition Management System* found that COTS
implementation in the FAA may not save the agency money over the
lifecycle of a program, given the higher costs for sustainment and
technology refresh. Research and lessons learned through the En Route
IPT experiences with COTS/NDI indicated that, in some cases, given the
current state of the market, the agency may not be able to avoid a
COTS/NDI strategy. Developmental programs are very costly. At the same
time, the government no longer drives the market, which gives impetus to
move to a COTS/NDI solution for sustainability purposes. The FAA, like
other government acquirers, may be forced in some cases to move towards
COTS/NDI and commercially available software (CAS) systems to meet user
requirements.

###### Problems during the solution implementation phase

Causes within this group resulted from activities in the solution
implementation phase of the program lifecycle. Two specific causes were
identified: Software development problems, and contractor performance
problems. Six of the ten programs experienced variances resulting from
one or both of these causes.

***6. Software Development Difficulties***. In a 1997 report, GAO stated
that the FAA's "processes for acquiring software...are ad hoc, sometimes
chaotic, and not repeatable across projects."[^9] Although improvements
have been made since that report, four of the ten programs experienced
difficulties with software development that resulted in or contributed
to cost and/or schedule variances. These variances resulted from
underestimation of lines of code required to meet user requirements and
delays in development and integration of software.

***7. Poor Contractor Performance***. This issue is closely related to
software development difficulties, since contractors are usually
responsible for developing software for FAA systems. Six of the ten
programs experienced poor contractor performance, resulting in cost
and/or schedule variances. Problems included difficulties in attracting
and retaining sufficient staff to perform the tasks, delays in software
development and integration, and inability to deliver required
functionality.

It should be noted that half of the programs in this evaluation remain
in the development stage of the program lifecycle and have not yet begun
to deploy the technology. Only one of the programs has been completely
deployed. Given this status, other causes for baseline instability may
exist that did not come to light during this evaluation.

###### Problems with multiple issues outside the IPT's control

Causes within this group also resulted from activities in the program
planning or solution implementation phase of the program lifecycle, but
they were related to issues outside the control of the IPTs. Two
specific causes were identified: unstable funding, and
interdependencies. Seven of the ten programs experienced variances
resulting from one or both of these causes.

***8. Unstable Funding**.* Funding stability was an important factor in
an IPT's ability to maintain stable baselines. Programs that received
substantially reduced funding in multiple years were not able to perform
the program mission on schedule and within budget. Funding instability
can result from internal (i.e., FAA-imposed) cuts or adjustments to the
program funding profile.[^10] Funds may also be adjusted external to the
agency by the Office of the Secretary of Transportation, the Office of
Management and Budget (OMB), or Congress.

The team found that the perception among the majority of program
personnel was that funding instability was the driving force behind the
inability to meet program baselines. Given that perception, the
evaluation team conducted an analysis of historical funding profiles for
all ten programs, focusing on external funding. The results are
displayed in Table 4.

**Table 4. Request versus Appropriation Fiscal Year 1996--Fiscal Year
2000 (\$M)**

  ------------- --------------- ------------------ ---------------- ---------------
  **Program**   **Requested**   **Appropriated**   **Difference**   **%
                                                                    Difference**

  **AMASS**     48.2            66.3               18.1             37.6%

  **Acquire**                                                       

  **BUEC**      26.5            23.6               --2.9            --11.0%

  **Common                                                          
  ARTS**                                                            

  **NIMS Phase  69.3            47.5               --21.7           --31.4%
  1**                                                               

  **OASIS**     71.1            52.4               --18.7           --26.4%

  **Oceanic                                                         
  Build 1**                                                         

  **STARS**     642.1           612.1              --30.0           --4.7%

  **WAAS**      488.5           499.7              11.2             2.3%

  **WARP**      89.7            91.9               2.1              2.4%
  ------------- --------------- ------------------ ---------------- ---------------

> **Notes:**

1.  Positive difference indicates appropriated more than requested.

2.  Negative difference indicates appropriated less than requested.

3.  Covers period Fiscal Year 1996--Fiscal Year 2000.

4.  No data for Common ARTS and Oceanic Build 1.

5.  ACQUIRE was funded internally.

The team discovered that funding instability indeed contributed greatly
to external baseline instability for some programs, while others
received nearly what was requested each year. Three programs in
particular were affected by external cuts. However, the analysis was
more complex than the data shown in Table 4. For example, although some
programs received within 10 percent of what was requested overall, the
table does not indicate funding fluctuations from year to year. These
fluctuations were significant for some programs in specific years. For
example, the WAAS program was cut \$28.1 million in Fiscal Year 2000.
Overall, this program seems to have received ample funding from
FY96-FY00. However, the cuts affected the program's ability to meet the
program baseline, since milestones for that year depended on timely
funding.

Internal to FAA, the analysis became even more complex. The team was
unable to develop a concise summary of internal funding history for the
programs. The number of layers, iterations, and caveats quickly became
unmanageable. It was also difficult to distinguish which cuts were made
as a result of Office of the Secretary of Transportation or OMB-imposed
funding request restrictions. However, the team concluded that internal
funding fluctuations may have created greater impact on baseline
stability than those imposed external to the agency. Again, some
programs appeared to have fared well in this process while others were
significantly affected by internal adjustments.

Overall, six[^11] programs experienced funding reductions or adjustments
that contributed to baseline variances. These will be discussed in terms
of internal and external funding instability.

*Internal Funding Instability*. Five of the ten programs experienced
internally-imposed funding reductions or adjustments to the funding
profile that contributed to baseline variances. These adjustments
resulted from the agency's efforts to address funding shortfalls. In
some cases, lower priority programs were cut or deferred to provide
funding for higher priority programs. In other cases, shortfalls in O&M
funding or in other F&E lines were addressed in the funded baselines,
increasing the costs for particular programs.

*External Funding Instability*. Four of the ten programs experienced
funding cuts from Congress that contributed to their inability to
maintain baseline stability.

***9. Interdependencies.*** Two programs experienced schedule variances
as a result of the failures of interdependent programs to meet key
milestones or performance standards. These schedule delays resulted from
either the requirement to wait for the required functionality to become
available or the lack of sufficient resources elsewhere in the agency to
work with multiple functionalities.

##### Underlying Causes

In addition to the documented causes discussed above, for which the
evaluation team was able to obtain data to support our conclusions, we
noted three underlying agency causes for baseline instability. We
discovered these causes solely through interviews with numerous IPT and
PT members. Interviewees were reluctant to discuss these underlying
causes without our assurance of anonymity. These causes included lack of
trust, lack of accountability, and lack of empowerment. These underlying
causes are complex and interrelated, and their relationships to
documented causes were not always readily apparent.

Although these causes have been reported many times by both internal and
external sources,[^12] analysis of program documentation revealed
relatively little supporting data to verify these interview opinions.
Nonetheless, the team received widespread reiteration of these root
causes in a majority of programs.

***1. Lack of Trust***. The team found indications of lack of trust at
several different levels within the organization. The most significant
of these was distrust between FAA lines of business, particularly the
Office of Research and Acquisitions (ARA) and the Office of Air Traffic
Services (ATS). Even among IPT members, the distinction between
operations and acquisition was keenly felt. Several ARA personnel
expressed opinions that the ATS representatives were not empowered and
could not make decisions for their organizations. A few ATS
representatives perceived that the ARA-led teams were secretive,
withholding information from ATS team members. This distrust seemed to
contribute to several of the causes for baseline variances, including
requirements changes and human factors issues.[^13]

Interviewees also expressed considerable frustration about management
decisions that negatively affected program baselines. These decisions
included publicizing cost baselines that were unrealistic or schedule
baselines that were artificially short. Additionally, decisions about
acquisition strategy and funding were made without the consent or
approval of the responsible IPT. Interviewees stated that this type of
management direction illustrated a lack of trust in subordinates and an
unwillingness to listen to the "experts" hired to make realistic
projections. This issue also relates to lack of empowerment, discussed
below.

***2. Lack of Accountability***. Interviewees stated that baselines were
subject to interference from those within the organization who had no
accountability for variances. The frustration was particularly
widespread when interviewees spoke of funding issues. Almost all
interviewees expressed strong disapproval of the current practice of
"salami slicing" (i.e., slicing the funding of most programs to adjust
to budget realities or to rescue other programs with variances). They
stated that decisions to cut many programs instead of terminating the
lowest priority programs were made by those having no accountability for
baseline stability.

Interviewees also expressed opinions that some lines of business felt
free to hold up programs by delaying time-critical decisions because
those lines of business were not accountable for program success. This
led to new requirements being introduced and led to an inability to
resolve interdependency issues.

***3. Lack of Empowerment***. As stated above, many interviewees
believed that members of IPTs were not sufficiently empowered to do the
job. Decisions reached by the IPT with the support of IPT
representatives were overturned by their parent organizations, leading
to additions of new requirements, requirements changes, and an inability
to resolve issues in a timely manner. Some of these changes were due to
fluctuations in personnel. When a new IPT representative appeared or a
new boss emerged somewhere up the organizational chain, many
interviewees suggested that all previous decisions became "up for
grabs."

The problems were compounded by the hierarchical structure in FAA. The
many layers of management fostered a controlling environment in which
some employees did not feel empowered to make decisions or were not held
accountable for the decisions they made.

## Finding #2: Acquisition documentation was not always consistent and needs to be better maintained.

During the evaluation, the team attempted to review program
documentation and found that some of the required documents for programs
were missing. Further investigation with the responsible FAA office,
IPTs, and other sources determined that, in several cases, the required
documentation could not be located at all. We also found that some
documentation was not accurate or consistent. For example, in one case,
the Joint Resources Council (JRC) minutes were inconsistent with other
program documentation, and this inconsistency led to confusion about
program status. In addition, eight programs that were baselined in the
SPIRE system were not reporting variances on a monthly basis.

Documentation requirements for acquisition programs are specified in the
AMS, Section 2.12. All programs are required to maintain six specific
documents (Mission Need Statement, Requirements Document, Investment
Analysis Report, Acquisition Program Baseline, Acquisition Strategy
Paper, and Integrated Program Plan) developed during the planning stages
of an acquisition. Guidance on variance reporting is provided in the
Integrated Baseline Establishment and Management (I-BEAM) Process
document, which instructs programs to report variances on a monthly
basis.

The FAA maintains a central repository for AMS-required program
documentation, located in the Document Control Center. This location is
required to contain approved versions of all applicable program
documentation. The SPIRE system has been established to collect program
progress data, including monthly variance reports. GAO has recommended
that the SPIRE system also contain electronic versions of all program
documentation. The process of implementing this recommendation is
underway but is not complete. To date, the documentation contained in
SPIRE is limited.

The lack of available documentation harms the FAA in several ways.
First, without documentation it becomes more difficult to track and
measure progress or determine what methods were used to make critical
program decisions. Second, this inability to locate the documentation
leads people to question the original purpose of developing the
document. If it cannot be located, it is obviously not in use. In the
case of many AMS-required documents (e.g., APBs and risk mitigation
strategies), if the document is not to be used, the resources spent on
its development may have been wasted. Finally, an inability to locate
required documentation reduces agency credibility in the eyes of
outsiders such as Congress and the aviation community. Inability to
provide an audit trail of programmatic and technical decision-making
creates the perception of lack of structure and accountability in these
areas. This lack of documentation has been noted by the GAO, the OIG,
and the Program Evaluation Division in previous reports.

## Finding #3. Agency guidance on rebaselining programs needs improvement.

While the agency has undertaken a significant number of program
rebaseline actions in the past two years, the process for modifying
baselines has not been consistently applied. Agency guidance related to
the rebaseline process, including what documentation is required for
these actions, is vague and incomplete. Considering that six of the ten
programs in our sample were rebaselined at least once since September
1998 and two additional programs have recently requested rebaseline
actions, we believe the agency needs to develop and implement clear and
specific guidelines on the rebaselining process.

Since the FAA began tracking program baselines as a part of AMS in 1996,
the agency has periodically rebaselined programs that exceed cost,
schedule, performance, or benefits baselines. Rebaseline actions approve
and document the changes to baseline parameters. These actions are
perceived as a way for a program to "start over" when circumstances no
longer make the current baseline realistic.

According to the FAA's I-BEAM Process document, dated March 1999,
"Rebaselinings will be approved by the JRC only if circumstances dictate
that the program can no longer be managed to its original baseline.
Changes must be so extensive that the original baseline no longer has a
value for measuring the performance of the agency with respect to its
objectives." In September 1999, the agency developed a draft revision to
the I-BEAM Process document that provides examples of events for which
rebaselining would be appropriate, including changes to the program at
contract award, changes in budget allocations, and changes in
requirements.

#### Evaluation of Rebaseline Actions in Our Sample

Six of the ten programs in our sample were rebaselined at least once
since September 1998. Table 5 contains specific data on these programs.
Two additional programs, AMASS and BUEC, have recently requested
re-baseline actions and are scheduled to appear before the JRC in March
2000. While the JRC approved all six rebaseline actions, the rebaseline
process and documentation requirements were not clear or consistent. For
example, it was not clear why one program needed to be rebaselined.
Also, while certain PTs updated the baseline parameters in the APBs,
other PTs considered the BMN sufficient documentation of the new
baseline parameters. In addition, there was confusion within the agency
about the effects of rebaselining. While many interviewees believed that
rebaselining allowed a clean start from the new baseline, others
realized that the agency tracked and reported variances from the
original baseline, irrespective of the new baseline.

Table 5. Reviewed Program Rebaselines

  -------- --------------- ---------------- ------------------- ---------------
                           **Original       **Most Recent       **Total Number
                           Baseline**       Rebaseline**        of
                                                                Rebaselines**

  **1**    **AMASS**       PS 10/98         N/A                 0

  **2**    **Acquire**     APB 12/96        N/A                 0

  **3**    **BUEC**        PS 10/98         N/A                 0

  **4**    **Common ARTS** APB 12/96        N/A                 0

  **5**    **NIMS Phase    APB 3/97         7/99                1
           I**                                                  

  **6**    **OASIS**       APB 4/97         5/99                1[^14]

  **7**    **Oceanic Build APB 6/96         9/98                1
           1**                                                  

  **8**    **STARS**       APB 3/96         In process          1

  **9**    **WAAS**        APB 1/98[^15]    9/99[^16]           1

  **10**   **WARP**        APB 3/95         8/99                1
  -------- --------------- ---------------- ------------------- ---------------

#### Justification for Rebaseline Actions

In response to ARA-1's question, "Under what circumstances should the
FAA rebaseline programs, and why?" the team examined possible reasons
from several different perspectives. We considered such philosophical
issues as whether programs should be rebaselined only when changes were
driven by events beyond the agency's control (e.g., funding reductions),
and whether rebaselines should be considered a means of "starting over"
for programs. We determined that rebaseline criteria should be based on
practical considerations. Baselines are tools for managing program
progress. When such tools are no longer useful, they should be revised.
Based on our review, we believe that the FAA should rebaseline programs
in two situations:

1.  To restructure a program when the program goals, methods, or
    strategy have changed to such an extent that the baseline no longer
    provides a realistic measure of the program progress. Three of the
    six rebaselined programs evaluated experienced significant changes
    to program structure or performance parameters as well as
    significant baseline variances. These programs needed to be
    rebaselined to allow the PTs to effectively manage to the programs'
    extended baseline parameters.

2.  To reestablish a manageable measure of progress when one or more
    baseline parameters have been breached to such a significant extent
    that those measures are no longer applicable. Two of the six
    rebaselined programs in our sample experienced baseline variances so
    large that the baseline was no longer a realistic measure of the
    program's progress. These programs needed to be rebaselined to
    establish a reasonable measure for which the current PT could be
    held accountable.

While the September 1999 draft revision to the I-BEAM Process document
provides examples of events in which rebaselining would be appropriate,
additional clarification needs to be developed and formally issued. This
guidance should clarify the role of rebaselining, the process for
requesting and documenting a rebaseline action, and the required
justification for rebaseline requests. In addition, the guidance should
include a mechanism for reviewing and approving rebaseline requests
prior to the JRC presentation.

# Other Matter---Multiple Agency Initiatives Addressing Baseline Instability Issues

During the evaluation, the team discovered multiple teams and
initiatives currently underway within the FAA that relate to baseline
stability issues. Some of these initiatives were designed to increase
baseline stability; others were not necessarily so designed but
nevertheless may produce that effect. All such initiatives discovered
during the evaluation are presented here for information purposes. Some
will be referred to in the recommendations section.

The team discovered these initiatives through referrals during
interviews, examination of agency reports such as the Program Evaluation
Division Report #1999-09, "Inventory of Agency Process Initiatives," and
other sources. However, the team could locate no central repository for
information on such agency initiatives. Therefore, the following
discussion may not include all applicable initiatives.

***1. Portfolio Management.*** This initiative began formally with a
memorandum signed in December 1999 by five FAA senior executives and was
designed to "improve our capital investment decisions." Based on work
completed by an Integrated Product Leadership Team sub-team, the
initiative will "aggregate investment candidates into funding categories
that correspond to the business model of the enterprise and allow senior
management to manage the capital investment portfolio as a whole with
the aim of increasing benefits and managing risks." The effort will
essentially organize FAA operations and acquisitions into "integrated
service teams." These teams will be led by a portfolio manager who will
be responsible for managing all the resources required to provide the
service. Senior FAA management will be responsible for making trade-offs
between the services.

If implemented widely, the effort has the potential to produce
far-reaching changes within the FAA. It could improve the agency's
ability to make sound capital investment decisions, manage program
performance and risks, and improve FAA's communication with Congress and
other stakeholders. On a deeper level, the effort could address the
underlying causes identified in this report.

The effort is still rather early in its development. In 2000, a working
group will be formed to examine the broad issues of portfolio
management, and a pilot will be initiated, using the Terminal/Surface
Separation Assurance Service. A schedule beyond 2000 has not yet been
developed.

***2. COTS/NDI.*** Efforts to improve the way the agency manages
COTS/NDI-based acquisition programs do not really constitute an
initiative. However, we found that, throughout the agency, specific
organizations have begun to recognize the far-reaching implications of
the agency's preference for COTS/NDI acquisition strategies and have
initiated efforts to manage the changes such a strategy entails. These
efforts appear to be fragmented and sometimes uncoordinated. They
include:

a.  The En Route IPT (AUA-200). This IPT seems to have taken the lead in
    ARA in terms of identifying COTS/NDI impacts on acquisition program
    management and execution, and has initiated several efforts aimed at
    changing the acquisition management processes to recognize the
    impacts of COTS/NDI. AUA-200 began by instituting plans and
    processes within the En Route IPT. These include:

-   Successful deployment of the COTS-based Display System Replacement
    (DSR) program. The DSR program ensured early user involvement and
    acceptance, effective in-service management planning and transition,
    budget stability, and technology refresh data requirements and
    planning.

-   The Host and Oceanic Computer System Replacement (HOCSR) program was
    the first to have the operations budget baselined and to include
    technology refresh of the computer as part of the program baseline.

-   Development of Section 5.0 of the En Route "Green Book" entitled
    "Sustainment and Technology Evolution Planning."

-   Development of the En Route Domain Infrastructure budget line. This
    line combines technology refresh, pre-planned product improvement
    activities, and sustainment for the en route domain, recognizing
    that COTS/NDI systems require constant and often short-term refresh.

-   Development of the Technology Refresh Planning Guide, dated January
    5, 1998.

-   The Voice Switching and Control System Sustainment and Technology
    Evolution Planning effort to develop and maintain system
    obsolescence profile for budget planning and defense.

> Additionally, AUA-200 has attempted to expand these ideas and lessons
> learned to other parts of the agency. Efforts include:

-   Recent demonstration of COTS-sensitive life-cycle modeling tool.

-   Development of a draft AUA position paper entitled "COTS and the
    FAA: Ready or Not?" This paper outlines actions the FAA needs to
    undertake to manage in a COTS environment.

-   Presentation to the Air Traffic Automation (ATA) Integrated
    Management Team entitled "Technology Refresh, Foundation for
    Strategic COTS Management," January 19, 2000.

b.  The Terminal IPT (AUA-300). The STARS program established a COTS
    manager position to ensure obsolescence and pre-planned product
    improvement planning is executed. This position is responsible to
    execute the contract option with the prime contractor specifically
    tailored to commercial product technology refresh planning.

c.  The Investment Analysis and Operations Research Division (ASD-400).
    ASD-400 began an effort to develop a COTS-based modeling approach
    for the NAS.

d.  The Office of Acquisitions (ASU). ASU established a flow chart for
    COTS/NDI hardware and custom software development and included it in
    the FAA Acquisition System Toolset. They are also currently in the
    process of adding COTS-related data to the individual flow chart
    steps as guidance.

e.  The NAS Operations Service (AOP). AOP released Policy 4560.1C in
    draft form for agency review in November 1998. The intent of the new
    policy was to capture the data requirements for COTS-based systems
    and to reflect the transition of the FAA Logistics Center to a fee
    for service organization. To date, the draft has not been updated.
    Responsibility for the update of this policy has recently been
    transferred to the NAS Logistics Property Management Division
    (AFZ-500).

f.  The NAS Logistics Property Management Division (AFZ-500). AFZ-500
    created a position for a Contractor Depot Logistics Support (CDLS)
    funding coordinator to prioritize funding for CDLS Contracts used to
    manage COTS-based systems and to identify standard data requirements
    for CDLS contracts.

g.  The Configuration Management Division (ACM-20). ACM-20 has recently
    updated the AMS and an FAA policy (Order 1800.66) to reflect the
    current NAS configuration management policy and process activities
    being implemented by the agency. The configuration management of
    COTS/NDI systems and equipment are addressed by the policy and life
    cycle process. There is further work underway in the development of
    NAS configuration management procedures that will provide a more
    detailed description of how to configuration manage COTS/NDI. The
    procedure development should be completed by May 2000.

We believe that these initiatives constitute a good start in attempting
to improve the way the agency handles COTS/NDI acquisitions.

***3. Operations Baseline Team.*** This group was established in August
1998 in response to recommendations made by the Budget Process Team. One
of the objectives in the Team's charter is "the development of a
recommended ATS Budget Formulation process that takes into account
actual and projected costs of providing services, actual and projected
costs of maintaining existing systems, results of investment decisions,
and the inter-relationship between the F&E and O&M budgets." The group
is currently engaged in breaking out operations funding for F&E programs
on an annual basis. It has selected 25 programs whose funding
requirements represent 80 percent of total affected O&M costs and has
begun to identify estimated O&M costs for each of those programs. The
group plans to develop the capability to report O&M requirements to
Congress on a yearly basis. Eventually, the O&M estimates will be
included in SPIRE. This initiative has the potential to create more
realistic O&M baselines, which would increase baseline stability.

***4. Human Factors Process Group.*** This group was formed in 1997 in
reaction to the STARS design and development deficiencies related to
human factors engineering. In addition to the technical reports rendered
by the effort, the group published two management or process related
reports. One related to general practices and considerations that were
found to be obstacles to achieving a good human factors accommodation of
two separate but related and often confused issues within the context of
AMS (i.e., human factors engineering and user and union participation in
system acquisitions). The group summarized the problems in six major
findings and recommended five broad actions to improve human factors
processes within FAA. The recommendations focused on placing more
emphasis on human factors engineering (especially by acquiring more
human factors expertise at the IPT level) and identifying formal ways to
obtain user and union involvement in the acquisition process. As a
result of these recommendations, a follow-on group was tasked to develop
a specific action plan for implementing the recommendations. The
implementation group has completed work on this action plan and will
publish the results in the near future. If successful, we believe that
this initiative may address some of the causes identified in this
report.

***5.*** ***Process Action Group---Customer Service Process Team.***
This group was formed in 1999 as a result of Common ARTS problems with
user acceptance at a specific field site. The group conducted on-site
interviews with field personnel and used the data to validate and refine
the customer service process documented by PT members. The group found
that field personnel have been articulating problems, but headquarters
has not always listened because field personnel may not have been
considered "customers." It also concluded that headquarters personnel
have been more concerned about pursuing aggressive implementation
schedule deadlines than ensuring that new systems fully meet customer
needs and expectations. The group further concluded that headquarters
personnel do not sufficiently understand customers, including their work
environments, requirements and objectives.

As a result of the study, the Customer Service Process Team is in the
process of developing an improved customer service process derived from
a combination of industry and FAA best practices and recommendations
from field customers. It will emphasize listening to the customer and
involving customers in the process earlier. The new process should be
ready for implementation by Fall 2000. The team will use the Terminal
IPT as a pilot, then initiate changes in AMS and the integrated
Capability Maturity Model.

We believe that, if successful, this initiative has the potential to
address some of the causes identified in this report.

***6. Development and Approval of a Five-Year Budget.*** The JRC
recently approved the agency's first five-year F&E budget in an effort
to attempt to stabilize funding both internally and externally by
planning farther into the future. A clear plan for the future allows
players both internal and external to the FAA more insight into the
agency's priorities and intentions. We believe this is a great start,
but it is contingent on annual appropriation. Also, the budget does not
include O&M or R,E&D funds, which are necessary for stable baselines.

***7. Integrated Capability Maturity Model.*** The agency has developed
and is using the FAA integrated Capability Maturity Model as a unified
approach for evaluating its processes and improving them to reach
maximum efficiency. A major initiative is currently underway to increase
the maturity level of selected major software-intensive programs. Many
of these programs have achieved integrated Capability Maturity Model
Level 2. We believe that, if successfully implemented across FAA
acquisition programs, this process may address some of the causes
identified in this report.

***8. Baseline Acquisition Programs at Contract Award.*** This
initiative would amend AMS to allow acquisition programs to be baselined
just before a major contract is awarded rather than at investment
decision. The rationale for the proposed change is that considerable
cost, schedule, and technical uncertainty still exists at a program's
investment decision point, and some of that uncertainty should be
eliminated between investment decision and contract award. The change
would thus lead to more stable baselines once they are approved.

Critics of the initiative pointed out that investment decisions without
acquisition program baseline approval would assume little importance,
and that few programs have experienced variances before contract award.
However, the initiative appears to have considerable energy behind it
and has proceeded to the stage of coordination with OMB and Congress.

The team found that only one program we reviewed (Acquire) experienced a
variance before contract award. This three-month schedule variance
occurred because the program office was dissatisfied with bids it had
received and re-issued the solicitation. In addition, one program not
included in our review (NEXCOM) has recently reported a variance before
contract award.

While implementing this initiative could result in more stable baselines
once they are set, it does not address the causes identified in this
report.

***9. Consolidation of JRC Meetings to Coincide with the Budget
Process.*** A recent informal effort has been initiated to reduce the
number of JRC meetings held throughout the year. This initiative would
combine most program-related JRCs into three major meetings planned in
accordance with the yearly budget cycle. Programs would have the ability
to appear before the JRC between these mass meetings if necessary but
would be urged to plan needed decisions around these structured
meetings.

This initiative has the potential to mitigate the effects of internal
and external funding instability through implementation of a more
structured planning and decision-making process. JRC members would have
the benefit of the most current budget information before making crucial
program decisions. The consolidation of several programs into one
meeting would also assist the JRC in making trade-offs between programs
due to resource scarcity.

This initiative appears to be underway, with the first mass JRC planned
for March 2000. The team could find no documentation of this effort and
is unsure if any changes are needed to document and institutionalize
this change. We believe this initiative has the potential to mitigate
some of the effects of baseline instability.

***10. Risk Management Added to AMS.*** The January 2000 version of AMS
includes new guidance to the agency on risk management. The guidance
stresses risk assessment and mitigation throughout the lifecycle of
every program. Risk assessment should begin in mission analysis and
should continue through the in-service management phase of each program.
If implemented throughout the agency, this new guidance has the
potential to address some of the causes identified in this report.

## Conclusion

As noted above, there are many existing groups and ongoing initiatives
in the agency. Many of these are worthwhile and, in some cases, we have
endorsed their recommendations in this report. However, the team also
discovered that this proliferation of groups, initiatives, and other
process change efforts produced significant frustration throughout the
organization. In particular, personnel participating in the IPT process
expressed the belief that the staffing of these groups, while important,
hindered the ability of the IPTs to produce their primary products.
Moreover, some of the groups appeared to overlap in their efforts;
others produced recommendations that might have the effect of creating
more layers of management or additional groups for coordination. There
appeared to be no central coordination mechanism among the efforts.

# Trends and Observations

In addition to the conclusions reached in our three findings, the team
noted several broader trends and observations that cut across the
findings. We also noted that some of the initiatives listed in the Other
Matter section may address the causes of baseline instability. The
additional trends and observations are noted below.

-   Causes for FAA acquisition program baseline variances follow
    distinct patterns, indicating systemic problems within the agency.
    Variances reported since the June 1999 cutoff for this report appear
    to be following the same patterns, indicating that, without
    intervention, the agency can expect more such variances to occur in
    future programs.

-   Of 42 baselined programs, 28 were initiated prior to AMS. None of
    these programs were planned under the more rigorous AMS procedures.
    Seven of these programs have reported variances of more than 10
    percent through July 1999. Subsequently, four additional pre‑AMS
    programs have reported variances greater than 10 percent.[^17] These
    data indicate that, without intervention, variances may be expected
    in additional pre-AMS programs.

-   Program planning issues such as establishing a realistic baseline,
    selecting a workable acquisition strategy, and conducting sound risk
    analysis affected half the programs. Seven of the ten programs
    reviewed were undertaken prior to AMS's introduction of a more
    rigorous planning process, but two of the three programs initiated
    under AMS experienced the same problems.

-   Within the causes occurring in the planning phase of the acquisition
    lifecycle, a recurring pattern emerged relating to difficulties in
    planning for COTS/NDI acquisitions. Although AMS stated a preference
    for COTS/NDI as an acquisition strategy, there appeared to be no
    overall agency recognition of the cultural, programmatic, or funding
    implications of such a strategy. The agency has undertaken several
    initiatives to improve its management of COTS/NDI technology. These
    efforts appeared to be somewhat fragmented and uncoordinated.

-   Research into both Department of Defense (DoD) and FAA programs
    indicated that baseline stability may be related to size and
    complexity of program. Programs maintaining baseline stability
    tended to be smaller and less complicated. Strategies used in DoD
    and in the En Route IPT included segmentation of programs; that is,
    program baselines were developed in smaller sections. When one
    segment neared completion, the estimate for the next segment was
    performed. Other strategies included evolutionary acquisition and
    spiral development.

-   Unstable funding was a problem that appeared to escalate in the last
    two years as budgets tightened. Six of ten programs had variances
    caused in whole or part by funding instability.

-   Poorly defined or new requirements continued to be a problem for
    programs we reviewed. Seven of the ten programs were affected by the
    inability to maintain stable requirements. The effects of improved
    requirements identification processes implemented by FAA in the past
    two years were not reflected in programs we reviewed, since each was
    initiated prior to these changes.

-   Human factors issues were a factor in four of the ten programs.
    Three of these programs were initiated prior to AMS; one was
    initiated under AMS. New FAA measures have been and are being
    developed to introduce a more rigorous human factors approach into
    program planning and execution processes. Some of these measures
    have yet to be implemented.

-   Software development continued to be a problem for six of ten
    programs. This problem has been identified previously, and the FAA
    is taking action to document and improve software development
    processes. However, these changes appeared to have little effect on
    the programs we reviewed.

-   Contractor performance was a problem in six of ten programs. For
    large, complex programs, the FAA continued to award contracts to
    relatively few vendors, creating staffing conflicts. The recent
    report *Independent Assessment of the Federal Aviation
    Administration's Acquisition Management System* found that "a
    uniform process for collecting and applying past performance
    qualifications in source selections is not evident" in the FAA.[^18]
    This evaluation confirmed that the past performance database is
    currently not populated, and that plans to add data for past
    contracts are uncertain. Moreover, the team became aware that the
    National Institutes of Health have a comprehensive contractor past
    performance database that federal agencies can access for a fee. The
    FAA currently does not use that database.

-   Program management--related lessons learned do not appear to be
    communicated among IPTs within the FAA. The FAA's "Acquisition
    Lessons Learned" database contains some contract-related data, but
    very little on program management issues.

# Recommendations

Achieving baseline stability is not an easy task. FAA programs are quite
complicated, span multiple years, and involve millions of dollars. The
systems acquired by the agency are sometimes state-of-the-art and always
involve lots of uncertainties. Between the time alternatives are
analyzed and the decisions based on those analyses are implemented,
changes in the environment, particularly in the area of the
fast-emerging technology, may make the decisions outdated even before
they can be implemented. This phenomenon, coupled with program funding
being uncertain and not completely within the control of the agency,
makes achieving baseline stability for the agency's programs nearly
impossible.

Recognizing this difficult task, we have worked collaboratively with
multiple parties within and outside the agency to develop
recommendations that will address this important issue. We do not
purport that these recommendations will totally solve our baseline
stability challenges, but we believe that they will help the agency
focus on those areas most within our control.

1.  Because the agency is not provided sufficient funds to cover all the
    modernization it would like to implement, it must manage resources
    from a corporate standpoint and not on a program-by-program basis.
    Accordingly, we recommend that the FAA continue to develop an
    integrated approach to the agency's entire portfolio and implement
    that approach as soon as possible. The portfolio management approach
    initiated within the agency may be a viable solution. Whatever form
    such an initiative takes, it should provide for:

```{=html}
<!-- -->
```
a.  The integration of service providers and service
    acquirers/developers into consolidated organizations centered around
    designated services.

b.  A streamlined budget process that requires managers to prioritize
    all required resources (including acquisition programs required to
    provide the service) at appropriate levels. The process should
    include both F&E and O&M costs.

c.  A process by which the agency can prioritize programs and fund only
    those programs for which sufficient resources are available.

d.  Empowerment of service level management to make resource and
    personnel decisions within the designated service.

e.  Accountability of management to provide specific services at a given
    resource level.

```{=html}
<!-- -->
```
2.  The agency has stated its preference for COTS/NDI solutions, and
    these solutions come with a host of new challenges. Therefore, we
    recommend that, when the agency employs a COTS/NDI acquisition
    strategy, the FAA takes specific steps to align the acquisition
    planning, implementation, sustainment, and budgeting processes with
    that strategy. These steps must be undertaken across lines of
    business and include:

```{=html}
<!-- -->
```
a.  Establish a common understanding of COTS/NDI and CAS product
    characteristics, risks, and challenges throughout the agency.
    Strategies for achieving this common understanding could include
    forums, seminars, written materials, brown bags lunches, and formal
    training.

b.  Perform an assessment of the investment analysis process to
    determine ways to modify the process so that it is better suited to
    COTS/NDI solutions. Examples of changes could be: obtain cost models
    that consider COTS/NDI factors, ensure appropriate analysis time,
    factor in sustainment and technology refresh, and determine and
    document F&E and O&M funding trade-offs.

c.  Identify changes to business practices required when a COTS/NDI/CAS
    strategy is pursued. Areas affected include planning, requirements
    definition, budgeting, contracting, systems engineering, test and
    integration, configuration management, human factors, maintenance,
    logistics, and general agency practices.

d.  Develop policy, guidance, process changes, and other documentation
    necessary to promulgate this information within the agency.

The En Route IPT is recommending that a new group be established under
the Integrated Product Leadership Team to spearhead these and other
actions. In light of the considerable frustration over multiple efforts
currently underway in other process improvement areas, we hesitate to
recommend another group be formed at this time. However, the importance
of this issue merits immediate action. Therefore, we also recommend that
such a group be formed to facilitate implementation of the above
recommendations.

3.  Because customer service is critical to the agency fulfilling its
    mission, we recommend that the FAA continue its efforts to improve
    customer service in the development and delivery of equipment and
    services. Specific actions include:

```{=html}
<!-- -->
```
a.  Recognize that the field users constitute an intermediate "customer"
    for the IPTs, and document this understanding in agency policy and
    guidance.

b.  Develop a formal process for customer involvement throughout the
    product development and delivery process. The process should
    culminate with formal agreements between IPTs, field
    representatives, and involved unions.

This recommendation could be facilitated by the efforts of the Customer
Service Process Team.

4.  Because early consideration of human factors issues is critical to
    baseline stability, we recommend that the FAA continue its efforts
    to improve integration of human factors requirements into all stages
    of program planning and solution implementation. Specific actions
    include:

```{=html}
<!-- -->
```
a.  Establish appropriate measures and mechanisms to ensure that all
    Integrated Requirements Teams, Investment Analysis Teams, and IPTs
    have qualified human factors expertise to participate as integral
    members of the team during all stages of analysis and program
    planning and development.

b.  Amend AMS policy and/or guidance to clarify roles and
    responsibilities for human factors activities in all phases of the
    acquisition lifecycle (see *The Management of Human Factors in FAA
    Acquisition Programs*, March 31, 1998, recommendation #2, for more
    details).

Efforts to achieve these steps could be facilitated by the work of the
Human Factors Process Group.

5.  Because air traffic control systems take many years to reach
    fruition and the environment may change significantly during this
    time, we recommend that the FAA take aggressive measures to
    implement its new policy on risk assessment and mitigation,
    including:

```{=html}
<!-- -->
```
a.  Develop detailed guidance for risk assessment and mitigation
    planning and execution and include this guidance in the FAA
    Acquisition System Toolset.

b.  Ensure all responsible personnel are aware of the guidance and of
    the importance of its implementation in both new and existing
    programs. Possible mechanisms include incorporation into AMS
    training and refresher courses.

c.  Incorporate risk assessment and mitigation strategies into executive
    level reporting requirements.

The AMS assigns responsibility for development of risk mitigation
processes to the various lines of business. We endorse that designation
but also recommend that coordination and review of these processes
should be performed by one central organization (e.g., the Integrated
Product Leadership Team).

6.  Because of the uncertainty that surrounds large, complex programs
    and the inherent difficulty built into the development of baselines
    with these conditions, we recommend that FAA IPTs consider
    innovative ways of structuring and conducting acquisitions to reduce
    cycle time, minimize risk, and maximize product utility, including:

```{=html}
<!-- -->
```
a.  Baseline segments of programs as opposed to baselining the entire
    program at investment decision.

b.  Employ spiral development techniques.

c.  Employ evolutionary acquisition and simulation based acquisition
    tools.

```{=html}
<!-- -->
```
7.  Because contractor performance is critical to the success of our
    operations and the agency has had problems with contractors in the
    past, we recommend that the FAA assess past performance for all
    contractors and use it as a factor in contract award. Specific
    actions include:

```{=html}
<!-- -->
```
a.  Evaluate the utility and cost/benefits of using the National
    Institutes of Health past performance database versus populating and
    maintaining the FAA Past Performance database.

b.  Populate the selected past performance database with data from past
    contracts as soon as possible.

c.  Develop and promulgate guidance for populating the database with
    results from existing or future contracts when available.

d.  Develop and implement a training mechanism to make PTs aware of the
    importance of past performance as a criterion in contract awards.

The Office of Acquisitions (ASU) is the point of contact for the Past
Performance database, and could be assigned responsibility for these
actions.

8.  Because learning from our past experiences is critical to improving
    our operations and ensuring that we gain from those experiences, we
    recommend that FAA PTs share information among themselves, both
    formally and informally, on lessons learned throughout a program's
    lifecycle. Specific actions include:

```{=html}
<!-- -->
```
a.  Populate the existing Acquisitions Lessons Learned database.

b.  Disseminate information about the database throughout the FAA
    acquisition community.

c.  Develop and promote an environment in which employees understand the
    benefits and feel free to exchange lessons learned information. This
    could be accomplished through a concerted effort, supported by top
    management, to:

```{=html}
<!-- -->
```
1)  Develop a forum for sharing such information (lessons learned
    meetings, lectures, brown bags, etc.).

2)  Encourage and reward participation in such efforts.

ASU is currently responsible for the Acquisitions Lessons Learned
database. These specific steps, however, could be integrated into
development of the FAA Acquisition Workforce Learning System, currently
the responsibility of the Office of Business Management (ABZ).

9.  Because accurate and complete data is imperative to the agency's
    ability to monitor and assess programs, we recommend that the FAA
    improve its management of program documentation, including:

```{=html}
<!-- -->
```
a.  Ensure that all required program documentation resides in the
    Document Control Center. The NAS Configuration Management and
    Evaluation Staff (ACM) is currently assigned responsibility for this
    task.

b.  Designate SPIRE as the central repository for electronic versions of
    FAA documentation. The NAS Programming and Financial Management
    Division (ASD-300) is assigned responsibility for SPIRE development
    and maintenance.

c.  Ensure that electronic versions are consistent with hard copies
    maintained in the Document Control Center.

```{=html}
<!-- -->
```
10. Because establishing and maintaining baselines is critical to the
    agency's credibility, we recommend that the FAA develop and
    implement a consistent policy on rebaselining programs. Specific
    actions include:

```{=html}
<!-- -->
```
a.  Clarify the role that re-baselining plays in the acquisition
    management process. This could be accomplished by ASD-300.

b.  Develop criteria and a process for re-baselining requests, and
    ensure they are followed. The first steps in identifying criteria
    and process have been taken by ASD-300 in the draft update to the
    I-BEAM document. These efforts should continue.

c.  Establish a review mechanism below the JRC to determine when a
    re-baseline request is appropriate. This could be accomplished by
    the FAA Acquisition Executive Advisory Board (FAB).

```{=html}
<!-- -->
```
11. Because of the proliferation of groups and initiatives within the
    agency that appear to be uncoordinated, we recommend that the FAA
    appoint a central mechanism within the agency to coordinate all
    groups, initiatives and efforts. This mechanism would ensure that
    groups are properly chartered, are serving a useful purpose, and do
    not duplicate or overlap with established groups. This mechanism
    could reside with the Associate Administrator for Information
    Services (AIO).

# Appendix A---Details for Causes of Baseline Instability

This appendix provides program by program details for the nine
documented causes of baseline instability.

***1. Poorly defined or new requirements***

The NIMS program did not initially understand requirements, and did not
have agreement on requirements and program philosophy from the user
prospective. The program used a spiral development process that
necessitated refining requirements on an ongoing basis. Although the
team had early customer involvement and adequate sponsor support, the
requirements were not identified at a level that facilitated fully
informed customer agreement. The program plan did not have the means to
quantify the evolution of requirements and the resulting impact on cost
and schedule baselines. More development was required than initially
anticipated, since COTS products did not fully meet the program
requirements. Additionally, information security (INFOSEC) requirements
were not well defined in the operational requirements document,
resulting in a later cost increase.

For the STARS, OASIS, and AMASS programs, the primary driver for
requirements changes was human factors requirements that were not
subjected to the appropriate level of analysis during the planning,
design, and development stages. See the discussion of human factors
below for more details. Additionally, the STARS program encountered new
requirements in several additional areas ranging from site adaptation to
conversion of remote towers to Terminal Radar Approach Control
Facilities (TRACONs). The OASIS program experienced increased costs due
to a requirement to purchase additional hardware. AMASS program costs
increased due to revised requirements for initial lay-in of spare parts
at both the depot and site levels.

Due in part to the extreme technical complexity of the program, the WAAS
program underestimated technical requirements, such as the need for
redundancy in the system's ground components. The program also failed to
document operational requirements, including the need to certify system
software, contributing to increases in both cost and schedule baselines.

The addition of a new requirement for additional Automatic Dependent
Surveillance way point reporting necessary to standardize procedures
with other nations' Civil Aviation Authorities caused a 10.5 percent
variance in schedule for the Oceanic Program (BMN #001). This added
requirement was later deleted when contractor performance issues led the
Automatic Dependent Surveillance to be deleted from the baseline,
descoping Oceanic Build 1 to Early Drop.

The WARP program initially established requirements for organic
maintenance of the system. However, Congressional and FAA funding cuts
led to an agreement with the contractor to reduce costs by eliminating
the organic maintenance requirements. Subsequently, the requirement was
re-introduced into the program, increasing costs by \$6.2 million.
Additionally, a new requirement was added to the program to add a Free
Flight Phase I interface to the FAA Bulk Weather Telecommunications
Gateway. These additional requirements contributed to the cost and
schedule variances experienced by the program.

2.  ***Human Factors Issues***

For the STARS and OASIS programs, the human factors issues were not
explored because the FAA had decided to adopt a policy of acquiring
COTS/NDI systems whenever possible, and the agency assumed the risk
associated with resolving at the time of deployment any deficiencies
that existed in the solution provided by the market place. Completion of
the changes resulted in increases to both cost and schedule for both
programs.

The AMASS program did not acquire a COTS/NDI system but still failed to
explore early indications of human factors deficiencies. The PT
negotiated a set of required changes, and implementation of these
changes resulted in increases to both cost and schedule.

The Common ARTS program encountered difficulties in the deployment phase
resulting from deployment of a software "national release" that was not
compatible with site-specific adaptations at a particular site. When the
software was introduced, unforeseen software glitches rendered the
software unsuitable to the users, resulting in a schedule delay for the
program while the fixes were completed.

***3. Unsuitable Acquisition Strategy***

The AMASS program elected to pursue concurrent development and
production phases, which resulted in a longer full scale development
period with associated increased costs. The strategy also contributed to
problems as a result of operational testing, causing further delays.

The NIMS program chose to conduct business process reengineering and
product acquisition simultaneously, using a spiral development approach.
This approach greatly increased software development risks for the
program and contributed to cost and schedule delays.

Both the STARS and OASIS programs chose to employ a COTS/NDI strategy,
in keeping with the guidance in AMS. However, the programs did not fully
understand the implications and risks of a COTS/NDI strategy and did not
take adequate measures to address those risks. For example, the programs
did not obtain field agreement that a commercially available system
would be acceptable, nor did they correctly estimate the costs
associated with COTS/NDI. This led to significant cost and schedule
delays for both programs.

***4. Unrealistic Baseline***

For the WAAS program, the FAA elected to present unrealistic cost and
schedule baselines, stating that program costs would be held to \$500
million even though internal cost estimates projected much higher
costs.[^19] Subsequent cost estimates did not include the costs of
leasing required for geostationary satellites. Further, the program cost
estimations did not use past performance criteria to refine the accuracy
of the cost information. The initial schedule was also unrealistic,
since it included concurrent software development and system acceptance
processes, which were not possible in practice. The WAAS program
experienced significant cost and schedule delays.

The original STARS program cost and schedule baseline was based on a
COTS product, and the schedule estimate compressed an estimated 32
months of development and testing into 25 months. When it became
apparent that a COTS product would not meet the requirements, a new
baseline was established. However, according to PT personnel, the
subsequent schedule of first ORD by December 1998 was not changed,
although the PT network logic schedules indicated that December 1999
would be a more realistic milestone. The PT did not meet the December
1998 milestone.

In an effort to meet politically driven milestones, the AMASS program
developed a schedule that presented concurrent development and
production phases. The schedule was unrealistic and resulted in
significant cost and schedule increases (see above discussion on
unsuitable acquisition strategy).

The NIMS program was the first to undergo an affordability analysis
under AMS. However, the assessment was forced to fit a predetermined
funding level, reducing the total funding from \$140 million to \$108
million. The performance baseline was not adjusted to reflect the
reduced funding. Thus, the baseline presented in the NIMS acquisition
program baseline document was unrealistic and was soon breached.

***5. Inadequate Risk Assessment***

Since WAAS was a pre-AMS program, an investment analysis was not
initially performed. The PT did not fully understand the technical,
institutional, and programmatic risks involved with the program before
the work began. The investment analysis conducted when WAAS was
rebaselined in January 1998 underestimated the risk of selecting another
government agency to acquire the geo-stationary satellites. This
understatement of risks contributed to both cost and schedule variances.

The NIMS program underestimated the risks of several programmatic
assumptions. The program assumed that prototype NIMS software could be
used directly for NIMS with changes, that operational control center
(OCC) systems would not be fully integrated, and that a minimal level of
security would be required. The risks in making these assumptions were
not independently assessed. The assumptions proved to be unrealistic,
resulting in cost and schedule variances.

Although these were the only documented cases of inadequate risk
analysis per se, the two other programs that used a COTS/NDI strategy
(STARS and OASIS) could also be placed in this category. Neither program
understood the risks of electing to acquire a COTS/NDI system to satisfy
the requirements.

***6. Software Development Difficulties***

The WARP program experienced delays in contractor development and
integration of software that resulted in a four-month slip to the
program.

The AMASS program experienced difficulties in software development that
delayed the test schedule by seven months and increased program costs.

The STARS program experienced software development difficulties,
including underestimation of the lines of code required to meet user
requirements and difficulties in contractor staffing to meet project
milestones.

The WAAS program experienced software development delays in both design
and integration, which resulted in non-delivery of critical software,
increasing cost and schedule.

***7. Poor Contractor Performance***

The AMASS contractor failed to meet full-scale development milestones
due to software development and integration problems. This inability to
deliver resulted in deferment of the test schedule, resulting in cost
increases and schedule delays for the program.

The Oceanic Build I program contractor was unable to deliver a key
component of the system (automatic dependent surveillance), forcing the
program to de-scope the requirements, removing the automatic dependent
surveillance component from the performance and benefits baseline. The
result was a significant decrease in benefits and performance for the
program.

The WARP and STARS program contractors experienced software development,
integration, and testing difficulties. Additionally, the STARS program
contractor was initially unable to staff the project adequately,
resulting in an inability to meet program milestones.

The initial WAAS program contractor failed to meet required milestones,
resulting in a termination of the contract. The project was awarded to a
second contractor, which also experienced technical difficulties in
software development and integration.

According to PT personnel, the Common ARTS contractor was insufficiently
staffed to meet program milestones due to removal of key personnel to
complete a bid on a larger FAA contract. The resulting delays resulted
in a restructuring of the contract from a cost-plus-fixed-fee to a
firm-fixed-price, causing a schedule variance.

***8. Funding Instability***

*Internal Funding Instability*. The BUEC program purchased equipment to
replace obsolete backup emergency communications equipment now installed
in the field. The equipment resides in warehouses; it will be removed
and installed by FAA personnel as resources permit. The program
experienced significant FAA-imposed funding cuts in both FY98 and FY99,
including a reduction from \$17.7 million to \$1.6 million in FY
99.[^20] The equipment remains in storage; the warranty expires in 2006.

The NIMS program began the solution implementation phase with a \$32.2
million shortfall imposed by the affordability assessment (see above
discussion on unrealistic baselines). In addition, the program
experienced \$13.6 million of internal cuts from FY 97--FY99. These
reductions, added to the external cuts that will b discussed below,
contributed significantly to NIMS cost and schedule variances.

Three programs (OASIS, WARP, and STARS) experienced cost growth due to
internally-imposed requirements to purchase equipment or services with
the program's F&E funds that were not originally intended to be a part
of the program requirements. These requirements arose because of funding
shortfalls in both F&E and O&M accounts. These three were the only
documented cases of this problem from our review. However, during the
interviews, several additional PTs expressed concerns about this issue
and indicated that additional impacts may be expected in the future.

The OASIS program variance was caused in part by the additional
requirement to replace flight service station consoles. BMN #035
reported a variance of \$15.8 million to replace flight service
automated system consoles "that were designed with little or no regard
for human factors. The flight service station console human interface
requirements are not part of the OASIS baseline." Replacing instead of
modifying existing consoles caused a variance to the OASIS F&E costs.

According to BMN #056, the WARP program experienced a \$16.8 million
cost growth stemming from a requirement "to accommodate
telecommunications funding shortfall due to O&M funding reduction in
FY99 and beyond."

The STARS program currently faces an issue in which facility upgrade
funds may be allocated to the STARS program due to F&E funding
instability in the NAS Transition and Implementation Directorate (ANS).
Many of these upgrades are required completely independently of the
STARS program requirements. If these additional requirements are
incorporated into the baseline, STARS costs will increase
significantly.[^21]

*External Funding Instability*. The BUEC program, already discussed
above, suffered a \$2.92 million cut from Congress in FY00 that
increased the program's inability to maintain its schedule.

In addition to the internal reductions mentioned above, the NIMS program
experienced \$12.7 million in Congressional cuts from FY98--FY00, and a
\$14.2 million cut from OMB in FY00.

Congress reduced the OASIS program funding by \$11.5 million in FY00.
The program, rebaselined for other reasons in May 1999, will return to
the JRC to request another rebaselining action. The PT does not believe
it can operate under the current APB due to this large funding
reduction.

The WAAS program has experienced several large external funding cuts,
including a substantial Congressional reduction in FY00. These
reductions were at least partially the result of the FAA's failure to
meet Congressional deadlines for reporting.[^22]

***9. Interdependencies***

The WARP program experienced variances as a result of interdependencies
with several FAA programs, including delays in Next Generation Weather
Radar (NEXRAD) testing, unavailability of the Display System Replacement
interface for testing, and National Airspace Data Interchange Network
(NADIN) telecommunications performance issues. The NADIN system could
not accommodate the WARP bandwidth requirements, and the subsequent
redesign caused a schedule delay and a cost increase for the WARP
program.

The Oceanic Build 1 program experienced a 16.3 percent schedule variance
due to resource issues at field facilities. The New York Air Route
Traffic Control Center lacked sufficient time and resources to support
the target date for Initial Operating Capability, due to higher
priorities for Host and Oceanic Computer System Replacement (HOCSR) and
Display System Replacement implementation.

# Appendix B---Reports Addressing Baseline Issues

During the course of our evaluation, we noted numerous reports that
addressed baseline and related issues. These reports are listed below.

+-------------+--------------------------------------+--------+--------+
| **Source**  | **Report Title**                     | **Re   | **Re   |
|             |                                      | port** | port** |
|             |                                      |        |        |
|             |                                      | **     | **Nu   |
|             |                                      | Date** | mber** |
+=============+======================================+========+========+
| ACM-10      | Evaluation of FAA Acquisition        | May-97 | N/A    |
|             | Reform, The First Year: April        |        |        |
|             | 1996-March 1997                      |        |        |
+-------------+--------------------------------------+--------+--------+
| ACM-10      | Evaluation of FAA Acquisition        | May-98 | 1      |
|             | Reform, The First Two Years: April   |        | 998-01 |
|             | 1996-March 1998                      |        |        |
+-------------+--------------------------------------+--------+--------+
| ACM-10      | Evaluation of FAA Acquisition        | May-99 | 1      |
|             | Reform, The First Three Years: April |        | 999-04 |
|             | 1996-March 1999                      |        |        |
+-------------+--------------------------------------+--------+--------+
| ACM-10      | FAA Acquisition Program Baseline     | Sep-99 | 1      |
|             | Stability: Comparison Between Growth |        | 999-05 |
|             | Rates for Programs Conducted Prior   |        |        |
|             | to Acquisition Reform and Those      |        |        |
|             | Conducted Under Acquisition Reform   |        |        |
+-------------+--------------------------------------+--------+--------+
| ASD-300     | Baseline Management Variance Report  | Apr-98 | N/A    |
|             | April 1998                           |        |        |
+-------------+--------------------------------------+--------+--------+
| ASD-300     | Baseline Management Variance Report  | Dec-99 | N/A    |
|             | December 1998                        |        |        |
+-------------+--------------------------------------+--------+--------+
| ASD-300     | Baseline Management Variance Report  | Jun-99 | N/A    |
|             | June 1999                            |        |        |
+-------------+--------------------------------------+--------+--------+
| ASD-300     | Baseline Management Variance Report  | Nov-99 | N/A    |
|             | November 1999                        |        |        |
+-------------+--------------------------------------+--------+--------+
| ASU-100     | Effectiveness of COTS Solutions      | Feb-99 | N/A    |
+-------------+--------------------------------------+--------+--------+
| Booz-Allen  | Independent Assessment of the FAA's  | Sep-97 | N/A    |
| & Hamilton  | Acquisition Management System        |        |        |
+-------------+--------------------------------------+--------+--------+
| Booz-Allen  | Independent Assessment of the        | Jul-99 | N/A    |
| & Hamilton  | Federal Aviation Administration\'s   |        |        |
|             | Acquisition Management System        |        |        |
+-------------+--------------------------------------+--------+--------+
| Customer    | Common ARTS Customer Service:        | Sep-99 | N/A    |
| Service     | Findings from FAA Field Personnel    |        |        |
| Process     | Interviews                           |        |        |
| Team        |                                      |        |        |
+-------------+--------------------------------------+--------+--------+
| DOT OIG     | FAA's Standard Terminal Automation   | Nov-97 | AV-19  |
|             | Replacement System (STARS)           |        | 98-012 |
+-------------+--------------------------------------+--------+--------+
| DOT OIG     | Air Traffic Control Modernization    | Mar-98 | AV-19  |
|             |                                      |        | 98-089 |
+-------------+--------------------------------------+--------+--------+
| DOT OIG     | Wide Area Augmentation System        | May-98 | AV-19  |
|             |                                      |        | 98-117 |
+-------------+--------------------------------------+--------+--------+
| DOT OIG     | FAA Financing and Cost Control       | Mar-99 | AV-19  |
|             |                                      |        | 99-066 |
+-------------+--------------------------------------+--------+--------+
| DOT OIG     | Air Traffic Control Modernization    | Mar-99 | AV-19  |
|             |                                      |        | 99-065 |
+-------------+--------------------------------------+--------+--------+
| FAA         | Acquisition Management System (AMS)  | Sep-99 | N/A    |
| Internal    | Schedule and Cost Performance        |        |        |
| Report -    |                                      |        |        |
| ASU-1       |                                      |        |        |
+-------------+--------------------------------------+--------+--------+
| FAA         | STARS Human Factors Management       | Dec-97 | N/A    |
| Internal    | Process                              |        |        |
| Report -    |                                      |        |        |
| STARS Human |                                      |        |        |
| Factors     |                                      |        |        |
| Process     |                                      |        |        |
| Group       |                                      |        |        |
+-------------+--------------------------------------+--------+--------+
| FAA         | The Management of Human Factors in   | Mar-98 | N/A    |
| Internal    | FAA Acquisition Programs             |        |        |
| Report -    |                                      |        |        |
| STARS Human |                                      |        |        |
| Factors     |                                      |        |        |
| Steering    |                                      |        |        |
| Committee   |                                      |        |        |
+-------------+--------------------------------------+--------+--------+
| GAO         | DOT's Budget-Management and          | Feb-88 | T      |
|             | Performance Issues Facing the        |        | -RCED- |
|             | Department in Fiscal Year 1999       |        | AIMD   |
|             |                                      |        | -98-76 |
+-------------+--------------------------------------+--------+--------+
| GAO         | National Airspace                    | May-95 | RCED   |
|             | System-Comprehensive FAA Plan for    |        | -95-26 |
|             | Global Positioning System is Needed  |        |        |
+-------------+--------------------------------------+--------+--------+
| GAO         | DOT's Budget-Challenges Facing the   | Mar-96 | N/A    |
|             | Department in Fiscal Year 1997 and   |        |        |
|             | Beyond                               |        |        |
+-------------+--------------------------------------+--------+--------+
| GAO         | Aviation Acquisition-A Comprehensive | Aug-96 | RCED-  |
|             | Strategy is Needed for Cultural      |        | 96-159 |
|             | Change at FAA                        |        |        |
+-------------+--------------------------------------+--------+--------+
| GAO         | Air Traffic Control-Improved Cost    | Jan-97 | AIMD   |
|             | Information Needed to Make Billion   |        | -97-20 |
|             | Dollar Modernization Investment      |        |        |
|             | Decisions                            |        |        |
+-------------+--------------------------------------+--------+--------+
| GAO         | Air Traffic Control -- Immature      | Mar-97 | AIMD   |
|             | Software Acquisition Processes       |        | -97-47 |
|             | Increase FAA System Acquisition      |        |        |
|             | Risks                                |        |        |
+-------------+--------------------------------------+--------+--------+
| GAO         | National Airspace                    | Oct-97 | T-RCED |
|             | System-Observations on the Wide Area |        | -98-12 |
|             | Augmentation System                  |        |        |
+-------------+--------------------------------------+--------+--------+
| GAO         | Air Traffic Control-Observations on  | Feb-98 | T      |
|             | FAA's Modernization Program          |        | -RCED- |
|             |                                      |        | AIMD   |
|             |                                      |        | -98-93 |
+-------------+--------------------------------------+--------+--------+
| GAO         | Air Traffic Control-Evolution and    | Mar-98 | T      |
|             | Status of FAA's Automation Program   |        | -RCED- |
|             |                                      |        | AIMD   |
|             |                                      |        | -98-85 |
+-------------+--------------------------------------+--------+--------+
| GAO         | National Airspace System-Status of   | Apr-98 | RCED   |
|             | Wide Area Augmentation System        |        | -98-79 |
|             | Project                              |        |        |
+-------------+--------------------------------------+--------+--------+
| GAO         | Air Traffic Control -- Status of     | Dec-98 | RCED   |
|             | FAA's Modernization Program          |        | -99-25 |
+-------------+--------------------------------------+--------+--------+
| GAO         | Air Traffic Control -- FAA's         | Apr-99 | RCED   |
|             | Modernization Investment Management  |        | -99-98 |
|             | Approach Could Be Strengthened       |        |        |
+-------------+--------------------------------------+--------+--------+
| National    | Avoiding Aviation Gridlock and       | Dec-97 | N/A    |
| Civil       | Reducing the Accident Rate-A         |        |        |
| Aviation    | Consensus for Change                 |        |        |
| Review      |                                      |        |        |
| Commission  |                                      |        |        |
+-------------+--------------------------------------+--------+--------+

# Appendix C---Acronyms

ACM FAA's NAS Configuration Management and Evaluation Staff

AMASS Airport Movement Area Safety System

AMS Acquisition Management System

APB Acquisition Program Baseline

ARA FAA's Office of Research and Acquisitions

ASU FAA's Office of Acquisitions

ATS FAA's Office of Air Traffic Services

BMN Baseline Management Notice

BUEC Backup Emergency Communications

CAS Commercially Available Software

CIP Capital Investment Plan

Common ARTS Common Automated Radar Terminal System

COTS/NDI Commercial Off-The-Shelf/Non-Developmental Item

DOT OIG Department of Transportation Office of Inspector General

FAA Federal Aviation Administration

FY Fiscal Year

F&E Facilities and Equipment

GAO General Accounting Office

I-BEAM Integrated Baseline Establishment and Management

IPT Integrated Product Team

JRC Joint Resources Council

NAS National Airspace System

NEXCOM Next Generation Air/Ground Communication System

NIMS NAS Infrastructure Management System

O&M Operations and Maintenance

OASIS Operational and Supportability Implementation System

OMB Office of Management and Budget

ORD Operational Readiness Demonstration

PT Product Team

R,E&D Research, Engineering, and Development

SEOAT Systems Engineering/Operational Analysis Team

SPIRE Simplified Program Information Reporting and Evaluation

STARS Standard Terminal Automated Replacement System

WAAS Wide Area Augmentation System

WARP Weather and Radar Processor

[^1]: In December 1999, the PT determined that the cost variance was
    146.76%.

[^2]: These programs were the Low-Level Windshear Alert System (LLWAS),
    the Airport Surveillance Radar (ASR-11) program, the Voice Recorder
    Replacement Program (VRRP), the Airport Surface Detection Equipment
    (ASDE) program, and the Next Generation Air-Ground Communication
    System (NEXCOM).

[^3]: Note that our sample lists seven pre-AMS programs with variances.
    One of these, Weather and Radar Processor (WARP), was not included
    in the June 1999 Variance Report. The WARP variance was reported in
    SPIRE in July 1999.

[^4]: The sample of 10 programs includes one additional program that
    reported a variance in July 1999.

[^5]: Although 42 programs resided in the FAA tracking system, only 34
    were reporting on a monthly basis at the time of this evaluation.

[^6]: This issue is closely related to human factors issues. Several of
    the programs experienced both of these problems.

[^7]: These programs did complete a Cost Benefit Analysis.

[^8]: Some FAA internal sources have recognized these risks. See "A Risk
    Management Approach for COTS/CAS Based Air Traffic Control Systems
    Acquisition and Sustainment." Presentation for National Aeronautics
    and Space Administration 2^nd^ Annual Workshop on Risk Management,
    October 28, 1999.

[^9]: *Immature Software Acquisition Processes Increase FAA System
    Acquisition Risks*, GAO/AIMD-97-47, March 1997.

[^10]: In some, cases, program funding was deferred until later years.
    These deferrals did not change the program's estimate at completion
    but still affected both cost and schedule.

[^11]: Note that five programs are affected by internal and four by
    external funding problems. However, several of these programs
    overlap, creating a total of six affected by funding overall.

[^12]: For example, see GAO Report RCED-96-159, *A Comprehensive
    Strategy for Culture Change is Needed at FAA*, August 1996; National
    Civil Aviation Review Commission, *Avoiding Aviation Gridlock and
    Reducing the Accident Rate-A Consensus for Change,* December 1997;
    GAO Report RCED-99-25, December 1998; STARS Human Factors Process
    Group, *STARS Human Factors Management Process,* December 8, 1997;
    and STARS Human Factors Steering Committee, *The Management of Human
    Factors in FAA Acquisition Programs,* March 31, 1998.

[^13]: See GAO Report RCED-96-159 and *The Management of Human Factors
    in FAA Acquisition Programs* for further evidence about this
    problem.

[^14]: The OASIS program has requested a second rebaseline as a result
    of Congressional budget cuts.

[^15]: This is the APB from which the agency is tracking WAAS costs.
    However, the original WAAS baseline was defined in 1994.

[^16]: Rebaseline was approved by the JRC pending final cost estimates.

[^17]: These variances are documented in the November 1999 *Baseline
    Management Variance Report*, and include changes to the LLWAS,
    ASR-11, VRRP, and ASDE programs.

[^18]: The report, dated June 1999, was prepared for the FAA by
    Booz-Allen & Hamilton, Inc.

[^19]: GAO Report: *DOT's Budget Management and Performance Issues
    Facing the Department in Fiscal Year 1999*, T-RCED/AIMD-98-76,
    February 12, 1998. We also confirmed the accuracy of this statement
    with program personnel.

[^20]: In FY00, the program was cut by Congress for the first time. See
    discussion on external cuts below.

[^21]: According to the PT, this decision is awaiting more concrete
    estimates of STARS operations costs.

[^22]: This information was initially gleaned from PT personnel, and was
    confirmed by Congressional language.