###### Evaluation of 

**FAA Acquisition Reform**

**The First Three Years:**

**April 1996 -- March 1999**

**Acknowledgments**

The data gathering and analysis activities that provided the basis for
this document were performed by members of the Program Evaluation Branch
with the assistance and support of other individuals. The members of
this evaluation team, which included FAA headquarters and regional
personnel and FAA support contractor personnel, were responsible for
planning the evaluation, executing the plan, and reporting the results.

The team would like to thank all who provided the many supporting
documents for our analysis and who participated in interviews and
surveys for their time and thoughtful comments. The evaluation team also
extends special thanks to those high level officials who reviewed the
various drafts of the report. These individuals represented the
following offices or lines of business:

-   Chief Counsel,

-   System Safety,

-   Air Traffic Services,

-   Civil Aviation Security,

-   Regulation and Certification, and

-   Research and Acquisitions.

# Executive Summary {#executive-summary .Part}

# Objective

The Congress granted FAA acquisition reform and mandated that FAA
develop an acquisition system that provides for more timely and
cost-effective acquisition of equipment and materials. The focus of
acquisition reform was on mission accomplishment, rather than compliance
with rules and processes. In response to this congressional mandate, FAA
developed the Acquisition Management System (AMS), which lays out the
agency's policy for all acquisitions.

The Program Evaluation Branch conducted this and two previous
evaluations of acquisition reform. The first was on the implementation
of the AMS policy, procedures, and infrastructure; the second was on
those key AMS processes, systems, and decision making bodies that were
critical to the success of acquisition reform. For this third
evaluation, the FAA Acquisition Executive tasked the Program Evaluation
Branch to review primarily how the agency is doing since acquisition
reform. While it is recognized that there are still improvements to be
made to AMS, that was not the focus of this evaluation. Instead, the two
main objectives were to determine (1) if, under acquisition reform, the
agency's procurement efforts were achieving faster awards, were
achieving competition and were meeting Major Procurement Program Goals;
and (2) if, under acquisition reform, the agency's programs were on
track to achieve success.

# Findings

Overall, the evaluation team found that the agency's procurement efforts
were achieving faster awards, were achieving competition, and were
meeting small business goals; however, FAA was failing to meet its goals
for awarding contracts to small business concerns owned and controlled
by socially and economically disadvantaged individuals. In general, the
evaluation team found that the agency's program results were on track to
achieve success in terms of ensuring programs support the FAA mission,
deliver planned product performance results, and meet customer needs,
but were not on track to meet cost and schedule baselines approved for
individual programs.

The evaluation team tried to measure whether there was a reduction in
the time and cost to acquire programs, but found that sufficient
comparable data were not available to reach a solid conclusion. This was
due to the nature of FAA's acquisitions, which can span over several
years, and the fact that FAA has been under acquisition reform for too
short a period of time to generate comparable data. Industry
improvements on the scale of acquisition reform can take up to seven
years to realize benefits, and FAA programs only came under acquisition
reform processes after April 1996. As a result, the agency is probably
four years away from determining whether acquisition reform has truly
been successful.

## Procurement Results

While procurement is only one part of an acquisition's lifecycle, it is
the acquisition area where the agency receives most of its outside
interest. As a part of the procurement review, the evaluation team first
analyzed the time to award contracts for a random sample of contracts
valued over \$100,000 and found that, under acquisition reform, the
average time to award contracts decreased approximately 55 percent. The
primary reasons for this dramatic improvement were the exemption from
the Federal Acquisition Regulation, which included a mandatory length of
time for a *Commerce Business Daily* announcement, and the requirement
that the government consider all offerors. Ultimately, the reduced
procurement time should contribute to a reduction in the overall
acquisition lifecycle.

Second, the team analyzed all contracts greater than \$25,000. This
revealed that there has been little change in competition since
acquisition reform and that most contract dollars were awarded through
competition. In fact, under acquisition reform, the percentage of
competitively awarded contracts increased slightly.

Finally, the evaluation team analyzed the Major Procurement Program
Goals for fiscal years 1997 and 1998. This indicated that, although FAA
was achieving its overall small business goal, it was not meeting the
other disadvantaged business and women owned business goals. However,
this finding needs be viewed with the understanding that some of FAA's
goals and all of FAA's accomplishments were higher than the national
goals and accomplishments. Generally, the evaluation team found that
some FAA personnel involved in the acquisition process were not aware of
the Major Procurement Program Goals and did not pursue making contract
awards to meet these goals. Also, the primary reason FAA did not meet
its SEDB award goals was tied to the fact that, prior to acquisition
reform, the principal method used for making SEDB awards was through
noncompetitive set-aside contracts. This allowed a faster procurement
award process, which encouraged SEDB awards. Acquisition reform
eliminated this noncompetitive set-aside contracting option and, thus,
the major incentive to make SEDB awards. The agency needs to improve
awareness of the Major Procurement Program Goals among all acquisition
personnel and develop some incentives for acquisition personnel to make
awards to disadvantaged businesses and women owned businesses.

## Program Results

The evaluation team reviewed a judgmental sample of 13 programs for
results to determine if the agency's programs, since acquisition reform,
were on track to achieve success. Results were evaluated in terms of
five parameters: supporting the FAA mission, delivering planned product
performance, meeting customer needs, meeting cost baselines, and meeting
schedule baselines. Ten of the programs reviewed were facilities and
equipment funded programs. Three were research, engineering, and
development funded programs, but inclusion of these three programs in
the conclusions was not possible in all cases. While most of the 13
programs were initiated before acquisition reform and most were too
early in the process to be completed, they were managed, at least in
part, under acquisition reform.

The evaluation team reviewed program results within the five parameters
and found that all sampled programs were consistent with the agency's
mission and long range strategic planning, and that 90 percent had
delivered or were on track to deliver planned product performance
results. In meeting customer needs, the agency was 70 percent
successful, with customer being defined as those both inside and outside
the FAA who will use or benefit from agency programs. This success may
be attributed in part to the broad engagement of customers in the
planning process. However, it is in the last two results, meeting cost
and schedule baselines, that the team found the agency is not
successful. Specifically, only 40 percent of the programs evaluated were
meeting cost baselines, and only 20 percent were meeting schedule
baselines.

Cost and schedule baselines are established to measure how the agency is
doing in meeting the planned budget and timeframes for acquiring the
product performance of the program. A determination of whether the
approved cost and schedule baselines were realistic was not made because
this evaluation was not intended to be an in-depth review of specific
programs. While the poor performance in meeting cost and schedule
baselines may not have resulted in a program failure, there were adverse
impacts. First, the agency was unable to deliver *all* products or
product performance and unable to meet *all* initially defined customer
needs. Next, there was a negative impact on other programs due to the
interdependency of FAA programs, and finally, a program with severe cost
and schedule baseline problems could come under consideration for
cancellation.

The evaluation team found there were three systemic problems that
contributed to growth in cost or schedule baselines. The first systemic
problem was the addition of new requirements or modification of the
initial program requirements to accommodate a new understanding. Some of
these requirement changes resulted from information obtained during
testing of the product. While requirement changes present problems, this
is not an activity that necessarily should be avoided. In a world with a
rapidly changing environment, it would be unreasonable to expect the
agency to maintain rigid requirements for a program planned to be
implemented in a decade or more. The second systemic problem was
technical complexity, which included software development and interface
issues and technical challenges related to commercial off‑the‑shelf
systems. The last systemic problem identified was funding shortfalls
that resulted from the initial funding levels either being inadequate or
becoming inadequate due to subsequent cuts. These funding shortfalls
occurred primarily because the budgeting process does not complement the
AMS lifecycle baseline process. The agency needs to take steps to reduce
the risk of unexpected requirements, especially those related to
software and interface issues and human factors processes, and to
mitigate the negative effects of budget cuts on programs.

## Recommendation Follow-up

In four previous evaluation reports related to acquisition reform, there
have been 46 recommendations made to management. In its review of the
status of these recommendations, the evaluation team found that
13 recommendations were completed, 30 were in progress, and 3 had no
action taken. While the agency is moving in the right direction in
implementing these recommendations, improvement is necessary. The two
previous evaluations conducted by the Program Evaluation Branch
specifically focused on AMS and made recommendations to improve the
processes within AMS. However, many of these recommendations have not
yet been completed. The uncompleted recommendations should be
scrutinized to ensure all possible efforts toward addressing the
recommendations are accomplished.

During this and previous evaluations, agency officials stated additional
improvements are needed to AMS. The evaluation team agrees. The agency
should begin improvement efforts by addressing the uncompleted
recommendations.

## Recommendations in this Report

The recommendations in this report were designed to highlight areas that
needed to be addressed by FAA management. The intent was not to dictate
specific solutions in the recommendations themselves, but rather to
allow the action officials flexibility in deciding the best approach to
solving a problem, since those officials best know their operations and
other factors that need to be considered. However, based on the
evaluation work, specific suggestions were included within this report
that management may find useful when addressing the recommendations. Any
solution by management that corrects the problems identified in the
findings would be acceptable. The ultimate goal is to ensure that
acquisition reform is successful. The Program Evaluation Branch will
track the recommendations with management, but the suggestions will not
be tracked.

In this report, the evaluation team recommends:

-   The Small Business Utilization Staff propose and pursue additional
    incentives or other opportunities to assist FAA in meeting its Major
    Procurement Program Goals.

-   The co-chairs of the Integrated Product Leadership Team work with
    the Integrated Product Team leads to establish and implement a plan
    to reduce the risk of unexpected requirements, including those
    related to software and interface issues and human factors
    processes.

-   The FAA Acquisition Executive take the lead for the Joint Resources
    Council to establish and implement a plan to mitigate negative
    effects of budget cuts on programs.

**Contents**

**Executive Summary [i](#executive-summary)**

Objective [i](#objective)

Findings [i](#findings)

Procurement Results [ii](#procurement-results)

Program Results [ii](#program-results)

Recommendation Follow-up [iii](#recommendation-follow-up)

Recommendations in this Report [iv](#recommendations-in-this-report)

Part IIntroduction [1](#part-iintroduction)

Background [3](#__RefHeading___Toc453556144)

Scope [3](#scope)

Methodology in General [4](#methodology-in-general)

Procurement Results (Chapter 1) [4](#procurement-results-chapter-1)

Program Results (Chapter 2) [5](#program-results-chapter-2)

Recommendation Follow-up on Previous Acquisition Reform Evaluations
(Chapter 3)
[6](#recommendation-follow-up-on-previous-acquisition-reform-evaluations-chapter-3)

Part IIEvaluation Results [7](#part-iievaluation-results)

Chapter 1Procurement Results [9](#__RefHeading___Toc453556151)

Finding 1: Contract awards were more timely under acquisition reform.
[9](#finding-1-contract-awards-were-more-timely-under-acquisition-reform.)

Finding 2: The percentage of contract dollars awarded through
competition increased slightly.
[11](#finding-2-the-percentage-of-contract-dollars-awarded-through-competition-increased-slightly.)

Competition of Awarded Contract Dollars
[12](#competition-of-awarded-contract-dollars)

Finding 3: FAA has met small business goals but not other Major
Procurement Program Goals.
[12](#finding-3-faa-has-met-small-business-goals-but-not-other-major-procurement-program-goals.)

Causes for Not Meeting the MPPG Goals
[17](#causes-for-not-meeting-the-mppg-goals)

Small Business Utilization Staff Responsibilities and Efforts
[17](#small-business-utilization-staff-responsibilities-and-efforts)

Potential Effects if Efforts Are Not Continued
[18](#potential-effects-if-efforts-are-not-continued)

Recommendation [19](#recommendation)

Chapter 2Program Results [21](#__RefHeading___Toc453556160)

Finding 4: Program product performance results were on track, but cost
and schedule were not.
[21](#finding-4-program-product-performance-results-were-on-track-but-cost-and-schedule-were-not.)

Results Element (1): Supporting the FAA Mission
[23](#results-element-1-supporting-the-faa-mission)

Results Element (2): On Track to Deliver Product Performance
[24](#results-element-2-on-track-to-deliver-product-performance)

Results Element (3): Meeting Customer Needs
[25](#results-element-3-meeting-customer-needs)

Results Elements (4) and (5): Meeting Cost and Schedule Baselines
[27](#results-elements-4-and-5-meeting-cost-and-schedule-baselines)

Impact of Not Meeting Cost or Schedule Baselines
[28](#impact-of-not-meeting-cost-or-schedule-baselines)

Systemic Problems Identified [30](#systemic-problems-identified)

Recommendations [34](#recommendations)

Finding 5: Acquisition Management System processes did not have a direct
impact on program results.
[36](#finding-5-acquisition-management-system-processes-did-not-have-a-direct-impact-on-program-results.)

Overall Impact of Acquisition Reform and AMS
[37](#overall-impact-of-acquisition-reform-and-ams)

Chapter 3Recommendation Follow-up on Previous Acquisition Reform
Evaluations [39](#__RefHeading___Toc453556171)

Part IIIAppendices [53](#part-iiiappendices)

Appendix AMethodology Detailed [55](#__RefHeading___Toc453556173)

Procurement Results (Chapter 1) [55](#procurement-results-chapter-1-1)

Timeliness [55](#timeliness)

Competition [56](#competition)

Major Procurement Program Goals [56](#major-procurement-program-goals)

Program Results (Chapter 2) [57](#program-results-chapter-2-1)

Results in Terms of FAA Mission, Product Performance, Customer Needs,
and Cost and Schedule Baselines
[58](#results-in-terms-of-faa-mission-product-performance-customer-needs-and-cost-and-schedule-baselines)

Impact of Acquisition Reform on Program Results
[59](#impact-of-acquisition-reform-on-program-results)

Recommendation Follow-up on Previous Acquisition Reform Evaluations
(Chapter 3)
[60](#recommendation-follow-up-on-previous-acquisition-reform-evaluations-chapter-3-1)

Appendix BSample Information [61](#__RefHeading___Toc453556182)

Appendix CMajor Procurement Program Goal Information
[63](#__RefHeading___Toc453556183)

Description of FAA's Disadvantaged Business Distinctions
[63](#description-of-faas-disadvantaged-business-distinctions)

FAA versus National Goals and Accomplishments
[64](#faa-versus-national-goals-and-accomplishments)

Appendix DRecommendations [67](#__RefHeading___Toc453556186)

Appendix EList of Acronyms [71](#__RefHeading___Toc453556187)

**List of Tables**

Table 1-1. Comparison of MPPG Goals and Accomplishments for Regions,
Centers, and Headquarters for Fiscal Year 1997
[15](#__RefHeading___Toc453556188)

Table 1-2. Comparison of MPPG Goals and Accomplishments for Regions,
Centers, and Headquarters for Fiscal Year 1998
[16](#__RefHeading___Toc453556189)

Table 1-3. Comparison of FAA and National MPPG Goals and Accomplishments
for Fiscal Year 1998 (as a percentage of total procurements)
[16](#__RefHeading___Toc453556190)

Table 2-1. Sampled Programs [22](#__RefHeading___Toc453556191)

Table 2-2. Program Results Under Acquisition Reform
[23](#__RefHeading___Toc453556192)

Table 2-3. Program Results Under Acquisition Reform, FAA Mission
[23](#__RefHeading___Toc453556193)

Table 2-4. Program Results Under Acquisition Reform, Product Performance
[24](#__RefHeading___Toc453556194)

Table 2-5. Program Results Under Acquisition Reform, Customer Needs
[26](#__RefHeading___Toc453556195)

Table 2-6. Program Results Under Acquisition Reform, Baselines
[27](#__RefHeading___Toc453556196)

Table 2-7. Programs Within 10 Percent Variance Range
[28](#__RefHeading___Toc453556197)

Table 2-8. Program Termination Requirements for Cost and Schedule
[29](#__RefHeading___Toc453556198)

Table 2-9. Systemic Problems Contributing to Cost and Schedule Growth
[31](#__RefHeading___Toc453556199)

Table 3-1. Recommendation Actions to FAA's Report for the First Year,
dated May 1997 [40](#__RefHeading___Toc453556200)

Table 3-2. Recommendation Actions to FAA's Report for the First Two
Years, dated May 29, 1998 [42](#__RefHeading___Toc453556201)

Table 3-3. Recommendation Actions to Booz•Allen & Hamilton's Independent
Assessment, dated September 19, 1997 [49](#__RefHeading___Toc453556202)

Table 3-4. Recommendation Actions to FAA's Report on Disadvantaged
Businesses, dated May 20, 1998 [52](#__RefHeading___Toc453556203)

Table B-1. Fiscal Year 1995 Timeliness Contract File Sample Set
[61](#__RefHeading___Toc453556204)

Table B-2. Fiscal Year 1997 Timeliness Contract File Sample Set
[61](#__RefHeading___Toc453556205)

Table B-3. Fiscal Year 1998 Timeliness Contract File Sample Set
[62](#__RefHeading___Toc453556206)

Table B-4. Sampled Programs Managed Under Acquisition Reform
[62](#__RefHeading___Toc453556207)

Table C-1. FAA Distinction Between SEDB and SDB Awards
[64](#__RefHeading___Toc453556208)

Table C-2. National versus FAA Goals for Small and Disadvantaged
Business Awards [65](#__RefHeading___Toc453556209)

Table C-3. Comparison of FAA and National MPPG Goals and Accomplishments
for Fiscal Year 1998 (as a percentage of total procurements)
[66](#__RefHeading___Toc453556210)

List of Figures

Figure 1-1. Average Number of Days from Contracting Officer's First
Action to Solicitation and from Solicitation to Contract Award
[10](#__RefHeading___Toc453556211)

Figure 1-2. Comparison of Competed and Non-Competed Contract Dollar
Values for Fiscal Years 1995, 1997, and 1998
[12](#__RefHeading___Toc453556212)

Figure 1-3. MPPG Goals and Accomplishments for Fiscal Year 1997
[14](#__RefHeading___Toc453556213)

Figure 1-4. MPPG Goals and Accomplishments for Fiscal Year 1998
[14](#__RefHeading___Toc453556214)

Figure 3-1. Status of Recommendations [39](#__RefHeading___Toc453556215)

Figure A-1. Procurement Cycle: Before and Since Acquisition Reform
[56](#__RefHeading___Toc453556216)

# Part IIntroduction {#part-iintroduction .Part}

[]{#__RefHeading___Toc453556144 .anchor}Background

On November 15, 1995, the President signed the Department of
Transportation and Related Agencies Appropriations Act of 1996 (Public
Law 104-50), which directed the FAA Administrator to:

> ...develop and implement... an acquisition management system for the
> Federal Aviation Administration that addresses the unique needs of the
> agency and, at a minimum, provides for more timely and cost-effective
> acquisitions of equipment and materials.

The FAA Reauthorization Act of 1996 (Public Law 104-264, dated October
9, 1996), expanded the acquisition reforms previously authorized by
Public Law 104-50 and provided the FAA Administrator increased autonomy
in carrying out the functions of the office and the authority to enter
into contracts, grants, cooperative agreements, and other transactions
with public and private entities on such terms and conditions as were
considered appropriate.

In response to the acquisition reform granted by these public laws, FAA
developed its Acquisition Management System (AMS), which went into
effect on April 1, 1996. In June 1997, FAA published a revised and
updated AMS policy document and has since maintained the current version
of all acquisition policies and procedures, guidance, templates, and
tools in the FAA Acquisition System Toolset located on the Internet
(http://fast.faa.gov). This site is the official repository for all AMS
documentation and is updated periodically under strict configuration
control.

With the advent of acquisition reform, the Associate Administrator for
Research and Acquisitions directed the Program Evaluation Branch to
report on the status and success of acquisition reform for the first
three years. An overall strategy for the three yearly evaluations was
established to provide progressively more detailed reviews as
acquisition reform and the AMS policy and procedures were put in place
and refined. The purpose of the first year's evaluation was to review
the implementation of the AMS policy and procedures and the supporting
infrastructure. The purpose of the second year's evaluation was to focus
on those key AMS processes, systems, and decision making bodies that
were critical to the success of acquisition reform. In this third year
evaluation, the purpose was to review the results of acquisition reform
as measured against ongoing acquisitions. As a result, the evaluation
team placed emphasis on procurement and program performance results
attributable to acquisition reform, not the AMS processes themselves.

# Scope

The Congress granted FAA acquisition reform and mandated that FAA
provide for more timely and cost-effective acquisition of equipment and
materials. The focus of acquisition reform is on mission accomplishment,
rather than compliance with processes. While FAA developed the AMS
policy to implement acquisition reform, the Program Evaluation Branch
was tasked to look at how the agency is actually doing since acquisition
reform and not necessarily whether the agency was complying with the
AMS. If the processes used conformed to AMS and produced successful
results, no changes to AMS would be necessary. However, if the processes
did not conform to AMS or, if conforming to AMS did not produce
successful results, the evaluation team would assess the procedures to
see if a change in AMS was warranted. The evaluation team recognized
there are various issues and concerns with improving the AMS policy;
however, the focus of this evaluation was not specifically on AMS or
processes per se, but was on the agency's procurement and program
*results* since acquisition reform. The three objectives below define
the focus of this report on acquisition reform. The evaluation team set
out to determine:

-   If, under acquisition reform, procurement efforts were achieving
    faster awards, were achieving competition, and were meeting Major
    Procurement Program Goals.

-   If, under acquisition reform, programs were on track to achieve
    success.

-   The status of the acquisition reform recommendations made in
    previous evaluation reports.

# Methodology in General

In order to conduct this review, the evaluation team employed various
evaluation techniques, practices, and procedures. The information below
is a general description of the primary steps taken to gather and
analyze information in order to evaluate the objectives. A more detailed
description can be found in Appendix A, and a list of contracts and
programs sampled can be found in Appendix B.

## Procurement Results (Chapter 1)

The evaluation team collected and analyzed data to determine if, since
acquisition reform, procurements were awarded faster, the degree of
contract competition did not diminish, and Major Procurement Program
Goals were achieved. Specifically, measurements were made of:

-   The time it took to award contracts. The evaluation team analyzed
    the time elapsed among three points in the procurement cycle and
    selected a random sample of contracts with a dollar value in excess
    of \$100,000. The contracts selected were from the last fiscal year
    (FY) before acquisition reform (FY 95), chosen as a base year, and
    compared to contracts from the first two full fiscal years (FY 97
    and FY 98) under acquisition reform. Because acquisition reform
    began in April 1996, during the middle of a fiscal year, FY 96
    contracts were not included in the analysis.

-   The degree of contract competition. The evaluation team analyzed the
    entire universe of contracts greater than \$25,000 to determine the
    extent of competition for contract dollars. Again, the evaluation
    team selected the last fiscal year before acquisition reform as a
    base year (FY 95) to compare to the first two full fiscal years
    (FY 97 and FY 98) under acquisition reform, and FY 96 contracts were
    not included in the analysis.

-   The achievement of Major Procurement Program Goals. The evaluation
    team analyzed the universe of data as reported by all FAA
    procurement offices on these goals for the first two full fiscal
    years under acquisition reform (FY 97 and FY 98). A comparison to
    pre-acquisition data was hampered because acquisition reform and
    updates made to the Small Business Utilization Staff's reporting
    procedures in FY 96 made data comparison more complicated. In
    addition, it was determined to be unnecessary in this evaluation
    because regardless of acquisition reform, there are specific Major
    Procurement Program Goals that need to be met.

## Program Results (Chapter 2)

In order to evaluate the success of programs, the evaluation team
reviewed program results in terms of five specific elements:
(1) supporting the FAA mission, (2) achieving planned product
performance, (3) meeting customer needs, (4) meeting cost baselines, and
(5) meeting schedule baselines. To conduct this review the team:

-   Obtained a universe of existing FAA programs and selected a
    judgmental sample of 10 facilities and equipment funded programs and
    3 research, engineering, and development funded programs to
    evaluate. A detailed list of the programs can be found in
    Appendix B, Table B-4. While these programs were managed under
    acquisition reform, most had completed a substantial part of the
    planning efforts prior to acquisition reform and most did not have
    fielded products at the time of this review.

```{=html}
<!-- -->
```
-   Interviewed Integrated Product Development System leaders, product
    or program customers, and financial and support personnel.

-   Analyzed and compared information from various documents, including
    Mission Need Statements, Integrated Program Plans, Acquisition
    Program Baselines, Requirements Documents, Acquisition Review data,
    Baseline Management Notices, status reviews, status reports, and
    monthly program reviews, as well as the National Airspace System
    Architecture.

-   Compared the program results attained with the processes followed
    for each program sampled. The team attempted to correlate the AMS
    processes used to program results. However, due to the short period
    of time since acquisition reform was implemented and the long term
    nature of agency programs, the evaluation team concluded it was
    still too early to state the effect that AMS or acquisition reform
    has on program results with certainty.

## Recommendation Follow-up on Previous Acquisition Reform Evaluations (Chapter 3)

The evaluation team identified all the recommendations from the two
previous internal acquisition reform evaluations: the *Evaluation of FAA
Acquisition Reform: The First Year: April 1996-March 1997*, and the
*Evaluation of FAA Acquisition Reform: The First Two Years: April
1996-March 1998* (which included two referenced recommendations from
another internal report, *Impact of Acquisition Reform on Awards to
Disadvantaged Businesses*). The team also identified the recommendations
from an external acquisition reform evaluation: the *Independent
Assessment of the FAA's Acquisition Management System*. The evaluation
team then sought information from each responsible FAA official on the
actions taken to implement the recommendations, evaluated the actions
taken, and determined the status to be reported as either completed, in
progress, or no action.

# Part IIEvaluation Results {#part-iievaluation-results .Part}

[]{#__RefHeading___Toc453556151 .anchor}Chapter 1Procurement Results

Under acquisition reform, FAA received procurement relief from the
Federal Acquisition Regulation. While procurement is only one part of an
acquisition's lifecycle, it is the area where the agency receives most
of its outside interest. Consequently, it was in the procurement area
that the evaluation team's objective was to determine if FAA was
achieving faster awards, was achieving competition, and was meeting
small business, women owned business, and disadvantaged business goals
under acquisition reform. For a random sample of contracts awarded that
were valued over \$100,000, the evaluation team analyzed the time
elapsed among three points in the procurement cycle and determined that,
since acquisition reform, FAA has shortened the time to award contracts.
An analysis of all the contracts in the Contract Information System
database for the fiscal years evaluated revealed that, under acquisition
reform, most contract dollars were awarded through competition. Finally,
an analysis of the Major Procurement Program Goals for fiscal years 1997
and 1998 indicated that, although FAA was achieving its overall small
business goals, it was not meeting the other disadvantaged business and
women owned business goals.

# Finding 1: Contract awards were more timely under acquisition reform.

The evaluation team collected and analyzed contract data to measure the
average time to award procurements and determine if there was any
progress toward achieving faster procurements. The time to award
procurements was defined in this review as the time from the contracting
officer's first action to contract award. Previous evaluation efforts
were too early in the implementation of acquisition reform to have
complete fiscal year data for comparison. Now that more time has
elapsed, the evaluation team was able to use two full fiscal years since
acquisition reform (FY 97 and FY 98) and compare those results with a
base year (FY 95), the latest full fiscal year before acquisition
reform.

Using a random sample[^1] selected from all contracts over \$100,000,
the evaluation team measured the time elapsed from the contracting
officer's first action to solicitation, from solicitation to contract
award, and from the contracting officer's first action to contract
award. The evaluation team did not analyze the time from the customer
generated procurement request to the contracting officer's first action,
which included events such as statement of work preparation, funding
availability and approval, and program delays. This period of time was
not reviewed because these events occurred prior to the contracting
officer's first action and were outside of the contracting officer's
responsibilities.

The evaluation team's analyses found that, under acquisition reform, the
time to award contracts over \$100,000 has decreased approximately
55 percent from the FY 95 level. Figure 1-1 presents a comparison of
FY 95 data to FY 97 and FY 98 data. The

comparison shows that procurement efforts achieved faster awards.
Specifically, under acquisition reform, FAA reduced by:

-   49 percent the average time from the contracting officer's first
    action to solicitation issuance (from 51 days in FY 95, down to
    30 days in FY 97, and down to 26 days in FY 98).

-   59 percent the average time from solicitation issuance to contract
    award (from 106 days in FY 95, down to 87 days in FY 97, and down to
    44 days in FY 98).

-   55 percent the average time from contracting officer's first action
    to contract award (from 156^1^ days in FY 95 down to 119^1^ days in
    FY 97, and down to 70 days in FY 98).

![](media/image1.wmf){width="5.747222222222222in"
height="3.4493055555555556in"}

[]{#__RefHeading___Toc453556211 .anchor}Figure 1-1. Average Number of
Days from Contracting Officer's First Action to Solicitation and from
Solicitation to Contract Award[^2]

The evaluation team attributed the success in creating a more
streamlined procurement process to three primary causes:

-   The exemption from the Federal Acquisition Regulation, which
    included a mandatory length of time for a *Commerce Business Daily*
    announcement and a less flexible source selection process. The
    Federal Acquisition Regulation required a minimum of 15 days for
    announcing the bid in the *Commerce Business Daily* and 30 days
    response time for receipt of bids and proposals. Under the agency's
    Acquisition Management System (AMS), there is no minimum time for
    announcement. Also, under the Federal Acquisition Regulation, the
    government must consider all offerors in the competitive range.
    Under AMS, FAA may negotiate with one, some, or all of the offerors.

-   Increased FAA personnel familiarization with AMS policy and
    guidelines in the second full and subsequent years of acquisition
    reform.

-   The application of improved communication methods, specifically,
    utilizing the Internet for online information, communications, and
    solicitations.

The reduced procurement time ultimately contributes toward a reduction
in time for the overall acquisition cycle. Additional benefits of faster
procurement time may include:

-   Helping achieve National Performance Review and public law
    objectives to reduce acquisition time.

-   Reducing administrative time and costs.

-   Achieving faster technology refreshment for existing systems.

-   Increasing customer satisfaction.

-   Improving FAA's credibility.

-   Allowing FAA's acquisition reform to serve as a model of faster
    procurement for other agencies to follow.

The evaluation team concluded that FAA procurement offices have been
successful at awarding contracts faster and should continue their
efforts. Areas of concentration should include:

-   Continuing effective AMS processes initiated under acquisition
    reform such as the Internet announcements and screening process.

-   Continuing to disseminate information about the AMS internally and
    externally.

-   Continuing employee and contractor AMS familiarization and follow-on
    training.

-   Increasing the use of new technology such as contract writing tools,
    Internet, etc.

-   Obtaining, evaluating, and responding to constructive customer and
    contractor feedback.

# Finding 2: The percentage of contract dollars awarded through competition increased slightly.

While acquisition reform exempted FAA from the Federal Acquisition
Regulation, the agency's AMS encouraged competition as the preferred
method of contracting, and the agency did not expect competition to
diminish as a result of acquisition reform. The evaluation team's
objective was to measure the extent of competition achieved by the
agency. Using the universe of contracts in the Contract Information
System database, the evaluation team analyzed and compared competition
data for all contract award dollars for FY 95, the latest full fiscal
year before acquisition reform, to all contract award dollars for FY 97
and FY 98, the first two full fiscal years since acquisition reform.

## Competition of Awarded Contract Dollars

The evaluation team found that, under acquisition reform, the percentage
of procurement dollars awarded through competition diminished three
percent in the first year, but recovered and increased by an additional
four percent in FY 98 as compared to the base year of FY 95. The results
are illustrated by fiscal year in Figure 1-2 for the three fiscal years
analyzed. The actual percentage results for each fiscal year analyzed
show that:

-   66 percent of contract dollars for FY 95 were awarded competitively.

-   63 percent of contract dollars for FY 97 were awarded competitively.

-   70 percent of contract dollars for FY 98 were awarded competitively.

The evaluation team concluded that FAA procurement offices should
continue their existing successful efforts to compete requirements for
products and services where possible.

![](media/image2.wmf){width="5.748611111111111in"
height="3.9305555555555554in"}

[]{#__RefHeading___Toc453556212 .anchor}Figure 1-2. Comparison of
Competed and Non-Competed Contract Dollar Values for Fiscal Years 1995,
1997, and 1998

# Finding 3: FAA has met small business goals but not other Major Procurement Program Goals.

Under the Small Business Act, the Congress mandated that federal
agencies establish goals, representing the nation's goals, for awarding
contracts to small business concerns in general, and to small business
concerns owned and controlled by socially and economically disadvantaged
individuals specifically. Although acquisition reform legislation
exempted the agency from the Small Business Act, FAA is required to
provide all reasonable opportunities to award contracts to small
business concerns and small business concerns owned and controlled by
socially and economically disadvantaged individuals. Thus, the
evaluation team's objective was to measure FAA's accomplishment of these
goals, and the team did so by examining the data FAA collects on all its
prime contract awards. The evaluation team did not review subcontracting
goals and accomplishments because this information is not consistently
reported or available. FAA segregates its data by the types of awards
made. Four of these award types are:

-   SEDB awards: Awards to disadvantaged businesses[^3] that hold
    current Section 8(a) certifications from the Small Business
    Administration and obtained contract award through an SEDB/8(a)
    set-aside competition.

-   SDB awards: Awards to disadvantaged businesses^1^ that hold current
    Section 8(a) certifications from the Small Business Administration,
    but obtained contract award through means other than an SEDB/8(a)
    set-aside competition, or awards to disadvantaged businesses^1^ that
    do not hold current Section 8(a) certifications from the Small
    Business Administration.

-   WOB awards: Awards to small business firms that are 51 percent owned
    and controlled by women, irrespective of their status as a
    disadvantaged business, i.e., disadvantaged businesses^1^ that also
    are women owned, are counted only in the WOB award category.

-   Small business awards: Awards to all small businesses. This includes
    SEDB, SDB, and WOB awards, as well as awards to other small and very
    small businesses.

In addition to collecting information on these awards, FAA establishes
annual goals for each of these four award categories. This report
collectively refers to the goals in these four award categories as Major
Procurement Program Goals (MPPG). Quarterly, FAA headquarters, regions,
and centers report on their accomplishments of these goals.

The evaluation team:

-   Reviewed the accomplishment of these four MPPG goals for the first
    two full fiscal years under acquisition reform (FY 97 and FY 98).

-   Reviewed the entire FAA accomplishment of these goals in addition to
    the individual accomplishments of headquarters, regions, and
    centers.

The evaluation team found that, in total, FAA successfully achieved its
small business goals for both FY 97 and FY 98. Though the agency
successfully achieved its SEDB goal for FY 97, it did not achieve the
SEDB goal for FY 98. Additionally, the FAA did not achieve the SDB or
the WOB goal for either FY 97 or FY 98. Figures 1-3 and 1-4 illustrate
this comparison.

![](media/image3.wmf){width="5.745138888888889in" height="3.65in"}

[]{#__RefHeading___Toc453556213 .anchor}Figure 1-3. MPPG Goals and
Accomplishments for Fiscal Year 1997

(percentage of contract dollars)

![](media/image4.wmf){width="5.748611111111111in"
height="3.6958333333333333in"}

[]{#__RefHeading___Toc453556214 .anchor}Figure 1-4. MPPG Goals and
Accomplishments for Fiscal Year 1998

(percentage of contract dollars)

A comparison of all FAA contract award dollars shows that, for FY 97 and
FY 98, the headquarters office awarded 64 percent of all contract
dollars, the two centers together awarded 20 percent of all dollars, and
all of the regions combined awarded 16 percent. Because the headquarters
office awards the majority of all FAA contract dollars, its ability to
meet MPPG goals significantly affects the agency's overall
accomplishment of these goals. Taking this into consideration, a further
comparison of MPPG goal achievement by the individual FAA offices
indicates that the headquarters office did not achieve a proportionate
percentage of the agency goals. Because the headquarters office's
percentage of total awards was so high, and its achievement of the goals
was so low, the agency as a whole did not meet its MPPG goals in certain
categories.

The data in Tables 1-1 and 1-2 indicate that, with respect to achieving
MPPG goals under acquisition reform, FAA headquarters missed SDB and WOB
goals in FY 97 and missed all MPPG goals in FY 98. When combined, FAA
regions exceeded all MPPG goals in FY 97 and FY 98, with the exception
of SEDB goals in FY 98. Finally, the combination of the two FAA centers
exceeded all MPPG goals in FY 97 and FY 98, with the exception of WOB
goals in FY 97.

[]{#__RefHeading___Toc453556188 .anchor}Table 1-1. Comparison of MPPG
Goals and Accomplishments for Regions, Centers, and Headquarters for
Fiscal Year 1997

+---+------+------+--------+--------+--------+--------+--------------+
|   |      |      | *      |        |        |        |              |
|   |      |      | *Major |        |        |        |              |
|   |      |      | Procu  |        |        |        |              |
|   |      |      | rement |        |        |        |              |
|   |      |      | P      |        |        |        |              |
|   |      |      | rogram |        |        |        |              |
|   |      |      | Goals  |        |        |        |              |
|   |      |      | and    |        |        |        |              |
|   |      |      | Accom  |        |        |        |              |
|   |      |      | plishm |        |        |        |              |
|   |      |      | ents** |        |        |        |              |
+---+------+------+--------+--------+--------+--------+--------------+
|   |      |      | *      | **SEDB | **SDB  | **WOB  | **Total      |
|   |      |      | *Small | Aw     | Aw     | Aw     | Obligated    |
|   |      |      | Bu     | ards** | ards** | ards** | Dollars**    |
|   |      |      | siness |        |        |        |              |
|   |      |      | Aw     |        |        |        |              |
|   |      |      | ards** |        |        |        |              |
+---+------+------+--------+--------+--------+--------+--------------+
| * |      |      | **20%  | **5    | **5%** | **5%** |              |
| * |      |      | **[^4] | %^1^** |        |        |              |
| G |      |      |        |        |        |        |              |
| o |      |      |        |        |        |        |              |
| a |      |      |        |        |        |        |              |
| l |      |      |        |        |        |        |              |
| s |      |      |        |        |        |        |              |
| i |      |      |        |        |        |        |              |
| n |      |      |        |        |        |        |              |
| P |      |      |        |        |        |        |              |
| e |      |      |        |        |        |        |              |
| r |      |      |        |        |        |        |              |
| c |      |      |        |        |        |        |              |
| e |      |      |        |        |        |        |              |
| n |      |      |        |        |        |        |              |
| t |      |      |        |        |        |        |              |
| o |      |      |        |        |        |        |              |
| f |      |      |        |        |        |        |              |
| T |      |      |        |        |        |        |              |
| o |      |      |        |        |        |        |              |
| t |      |      |        |        |        |        |              |
| a |      |      |        |        |        |        |              |
| l |      |      |        |        |        |        |              |
| O |      |      |        |        |        |        |              |
| b |      |      |        |        |        |        |              |
| l |      |      |        |        |        |        |              |
| i |      |      |        |        |        |        |              |
| g |      |      |        |        |        |        |              |
| a |      |      |        |        |        |        |              |
| t |      |      |        |        |        |        |              |
| e |      |      |        |        |        |        |              |
| d |      |      |        |        |        |        |              |
| D |      |      |        |        |        |        |              |
| o |      |      |        |        |        |        |              |
| l |      |      |        |        |        |        |              |
| l |      |      |        |        |        |        |              |
| a |      |      |        |        |        |        |              |
| r |      |      |        |        |        |        |              |
| s |      |      |        |        |        |        |              |
| * |      |      |        |        |        |        |              |
| * |      |      |        |        |        |        |              |
+---+------+------+--------+--------+--------+--------+--------------+
| * |      |      |        |        |        |        |              |
| * |      |      |        |        |        |        |              |
| A |      |      |        |        |        |        |              |
| c |      |      |        |        |        |        |              |
| c |      |      |        |        |        |        |              |
| o |      |      |        |        |        |        |              |
| m |      |      |        |        |        |        |              |
| p |      |      |        |        |        |        |              |
| l |      |      |        |        |        |        |              |
| i |      |      |        |        |        |        |              |
| s |      |      |        |        |        |        |              |
| h |      |      |        |        |        |        |              |
| m |      |      |        |        |        |        |              |
| e |      |      |        |        |        |        |              |
| n |      |      |        |        |        |        |              |
| t |      |      |        |        |        |        |              |
| s |      |      |        |        |        |        |              |
| * |      |      |        |        |        |        |              |
| * |      |      |        |        |        |        |              |
+---+------+------+--------+--------+--------+--------+--------------+
| > | **9  |      | 71.1%  | 5.3%   | 8.4%   | 12.0%  | \$339.9M     |
|   | R    |      |        |        |        |        |              |
| * | egio |      |        |        |        |        |              |
| * | ns** |      |        |        |        |        |              |
| O |      |      |        |        |        |        |              |
| r |      |      |        |        |        |        |              |
| g |      |      |        |        |        |        |              |
| a |      |      |        |        |        |        |              |
| n |      |      |        |        |        |        |              |
| i |      |      |        |        |        |        |              |
| z |      |      |        |        |        |        |              |
| a |      |      |        |        |        |        |              |
| t |      |      |        |        |        |        |              |
| i |      |      |        |        |        |        |              |
| o |      |      |        |        |        |        |              |
| n |      |      |        |        |        |        |              |
| s |      |      |        |        |        |        |              |
| * |      |      |        |        |        |        |              |
| * |      |      |        |        |        |        |              |
+---+------+------+--------+--------+--------+--------+--------------+
|   | **2  |      | 46.8%  | 17.8%  | 5.1%   | 3.7%   | \$413.9M     |
|   | C    |      |        |        |        |        |              |
|   | ente |      |        |        |        |        |              |
|   | rs** |      |        |        |        |        |              |
+---+------+------+--------+--------+--------+--------+--------------+
|   | *    |      | 25.3%  | 7.3%   | 1.6%   | 1.3%   | \$1,327.0M   |
|   | *FAA |      |        |        |        |        |              |
|   | He   |      |        |        |        |        |              |
|   | adqu |      |        |        |        |        |              |
|   | arte |      |        |        |        |        |              |
|   | rs** |      |        |        |        |        |              |
+---+------+------+--------+--------+--------+--------+--------------+
|   | *    |      | **3    | **     | **     | **     | **           |
|   | *All |      | 7.0%** | 9.1%** | 3.4%** | 3.5%** | \$2,080.8M** |
|   | F    |      |        |        |        |        |              |
|   | AA** |      |        |        |        |        |              |
+---+------+------+--------+--------+--------+--------+--------------+

Shaded area indicates missed goal.

[]{#__RefHeading___Toc453556189 .anchor}Table 1-2. Comparison of MPPG
Goals and Accomplishments for Regions, Centers, and Headquarters for
Fiscal Year 1998

+---+------+------+--------+--------+--------+--------+--------------+
|   |      |      | **MPPG |        |        |        |              |
|   |      |      | Goals  |        |        |        |              |
|   |      |      | and    |        |        |        |              |
|   |      |      | Accom  |        |        |        |              |
|   |      |      | plishm |        |        |        |              |
|   |      |      | ents** |        |        |        |              |
+---+------+------+--------+--------+--------+--------+--------------+
|   |      |      | *      | **SEDB | **SDB  | **WOB  | **Total      |
|   |      |      | *Small | Aw     | Aw     | Aw     | Obligated    |
|   |      |      | Bu     | ards** | ards** | ards** | Dollars**    |
|   |      |      | siness |        |        |        |              |
|   |      |      | Aw     |        |        |        |              |
|   |      |      | ards** |        |        |        |              |
+---+------+------+--------+--------+--------+--------+--------------+
| * |      |      | **25%  | **10   | **5%** | **5%** |              |
| * |      |      | **[^5] | %^1^** |        |        |              |
| G |      |      |        |        |        |        |              |
| o |      |      |        |        |        |        |              |
| a |      |      |        |        |        |        |              |
| l |      |      |        |        |        |        |              |
| s |      |      |        |        |        |        |              |
| i |      |      |        |        |        |        |              |
| n |      |      |        |        |        |        |              |
| P |      |      |        |        |        |        |              |
| e |      |      |        |        |        |        |              |
| r |      |      |        |        |        |        |              |
| c |      |      |        |        |        |        |              |
| e |      |      |        |        |        |        |              |
| n |      |      |        |        |        |        |              |
| t |      |      |        |        |        |        |              |
| o |      |      |        |        |        |        |              |
| f |      |      |        |        |        |        |              |
| T |      |      |        |        |        |        |              |
| o |      |      |        |        |        |        |              |
| t |      |      |        |        |        |        |              |
| a |      |      |        |        |        |        |              |
| l |      |      |        |        |        |        |              |
| O |      |      |        |        |        |        |              |
| b |      |      |        |        |        |        |              |
| l |      |      |        |        |        |        |              |
| i |      |      |        |        |        |        |              |
| g |      |      |        |        |        |        |              |
| a |      |      |        |        |        |        |              |
| t |      |      |        |        |        |        |              |
| e |      |      |        |        |        |        |              |
| d |      |      |        |        |        |        |              |
| D |      |      |        |        |        |        |              |
| o |      |      |        |        |        |        |              |
| l |      |      |        |        |        |        |              |
| l |      |      |        |        |        |        |              |
| a |      |      |        |        |        |        |              |
| r |      |      |        |        |        |        |              |
| s |      |      |        |        |        |        |              |
| * |      |      |        |        |        |        |              |
| * |      |      |        |        |        |        |              |
+---+------+------+--------+--------+--------+--------+--------------+
| * |      |      |        |        |        |        |              |
| * |      |      |        |        |        |        |              |
| A |      |      |        |        |        |        |              |
| c |      |      |        |        |        |        |              |
| c |      |      |        |        |        |        |              |
| o |      |      |        |        |        |        |              |
| m |      |      |        |        |        |        |              |
| p |      |      |        |        |        |        |              |
| l |      |      |        |        |        |        |              |
| i |      |      |        |        |        |        |              |
| s |      |      |        |        |        |        |              |
| h |      |      |        |        |        |        |              |
| m |      |      |        |        |        |        |              |
| e |      |      |        |        |        |        |              |
| n |      |      |        |        |        |        |              |
| t |      |      |        |        |        |        |              |
| s |      |      |        |        |        |        |              |
| * |      |      |        |        |        |        |              |
| * |      |      |        |        |        |        |              |
+---+------+------+--------+--------+--------+--------+--------------+
| > | **9  |      | 60.5%  | 3.9%   | 6.8%   | 8.9%   | \$306.0M     |
|   | R    |      |        |        |        |        |              |
| * | egio |      |        |        |        |        |              |
| * | ns** |      |        |        |        |        |              |
| O |      |      |        |        |        |        |              |
| r |      |      |        |        |        |        |              |
| g |      |      |        |        |        |        |              |
| a |      |      |        |        |        |        |              |
| n |      |      |        |        |        |        |              |
| i |      |      |        |        |        |        |              |
| z |      |      |        |        |        |        |              |
| a |      |      |        |        |        |        |              |
| t |      |      |        |        |        |        |              |
| i |      |      |        |        |        |        |              |
| o |      |      |        |        |        |        |              |
| n |      |      |        |        |        |        |              |
| s |      |      |        |        |        |        |              |
| * |      |      |        |        |        |        |              |
| * |      |      |        |        |        |        |              |
+---+------+------+--------+--------+--------+--------+--------------+
|   | **2  |      | 55.0%  | 19.9%  | 8.3%   | 7.5%   | \$387.0M     |
|   | C    |      |        |        |        |        |              |
|   | ente |      |        |        |        |        |              |
|   | rs** |      |        |        |        |        |              |
+---+------+------+--------+--------+--------+--------+--------------+
|   | *    |      | 16.5%  | 6.2%   | 2.0%   | 0.7%   | \$1,223.0M   |
|   | *FAA |      |        |        |        |        |              |
|   | He   |      |        |        |        |        |              |
|   | adqu |      |        |        |        |        |              |
|   | arte |      |        |        |        |        |              |
|   | rs** |      |        |        |        |        |              |
+---+------+------+--------+--------+--------+--------+--------------+
|   | *    |      | **3    | **     | **     | **     | **           |
|   | *All |      | 1.3%** | 8.6%** | 4.1%** | 3.4%** | \$1,916.0M** |
|   | F    |      |        |        |        |        |              |
|   | AA** |      |        |        |        |        |              |
+---+------+------+--------+--------+--------+--------+--------------+

Shaded area indicates missed goal.

Although FAA is failing to meet many of its MPPG goals, a more complete
representation of FAA's goals and accomplishments can be understood only
in relation to the national goals and accomplishments. While some FAA
MPPG goals were reduced since acquisition reform, FAA's small business
goals and SEDB goals have been set 5 to 10 percent higher than the
national goals in 3 of the last 4 years. In addition, during FY 98, FAA
accomplished a higher percentage of its total procurement awards to
small business, SEDB, SDB, and WOB than the national accomplishments for
those same award categories. See Table 1-3 and Appendix C for further
details.

[]{#__RefHeading___Toc453556190 .anchor}Table 1-3. Comparison of FAA and
National MPPG Goals and Accomplishments for Fiscal Year 1998 (as a
percentage of total procurements)

+---------------+--------+-----------------+--------+-----------------+
|               | #      |                 | **Nati |                 |
|               | ###### |                 | onal** |                 |
|               | ## FAA |                 |        |                 |
+---------------+--------+-----------------+--------+-----------------+
|               | **G    | **Ac            | **G    | **Ac            |
|               | oals** | complishments** | oals** | complishments** |
+---------------+--------+-----------------+--------+-----------------+
| **Small       | 25%    | 31.3%           | 20%    | 23.39%          |
| Business      |        |                 |        |                 |
| Awards**      |        |                 |        |                 |
+---------------+--------+-----------------+--------+-----------------+
| **SEDB        | 10%    | 8.6%            | 5%     | 3.57%           |
| Awards**      |        |                 |        |                 |
+---------------+--------+-----------------+--------+-----------------+
| **SDB         | 5%     | 4.1%            | 5%     | 2.91%           |
| Awards**      |        |                 |        |                 |
+---------------+--------+-----------------+--------+-----------------+
| **WOB         | 5%     | 3.4%            | 5%     | 2.21%           |
| Awards**      |        |                 |        |                 |
+---------------+--------+-----------------+--------+-----------------+

Shaded area indicates higher FAA goal/accomplishment.

## Causes for Not Meeting the MPPG Goals

Based on previous year's work the evaluation team performed in this area
and interviews with agency directors, regional administrators, managers,
small business representatives, contracting officers, and program
managers/engineers at FAA headquarters, one center, and two regions, the
following causes were identified:

-   Before acquisition reform the primary method used for making SEDB
    awards was though noncompetitive set-aside contracts. Under a
    noncompetitive set-aside contract, contracting officers could award
    contracts through a faster process than with competitive awards and
    could award contracts without competition. This made the
    noncompetitive set-aside process quick, and many contracting
    officers and Integrated Product Teams preferred this process when it
    was reasonable for the circumstances. Under acquisition reform, this
    non-competitive set-aside option was eliminated and with it the
    significant incentive to make SEDB awards.

-   Acquisition reform relaxed the requirement for contracting officers
    and Integrated Product Teams to coordinate through the Small
    Business Utilization Staff. Prior to acquisition reform, contracting
    officers were required to coordinate most contract requirements
    through the Small Business Utilization Staff. As a result of not
    coordinating with this Staff, the agency may have lost opportunities
    to identify contracts for SEDB/8(a) competitive set-aside awards.
    The Small Business Utilization Staff maintains a database of
    eligible disadvantaged businesses holding Small Business
    Administration Section 8(a) certifications and may be able to
    identify MPPG goal opportunities.

-   While most agency directors, regional administrators and managers,
    and contracting officers were aware of the existence and importance
    of MPPG goals, technical managers and program managers were not.
    This difference in the level of awareness of MPPG goals may be
    because procurement personnel focus their efforts on contracting
    issues and have an MPPG goal element included in their annual
    performance appraisals. All the technical and program managers
    interviewed focused their efforts on the program itself and did not
    have a comparable element for MPPG goals included in their
    performance appraisals. This different focus on the part of
    technical and program managers was significant because both groups
    are an integral part of awarding FAA contracts.

## Small Business Utilization Staff Responsibilities and Efforts

The FAA's Small Business Utilization Staff is responsible for:

-   Implementing FAA's small business program, which includes the
    utilization of small businesses and small businesses owned and
    controlled by socially and economically disadvantaged individuals.

-   Establishing mechanisms for monitoring and evaluating the
    effectiveness of the small business program.

-   Ensuring FAA-wide implementation and accomplishment of the small
    business program objectives.

-   Implementing and monitoring the FAA Mentor-Protégé Program, which
    establishes incentives to prime contractors to assist socially and
    economically disadvantaged businesses, among others, in enhancing
    their capabilities to perform FAA contracts. This assistance by
    mentor businesses is expected to increase the overall number of
    disadvantaged businesses that receive FAA contract awards.

In addition to fulfilling these responsibilities, the Small Business
Utilization Staff has undertaken outreach initiatives to improve MPPG
goal performance. The Staff coordinated with the sponsors of small
business conferences for vendors and contractors to assist and educate
vendors and contractors on doing business with FAA. The first Small
Business Conference was held at the William J. Hughes Technical Center
in October 1997. Subsequent Small Business Conferences have been held in
Los Angeles, California, in March 1998; Atlanta, Georgia, in September
1998; and Tacoma, Washington, in March 1999. Four other conferences are
scheduled: a Small Business Symposium at Chicago, Illinois, in
June 1999; a Women's Business Initiative Workshop in August 1999 at FAA
headquarters in Washington, D.C.; the Third Year East Coast Conference
at New York, New York, in October 1999; and a Small Business Conference
in Oklahoma City, Oklahoma, in Spring 2000. The purpose of these
outreach initiatives is to assist the agency in aggressively reaching
out to the small business communities and to reinforce the agency's good
faith commitment to reaching its MPPG goals.

In addition to these outreach initiatives, the Small Business
Utilization Staff was instrumental in assisting FAA with its FY 98 award
of a contract for Broad Information Technology Services (BITS) to
15 small business contractors, 7 of which are disadvantaged businesses
that hold current Section 8(a) certifications from the Small Business
Administration. Awards may be made to these prior approved contractors
for information technology services as the need arises over the next
five years. Due to the large number of small business contractors and
subcontractors who may receive awards and the anticipated large dollar
value, this Small Business Utilization Staff effort may greatly improve
the chances the agency may achieve its MPPG goals over the next five
years.

## Potential Effects if Efforts Are Not Continued

The evaluation team concluded that, even though the centers and most of
the regions were achieving MPPG goals, they need to continue their
efforts to ensure the agency can meet these goals. FAA's continued
failure to meet MPPG goals, especially by headquarters, could possibly
result in significant effects such as increased complaints from small
businesses that they have not been provided reasonable opportunities
under acquisition reform, or the loss of a positive public image that
FAA is making a good faith effort to support MPPG goals.

### Recommendation

The evaluation team recommends:

**3A** The Small Business Utilization Staff propose and pursue
additional incentives or other opportunities to assist FAA in meeting
its Major Procurement Program Goals.

#### Suggested Solutions:

1)  The FAA Administrator could reinstate the requirement to involve the
    Small Business Representative in all awards by amending AMS Section
    3.2.1.3.4 to state "\... individuals. However, for procurement
    requirements over \$100,000, the requesting sponsor should
    coordinate with the FAA Small Business Representative designee prior
    to advertising the announcement. FAA will work with the
    Department\...."

2)  The Small Business Utilization Staff could initiate additional MPPG
    goal awareness through:

> a\) Holding informal workshops that include the mechanics of meeting
> MPPG goals and targeting the training to technical and program
> managers. This could be performed at Air Traffic Control Association
> events such as symposiums or conferences, or at other conferences or
> sessions frequently attended by technical and program managers.

b)  Encouraging the increased participation of technical and program
    managers at procurement or other conferences by inviting them as
    speakers or targeting activities at these conferences to the FAA
    technical and program managers.

c)  Holding workshops for Integrated Product Team leads to encourage
    their assistance in attracting prime contracting firms to the
    Mentor-Protégé Program and using this program when awarding
    contracts.

```{=html}
<!-- -->
```
3)  The Small Business Utilization Staff could pursue with FAA managers
    the requirement to include MPPG goals in performance appraisals for
    procurement personnel as well as technical and program managers. The
    evaluation team found that these goals already existed in many
    procurement personnel performance appraisals.

[]{#__RefHeading___Toc453556160 .anchor}Chapter 2Program Results

The evaluation team's objective was to determine if programs were on
track to achieve success. This objective included determining if FAA
will obtain what was planned for and determining if acquisition reform
was working. To meet the objective, the evaluation team reviewed program
results in terms of supporting the FAA mission, delivering the planned
product performance, meeting customer needs, and cost and schedule
baselines. A judgmental sample of 13 programs from two of FAA's three
major appropriation funds used for acquisitions was selected. All of
these programs were managed under acquisition reform, but most had been
initiated before acquisition reform began and were far from completed.
The evaluation team found that, under acquisition reform, the agency is
on track to achieve success in ensuring programs support the FAA
mission, deliver planned product performance results, and meet customer
needs. However, the agency is not on track to meet cost and schedule
baselines approved for individual programs.

In addition, the evaluation team performed a correlation analysis of
program results to the Acquisition Management System (AMS) processes for
the sampled programs. Based on this analysis, the evaluation team was
not able to find a direct link, either positive or negative, between AMS
and program results, but did find improvements in documentation and
record-keeping since acquisition reform. Acquisition reform began just
three years ago. Since industry improvements like acquisition reform can
take up to seven years to be realized, and FAA programs are long term in
nature, determining the full success of acquisition reform may take
another four years.

# Finding 4: Program product performance results were on track, but cost and schedule were not.

To determine the level of program success under acquisition reform, the
evaluation team looked at program results in terms of five specific
elements: (1) supporting the FAA mission, (2) delivering product
performance, (3) meeting customer needs, (4) meeting cost baselines, and
(5) meeting schedule baselines. Developing products that meet the
mission and achieve the desired performance for the customer is
paramount in the public sector, i.e., federal agencies. In contrast, the
primary concern for the private sector, i.e., commercial enterprises, is
cost, which is frequently impacted by schedule. The general public
watches how much an agency spends for what it buys and when it is
received. Thus, while not as important as the mission accomplishment,
the cost and schedule baselines are perhaps the most visible area and
the most likely to receive public criticism when not met.

As shown in Table 2-1, the evaluation team selected a sample of 10
programs funded with facilities and equipment dollars and 3 programs
funded with research, engineering, and development dollars. When
separating program activities into planning and lifecycle management,
the evaluation team determined all 13 programs were managed, at least in
part, under acquisition reform, but most had been planned prior to the
implementation of acquisition reform. Because agency programs are long
term in nature and acquisition reform has been in place just three
years, the majority of the sampled programs had completed a substantial
part of their planning efforts prior to the implementation of
acquisition reform. Substantial planning efforts after the
implementation of acquisition reform in April 1996 were conducted for
only three programs in the sample.

[]{#__RefHeading___Toc453556191 .anchor}Table 2-1. Sampled Programs

+-----+-----------------------------------+----------------------------+
|     | **Program Name**                  | Primary Funding Source     |
+=====+===================================+============================+
| 1   | ACQUIRE                           | Facilities & Equipment     |
+-----+-----------------------------------+----------------------------+
| 2   | Advanced Aviation Security        | Facilities & Equipment     |
|     | Initiatives                       |                            |
+-----+-----------------------------------+----------------------------+
| 3   | Airport Surveillance Radar        | Facilities & Equipment     |
|     | (ASR-11)                          |                            |
+-----+-----------------------------------+----------------------------+
| 4   | Integrated Terminal Weather       | Facilities & Equipment     |
|     | System (ITWS)                     |                            |
+-----+-----------------------------------+----------------------------+
| 5   | National Airspace System          | Facilities & Equipment     |
|     | Infrastructure Management System  |                            |
|     | (NIMS)                            |                            |
+-----+-----------------------------------+----------------------------+
| 6   | Operational and Supportability    | Facilities & Equipment     |
|     | Implementation System (OASIS)     |                            |
+-----+-----------------------------------+----------------------------+
| 7   | Standard Terminal Automation      | Facilities & Equipment     |
|     | Replacement System (STARS)        |                            |
+-----+-----------------------------------+----------------------------+
| 8   | Weather and Radar Processor       | Facilities & Equipment     |
|     | (WARP)                            |                            |
+-----+-----------------------------------+----------------------------+
| 9   | Wide Area Augmentation System     | Facilities & Equipment     |
|     | (WAAS)                            |                            |
+-----+-----------------------------------+----------------------------+
| 10  | Year 2000 Computer Problem        | Facilities & Equipment     |
|     |                                   |                            |
|     | (Program Office)                  |                            |
+-----+-----------------------------------+----------------------------+
| 1   | Explosives and Weapons Detection  | Research, Engineering &    |
|     |                                   | Development                |
+-----+-----------------------------------+----------------------------+
| 2   | Navigation                        | Research, Engineering &    |
|     |                                   | Development                |
+-----+-----------------------------------+----------------------------+
| 3   | Weather Program                   | Research, Engineering &    |
|     |                                   | Development                |
+-----+-----------------------------------+----------------------------+
| **1 | **Total Programs in Sample**      |                            |
| 3** |                                   |                            |
+-----+-----------------------------------+----------------------------+

The facilities and equipment funded programs exist to acquire, deploy,
and improve air navigation systems and are successful upon the
deployment of the specific systems. In contrast, research, engineering,
and development funded programs exist to explore cutting edge
technology, to draft standards and specifications, and to prototype
potential solutions. These programs can be successful even if they
result in a determination that a particular technology is not suitable
for implementation.

Where applicable, the results of the three sampled research,
engineering, and development funded programs were included in the
conclusions. However, the primary conclusions were based on results from
the 10 sampled facilities and equipment funded programs. In the results,
all the sampled programs were considered equally; there were no weights
applied for program size, dollar value, scheduled time, or complexity of
the products or systems developed. Based on the evaluation of these 10
programs, the team found that 100 percent fully supported and were
consistent with the FAA mission, 90 percent were on track to deliver
product performance results, and 70 percent were meeting customer needs.
However, only 40 percent were within cost baselines, and only 20 percent
were able to maintain their planned schedule baselines. Table 2-2 below
presents the results of the five specific elements reviewed.

[]{#__RefHeading___Toc453556192 .anchor}Table 2-2. Program Results Under
Acquisition Reform

  -------------- ---------------- ------------- ------------ -------------
  **Supporting   **On Track to    **Meeting     **Meeting    **Meeting
  FAA Mission**  Deliver Product  Customer      Cost         Schedule
                 Performance**    Needs**       Baseline**   Baseline**

  100%           90%              70%           40%          20%
  -------------- ---------------- ------------- ------------ -------------

## Results Element (1): Supporting the FAA Mission

In 1994, the FAA mission was to provide a safe, secure, and efficient
global aviation system that contributed to national security and the
promotion of United States aviation. In 1996, the agency's mission was
widened to address aerospace rather than simply aviation. "Aerospace" in
this definition included the *space* comprising the earth's atmosphere
and the space beyond, while "aviation" specifically referred to the
operation of various air vehicles as well as their manufacture,
development, and design. The agency was criticized for its dual role to
promote the aerospace industry while also being responsible for
regulating aerospace safety. As a result, in 1998 FAA's mission was
narrowed to aerospace safety. Today, the FAA mission is to provide a
safe, secure, and efficient global aerospace system that contributes to
national security and the promotion of United States aerospace safety.

In this review, the evaluation team looked at the broad purposes of the
sampled programs as defined in their Mission Need Statements and
Integrated Program Plans to determine whether each program was
consistent with the current FAA mission. Also, the program information
was compared to the National Airspace System Architecture, the Capital
Investment Plan, and the Research, Engineering and Development Plan to
ensure consistency with FAA's long range strategic planning. Table 2-3
below highlights the evaluation results for supporting the FAA mission.

[]{#__RefHeading___Toc453556193 .anchor}Table 2-3. Program Results Under
Acquisition Reform, FAA Mission

  -------------- ---------------- ------------- ------------ -------------
  **Supporting   On Track to      Meeting       Meeting Cost Meeting
  FAA Mission**  Deliver Product  Customer      Baseline     Schedule
                 Performance      Needs                      Baseline

  **100%**       90%              70%           40%          20%
  -------------- ---------------- ------------- ------------ -------------

All 13 (100 percent) sampled programs were consistent with the FAA
mission. This included the 10 facilities and equipment funded programs
and the 3 research, engineering, and development funded programs. In
addition, all 13 sampled programs (100 percent) were consistent with the
agency's long range strategic planning.

## Results Element (2): On Track to Deliver Product Performance

Acquisition reform focuses on mission accomplishment rather than
compliance with rules and regulations, and as a result, AMS grants FAA
personnel greater flexibility while holding the acquisition teams
accountable for the results. The key measure of mission accomplishment
is performance results. The evaluation team defined a successful product
performance result as one where the new system performs successfully or,
in the case of products that have not yet been fielded, is expected to
perform successfully. While achieving successful product performance
results within planned cost and schedule baselines is the ultimate goal
of acquisition reform, the evaluation team considered cost and schedule
results separately from achieving product performance for the purposes
of this evaluation.

The evaluation team interviewed Integrated Product Team leads, Product
Team leads, and customers to determine whether each program had met or
was projected to meet the planned program performance results. Baseline
documentation was reviewed to determine if technical performance and
benefits will fail to achieve at least 50 percent of the performance
goal as established for the program. Customers were interviewed, where
appropriate, to verify whether the fielded product fulfilled the
established requirements. Also, Acquisition Review data and reports were
compared to Requirements Documents to verify whether the intended
purpose was accomplished. Table 2-4 below highlights the evaluation
results for delivering planned product performance.

[]{#__RefHeading___Toc453556194 .anchor}Table 2-4. Program Results Under
Acquisition Reform, Product Performance

  -------------- ---------------- ------------- ------------ -------------
  Supporting FAA **On Track to    Meeting       Meeting Cost Meeting
  Mission        Deliver Product  Customer      Baseline     Schedule
                 Performance**    Needs                      Baseline

  100%           **90%**          70%           40%          20%
  -------------- ---------------- ------------- ------------ -------------

Since research, engineering, and development programs are designed for a
purpose other than producing a specific product, the evaluation team did
not attempt to measure the level of product performance results attained
for the three programs of that type in the sample. However, 9 of the 10
(90 percent) facilities and equipment funded programs had already
delivered or were on track to deliver planned product performance
results. Specifically:

-   The Advanced Aviation Securities Initiative program met product
    performance requirements within the available technology.

```{=html}
<!-- -->
```
-   The WARP program completed its initial stage of planned product
    results with a commercial system.

-   The ASR-11 program product delivery to the test and evaluation site
    will have been accomplished by June 1999.

-   The ITWS, OASIS, WAAS, and the Program Office of the Year 2000
    Computer Problem were all projected to achieve the desired product
    performance results.

-   The ACQUIRE program, which replaced two previous procurement
    systems, will meet the product requirement to improve system
    efficiency over previous systems.

-   The STARS program is moving forward with a revised planning effort
    to accommodate computer-human interface issues.

NIMS was the one program not on track to deliver planned product
performance results. This occurred primarily due to insufficient
baseline funding of Phase 1 as well as continued cuts in the approved
baseline funding stream, which led to continuous replanning activities.
Both a business process reengineering and product acquisition were
performed simultaneously, using a spiral development approach. Under
this approach, system requirements were defined as the program evolved
and were not fully defined prior to contract award. The evaluation team
believes this approach lent itself to the increased risk of cost and
schedule growth later in the program. While the NIMS program followed
AMS to conduct an affordability assessment and complete a risk
mitigation strategy, the affordability assessment was forced to fit a
predetermined funding level, and the risk mitigation strategy was
insufficient to address uncertainties in requirements. Other impacts to
the program were: (1) while prototyping was accomplished, the data
produced were not fully used to determine the feasibility of system
design, and (2) the NIMS Product Team and the operations customer did
not always work together in a cohesive manner.

## Results Element (3): Meeting Customer Needs

The evaluation team defined "customer" as those entities or individuals
both inside and outside the FAA who will use or benefit from agency
programs. This would include air carriers, airports, and other
government agencies such as the Department of Defense, as well as
employees and sponsoring lines of business within the FAA.

To evaluate whether the sampled programs were meeting customer needs,
the evaluation team first identified the lines of business supporting or
benefiting from these programs. Integrated Product Team and Investment
Analysis Team member lists were reviewed to identify and interview
customers who may have been active members on the program. Interviews
were conducted to determine customer involvement in the planning process
and to learn whether they believed their participation was effective and
meaningful. Mission Need Statements, investment analysis documents, and
Requirements Documents were compared to verify whether customer needs
were identified, documented, and addressed in the planning process. The
evaluation team also interviewed Integrated Product Development System
team leads to learn how customer needs were revalidated, how changing
needs were identified and prioritized, and how changing needs were
incorporated into the product. For programs with a fielded product, the
evaluation team interviewed customers to find out whether they were
satisfied with the product performance results. Table 2-5 below
highlights the evaluation results for meeting customer needs.

[]{#__RefHeading___Toc453556195 .anchor}Table 2-5. Program Results Under
Acquisition Reform, Customer Needs

  -------------- ---------------- ------------- ------------ -------------
  Supporting FAA On Track to      **Meeting     Meeting Cost Meeting
  Mission        Deliver Product  Customer      Baseline     Schedule
                 Performance      Needs**                    Baseline

  100%           90%              **70%**       40%          20%
  -------------- ---------------- ------------- ------------ -------------

Seven of the 10 (70 percent) facilities and equipment funded programs
were effective in meeting customer needs. The evaluation team also found
that, for the three research, engineering, and development funded
programs, customers were involved in the planning process, and customer
needs were identified and documented; however, customer participation in
these programs was more limited.

The success in meeting customer needs may be attributed in part to the
extensive involvement of customers in the planning process. For example,
the ITWS Product Team actively involved focus groups representing the
customer community in the development process from its inception. Focus
group ideas and concerns were incorporated into the ITWS product. This
practice resulted in a highly satisfied group of ITWS product customers,
including air traffic control supervisors, traffic management unit
coordinators, and air traffic controllers. The ITWS program had strong
industry support. The Advanced Aviation Security Initiatives program
also involved customers actively. While air carrier customer
participation was voluntary, it was strongly encouraged. As a result of
this good working relationship, customer concerns and technical issues
were resolved to the satisfaction of those involved.

Three facilities and equipment funded programs did not achieve customer
satisfaction:

-   ACQUIRE customers did not support the solution selected for ACQUIRE
    and did not believe their specific requirements were met. This
    occurred because customer input was not incorporated during
    development and testing. The evaluation team found customers did not
    provide timely input during requirements and testing, and customer
    needs were only partially documented. In addition, numerous system
    change requests were submitted but not adequately addressed.

```{=html}
<!-- -->
```
-   The NIMS program had early customer involvement and had adequate
    support from the sponsoring line of business. Using the spiral
    development approach, however, the system requirements were to be
    defined as the program evolved. As a result, identified customer
    needs were not fully clarified. The NIMS product plan did not
    account for this development approach.

-   The STARS program Product Team had not gained the full support of
    the union members in addressing computer-human interface issues.
    Although this issue has since been resolved, the evaluation team
    counted STARS among the 30 percent that did not meet customer needs.

## Results Elements (4) and (5): Meeting Cost and Schedule Baselines

Baselines are developed to establish an initial basis from which to
measure success in achieving the planned objectives. The agency develops
baselines for cost, schedule, technical performance, and benefits.
Baselines for cost represent the budget objective and those for schedule
establish the expected timeframe for accomplishing critical or key
steps.

To evaluate whether the sampled programs met cost and schedule
baselines, the evaluation team obtained baseline information and, if
available, the Joint Resources Council approved formal baselines for
each of the 10 facilities and equipment funded programs. If formal
baselines had not been established, the evaluation team relied on
project plans to evaluate the program's success in meeting the planned
cost and schedule objectives. Management reporting documents, including
Baseline Management Notices, status reviews, status reports, and monthly
program reviews, were obtained to measure the level of deviation from
the initial baseline to the current or projected baseline. It was beyond
the scope of this evaluation for the team to assess whether or not the
approved baselines were realistic. To do this would require in-depth,
individual program reviews.

Baseline information contained in the Research Project Description was
collected for the research, engineering, and development funded
programs. The cost baselines in the Research Project Descriptions
reflect the level of funding the program requested and do not represent
a budget or cost agreement. As a result, the evaluation team did not
attempt to measure the rate of baseline deviation for the research,
engineering, and development funded programs. Table 2-6 highlights the
evaluation results for meeting cost and schedule baselines.

[]{#__RefHeading___Toc453556196 .anchor}Table 2-6. Program Results Under
Acquisition Reform, Baselines

  -------------- ---------------- ------------- ------------ -------------
  Supporting FAA On Track to      Meeting       **Meeting    **Meeting
  Mission        Deliver Product  Customer      Cost         Schedule
                 Performance      Needs         Baseline**   Baseline**

  100%           90%              70%           **40%**      **20%**
  -------------- ---------------- ------------- ------------ -------------

At the time of this review, only 4 of the 10 (40 percent) facilities and
equipment funded programs were meeting their baselines for cost, and
only 2 (20 percent) had not already experienced a schedule slip.
However, as shown in Table 2-7, 6 of the 10 (60 percent) were within a
variance range of 10 percent for cost, and 5 of the 10 (50 percent) were
within a variance range of 10 percent for projected schedule. A variance
range of 10 percent was considered acceptable because the FAA
Administrator is not required to *consider* terminating programs until
the cost or schedule baselines experience a variance greater than 10
percent.

[]{#__RefHeading___Toc453556197 .anchor}Table 2-7. Programs Within 10
Percent Variance Range

  ----------------------------------- -----------------------------------
  **Within 10% Variance of Cost       **Within 10% Variance of Schedule
  Baseline**                          Baseline**

  60%                                 50%
  ----------------------------------- -----------------------------------

### Impact of Not Meeting Cost or Schedule Baselines

The evaluation team noted that not meeting cost or schedule baselines
did not always have a severe effect on program results and may not have
resulted in program failure, but it did have an impact. The evaluation
team found three primary impacts related to not meeting cost or schedule
baselines. First, even though programs were delivering product
performance and generally meeting customer needs, programs that were not
meeting cost or schedule baselines were unable to deliver *all* products
or product performance requirements or could not meet *all* initially
defined customer needs. Second, because of the interdependency of
programs, schedule delays and cost increases in one program had an
adverse impact on other programs. Third, programs experiencing great
difficulty in meeting cost or schedule baselines could be considered for
cancellation.

#### Unable to Deliver ALL Products or Meet ALL Customer Needs

Whether or not programs were on track to deliver product performance or
customers were generally satisfied, if programs could not maintain cost
and schedule baselines, some part of the program plan could not be
delivered. This occurred in the WAAS, STARS, and NIMS programs.
Specifically:

-   The WAAS program was on track to deliver planned product performance
    and the WAAS Phase 1 hardware was installed throughout the National
    Airspace System. However, because of technical difficulties
    associated with the software design and integration, operational
    software had not been delivered. As a result, implementation of WAAS
    Phase 2 and Phase 3 requirements will be delayed. The WAAS program
    was within a 10 percent variance range for schedule, but not for
    cost.

-   The STARS program was on track to deliver product performance after
    a replanning effort. The STARS program could not be maintained to
    deliver the product performance as originally planned because of
    significant costs to remedy computer-human interface problems and
    delays created by those problems. The system would not support
    customer needs until identified computer-human interface issues were
    corrected. In April 1999, the FAA, along with the National Air
    Traffic Controllers Association and the Professional Airways System
    Specialists, announced a revised implementation plan for the STARS
    program. The revised plan will focus on developing the full STARS as
    soon as possible while simultaneously meeting short-term
    requirements for controller displays at a small number of FAA
    facilities. The STARS program was not within a 10 percent variance
    range for cost.

-   NIMS Phase 1 was not on track to deliver the product performance
    sought by the customer. This program was scaled back to match the
    output possible with the available funding. As a result, only a
    limited subset of the original NIMS Phase 1 product performance will
    be delivered. The NIMS program was not within a 10 percent variance
    range for either cost or schedule.

#### Interdependencies and Adverse Impact on Other Programs

Most FAA programs are interdependent. Because of these
interdependencies, cost increases and schedule delays for one program
also have an adverse impact on other programs or products. For example,
additional Year 2000 testing requirements will delay the WARP program
schedule for Stage 1/2. This will delay the availability of next
generation weather radar on controller displays. The initial stage of
WARP, Stage 0, was able to deploy three to six months ahead of schedule,
but the following stages (Stages 1 and 2, which were combined into a
single Stage 1/2, and Stage 3) will include system enhancements and
National Airspace System interfaces. There is an anticipated schedule
delay in the operational testing and evaluation of WARP because of the
unavailability of the Display System Replacement interface. Also, a
requirement was added to develop the initial capability of the FAA bulk
weather telecommunications gateway to support Free Flight Phase One. As
a result of this schedule slip, en route controllers will not have the
next generation weather radar data available on their scopes via the
Display System Replacement until later than originally planned.

#### Potential Cancellation

Programs experiencing great difficulty in meeting cost or schedule
baselines may be considered for cancellation. Public Law 104-264
requires certain actions by the FAA Administrator when any facilities
and equipment acquisition program initiated after November 1996 breaches
its baseline. Specifically, the FAA Administrator shall terminate that
program if it is more than 50 percent over cost or more than 50 percent
behind schedule as determined by the cost and schedule goals[^6]
established for the program. However, there is an exception in the law
in which the FAA Administrator has the authority to continue an
acquisition program if that termination would be inconsistent with the
development or operation of the national air transportation system in a
safe and efficient manner. In addition, this public law states that the
FAA Administrator shall *consider* terminating any substantial
acquisition program that is more than 10 percent over cost or more than
10 percent behind schedule as determined by the cost and schedule
goals^1^ established for the program. Research, engineering, and
development funded programs are not subject to these termination
criteria. Table 2-8 shows the relationship between the baseline variance
requiring program termination and one that requires only *consideration*
of program termination.

[]{#__RefHeading___Toc453556198 .anchor}Table 2-8. Program Termination
Requirements for Cost and Schedule

+--------------+---------------------------+--------------------------+
| Baseline     | Consider Terminating      | Terminate Program \*     |
| Element      | Program \*                |                          |
+==============+===========================+==========================+
| > Cost       | \>10% over cost goal      | \>50% over cost goal     |
+--------------+---------------------------+--------------------------+
| > Schedule   | \>10% behind schedule     | \>50% behind schedule    |
|              | goal                      | goal                     |
+--------------+---------------------------+--------------------------+
| \* Public    |                           |                          |
| Law 104-264, |                           |                          |
| Federal      |                           |                          |
| Aviation     |                           |                          |
| Rea          |                           |                          |
| uthorization |                           |                          |
| Act of 1996. |                           |                          |
+--------------+---------------------------+--------------------------+

Both the NIMS program and the STARS program have projected cost and
schedule variances in excess of 10 percent. The NIMS program breached
the 50 percent thresholds for both cost and schedule growth. The
Associate Administrator for Research and Acquisitions transmitted a
breach notice that included the NIMS program to the FAA Administrator in
March 1999. Also, the STARS program breached its cost and schedule
baselines. The FAA Administrator asked for continued support for the
STARS program and emphasized that the agency is committed to a
successful implementation of this program.

### Systemic Problems Identified

Based on data collected for the 10 facilities and equipment funded
programs, the evaluation team found three systemic problems that
contributed to the difficulty agency programs were having in meeting
cost or schedule baselines. Those systemic problems included: (1) new or
poorly understood requirements, (2) underestimating technical
complexity, and (3) funding shortfalls. While all programs may have
experienced one or more of these systemic problems, for this report the
evaluation team identified only those programs where the inability to
meet cost or schedule baselines was directly attributed to one of these
three factors. The evaluation team limited the results to the
10 facilities and equipment funded programs because only these programs
are subject to termination if cost or schedule goals are not met.
Because acquisition reform has been in place only three years, and most
of these programs have been in existence longer than that time, most
sampled programs had substantial planning efforts before AMS processes
took effect. Some of the systemic problems identified were attributed to
actions that occurred prior to acquisition reform. Table 2-9 illustrates
the problems experienced by programs sampled. Specifically, the
evaluation team found that, of the 10 programs:

-   5 had difficulty meeting cost or schedule projections because of new
    or poorly defined requirements.

-   5 had cost increases or schedule delays resulting from
    underestimated technical complexities.

-   3 experienced significant problems as a result of funding
    shortfalls.

[]{#__RefHeading___Toc453556199 .anchor}Table 2-9. Systemic Problems
Contributing to Cost and Schedule Growth

+--------------------------+--------------+----------------+----------+
|                          | ##           |                |          |
|                          | ####### Syst |                |          |
|                          | emic Problem |                |          |
+==========================+==============+================+==========+
| **Program**              | **New or     | **             | *        |
|                          | Poorly       | Underestimated | *Funding |
|                          | Understood   | Technical      | Shor     |
|                          | Re           | Complexities** | tfalls** |
|                          | quirements** |                |          |
+--------------------------+--------------+----------------+----------+
| ACQUIRE                  | X            |                |          |
+--------------------------+--------------+----------------+----------+
| Advanced Aviation        |              |                | X        |
| Security Initiatives     |              |                |          |
+--------------------------+--------------+----------------+----------+
| ASR-11                   | X            |                | X        |
+--------------------------+--------------+----------------+----------+
| ITWS                     |              | X              |          |
+--------------------------+--------------+----------------+----------+
| NIMS                     | X            | X              | X        |
+--------------------------+--------------+----------------+----------+
| OASIS                    | X            |                |          |
+--------------------------+--------------+----------------+----------+
| STARS                    | X            | X              |          |
+--------------------------+--------------+----------------+----------+
| WARP                     |              | X              |          |
+--------------------------+--------------+----------------+----------+
| WAAS                     |              | X              |          |
+--------------------------+--------------+----------------+----------+
| Year 2000 Computer       |              |                |          |
| Problem (Program Office) |              |                |          |
+--------------------------+--------------+----------------+----------+

#### New or Poorly Understood Requirements

Adding new requirements or modifying initial requirements caused an
increase in cost or schedule for 5 of the 10 sampled programs. In some
cases, the additional requirements resulted from information obtained
during testing of the product. Adding or modifying requirements is not
presented here as an activity that should be avoided. In a rapidly
changing technological environment, it would be unreasonable to expect
the agency to establish a set of rigid requirements today for programs
that will be planned and implemented over the next decade or more.
However, the evaluation team did identify changing requirements as an
area that affected cost or schedule growth for half the sampled
programs. The agency and program offices should be aware that
requirements definition is a critical planning element, and changing
requirements may have a significant impact on a program's ability to
maintain cost and schedule baselines.

-   The ACQUIRE program was a commercial off-the-shelf procurement
    system to replace the System for Acquisition Management and the
    Procurement Automation System. The ACQUIRE Product Team discovered
    the workstations were unsuited to handle the large disk space
    required. As a result, a new product, which was an unforeseen
    requirement, was purchased and installed to accommodate the system.

-   The STARS program had unplanned custom software development as a
    result of unrecognized computer-human interface requirements. These
    poorly understood interface requirements represented a change in the
    STARS program technical baseline, one of the four baselines used by
    the agency to measure a program's progress. According to the draft
    Baseline Management Notice dated November 1998, this change to the
    technical baseline had the potential to increase development costs
    an additional \$293.9 million.

-   The NIMS program used a spiral development process which
    necessitated refining requirements on an ongoing basis. The program
    plan did not have the means to quantify the evolution of
    requirements and the resulting impact on cost and schedule
    baselines.

-   Four new requirements were projected for the ASR-11 program, which
    will result in increased cost and schedule growth. This is a joint
    program led by the Department of Defense. The program followed
    Federal Acquisition Regulation guidelines and was not under
    acquisition reform during the planning process. However, these new
    requirements will affect cost and schedule.

-   New requirements were also added to the OASIS program. This program
    was intended to be a commercial-off-the-shelf program, but evolved
    into a developmental program.

#### Underestimating Technical Complexity

The evaluation team defined technical complexity as software development
and interface issues and challenges related to commercial off-the-shelf
systems. Based on this definition, the evaluation team found 5 of 10
programs had underestimated the technical complexities surrounding the
programs, which resulted in overly optimistic cost or schedule goals.
While cost or schedule growth resulting from underestimated complexities
was not always significant, it did affect half the sampled programs.

Unexpected software development challenges and interface problems caused
some cost or schedule growth in the WARP, WAAS, ITWS, and STARS
programs. The STARS program and the NIMS program also had additional
cost or schedule growth from technical challenges when commercial
off-the-shelf items were not sufficient (while commercial off-the-shelf
items were also not sufficient for the OASIS program, and that
insufficiency resulted in additional requirements, the evaluation team
captured that in the previous category for new or poorly understood
requirements and did not count it again here as a technical complexity
issue). The evaluation team found that these technical complexities
increased cost, delayed schedules, and had the potential to interfere
with accomplishing program performance results. Specifically:

-   The WARP program software development Stage 1/2 was slower than
    expected. The contractor experienced software development
    difficulties, causing the software code to be completed four months
    later than originally planned. The contractor then had difficulty
    with software integration and testing, which further delayed the
    test readiness review, factory acceptance testing, and site
    acceptance testing. The original communication design did not meet
    the program's requirements. Rewriting the software code to
    accommodate the design change resulted in an addition of eight weeks
    to the schedule.

-   The WAAS program office underestimated the complexity of the
    correction and verification software system needed. The contractor
    for WAAS experienced technical difficulties with software design and
    integration, which resulted in non-delivery of critical software.
    The missed delivery, together with a minor imapct from funding
    reductions, prompted an estimated 14-month delay of the WAAS initial
    operating capability. This schedule growth was still within a 10
    percent variance range for schedule at the time of this review. Cost
    growth was also associated with these technical difficulties and
    with the schedule delay. The WAAS program exceeded a 10 percent
    variance range for cost.

-   The ITWS program experienced technical complexity related to
    software development and integration impacting cost and schedule.
    The program office anticipates additional schedule delays due to the
    complexity of the National Weather Service filter unit development.
    Software productivity, which has been slower than planned, is also
    impacting cost.

-   The STARS program had substantial unplanned custom software
    development to accommodate computer-human interface requirements.
    Cost and schedule baselines could not be met because of these new
    human factor activities. The STARS contract was awarded in September
    1996 primarily as a commercial off‑the‑shelf system. The magnitude
    of unplanned custom software development to accommodate new
    computer-human interface requirements and other requirements for
    STARS took the program out of the commercial off‑the‑shelf realm and
    made it a developmental program. The FAA determined that the cost
    and schedule impacts for the STARS program were consistent with a
    developmental program of this size.

-   Some of the NIMS program cost growth was attributed to
    underestimating commercial off‑the‑shelf costs and complexities.
    NIMS also had information security costs and startup costs for
    Phase 2 that were not included in the original Acquisition Program
    Baseline.

#### Funding Shortfalls

Funding shortfalls prevented some programs from meeting their cost
baselines and caused programs to scale back on accomplishing planned
product performance results. In two cases, funding levels did not remain
adequate to accomplish all planned product performance results. In
another case, the funding level was not initially adequate to accomplish
all planned product performance results. Funding shortfalls occurred
primarily because the budgeting process does not complement the AMS
process for establishing and maintaining cost baselines. The budgeting
process is generally completed on an annual basis and without firm, long
term commitments to individual programs. In this process, FAA determines
its program priorities and then submits its budget request to the
Department of Transportation to be included in the Department's budget
request. This is then submitted to the Office of Management and Budget.
The budget eventually is submitted to the Congress, where funds are
actually appropriated. Program funding can be changed at any point by
the FAA, the Department of Transportation, the Office of Management and
Budget, or the Congress. In contrast, the AMS process requires program
cost baselines for multiple years, making funding assumptions for each
program throughout its full lifecycle, which usually spans a period
greater than one year. For the sampled programs, the evaluation team
found the following:

-   The ASR-11 program experienced a \$47.4 million reduction in
    formulating FAA's FY 00 budget request. As a result, the program
    will defer acquiring up to 11 radar systems from its planned
    procurements. Although the ASR-11 program had an approved
    Acquisition Program Baseline, the program received lower levels of
    funding and was faced with continual budget cuts.

-   The NIMS program experienced funding instability from the beginning
    of the program. In addition to insufficient initial funding, the
    NIMS program experienced continual funding cuts. The evaluation team
    found the NIMS program office forecasted a 58 percent increase in
    cost and a 123 percent increase in schedule, attributed
    predominantly to funding cuts cumulating \$39 million through FY 00.

Funding also was indicated as a cause for schedule delays. Significant
funding delays impacted the Advanced Aviation Security Initiatives
program's ability to meet its schedules. Public Law 104-208 provided for
\$144.2 million in funds to purchase and assist in installation of
advanced security equipment. Subsequent to this law, the FY 98 budget
for the Advanced Aviation Security Initiatives program was reduced to
zero. Eventually, the President signed an emergency supplement of \$100
million for FY 99, but the funds were not released to the program until
air carriers agreed to maintain the equipment and provide training for
operators. The program office did not receive verification of this
agreement from all air carriers until the end of January 1999. At that
time, the Advanced Aviation Security Initiatives program received only
\$50 million of the planned \$100 million. These funding delays impacted
the planned schedule. There was insufficient FY 98 funding for
installation of units already procured for the Advanced Aviation
Security Initiatives program, and site preparation work for additional
installations was delayed due to lack of FY 99 funds.

While other programs also experienced funding cuts, those funding issues
were not considered significant or were not the primary cause of a
program's challenges in meeting cost or schedule baselines or in
achieving product performance results. As an example, FY 99 funding for
the ITWS program was below the program's current baseline, resulting in
the ITWS program office anticipating a schedule delay of at least two
years for preplanned product improvements. The ITWS Product Team did not
consider this critical in meeting the overall product performance
results. Also, while the STARS program had more serious funding
problems, these shortfalls were caused by changes in the program's
requirements and were unrelated to initial budgeting issues.

### Recommendations

The evaluation team recommends:

**4A** The co-chairs of the Integrated Product Leadership Team work with
the Integrated Product Team leads to establish and implement a plan to
reduce the risk of unexpected requirements, including those related to
software and interface issues and human factors processes.

#### Suggested Solutions:

1)  As required under the Acquisition Management System, each Integrated
    Product Team should ensure the right team composition is assigned to
    the program. If that is not feasible, Product Teams working with a
    potential software solution could coordinate with a knowledgeable
    central information system office during the planning stages of a
    program to ensure all known areas are considered. For administrative
    systems and programs, the Chief Information Officer could be
    responsible for providing this specific guidance. For air traffic
    control systems, the Air Traffic Services line of business could be
    responsible for providing this specific guidance.

2)  The Integrated Product Team leads could work together to develop an
    ad hoc peer review system to leverage program experiences, act as
    advisory bodies, and provide additional guidance during the planning
    phase from knowledgeable FAA and contractor personnel unrelated to
    the specific program. The peer review team, however, should not
    serve as an approval board, which could create delays in the system
    development process. This peer review system may help with
    identifying cost, schedule, technical, and risk areas early in the
    system development process. The peer review system may also be
    involved during critical design reviews and test readiness reviews.
    The elements of the peer review should be determined to provide the
    most useful feedback for the Integrated Product Teams. For example,
    the use of commercial off-the-shelf items when computer-human
    interface issues are involved could be considered for a peer review.

3)  The Integrated Product Teams could be required to develop and review
    lessons learned in the FAA Acquisition System Toolset (FAST)
    specifically related to requirements, software, and interface issues
    experienced by other teams on a regular basis throughout the
    program's lifecycle. Also, Integrated Product Teams could be
    required to enter all experiences, positive and negative, in FAST.

**4B** The FAA Acquisition Executive take the lead for the Joint
Resources Council to establish and implement a plan to mitigate negative
effects of budget cuts on programs.

#### Suggested Solutions:

1)  The System Engineering/Operational Analysis Team (SEOAT) could
    recommend budget cuts from only the lowest priority ranked programs,
    e.g., 25 percent. The budgets for the remaining programs,
    e.g., 75 percent, could then remain intact. Also, the SEOAT could
    recommend eliminating low priority programs to preserve the funding
    to higher priority programs. The interdependency of programs would
    have to be considered for any potential changes or cancellations.

2)  The Joint Resources Council could hold meetings specifically
    designated to decide if rebaselining for specific programs as a
    result of funding shortfalls is necessary. These special Joint
    Resources Council meetings should be held at least once, but no more
    than twice, each year.

3)  Each Integrated Product Team could establish and work toward a
    specified percentage below 100 percent of the approved acquisition
    program cost baseline. Incentives to meet the lower goals could be
    developed.

4)  The Product Teams impacted by budget cuts could be permitted to
    reevaluate, and possibly reduce, the number of products or systems
    they are expected to deliver. In addition, the Product Team could
    identify training or support areas to be reduced to accommodate
    budget cuts.

# Finding 5: Acquisition Management System processes did not have a direct impact on program results.

The evaluation team compared the program results attained with the AMS
processes followed for each sampled program to evaluate whether
identified program strengths and weaknesses were attributable to changes
brought about by acquisition reform. The evaluation team:

-   Documented whether formal baselines were established and approved
    and whether they included all funding types.

-   Identified whether processes were in place to monitor the program's
    ability to meet planned program performance and whether the program
    established and followed a formal process for monitoring contractor
    performance.

-   Verified whether affordability assessments for investment decisions
    were prepared and whether they included all funding requirement
    sources and all lifecycle phases.

-   Confirmed whether program plans included risk assessments and risk
    mitigation strategies.

-   Verified whether sponsoring lines of business were involved in the
    planning process.

-   Reviewed Mission Need Statements, Requirements Documents, and
    investment analysis reports.

-   Reviewed memoranda from the Joint Resources Council relating to
    program funding and solution selection.

-   Verified whether multiple levels of management reviews were active
    for the programs.

For the purposes of correlating AMS processes to program results, the
evaluation team limited its review of the above activities to verifying
whether the processes were completed and documented. The quality of the
data provided and the level of effort afforded each process were not
evaluated except in those cases where the program did not achieve
positive results.

Since processes for research, engineering, and development funded
programs were not addressed specifically in AMS when it was implemented
in April 1996, the evaluation team restricted the correlation review
activity to the 10 facilities and equipment funded programs. FAA is in
the process of developing specific processes for the research,
engineering, and development funded programs and will incorporate those
processes into AMS when completed.

Based on the review of activities completed and results attained for the
10 programs, the evaluation team did not find that AMS processes had a
direct impact on program results. Neither a strong positive nor a strong
negative causal relationship was found between program results and AMS
processes. All programs in the sample supported the FAA mission,
regardless of whether they followed AMS. Most of the sampled programs
were on track to deliver product performance results and meet customer
needs. Some of these programs followed AMS processes, and some did not.
The evaluation team found that following AMS processes did not guarantee
delivering product performance results, and not following AMS processes
did not preclude success. Fewer than half of the facilities and
equipment funded programs were able to meet cost and schedule baselines.
This occurred even though all but two programs had established formal
baselines and all programs had a process in place to monitor government
program performance and contractor performance.

## Overall Impact of Acquisition Reform and AMS

The evaluation team did attempt to correlate AMS processes to program
results but was unable to link them. The reader should not attempt to
draw firm conclusions about the impact of acquisition reform or AMS,
either positive or negative, on program results based on this
correlation effort. Agency programs are long term in nature and stretch
well beyond the three years that acquisition reform has been in place.
In addition, while all the sampled programs were managed under
acquisition reform, most had completed their planning efforts before
acquisition reform was in place. Only three programs (the Program Office
of the Year 2000 Computer Problem program, the ACQUIRE program, and the
Advanced Aviation Security Initiatives program) conducted substantial
planning efforts after the implementation of acquisition reform. The
other programs had completed planning efforts prior to acquisition
reform. In addition, most programs evaluated did not have a fielded
product at the time of this review.

In any industry or agency, developing and implementing a completely new
process or system could reasonably be expected to take several years
before the full benefits could be realized. Based on the average
duration for process innovation cited in the Program Evaluation Branch's
*Evaluation of FAA Acquisition Reform: The First Two Years:
April 1996-March 1998*, dated May 1998, industry improvement efforts
averaged 4.2 years, and government organizations' efforts averaged 5.1
years. The range of duration can span from 3.5 to 7 years before full
benefits are realized. This is the case with acquisition reform. After
the third year, it is still too early to state the effect of acquisition
reform on program results with absolute certainty.

The evaluation team did find that the use of AMS resulted in definite
improvements in some areas. For example, the evaluation team found more
supporting documentation for programs managed under acquisition reform
than for programs managed prior to acquisition reform. In addition,
these supporting documents tended to be more consistent among the
various programs across the agency, which will increase the usefulness
of the data for decision making purposes.

[]{#__RefHeading___Toc453556171 .anchor}Chapter 3Recommendation
Follow-up on Previous Acquisition Reform Evaluations

This chapter addresses the status of recommendations found in four
previous evaluation reports related to acquisition reform. Those reports
are: the *Evaluation of FAA Acquisition Reform: The First Year: April
1996-March 1997*, prepared by the FAA Program Evaluation Branch; the
*Evaluation of FAA Acquisition Reform: The First Two Years: April
1996-March 1998*, also prepared by the FAA Program Evaluation Branch;
the contractor report, *Independent Assessment of the FAA's Acquisition
Management System,* prepared by Booz•Allen & Hamilton, Inc.; and the
*Impact of Acquisition Reform on Awards to Disadvantaged Businesses,*
prepared by the FAA Program Evaluation Branch. Only recommendations 1A
and 1B from the disadvantaged business report are discussed in this
report, because these were the only two recommendations in that report
relating to acquisition reform.

To determine the status of these recommendations, the evaluation team
contacted the responsible offices, obtained information on the
implementation status, and, wherever possible, verified the status.
Based on this information, the evaluation team categorized each
recommendation as completed, in progress, or having no action. The
evaluation team determined that, of the 46 recommendations, 13 were
completed, 30 were in progress, and 3 had no action. To understand the
context of all these recommendations, please refer to the appropriate
report. Figure 3-1 illustrates the status of the recommendations, and
specific information can be found in Tables 3-1, 3‑2, 3-3, and 3-4. The
following definitions were used:

-   Completed---No further action is required to address the
    recommendation.

-   In Progress---There is an ongoing effort to address the
    recommendation.

-   No Action---There has been no action to address the recommendation.

![](media/image5.wmf){width="2.748611111111111in"
height="2.4298611111111112in"}

[]{#__RefHeading___Toc453556215 .anchor}Figure 3-1. Status of
Recommendations

[]{#__RefHeading___Toc453556200 .anchor}Table 3-1. Recommendation
Actions to FAA's Report for the First Year, dated May 1997

  --------------------------------------------------------------------------------
  **Rec.   Recommendation           **Status**   **Rationale for Status**
  \#**                                           
  -------- ------------------------ ------------ ---------------------------------
  \[1\]    Establish a focused,     In Progress  A performance plan for the
           rigorous plan for the                 Research and Acquisitions line of
           Integrated Product Teams              business has been established
           to reduce the time to                 with a goal to "...reduce the
           field new systems by 50               cost of acquisitions by 20% and
           percent and reduce                    the time required to acquire and
           acquisition costs by 20               commission systems by 50%..."
           percent. Establish an                 Annual and quarterly reports that
           appropriate, effective,               contain the status of this goal
           reporting system to                   are issued.
           monitor progress.                     

  \[2\]    Obtain the               Completed    On June 2, 1997, the Acting
           Administrator's approval              Administrator approved the FAA
           of the revised                        Acquisition Management System
           Acquisition Management                revision.
           System core policy.                   

  \[3\]    Develop an integrated    In Progress  FAA has numerous processes,
           metrics program and                   reports, and organizations that
           educate the workforce on              focus on metrics, but there is
           its use and value.                    still no integrated metrics
                                                 program for the agency.

  \[4\]    Establish and enforce a  In Progress  The Integrated Baseline
           baseline management                   Establishment and Management
           policy and procedures,                (I‑BEAM) concept was approved by
           and expedite the                      the Joint Resources Council as an
           establishment of                      integrated, multi-level process
           baselines for legacy                  for establishing and managing
           programs.                             program baselines and the
                                                 guidelines were published in
                                                 March 1999. Some additional
                                                 baselines have been established,
                                                 including baselines for legacy
                                                 programs, but not all programs
                                                 have baselines.

  \[5\]    Complete the solution    In Progress  Solution implementation guidance
           implementation guidance               has been completed, and
           for systems/software as               additional guidance is being
           soon as possible and                  developed.
           expedite the remaining                
           processes, guidance, and              
           tools.                                
  --------------------------------------------------------------------------------

Table 3-1. Recommendation Actions to FAA's Report for the First Year,
dated May 1997 (continued)

  ---------------------------------------------------------------------------------
  **Rec.   Recommendation            **Status**   **Rationale for Status**
  \#**                                            
  -------- ------------------------- ------------ ---------------------------------
  \[6\]    Prioritize programs,      In Progress  Some prioritization does occur
           functions and processes                when program resources are
           and reallocate resources               reallocated. The evaluation team
           accordingly, eliminating               particularly noted this with the
           low-priorities.                        Systems Engineering/Operational
           Centralize processes                   Analysis Team. However, there is
           where efficiency or                    no evidence of a formalized,
           effectiveness gains are                centralized structure to
           possible.                              implement this recommendation.

  \[7\]    Merge the JRC \[Joint     In Progress  There have been discussions about
           Resources Council\] and                merging these Councils, but there
           RMC \[Resource Management              continues to be opposition to
           Council\] into a single                this recommendation. No final
           Assistant and Associate                decision on merging the Councils
           Administrator-level body               has been made.
           to optimize investment                 
           decisions and resource                 
           allocation across all FAA              
           appropriations. An                     
           expanded SEOAT \[Systems               
           Engineering/Operational                
           Analysis Team\] could                  
           support the merged                     
           council.                               

  \[8\]    Fully fund and staff the  Completed    The Office of Dispute Resolution
           ODR \[Office of Dispute                is now funded and staffed, and
           Resolution\] and complete              the processes and procedures have
           the required processes                 been completed.
           and procedures.                        

  \[9\]    Determine the             Completed    SEDB guidance has been
           requirements for, and                  established and implemented, and
           articulate the benefit of              the Small Business Utilization
           SEDB contracts and                     Staff has articulated the
           implement guidance                     benefits of SEDB awards at
           accordingly.                           forums, workshops, and
                                                  conferences.
  ---------------------------------------------------------------------------------

[]{#__RefHeading___Toc453556201 .anchor}Table 3-2. Recommendation
Actions to FAA's Report for the First Two Years, dated May 29, 1998

  -------------------------------------------------------------------------------
  **Rec.   **Recommendation**              **Status**   **Rationale for Status**
  \#**                                                  
  -------- ------------------------------- ------------ -------------------------
  1A       The FAA Administrator ensure    In Progress  The Mission Analysis
           that all critical needs and                  Steering Group has
           opportunities emerge from                    developed a methodology
           structured mission analysis,                 to assign priorities to
           are developed into Mission Need              Mission Need Statements,
           Statements with a recommended                and a proposal for
           prioritization, are reviewed                 testing has been
           for validation, and, if                      approved. Plans are in
           approved are assigned a                      process to address the
           prioritization relative to all               other issues in this
           other approved Mission Need                  recommendation.
           Statements, in accordance with               
           AMS \[Acquisition Management                 
           System\].                                    

  1B       The chairperson of the Mission  In Progress  The Mission Analysis
           Analysis Steering Group prepare              Steering Group prepared
           guidance on how to prioritize                prioritization guidelines
           new and existing mission needs               which have been approved
           for Joint Resources Council                  for limited testing.
           approval, and ensure the                     
           approved guidance is                         
           implemented.                                 

  2A       The chairperson of the Mission  In Progress  Steps have been taken for
           Analysis Steering Group ensure               representatives to
           the Mission Analysis Steering                fulfill their
           Group representatives (both                  responsibilities, and
           members and their alternates)                more members are
           fulfill their responsibilities               attending meetings.
           to the Mission Analysis                      Action has been taken to
           Steering Group, to include                   ensure mission analysis
           ensuring the AMS \[Acquisition               work, such as the
           Management System\] mission                  National Airspace System
           analysis process is followed                 Architecture, is
           and that all mission analysis                coordinated with the
           work in the agency, including                Mission Analysis Steering
           the National Airspace System                 Group.
           Architecture, is coordinated                 
           with the Mission Analysis                    
           Steering Group.                              
  -------------------------------------------------------------------------------

Table 3-2. Recommendation Actions to FAA's Report for the First Two
Years, dated May 29, 1998 (continued)

  -------------------------------------------------------------------------------
  **Rec.   **Recommendation**              **Status**   **Rationale for Status**
  \#**                                                  
  -------- ------------------------------- ------------ -------------------------
  2B       The Associate Administrator for In Progress  Some discussions have
           each line of business ensure                 taken place with no
           sufficient qualified and                     definitive outcome.
           trained staff and the                        
           appropriate resources are                    
           available to perform mission                 
           analysis work or delegate this               
           responsibility and authority to              
           the Mission Analysis Steering                
           Group or Air Traffic Services                
           Mission Analysis team as                     
           appropriate and agreed upon.                 

  3A       The chairperson of the Mission  In Progress  Guidance for revalidating
           Analysis Steering Group develop              mission needs is expected
           guidance and procedures on how               to be finalized in June
           to conduct and document                      1999.
           revalidation of mission needs,               
           and establish a plan with a                  
           schedule for accomplishing                   
           revalidations.                               

  3B       The chairperson of the Mission  In Progress  Guidance on establishing
           Analysis Steering Group develop              a central repository is
           guidance to establish a central              expected to be finalized
           repository for historical                    in June 1999.
           records of all supporting                    
           documentation for submitted                  
           Mission Need Statements, and to              
           ensure each line of business                 
           submits all acquisition                      
           planning and control supporting              
           documentation to this central                
           repository.                                  

  4A       The FAA Acquisition Executive   In Progress  Some discussions have
           require the changed direction                taken place with no
           and rationale for any                        definitive outcome.
           intervention in the investment               
           analysis process be documented               
           at the time of the                           
           intervention, and ensure this                
           documentation is included in                 
           the Investment Analysis Report               
           used at the Investment                       
           Decision.                                    
  -------------------------------------------------------------------------------

Table 3-2. Recommendation Actions to FAA's Report for the First Two
Years, dated May 29, 1998 (continued)

  -------------------------------------------------------------------------------
  **Rec.   **Recommendation**              **Status**   **Rationale for Status**
  \#**                                                  
  -------- ------------------------------- ------------ -------------------------
  5A       The chairperson of the          In Progress  A revision is under
           Acquisition System Advisory                  development.
           Group revise the AMS                         
           \[Acquisition Management                     
           System\] to allow the Systems                
           Engineering/Operational                      
           Analysis Team, in conjunction                
           with the Investment Analysis                 
           Team, flexibility to determine,              
           on a case-by-case basis, how                 
           many viable candidate solutions              
           will have an affordability                   
           assessment completed.                        

  6A       The FAA Administrator           No Action    The Office of
           \[subsequently delegated to the              Acquisitions, ASU,
           Associate Administrator for                  believes it is
           Research and Acquisitions\]                  unnecessary to define
           ensure definitions are                       these terms, since FAA's
           developed for "program,"                     acquisition policy
           "substantial program," and                   applies to all programs,
           "major systems acquisitions,"                not just certain
           and that all FAA programs are                categories of programs.
           designated in the appropriate                The evaluation team
           category.                                    agrees in part, but the
                                                        definitions are needed in
                                                        order for FAA to
                                                        determine compliance with
                                                        other regulations and
                                                        laws, including Office of
                                                        Management and Budget
                                                        Circulars.

  6B       The FAA Acquisition Executive   In Progress  Additional baselines have
           ensure every program, existing               been established, but
           or new, has an Acquisition                   many programs still need
           Program Baseline as required by              baselines.
           AMS \[Acquisition Management                 
           System\], or an alternate                    
           baseline document, that                      
           includes all applicable cost                 
           (Facilities and Equipment;                   
           Research, Engineering and                    
           Development; and Operations and              
           Maintenance), schedule,                      
           benefits, and performance                    
           parameters.                                  
  -------------------------------------------------------------------------------

Table 3-2. Recommendation Actions to FAA's Report for the First Two
Years, dated May 29, 1998 (continued)

  -------------------------------------------------------------------------------
  **Rec.   **Recommendation**              **Status**   **Rationale for Status**
  \#**                                                  
  -------- ------------------------------- ------------ -------------------------
  6C       The FAA Administrator           No Action    The evaluation team found
           \[subsequently delegated to the              no action has been taken.
           Associate Administrator for                  
           Research and Acquisitions\]                  
           require those organizations or               
           individuals with signature                   
           authority for funding,                       
           including Facilities and                     
           Equipment; Research,                         
           Engineering, and Development;                
           and Operations and Maintenance,              
           provide all the funding data                 
           and their signatures to                      
           authorize and commit these                   
           funds for each Acquisition                   
           Program Baseline.                            

  7A       The FAA Administrator designate In Progress  A memorandum designating
           a single focal point for all                 the National Airspace
           program baseline management and              System Programming and
           reporting activities.                        Financial Management
                                                        Office, ASD-300, as the
                                                        single focal point is
                                                        being drafted for the FAA
                                                        Acquisition Executive's
                                                        signature.

  7B       The FAA Administrator direct    In Progress  The FAA Acquisition
           the development of an adequate               Executive states that
           tracking and validation system               benefits tracking should
           for benefits, or revise the AMS              remain in the Acquisition
           \[Acquisition Management                     Management System, and
           System\] if it is determined                 the agency plans to track
           the benefits baselining                      benefits where it is
           requirement should not be                    feasible. However, he
           retained.                                    states that it is not
                                                        feasible for all programs
                                                        to track and validate
                                                        benefits after the
                                                        initial baselining
                                                        action.

  7C       The FAA Administrator design    Completed    The Integrated Baseline
           and implement procedures for                 Establishment and
           tracking and reporting program               Management (I‑BEAM)
           status against all approved                  concept and guidelines
           baselines.                                   were established,
                                                        approved, and finalized
                                                        to manage program
                                                        baselines for cost,
                                                        schedule, benefits, and
                                                        performance.
  -------------------------------------------------------------------------------

Table 3-2. Recommendation Actions to FAA's Report for the First Two
Years, dated May 29, 1998 (continued)

  -------------------------------------------------------------------------------
  **Rec.   **Recommendation**              **Status**   **Rationale for Status**
  \#**                                                  
  -------- ------------------------------- ------------ -------------------------
  8A       The FAA Administrator combine   In Progress  There have been
           the responsibility and                       discussions about merging
           authority for full lifecycle                 the Joint Resources
           funding under one                            Council and the Resource
           Associate/Assistant                          Management Council, but
           Administrator level body and                 there continues to be
           clearly define its role,                     opposition to this
           responsibilities, and                        recommendation. No final
           authority.                                   decision on merging these
                                                        Councils has been made.

  8B       The FAA Acquisition Executive   Completed    The problem of
           ensure each Joint Resources                  rescheduling meetings due
           Council member designates a                  to the absence of one
           single alternate and empowers                member has been addressed
           this alternate to act in                     and it is no longer an
           his/her place, and each key                  issue.
           Joint Resources Council meeting              
           participant designates a single              
           alternate and empowers this                  
           alternate to act in his/her                  
           place in order to reduce                     
           rescheduling caused by calendar              
           conflicts and to maintain                    
           consistency at meetings. These               
           alternates should possess                    
           sufficient knowledge of the                  
           duties and responsibilities of               
           the Joint Resources Council                  
           member or key participant and                
           should have the appropriate                  
           corporate level perspective to               
           serve in this position.                      

  9A       The FAA Acquisition Executive   In Progress  Revised guidance was
           revise the FAA Joint Resources               drafted in February 1999
           Council Guidance requirement to              to state that the Records
           issue Records of Decision                    of Decision will be
           within five days to a realistic              provided within 21 days
           period of time; and, develop                 instead of 5. The revised
           and document a process for                   guidance will be
           timely preparation, approval,                disseminated.
           and dissemination of Joint                   
           Resources Council Records of                 
           Decision, and ensure that                    
           process is followed.                         
  -------------------------------------------------------------------------------

Table 3-2. Recommendation Actions to FAA's Report for the First Two
Years, dated May 29, 1998 (continued)

  -------------------------------------------------------------------------------
  **Rec.   **Recommendation**              **Status**   **Rationale for Status**
  \#**                                                  
  -------- ------------------------------- ------------ -------------------------
  9B       The FAA Acquisition Executive   Completed    Action items are reviewed
           establish an effective means to              during Joint Resources
           capture assigned action items                Council meetings and are
           during Joint Resources Council               recorded and disseminated
           meetings (including what is                  in Records of Decision. A
           required, who is responsible,                database has been
           and when the action is to be                 implemented to track and
           completed) to ensure members                 close the action items.
           and participants understand and              
           agree to all action items                    
           assigned; and define and                     
           implement a process to track,                
           follow up on, close, and                     
           disseminate results of action                
           items after Joint Resources                  
           Council meetings.                            

  10A      The FAA Administrator           No Action    The FAA Acquisition
           \[subsequently delegated to the              Executive agreed with
           Associate Administrator for                  stressing quality
           Research and Acquisitions\]                  direction and processes
           ensure that Integrated Product               for a stronger team but
           Teams/Product Teams and                      states that the
           functional                                   recommendation is already
           management/organizations                     incorporated within the
           jointly select team members,                 framework and structure
           document agreements clarifying               of the Integrated Product
           team members' roles and                      Development System and
           responsibilities, and evaluate               that no action is needed.
           team member performance.                     The evaluation team does
                                                        not agree and concludes
                                                        that more commitment in
                                                        this area is needed.

  10B      The chairperson of the          In Progress  The Integrated Product
           Integrated Product Leadership                Leadership Team co‑leads
           Team ensure issues raised                    agreed to address this
           within the Integrated Product                recommendation by
           Development System structure                 establishing subteams to
           are resolved.                                raise and resolve
                                                        Integrated Product
                                                        Development System
                                                        issues.
  -------------------------------------------------------------------------------

Table 3-2. Recommendation Actions to FAA's Report for the First Two
Years, dated May 29, 1998 (continued)

  -------------------------------------------------------------------------------
  **Rec.   **Recommendation**              **Status**   **Rationale for Status**
  \#**                                                  
  -------- ------------------------------- ------------ -------------------------
  12A      The Special Assistant, Small    In Progress  Guidance has been
           Business Utilization Staff,                  developed and is
           ensure that FAA Regions,                     available; however, since
           Centers, and Headquarters have               the ACQUIRE system became
           accurate, current guidance for               operational, the Small
           development of Major                         Business Utilization
           Procurement Program Goals                    Staff has not received
           reports and comply with that                 any FY 99 data to confirm
           guidance.                                    compliance with the
                                                        guidelines.

  14A      The FAA Administrator ensure    In Progress  Some activities have been
           that the agency performs better              initiated to improve
           planning for its future funding              planning; however, these
           needs to avoid constant funding              activities have not
           reductions to on-going                       mitigated funding
           programs. This would involve                 reductions to ongoing
           improving the agency's (1)                   programs.
           mission analysis capabilities,               
           which must be tied strongly to               
           the agency's approved National               
           Airspace System Architecture;                
           (2) investment analysis                      
           capabilities; and (3) managing               
           of program cost, schedule,                   
           performance, and benefits                    
           baselines.                                   
  -------------------------------------------------------------------------------

[]{#__RefHeading___Toc453556202 .anchor}Table 3-3. Recommendation
Actions to Booz•Allen & Hamilton's Independent Assessment, dated
September 19, 1997

  -------------------------------------------------------------------------------
  **Rec.   **Recommendation**              **Status**   **Rationale for Status**
  \#**                                                  
  -------- ------------------------------- ------------ -------------------------
  1        Continue AMS \[Acquisition      Completed    The Acquisition
           Management System\]                          Management System policy
           implementation and evaluation                document and other
           as planned.                                  guidance and automated
                                                        tools are continually
                                                        being upgraded and
                                                        improved.

  2        Communicate new and planned     Completed    FAA Acquisition System
           features of the FAST \[FAA                   Toolset (FAST) changes
           Acquisition System Toolset\] to              and updates are available
           its users.                                   to FAST users via the FAA
                                                        site on the Internet. All
                                                        FAST users have Internet
                                                        access.

  3        Provide specialized training    Completed    The Office of
           focused on AMS \[Acquisition                 Acquisitions, ASU, has
           Management System\] and its                  developed and conducted
           lifecycle approach.                          various training classes
                                                        on the Acquisition
                                                        Management System policy
                                                        and lifecycle approach
                                                        and is in the process of
                                                        developing computer-based
                                                        instruction courses.

  4        Communicate protest and dispute In Progress  Protest and dispute
           information agency wide.                     information is available
                                                        via the FAA site on the
                                                        Internet. However,
                                                        because some FAA field
                                                        offices have neither
                                                        Internet access, nor
                                                        knowledge of other means
                                                        to obtain protest and
                                                        dispute information, this
                                                        recommendation has not
                                                        been completed
                                                        agency-wide.

  5        Develop standardized contract   Completed    Checklists have been
           file documentation.                          developed to help
                                                        contracting officers
                                                        standardize contract file
                                                        documentation.
  -------------------------------------------------------------------------------

**Table 3-3. Recommendation Actions to Booz•**Allen & Hamilton's
Independent Assessment, dated September 19, 1997(continued)

  -------------------------------------------------------------------------------
  **Rec.   **Recommendation**              **Status**   **Rationale for Status**
  \#**                                                  
  -------- ------------------------------- ------------ -------------------------
  6        Define roles and                In Progress  Some of the roles and
           responsibilities across all                  responsibilities have
           lifecycle phases.                            been defined for the
                                                        mission analysis and
                                                        investment analysis
                                                        phases, but more work is
                                                        needed on other lifecycle
                                                        phases.

  7        Establish quantitative goals    In Progress  Specific program goals
           and metrics.                                 and metrics have been
                                                        implemented, but
                                                        quantitative goals and
                                                        metrics have not been
                                                        implemented for lifecycle
                                                        phases.

  8        Implement a centralized         Completed    While there have been
           database for contracting.                    concerns from users, the
                                                        ACQUIRE system has been
                                                        implemented and other
                                                        contracting systems have
                                                        decommissioned.

  9        Determine the extent and cause  Completed    An evaluation was
           of any decrease in contract                  conducted on this issue
           awards to SEDB concerns.                     and addressed this
                                                        recommendation (see the
                                                        FAA report, *Impact of
                                                        Acquisition Reform on
                                                        Awards to Disadvantaged
                                                        Businesses*, dated May
                                                        20, 1998).

  10       Establish clear authority and   Completed    The Office of Dispute
           provide adequate resources to                Resolution was authorized
           staff Office of Dispute                      by the FAA Administrator
           Resolution.                                  and has been fully
                                                        staffed.

  11       Develop and implement an        In Progress  Some activities have been
           integrated approach to budget                initiated to improve
           planning and management.                     planning; however, these
                                                        activities have not
                                                        mitigated funding
                                                        reductions to ongoing
                                                        programs.
  -------------------------------------------------------------------------------

**Table 3-3. Recommendation Actions to Booz•**Allen & Hamilton's
Independent Assessment, dated September 19, 1997(continued)

  -------------------------------------------------------------------------------
  **Rec.   **Recommendation**              **Status**   **Rationale for Status**
  \#**                                                  
  -------- ------------------------------- ------------ -------------------------
  12       Develop and implement outreach  In Progress  FAA has developed and
           program for external                         implemented outreach
           stakeholders.                                programs for external
                                                        customers, such as the
                                                        effort to obtain industry
                                                        input for the National
                                                        Airspace System
                                                        Architecture Version 4.0
                                                        and the Small Business
                                                        Utilization Staff work to
                                                        reach out to small
                                                        business concerns and
                                                        small business concerns
                                                        owned and controlled by
                                                        socially and economically
                                                        disadvantaged
                                                        individuals.

  13       Develop an overarching          In Progress  Steps have been taken to
           management process that guides               link various agency
           various reform and process                   reforms and process
           change initiatives.                          change initiatives.
                                                        However, no evidence of
                                                        an overarching management
                                                        process that integrates
                                                        all initiatives was
                                                        found.
  -------------------------------------------------------------------------------

[]{#__RefHeading___Toc453556203 .anchor}Table 3-4. Recommendation
Actions to FAA's Report on Disadvantaged Businesses, dated May 20, 1998

  -----------------------------------------------------------------------------
  **Rec.   **Recommendation**              Status     **Rationale for Status**
  \#**                                                
  -------- ------------------------------- ---------- -------------------------
  1A       The FAA currently has only one  In         The Small Business
           method for meeting SEDB         Progress   Utilization Staff has
           requirements and the potential             been successful in
           success of that method is tied             encouraging some
           to the functions of the FAA                coordination with its
           Small Business Utilization                 staff (see Finding 3 in
           Staff. Therefore, we recommend             this report). In
           the FAA Administrator establish            addition, some lines of
           incentives to encourage both               business have included an
           contracting officers and                   element in their
           Integrated Product Teams to                employees' performance
           coordinate with the FAA Small              appraisals to award
           Business Utilization Staff to              specified percentages of
           ensure the FAA has every                   total contract dollars to
           opportunity to identify awards             meet Major Procurement
           suitable for SEDB awards.                  Program Goals.

  1B       We recommend the FAA            In         The Associate
           Administrator adjust the        Progress   Administrator for
           criteria for counting awards to            Research and
           disadvantaged businesses                   Acquisitions, the Special
           holding a Small Business                   Assistant for Small
           Administration Section 8(a)                Business, and the
           certification by including all             Director of Acquisitions,
           awards to these businesses in              have agreed to categorize
           the SEDB award category                    all contract awards to
           regardless of how those awards             disadvantaged businesses
           were received.                             that hold Small Business
                                                      Administration 8(a)
                                                      certification as SEDB
                                                      awards, irrespective of
                                                      how the award was made.
                                                      However, implementation
                                                      of the change is still
                                                      being finalized.
  -----------------------------------------------------------------------------

# Part IIIAppendices {#part-iiiappendices .Part}

[]{#__RefHeading___Toc453556173 .anchor}Appendix AMethodology Detailed

In order to conduct this review, the evaluation team employed various
techniques, practices, and procedures. The different methodologies used
by the evaluation team are described here in the order the evaluation
results are presented in the report.

# Procurement Results (Chapter 1)

The evaluation team's objective was to determine if, under acquisition
reform, procurement efforts were achieving faster awards, were achieving
competition, and were meeting Major Procurement Program Goals (MPPG). A
different methodology was used for each part of the objective. For
timeliness and competition, the team compared data for the last full
fiscal year before acquisition reform (FY 95) as a base year to the two
full fiscal years after acquisition reform (FY 97 and FY 98). This is a
departure from the methodology used for the first two FAA evaluations of
acquisition reform. The first two evaluation efforts used partial fiscal
year data and focused on shorter segments of time because insufficient
data existed to compare full multiple fiscal years. For this report,
full fiscal years were used for clarity of comparison. Because
acquisition reform began in the middle of FY 96, data from FY 96 was not
used. Using complete fiscal years to identify data allowed more precise
data identification, easier data collection, and consistency with agency
records. In examining the achievement of FAA's MPPG goals, the
evaluation team concentrated on whether the FAA was meeting its goals
and only analyzed the data for the two complete fiscal years (FY 97 and
FY 98) after acquisition reform. This timeframe was selected because of
available data and to be consistent with the agency's reports to the
Department of Transportation. In addition, acquisition reform and
updates made to the Small Business Utilization Staff's reporting
procedures in FY 96 made data comparison for dates before April 1996
more complicated.

## Timeliness

The evaluation team defined four measurable dates in the procurement
process: (1) initiation of a procurement request; (2) first action taken
by the contracting officer, which is usually a public announcement of
the procurement; (3) formal request for an offer of proposal; and
(4) the procurement award. Figure A-1 illustrates the procurement cycle
before and after acquisition reform. In evaluating the impact of
acquisition reform on the procurement process, the team determined
timeliness based on data collected for the last three measurable dates
in the procurement process. Since the Contract Information System only
includes the award amount and award date and does not include dates for
the procurement request, first contracting officer action, or request
for a proposal, it was not realistic to collect data for the entire
universe of contracts. Consequently, the team identified a random
statistical sample from the universe of contracts over \$100,000
included in the Contract Information System for FY 95, FY 97, and FY 98.
A list of the specific contracts selected in this random sample can be
found in Appendix B, Tables B-1, B-2, and B-3. Information was collected
from contracting officers and contract files for the sampled contracts
to support the three measurable dates. Based on the obtainable
information from procurement records, the evaluation team determined the
award time for periods before and after acquisition reform and compared
them to determine whether procurement efforts were achieving faster
awards.

![](media/image6.wmf){width="5.7444444444444445in"
height="2.604861111111111in"}

[]{#__RefHeading___Toc453556216 .anchor}Figure A-1. Procurement Cycle:
Before and Since Acquisition Reform

## Competition

To determine the impact on competition, the evaluation team compared the
dollars awarded for competitive and non-competitive awards, including
their modifications during that year for a base fiscal year (FY 95)
before acquisition reform to the first two full fiscal years (FY 97 and
FY 98) after acquisition reform. The evaluation team used automated data
from Contract Information System database for all contracts with a value
over \$25,000. Based on the comparison, the team determined the extent
of competition achieved.

## Major Procurement Program Goals

Acquisition Management System (AMS) policy Section 3.6.1.2, requires the
FAA to "...implement and aggressively strive to provide small businesses
and small businesses owned and controlled by socially and economically
disadvantaged individuals attainable and reasonable opportunities to
participate as prime contractors and subcontractors for products and
services procured by the FAA." AMS policy Section 3.6.1.3.1, also
requires the FAA to establish measurable annual agency-wide procurement
program goals for these businesses. The FAA established goals between 5
and 25 percent for these categories for each year.

The evaluation team reviewed all FAA region, center, and headquarters
procurement office MPPG data reported to the Small Business Utilization
Staff for FY 97 and FY 98 and confirmed the totals in these MPPG reports
with the Small Business Utilization Staff.

Based on a comparative analysis of reported data and agency goals, the
evaluation team determined the extent of FAA MPPG goal achievement.
Regions, centers, and headquarters offices where MPPG goals were not
being met were visited and personnel were interviewed with the express
intent of determining reasons for not achieving MPPG goals as well as
any potential consequences. Using dollar amount of awards as the
measurement, additional data were compiled to summarize acquisition
reform's impact on achieving these goals.

# Program Results (Chapter 2)

The evaluation team's objective was to determine if, under acquisition
reform, programs were on track to achieve success. To make this
determination, the evaluation team considered program results in terms
of five specific elements: (1) supporting the FAA mission, (2)
delivering product performance, (3) meeting customer needs, (4) meeting
cost baselines, and (5) meeting schedule baselines. The evaluation team
attempted to correlate program results attained to Acquisition
Management System (AMS) processes to measure the impact of acquisition
reform on program results.

The evaluation team based its findings on analyses performed on a
judgmental sample of programs. To develop the sample, the evaluation
team established the universe of programs managed under acquisition
reform by using all programs listed in the Capital Investment Plan,
which captures facilities and equipment funded programs; and all
programs listed in the Research, Engineering and Development Plan, which
captures research, engineering, and development funded programs. No
document currently exists to reflect operations and maintenance funded
programs, so the evaluation team reviewed the FY 99 President's Budget
Submission and conducted interviews with various levels of agency
personnel. However, the team was unable to identify programs
specifically funded with operations and maintenance appropriations.

Only programs with a contract award between January 1996 and January
1998 were considered for sample selection. The January 1996 date was
established to ensure programs selected were managed under acquisition
reform. The January 1998 date was established to ensure at least one
year of program data would be available for analysis. The judgmental
sample included 10 facilities and equipment funded programs and
3 research, engineering, and development funded programs. The list of
the sampled programs and their Capital Investment Plan number or
Research, Engineering and Development Plan number can be found in
Appendix B, Table B-4. The sample covered FAA mission areas of safety,
security, system efficiency, environment, productivity, and capacity.
Because acquisition reform has been in place only three years and FAA
programs are typically long term in nature, most programs in the sample
were initiated prior to the implementation of acquisition reform. While
these programs were managed under acquisition reform, most had completed
a substantial part of the planning efforts prior to the implementation
of AMS, and most of the sampled programs did not have fielded products
at the time of this review. In the results, all the sampled programs
were considered equally; there were no weights applied for program size,
dollar value, scheduled time, or complexity of the products developed.

## Results in Terms of FAA Mission, Product Performance, Customer Needs, and Cost and Schedule Baselines

The evaluation team looked at the broad purposes of the sampled programs
as defined in their Mission Need Statements and Integrated Program Plans
to determine whether each program was consistent with the current FAA
mission. Also, the program information was compared to the National
Airspace System Architecture, the Capital Investment Plan, and the
Research, Engineering and Development Plan to ensure consistency with
FAA's long range strategic planning. Integrated Product Team leads and
Product Team leads were interviewed to determine whether each program
had met or was projected to meet the planned program performance
results. Baseline documentation was reviewed to verify whether technical
performance and benefits will fail to achieve at least 50 percent of the
performance goal as established for the program. Where appropriate,
customers were interviewed to verify whether the fielded product
fulfilled the requirements established. Also, Acquisition Review data
and reports were compared to Requirements Documents to verify whether
the intended purpose was accomplished. Since research, engineering, and
development programs are designed for a purpose other than producing a
specific product, the evaluation team did not attempt to measure the
level of product performance results attained for the three programs of
that type in the sample.

To evaluate whether the sampled programs were meeting customer needs,
the evaluation team first identified the lines of business supporting or
benefiting from the these programs. Integrated Product Team and
Investment Analysis Team member lists were reviewed to identify and
interview customers who may have been active members on the program.
Interviews were conducted to determine customer involvement in the
planning process and to learn whether they believed their participation
was effective and meaningful. Mission Need Statements, investment
analysis documents, and Requirements Documents were compared to verify
whether customer needs were identified, documented, and addressed in the
planning process. The evaluation team also interviewed Integrated
Product Development System team leads to learn how customer needs were
revalidated, how changing needs were identified and prioritized, and how
changing needs were incorporated into the product. For programs with a
fielded product, the evaluation team interviewed customers to find out
whether they were satisfied with the product performance results.

In evaluating whether the sampled programs met cost and schedule
baselines, the evaluation team obtained baseline information and, if
available, the Joint Resources Council approved formal baselines for
each of the 10 facilities and equipment funded programs. If formal
baselines had not been established, the evaluation team relied on
project plans to evaluate the program's success in meeting the planned
cost and schedule objectives. Management reporting documents, including
Baseline Management Notices, status reviews, status reports, and monthly
program reviews, were obtained to measure the level of deviation from
the initial baseline to the current or projected baseline. It was beyond
the scope of this evaluation for the team to assess whether or not the
approved baselines were realistic. To do this would require in-depth,
individual program reviews. Also, baseline information contained in the
Research Project Description was collected for the research,
engineering, and development funded programs. The cost baselines in the
Research Project Descriptions reflect the level of funding the program
requested and do not represent a budget or cost agreement. As a result,
the evaluation team did not attempt to measure the rate of baseline
deviation for the research, engineering, and development funded
programs.

## Impact of Acquisition Reform on Program Results

The evaluation team compared the program results attained in the section
above with the processes followed during planning and lifecycle
management for each program. Under planning, consideration was given to
the processes followed for mission analysis, requirements definition,
affordability assessment, risk assessment, baseline development, and
line of business involvement. Under lifecycle management, consideration
was given to the processes followed for monitoring government and
contractor performance, managing baselines, reporting and reviewing
progress, and receiving executive level support and oversight. Since
processes for research, engineering, and development funded programs
were not specifically addressed in AMS, the evaluation team restricted
the correlation review activity to the 10 facilities and equipment
funded programs.

The evaluation team compared the program results attained with the
processes followed during planning and lifecycle management for each
program to evaluate whether identified program strengths and weaknesses
were attributable to changes brought about by acquisition reform. The
evaluation team:

-   Documented whether formal baselines were established and approved
    and whether they included all funding types.

-   Identified whether processes were in place to monitor the program's
    ability to meet planned program performance and whether the program
    established and followed a formal process for monitoring contractor
    performance.

-   Verified whether affordability assessments were prepared and whether
    they included all funding requirement sources and all lifecycle
    phases.

-   Confirmed whether program plans included risk assessments and risk
    mitigation strategies.

-   Verified whether sponsoring lines of business were involved in the
    planning process.

-   Reviewed Mission Need Statements, Requirements Documents, and
    investment analysis reports.

-   Reviewed memoranda from the Joint Resources Council relating to
    program funding and solution selection.

-   Verified whether multiple levels of management reviews were active
    for the programs.

For the purposes of correlating AMS processes to program results, the
evaluation team limited its review of the above activities to verifying
whether the processes were completed and documented. The quality of the
data provided or the level of effort afforded each process were not
evaluated except in those cases where the program did not achieve
positive results.

# Recommendation Follow-up on Previous Acquisition Reform Evaluations (Chapter 3)

The evaluation team's objective was to determine the status of
recommendations made in the previous acquisition reform evaluations. In
order to do this, the evaluation team conducted the following steps:

-   Identified and clarified each recommendation.

-   Identified the organization responsible for implementing the
    recommendation.

-   Identified the individual or position within the organization
    responsible for completing the action.

-   Contacted the action officials to determine the status or to
    identify other sources for obtaining information.

-   Confirmed, to the extent possible, the status of the
    recommendations.

The evaluation team determined the status only of each recommendation.
The results and the quality of the actions taken to complete the
recommendations were not evaluated.

[]{#__RefHeading___Toc453556182 .anchor}Appendix BSample Information

The tables in this appendix identify those contracts and programs that
were sampled for this evaluation. Tables B-1, B-2, and B-3 contain the
contract numbers for the samples used to evaluate timeliness as reported
in Chapter 1, Finding 1. Table B-4 contains a list of programs and their
Capital Investment Plan or Research, Engineering and Development Plan
numbers for the sample used to evaluate program results in Chapter 2.

[]{#__RefHeading___Toc453556204 .anchor}Table B-1. Fiscal Year 1995
Timeliness Contract File Sample Set

+-----------------+-----------------+-----------------+-----------------+
| > **Contract    | > **Contract    | > **Contract    | > **Contract    |
| > Number**      | > Number**      | > Number**      | > Number**      |
+-----------------+-----------------+-----------------+-----------------+
| >               | >               | >               | >               |
|  DTFA0195C00005 |  DTFA0395C00016 |  DTFA0995C25028 |  DTFA1195C00110 |
+-----------------+-----------------+-----------------+-----------------+
| >               | >               | >               | >               |
|  DTFA0195C00007 |  DTFA0395D00024 |  DTFA1195C00105 |  DTFA1195C00169 |
+-----------------+-----------------+-----------------+-----------------+
| >               | >               | >               | >               |
|  DTFA0195C00015 |  DTFA0495C10013 |  DTFA1195C00122 |  DTFA1295C00009 |
+-----------------+-----------------+-----------------+-----------------+
| >               | >               | >               | >               |
|  DTFA0195C00016 |  DTFA0595C75010 |  DTFA1195C00158 |  DTFA0695C30049 |
+-----------------+-----------------+-----------------+-----------------+
| >               | >               | >               | >               |
|  DTFA0195C01029 |  DTFA0695C30017 |  DTFA1195C00162 |  DTFA0695C30115 |
+-----------------+-----------------+-----------------+-----------------+
| >               | >               | >               | >               |
|  DTFA0195C01030 |  DTFA0695C50005 |  DTFA1495C33567 |  DTFA0695C50043 |
+-----------------+-----------------+-----------------+-----------------+
| >               | >               | >               | >               |
|  DTFA0295D95053 |  DTFA0695C50014 |  DTFA1495C33660 |  DTFA0795C03178 |
+-----------------+-----------------+-----------------+-----------------+
| >               | >               | >               | >               |
|  DTFA0295D95521 |  DTFA0795C03118 |  DTFA0195C00009 |  DTFA0195C00049 |
+-----------------+-----------------+-----------------+-----------------+
| >               | >               | >               | >               |
|  DTFA0295D95544 |  DTFA0795C03119 |  DTFA0295D95535 |  DTFA0795C03190 |
+-----------------+-----------------+-----------------+-----------------+
| >               | >               | >               | >               |
|  DTFA0295D95551 |  DTFA0795C03154 |  DTFA0395D00025 |  DTFA0795C03148 |
+-----------------+-----------------+-----------------+-----------------+
| >               | >               | >               |                 |
|  DTFA0395C00004 |  DTFA0895D04796 |  DTFA0795C03157 |                 |
+-----------------+-----------------+-----------------+-----------------+

[]{#__RefHeading___Toc453556205 .anchor}Table B-2. Fiscal Year 1997
Timeliness Contract File Sample Set

+-----------------+-----------------+-----------------+-----------------+
| > **Contract    | > **Contract    | > **Contract    | > **Contract    |
| > Number**      | > Number**      | > Number**      | > Number**      |
+-----------------+-----------------+-----------------+-----------------+
| >               | >               | >               | >               |
|  DTFA0197C00010 |  DTFA0297P80460 |  DTFA0797C03285 |  DTFA0197P07062 |
+-----------------+-----------------+-----------------+-----------------+
| >               | >               | >               | >               |
|  DTFA0197C00031 |  DTFA0297P80631 |  DTFA0797C03333 |  DTFA0297C97058 |
+-----------------+-----------------+-----------------+-----------------+
| >               | >               | >               | >               |
|  DTFA0197C00048 |  DTFA0397D00001 |  DTFA0897C02925 |  DTFA0297P80002 |
+-----------------+-----------------+-----------------+-----------------+
| >               | >               | >               | >               |
|  DTFA0197P07063 |  DTFA0397P00321 |  DTFA0897C03247 |  DTFA1197C00130 |
+-----------------+-----------------+-----------------+-----------------+
| >               | >               | >               | >               |
|  DTFA0197P20316 |  DTFA0497C10011 |  DTFA0897C03623 |  DTFA1297C00096 |
+-----------------+-----------------+-----------------+-----------------+
| >               | >               | >               | >               |
|  DTFA0297D97067 |  DTFA0597C57289 |  DTFA0897P06519 |  DTFA0297D97507 |
+-----------------+-----------------+-----------------+-----------------+
| >               | >               | >               | >               |
|  DTFA0297D97070 |  DTFA0697C30043 |  DTFA0997C27005 |  DTFA0797C03290 |
+-----------------+-----------------+-----------------+-----------------+
| >               | >               | >               | >               |
|  DTFA0297D97500 |  DTFA0697C30061 |  DTFA0997C27006 |  DTFA0397C00031 |
+-----------------+-----------------+-----------------+-----------------+
| >               | >               | >               |                 |
|  DTFA0297D97515 |  DTFA0697C50027 |  DTFA0997C27012 |                 |
+-----------------+-----------------+-----------------+-----------------+
| >               | >               | >               |                 |
|  DTFA0297D97528 |  DTFA0797C03257 |  DTFA1297C00088 |                 |
+-----------------+-----------------+-----------------+-----------------+
| >               | >               | >               |                 |
|  DTFA0297D97530 |  DTFA0797C03265 |  DTFA1497C33893 |                 |
+-----------------+-----------------+-----------------+-----------------+

[]{#__RefHeading___Toc453556206 .anchor}Table B-3. Fiscal Year 1998
Timeliness Contract File Sample Set

+-----------------+-----------------+-----------------+-----------------+
| > **Contract    | > **Contract    | > **Contract    | > **Contract    |
| > Number**      | > Number**      | > Number**      | > Number**      |
+-----------------+-----------------+-----------------+-----------------+
| >               | >               | >               | >               |
|  DTFA0198C00044 |  DTFA0298C98022 |  DTFA0698D30011 |  DTFA0998C28030 |
+-----------------+-----------------+-----------------+-----------------+
| >               | >               | >               | >               |
|  DTFA0198C00057 |  DTFA0298D98565 |  DTFA0798C03370 |  DTFA0998C28032 |
+-----------------+-----------------+-----------------+-----------------+
| >               | >               | >               | >               |
|  DTFA0198C00092 |  DTFA0298P80615 |  DTFA0798C03378 |  DTFA0998C28035 |
+-----------------+-----------------+-----------------+-----------------+
| >               | >               | >               | >               |
|  DTFA0198D03000 |  DTFA0398P00090 |  DTFA0898C03604 |  DTFA1198C00106 |
+-----------------+-----------------+-----------------+-----------------+
| >               | >               | >               | >               |
|  DTFA0198P05067 |  DTFA0398P00273 |  DTFA0898C03635 |  DTFA1198C00107 |
+-----------------+-----------------+-----------------+-----------------+
| >               | >               | >               | >               |
|  DTFA0198P05154 |  DTFA0398P00285 |  DTFA0898P06508 |  DTFA1198C00120 |
+-----------------+-----------------+-----------------+-----------------+
| >               | >               | >               | >               |
|  DTFA0198P07064 |  DTFA0398P00292 |  DTFA0898P06520 |  DTFA1498P10005 |
+-----------------+-----------------+-----------------+-----------------+
| >               | >               | >               | >               |
|  DTFA0198P07141 |  DTFA0498P10027 |  DTFA0898P06523 |  DTFA0298P40391 |
+-----------------+-----------------+-----------------+-----------------+
| >               | >               | >               | >               |
|  DTFA0198P09114 |  DTFA0598C57305 |  DTFA0998C28014 |  DTFA0798C03329 |
+-----------------+-----------------+-----------------+-----------------+
| >               | >               | >               | >               |
|  DTFA0198P09157 |  DTFA0698C50011 |  DTFA0998C28018 |  DTFA0998C28006 |
+-----------------+-----------------+-----------------+-----------------+
| >               | >               | >               | >               |
|  DTFA0298C98021 |  DTFA0698C50038 |  DTFA0998C28026 |  DTFA1198C00129 |
+-----------------+-----------------+-----------------+-----------------+

[]{#__RefHeading___Toc453556207 .anchor}Table B-4. Sampled Programs
Managed Under Acquisition Reform

  ------------------------------------------------------------------------
  **CIP / R,E&D  **Program Name**                   **Primary Funding
  Plan Number**                                     Source**
  -------------- ---------------------------------- ----------------------
  M08.19         ACQUIRE                            Facilities & Equipment

  M33            Advanced Aviation Security         Facilities & Equipment
                 Initiatives                        

  S03.02         Airport Surveillance Radar         Facilities & Equipment
                 (ASR-11)                           

  W07            Integrated Terminal Weather System Facilities & Equipment
                 (ITWS)                             

  M07            National Airspace System           Facilities & Equipment
                 Infrastructure Management System   
                 (NIMS)                             

  A07            Operational and Supportability     Facilities & Equipment
                 Implementation System (OASIS)      

  A04            Standard Terminal Automation       Facilities & Equipment
                 Replacement System (STARS)         

  W04            Weather and Radar Processor (WARP) Facilities & Equipment

  N12.01         Wide Area Augmentation System      Facilities & Equipment
                 (WAAS)                             

  M08.18         Year 2000 Computer Problem         Facilities & Equipment
                 (Program Office)                   

  A07a           Explosives and Weapons Detection   Research, Engineering
                                                    & Development

  A03b           Navigation                         Research, Engineering
                                                    & Development

  A04a           Weather Program                    Research, Engineering
                                                    & Development
  ------------------------------------------------------------------------

[]{#__RefHeading___Toc453556183 .anchor}Appendix CMajor Procurement
Program Goal Information

# Description of FAA's Disadvantaged Business Distinctions

Technically, "small business concerns owned and controlled by socially
and economically disadvantaged individuals" are considered disadvantaged
businesses. This category is further divided into two basic types of
disadvantaged businesses. Disadvantaged businesses are small business
concerns owned and controlled by socially and economically disadvantaged
individuals that either:

-   hold current certification from the Small Business Administration
    under Section 8(a) of the Small Business Act; or

-   do not hold the Small Business Administration's Section 8(a)
    certification, but are self certified[^7].

Disadvantaged businesses can receive awards from FAA in various
ways[^8], including, but not limited to:

-   full and open competition or unrestricted,

-   sole or single source award,

-   small business or very small business set-aside competition, or

-   SEDB/8(a) set-aside competition.

Only those disadvantaged businesses that hold current certification from
the Small Business Administration are eligible to compete in an
SEDB/8(a) set-aside competition.

FAA collects data on awards made to all small businesses. FAA makes a
distinction between awards made to the two basic types of disadvantaged
businesses. Disadvantaged businesses can receive awards either through
SEDB/8(a) set-aside competition or through any other process. Awards to
disadvantaged businesses through SEDB/8(a) set-aside competition were
categorized by FAA as "SEDB awards." Awards to disadvantaged businesses
through any other process, whether or not those disadvantaged businesses
held a current certification from the Small Business Administration,
were categorized by FAA as "SDB awards." See Table C-1 for distinctions
between SEDB and SDB awards.

[]{#__RefHeading___Toc453556208 .anchor}Table C-1. FAA Distinction
Between SEDB and SDB Awards

  ------------------- ---------------------------------------------------
  **SEDB Awards**     Awards to disadvantaged businesses that hold
                      current certifications from the Small Business
                      Administration ***and*** obtained contract award
                      through an SEDB/8(a) set-aside competition.

  **SDB Awards**      Awards to disadvantaged businesses that hold
                      current certifications from the Small Business
                      Administration, but obtained contract award through
                      means *other than* an SEDB/8(a) set-aside
                      competition.

                      ***or***

                      Awards to disadvantaged businesses that do not hold
                      current certifications from the Small Business
                      Administration.
  ------------------- ---------------------------------------------------

During the fourth quarter of FY 99, FAA will change the way it
categorizes awards to disadvantaged businesses that hold certifications
under Section 8(a) of the Small Business Act. From that point forward,
FAA will include in the "SEDB/8(a) award"\
category all awards to disadvantaged businesses holding 8(a)
certifications from the Small Business Administration irrespective of
how the disadvantaged businesses received the awards.

# FAA versus National Goals and Accomplishments

In light of FAA not achieving some of its Major Procurement Program
Goals (MPPG), the evaluation team analyzed how the government was doing
in general on goals and accomplishments. Information for the FAA and
national goals and accomplishments was obtained from the FAA Small
Business Utilization Staff and the *Federal Procurement Report: Fiscal
Year 1998 through Fourth Quarter*. In comparing that data, several
observations can be made.

First, FAA's goals for small business awards and SEDB awards have
routinely been established at or above the national goals since FY 95.
FAA's goals for SDB and women owned business (WOB) awards have been
established at the national goals. See Table C-2 for a comparison.

[]{#__RefHeading___Toc453556209 .anchor}Table C-2. National versus FAA
Goals for Small and Disadvantaged Business Awards

+-----------------------------+---------+---------+---------+---------+
|                             | **FY    | **FY    | **FY    | **FY    |
|                             | 95**    | 96**    | 97**    | 98**    |
+-----------------------------+---------+---------+---------+---------+
| **Small Business Awards:**  |         |         |         |         |
+-----------------------------+---------+---------+---------+---------+
| > National Goal             | 20%     | 20%     | 20%     | 20%     |
+-----------------------------+---------+---------+---------+---------+
| > FAA Goal                  | 25%     | 25%     | 20%     | 25%     |
+-----------------------------+---------+---------+---------+---------+
| **SEDB Awards:**            |         |         |         |         |
+-----------------------------+---------+---------+---------+---------+
| > National Goal             | 5%      | 5%      | 5%      | 5%      |
+-----------------------------+---------+---------+---------+---------+
| > FAA Goal                  | 15%     | 15%     | 5%      | 10%     |
+-----------------------------+---------+---------+---------+---------+
| **SDB Awards:**             |         |         |         |         |
+-----------------------------+---------+---------+---------+---------+
| > National Goal             | 5%      | 5%      | 5%      | 5%      |
+-----------------------------+---------+---------+---------+---------+
| > FAA Goal                  | 5%      | 5%      | 5%      | 5%      |
+-----------------------------+---------+---------+---------+---------+
| **WOB Awards:**             |         |         |         |         |
+-----------------------------+---------+---------+---------+---------+
| > National Goal             | 5%      | 5%      | 5%      | 5%      |
+-----------------------------+---------+---------+---------+---------+
| > FAA Goal                  | 5%      | 5%      | 5%      | 5%      |
+-----------------------------+---------+---------+---------+---------+

Shaded area indicates higher FAA goal/accomplishment.

The small business goal can be met by a combination of SEDB, SDB, and
WOB awards as well as awards to other small and very small businesses.
Although other small and very small businesses are not reported by FAA
as an MPPG category, they are included within the small business total
(see below).

  ----------- -----------------------------------------------------------
  \+          SEDB Awards

  \+          SDB Awards

  \+          WOB Awards

  \+          Other Small Business and Very Small Business Awards

  =           Total of All Small Business Awards
  ----------- -----------------------------------------------------------

Second, with the exception of FY 97, FAA MPPG goals were higher than the
national goals for small business awards and SEDB awards. Third, as
shown in Table C-3, in FY 98, FAA awarded a higher percentage of its
total procurements as MPPG awards than did the rest of the government.
Although comparison to national goals and accomplishments does not
absolve FAA of responsibility in failing to meet its own MPPG goals, it
does give a broader view of the goals and accomplishments and puts FAA's
accomplishments in that wider setting. A detailed breakdown of the FY 98
information is shown in Table C-3.

[]{#__RefHeading___Toc453556210 .anchor}Table C-3. Comparison of FAA and
National MPPG Goals and Accomplishments for Fiscal Year 1998 (as a
percentage of total procurements)

+---------------+--------+-----------------+--------+-----------------+
|               | #      |                 | **Nati |                 |
|               | ###### |                 | onal** |                 |
|               | ## FAA |                 |        |                 |
+---------------+--------+-----------------+--------+-----------------+
|               | **G    | **Ac            | **G    | **Ac            |
|               | oals** | complishments** | oals** | complishments** |
+---------------+--------+-----------------+--------+-----------------+
| **Small       | 25%    | 31.3%           | 20%    | 23.39%          |
| Business      |        |                 |        |                 |
| Awards**      |        |                 |        |                 |
+---------------+--------+-----------------+--------+-----------------+
| **SEDB        | 10%    | 8.6%            | 5%     | 3.57%           |
| Awards**      |        |                 |        |                 |
+---------------+--------+-----------------+--------+-----------------+
| **SDB         | 5%     | 4.1%            | 5%     | 2.91%           |
| Awards**      |        |                 |        |                 |
+---------------+--------+-----------------+--------+-----------------+
| **WOB         | 5%     | 3.4%            | 5%     | 2.21%           |
| Awards**      |        |                 |        |                 |
+---------------+--------+-----------------+--------+-----------------+

Shaded area indicates higher FAA goal/accomplishment.

[]{#__RefHeading___Toc453556186 .anchor}Appendix DRecommendations

The following lists all the recommendations included in this report. The
recommendation numbers correlate to the finding numbers, and not all of
the findings had recommendations.

These recommendations were intentionally broad and not specific. The
purpose of these recommendations was to highlight areas that needed to
be addressed by FAA management. The intent was not to dictate specific
solutions in the recommendations themselves. However, based on its
evaluation work, the team included specific suggestions that management
may choose to pursue. The suggestions are not part of the broad
recommendations and should allow the action officials flexibility in
deciding the best approach to solving a problem, since those officials
best know their operations and other factors that need to be considered.
The Program Evaluation Branch will follow-up on the recommendations, not
the suggestions. Any solution that corrects the problems identified in
the findings would be acceptable. The ultimate goal is to ensure that
acquisition reform continues to be successful.

The evaluation team recommends:

**3A** The Small Business Utilization Staff propose and pursue
additional incentives or other opportunities to assist FAA in meeting
its Major Procurement Program Goals.

#### Suggested Solutions:

1)  The FAA Administrator could reinstate the requirement to involve the
    Small Business Representative in all awards by amending AMS Section
    3.2.1.3.4 to state "\... individuals. However, for procurement
    requirements over \$100,000, the requesting sponsor should
    coordinate with the FAA Small Business Representative designee prior
    to advertising the announcement. FAA will work with the
    Department\...."

2)  The Small Business Utilization Staff could initiate additional Major
    Procurement Program Goal (MPPG) awareness through:

```{=html}
<!-- -->
```
a)  Holding informal workshops that include the mechanics of meeting
    MPPG goals and targeting the training to technical and program
    managers. This could be performed at Air Traffic Control Association
    events such as symposiums or conferences, or at other conferences or
    sessions frequently attended by technical and program managers.

b)  Encouraging the increased participation of technical and program
    managers at procurement or other conferences by inviting them as
    speakers or targeting activities at these conferences to the FAA
    technical and program managers.

c)  Holding workshops for Integrated Product Team leads to encourage
    their assistance in attracting prime contracting firms to the
    Mentor-Protégé Program and using this program when awarding
    contracts.

```{=html}
<!-- -->
```
3)  The Small Business Utilization Staff could pursue with FAA managers
    the requirement to include MPPG goals in performance appraisals for
    procurement personnel as well as technical and program managers. The
    evaluation team found that these goals already existed in many
    procurement personnel performance appraisals.

**4A** The co-chairs of the Integrated Product Leadership Team work with
the Integrated Product Team leads to establish and implement a plan to
reduce the risk of unexpected requirements, including those related to
software and interface issues and human factors processes.

#### Suggested Solutions:

1)  As required under the Acquisition Management System, each Integrated
    Product Team should ensure the right team composition is assigned to
    the program. If that is not feasible, Product Teams working with a
    potential software solution could coordinate with a knowledgeable
    central information system office during the planning stages of a
    program to ensure all known areas are considered. For administrative
    systems and programs, the Chief Information Officer could be
    responsible for providing this specific guidance. For air traffic
    control systems, the Air Traffic Services line of business could be
    responsible for providing this specific guidance.

2)  The Integrated Product Team leads could work together to develop an
    ad hoc peer review system to leverage program experiences, act as
    advisory bodies, and provide additional guidance during the planning
    phase from knowledgeable FAA and contractor personnel unrelated to
    the specific program. The peer review team, however, should not
    serve as an approval board, which could create delays in the system
    development process. This peer review system may help with
    identifying cost, schedule, technical, and risk areas early in the
    system development process. The peer review system may also be
    involved during critical design reviews and test readiness reviews.
    The elements of the peer review should be determined to provide the
    most useful feedback for the Integrated Product Teams. For example,
    the use of commercial off-the-shelf items when computer-human
    interface issues are involved could be considered for a peer review.

3)  The Integrated Product Teams could be required to develop and review
    lessons learned in the FAA Acquisition System Toolset (FAST)
    specifically related to requirements, software, and interface issues
    experienced by other teams on a regular basis throughout the
    program's lifecycle. Also, Integrated Product Teams could be
    required to enter all experiences, positive and negative, in FAST.

**4B** The FAA Acquisition Executive take the lead for the Joint
Resources Council to establish and implement a plan to mitigate negative
effects of budget cuts on programs.

#### Suggested Solutions:

1)  The System Engineering/Operational Analysis Team (SEOAT) could
    recommend budget cuts from only the lowest priority ranked programs,
    e.g., 25 percent. The budgets for the remaining programs, e.g.,
    75 percent, could then remain intact. Also, the SEOAT could
    recommend eliminating low priority programs to preserve the funding
    to higher priority programs. The interdependency of programs would
    have to be considered for any potential changes or cancellations.

2)  The Joint Resources Council could hold meetings specifically
    designated to decide if rebaselining for specific programs as a
    result of funding shortfalls is necessary. These special Joint
    Resources Council meetings should be held at least once, but no more
    than twice, each year.

3)  Each Integrated Product Team could establish and work toward a
    specified percentage below 100 percent of the approved acquisition
    program cost baseline. Incentives to meet the lower goals could be
    developed.

4)  The Product Teams impacted by budget cuts could be permitted to
    reevaluate, and possibly reduce, the number of products or systems
    they are expected to deliver. In addition, the Product Team could
    identify training or support areas to be reduced to accommodate
    budget cuts.

[]{#__RefHeading___Toc453556187 .anchor}Appendix EList of Acronyms

  ------------ ----------------------------------------------------------
  ACQUIRE      This is not an acronym. Instead, it is the name of a
               commercial off‑the‑shelf procurement system to replace
               both the System for Acquisition Management and the
               Procurement Automation System.

  AMS          Acquisition Management System

  ASR-11       Airport Surveillance Radar‑11

  FAA          Federal Aviation Administration

  FAST         FAA Acquisition System Toolset

  FY           Fiscal Year

  ITWS         Integrated Terminal Weather System

  MPPG         Major Procurement Program Goal

  NIMS         National Airspace System Infrastructure Management System

  OASIS        Operational and Supportability Implementation System

  SEOAT        Systems Engineering/Operational Analysis Team

  STARS        Standard Terminal Automation Replacement System

  WAAS         Wide Area Augmentation System

  WARP         Weather and Radar Processor

  WOB          Women Owned Business
  ------------ ----------------------------------------------------------

[^1]: The sampled contract numbers are listed in Appendix B, Tables B-1,
    B-2, and B-3.

[^2]: The sums of the two segments of time for FY 95 and FY 97 are off
    by one and two days, respectively, from the total average time
    because of variation in sample sizes and averaging within each of
    the segments of time used to determine procurement timeliness.

[^3]: "Disadvantaged businesses" refers to "small business concerns
    owned and controlled by socially and economically disadvantaged
    individuals." See Appendix C for further information on
    disadvantaged businesses.

[^4]: The FY 98 goals for small business awards and SEDB awards were
    five percent higher than the same goals for FY 97.

[^5]: The FY 98 goals for small business awards and SEDB awards were
    five percent higher than the same goals for FY 97.

[^6]: The law also covers requirements for terminating a program for
    performance goals.

[^7]: Recently, there has been a change in the rules that allow
    disadvantaged business firms to self certify their status. Because
    the fiscal years evaluated occurred prior to this change, the team
    did not address those changes in this report.

[^8]: The terms for contract awards were different under the Federal
    Acquisition Regulation and the Acquisition Management System, so
    both terms are listed here.