**Acknowledgments**

The data gathering and analysis that provided the basis for this
document were performed by the members of the Program Evaluation Branch
with the assistance and support of other individuals. The members of
this evaluation team, which included FAA Headquarters and Regional
personnel and FAA support contractor personnel, were responsible for
planning the evaluation, executing the plan, and reporting the results.

The team would like to thank all of those interviewed and surveyed for
their time and their suggestions. The evaluation team also extends
special thanks to those high level officials who reviewed the various
drafts of the report. These individuals, who represented the Research
and Acquisitions and Air Traffic Services lines of business and the
Office of the General Counsel, provided valuable inputs.

# Executive Summary {#executive-summary .Chapter}

On November 15, 1995, the President signed the Department of
Transportation and Related Agencies Appropriations Act of 1996 (Public
Law 104--50), which directed the FAA Administrator to:

...develop and implement...an Acquisition Management System for the
Federal Aviation Administration that addresses the unique needs of the
agency and, at a minimum, provides for more timely and cost-effective
acquisition of equipment and materials.

The FAA Reauthorization Act of 1996 (Public Law 104--264, dated October
9, 1996), expanding the procurement reforms previously authorized by
Public Law 104--50, provided the FAA Administrator increased autonomy in
carrying out the functions of the office and the authority to enter into
contracts, grants, cooperative agreements, and other transactions with
public and private entities on such terms and conditions as were
considered appropriate.

The FAA developed its Acquisition Management System (AMS) in response to
Public Laws 104--50 and 104--264. On April 1, 1996, the system went into
effect. In June 1997, the FAA published a revised and updated version of
the AMS policy document developed to further clarify and refine basic
concepts of AMS.

This report presents the results of the second annual evaluation of FAA
acquisition reform. When acquisition reform was initiated, the Associate
Administrator for Research and Acquisitions directed the Program
Evaluation Branch to report annually on the status of acquisition
reform. The FAA Program Evaluation Branch reported the results of the
implementation of AMS in the first year in its report, "Evaluation of
FAA Acquisition Reform, The First Year: April 1996--March 1997," dated
May 1997.

The first year evaluation report observed the progress of AMS
implementation, the impact of AMS on the FAA, and the effectiveness of
some parts of the acquisition process. The second annual evaluation
covered the two-year period April 1996--

March 1998, and assessed whether the FAA was on track to achieve its
goal of more timely and cost-effective acquisitions of equipment and
materials. The second annual evaluation was also planned and performed,
and results were reported, under the direction of the FAA Program
Evaluation Branch.

The objective for the second annual evaluation of AMS was to determine
whether the FAA was on track to achieve more timely and cost-effective
acquisitions of equipment and materials. To make that determination, the
evaluation team chose six component areas and developed an evaluation
objective for each area. The six critical areas represented key
processes, decision making bodies, and systems within acquisition
reform, and included: (1) mission analysis, (2) investment analysis, (3)
baseline management, (4) the Joint Resources Council (JRC), (5) the
Integrated Product Development System (IPDS), and (6) procurement. The
following table lists the objectives for each of the six component areas
evaluated.

This second annual report covers the first two years of AMS. During the
first year review, the evaluation team relied more on perceptions and
opinions of affected and surveyed individuals because AMS was so new and
little documentation existed at the time of the review. During the
second year review, the evaluation team relied more on supporting
documentation and evidence.

Second year evaluation activities addressed those AMS processes,
systems, and decision making bodies that were crucial to the success of
acquisition reform. The Program Evaluation Branch relied on a variety of
data sources for its analysis of acquisition reform progress in the FAA,
including:

-   reviews of AMS policy, guidance, and procedure documents;

-   interviews and surveys of FAA and contractor personnel;

-   reviews of meeting minutes and Records of Decision from the JRC, the
    Systems Engineering/Operational Analysis Team (SEOAT), and IPDS
    teams;

-   analyses of outputs from FAA databases and information systems;

-   analyses of AMS-related program documents (e.g., Mission Need
    Statements, Investment Analysis Reports, Acquisition Program
    Baselines (APBs) and other baseline documents, Requirements
    Documents, Integrated Program Plans, Acquisition Strategy Papers);
    and

-   reviews of IPDS organizational charters and Integrated Product
    Team/Product Team (IPT/PT) Plans.

In addition, samples were selected and reviewed within each of the six
evaluation areas. The samples selected for all six areas also included
four selected acquisition programs that moved through various stages of
the lifecycle acquisition process. These four programs included:

-   Air Traffic Control Beacon Interrogator Replacement (ATCBI--R),

-   National Airspace System Infrastructure Management System (NIMS),

-   Operational and Supportability-Implementation System (OASIS), and

-   Standard Terminal Automation Replacement System (STARS).

These four programs were selected because, in addition to providing the
evaluation team with a common set of acquisition programs from which to
measure results, they represented high-visibility FAA acquisitions being
accomplished through the new AMS planning, analysis, and execution
processes.

# Results

AMS has been in place for two years, and the FAA has made significant
progress toward implementing procedures designed to achieve cost and
schedule goals. After the second year, the AMS process was moving in the
right direction, but it was still too early to validate the success of
acquisition reform.

The agency has achieved more timely acquisitions, but further
improvements are necessary if acquisition reform is going to meet the
agency's ambitious goals of reducing the time to field quality products
and services by 50 percent and the cost by 20 percent in the third year.
It is essential that improvements identified be fully implemented
quickly so the effects can, at the very least, be measured to determine
if the FAA's goals have been realized.

# Progress

Continuous improvements from the first year evaluation report were
identified in every major area. Following are some of the achievements
and highlighted improvements implemented under AMS:

-   FAA employees, managers, and executives were serious about
    acquisition reform and AMS, and they made diligent efforts to follow
    the AMS processes.

-   Cross-functional, multi-disciplinary teams did exist and worked
    together.

-   Major procurements were more timely.

-   Most FAA employees endorsed and accepted the IPDS structure.

-   Content of JRC Records of Decision improved significantly.

-   The agency held numerous acquisition reform and AMS training classes
    in the various AMS processes and concepts.

-   Many FAA Investment Analysis Teams formed according to AMS policy.

-   The percentage of FAA Facilities and Equipment funded programs
    having JRC-approved baseline documentation increased from 13 percent
    reported in the first year of AMS to 54 percent in the second year
    of AMS.

-   The IPDS structure was implemented, acquisitions were included
    within this structure, and almost half of the IPT/PTs had a formal
    plan for guidance.

-   After an initial decline immediately following the implementation of
    AMS, the FAA made progress toward achieving socio-economic goals.

-   The FAA increased the percentage of major procurements awarded
    through competition.

# Improvements Needed

Although there were continuous improvements during the second year of
AMS, the evaluation team found additional improvements were necessary.
Specifically, the team found that improvements were needed:

-   in the mission analysis process, including improvements in
    identifying, validating, and prioritizing mission needs; receiving
    full support from all lines of business; and coordinating between
    the Mission Analysis Steering Group (MASG) and the National Airspace
    System (NAS) Architecture core team;

-   in the investment analysis process in documenting management
    intervention in solution identification;

-   to AMS in addressing SEOAT activities dealing with affordability
    assessments;

-   in baseline management, including establishing complete baselines
    and compiling and tracking baseline information to assist the agency
    in measuring and monitoring a program's progress;

-   in the JRC activities to ensure Operations and Maintenance funds
    were fully considered in lifecycle funding decisions and to ensure
    action items were tracked and monitored;

-   to the FAA Joint Resources Council Guidance to develop a realistic
    time frame for disseminating Records of Decisions;

-   in the IPDS structure to ensure IPDS was fully implemented, operated
    free of unnecessary constraints, and was effective in supporting
    successful acquisitions;

-   in the procurement segment to ensure the accuracy and completeness
    of data used to measure results; and

-   to the FAA's overall planning process.

# Findings

**Finding 1: Improvements were needed in identifying, validating, and
prioritizing mission needs.**

The agency did identify critical needs and did move forward under AMS to
address these critical needs. We found the agency conducted mission
analysis on the six programs reviewed in our sample. We also found the
agency had documented 329 mission needs in Mission Need Statements.
However the agency did not follow the AMS process in validating and
prioritizing *all* critical needs that were identified and structured
under mission analysis. Additionally, in some cases, a structured
mission analysis was not conducted and a Mission Need Statement was not
prepared.

This occurred because: (1) some critical needs had been previously
addressed in programs approved prior to AMS, (2) the lines of business
did not always recognize the necessity for identifying and developing
Mission Need Statements, and (3) the MASG had not developed guidance on
how to prioritize new and existing mission needs.

When critical needs were identified and validated, the agency was better
able to plan for the most cost-effective use of resources and provide
support for future budget and resource requests. Prioritizing agency
needs would also improve the FAA's ability to plan effectively.

**Finding 2: Improvements were needed in support of mission analysis and
in coordination between the Mission Analysis Steering Group and the
National Airspace System Architecture core team.**

We found the Associate Administrators for the two lines of business
heavily involved in the acquisition process, Air Traffic Services and
Research and Acquisitions, complied with AMS and supported the mission
analysis process. The other five lines of business did not have the
direct need to become as active in mission analysis and did not have the
resources to support mission analysis to the level described by AMS. We
found those five lines of business had not established mission analysis
teams and had not provided their staffs with sufficient training in the
mission analysis area. Recognizing this difficulty, the MASG and Air
Traffic Services indicated their willingness to provide the needed
support in the area of mission analysis to these other lines of
business.

We also found that while many MASG members supported the mission
analysis process, as many as half of the MASG members were not fully
involved, indicating mission analysis was given a low priority by the
MASG members and by their respective lines of business.

In addition, improvement was needed in coordination between the NAS
Architecture core team and the MASG. We found the NAS Architecture core
team did not coordinate its critical needs and opportunities with the
MASG in preparation of the NAS Architecture Version 3.0. In turn, the
MASG did not coordinate mission analysis with the NAS Architecture core
team for critical needs and opportunities. Improving coordination
between the NAS Architecture Core team and the MASG would reduce the
likelihood of duplicated mission analysis effort, improve capability to
identify all needs and eliminate conflicting overall FAA mission needs,
and provide for the more efficient use of agency resources.

**Finding 3: Management control improvements were needed for the FAA
mission analysis process.**

We found the FAA had not established internal controls to ensure Mission
Need Statements would be revalidated, when necessary. As a result, the
agency did not know if 56 mission needs were still valid and up to date,
or if four new Mission Need Statements duplicated needs identified on
prior Mission Need Statements.

We also found the FAA did not establish internal controls to ensure all
historical records related to mission needs were collected in a central
repository. We found the agency did establish an official repository for
all JRC-approved documents in the Evaluation and Configuration
Management Division (ASD-200) and an unofficial repository for other
acquisition planning and control documents in the National Airspace
System Concept Development Branch (ASD-130). However, lines of business
did not maintain or provide copies of all supporting documentation for
acquisition planning and control documents to the unofficial repository.
This occurred because the FAA (1) had not formally established a
repository for historical records, other than for JRC-approved
documents, relating to all submitted Mission Need Statements, and (2)
had not established an internal control to ensure the lines of business
maintained all supporting records and provided these supporting
documents to a central repository.

**Finding 4: Investment Analysis Teams formed according to AMS, but
investment analysis actions were not fully documented.**

The FAA Investment Analysis Teams were formed according to AMS policy in
five of the six sample programs reviewed. However, we found the sponsor
or upper level management identified or limited solutions prior to the
completion of the alternatives analysis process in two of the six cases,
and paperwork prepared to support the investment analysis process did
not necessarily document what actually occurred. For example, in one
case, the Requirements Document was written by the sponsor to drive a
specific solution, and the sponsor was reluctant to negotiate and
identify a core set of top level requirements. In another case, the
solution for a high priority program was identified by upper level
management while the Investment Analysis Team was still evaluating
different options.

When upper level management intervened in the investment analysis
process, it limited the scope of the investment analysis but did not
reduce the level of effort required by the Investment Analysis Team.
Because management intervention was not fully documented, it was not
clear whether there was a problem within the investment analysis process
or whether other concerns drove management's need to intervene. In
addition, when the intervention was not fully documented, the full
decision body may not have been aware an intervention occurred.

**Finding 5: SEOAT affordability assessment requirements under AMS need
to be revised.**

We found the SEOAT affordability assessment activities did not comply
with AMS and may not be able to comply with AMS. AMS required the SEOAT
to perform an affordability assessment on all candidate solutions. In
most cases, the SEOAT performed an affordability assessment on the
recommended candidate solution only. This occurred because time and
resource constraints did not permit the level of effort necessary to
perform an affordability assessment on each candidate solution.

**Finding 6: The FAA did not establish complete baselines.**

During this review, the evaluation team discovered the FAA had not
established definitions for, or designations of, "programs,"
"substantial programs," or "major systems acquisitions," which were
essential in identifying the number of programs requiring an APB. In
addition, since the FAA Administrator was required to consider
terminating "substantial programs" that failed to meet defined goals,
the agency needed to define what a "substantial program" was and set
goals for those programs. The agency also was required to report on
"major systems acquisitions" and needed to define that term in order to
comply.

Although programs were not defined by the agency, the evaluation team
defined and identified 94 Capital Investment Plan programs that should
have had APBs. The Capital Investment Plan included Facilities and
Equipment (F&E) funded programs; we found no JRC-approved baseline
documentation for Research, Engineering and Development (R,E&D) only or
Operations and Maintenance (O&M) only programs. For the 94 Capital
Investment Plan programs, the evaluation team found 51 (54 percent) had
some form of baseline documentation: 5 programs had APBs and 46 had
Parameter Sheets. While the other 43 programs had no JRC-approved
baseline documentation at all, the 54 percent that did have JRC-approved
baseline documentation represented a dramatic improvement over the 13
percent that had JRC-approved baseline documentation in the first year
of AMS.

Of the 51 programs with JRC-approved baseline documentation, we found
those documents did not always include enough information to measure and
monitor the program's performance. The evaluation team found that
baseline data included in APBs was significantly more complete and
comprehensive than baseline data in Parameter Sheets. For example,
performance baselines were complete in APBs, but were incomplete in most
Parameter Sheets. Benefits baselines were complete in most APBs, but
were incomplete in most Parameter Sheets. Schedule baselines were
incomplete for most APBs, but were incomplete for all Parameter Sheets.
Cost baselines for all fundings appeared complete for APBs, but was
incomplete for all Parameter Sheets.

The evaluation team found R,E&D funding was frequently not included in
baseline documents because it was regarded as a "sunk cost" by the time
the investment decision was made and the APB was approved. In addition,
O&M funding was not always included on the baseline documents because
(1) although O&M funding was provided by the sponsor for each program
baseline, it was unclear which organization, if

any, was responsible for all planning and programming for O&M funds; (2)
little, if any, current or historical O&M funding data existed, and what
was available tended to be unreliable; and (3) the FAA did not have a
cost accounting system capable of identifying real O&M costs for
existing systems or projected O&M costs for future new systems. As a
result, there was very little credibility associated with the O&M cost
estimates contained in APBs and Parameter Sheets. These poor or
inaccurate cost estimates contributed to the incompleteness of the
baseline documentation and limited the FAA's ability to make sound
investment decisions based on the economic merits of a proposal.

Even though APBs were more complete than Parameter Sheets, neither type
of document was always sufficient to measure a program's progress or to
allow decision makers to make sound decisions on continuing investments.
Approved baseline documentation was intended to provide a mutual
agreement between FAA management, represented by the JRC, and the
provider, which was typically the IPT. The absence of formal baseline
documentation against which to measure progress restricted decision
makers' abilities to make sound decisions on continued investment.

**Finding 7: Baseline data was not compiled and tracked to assist the
agency in measuring and monitoring a program's progress.**

The FAA collected and reported numerous baseline data, but did not
collect or track complete or consistent baseline data in order to
measure and monitor a program's progress. This occurred because the
numerous systems and processes used for tracking program status were not
integrated, and the FAA did not ensure a standard set of data was
collected for comparisons. Also, the data in these systems and processes
did not necessarily reflect current JRC-approved program baselines. As a
result, there was no thorough oversight or tracking of some baselines,
and there was no commitment to periodic revalidation of baselines. In
addition, FAA management received limited visibility into program status
after the investment decision.

**Finding 8: The Joint Resources Council did not make fully supported
and timely decisions.**

We found the JRC did not make fully supported decisions when it
committed the FAA to full lifecycle funding without consistently
considering O&M funds. Although the JRC had the responsibility to make
decisions committing the FAA to full lifecycle funding, the JRC did not
have the data needed to commit the FAA's O&M funding. The FAA did not
have a cost accounting system in place that enabled identification of
realistic O&M costs. The agency was developing a cost accounting system,
which is expected to be in place by October 1, 1998. Without an
operational cost accounting system or an effective financial baseline
for the required O&M funds, the FAA could not determine if a program was
affordable over its full lifecycle. There was also confusion over the
role and authority of the Resource Management Council.

In addition, while we found JRC meeting attendance was generally good,
meetings were frequently rescheduled, which delayed JRC decisions. JRC
meetings were rescheduled when a key JRC member requested the meeting be
rescheduled, on a case-by-case basis if key participants were not ready
for the meeting or had a conflicting obligation, or the required
documentation was not provided three days prior to the meeting. When JRC
meetings were rescheduled, decisions that would have been made at those
meetings were delayed. This had the potential to impact other phases of
the acquisition process, program schedules and contractual activities.
It also caused staff time to be lost.

**Finding 9: The Joint Resources Council fully documented its decisions
and rationale for decisions; however, timeliness in disseminating
information and tracking action items needed improvement.**

We found the JRC fully documented its decisions and rationale for
decisions during the second year of AMS. During the second year of AMS,
JRC Records of Decision were expanded from a one-page document with
little information to a full explanation of JRC decisions, including
rationale, discussion, and participant quotes. With improved JRC Records
of Decision, the FAA had a better historical record of what occurred at
JRC meetings and better documentation of the rationale for decisions
made.

Timeliness in disseminating information and tracking action items,
however, needed improvement. The FAA Joint Resources Council Guidance
required the JRC Record of Decision to be prepared and issued within
five working days. We found it took an average of 36 working days to
disseminate these Records of Decision during the first year of AMS and
36.9 working days the second year of AMS. While we believe 36 working
days was too long, 5 working days was unrealistic.

We also found there was no formal process in place to track action items
assigned at JRC meetings during the time of our review. The informal
process being used to track action items assigned at JRC meetings was
not adequate, and some JRC meetings adjourned without a clear
understanding among JRC members and participants of the specific action
items assigned, including what was required, who was responsible, and
when the action was to be completed. JRC members and participants were
sometimes confused about assigned action items, responsible
organizations, and due dates. Although action items were recorded in the
JRC Records of Decision, they were not always captured effectively
*during* the meetings to ensure JRC members and participants understood
and agreed to them. Because JRC Records of Decision were the only formal
documentation of JRC decisions, until they were signed and disseminated,
some actions required by the JRC decision were not implemented. As a
result, actions may not have been completed appropriately. In addition,
without a tracking process to close out completed action items, the JRC
could not monitor action items assigned. Before we completed our review,
we confirmed that an automated system to track, follow up on, and close
assigned action items was being designed to address some of these
concerns.

**Finding 10: Some constraints impacted the support of successful
acquisitions within the Integrated Product Development System.**

The agency initiated IPDS prior to acquisition reform and formally
incorporated IPDS into the acquisition reform process during the
development of AMS. Technically, an IPDS structure can consist of
something as informal as the contracting officer and someone requesting
the product. Using this as a basis, we concluded that the IPDS structure
was implemented. Irrespective of that broad definition, we found that
most acquisitions were included within the formal IPDS structure, and
almost half of the IPT/PTs had a formal plan for guidance. There were
some constraints, however, which impacted IPDS in supporting successful
acquisitions. There were also areas where AMS requirements were more
stringent than necessary under the circumstances.

Constraints impacting the ability of the IPDS structure to support
successful acquisitions included: (1) conflicts between the horizontal
(IPDS team lifecycle responsibilities concept) and vertical (line of
business functional responsibilities concept) structures, (2) the lack
of authority of IPT/PT leadership over all team members, and (3) an
inability of IPDS teams to resolve major agency issues. These
constraints delayed decisions, which could have prolonged the
acquisition process and, in turn, could have increased acquisition
costs.

In addition, we found some areas where AMS requirements or IPDS Team
Plan Guidelines were more stringent than necessary under the
circumstances. For example, we found not all acquisitions were suitable
for inclusion within the formal IPT structure as required by AMS.
Acquisitions such as facilities or services may not need to be under the
IPT structure; however, there were no guidelines to clarify which
acquisitions did not need to be under a formal IPT structure. The IPDS
Team Plan Guidelines required IPT/PTs to have formal plans. However,
some IPT/PTs may not need to be under a formal plan at this time. We
found some IPT/PTs were able to manage acquisitions without a formal
plan. We did find, however, that 25 out of 53 (47 percent) of the
IPT/PTs were under formal plans during the second year under AMS,
representing a marked improvement over the first year of AMS. The goal
to have a formal plan for all formal IPDS teams is a good one and should
continue to be implemented.

**Finding 11: Major procurements were more timely under AMS.**

Major procurements were more timely under AMS while costs to conduct
procurements changed very little. Based on analysis of the random sample
selected, the evaluation team determined contracts were awarded an
average of 56.2 days faster after the implementation of AMS while
dollars expended to award contract actions dropped from \$8 million to
\$7.6 million. The overall goal of FAA acquisition reform was to reduce
the time to acquire and field new systems by 50 percent and to reduce
the costs of acquisitions by 20 percent, both by April 1999. These
results would be measured from the investment decision through first
commissioning of a new system. The portion of procurement timeliness
covered in this section of the AMS evaluation report represents the
procurement segment of the acquisition process only.

While major procurements were more timely under AMS, and costs to
conduct procurements changed very little, the evaluation team was still
unable to quantify whether procurements were more cost-effective because
cost-effectiveness includes more than just quicker (more timely)
procurements. Cost-effectiveness also includes costs to conduct the
procurements, the cost of the procurements themselves, and all other
acquisition costs. Because sufficient information did not exist to
analyze costs of acquisitions, we looked only at the costs to conduct
the procurements. We found dollars expended to award contract actions
dropped from \$8 million to \$7.6 million while contract dollars awarded
dropped from \$551 million to \$495 million. Since the implementation of
AMS, total dollars expended for procurement operations, total number of
employees conducting procurement operations, and total dollars awarded
for contracts all decreased. However, without a detailed cost accounting
system to (1) segregate procurement operations conducted by procurement
personnel but not related to procurement award activities, such as
contract administration, etc., from the total procurement expense; and
(2) capture full costs of acquisitions, we were unable to make a
conclusion on whether procurements were more cost-effective.

**Finding 12: The FAA appeared to have made progress toward achieving
socio-economic goals.**

After an initial decline immediately following the implementation of
AMS, the FAA appeared to have made progress toward achieving
socio-economic goals. However, because the data used to report progress
in this area was incomplete or inaccurate, the evaluation team could not
accurately determine the true impact of AMS on socio-economic goals.

Although awards to businesses identified in the socio-economic program
goals decreased in the six-month period immediately following the
implementation of AMS, the FAA made steady improvement in awards to
those businesses after the initial six-month period. For example, the
percentage of awards to small businesses in general decreased from 27
percent in the last six-month period prior to AMS to 18 percent for the
six-month period immediately following the implementation of AMS. By
contrast, when the evaluation team compared the results from the first
six-month period following the implementation of AMS to the results from
the full seven quarters following AMS, we determined that the percentage
of awards to small businesses increased from 18 percent to 52 percent.
Similarly, set-aside awards to disadvantaged businesses certified by the
Small Business Administration under Section 8(a) (referred to as SEDB
awards) increased from 10 percent for the first six-month period to 12
percent for the full seven quarters following AMS, and awards to
disadvantaged businesses that were self-certified or certified by the
Small Business Administration but received awards through a process
other than the set-aside process (referred to as SDB awards), increased
from 2 percent to 3 percent.

The evaluation team identified these improvements based on information
from Major Procurement Program Goals reports submitted to the FAA's
Small Business Utilization Staff. However, we determined information
submitted for the Major Procurement Program Goals reports was frequently
incomplete or inaccurate. In addition, reporting was inconsistent among
the FAA Regions, Centers, and Headquarters.

In a separate report entitled "Impact of Acquisition Reform on Awards to
Disadvantaged Businesses" issued May 20, 1998, the FAA's Program
Evaluation Branch also identified inconsistencies in the data and
weaknesses in the systems used to collect and report the FAA's progress
in meeting socio-economic goals.

Without accurate data, the FAA cannot know whether or not it is meeting
established socio-economic goals, whether additional efforts to meet
those goals are necessary, or whether any additional efforts put in
place were effective.

**Finding 13: The percentage of contract dollars awarded through
competition increased.**

The FAA increased the percentage of major procurement contracts awarded
through competition. To determine the level of competition under AMS,
the evaluation team collected and compared data from the Contract
Information System database for the eight quarters pre-AMS and the seven
quarters under AMS. The Contract Information System contained data for
procurements over \$25,000 and identified whether the contracts were
awarded through competition or through a noncompetitive process. After
averaging to allow for the different number of quarters for the pre-AMS
and AMS periods, we found awards through competition increased from 64
percent before AMS to 70 percent under AMS.

**Finding 14: The FAA's overall planning process needed improvement.**

While performing the six component evaluations, the evaluation team
identified one issue that crossed almost all acquisition processes and
functional areas. Because of the interrelated and recurring nature of
this issue, we separated it from the rest of the evaluation chapters.

We found that the agency cut funding to multiple and, in some cases,
high priority programs to meet its funding levels and changing
priorities because it had not properly planned for these issues. During
the past year, there were two major issues that affected funding levels
of numerous agency programs. First, the agency assumed it would receive
a fiscal year 1999 funding level that was almost \$500 million greater
than what was agreed to by the Office of Management and Budget (OMB).
Second, the agency did not anticipate \$167 million of reprogramming
priorities that surfaced in fiscal year 1998 after the agency received
its budget for the year.

The FAA planned and approved program funding profiles within the agency
and submitted its fiscal year 1999 budget request to the OMB for almost
\$500 million higher than the OMB outyear-target-level guidance (i.e.,
OMB guidance on the maximum funding level to request from the Congress).
In conjunction, the FAA prepared the NAS Architecture with the
assumption that OMB would submit this higher request to Congress. The
OMB denied the FAA's higher budget request to Congress. When this was
disallowed, many Capital Investment Plan lines had to be reworked. The
FAA's submission of a budget request higher than the OMB
outyear-target-level was a one time action on FAA's part and will not
likely reoccur.

In the case of the fiscal year 1998 appropriation from Congress, four
unplanned emerging priorities surfaced. These four priorities were (1)
Year 2000, (2) Host and Oceanic Computer System Replacement (HOCSR), (3)
additional Standard Terminal Automation Replacement System (STARS)
requirements, and (4) additional security funding. These four unplanned
emerging priorities resulted in a \$167 million shortage which forced
the FAA to reprogram funding on 35 of 113 programs. One of the four
unplanned emerging priorities (the security issue) was outside of the
FAA's control; however, the other three issues were under the FAA's
control and required better planning.

Because the agency was not able to plan based on realistic funding
levels and did not anticipate the emerging priority needs, the agency
had to adjust cost baselines and reprogram funding from numerous
programs. These actions resulted in deferring the planned implementation
of crucial programs and diminished trust between planners, developers,
and decision makers when their program work was undermined by deviations
from baseline expectations and agreements.

# Recommendations

The following lists all the recommendations included in this report. The
recommendation numbers correlate to the finding numbers; not all of the
findings had recommendations. We limited our recommendations to specific
problems identified during the evaluation. Our evaluation was limited to
acquisition reform and did not address all of the FAA's operations.
While we believe there are other recommendations that could help the FAA
improve its overall operations, we limited our recommendations in this
report to those issues we found within acquisition reform. In addition,
we were intentionally broad and not specific in our recommendations. The
purpose of our recommendations was to highlight areas that needed to be
addressed by the FAA management. Any solution that corrects the problems
identified in our findings would be acceptable. The ultimate goal is to
ensure that acquisition reform succeeds.

The evaluation team recommends:

**1A** The FAA Administrator ensure that *all* *critical* needs and
opportunities emerge from structured mission analysis, are developed
into Mission Need Statements with a recommended prioritization, are
reviewed for validation, and, if approved, are assigned a prioritization
relative to all other approved Mission Need Statements, in accordance
with AMS.

**1B** The chairperson of the Mission Analysis Steering Group prepare
guidance on how to prioritize new and existing mission needs for Joint
Resources Council approval, and ensure approved guidance is implemented.

**2A** The chairperson of the Mission Analysis Steering Group ensure the
Mission Analysis Steering Group representatives (both members and their
alternates) fulfill their responsibilities to the Mission Analysis
Steering Group, to include ensuring the AMS mission analysis process is
followed and that all mission analysis work in the agency, including the
National Airspace System Architecture, is coordinated with the Mission
Analysis Steering Group.

**2B** The Associate Administrator for each line of business ensure
sufficient qualified and trained staff and the appropriate resources are
available to perform mission analysis work *or* delegate this
responsibility and authority to the Mission Analysis Steering Group or
Air Traffic Services mission analysis team as appropriate and agreed
upon.

**3A** The chairperson of the Mission Analysis Steering Group develop
guidance and procedures on how to conduct and document revalidation of
mission needs, and establish a plan with a schedule for accomplishing
revalidations.

**3B** The chairperson of the Mission Analysis Steering Group develop
guidance to establish a central repository for historical records of all
supporting documentation for submitted Mission Need Statements, and to
ensure each line of business submits all acquisition planning and
control supporting documentation to this central repository.

**4A** The FAA Acquisition Executive require the changed direction and
rationale for any intervention in the investment analysis process be
documented at the time of the intervention, and ensure this
documentation is included in the Investment Analysis Report used at the
investment decision.

**5A** The chairperson of the Acquisition System Advisory Group revise
the AMS to allow the Systems Engineering/Operational Analysis Team, in
conjunction with the Investment Analysis Team, flexibility to determine,
on a case-by-case basis, how many viable candidate solutions will have
an affordability assessment completed.

**6A** The FAA Administrator ensure definitions are developed for
"program," "substantial program," and "major systems acquisitions," and
that all FAA programs are designated in the appropriate category.

**6B** The FAA Acquisition Executive ensure every program, existing or
new, has an Acquisition Program Baseline as required by the AMS, or an
alternate baseline document, that includes all applicable cost
(Facilities and Equipment; Research, Engineering and Development; and
Operations and Maintenance), schedule, benefits, and performance
parameters.

**6C** The FAA Administrator require those organizations or individuals
with signature authority for funding, including Facilities and
Equipment; Research, Engineering and Development; and Operations and
Maintenance, provide all the funding data and their signatures to
authorize and commit these funds for each Acquisition Program Baseline.

**7A** The FAA Administrator designate a single focal point for all
program baseline management and reporting activities.

**7B** The FAA Administrator direct the development of an adequate
tracking and validation system for benefits, or revise the AMS if it is
determined the benefits baselining requirement should not be retained.

**7C** The FAA Administrator design and implement procedures for
tracking and reporting program status against all approved baselines.

**8A** The FAA Administrator combine the responsibility and authority
for full lifecycle funding under one Associate/Assistant Administrator
level body and clearly define its role, responsibilities, and authority.

**8B** The FAA Acquisition Executive ensure each Joint Resources Council
member designates a single alternate and empowers this alternate to act
in his/her place, and each key Joint Resources Council meeting
participant designates a single alternate and empowers this alternate to
act in his/her place in order to reduce rescheduling caused by calendar
conflicts and to maintain consistency at meetings. These alternates
should possess sufficient knowledge of the duties and responsibilities
of the Joint Resources Council member or key participant and should have
the appropriate corporate level perspective to serve in this position.

**9A** The FAA Acquisition Executive revise the FAA Joint Resources
Council Guidance requirement to issue Records of Decision within five
days to a realistic period of time; and, develop and document a process
for the timely preparation, approval, and dissemination of Joint
Resources Council Records of Decision, and ensure that process is
followed.

**9B** The FAA Acquisition Executive establish an effective means to
capture assigned action items *during* Joint Resources Council meetings
(including what is required, who is responsible, and when the action is
to be completed) to ensure members and participants understand and agree
to all action items assigned; and define and implement a process to
track, follow up, close, and disseminate results of action items *after*
Joint Resources Council meetings.

**10A** The FAA Administrator ensure that Integrated Product
Teams/Product Teams and functional management/organizations jointly
select team members, document agreements clarifying team members' roles
and responsibilities, and evaluate team member performance.

**10B** The chairperson of the Integrated Product Leadership Team ensure
issues raised within the Integrated Product Development System structure
are resolved.

**12A** The Special Assistant, Small Business Utilization Staff, ensure
that FAA Regions, Centers, and Headquarters have accurate, current
guidance for development of Major Procurement Program Goals reports and
comply with that guidance.

> Additional recommendations related to achieving socio-economic goals
> were addressed in Report #1998-01, "Impact of Acquisition Reform on
> Awards to Disadvantaged Businesses," dated May 20, 1998.

**14A** The FAA Administrator ensure that the agency performs better
planning for its future funding needs to avoid constant funding
reductions to on-going programs. This would involve improving the
agency's (1) mission analysis capabilities, which must be tied strongly
to the agency's approved National Airspace System Architecture; (2)
investment analysis capabilities; and (3) managing of program cost,
schedule, performance, and benefits baselines.

**Contents**

**Executive Summary [i](#executive-summary)**

Results [iii](#results)

Progress [iii](#progress)

Improvements Needed [iv](#improvements-needed)

Findings [v](#findings)

Recommendations [xiii](#recommendations)

Second Year Evaluation Overview [1](#second-year-evaluation-overview)

Evaluation Objectives [1](#evaluation-objectives)

Scope [2](#scope)

Methodology [3](#methodology)

Overall Evaluation Results in Brief
[3](#overall-evaluation-results-in-brief)

Follow-Up from the First Year Evaluation
[6](#follow-up-from-the-first-year-evaluation)

Chapter 1Mission Analysis [7](#chapter-1mission-analysis)

Evaluation Objective [7](#evaluation-objective)

Scope and Methodology [7](#scope-and-methodology)

Mission Analysis Results in Brief
[8](#mission-analysis-results-in-brief)

Finding 1: Improvements were needed in identifying, validating, and
prioritizing mission needs.
[8](#finding-1-improvements-were-needed-in-identifying-validating-and-prioritizing-mission-needs.)

Finding 2: Improvements were needed in support of mission analysis and
in coordination between the Mission Analysis Steering Group and the
National Airspace System Architecture core team.
[10](#finding-2-improvements-were-needed-in-support-of-mission-analysis-and-in-coordination-between-the-mission-analysis-steering-group-and-the-national-airspace-system-architecture-core-team.)

Finding 3: Management control improvements were needed for the FAA
mission analysis process.
[11](#finding-3-management-control-improvements-were-needed-for-the-faa-mission-analysis-process.)

Chapter 2Investment Analysis [13](#chapter-2investment-analysis)

Evaluation Objective [13](#evaluation-objective-1)

Scope and Methodology [13](#scope-and-methodology-1)

Investment Analysis Results in Brief
[14](#investment-analysis-results-in-brief)

Finding 4: Investment Analysis Teams formed according to AMS, but
investment analysis actions were not fully documented.
[14](#finding-4-investment-analysis-teams-formed-according-to-ams-but-investment-analysis-actions-were-not-fully-documented.)

Finding 5: SEOAT affordability assessment requirements under AMS need to
be revised.
[15](#finding-5-seoat-affordability-assessment-requirements-under-ams-need-to-be-revised.)

Chapter 3Baseline Management [17](#chapter-3baseline-management)

Evaluation Objective [17](#evaluation-objective-2)

Scope and Methodology [17](#scope-and-methodology-2)

Baseline Management Results in Brief
[17](#baseline-management-results-in-brief)

Finding 6: The FAA did not establish complete baselines.
[18](#finding-6-the-faa-did-not-establish-complete-baselines.)

Finding 7: Baseline data was not compiled and tracked to assist the
agency in measuring and monitoring a program's progress.
[21](#finding-7-baseline-data-was-not-compiled-and-tracked-to-assist-the-agency-in-measuring-and-monitoring-a-programs-progress.)

Chapter 4The Joint Resources Council
[25](#chapter-4the-joint-resources-council)

Evaluation Objective [25](#evaluation-objective-3)

Scope and Methodology [25](#scope-and-methodology-3)

Joint Resources Council Results in Brief
[26](#joint-resources-council-results-in-brief)

Finding 8: The Joint Resources Council did not make fully supported and
timely decisions.
[26](#finding-8-the-joint-resources-council-did-not-make-fully-supported-and-timely-decisions.)

Finding 9: The Joint Resources Council fully documented its decisions
and rationale for decisions; however, timeliness in disseminating
information and tracking action items needed improvement.
[28](#finding-9-the-joint-resources-council-fully-documented-its-decisions-and-rationale-for-decisions-however-timeliness-in-disseminating-information-and-tracking-action-items-needed-improvement.)

Chapter 5The Integrated Product Development System
[31](#chapter-5the-integrated-product-development-system)

Evaluation Objective [32](#evaluation-objective-4)

Scope and Methodology [32](#scope-and-methodology-4)

Integrated Product Development System Results in Brief
[32](#integrated-product-development-system-results-in-brief)

Finding 10: Some constraints impacted the support of successful
acquisitions within the Integrated Product Development System.
[33](#finding-10-some-constraints-impacted-the-support-of-successful-acquisitions-within-the-integrated-product-development-system.)

Chapter 6Procurement [37](#__RefHeading___Toc421596219)

Evaluation Objective [37](#evaluation-objective-5)

Scope and Methodology [37](#scope-and-methodology-5)

Procurement Results in Brief [38](#procurement-results-in-brief)

Finding 11: Major procurements were more timely under AMS.
[38](#finding-11-major-procurements-were-more-timely-under-ams.)

Finding 12: The FAA appeared to have made progress toward achieving
socio-economic goals.
[39](#finding-12-the-faa-appeared-to-have-made-progress-toward-achieving-socio-economic-goals.)

Finding 13: The percentage of contract dollars awarded through
competition increased.
[42](#finding-13-the-percentage-of-contract-dollars-awarded-through-competition-increased.)

Chapter 7Interrelated and Recurring Issue
[43](#chapter-7interrelated-and-recurring-issue)

Finding 14: The FAA's overall planning process needed improvement.
[43](#finding-14-the-faas-overall-planning-process-needed-improvement.)

Recommendations [45](#recommendations-11)

Appendix ACriteria A--[1](#appendix-acriteria)

Appendix BList of Acronyms B--[1](#appendix-blist-of-acronyms)

**List of Tables**

Table 1--1. Evaluation Objectives for Component Areas

Table 1--2. AMS Training Statistics

Table 1--3. Status of First Year Evaluation Recommendations

Table 3--1. Completeness of Baseline Documents

Table 3--2. Program Reporting

Table 6--1. Socio-economic Program Goals: Comparison of Six-month
Periods Preceding and Following AMS

Table 6--2. Socio-economic Program Goals: Comparison of First Six-month
Period to First Seven Quarters Following AMS

Table 7--1. High Priority Programs With Reprogramming Reductions

**List of Figures**

[Figure 3--1. Baseline Document Status 19]{.smallcaps}

[Figure 5--1. IPDS Structure 31]{.smallcaps}

[Figure 5--2. Vertical vs. Horizontal Structure 34]{.smallcaps}

# Second Year Evaluation Overview {#second-year-evaluation-overview .Chapter}

On November 15, 1995, the President signed the Department of
Transportation and Related Agencies Appropriations Act of 1996 (Public
Law 104--50), which directed the FAA Administrator to:

...develop and implement...an Acquisition Management System for the
Federal Aviation Administration that addresses the unique needs of the
agency and, at a minimum, provides for more timely and cost-effective
acquisition of equipment and materials.

The FAA Reauthorization Act of 1996 (Public Law 104--264, dated October
9, 1996), expanding the procurement reforms previously authorized by
Public Law 104--50, provided the FAA Administrator increased autonomy in
carrying out the functions of the office and the authority to enter into
contracts, grants, cooperative agreements, and other transactions with
public and private entities on such terms and conditions as were
considered appropriate.

The FAA developed its Acquisition Management System (AMS) in response to
Public Laws 104--50 and 104--264. On April 1, 1996, the system went into
effect. In June 1997, the FAA published a revised and updated version of
the AMS policy document developed to further clarify and refine basic
concepts of AMS. The current version of all AMS policy, guidance,
templates, and tools may be found in the FAA Acquisition System Toolset
(FAST) located on the World Wide Web at http://fast.faa.gov. FAST is the
official repository for all AMS documentation and is updated monthly
under strict configuration control.

# Evaluation Objectives

In response to the congressional mandate to provide for more timely and
cost-effective acquisition of equipment and materials, the Program
Evaluation Branch set as its overall objective to determine whether the
FAA was on track to achieve more timely and cost-effective acquisitions
of equipment and materials. The evaluation team chose six component
areas and developed evaluation objectives for each area to demonstrate
whether the overall objective was achieved. These critical areas, which
represented key processes, decision making bodies, and systems within
acquisition reform, included:

-   mission analysis,

-   investment analysis,

-   baseline management,

-   the Joint Resources Council (JRC),

-   the Integrated Product Development System (IPDS), and

-   procurement.

Table 1--1 lists the objective for each of the six component areas
evaluated.

# Scope

The second annual report generally covered the period between April 1,
1996, and March 30, 1998. Each of the six segments, however, covered a
particular time frame within this two-year period. It remained difficult
to measure the results of acquisition reform after only two years under
AMS, since no acquisitions moved totally through the full lifecycle
within that time. As a result, the Program Evaluation Branch focused its
efforts on key lifecycle processes, systems, and decision making bodies
within the FAA. The first year evaluation report observed the progress
of AMS implementation, the impact of AMS on the FAA, and the
effectiveness of some parts of the acquisition process. This second year
report assessed whether the FAA was on track to achieve its goal of more
timely and cost-effective acquisitions of equipment and materials, and
the third year evaluation will attempt to report on whether the agency
has reached that goal. Because AMS was so new and little documentation
existed at the time of the first year review, the evaluation team at
that time relied more on perceptions and opinions of affected and
surveyed individuals. During the second year review, the evaluation team
relied more on supporting documentation and evidence.

In addition to the individual samples selected within the six evaluation
areas, the evaluation team reviewed four selected acquisition programs
as they passed through each phase of the lifecycle acquisition process.
These four programs included:

-   Air Traffic Control Beacon Interrogator Replacement (ATCBI--R),

-   National Airspace System Infrastructure Management System (NIMS),

-   Operational and Supportability Implementation System (OASIS), and

-   Standard Terminal Automation Replacement System (STARS).

The team selected these four programs because they represented
high-visibility FAA acquisitions being accomplished through the new AMS
planning, analysis, and execution processes. These programs provided the
evaluation team with a common set of acquisition programs from which to
measure acquisition reform. These programs were in various stages of the
acquisition process when AMS was implemented; reviewing these programs
provided concrete and illustrative examples of how AMS worked.

# Methodology

Second year evaluation activities addressed those AMS processes,
systems, and decision making bodies that were crucial to the success of
acquisition reform. The Program Evaluation Branch relied on a variety of
data sources for its analysis of acquisition reform progress in the FAA,
including the following:

-   reviews of AMS policy, guidance, and procedure documents;

-   interviews and surveys of FAA personnel;

-   reviews of meeting minutes and Records of Decision from the JRC, the
    Systems Engineering/Operational Analysis Team (SEOAT), and IPDS
    teams;

-   analyses of outputs from FAA databases and information systems;

-   analyses of AMS-related program documents (e.g., Mission Need
    Statements, Investment Analysis Reports, Acquisition Program
    Baselines and other baseline documents, Requirements Documents,
    Integrated Program Plans, Acquisition Strategy Papers); and

-   reviews of IPDS organizational charters and Integrated Product
    Team/Product Team Plans.

# Overall Evaluation Results in Brief

AMS has been in place for two years, and the FAA has made significant
progress toward implementing procedures designed to achieve cost and
schedule goals. Continuous improvements from the first year evaluation
report were identified in every major area. Following are some of the
achievements and highlighted improvements implemented under AMS:

-   FAA employees, managers, and executives were serious about
    acquisition reform and AMS and made diligent efforts to follow the
    AMS processes.

-   Cross-functional, multi-disciplinary teams did exist and worked
    together.

-   Major procurements were more timely.

-   Most FAA employees endorsed and accepted the IPDS structure, which
    established a framework in which to resolve issues and brought teams
    and people together to solve problems sooner.

-   Content of JRC Records of Decision improved significantly. During
    the first year of AMS, the Records of Decision were one-page
    documents and included only the JRC decisions with little reference
    to what led to those decisions. During the second year of AMS, the
    Records of Decision included JRC decisions, rationale, and
    discussion. Improved Records of Decision resulted in FAA having a
    better historical record of what occurred at JRC meetings and better
    documentation of the rationale for decisions.

-   The agency held numerous acquisition reform and AMS training classes
    following the establishment of AMS. FAA employees and contract
    assistance personnel attended training courses in various AMS
    processes and concepts. Table 1--2 includes some of the courses
    offered and the number of persons in attendance for each respective
    course.

-   Many FAA Investment Analysis Teams formed according to AMS policy.

-   The percentage of FAA Facilities and Equipment funded programs
    having JRC-approved baseline documentation increased from 13 percent
    reported in the first year of AMS to 54 percent in the second year
    of AMS.

-   The IPDS structure was implemented, acquisitions were included
    within this structure, and almost half of the Integrated Product
    Teams/Product Teams had a formal plan for guidance.

-   After an initial decline immediately following the implementation of
    AMS, the FAA made progress toward achieving socio-economic goals.

-   The FAA increased the percent of major procurements awarded through
    competition under AMS.

In any industry or agency, developing and implementing a completely new
process or system could reasonably be expected to take several years
before the full benefits could be realized. Based on the average
duration for process innovation cited in the Program Evaluation Branch's
Interim Acquisition Reform Report, dated March 1997, industry
improvement efforts averaged 4.2 years and government organization's
efforts averaged 5.1 years. The range of durations spanned from 3.5 to 7
years before full benefits were realized. This is the case with AMS.
After the second year, the FAA was moving in the right direction, but it
was still too early to validate the success of acquisition reform. The
agency has achieved more timely acquisitions, but further improvements
are necessary before acquisition reform can meet the agency's ambitious
goals of reducing the time to field quality products and services by 50
percent and the cost by 20 percent. The agency was extremely optimistic
in establishing the timeline for meeting these goals at three years.
Because of the ambitious goals and the optimistic time frame, it is
essential that improvements identified be fully implemented quickly so
that the effects can, at the very least, be measured within the third
year to determine if the FAA's goals have been realized.

Although there were continuous improvements during the second year of
AMS, the evaluation team found additional improvements were necessary.
Specifically, the team found that improvements were needed:

-   in the mission analysis process, including improvements in
    identifying, validating, and prioritizing mission needs; receiving
    full support from all lines of business; and coordinating between
    the Mission Analysis Steering Group and the National Airspace System
    (NAS) Architecture core team;

-   in the investment analysis process in documenting management
    intervention in solution identification;

-   to AMS in addressing Systems Engineering/Operational Analysis Team
    activities dealing with affordability assessments;

-   in baseline management, including establishing complete baselines
    and compiling and tracking baseline information to assist the agency
    in measuring and monitoring a program's progress;

-   in the JRC activities to ensure Operations and Maintenance funds
    were fully considered in lifecycle funding decisions and to ensure
    action items were tracked and monitored;

-   to the FAA Joint Resources Council Guidance to develop a realistic
    time frame for disseminating Records of Decisions;

-   in the IPDS structure to ensure IPDS was fully implemented, operated
    free of unnecessary constraints, and was effective in supporting
    successful acquisitions;

-   in the procurement segment to ensure the accuracy and completeness
    of data used to measure results; and

-   to the FAA's overall planning process.

# Follow-Up from the First Year Evaluation

The first year evaluation report issued in May 1997 contained nine
recommendations to the FAA. These recommendations and their completion
status are listed in Table 1--3. Various FAA officials provided
recommendation status information to the evaluation team; the team did
not confirm this status information independently. When recommendations
addressed one of the six areas evaluated for this second year report,
the team looked into them further and made additional recommendations
where appropriate.

# Chapter 1Mission Analysis {#chapter-1mission-analysis .Chapter}

Mission analysis within the FAA was planned as the continuous analytical
activity to evaluate the capacity of agency assets to satisfy existing
and emerging demands for services. It was designed to focus on the
National Airspace System and to address all other agency critical needs
and opportunities. Mission analysis was intended to enable the agency to
determine and prioritize its most critical capability shortfalls and its
best technological opportunities for improving the National Airspace
System overall safety, security, capacity, efficiency, and effectiveness
in providing services to its customers. As such, mission analysis was
expected to be conducted within the framework of, and to contribute
strongly to, the National Airspace System (NAS) Architecture, which was
the long-range plan defining the needs of the National Airspace System
and encompassing the long-range strategic goals of the agency.

# Evaluation Objective

Determine whether the FAA's mission analysis process identified and
prioritized shortfalls and opportunities for meeting its mission.

# Scope and Methodology

In order to answer the evaluation objective, the evaluation team
performed an assessment of quality and compliance with AMS for two of
the six Mission Need Statements approved under AMS during the period
from April 1, 1996, to December 31, 1997, \[En Route/Oceanic Domain and
Workforce Information Next Generation System (WINGS)\] and for four
Mission Need Statements approved prior to AMS for the four selected
acquisition programs \[Air Traffic Control Beacon Interrogator
Replacement (ATCBI--R), NAS Infrastructure Management System (NIMS),
Operational and Supportability Implementation System (OASIS), and
Standard Terminal Automation Replacement System (STARS)\]. The team also
assessed how the agency's lines of business conducted mission analysis,
reviewed the relationship between the NAS Architecture and the mission
analysis process, reviewed the operations of the Mission Analysis
Steering Group (MASG), and reviewed the draft MASG charter.

The evaluation team gathered data from the following sources:

-   AMS policy,

-   Mission Need Statements,

-   National Airspace System Mission Need Statement Database,

-   Mission Analysis Process Guidelines,

-   interviews with individuals actively involved with the NAS
    Architecture, and

-   interviews and surveys of the MASG representatives from each of the
    seven lines of business (Air Traffic Services, Airports, Regulation
    and Certification, Civil Aviation Security, Administration,
    Commercial Space Transportation, and Research and Acquisitions).

# Mission Analysis Results in Brief

The agency identified critical needs and moved forward under AMS to
address these critical needs. We found the agency conducted mission
analysis on the six programs reviewed in our sample. We also found the
agency had documented 329 mission needs in Mission Need Statements.
However, the agency did not follow the AMS process in validating and
prioritizing *all* critical needs that were identified and structured
under mission analysis. Additionally, in some cases, a structured
mission analysis was not conducted and a Mission Need Statement was not
prepared. \[See finding 1 for details.\]

We found two Associate Administrators for FAA's lines of business most
heavily involved in the acquisition process supported the mission
analysis process. In addition, many MASG members, who were
representatives from the various lines of business, supported the
mission analysis process. However, we found some lines of business did
not fully implement the mission analysis process and some MASG members
were not fully involved. In addition, improvements were needed in
coordination between the NAS Architecture core team and the MASG. \[See
finding 2 for details.\]

We also found management control improvements were needed to ensure
Mission Need Statements were revalidated as necessary and to assure all
historical records related to mission needs were collected in a central
repository. The FAA needed to establish internal controls to ensure
Mission Need Statements would be revalidated when necessary. Without
this revalidation, the agency did not know the status of all mission
needs. The agency did create an official repository for all mission
analysis documents approved by the Joint Resources Council (JRC), but
did not establish controls to ensure all unofficial supporting
documentation related to mission needs were also maintained as
historical records. \[See finding 3 for details.\]

## Finding 1: Improvements were needed in identifying, validating, and prioritizing mission needs.

The agency did identify critical needs and did move forward under AMS to
address these critical needs. We found the agency conducted mission
analysis on the six programs reviewed in our sample. We also found the
agency had documented 329 mission needs in Mission Need Statements.
However, the agency did not follow the AMS process in validating and
prioritizing *all* critical needs that were identified and structured
under mission analysis. Additionally, in some cases, a structured
mission analysis was not conducted and a Mission Need Statement was not
prepared.

AMS Sections 2.3 through 2.3.4:

-   required each line of business to perform mission analysis for all
    critical agency needs and opportunities within its business area and
    to sponsor a Mission Need Statement for those needs and
    opportunities which emerged from the structured mission analysis
    process;

-   tasked the MASG to formulate action in response to mission needs
    arising outside the lines of business and to recommend a priority
    ranking for all Mission Need Statements to the JRC; and

-   designated the JRC as the responsible authority to approve Mission
    Need Statements and assign a priority rank to each approved Mission
    Need Statement relative to all approved mission needs.

We found the agency conducted mission analysis on the six programs
reviewed in our sample. In addition, the agency validated and documented
critical needs in a Mission Need Statement for some emerging critical
programs, such as the Host and Oceanic Computer System Replacement
(HOCSR). However, we also found a structured mission analysis was not
conducted and a Mission Need Statement was not prepared, validated, or
prioritized for other critical needs and opportunities. For example, the
Year 2000 and Flight 2000 initiatives were critical needs for which the
agency planned, but which were not validated or documented in the form
of a Mission Need Statement. Year 2000 was a recognized need, and the
agency moved forward in addressing this need, but it was not documented
in a Mission Need Statement. Flight 2000 was a Research, Engineering and
Development funded program that did not have a Mission Need Statement.
Under AMS, when a Mission Need Statement was not prepared, it could have
resulted in a delay in the entire acquisition process or programs may
have been initiated without any analysis of a need for the program.

Critical needs were not always structured under the AMS mission analysis
process because some of these needs had been previously addressed in
programs that were already approved prior to AMS. In addition, because
agency needs, both critical and noncritical, were many and varied, the
lines of business did not always recognize the necessity to identify and
develop Mission Need Statements for *all* critical needs and
opportunities.

While we found the agency had developed and documented mission needs in
329 Mission Need Statements in the National Airspace System Mission Need
Statement Data Base, none of these Mission Need Statements had a
recommended priority ranking by the MASG, and none had been assigned a
prioritization by the JRC. This occurred because the MASG had not
developed guidance on how to prioritize new and existing mission needs.

When critical needs were identified and validated, the agency was better
able to plan for the most cost-effective use of resources and to provide
support for future budget and resource requests. If a critical need was
not recognized at the earliest stage, whether or not it was fully
articulated, the agency could have lost an opportunity to address and
plan for the need before it became a crisis. Prioritizing agency needs
would also improve the FAA's ability to plan effectively.

### Recommendations

We recommend:

**1A** The FAA Administrator ensure that *all* *critical* needs and
opportunities emerge from structured mission analysis, are developed
into Mission Need Statements with a recommended prioritization, are
reviewed for validation, and, if approved, are assigned a prioritization
relative to all other approved Mission Need Statements, in accordance
with AMS.

**1B** The chairperson of the Mission Analysis Steering Group prepare
guidance on how to prioritize new and existing mission needs for Joint
Resources Council approval, and ensure the approved guidance is
implemented.

## Finding 2: Improvements were needed in support of mission analysis and in coordination between the Mission Analysis Steering Group and the National Airspace System Architecture core team.

We found the Associate Administrators for the FAA lines of business most
heavily involved in the acquisition process supported the mission
analysis process. In addition, many MASG members, who were
representatives from the various lines of business, supported the
mission analysis process. However, we found some lines of business did
not fully implement the mission analysis process and some MASG members
were not fully involved. In addition, improvements were needed in
coordination between the NAS Architecture core team and the MASG.

AMS Section 2.3.3 and Appendix A required that each line of business
provide a representative to the MASG and perform mission analysis using
staffs of qualified analysts. The Mission Analysis Process Guideline,
Version 1.0, Section 2.3 recommended each FAA line of business establish
a mission analysis team to conduct mission analysis and to develop
Mission Need Statements.

We found the Associate Administrators for the two lines of business
heavily involved in the acquisition process, Air Traffic Services and
Research and Acquisitions, complied with AMS and supported the mission
analysis process. The other five lines of business did not have the
direct need to become as active in mission analysis and did not have the
resources to support mission analysis to the level described by AMS. We
found the Associate Administrators for five of the seven lines of
business had not established mission analysis teams. These lines of
business were also unable to provide their staffs with sufficient
training in the mission analysis area. Recognizing this difficulty, the
MASG and Air Traffic Services indicated their willingness to provide the
needed support in the area of mission analysis to these other lines of
business.

While lines of business did provide representatives to the MASG, as many
as half of the members were not fully involved. We found, on average,
only four of the nine members attended the 16 MASG meetings held during
the evaluation period. This low level of attendance indicated mission
analysis was given a low priority by the MASG members and by their
respective lines of business.

In addition, improvement was needed in coordination between the NAS
Architecture core team and the MASG. AMS Section 2.3.3 required the FAA
to conduct mission analysis within the framework of the NAS
Architecture. We found the NAS Architecture core team did not coordinate
its critical needs and opportunities with the MASG in preparation of the
NAS Architecture Version 3.0. In turn, the MASG did not coordinate
mission analysis with the NAS Architecture core team for critical needs
and opportunities.

The NAS Architecture was established as the FAA's long-range plan to
define the needs of the National Airspace System over the next 15--20
years. It was intended to provide the framework for mission analysis.
Improving coordination between the NAS Architecture core team and the
MASG would reduce the likelihood of duplicated mission analysis effort,
improve capability to identify all needs and eliminate conflicting
overall FAA mission needs, and provide for the more efficient use of
agency resources.

### Recommendations

We recommend:

**2A** The chairperson of the Mission Analysis Steering Group ensure the
Mission Analysis Steering Group representatives (both members and their
alternates) fulfill their responsibilities to the Mission Analysis
Steering Group, to include ensuring the AMS mission analysis process is
followed and that all mission analysis work in the agency, including the
National Airspace System Architecture, is coordinated with the Mission
Analysis Steering Group.

**2B** The Associate Administrator for each line of business ensure
sufficient qualified and trained staff and the appropriate resources are
available to perform mission analysis work *or* delegate this
responsibility and authority to the Mission Analysis Steering Group or
Air Traffic Services mission analysis team as appropriate and agreed
upon.

## Finding 3: Management control improvements were needed for the FAA mission analysis process.

Management control improvements were needed to ensure Mission Need
Statements were revalidated as necessary and to assure all historical
records related to mission needs were collected in a central repository.

Revalidated Mission Need Statements were required under AMS. AMS Section
2.7 and Appendix A required Mission Need Statements to be revalidated by
the Associate Administrators of the lines of business as necessary
during various stages of the lifecycle acquisition management process.
We found the FAA did not establish internal controls to ensure Mission
Need Statements would be revalidated when necessary. As a result, the
agency did not know:

-   if 56 mission needs were still valid and up to date, or

-   if four new Mission Need Statements duplicated needs identified on
    prior Mission Need Statements.

    Maintaining historical records and documentation was only partially
    required under AMS. AMS Appendix B required the maintenance of a
    central repository of all approved acquisition planning and control
    documents within the Evaluation and Configuration Management
    Division (ASD--200). We found the agency did establish an official
    repository for all JRC-approved documents in the Evaluation and
    Configuration Management Division (ASD--200). Although AMS did not
    address other acquisition planning and control documents, we found
    the agency also established an unofficial repository in the National
    Airspace System Concept Development Branch (ASD--130) for these
    documents. Both repositories provided a reference of historical
    acquisition documents.

    We found the lines of business appropriately relied on the
    Evaluation and Configuration Management Division (ASD--200) to
    collect and maintain official records relating to JRC-approved
    Mission Need Statements. The lines of business, therefore, did not
    maintain or provide copies of all supporting documentation for
    acquisition planning and control documents. This occurred because
    the FAA (1) had not formally established a repository for historical
    records, other than for JRC-approved documents, relating to all
    submitted Mission Need Statements, and (2) had not established an
    internal control to ensure the lines of business maintained all
    supporting records and provided these supporting documents to a
    central repository. With improved historical mission need records,
    the FAA would be better able to:

-   identify the population of mission needs and of submitted Mission
    Need Statements in order to track the need through the acquisition
    process;

-   share and use mission need information throughout the FAA, reducing
    duplication of effort; and

-   conduct current and future mission analysis, revalidate mission
    needs, or prioritize mission needs agency-wide with the benefit of
    historical data.

### Recommendations

We recommend the chairperson of the Mission Analysis Steering Group:

**3A** Develop guidance and procedures on how to conduct and document
revalidation of mission needs, and establish a plan with a schedule for
accomplishing revalidations.

**3B** Develop guidance to establish a central repository for historical
records of all supporting documentation for submitted Mission Need
Statements, and to ensure each line of business submits all acquisition
planning and control supporting documentation to this central
repository.

# Chapter 2Investment Analysis {#chapter-2investment-analysis .Chapter}

Investment analysis was designed to generate information for the Joint
Resources Council (JRC) to use at investment decision to determine the
best overall solution for satisfying a mission need. Investment analysis
was planned as a partnership between the sponsoring and acquiring
organizations to ensure critical needs of the user and customer were
satisfied by an affordable solution. Investment analysis was structured
to translate mission needs into top-level performance and supportability
requirements by: (1) conducting a thorough market analysis; (2)
performing alternative solutions analysis; (3) developing an
affordability assessment to determine the affordability of identified
solutions; and (4) quantifying the cost, schedule, performance, and
benefits baselines for the solution. Investment Analysis Teams were
expected to analyze viable alternative solutions to mission needs
thoroughly and equally. The role of the Systems Engineering/Operational
Analysis Team (SEOAT) in investment analysis was to perform
affordability assessments of all candidate solutions to a mission need
and to maintain a relative priority listing of all agency programs based
on standard evaluation criteria approved by the JRC.

# Evaluation Objective

Determine whether the investment analysis process, including SEOAT
activities, complied with AMS and provided adequate support to permit
sound investment decisions.

# Scope and Methodology

The evaluation team selected and reviewed six programs that were in the
investment analysis process between April 1, 1996, and January 9, 1998.
The sample selection did not require the programs to have moved
completely through the investment analysis process during that time,
only that they were somewhere in the investment analysis process between
those dates. The six programs included three of the four selected
acquisition programs: Air Traffic Control Beacon Interrogator
Replacement (ATCBI--R), National Airspace System Infrastructure
Management System (NIMS), and Operational and Supportability
Implementation System (OASIS). The fourth selected acquisition program,
Standard Terminal Automation Replacement System (STARS), was not
included in this review because STARS was already beyond investment
decision when AMS was implemented. In addition to the three selected
acquisition programs, the evaluation team reviewed Workforce Information
Next Generation System (WINGS), Modernization of Procurement Automation
(ACQUIRE), and Host and Oceanic Computer System Replacement (HOCSR).
WINGS was selected because it had gone through the investment analysis
process without leading to an investment decision. ACQUIRE was selected
because it had both a Mission Need Statement approval and investment
decision during the period of our review. HOCSR was selected because it
had a financial impact on other programs evaluated during this review.
The team evaluated the overall investment analysis process, including
team formation, alternative solutions generation and analysis, market
analysis, affordability assessment, and the JRC decision process. The
team also reviewed SEOAT activities supporting the investment analysis
process. The evaluation team gathered data from the following sources:

-   AMS policy;

-   Investment Analysis Guidelines;

-   program documentation (Mission Need Statements, Requirements
    Documents, Affordability Analyses, and Acquisition Program
    Baselines);

-   JRC presentations, briefing materials, and Records of Decision;

-   Investment Analysis Reports; and

-   interviews with FAA personnel involved in investment analysis,
    including:

```{=html}
<!-- -->
```
-   Integrated Product Team members;

-   FAA and contractor investment analysis personnel;

-   representatives from agency financial management, including SEOAT
    members; and,

-   representatives from the Architecture and Systems Engineering
    Division.

# Investment Analysis Results in Brief

The FAA Investment Analysis Teams were formed according to AMS policy in
most cases. However, we found solutions were sometimes selected or
limited by upper level management prior to the completion of the
alternatives analysis process. When this occurred, the reasons and
justification were not documented in the file. As a result, we could not
determine whether there was a problem with the investment analysis
process or whether other concerns drove management's need to intervene.
Without written documentation to explain the rationale for the
interventions, the FAA cannot determine what changes, if any, should be
made to AMS to address the investment analysis process. \[See finding 4
for details.\]

We also found SEOAT affordability assessment requirements under AMS
needed to be revised. SEOAT affordability assessment activities did not
comply with AMS and may not be able to comply with AMS. AMS required the
SEOAT to perform an affordability assessment on all candidate solutions.
We found this task was difficult, if not impossible, to accomplish with
the available resources. \[See finding 5 for details.\]

## Finding 4: Investment Analysis Teams formed according to AMS, but investment analysis actions were not fully documented.

The FAA Investment Analysis Teams were formed according to AMS policy in
five of the six sample programs reviewed. The Investment Analysis Teams
created matrix partnerships and developed and analyzed alternatives in
light of requirements definitions. However, we found two of the
solutions from the six programs reviewed were selected or limited prior
to the completion of the alternatives analysis process, and paperwork
prepared to support the investment analysis process did not necessarily
document what actually occurred.

AMS Section 2.4 described investment analysis as the process to generate
information used by the JRC at investment decision to determine the best
overall solution for satisfying a mission need. AMS Section 2.4 further
described investment analysis as a partnership between the sponsoring
and acquiring organizations to ensure the critical needs of the user and
customer were satisfied by an affordable solution. AMS Section 2.4.2
identified principal activities of investment analysis, including
identifying alternatives, surveying the market and analyzing
alternatives. In analyzing alternatives, AMS Section 2.4.2 tasked the
Investment Analysis Team with determining "the most advantageous and
reasonable solution to a core set of top-level requirements, *not
necessarily all initial requirements.*" \[Emphasis included in AMS
document.\]

We found the sponsor or upper level management identified or limited
solutions prior to the completion of the alternatives analysis process
in two of the six cases reviewed. For example, in the case of ATCBI--R,
the Requirements Document was written by the sponsor to drive a specific
solution, and the sponsor was reluctant to negotiate and identify a core
set of top level requirements. In HOCSR, which was a high priority
project, the solution was identified by upper level management while the
Investment Analysis Team was still evaluating different options.

When upper level management intervened in the investment analysis
process, it limited the scope of the investment analysis but did not
reduce the level of effort required by the Investment Analysis Team.
Because management intervention was not fully documented, it was not
clear whether there was a problem within the investment analysis process
or whether other concerns drove management's need to intervene. Without
written documentation to explain the rationale for the interventions,
FAA cannot determine what changes, if any, should be made to AMS to
address the investment analysis process. In addition, when the
intervention was not fully documented, the full decision body may not
have been aware an intervention occurred. At least one JRC member stated
it would have affected the decision if that person had known an
intervention had occurred.

### Recommendations

We recommend the FAA Acquisition Executive:

**4A** Require the changed direction and rationale for any intervention
in the investment analysis process be documented at the time of the
intervention, and ensure this documentation is included in the
Investment Analysis Report used at the investment decision.

## Finding 5: SEOAT affordability assessment requirements under AMS need to be revised.

SEOAT affordability assessment requirements under AMS need to be
revised. SEOAT affordability assessment activities did not comply with
AMS and may not be able to comply with AMS. The SEOAT was required to
perform an affordability assessment on all candidate solutions. We found
this task was difficult, if not impossible, to accomplish with the
available resources.

AMS Section 2.4 required the SEOAT to perform an affordability
assessment of *all* candidate solutions to a mission need.

SEOAT affordability assessment activities were not able to comply with
AMS. In most cases, the SEOAT performed an affordability assessment on
the recommended candidate solution only. The SEOAT could not complete
affordability assessments on *each* candidate solution because time and
resource constraints did not permit that level of effort. In addition,
there were conflicting instructions for SEOAT activities between AMS and
the Investment Analysis Guidelines. AMS Section 2.4 stated, "The...SEOAT
performs an affordability assessment of all candidate solutions to a
mission need..." and Section 2.4.2 stated, "The SEOAT assesses the
affordability of each candidate against all other programs in the
agency's financial baseline based on their relative priority." However,
Section 2.2.4.1 of the Investment Analysis Guidelines stated, "It is
probable that the SEOAT will only perform the affordability analysis on
the recommended candidate solution unless the \[Investment Analysis
Team\] or the SEOAT requests otherwise." We found the SEOAT followed the
Investment Analysis Guidelines rather than AMS; we determined this was
reasonable.

### Recommendations

We recommend the chairperson of the Acquisition System Advisory Group:

**5A** Revise the AMS to allow the Systems Engineering/Operational
Analysis Team, in conjunction with the Investment Analysis Team,
flexibility to determine, on a case-by-case basis, how many viable
candidate solutions will have an affordability assessment completed.

# Chapter 3Baseline Management {#chapter-3baseline-management .Chapter}

Under AMS, when the Joint Resources Council (JRC) established an
acquisition program at the investment decision, the JRC was also
expected to approve performance and benefit objectives to be achieved
within strict cost and schedule boundaries, as defined in the
Acquisition Program Baseline (APB). Performance, benefit, cost, and
schedule were the four parameters comprising a program's baseline; these
baselines were intended to be the control elements enabling the agency
to plan realistically when committing to full funding for new programs.
Individual program baselines were built from agency planning and
budgeting profiles. AMS required every acquisition to have an APB
approved by the JRC. For new acquisition programs, the APB was to be
established at the investment decision. For existing programs without a
baseline, the responsible manager was required to propose and obtain JRC
approval of an APB.

# Evaluation Objective

Determine whether the FAA established effective baselines and used each
baseline as a primary tool to measure and monitor a program's progress.

# Scope and Methodology

The evaluation team assessed the status and completeness of baselines
for programs as of January 23, 1998. We used the FAA's Capital
Investment Plan and identified 113 programs funded with Facilities and
Equipment (F&E) funds. Of these 113 programs, 19 were still in
investment analysis and were not expected to have an APB. The remaining
94 programs, however, should have had an APB. Our review focused on
these 94 programs. With respect to these programs, the team reviewed the
general quality of baselines, compliance of baselines with AMS, and both
the agency-level and operational-level management of baselines.

The evaluation team gathered data from the following sources:

-   AMS policy;

-   AMS guidance documents;

-   APBs;

-   Parameter Sheets, which represented JRC-approved interim baseline
    documents; and

-   interviews with Integrated Product Team (IPT) and Product Team (PT)
    leads from a sample of programs.

# Baseline Management Results in Brief

The FAA had not defined "programs," "substantial programs," or "major
systems acquisitions" in order to comply with congressional and Office
of Management and Budget reporting requirements, and had not established
JRC-approved baselines for all programs. Those baselines that were
JRC-approved were usually not comprehensive. (For example, baselines did
not always contain complete information on cost, schedule, performance,
and benefits.) In addition, the JRC allowed Parameter Sheets to
substitute for required APBs in some existing programs, and permitted
other programs to avoid developing any JRC-approved baseline document.
Parameter Sheets were used in lieu of APBs because the JRC needed a way
to expedite baselining for existing programs. However, Parameter Sheets
were not fully effective because essential elements, such as Research,
Engineering and Development (R,E&D) funding and Operations and
Maintenance (O&M) funding, were not included in the majority of the
Parameter Sheets reviewed. As a result, the FAA did not always have
formal baseline documentation against which to measure progress, and
decision makers were limited in their abilities to make sound decisions
on continuing investments. \[See finding 6 for details.\]

We also found the FAA collected incomplete or inconsistent baseline data
to measure and monitor program progress. This occurred because the
numerous systems and processes used for tracking program status were not
integrated and the FAA did not ensure a standard set of data was
collected for comparisons. Also, the data in these systems and processes
did not necessarily reflect current JRC-approved program baselines. As a
result, there was no thorough oversight or tracking of some baselines,
and there was no commitment to periodic revalidation of baselines. In
addition, FAA management received limited visibility into program status
after the investment decision. \[See finding 7 for details.\]

## Finding 6: The FAA did not establish complete baselines.

The FAA had not defined "programs," "substantial programs," or "major
systems acquisitions;" and did not establish baselines for all programs.
During this review, the evaluation team discovered the FAA had not
established definitions for, or designations of, "programs,"
"substantial programs," or "major systems acquisitions," which made it
difficult to identify the number of programs in existence. Because all
programs required an APB, a definition of what constituted a "program"
was essential. Since the FAA Administrator was required to consider
terminating "substantial programs" that fail to meet defined goals, the
agency needed to define what a "substantial program" was and set goals
for those programs. In addition, since the agency was required to report
on "major systems acquisitions," the agency needed to define that term
and identify those programs.

Although programs were not defined by the agency, we defined for our
purposes what a program was and identified 94 Capital Investment Plan
programs that should have had APBs. The Capital Investment Plan included
F&E funded programs; we found no JRC-approved baseline documentation for
R,E&D only or O&M only programs. For these 94 Capital Investment Plan
programs, the evaluation team found 51 (54 percent) had some form of
baseline documentation: 5 programs had APBs and 46 had Parameter Sheets.
While the other 43 programs had no JRC-approved baseline documentation
at all, the 54 percent that did have JRC-approved baseline documentation
represented a dramatic improvement over the 13 percent that had
JRC-approved baseline documentation in the first year of AMS.

Of the 51 programs with JRC-approved baseline documentation, we found
those documents did not always include enough information to measure and
monitor the programs' performance. The evaluation team found that
baseline data included in APBs was significantly more comprehensive than
Parameter Sheets, but neither APBs nor Parameter Sheets were always in
compliance with the AMS. For example, 25 of 29 (86 percent) sampled
baseline documents did not contain all the cost data, and 27 of 29 (93
percent) did not contain enough scheduling information to measure or
monitor program performance.

Public Law 104--264 required the FAA Administrator to consider
terminating "substantial programs" that failed to meet defined goals.
Office of Management and Budget Circulars A--109 and A--11 required
designation and reporting on "major systems acquisitions."

AMS Section 2.11.3 required every acquisition to have an APB approved by
the JRC. According to AMS, an individual APB defined a program's cost
and schedule boundaries, and performance and benefits objectives. These
four parameters comprised a program's baseline. For new acquisition
programs, AMS Section 2.11.3 required a baseline to be established at
the investment decision and required all programs to have an APB. For
existing programs or Capital Investment Plan lines without a baseline,
AMS required the responsible manager to propose and obtain JRC approval
of a baseline.

On June 13, 1997, the JRC approved a plan that allowed exceptions to the
AMS requirement for all programs to have an APB. Specifically, the JRC
decided that programs with low F&E funding totals, programs that were
nearly completed, and programs representing certain level-of-effort
criteria (e.g., Air Route Traffic Control Center Modifications and the
Aeronautical Center Lease) did not require JRC-approved baselines. With
this decision, the JRC intended to reduce to a manageable level the
number of programs requiring documented and approved baselines.

To expedite baselining for existing programs, the JRC also decided not
to enforce the AMS requirement for an APB and approved the Parameter
Sheet as an alternate baseline document. The Parameter Sheet was
considered an interim baseline document, and no written guidance or
standards were developed for its preparation.

To assess the effectiveness of the baselines, the evaluation team
selected a sample of 29 programs from the 94 Capital Investment Plan
programs. Of the 29 programs selected for the sample, 5 had a
JRC-approved APB, 19 had a JRC-approved Parameter Sheet, and 5 had no
JRC-approved baseline document. Using this sample, the team evaluated
the quality of the baseline data for the four baseline parameters (cost,
schedule, performance, and benefits), to determine if there was enough
information to measure the program's progress or to allow decision
makers to make sound decisions on continuing
investments.![](media/image1.wmf){width="3.6368055555555556in"
height="1.8694444444444445in"}

The team found the evaluated APBs were significantly more complete than
the evaluated Parameter Sheets. Performance baselines were complete in
APBs, but were incomplete in most Parameter Sheets. Benefits baselines
were complete in most APBs, but were incomplete in most Parameter
Sheets. Schedule baselines were incomplete for most APBs, but were
incomplete for all Parameter Sheets. Cost baselines for all funding
appeared complete for APBs, but were incomplete for all Parameter
Sheets. Specifically, the evaluation team found R,E&D funding was
frequently not included in baseline documents because it was regarded as
a "sunk cost" by the time the investment decision was made and the APB
was approved. In addition, O&M funding was not always included on the
baseline documents because (1) although O&M funding was provided by the
sponsor for each program baseline, it was unclear which organization, if
any, was responsible for all planning and programming for O&M funds; (2)
little, if any, current or historical O&M funding data existed, and what
was available tended to be unreliable; and (3) the FAA did not have a
cost accounting system capable of identifying real O&M costs for
existing systems or projected O&M costs for future new systems. As a
result, there was very little credibility associated with the O&M cost
estimates contained in APBs and Parameter Sheets. These poor or
inaccurate cost estimates contributed to the incompleteness of the
baseline documentation and limited the FAA's ability to make sound
investment decisions based on the economic merits of a proposal. (See
Table 3--1 for the evaluation team's assessment of baseline
documentation completeness for the sampled documents.)

While APBs were prepared with input from IPT/PT members, Parameter
Sheets were prepared by Research and Acquisitions line of business
personnel that did not always have sufficient baseline data. In
addition, while APBs required signatures from the FAA Acquisition
Executive, the IPT/PT, and the sponsor, Parameter Sheets required no
approving signatures signifying concurrence and agreement by FAA
management, the IPT/PT, or the sponsor.

Even though APBs were more complete than Parameter Sheets, neither type
of document was always sufficient to measure a program's progress or to
allow decision makers to make sound decisions on continuing investments.
Baseline documents did not always provide an adequate depiction of
lifecycle costs or lifecycle linkage. Although F&E estimates were
usually complete, O&M funding requirements were poorly documented and
often did not relate chronologically to the development cycle. In
addition, R,E&D costs were not included in any Parameter Sheet. No
baseline documentation existed for programs requiring only R,E&D funds,
such as the Aging Aircraft program, or for programs requiring only O&M
funds, such as the Leased Interfacility National Airspace System
Communication System.

Approved baseline documentation was intended to provide a mutual
agreement between FAA management, represented by the JRC, and the
provider, which was typically the IPT. Because the JRC permitted
specific programs to avoid developing an approved baseline, many
programs had no baseline documentation. As a result, the FAA was
restricted from having adequate program control to measure and monitor
program progress. The absence of formal baseline documentation against
which to measure progress restricted decision makers' abilities to make
sound decisions on continued investment. In addition, when Parameter
Sheets were allowed to substitute for APBs, the amount of data was
reduced, which also limited the ability of FAA to make sound decisions
based on those baselines.

### Recommendations {#recommendations-6 .unnumbered}

We recommend:

**6A** The FAA Administrator ensure definitions are developed for
"program," "substantial program," and "major systems acquisitions," and
that all FAA programs are designated in the appropriate category.

**6B** The FAA Acquisition Executive ensure every program, existing or
new, has an Acquisition Program Baseline as required by AMS, or an
alternate baseline document, that includes all applicable cost
(Facilities and Equipment; Research, Engineering and Development; and
Operations and Maintenance), schedule, benefits, and performance
parameters.

**6C** The FAA Administrator require those organizations or individuals
with signature authority for funding, including Facilities and
Equipment; Research, Engineering and Development; and Operations and
Maintenance, provide all the funding data and their signatures to
authorize and commit these funds for each Acquisition Program Baseline.

## Finding 7: Baseline data was not compiled and tracked to assist the agency in measuring and monitoring a program's progress.

The FAA collected and reported numerous baseline data, but did not
collect or track complete or consistent baseline data in order to
measure and monitor a program's progress. The evaluation team found that
although the FAA had numerous systems and processes for collecting and
reporting baselines, data to monitor a program's progress was not useful
to the agency in measuring, monitoring and comparing all programs'
progress.

The AMS policy document did not specifically address baseline status
reporting requirements, but AMS Section 1.13 did require metrics and
indicators to be established at the investment decision for each
acquisition program and actual performance and benefits to be measured
and reported against planned values. Also, AMS Section 2.5.5 required
the reporting of baseline status, including cost, schedule, performance,
and benefits, as part of acquisition reviews.

Public Law 104--264 required the Administrator to report to Congress on
any program that was to be continued if it was more than 50 percent over
the cost goal established for the program, it failed to achieve at least
50 percent of the performance goals established for the program, or it
was more than 50 percent behind schedule as determined in the schedule
boundary established for the program.

The Capital Programming Guide, Supplement to OMB Circular A--11, Part 3,
Section IV: Step IV.3, required the planning and procurement process to
be evaluated 3 to 12 months after a new asset became operational to
determine whether benefits from the new asset had been accurately
predicted.

The agency had numerous systems and processes for collecting, reporting
and tracking program status and for measuring it against the established
baselines. No commonality existed among reporting systems; however, and
there was little ability to interface systems with each other or with
agency-level systems already in place, such as the Master Schedule
System. In addition, reporting system interfaces with other
acquisition-related activities were poorly defined. The agency's
numerous systems and processes contained inconsistent, conflicting, or
incomplete data, and did not reflect all baseline parameters from the
JRC-approved baseline documents. A comparison of a sample of some of
these reports and reviews, and the baseline elements they included, is
shown in Table 3--2.

These various systems and reports were established because policy and
guidance responsibilities were diffused within the FAA, many FAA
organizations levied separate reporting requirements on the IPTs and
programs, and there was no centralized coordination of these information
needs.

With respect to the benefits baselines, there was no thorough oversight
or tracking and no commitment to its periodic revalidation. Tracking
benefits baselines with these multiple systems and processes was
difficult, and IPTs lacked confidence in the results of revalidated
benefits. The FAA did not establish a requirement to revalidate, track,
and report benefits status.

With respect to schedule baselines, the agency had no reliable
monitoring data that correlated to program baseline documents. As a
result, the FAA could not compare programs and could not capture
historical data. Without defined schedule milestones, the FAA also could
not establish consistent schedule baselines to monitor progress
throughout program lifecycles. Without consistent schedule baselines,
the agency could not track the acquisition reform goal of 50 percent
schedule reduction or the congressional reporting requirements.

Multiple and inconsistent reporting requirements resulted in a lack of
credibility for FAA data. In addition, collecting and analyzing data and
preparing these multiple reports created a significant workload for the
IPTs and for FAA management. Because no one report tracked all baseline
data, and FAA management did not direct all program status reporting to
be tracked to the JRC-approved baselines, there was no single
agency-wide approach for comparing actual performance against
JRC-approved baselines. As a result, the FAA had no reliable early
warning of schedule and cost baseline problems, no interrelated impact
assessment, and no forecast of cost growth to provide a reliable program
cost estimate at completion.

### Recommendations

We recommend the FAA Administrator:

**7A** Designate a single focal point for all program baseline
management and reporting activities.

**7B** Direct the development of an adequate tracking and validation
system for benefits, or revise the AMS if it is determined the benefits
baselining requirement should not be retained.

**7C** Design and implement procedures for tracking and reporting
program status against all approved baselines.

# Chapter 4The Joint Resources Council {#chapter-4the-joint-resources-council .Chapter}

The Joint Resources Council (JRC) was the FAA body responsible for
making corporate level acquisition decisions. Membership of the JRC
consisted of:

-   Associate Administrators representing all the FAA lines of business
    (Air Traffic Services, Airports, Regulation and Certification, Civil
    Aviation Security, Administration, Commercial Space Transportation,
    and Research and Acquisitions);

-   Assistant Administrators for System Safety and for Policy, Planning
    and International Aviation;

-   FAA Acquisition Executive;

-   Chief Financial Officer; and

-   Legal Counsel.

In addition to these JRC members, JRC meetings were also attended by key
participants responsible for briefing the JRC members on a particular
issue relevant to that JRC meeting. These key participants often
included Integrated Product Team leads, Investment Analysis Team leads,
Systems Engineering/Operational Analysis Team representatives, or
others.

The deliberations of the JRC were focused on five investment-related
decisions: (1) mission need decisions, which determined what
capabilities the agency would pursue; (2) investment decisions, which
determined what solutions the agency would approve and fund to fulfill
mission needs; (3) Acquisition Program Baseline change decisions, which
altered the cost, schedule, performance and/or benefits baselines
established at the investment decision; (4) Research, Engineering and
Development and Facilities and Equipment annual budget submission
decisions; and (5) National Airspace System (NAS) Architecture baseline
decisions. The JRC also participated in development of the agency's
annual operations budget submission.

# Evaluation Objective

Determine whether the JRC made fully supported decisions in a timely
manner and documented its decisions, rationale, and assigned action
items.

# Scope and Methodology

The evaluation team measured the effectiveness of JRC processes; whether
and to what extent the processes had been implemented; and what issues,
potential issues, and barriers existed to success. The evaluation
covered all JRC meetings and events that occurred between April 1, 1996,
and January 31, 1998. Of the 55 JRC meetings held during this time
period, the evaluation team reviewed documentation and results from a
sample of 24 meetings (44 percent). The sample included one JRC
baselining meeting; a JRC budget meeting; all JRC meetings related to
the four selected acquisition programs \[Air Traffic Control Beacon
Interrogator Replacement (ATCBI--R), NAS Infrastructure Management
System (NIMS), Operational and Supportability Implementation System
(OASIS), and Standard Terminal Automation Replacement System (STARS)\];
as well as other JRC meetings. The evaluation team gathered data from
the following sources:

-   AMS policy;

-   FAA Joint Resources Council Guidance;

-   JRC briefing packages, scheduling data, and Records of Decision;

-   interviews with JRC members, Integrated Product Team leads, and
    other JRC key participants; and

-   action items assigned by the JRC.

# Joint Resources Council Results in Brief

We found the JRC did not make fully supported decisions when it
committed the FAA to full lifecycle funding without consistently
considering Operations and Maintenance (O&M) funds. Although the JRC had
the responsibility to make decisions committing the FAA to full
lifecycle funding, the JRC did not have the data needed to commit the
FAA's O&M funding. The FAA did not have a cost accounting system in
place that enabled identification of realistic O&M costs. The agency was
in the process of developing a cost accounting system, which is expected
to be in place by October 1, 1998. Without an operational cost
accounting system or an effective financial baseline for the required
O&M funds, the FAA could not determine if a program was affordable over
its full lifecycle. There was also confusion over the role and authority
of the Resource Management Council. In addition, while we found JRC
meeting attendance was generally good, meetings were frequently
rescheduled, which delayed JRC decisions. \[See finding 8 for details.\]

We found the JRC fully documented its decisions and rationale for
decisions during the second year of AMS. During the second year of AMS,
JRC Records of Decision were expanded from a one-page document with
little information to a full explanation of JRC decisions, including
rationale, discussion, and participant quotes. With improved JRC Records
of Decision, the FAA had a better historical record of what occurred at
JRC meetings and better documentation of the rationale for decisions
made. Timeliness in disseminating information and tracking action items,
however, needed improvement. \[See finding 9 for details.\]

## Finding 8: The Joint Resources Council did not make fully supported and timely decisions.

We found the JRC did not make fully supported decisions when it
committed the FAA to full lifecycle funding without considering O&M
funds. We also found that, while the JRC meeting attendance was
generally good, meetings were frequently rescheduled, delaying some
decisions.

### Lifecycle Funding

AMS Appendix A gave the JRC responsibility to commit the FAA to full
lifecycle funding of programs. Lifecycle funding included all
acquisition, development, and operational costs. We found the JRC did
not fully consider operational costs, which were included within the
FAA's O&M funds, when the JRC made decisions. The JRC did not fully
consider operational costs for two reasons:

-   First, while the JRC had the responsibility to make decisions
    committing the FAA to full lifecycle funding, it did not have the
    data needed to commit the FAA's O&M funding. In addition, there was
    confusion over the role and authority of the Resource Management
    Council, which made decisions on O&M funds.

-   Second, the FAA did not have a cost accounting system and had not
    developed and maintained an operations financial baseline to
    determine current and future operational costs. The basic framework
    that existed for F&E funding did not exist for O&M funding. This
    problem was unrelated to AMS and could not be corrected entirely by
    AMS processes. The FAA was developing a cost accounting system,
    which is expected to be in place by October 1, 1998.

When the JRC made investment decisions, the associated required O&M
funds commitment did not have to be honored by the Resource Management
Council. This split in funding decision making negatively affected
lifecycle planning, data quality, and decision making in the investment
analysis, baseline management, and Integrated Product Development System
processes. Without an operational cost accounting system or an effective
financial baseline for the required O&M funds, the FAA could not
determine if a program was affordable over its full lifecycle.

### JRC Meeting Attendance

The JRC meeting attendance by members was generally good. However, we
found meetings were frequently rescheduled, delaying some decisions.

The FAA Joint Resources Council Guidance required that:

-   JRC meetings be scheduled around the calendars of the FAA
    Acquisition Executive, the sponsoring Associate Administrator and
    the Associate Administrator for Air Traffic Services;

-   the Chief Financial Officer be present when a major program was
    involved; and

-   required documentation be provided to JRC members at least three
    full working days in advance of the JRC meetings.

JRC meetings were rescheduled when a key JRC member requested the
meeting be rescheduled, on a case-by-case basis if key participants were
not ready for the meeting or had a conflicting obligation, or if the
required documentation was not provided three days prior to the meeting.

When JRC meetings were rescheduled, decisions that would have been made
at those meetings were delayed. This had the potential to impact other
phases of the acquisition process, program schedules, and contractual
activities. It also caused staff time to be lost.

***Recommendations***

We recommend:

**8A** The FAA Administrator combine the responsibility and authority
for full lifecycle funding under one Associate/Assistant Administrator
level body and clearly define its role, responsibilities, and authority.

**8B** The FAA Acquisition Executive ensure each Joint Resources Council
member designates a single alternate and empowers this alternate to act
in his/her place, and each key Joint Resources Council meeting
participant designates a single alternate and empowers this alternate to
act in his/her place in order to reduce rescheduling caused by calendar
conflicts and to maintain consistency at meetings. These alternates
should possess sufficient knowledge of the duties and responsibilities
of the Joint Resources Council member or key participant and should have
the appropriate corporate level perspective to serve in this position.

## Finding 9: The Joint Resources Council fully documented its decisions and rationale for decisions; however, timeliness in disseminating information and tracking action items needed improvement.

The JRC fully documented decisions and rationale for decisions during
the second year of AMS. Timeliness in disseminating information and
tracking action items, however, needed improvement.

The FAA Joint Resources Council Guidance designated the Evaluation and
Configuration Management Division (ASD--200) as the JRC's executive
secretariat and required it to prepare and issue the JRC Records of
Decision. These JRC Records of Decision represented the minutes of the
JRC meetings and documented JRC decisions and the rationale for the
decisions. These minutes, or JRC Records of Decision, were issued after
JRC meetings. The content of these JRC Records of Decision improved
significantly during the second year under AMS. During the first year of
AMS, the JRC Records of Decision were one-page documents that included
only the JRC decisions with little reference to what led to those
decisions. During the second year of AMS, the JRC Records of Decision
included JRC decisions, rationale, discussion, and participant quotes.
With improved JRC Records of Decision, the FAA had a better historical
record of what occurred at JRC meetings and better documentation of the
rationale for decisions made.

While the quality of the JRC Records of Decision improved significantly,
timeliness in disseminating these documents did not improve. The FAA
Joint Resources Council Guidance required the JRC Records of Decision to
be prepared and issued within five working days. We found it took an
average of 36 working days to disseminate these Records of Decision
during the first year of AMS and 36.9 working days the second year of
AMS. On average for the first two years under AMS, it took 36.4 working
days to disseminate JRC Records of Decision after the meetings. While we
believe 36 working days was too long, 5 working days was unrealistic. We
found the time to disseminate documents was extended because the agency
had no efficient process for the timely preparation, approval, and
dissemination of JRC Records of Decision. The JRC Records of Decision
were reviewed by many levels before being signed and issued in final
form, and the number of days within which respondents were required to
reply was not specified. As a result, turnaround time varied greatly
between and among the different levels of review.

We found there was no formal process in place to track action items
assigned at JRC meetings during the time of our review. The informal
process being used to track action items assigned at JRC meetings was
not adequate, and some JRC meetings adjourned without a clear
understanding among JRC members and participants of the specific action
items assigned, including what was required, who was responsible, and
when the action was to be completed. JRC members and participants were
sometimes confused about assigned action items, responsible
organizations, and due dates. Although action items were recorded in the
JRC Records of Decision, they were not always captured effectively
*during* the meetings to ensure JRC members and participants understood
and agreed to them. Because JRC Records of Decision were the only formal
documentation of JRC decisions, until they were signed and disseminated,
some actions required by the JRC decision were not implemented. As a
result, actions may not have been completed appropriately. We found
one-third (60 of 184) of the action items assigned during JRC meetings
since the implementation of AMS remained open and were overdue. Because
a formal tracking process did not exist, the evaluation team could not
identify the exact area where delays occurred. In addition, without a
tracking process to close out completed action items, the JRC could not
monitor action items assigned. Before we completed our review, we
confirmed that an automated system to track, follow up on, and close
assigned action items was being designed to address some of these
concerns, but not all of them.

### Recommendations

We recommend the FAA Acquisition Executive:

**9A** Revise the FAA Joint Resources Council Guidance requirement to
issue Records of Decision within five days to a realistic period of
time; and, develop and document a process for the timely preparation,
approval, and dissemination of Joint Resources Council Records of
Decision, and ensure that process is followed.

**9B** Establish an effective means to capture assigned action items
*during* Joint Resources Council meetings (including what is required,
who is responsible, and when the action is to be completed) to ensure
members and participants understand and agree to all action items
assigned; and define and implement a process to track, follow up on,
close, and disseminate results of action items *after* Joint Resources
Council meetings.

# Chapter 5The Integrated Product Development System {#chapter-5the-integrated-product-development-system .Chapter}

The Integrated Product Development System (IPDS) was designed as the
implementing arm for the lifecycle acquisition management system. The
philosophy of IPDS was to team functional disciplines systematically and
thereby integrate and apply all relevant processes to produce an
effective, efficient product or service that satisfied customer needs.
The system, built on the concept of "teams leading teams," stressed
cutting across organizational boundaries and emphasized full lifecycle
responsibility from program inception to disposal of products, and
termination of services.

IPDS was initiated prior to acquisition reform and was formally made a
part of the FAA's acquisition reform process during the development of
AMS. There were four levels of teams within IPDS composed of Integrated
Product Leadership Team, Integrated Management Team, Integrated Product
Team (IPT), and Product Team (PT) (i.e., a sub-IPT when there were
multiple products or services requiring a division of the IPT for
manageability). FAA Headquarters implemented an IPDS structure composed
of 1 Integrated Product Leadership Team overseeing 3 Integrated
Management Teams composed of 17 IPTs, many of which were subdivided into
a total of 36 PTs. Figure 5--1 illustrates the IPDS
structure.![](media/image2.wmf){width="5.991666666666666in"
height="4.091666666666667in"}

# Evaluation Objective

Determine whether the IPDS structure for acquisition programs was fully
implemented, operated free of unnecessary constraints, and was effective
in supporting successful acquisitions.

# Scope and Methodology

The evaluation team reviewed the IPDS put into place between April 1,
1996, and January 31, 1998. The evaluation team gathered data from the
following sources:

-   AMS policy;

-   IPDS Team Plan Guidelines;

-   Integrated Product Leadership Team and Integrated Management Team
    Charters;

-   IPT and PT Plans;

-   Integrated Product Leadership Team and Integrated Management Team
    meeting minutes;

-   IPDS survey of agency organizations; and

-   interviews with Integrated Product Leadership Team, Integrated
    Management Team, IPT/PT, and Implementation Working Group members.

# Integrated Product Development System Results in Brief

The agency initiated IPDS prior to acquisition reform and formally
incorporated IPDS into the acquisition reform process during the
development of AMS. Technically, an IPDS structure can consist of
something as informal as the contracting officer and someone requesting
the product. Using this as a basis, we concluded that the IPDS structure
was implemented. Irrespective of that broad definition, we found that
most acquisitions were included within the formal IPDS structure, and
almost half of the IPT/PTs had a formal plan for guidance. There were
some constraints, however, which impacted IPDS in supporting successful
acquisitions. There were also areas where AMS requirements were more
stringent than necessary under the circumstances.

Constraints impacting the ability of the IPDS structure to support
successful acquisitions included: (1) conflicts between the horizontal
(IPDS team lifecycle responsibilities concept) and vertical (line of
business functional responsibilities concept) structures, (2) the lack
of authority of IPT/PT leadership over all team members, and (3) an
inability of IPDS teams to resolve major agency issues. These
constraints delayed decisions, which could have prolonged the
acquisition process and, in turn, could have increased acquisition
costs. \[See finding 10 for details.\]

In addition, we found some areas where AMS requirements or IPDS Team
Plan Guidelines were more stringent than necessary under the
circumstances. For example, we found not all acquisitions were suitable
for inclusion within the formal IPT structure as required by AMS.
Acquisitions such as facilities or services may not need to be under the
IPT structure; however, there were no guidelines to clarify which
acquisitions did not need to be under a formal IPT structure. The IPDS
Team Plan Guidelines required IPT/PTs to have formal plans. However,
some IPT/PTs may not need to be under a formal plan at this time. We
found some IPT/PTs were small and were able to manage acquisitions
without a formal plan. We did find, however, that 25 out of 53 (47
percent) of the IPT/PTs were under formal plans during the second year
under AMS, representing a marked improvement over the first year of AMS.
The goal to have a formal plan for all formal IPDS teams is a good one
and should continue to be implemented.

## Finding 10: Some constraints impacted the support of successful acquisitions within the Integrated Product Development System.

The evaluation team found some constraints impacted the ability of IPDS
to support successful acquisitions. Conflicts between the horizontal and
vertical structures created the primary constraint to the IPDS.
Horizontal structure refers to the IPDS concept of teams with full
lifecycle responsibilities over acquisitions. Vertical structure refers
to the concept of individuals with functional responsibilities to their
FAA lines of business. Further constraints included a lack of IPT/PT
authority over all team members and the inability of IPDS teams to
resolve major agency issues.

AMS Section 1.11 and Appendix A:

-   defined the philosophy of IPDS was to team functional disciplines
    systematically, thereby integrating and applying all relevant
    processes to produce an effective and efficient product or service
    that satisfied customer needs;

-   empowered teams at the lowest level to make binding, team-based
    decisions on their programs;

-   tasked the Integrated Product Leadership Team with oversight of the
    entire IPDS operation and resolution of high level, cross-domain
    issues requiring senior management assistance and support; and

-   required the Joint Resources Council (JRC) to resolve issues not
    resolved within the IPDS structure.

### Horizontal vs. Vertical

Agency implementation of IPDS placed a horizontal organizational
structure over an existing vertical organizational structure (see Figure
5--2) without reconciling conflicts or overlaps. In some cases, this
resulted in team members experiencing conflicts in goals,
responsibilities, and priorities. For example, IPT lifecycle
responsibilities prescribed by AMS were perceived to be in conflict with
functional responsibilities. In addition, Integrated Management Team and
Integrated Product Leadership Team members stated they spent 75 percent
of their time on vertical activities as opposed to horizontal
activities.

Conflicts and overlaps between the horizontal and vertical structures
increased with the perception that IPDS was an initiative of the
Research and Acquisitions line of business. This perception was
reinforced by the fact that the Research and Acquisitions line of
business (1) filled most IPT lead positions, (2) housed most IPTs and
team members physically, and (3) led most acquisition processes.
Conflicts and overlaps, combined with this perception that IPDS was an
initiative of the Research and Acquisitions line of business, made
achievement of team consensus difficult and extended the decision making
process.

### IPT/PT Authority![](media/image3.wmf){width="4.4534722222222225in" height="3.1930555555555555in"}

IPTs/PTs lacked appropriate authority over all of their team members,
further hindering IPDS effectiveness. IPT/PTs included members who were
not equally empowered. Teams depended on their members to make decisions
that would be supported by the vertical organizations they represented,
but there was tension between the teams and the vertical organizations.
When team members were not fully empowered, representatives had to
consult with their vertical organizations on all issues and it took
longer to make decisions. Team leads had no formal input into all team
members' performance appraisals, and team members were formally
accountable only to their vertical organizations. The evaluation team
found IPT/PT leads believed they were held accountable by the agency for
program performance, while team members believed they were not held
accountable for much of anything. Team leads proposed that all team
member performance appraisals include the same critical element related
to a team's overall performance.

### Resolution of Major Issues

Major issues raised to the Integrated Product Leadership Team were not
always resolved to the satisfaction of the IPT/PTs. In one example, team
members experienced conflicts between their teams and their vertical
organizations regarding second level maintenance issues. In this case,
the IPT believed it was more cost-effective to replace equipment than to
repair it. But the lines of business believed it was less costly to
repair or replace part of the equipment than to replace it all. The
Integrated Product Leadership Team may not have been sufficiently
empowered to resolve the second level maintenance issue. Resolution
would have required the full integration of vertical and horizontal
roles and responsibilities, which also would have required strong
leadership commitment from senior FAA management. When major issues such
as this were not resolved, it was difficult for IPTs/PTs to make key
decisions regarding full lifecycle responsibility.

### Recommendations

We recommend:

**10A** The FAA Administrator ensure that Integrated Product
Teams/Product Teams and functional management/organizations jointly
select team members, document agreements clarifying team members' roles
and responsibilities, and evaluate team member performance.

**10B** The chairperson of the Integrated Product Leadership Team ensure
issues raised within the Integrated Product Development System structure
are resolved.

# Chapter 6Procurement {#chapter-6procurement .Chapter}

The goal of the FAA procurement system was to obtain high quality
products, services, and real property in a timely, cost-effective
manner, and at prices that were fair and reasonable. The procurement
system allowed the agency to be innovative and creative so the best
vendor could be selected to implement a solution. The procurement system
was established as an integral part of the acquisition process focusing
primarily on identifying sources, awarding contracts, and administering
contracts. Under AMS, the procurement system was designed to:

-   emphasize competition as the preferred method of contracting,

-   enable selection of the contractor with the best value,

-   provide attainable and reasonable opportunities for small
    businesses,

-   promote open communications with industry,

-   provide an internal protest forum, and

-   promote sound business judgment and flexibility.

# Evaluation Objective

Determine whether procurements under AMS resulted in more cost-effective
procurements and whether there were impacts on socio-economic goals and
on competition.

# Scope and Methodology

The evaluation team compared procurement records and data from all FAA
Regions, Centers, and Headquarters for eight quarters pre-AMS (from
April 1, 1994, through March 31, 1996), and seven quarters under AMS
(from April 1, 1996, through December 31, 1997). At the time of our
review, data was unavailable beyond this time period. The evaluation
team compared award time and cost to the value of award dollars for each
period, measured the percent of contract dollars awarded to businesses
identified in socio-economic program goals against the established
goals, and identified the percentage of competitive and single-source
awards for both periods.

To assess procurement timeliness, the evaluation team reviewed a data
list of contracts from the Contract Information System for the
evaluation time periods. The team eliminated duplicate contract numbers
and added additional contracts from the System for Acquisition
Management. From the consolidated list, the evaluation team identified a
random sample of contracts based on a criteria of 90 percent accuracy
with a confidence level of plus-or-minus 5 percent. The random sample of
86 contracts included 41 contracts from the pre-AMS period and 45
contracts from the AMS period. For pre-AMS actions, the team computed
the average time from the issuing of a Request for Proposals to award of
the contract. For AMS actions, the team computed the average time from
the issuing the first Screening Information Request used for initial
screening of vendors to award of the contract.

The evaluation team gathered data from the following sources:

-   AMS policy;

-   FAA procurement personnel, including:

```{=html}
<!-- -->
```
-   Office of Acquisitions,

-   Small Business Utilization Staff,

-   procurement office managers and procurement administrative officers,

-   contracting officers and contracting officers' representatives, and

-   program managers;

```{=html}
<!-- -->
```
-   Contract Information System;

-   System for Acquisition Management; and

-   Major Procurement Program Goals reports.

# Procurement Results in Brief

Major procurements were more timely under AMS while costs to conduct
procurements changed very little. Based on analysis of the random sample
selected, the evaluation team found contracts were awarded an average of
56.2 days faster after the implementation of AMS. At the same time,
dollars expended to award contract actions dropped from \$8 million to
\$7.6 million. \[See finding 11 for details.\]

After an initial decline immediately following the implementation of
AMS, the FAA made progress toward achieving socio-economic goals.
However, because the data used to report progress in this area was
incomplete or inaccurate, the evaluation team could not accurately
determine the true impact of AMS on socio-economic goals. \[See finding
12 for details.\]

The FAA also increased the percent of major procurements awarded through
competition under AMS. \[See finding 13 for details.\]

## Finding 11: Major procurements were more timely under AMS.

Major procurements were more timely under AMS while costs to conduct
procurements changed very little. Based on analysis of the random sample
selected, the evaluation team determined contracts were awarded an
average of 56.2 days faster after the implementation of AMS while
dollars expended to award contract actions dropped from \$8 million to
\$7.6 million.

AMS Section 1.1 described reducing the time and decreasing the cost of
delivering needed services as two intended purposes of acquisition
reform. The overall goal of FAA acquisition reform was to reduce the
time to acquire and field new systems by 50 percent and to reduce the
costs of acquisitions by 20 percent, both by April 1999. These results
would be measured from the investment decision through first
commissioning of a new system. The portion of procurement timeliness
covered in this section of the AMS evaluation report represents the
procurement segment of the acquisition process only.

The evaluation team defined time to award procurement actions as the
time from solicitation for a proposal to contract award. Based on the
sample data, the average elapsed time from issuing a Request for
Proposals or Screening Information Request to awarding the contract was
119.8 days pre-AMS and 63.6 days under AMS, representing a 47 percent
reduction in the time required for this phase of the acquisition
process.

While major procurements were more timely under AMS, and costs to
conduct procurements changed very little, the evaluation team was still
unable to quantify whether procurements were more cost-effective because
cost-effectiveness includes more than just quicker (more timely)
procurements. Cost-effectiveness also includes costs to conduct the
procurements, the cost of the procurements themselves, and all other
acquisition costs. Because sufficient information did not exist to
analyze costs of acquisitions, we looked only at the costs to conduct
the procurements.

To determine whether the costs to conduct procurements were more
cost-effective after the implementation of AMS, the evaluation team
examined the time to award procurement actions, the total expenditures
for procurement operations, and contract dollars awarded. Total
expenditures for procurement operations were defined in terms of dollars
spent for FAA employees and for contracted assistance personnel used to
conduct procurement operations. The evaluation team found dollars
expended to award contract actions dropped from \$8 million to \$7.6
million while contract dollars awarded dropped from \$551 million to
\$495 million.

Although we do not know the exact causes, since the implementation of
AMS, total dollars expended for procurement operations, total number of
employees conducting procurement operations, and total dollars awarded
for contracts all decreased. However, without a detailed cost accounting
system to (1) segregate procurement operations conducted by procurement
personnel but not related to procurement award activities, such as
contract administration, etc., from the total procurement expense; and
(2) capture full costs of acquisitions, we were unable to make a
conclusion on whether procurements were more cost-effective.

## Finding 12: The FAA appeared to have made progress toward achieving socio-economic goals.

After an initial decline immediately following the implementation of
AMS, the FAA appeared to have made progress toward achieving
socio-economic goals. However, because the data used to report progress
in this area was incomplete or inaccurate, the evaluation team could not
accurately determine the true impact of AMS on socio-economic goals.

Congress mandated that Federal agencies establish goals, representing
the nation's social goals, for awarding contracts to small business
concerns in general, and to small business concerns owned and controlled
by socially and economically disadvantaged individuals specifically.
Although the Department of Transportation and Related Agencies
Appropriation Act of 1996 exempted the FAA from the Federal Acquisition
Regulation and from the Small Business Act, AMS Section 3.6.1.2 required
the FAA to "implement and aggressively strive to provide small
businesses and small businesses owned and controlled by socially and
economically disadvantaged individuals attainable and reasonable
opportunities to participate as prime contractors and subcontractors for
products and services procured by the FAA." AMS Section 3.6.1.3.1
required the FAA to establish measurable annual agency-wide procurement
program goals for these businesses. The FAA established goals between 5
and 25 percent for these categories.

Awards to businesses identified in the socio-economic program goals
decreased in the six-month period immediately following the
implementation of AMS. However, the evaluation team found the FAA made
steady improvement in awards to those businesses after the initial
six-month period. To make this comparison, we used Major Procurement
Program Goals reports maintained by the FAA's Small Business Utilization
Staff. This Staff had the responsibility for consolidating and reporting
the FAA's progress in meeting socio-economic goals. As shown in Table
6--1 below, we compared the first six-month period following the
implementation of AMS to the last six-month period before AMS and
determined that the percentage of awards to all small businesses
identified in the FAA's socio-economic goals declined immediately
following the implementation of AMS. For example, the percentage of
awards to small businesses in general decreased from 27 percent in the
last six-month period prior to AMS to 18 percent for the six-month
period immediately following the implementation of AMS. Similarly,
set-aside awards to disadvantaged businesses certified by the Small
Business Administration under Section 8(a) (referred to as SEDB awards)
decreased from 14 percent for the last six-month period prior to AMS to
10 percent for the first six-month period immediately following the
implementation of AMS, and awards to disadvantaged businesses that were
self-certified or certified by the Small Business Administration but
received awards through a process other than the set-aside process
(referred to as SDB awards) decreased from 23 percent to 2 percent. The
FAA attributed the initial decline in awards to businesses identified in
the socio-economic program goals in part to the confusion surrounding
AMS set-aside programs and to the industry's lack of familiarity with
the new procurement process under AMS.

By contrast, when the evaluation team compared the results from the
first six-month period following the implementation of AMS to the
results from the full seven quarters following AMS, we determined that
the percentages of awards increased for all three categories. As shown
in Table 6--2, the percentage of awards to small businesses increased
from 18 percent in the first six-month period following AMS to 52
percent for the full seven quarters following AMS. Similarly, set-aside
awards to disadvantaged businesses certified by the Small Business
Administration under Section 8(a) (referred to as SEDB awards) increased
from 10 percent for the first six-month period to 12 percent for the
full seven quarters following AMS, and awards to disadvantaged
businesses that were self-certified or certified by the Small Business
Administration but received awards through a process other than the
set-aside process (referred to as SDB awards), increased from 2 percent
to 3 percent.

The evaluation team identified these improvements based on information
from Major Procurement Program Goals reports submitted to the Small
Business Utilization Staff. However, we determined information submitted
for the Major Procurement Program Goals reports was frequently
incomplete or inaccurate. In addition, reporting was inconsistent among
the FAA Regions, Centers, and Headquarters. The evaluation team found
some Major Procurement Program Goals reports were incomplete or
inaccurate. For example, we found awards from credit card purchases were
not reported, commas were used when decimal points were appropriate, and
totals did not always add up correctly. We also found individuals
submitting the report information did not always understand the
definitions or the format of the Major Procurement Program Goals
reports. In addition, FAA Regions, Centers, and Headquarters were not
consistent in capturing and reporting information on the Major
Procurement Program Goals reports. Some sites reported minority awards
from credit card purchases based on a percentage established by that
site, while other sites did not report credit card activity at all. Some
sites reported awards in actual dollars, while some rounded to the
nearest thousand, and others rounded to the nearest million. Some sites
reported subcontracting award results, while others did not. In
addition, not all sites recorded the same data in the same categories on
the Major Procurement Program Goals reports.

In a separate report entitled "Impact of Acquisition Reform on Awards to
Disadvantaged Businesses" issued May 20, 1998, the FAA Program
Evaluation Branch also identified inconsistencies in the data and
weaknesses in the systems used to collect and report the FAA's progress
in meeting socio-economic goals. Specifically, the report stated that
multiple systems were used to collect contract award information, and
information collected to produce the Major Procurement Program Goals
reports was inconsistent throughout the FAA. As a result, the Program
Evaluation Branch limited the scope of its review for that report to one
system and one location. The May 20, 1998, report discussed the FAA's
progress in meeting socio-economic goals for FAA Headquarters only,
while this AMS evaluation report includes a review of FAA's progress
agency-wide. Because there were substantial inconsistencies in
collecting and reporting information among the FAA Regions, Centers, and
Headquarters, neither report was able to quantify the impact of
acquisition reform on socio-economic goals accurately. Although there
was some initial confusion in data collection under AMS, the FAA's data
collection and reporting inaccuracies were not created by AMS; they were
only highlighted by AMS.

As a result of the errors and inconsistencies in the reporting process,
socio-economic program goal data reported to the Office of the Secretary
of Transportation and the Small Business Administration may not have
been accurate. Without accurate data, the FAA cannot know whether or not
it is meeting established socio-economic goals, whether additional
efforts to meet those goals are necessary, or whether any additional
efforts put in place were effective.

### Recommendations

We recommend:

**12A** The Special Assistant, Small Business Utilization Staff, ensure
that FAA Regions, Centers, and Headquarters have accurate, current
guidance for development of Major Procurement Program Goals reports and
comply with that guidance.

> Additional recommendations related to achieving socio-economic goals
> were addressed in Report #1998--01, "Impact of Acquisition Reform on
> Awards to Disadvantaged Businesses," dated May 20, 1998.

## Finding 13: The percentage of contract dollars awarded through competition increased.

The FAA increased the percentage of major procurement contracts awarded
through competition. To determine the level of competition under AMS,
the evaluation team collected and compared data from the Contract
Information System database for the eight quarters pre-AMS and the seven
quarters under AMS. The Contract Information System contained data for
procurements over \$25,000 and identified whether the contracts were
awarded through competition or through a noncompetitive process. After
averaging to allow for the different number of quarters for the pre-AMS
and AMS periods, we found awards through competition increased from 64
percent before AMS to 70 percent under AMS.

# Chapter 7Interrelated and Recurring Issue {#chapter-7interrelated-and-recurring-issue .Chapter}

While performing the six component evaluations, the evaluation team
identified one issue that crossed almost all acquisition processes and
functional areas. Because of the interrelated and recurring nature of
this issue we separated it from the rest of the evaluation chapters.

We found that the agency cut funding to multiple and, in some cases,
high priority programs to meet its funding levels and changing
priorities because it had not properly planned for these issues. This
resulted in delayed implementation of crucial programs and diminished
trust between planners, developers, and decision makers when their
program work was undermined by deviations from baseline expectations and
agreements.

## Finding 14: The FAA's overall planning process needed improvement.

During the past year, there were two major issues that affected funding
levels of numerous agency programs. First, the agency assumed it would
receive a fiscal year 1999 funding level that was almost \$500 million
greater than what was agreed to by the Office of Management and Budget
(OMB). Second, the agency did not anticipate \$167 million of
reprogramming priorities that surfaced in fiscal year 1998 after the
agency received its budget for the year.

The FAA planned and approved program funding profiles within the agency
and submitted its fiscal year 1999 budget request to the OMB for almost
\$500 million higher than the OMB outyear-target-level guidance (i.e.,
OMB guidance on the maximum funding level to request from the Congress).
In conjunction, the FAA prepared the National Airspace System (NAS)
Architecture with the assumption that OMB would submit this higher
request to Congress. The OMB denied the FAA's higher budget request to
Congress. When this was disallowed, many Capital Investment Plan lines
had to be reworked.

In the case of the fiscal year 1998 appropriation from Congress, four
unplanned emerging priorities surfaced. These four priorities were (1)
Year 2000, a computer problem; (2) Host and Oceanic Computer System
Replacement (HOCSR); (3) additional Standard Terminal Automation
Replacement System (STARS) requirements; and (4) additional security
funding. These four unplanned emerging priorities resulted in a \$167
million shortage which forced the FAA to reprogram funding on 35 of 113
programs. These programs already had Capital Investment Plan lines
established. (As discussed in chapter 3, not all of the agency's
programs had JRC-approved baselines. However, program planners,
developers, and decision makers had expectations of a certain level of
program funding.) The reprogramming reductions impacted high priority
programs and could affect the success of the programs and the agency's
acquisition reform efforts. Some examples of these funding cuts for ten
high priority programs are shown in Table 7--1.

Because the agency was not able to plan based on realistic funding
levels and did not anticipate the emerging priority needs, the agency
had to adjust cost baselines and reprogram funding from numerous
programs. These actions resulted in deferring the planned implementation
of crucial programs \[e.g., Next Generation VHF A/G communication System
(NEXCOM) and Long Range Radar (LRR) Improvements\] and diminished trust
between planners, developers, and decision makers when their program
work was undermined by deviations from baseline expectations and
agreements.

The FAA's submission of a budget request higher than the OMB
outyear-target-level was a one time action on the FAA's part and will
not likely reoccur. One of the four unplanned emerging priorities (the
security issue) was outside of the FAA's control; however, the other
three issues were under the FAA's control and required better planning.

### Recommendation

We recommend the FAA Administrator:

**14A** Ensure that the agency performs better planning for its future
funding needs to avoid constant funding reductions to on-going programs.
This would involve improving the agency's (1) mission analysis
capabilities, which must be tied strongly to the agency's approved
National Airspace System Architecture; (2) investment analysis
capabilities; and (3) managing of program cost, schedule, performance,
and benefits baselines.

# Recommendations {#recommendations-11 .Chapter}

The following lists all the recommendations included in this report. The
recommendation numbers correlate to the finding numbers; not all of the
findings had recommendations.

We limited our recommendations to specific problems identified during
the evaluation. Our evaluation was limited to acquisition reform and did
not address all of the FAA's operations. While we believe there are
other recommendations that could help the FAA improve its overall
operations, we limited our recommendations in this report to those
issues we found within acquisition reform. For example, one
recommendation was to combine the responsibility and authority for full
lifecycle funding of programs under one Associate/Assistant
Administrator level body. We did not address Airport Improvement Program
funds in this evaluation, so we did not address including those funds in
the recommendation. The FAA Administrator may decide it is necessary to
expand this recommendation to include *all* funds. We believe the
expansion of this recommendation, along with the expansion of other
recommendations, may be appropriate.

In addition, we were intentionally broad and not specific in our
recommendations. The purpose of our recommendations was to highlight
areas that needed to be addressed by the FAA management. It was not our
intent to dictate specific solutions. We believe it is best to allow the
action officials flexibility in deciding the best approach to solving a
problem, since those officials best know their operations and other
factors that need to be considered. Any solution that corrects the
problems identified in our findings would be acceptable. The ultimate
goal is to ensure that acquisition reform succeeds.

The evaluation team recommends:

**1A** The FAA Administrator ensure that *all* *critical* needs and
opportunities emerge from structured mission analysis, are developed
into Mission Need Statements with a recommended prioritization, are
reviewed for validation, and, if approved, are assigned a prioritization
relative to all other approved Mission Need Statements, in accordance
with AMS.

**1B** The chairperson of the Mission Analysis Steering Group prepare
guidance on how to prioritize new and existing mission needs for Joint
Resources Council approval, and ensure the approved guidance is
implemented.

**2A** The chairperson of the Mission Analysis Steering Group ensure the
Mission Analysis Steering Group representatives (both members and their
alternates) fulfill their responsibilities to the Mission Analysis
Steering Group, to include ensuring the AMS mission analysis process is
followed and that all mission analysis work in the agency, including the
National Airspace System Architecture, is coordinated with the Mission
Analysis Steering Group.

**2B** The Associate Administrator for each line of business ensure
sufficient qualified and trained staff and the appropriate resources are
available to perform mission analysis work *or* delegate this
responsibility and authority to the Mission Analysis Steering Group or
Air Traffic Services mission analysis team as appropriate and agreed
upon.

**3A** The chairperson of the Mission Analysis Steering Group develop
guidance and procedures on how to conduct and document revalidation of
mission needs, and establish a plan with a schedule for accomplishing
revalidations.

**3B** The chairperson of the Mission Analysis Steering Group develop
guidance to establish a central repository for historical records of all
supporting documentation for submitted Mission Need Statements, and to
ensure each line of business submits all acquisition planning and
control supporting documentation to this central repository.

**4A** The FAA Acquisition Executive require the changed direction and
rationale for any intervention in the investment analysis process be
documented at the time of the intervention, and ensure this
documentation is included in the Investment Analysis Report used at the
investment decision.

**5A** The chairperson of the Acquisition System Advisory Group revise
the AMS to allow the Systems Engineering/Operational Analysis Team, in
conjunction with the Investment Analysis Team, flexibility to determine,
on a case-by-case basis, how many viable candidate solutions will have
an affordability assessment completed.

**6A** The FAA Administrator ensure definitions are developed for
"program," "substantial program," and "major systems acquisitions," and
that all FAA programs are designated in the appropriate category.

**6B** The FAA Acquisition Executive ensure every program, existing or
new, has an Acquisition Program Baseline as required by AMS, or an
alternate baseline document, that includes all applicable cost
(Facilities and Equipment; Research, Engineering and Development; and
Operations and Maintenance), schedule, benefits, and performance
parameters.

**6C** The FAA Administrator require those organizations or individuals
with signature authority for funding, including Facilities and
Equipment; Research, Engineering, and Development; and Operations and
Maintenance, provide all the funding data and their signatures to
authorize and commit these funds for each Acquisition Program Baseline.

**7A** The FAA Administrator designate a single focal point for all
program baseline management and reporting activities.

**7B** The FAA Administrator direct the development of an adequate
tracking and validation system for benefits, or revise the AMS if it is
determined the benefits baselining requirement should not be retained.

**7C** The FAA Administrator design and implement procedures for
tracking and reporting program status against all approved baselines.

**8A** The FAA Administrator combine the responsibility and authority
for full lifecycle funding under one Associate/Assistant Administrator
level body and clearly define its role, responsibilities, and authority.

**8B** The FAA Acquisition Executive ensure each Joint Resources Council
member designates a single alternate and empowers this alternate to act
in his/her place, and each key Joint Resources Council meeting
participant designates a single alternate and empowers this alternate to
act in his/her place in order to reduce rescheduling caused by calendar
conflicts and to maintain consistency at meetings. These alternates
should possess sufficient knowledge of the duties and responsibilities
of the Joint Resources Council member or key participant and should have
the appropriate corporate level perspective to serve in this position.

**9A** The FAA Acquisition Executive revise the FAA Joint Resources
Council Guidance requirement to issue Records of Decision within five
days to a realistic period of time; and, develop and document a process
for the timely preparation, approval, and dissemination of Joint
Resources Council Records of Decision, and ensure that process is
followed.

**9B** The FAA Acquisition Executive establish an effective means to
capture assigned action items *during* Joint Resources Council meetings
(including what is required, who is responsible, and when the action is
to be completed) to ensure members and participants understand and agree
to all action items assigned; and define and implement a process to
track, follow up on, close, and disseminate results of action items
*after* Joint Resources Council meetings.

**10A** The FAA Administrator ensure that Integrated Product
Teams/Product Teams and functional management/organizations jointly
select team members, document agreements clarifying team members' roles
and responsibilities, and evaluate team member performance.

**10B** The chairperson of the Integrated Product Leadership Team ensure
issues raised within the Integrated Product Development System structure
are resolved.

**12A** The Special Assistant, Small Business Utilization Staff, ensure
that FAA Regions, Centers, and Headquarters have accurate, current
guidance for development of Major Procurement Program Goals reports and
comply with that guidance.

> Additional recommendations related to achieving socio-economic goals
> were addressed in Report #1998--01, "Impact of Acquisition Reform on
> Awards to Disadvantaged Businesses," dated May 20, 1998.

**14A** The FAA Administrator ensure that the agency performs better
planning for its future funding needs to avoid constant funding
reductions to on-going programs. This would involve improving the
agency's (1) mission analysis capabilities, which must be tied strongly
to the agency's approved National Airspace System Architecture; (2)
investment analysis capabilities; and (3) managing of program cost,
schedule, performance, and benefits baselines.

# Appendix ACriteria {#appendix-acriteria .Chapter}

The following documents and citations formed the criteria against which
the evaluation team measured FAA acquisition reform efforts and
processes. Documents include (in order of precedence) public laws,
Office of Management and Budget circulars, the AMS policy document, and
AMS guidance documents. In this evaluation report, all references to the
AMS policy document are to the June 1997 version.

**Chapter 1Mission Analysis**

**Finding 1**

AMS Section 2.3: "Mission analysis is a strong, forward-looking, and
continuous analytical activity that evaluates the capacity of agency
assets to satisfy existing and emerging demands for services...Mission
analysis enables the agency to determine and prioritize its most
critical capability shortfalls and best technology opportunities...All
Mission Need Statements will emerge from structured mission analysis."

AMS Section 2.3.3: "Each FAA line of business...performs mission
analysis for its business area...The Mission Analysis Steering Group
coordinates mission analysis among the lines of business; recommends to
the Joint Resources Council a priority ranking for each Mission Need
Statement; and formulates action in response to mission needs arising
outside the lines of business (e.g., system safety, environmental,
international)."

AMS Section 2.3.4: "An FAA line of business must sponsor a Mission Need
Statement...The Joint Resources Council assigns a priority rank to each
approved Mission Need Statement relative to all approved mission needs."

**Finding 2**

AMS Section 2.3.3: "Each FAA line of business...performs mission
analysis for its business area using staffs of qualified analysts. These
analysts conduct mission analysis within the broad framework of NAS
Architecture, congressional mandates, and agency strategic planning. The
Mission Analysis Steering Group coordinates mission analysis among the
lines of business; recommends to the Joint Resources Council a priority
ranking for each Mission Need Statement; and formulates action in
response to mission needs arising outside the lines of business (e.g.,
system safety, environmental, international)."

AMS Appendix A: "The Mission Analysis Steering Group is composed of
representatives from each line of business."

AMS Mission Analysis Process Guideline Section 2.3: "Sponsoring lines of
business are normally responsible for:...

-   Establishing a line of business Mission Analysis Team (MAT) and
    supporting staff to conduct mission analysis and to develop Mission
    Need Statements."

**Finding 3**

AMS Section 2.7: "*Mission need must be revalidated (and updated if the
need has changed) by the mission analysis staff of the original
sponsoring line of business*..." \[sic\]

AMS Appendix A: The Line of Business Associate Administrator:

-   "Revalidates Mission Need Statements as required during various
    stages of the lifecycle acquisition management process."

AMS Appendix B: "This appendix contains the purpose, description,
approval authority, distribution, and content for the six mandatory
acquisition planning and control documents in the Lifecycle Acquisition
Management System...Send a copy \[of the approved Mission Need
Statement\] to ASD--200, Program Evaluation Office, which maintains a
central repository of approved acquisition documents for the Joint
Resources Council."

**Chapter 2Investment Analysis**

**Finding 4**

AMS Section 2.4: "Investment analysis generates the information used by
the Joint Resources Council at the investment decision to determine the
best overall solution for satisfying a mission need. It is conducted as
a partnership between the sponsoring and acquiring organizations to
ensure the critical needs of the user and customer are satisfied by an
affordable solution. ... The intent of investment analysis is to define
in functional and performance terms the capability the agency must have
to satisfy mission need, and to determine and baseline the best overall
solution(s) for achieving that capability."

AMS Section 2.4.2: "The principal activities of investment analysis are:

-   **Identify alternatives and survey the market**...

-   **Analyze alternatives**...This process involves multiple iterations
    back through requirements...to determine the most advantageous and
    reasonable solution to a core set of top-level requirements, *not
    necessarily all initial requirements*." \[sic\]

**Finding 5**

AMS Section 2.4: "...The Systems Engineering/Operational Analysis Team
(SEOAT) performs an affordability assessment of all candidate solutions
to a mission need..."

AMS Section 2.4.2: "...The SEOAT assesses the affordability of each
candidate against all other programs in the agency's financial baseline
based on their relative priority..."

Investment Analysis Guidelines Section 2.2.4.1: "It is probable that the
SEOAT will only perform the affordability analysis on the recommended
candidate solution unless the \[Investment Analysis Team\] or the SEOAT
requests otherwise."

**Chapter 3Baseline Management**

**Finding 6**

Public Law 104--264 Subtitle B Section 252: "The Administrator shall
consider terminating...any substantial acquisition program..."

Office of Management and Budget Circular A--11: "Exhibit 300B covers
major acquisitionsthose requiring special management attention because
of their importance to the agency mission; high development, operating,
or maintenance costs; high risk; high return; or their significant role
in the administration of agency programs, finances, property, or other
resources...Agencies should report on all major acquisitions."

Office of Management and Budget Circular A--109 Section 5.h: "Major
system acquisition programs are those programs that (1) are directed at
and critical to fulfilling an agency mission, (2) entail the allocation
of relatively large resources, and (3) warrant special management
attention."

AMS Section 2.11.3: "The Acquisition Management System requires *every*
acquisition to have an Acquisition Program Baseline approved by the
Joint Resources Council. For new acquisition programs, the baseline is
established at the investment decision. For existing programs or
\[Capital Investment Plan\] lines without a baseline, the responsible
manager must propose and obtain Joint Resources Council approval or a
baseline." \[sic\]

"Key" statement in the margin of AMS Section 2.11.3: "All programs must
have an Acquisition Program Baseline."

**Finding 7**

Public Law 104--264 Subtitle B Section 252: "The Administrator of the
Federal Aviation Administration shall terminate any acquisition program
initiated after the date of the enactment of the Air Traffic Management
System Performance Improvement Act of 1996 and funded under the
Facilities and Equipment account that

> \(1\) is more than 50 percent over the cost goal established for the
> program;
>
> \(2\) fails to achieve at least 50 percent of the performance goals
> established for the program; or
>
> \(3\) is more than 50 percent behind schedule as determined in
> accordance with the schedule goal established for the program."

Public Law 104--264 Subtitle B Section 252: "...if the Administrator
determines that termination \[of a program\] would be inconsistent with
the development or operation of the national air transportation system
in a safe and efficient manner...\[ and decides to continue a program
that fails to meet its cost, performance, or schedule goal by more than
50 percent\] ...the Administrator shall transmit a copy of the
determination, together with a statement of the basis for the
determination..." to the Congress.

The Capital Programming Guide of OMB Circular A--11, Section IV: Step
IV.3: "Three to twelve months after a new asset becomes operational, the
planning and procurement process should be evaluated to determine
whether they accurately predicted the benefits to be derived from the
new asset."

AMS Section 1.13: "In compliance with the Government Performance and
Results Act of 1993 (GPRA), the Acquisition Management System requires
performance indicators and metrics to be established at the investment
decision for each acquisition program. Actual performance and benefits
will be measured and reported against planned values."

AMS Section 2.5.5: "Topics for \[acquisition\] reviews should include
baseline status (cost, schedule, performance, benefits)..."

**Chapter 4The Joint Resources Council**

**Finding 8**

AMS Appendix A: The JRC roles and responsibilities include committing
"...the agency to full lifecycle funding of the program."

The FAA Joint Resources Council Guidance, October 1996, (p.6): "If a
major program is involved and the Chief Financial Officer cannot attend,
the JRC meeting will be canceled."

The FAA Joint Resources Council Guidance (p.7): "JRC meetings will be
scheduled around the calendars of the FAE, the Sponsoring Associate
Administrator, and the Associate Administrator for Air Traffic
Services."

The FAA Joint Resources Council Guidance (p.8): Requires the meeting to
be canceled "...if the required material has not been provided at least
three full working days in advance of the meeting."

**Finding 9**

The FAA Joint Resources Council Guidance (p. 5): "The Program Evaluation
Division is the executive secretariat for the JRC...The Division
prepares and issues the final decision record and action items within
five working days of the JRC meeting. The Division also maintains files
which document all JRC decisions."

**Chapter 5The Integrated Product Development System**

**Results in Brief**

IPDS Team Plan Guidelines Section 1: "...PTs will prepare stand-alone
Product Team Plans, consistent with these guidelines, which will serve
as their governing 'rule book'."

IPDS Team Plan Guidelines Section 4: "The IPT...Plan should address the
same topics, in the same order as the Product Team Plans."

**Finding 10**

AMS Section 1.11: "The philosophy of the Integrated Product Development
System (IPDS) is to team functional disciplines systematically, and
thereby integrate and apply all relevant processes to produce an
effective and efficient product or service that satisfies customer
needs. This system is built on the concept of 'teams leading teams,' and
requires a cultural and organizational focus that understands and
accommodates the mechanics and dynamics of team operation. It stresses
cutting across organizational 'stovepipes,' and emphasizes full
lifecycle responsibility from program inception to disposal for products
and termination for services."

AMS Section 1.11: "IPTs or PTs, as appropriate, are the primary
lifecycle acquisition teams. They are empowered at the lowest level to
make binding, team-based decisions on the programs within their purview
in the interests of all stakeholders, users, and customers in
accomplishing their mission..."

AMS Section 1.11: "The Integrated Product Leadership Team, consisting of
director-level managers, oversees the entire IPDS operation and resolves
high-level, cross-domain issues requiring senior management assistance
and support."

AMS Appendix A: The Joint Resources Council

-   "Resolves issues not resolved within the Integrated Product
    Development System structure."

**Chapter 6Procurement**

**Finding 11**

AMS Section 1.1: AMS "...is intended to simplify, integrate, and unify
the elements of lifecycle acquisition management into an efficient and
effective system that increases the quality, reduces the time, and
decreases the cost of delivering needed services to its customers."

**Finding 12**

Public Law 104--50 Section 348: "The following provisions of Federal
acquisition law shall not apply to the new acquisition management system
developed and implemented pursuant to subsection (a):...

\(4\) The Small Business Act (15 U.S.C. 631 et seq.), except that all
reasonable opportunities to be awarded contracts shall be provided to
small business concerns and small business concerns owned and controlled
by socially and economically disadvantaged individuals."

AMS Section 3.6.1.2: "The FAA shall implement and aggressively strive to
provide small businesses and small businesses owned and controlled by
socially and economically disadvantaged individuals attainable and
reasonable opportunities to participate as prime contractors and
subcontractors for the products and services procured by the FAA."

AMS Section 3.6.1.3.1: "Prior to the end of each fiscal year, measurable
annual FAA wide major procurement program goals (including
subcontracting goals) will be established to provide attainable and
reasonable opportunities for small businesses and small businesses owned
and controlled by socially and economically disadvantaged individuals to
participate in contracts awarded by the FAA for the next fiscal year."

# Appendix BList of Acronyms {#appendix-blist-of-acronyms .Chapter}

AMS Acquisition Management System

APB Acquisition Program Baseline

ASD Office of System Architecture and Investment Analysis

ATCBI--R Air Traffic Control Beacon Interrogator Replacement

F&E Facilities and Equipment

FAA Federal Aviation Administration

FAST FAA Acquisition System Toolset

HOCSR Host and Oceanic Computer System Replacement

IPDS Integrated Product Development System

IPT Integrated Product Team

JRC Joint Resources Council

MASG Mission Analysis Steering Group

NAS National Airspace System

NIMS NAS Infrastructure Management System

OASIS Operational and Supportability Implementation System

O&M Operations and Maintenance

OMB Office of Management and Budget

PT Product Team

R,E&D Research, Engineering and Development

SEOAT Systems Engineering/Operational Analysis Team

STARS Standard Terminal Automation Replacement System

WINGS Workforce Information Next Generation System