![](media/image1.emf)

Acknowledgments

The data gathering and analysis that provided the basis for this
document was performed by the members of the Acquisition Management
System Evaluation Team. The members were responsible for planning their
evaluations, executing the plans, and reporting the data.

The team would like to thank all of those interviewed for their time,
their candor, and their suggestions. Special thanks is also extended to
those who reviewed the various drafts; their inputs were greatly
appreciated.

# Executive Summary  {#executive-summary .Chapter}

On November 15, 1995, the President signed the Department of
Transportation and Related Agencies Appropriations Act of 1996, which
directed the Administrator of the Federal Aviation Administration (FAA)
to:

...develop and implement...an Acquisition Management System for the
Federal Aviation Administration that addresses the unique needs of the
agency and, at a minimum, provides for more timely and cost effective
acquisition of equipment and materials.

After 4 months of intensive effort involving nearly 350 people,
Administrator David Hinson established the new Acquisition Management
System (AMS) on April 1, 1996. This internal FAA evaluation of the first
year under AMS finds:

-   Measurable progress in implementing the AMS

-   Reduced procurement times

-   An increase in obligations to small businesses, and a decrease in
    obligations to socially and economically disadvantaged businesses

-   Cost savings to industry

-   Problems with the new dispute resolution process

-   A lack of consistent measurement capability

-   Minimal progress in establishing a full lifecycle perspective.

# Purpose and Methodology

The Associate Administrator for Research and Acquisitions led the reform
effort and directed the ASD program evaluation staff to report annually
on the status of acquisition reform. These annual evaluations will:

-   Provide an objective view of acquisition reform implementation and
    effectiveness

-   Support implementation and improvement of acquisition reform
    activities

```{=html}
<!-- -->
```
-   Determine the impact of acquisition reform on the FAA, its
    customers, and its suppliers.

An overall strategy for the yearly evaluations was established to
provide progressively more detailed information as the AMS matures. The
first year's evaluation activities concentrate on the implementation of
the AMS and the supporting infrastructure. In the second year, the
evaluation will focus on the effectiveness and efficiency of the new and
re-engineered processes. The third year evaluation will look at the
overall results of the acquisition reform against established goals and
expected outcomes.

The primary means of data collection was approximately 200 interviews
conducted between January 15 and April 30, 1997. Other sources included
reports and documentation from FAA, congressional testimony and reports,
independent reviews, and media coverage.

# Background 

Congress has been concerned about the state of the air traffic control
system and the inability of the FAA to install systems on time and
within cost. In response to these problems, Congress passed Public Laws
104-50 and 104-264 and gave the FAA the opportunity and authority to
enact major reforms in both the acquisition and personnel management
areas. This legislation exempted the FAA from many of the existing
federal procurement laws and regulations, including the Federal
Acquisition Regulation, and directed the FAA to develop and implement
new acquisition and personnel systems. The FAA established a number of
design and implementation teams and convened a Blue Ribbon Panel to
provide outside expertise.

FAA's new AMS establishes high-level policy and guidance for all aspects
of the acquisition lifecycle from mission needs determination through
the procurement, operation and retirement of solutions that meet those
needs. Conceptually, the new system replaces extensive rules with a core
acquisition policy document. This policy is augmented by a set of
generic processes, extensive guidance, and acquisition aids from which
practitioners can formulate a tailored acquisition process to meet
specific needs. All of these components will reside in an automated FAA
Acquisition System Toolset (FAST), which is available via the Internet.

# Progress

-   Interviews indicated that across the agency there is widespread
    knowledge of AMS and acquisition reform, its goals, and its
    philosophy.

-   The philosophy and policy established for AMS address the
    fundamental problems that precipitated reform.

-   The approved AMS policy was delivered as a report to Congress in
    April, 1996. A scheduled October revision has been widely
    distributed but not yet signed.

-   The Office of Business Management (ABZ) was established to focus on
    business processes including workforce learning.

-   Approximately half of the generic processes and guidance that
    support the core policy of AMS have been completed, approved, and
    entered into FAST.

-   Detailed guidelines for conducting mission analysis and investment
    analysis have been created and approved.

-   Mission analysis staffs have been established within several of the
    FAA lines of business. The Mission Analysis Steering Group
    coordinates the analyses performed across all lines of business.

-   The Joint Resources Council (JRC) has been established and is
    meeting regularly to make corporate-level decisions and optimize
    resource allocation.

-   Several multi-organization Investment Analysis Teams are working to
    provide a basis for major strategic investment decisions.

-   New procurement processes with automated guidance and tools are
    available to all FAA procurement employees.

-   A comprehensive system of procurement performance measures has been
    identified, and their collection has been initiated.

# Findings

-   The time from announcement to award for contracts over \$100,000 has
    been reduced from a pre-AMS minimum of 90 days to an average of less
    than 60 days.

-   AMS has reduced bid and proposal costs for vendors by as much as 40
    percent.

-   While AMS implementation is widespread and continuing, only 48
    percent of the milestones due to be accomplished by April 1, 1997,
    have been achieved.

-   The two primary AMS goalsachieve 20 percent cost reduction and
    achieve 50 percent schedule reductionhave not been defined, planned
    for, or tracked.

-   There is no evidence of consistent acquisition program measurement
    across the organization. Most programs lack JRC-approved acquisition
    program baselines.

-   The dispute resolution process is not well defined, not permanently
    staffed, and has nearly three times the number of disputes than the
    same period before AMS with one third of these protests unresolved.

-   Although overall awards to small businesses have increased, awards
    to small and economically disadvantaged businesses have dropped 72
    percent.

-   The establishment of the Resource Management Council (RMC) as a
    second separate corporate level decision making body is inconsistent
    with the AMS principle of unified agency-wide planning, programming,
    and budgeting within a long-range strategic framework.

-   Tension is perceived on boundary issues within the Integrated
    Product Development System (IPDS); "The horizontal notion of
    empowered teams bumps up against the vertical notion of functional
    responsibility."

-   Many believe a lack of progress in personnel reform is slowing
    acquisition reform.

-   Interviews showed wide-spread industry acceptance of AMS and
    acquisition reform, its goals, and its philosophy as a better way to
    do business.

# Recommendations 

-   Establish a focused, rigorous plan for the IPTs to reduce the time
    to field new systems by 50 percent and reduce acquisition costs by
    20 percent. Establish an appropriate, effective reporting system to
    monitor progress.

-   Obtain the Administrator\'s approval of the revised AMS core policy.

-   Develop an integrated metrics program and educate the workforce on
    its use and value.

-   Establish and enforce baseline management policy and procedures, and
    expedite the establishment of baselines for legacy programs.

-   Complete the solution implementation guidance for systems/software
    as soon as possible and expedite the remaining processes, guidance,
    and tools.

-   Prioritize programs, functions, and processes and reallocate
    resources accordingly, eliminating low-priorities. Centralize
    processes where efficiency or effectiveness gains are possible.

-   Merge the JRC and RMC into a single Assistant and Associate
    Administrator-level body to optimize investment decisions and
    resource allocation across all FAA appropriations. An expanded SEOAT
    could support the merged council.

-   Fully fund and staff the Office of Dispute Resolution and complete
    the required processes and procedures.

```{=html}
<!-- -->
```
-   Determine the requirements for, and articulate the benefit of, small
    economically disadvantaged business contracts and implement guidance
    accordingly.

Contents

**[Executive Summary]{.smallcaps}**

Purpose and Methodology

Background

Progress

Findings

Recommendations

Chapter 1Evaluation Approach

Objectives

Scope

Methodology

Structure of the Report

Chapter 2Acquisition Reform

Background

Congressional Intent

Problems Driving Acquisition Reform

Acquisition Reform Goals

Relationship of Acquisition Reform to Personnel and Financial Reform

Chapter 3Acquisition Management System Status

Evaluation Results

Progress

Findings

Barriers

Recommendations

Discussion

The Acquisition Management System

Completeness With Respect To The Driving Problems

Implementation Progress Against Milestones

Measuring Implementation Progress Through Documentation

Chapter 4FAA and Industry Experiences

FAA Perceptions and Responses

Progress

Findings

Barriers

Discussion

Flexibility

Culture

Empowerment and Accountability

Metrics and Measures

Industry Perceptions and Responses

Progress

Findings

Barrier

Discussion

Impact of AMS on Small Economically Disadvantaged Businesses

Chapter 5---Lifecycle Acquisition Management

Evaluation Results

Progress

Findings

Barriers

Recommendations

Discussion

Core Policy

Joint Resources Council

Metrics and Performance Measurement

Program Baselines

Integrated Product Development System/Lead the Fleet Programs

Human Resource Concerns

Chapter 6Procurement

Evaluation Results

Progress

Findings

Barriers

Recommendations

Discussion

Procurement Guidance and Processes

Communication

Training

Tracking and Metrics

Process Flexibility

Timeliness

Competition

Credit Cards

Socio-Economic and Other Policies and Programs

Dispute Resolution

Chapter 7Conclusions and Recommendations

Management Control

Planning

Acquisition and Procurement Process

Financial Resources

Conclusions

Recommendations

AppendixList of Acronyms

List of Tables

Table 2-1. Summary of Problems

Table 3-1. Summary of AMS Definitions

Table 3-2. Summary of AMS Completeness

Table 3-3. Strategic Milestone Status and Risks

Table 3-4. Status of AMS Policy Elements

Table 4-1. Workforce Core Questions

Table 4-2. Industry Core Questions

Table 5-1. JRC Decisions

Table 5-2. Summary Baseline Inventory

Table 6-1. Procurement Metrics Objectives, Measures, and Status

Table 6-2 Competitive vs. Single Source Awards

Table 6-3. Summary of Credit Card Data

Table 6-4. Comparison of Small Business Obligations

Table 7-1. Summary of AMS Effectiveness Against Problems

Table 7-2. Evaluation Team Assessment of AMS Effectiveness

List of Figures

Figure 1-1. Three-year Evaluation Strategy 2

Figure 3-1. AMS Strategic Milestone Status 13

Figure 3-2. AMS Tactical Milestone Status 14

Figure 6-1. Average Procurement Time 33

# Chapter 1Evaluation Approach {#chapter-1evaluation-approach .Chapter}

# Objectives 

The Associate Administrator for Research and Acquisitions directed the
program evaluation staff to report annually on the status of acquisition
reform. This series of reports is intended to:

-   Provide an objective view of acquisition reform implementation and
    its effectiveness

-   Support the implementation and improvement of acquisition reform
    activities

```{=html}
<!-- -->
```
-   Determine the impact of acquisition reform on the FAA, its
    customers, and its suppliers.

# Scope 

This first annual report covers the period between April 1, 1996, and
March 30, 1997. It is difficult to measure the results of acquisition
reform over only one year, because very few acquisitions move through
the lifecycle within that time. What can be observed is the progress of
Acquisition Management System (AMS) implementation, the impact on the
organization, and the demonstrated effectiveness of some parts of the
acquisition process.

# Methodology 

An overall strategy for the yearly evaluations was established to
provide progressively more detailed information as the AMS matures
(Figure 1-1). The evaluation activities in the first year concentrate on
the implementation of the AMS and the supporting infrastructure. In the
second year, the evaluation will focus on the effectiveness and
efficiency of the new and re-engineered processes. The final evaluation
will look at the overall results of the acquisition reform in terms of
the established goals and expected outcomes.

Four questions were established for the first report:

1.  What is Acquisition Reform?

2.  How far has the implementation progressed?

3.  How effective have the changes been?

4.  What are the barriers to Acquisition Reform success?

To answer these questions, the evaluation team studied the following
areas:

-   The scope of acquisition reform

-   The completeness of AMS and its consistency with respect to
    identified problems

-   The progress of AMS implementation against scheduled milestones

-   The Lead the Fleet (LTF) program

-   The impact of AMS on procurement practices

-   The development of acquisition program baselines

```{=html}
<!-- -->
```
-   The existing and planned metrics and measures.

Figure 1-1. Three-year Evaluation Strategy

The team also conducted nearly 200 interviews with AMS users in the FAA
and industry to measure understanding of acquisition reform goals, AMS
policy, and new processes as well as identify AMS successes, problems,
and barriers associated with AMS.

The report presents four kinds of information resulting from the AMS
evaluation activities. These are:

1.  Documented information drawn from published sources, databases,
    files, and direct observation. The definition, driving problems,
    baseline statistics, status of documentation, and procurement
    statistics are examples of this type of information.

2.  Perceptions drawn from interviews. This data is not necessarily
    corroborated by evaluation, but represents the views of those
    interviewed.

3.  Analytical opinions of the evaluation team. This information
    represents the informed judgment of the evaluation team, and is
    exemplified by the completeness and consistency information, and the
    observations on process effectiveness.

4.  Conclusions and recommendations. This information is the result of
    synthesizing the previous three kinds of information.

# Structure of the Report

Chapter 2 provides background and context for the FAA's acquisition
reform effort, focusing on discerning congressional intent, legislative
changes to procedures, and the problems which drove acquisition reform.

Chapter 3 addresses the status of AMS implementation. Key components of
the new system are defined, and the completeness of AMS with respect to
the driving problems is assessed. Implementation milestones are
discussed, and implementation progress is estimated according to the
status of supporting AMS documentation.

Chapter 4 summarizes the data gathered from the interviews. Perceptions
of those interviewed are used to assess progress and generate findings.

Chapter 5 evaluates acquisition reform implementation in terms of
lifecycle acquisition management and focuses on key elements and
processes instituted by the new system.

Chapter 6 focuses on procurement, the area with the most radical changes
to the traditional way of conducting business.

Chapter 7 presents the conclusions and recommendations of the Evaluation
Team.

# Chapter 2Acquisition Reform {#chapter-2acquisition-reform .Chapter}

# Background

On November 15, 1995, the President signed the Department of
Transportation and Related Agencies Appropriations Act of 1996, which
directed the FAA Administrator to:

...develop and implement...an Acquisition Management System for the
Federal Aviation Administration that addresses the unique needs of the
agency and, at a minimum, provides for more timely and cost effective
acquisition of equipment and materials.

Section 348 of Public Law 104-50 also stated that the following
provisions of acquisition law "shall not apply" to the new acquisition
management system:

-   Title III of the Federal Property and Administrative Services Act of
    1949 (41 U.S.C. 252-266);

-   The Office of Federal Procurement Policy Act (41 U.S.C. 401 et
    seq.);

-   The Federal Acquisition Streamlining Act of 1994 (Public Law
    103-355);

-   The Small Business Act (15 U.S.C. 631 et seq.), except that all
    reasonable opportunities to be awarded contracts shall be provided
    to small business concerns and small business concerns owned and
    controlled by socially and economically disadvantaged individuals;

-   The Competition in Contracting Act;

-   Subchapter V of chapter 35 of title 31, relating to the procurement
    protest system;

-   The Brooks Automatic Data Processing Act (40 U.S.C. 759); and

```{=html}
<!-- -->
```
-   The Federal Acquisition Regulation and any laws not listed above
    which provide authority to promulgate regulations in the Federal
    Acquisition Regulation.

Expanding the procurement reforms previously authorized by Public Law
104-50, the FAA Reauthorization Act of 1996 (Public Law 104-264, October
9, 1996) provides the FAA Administrator autonomy in carrying out the
functions of the Administrator and the authority to enter into
contracts, grants, cooperative agreements, and other transactions with
public and private entities on such terms and conditions as are
considered appropriate.

Public Law 104-264 also requires the Administrator to terminate those
programs initiated after October, 1996 which are 50 percent over cost,
schedule, or performance goals and consider terminating any substantial
program which is 10 percent over cost and schedule goals or fail to
achieve at least 90 percent of their performance goals.

The FAA convened a Blue Ribbon Panel to provide outside expertise in
reforming its acquisition system. The panel met regularly with the
acquisition reform working group during the design of the AMS to provide
insight and criticism.

The FAA developed its Acquisition Management System (AMS) in response to
Public Laws 104-50 and 104-264. AMS became effective on April 1, 1996.

# Congressional Intent

In congressional testimony and reports, aviation industry and government
leaders identified the following concerns influencing reform
legislation:

-   A reliable, safe, effective air traffic control system is needed to
    meet user needs. Delays caused by an antiquated air traffic control
    system cost the airlines and public too much.

-   The FAA has failed to keep its air traffic control modernization
    program on track.

-   Something must be done, for the present system is not working.

-   Procurement reform is imperative to allow the FAA to conduct
    business in a manner similar to industry. The Federal Acquisition
    Regulation (FAR) is too restrictive and causes undue delays in
    acquiring systems efficiently and effectively.

-   The FAA must have a stable budget to buy technology able to support
    and maintain the air traffic control system.

```{=html}
<!-- -->
```
-   Personnel reform is required because a qualified, highly motivated
    workforce is necessary for change to meet with success.

# Problems Driving Acquisition Reform

To successfully evaluate AMS, it is important to understand the nature
of acknowledged FAA difficulties in successfully acquiring systems.
Reports by the General Accounting Office (GAO) and the Center for Naval
Analyses attributed Advanced Automation System cost and schedule growth
to poor management, requirements creep, poor planning, underestimation
of technical complexity, and poor contractor oversight.

In addition, three other programs (the Automated Surface Observing
System (ASOS), the Air Route Surveillance Radar-4 (ARSR-4), and the Mode
Select (Mode S) programs) were criticized for failing to meet
performance specifications. A 1993 report by the National Performance
Review criticized the Federal Acquisition Regulation and defined the
federal procurement process as an extraordinary example of red tape and
a burden to the FAA.

In 1996, the GAO stated the FAA's Òorganizational cultureÓ has
contributed to acquisition problems, and Òits acquisitions were impaired
because employees acted in ways that did not reflect a strong commitment
to mission focus, accountability, coordination, and adaptability." Table
2-1 summarizes these problems which faced FAA acquisition programs.

Table 2-1. Summary of Problems

  ---------------------- ------------------ -----------------------------------------------------
  **Problem**            **Description**    

  **Management Control -                    
  The effectiveness of                      
  program management.**                     

  Widely diffused        Roles and          
  responsibility and     responsibilities   
  accountability         in decision making 
                         process are not    
                         adequately         
                         defined.           
                         Accountability is  
                         not well-defined   
                         or enforced for    
                         decisions.         

  Poor coordination                         There is a lack of coordination among organizations
                                            that participate in system acquisitions.

  Poor contract          There is a lack of 
  management/oversight   oversight of       
                         contractor         
                         performance by FAA 
                         program managers.  
                         Testing program    
                         flawed.            
                         Inexperience in    
                         managing large     
                         programs.          

  **Planning - The use                      
  of analytical efforts                     
  to support capital                        
  investment,                               
  acquisition, and                          
  fielding decisions.**                     

  Inefficient and        There is a need to 
  ineffective planning   develop a more     
  processes              efficient planning 
                         process and a      
                         stable road map    
                         for the            
                         modernization of   
                         the air traffic    
                         control system.    

  Lack of understanding  Acquisitions have  
  of customer needs      proceeded to       
                         production without 
                         the agency knowing 
                         if the system      
                         meets user or      
                         customer needs.    

  Weak mission focus,    FAA mission        
  requirements analysis  analysis is not    
  and control            consistently       
                         performed. Mission 
                         Need Statements    
                         are missing,       
                         inadequate and/or  
                         unsupported.       
                         Analyses are       
                         ignored, reworked  
                         or redone to fit   
                         defined program    
                         goals.             

  Inability to recognize The AAS program    
  technical complexity   was overly         
                         ambitious. FAA and 
                         contractors have   
                         underestimated     
                         amount of work.    
                         Difficulties in    
                         developing         
                         software.          

  Inability to provide   FAA does not have  
  good estimates         the necessary      
                         tools or           
                         historical data to 
                         provide good cost  
                         or schedule        
                         estimates.         

  Inability to control   Requirements tend  
  systems requirements   to float and come  
                         from various       
                         sources.           

  **Acquisition and                         
  Procurement Process -                     
  The processes to                          
  acquire and field                         
  systems.**                                

  Inability to obligate  FAA has been       
  or spend appropriated  unable to          
  funding in timely      obligate/spend     
  manner                 appropriated funds 
                         due to             
                         bureaucratic       
                         acquisition        
                         process, poor      
                         planning and poor  
                         management.        

  Inability to field     Systems have not   
  systems in timely      been fielded on    
  fashion                time. Equipment is 
                         obsolete or fails  
                         to meet needs by   
                         the time it is     
                         delivered to the   
                         field.             

  Inability to field     There is a         
  current technology     tendency to rely   
  systems                upon development   
                         efforts rather     
                         than available     
                         commercial         
                         products.          

  Long, bureaucratic and The FAA's          
  difficult procurement  bureaucratic       
  and acquisition cycle  procurement        
                         policies lead to   
                         long, drawn-out    
                         acquisitions. The  
                         protest system is  
                         causing undue      
                         delays. The        
                         acquisition        
                         process is         
                         predicated on a    
                         20-year lifecycle  
                         despite the fact   
                         that technology    
                         changes in 3 to 5  
                         years. Too many    
                         levels of review.  
                         Approval of major  
                         acquisition        
                         program key        
                         decision points    
                         outside of the     
                         FAA. Too many      
                         documents required 
                         to support         
                         acquisition, some  
                         of which contain   
                         redundant          
                         information.       

  **Human Resources -                       
  The workforce needed                      
  to successfully                           
  acquire systems.**                        

  Workforce lacks        Inability to       
  appropriate            manage large,      
  competencies           complex            
                         acquisitions. Lack 
                         of system          
                         engineering        
                         skills.            

  Lack of incentives to  Bad news is not    
  reinforce desired      welcomed or        
  behavior               reported for fear  
                         of reprisals.      
                         Federal employees  
                         obtain no reward   
                         for efficiency.    

  Senior Management      Stable leadership  
  turnover               is a key factor in 
                         sustaining system  
                         acquisitions. At   
                         the FAA, changing  
                         leadership at the  
                         senior levels has  
                         caused reform to   
                         proceed by fits    
                         and starts and     
                         prevents a         
                         sustained          
                         consistent         
                         approach. Major    
                         decisions delayed  
                         due to leadership  
                         changes.           

  **Financial                               
  Resources - The                           
  budgeting process to                      
  support the                               
  acquisition process.**                    

  Ineffective budget     There is a need    
  process                for a stable       
                         funding stream to  
                         support major      
                         system             
                         acquisitions.      
                         Difficult to       
                         perform long range 
                         planning with      
                         year-to-year       
                         appropriations.    
                         Budget process     
                         (affordability)    
                         drives system      
                         acquisition        
                         strategy versus a  
                         basis of need or   
                         sound business     
                         practices. Need to 
                         establish linkage  
                         between            
                         acquisition        
                         process and the    
                         budget process.    
  ---------------------- ------------------ -----------------------------------------------------

# Acquisition Reform Goals

The goals identified by Congress, GAO, and the airline industry for FAA
acquisition reform are substantial. The ultimate outcome is a National
Airspace System capable of safely handling the increasing traffic load
in a budget-constrained environment. A summary of FAA acquisition reform
goals is taken from the Acquisition Reform Implementation Action Plan of
June 12, 1996:

The intent of acquisition reform is to reduce the time to field quality
products and services by 50% and the cost by 20% within three years.

The 50 percent time reduction objective is stated in the AMS core policy
document; the 20 percent cost reduction and the three-year timeframe
were goals established by FAA management.

# Relationship of Acquisition Reform to Personnel and Financial Reform

In the *Report of the Blue Ribbon Panel on FAA Acquisition Reform*, two
of the four major reform areas called out as necessary to ensure that
the FAA acquisition system supports its mission were personnel and
financial reform.

The new AMS is based on flexible guidance with a minimization of
mandatory policy. To operate successfully, this system depends on the
knowledge and skills of the users. Personnel reform, with its improved
hiring, performance management, compensation, incentive, and training
strategies will allow the FAA to upgrade and maintain its workforce.

Successful implementation of acquisition reform also requires a new
budget process. Budget reform is needed to ensure adequate and stable
multi-year funding for major acquisition programs, as well as
flexibility for managing funds without the constraints imposed by line
item program allocations and multiple appropriation sources.

# Chapter 3Acquisition Management System Status {#chapter-3acquisition-management-system-status .Chapter}

This chapter reviews the AMS as it relates to the problems to be solved
and discusses the progress made in its implementation and use. The
evaluation team analyzed the AMS and determined how well the system, as
defined and currently implemented, addresses the list of problems
identified in Chapter 2 of this evaluation. The team also compared
observable progress with planned AMS and Acquisition Reform
Implementation Team (ARIT) milestone schedules. Finally, the status of
the AMS policy and supporting documentation was used to estimate
implementation progress.

# Evaluation Results

## Progress

-   The philosophy and policy established for AMS largely addresses the
    fundamental problems that precipitated reform. The implementation of
    infrastructure and guidance is ongoing.

-   The FAA Acquisition System Toolbox (FAST) software shell is
    implemented, and its content continues to be updated as AMS
    guidance, processes, and information become available. Approximately
    50 percent of the guidance and process documentation has been
    generated and is available in FAST.

-   The Office of Business Management (ABZ) was established in April,
    1997, to focus on business processes including workforce learning.

```{=html}
<!-- -->
```
-   Acquisition Leader Learning has been initiated.

## Findings 

-   AMS completely or partially addresses 15 of the 17 driving problems
    identified by the evaluation team. Of the remaining two problems,
    one is primarily addressed by personnel reform, and the other is not
    within the scope of AMS.

-   AMS implementation is behind the established schedule; however, the
    schedule does not effectively measure implementation progress.
    Strategic and tactical milestones have been identified, but tracking
    progress against them is difficult.

-   Systematic metrics development is not evident except in the area of
    procurement.

```{=html}
<!-- -->
```
-   There is no effective plan for achieving or measuring the key
    strategic goals of AMS---a 50 percent reduction in the time to field
    new systems and a 20 percent reduction in acquisition costs.

-   Milestones were established without coordination with the
    practitioners responsible for the tasks.

## Barriers

-   It is difficult to precisely define the strategic goals.

-   There are no clear, common progress indicators regarding schedule
    and costs, nor is there automated support for compiling, reporting
    or analyzing the existing data.

## Recommendations

-   Establish a focused, rigorous plan for the Integrated Product Teams
    (IPTs) to reduce the time to field new systems by 50 percent and
    reduce acquisition costs by 20 percent. Establish an appropriate,
    effective reporting system to monitor progress.

# Discussion

## The Acquisition Management System

FAA's new AMS established policy and guidance for all aspects of the
acquisition lifecycle, from mission needs determination through the
procurement, operation, and retirement of solutions that meet those
needs. The new system consists of an acquisition policy document, a set
of generic processes, and guidance and acquisition aids from which
practitioners can formulate a tailored process to meet their needs. All
of these components reside in the automated FAST which is on the
Internet and available to employees. The AMS moves acquisition from a
rigid, rule-based process to a minimal policy, common sense-driven
process supported by guidance and tools. The definitions of the AMS
components in Table 3-1 are taken from the April 1, 1996, document.

## Completeness With Respect To The Driving Problems

In order to evaluate the completeness of AMS against the reasons for
acquisition reform, the evaluation team compared the components to
problems identified in the definition of acquisition reform. Team
members evaluated how completely the AMS, as defined by its policy,
guidance and tools, addressed each problem. The adequacy of each
solution was assessed in terms of completeness, clarity, scope, and
feasibility within the FAA acquisition environment.

Major changes have begun to enhance the agency\'s ability to accomplish
its acquisition management mission. The philosophy and policy
established for AMS largely address the driving problems, but the
implementation and infrastructure which allow the policy to work are not
complete. Table 3-2 summarizes the team's assessment of AMS
completeness. Each problem was evaluated as to how fully the policy
addressed the problem and the extent of guidance available and approved.
It should be noted that the existing FAA orders which are affected by
AMS have not been completely brought into conformance with the new
policy. The rationale for the determination is included in the table.

Table 3-1. Summary of AMS Definitions

+-----------------------------------------------------------------------+
| ::: Definition                                                        |
| **Acquisition Program Baseline (APB)**Established at the investment  |
| decision, it establishes the *critical* performance parameters and    |
| benefits that a program must achieve, and sets boundaries for cost    |
| and schedule within which the program is authorized to proceed. IPTs  |
| must submit baseline change requests to the JRC whenever breaches are |
| anticipated or congressional actions impact cost, schedule, or        |
| performance.                                                          |
| :::                                                                   |
+-----------------------------------------------------------------------+
| ::: Definition                                                        |
| **Integrated Product Development System (IPDS)**The implementing arm |
| of the lifecycle management system. The systematic teaming of         |
| functional disciplines to integrate and apply all relevant processes  |
| to produce an effective and efficient product or service that         |
| satisfies customer or user needs. Based on a "teams leading teams"    |
| approach.                                                             |
| :::                                                                   |
+-----------------------------------------------------------------------+
| ::: Definition                                                        |
| **Investment Analysis**Generates the information used by the JRC at  |
| the investment decision to determine the best overall means for       |
| satisfying mission need. It is conducted as a joint partnership       |
| between the sponsoring and acquiring organizations to ensure the      |
| critical needs of the user are satisfied by a solution that is        |
| affordable.                                                           |
| :::                                                                   |
+-----------------------------------------------------------------------+
| ::: Definition                                                        |
| **Joint Resources Council (JRC)**A body responsible for making       |
| corporate level acquisition decisions. Membership includes the        |
| Acquisition Executive, the Assistant Administrators for System Safety |
| and for Policy, Planning, and International Aviation, the Associate   |
| Administrators representing all lines of business investment areas of |
| the agency, the Director of the Office of Financial Services, and     |
| legal counsel.                                                        |
| :::                                                                   |
+-----------------------------------------------------------------------+
| ::: Definition                                                        |
| **Learning System**A learning- based methodology aimed at increasing |
| the effectiveness and productivity of the workforce. It has three     |
| major characteristics: mission driven, competency based, and goal     |
| oriented.                                                             |
| :::                                                                   |
+-----------------------------------------------------------------------+
| ::: Definition                                                        |
| **Long Range Planning**Ties together the various programs and        |
| activities underway within the agency to provide the services needed  |
| by its users and customers. Provides a framework for mission          |
| analysis, investment analysis, and program implementation and evolves |
| in response to these activities.                                      |
| :::                                                                   |
+-----------------------------------------------------------------------+
| ::: Definition                                                        |
| **Measurement System**Metrics of critical measures of performance    |
| such as time, cost, customer satisfaction, and product/service        |
| quality to assess progress and identify additional opportunities or   |
| improvement. Indicators of the status of a project or procurement.    |
| Metrics are generally quantitative but can be qualitative.            |
| :::                                                                   |
+-----------------------------------------------------------------------+
| ::: Definition                                                        |
| **Mission Analysis**A strong, full-time, forward-looking analytical  |
| activity that evaluates the capacity of the NAS to sustain existing   |
| and satisfy emerging demands for services. It identifies both         |
| critical capability shortfalls that must be resolved and              |
| technological opportunities for improving the safety, efficiency, or  |
| effectiveness of NAS operations.                                      |
| :::                                                                   |
+-----------------------------------------------------------------------+
| ::: Definition                                                        |
| **Procurement System**Those activities associated with the           |
| preparation of contract awards and the administration of those        |
| contracts.                                                            |
| :::                                                                   |
+-----------------------------------------------------------------------+
| **Solution Implementation**The phase of the acquisition lifecycle    |
| that begins after the JRC selects the most advantageous solution and  |
| approves an acquisition program. It ends when the new capability is   |
| ready to go into service. It will vary widely depending on the        |
| acquisition category and the nature of the solution selected at the   |
| investment decision.                                                  |
+-----------------------------------------------------------------------+

Table 3-2. Summary of AMS Completeness

  -----------------------------------------------------------------------------------------------------------------------------------------------------
  **Problem**                  ![](media/image4.emf)   ![](media/image5.emf)                **Rationale**      
  ----------------------- ---- ----------------------- ----------------------- ------------ ------------------ ----------------------------------------
  **Management Control**                                                                                       

  Widely diffused                                                                         While the          
  responsibility and                                                                        organizational     
  accountability.                                                                           elements and roles 
                                                                                            and                
                                                                                            responsibilities   
                                                                                            for the JRC and    
                                                                                            the Integrated     
                                                                                            Product Team (IPT) 
                                                                                            are solidifying,   
                                                                                            the roles and      
                                                                                            responsibilities   
                                                                                            for the Integrated 
                                                                                            Management Team,   
                                                                                            Integrated Product 
                                                                                            Leadership Team,   
                                                                                            and the various    
                                                                                            functional         
                                                                                            disciplines,       
                                                                                            (e.g., systems     
                                                                                            engineering, test  
                                                                                            and evaluation)    
                                                                                            require further    
                                                                                            definition.        

  Poor coordination.                                                                      The                
                                                                                            cross-functional   
                                                                                            IPDS approach      
                                                                                            enhances           
                                                                                            coordination.      

  Poor contract                                                                           Although the IPTs  
  management/oversight.                                                                     have been          
                                                                                            empowered for      
                                                                                            program decision   
                                                                                            making, and        
                                                                                            baseline           
                                                                                            management has     
                                                                                            been strongly      
                                                                                            emphasized, very   
                                                                                            little of the      
                                                                                            procedural         
                                                                                            infrastructure is  
                                                                                            defined to ensure  
                                                                                            changes in program 
                                                                                            management         
                                                                                            practice. No       
                                                                                            consistent         
                                                                                            management         
                                                                                            information        
                                                                                            collection or      
                                                                                            reporting policy   
                                                                                            exists.            

  **Planning**                                                                                                 

  Inefficient and                                                                         The AMS requires   
  ineffective planning                                                                      the integration of 
  processes.                                                                                mid- and           
                                                                                            short-range        
                                                                                            resource plans     
                                                                                            into a             
                                                                                            comprehensive,     
                                                                                            formal long-range  
                                                                                            planning           
                                                                                            framework.         

  Lack of understanding                                                                   From the outset of 
  of customer needs.                                                                        investment         
                                                                                            analysis through   
                                                                                            the program's      
                                                                                            lifecycle, the     
                                                                                            IPDS approach      
                                                                                            fosters a          
                                                                                            partnership        
                                                                                            between the        
                                                                                            sponsoring and     
                                                                                            acquiring          
                                                                                            organizations.     

  Weak mission focus,                                                                     Planning and       
  requirements analysis                                                                     control of         
  and control.                                                                              requirements is    
                                                                                            generally          
                                                                                            addressed;         
                                                                                            however,           
                                                                                            clarification is   
                                                                                            needed for the     
                                                                                            relationship       
                                                                                            between the        
                                                                                            Requirements       
                                                                                            Document and the   
                                                                                            Acquisition        
                                                                                            Program Baseline,  
                                                                                            specifically as    
                                                                                            they relate to     
                                                                                            unsatisfied        
                                                                                            requirements       
                                                                                            specified in the   
                                                                                            Requirements       
                                                                                            Document.          

  Inability to recognize                                                                  There is a formal  
  technical complexity.                                                                     investment         
                                                                                            analysis process,  
                                                                                            which is           
                                                                                            structured to      
                                                                                            translate mission  
                                                                                            need into          
                                                                                            top-level          
                                                                                            performance and    
                                                                                            supportability     
                                                                                            requirements. The  
                                                                                            move toward        
                                                                                            COTS/NDI will      
                                                                                            reduce risk and    
                                                                                            minimize technical 
                                                                                            complexity.        

  Inability to provide                                                                    Complete resource  
  good estimates.                                                                           and time estimates 
                                                                                            are required       
                                                                                            during investment  
                                                                                            analysis. These    
                                                                                            estimates form the 
                                                                                            basis for the cost 
                                                                                            and schedule       
                                                                                            boundaries in the  
                                                                                            APB. A mechanism   
                                                                                            for capturing      
                                                                                            actual results is  
                                                                                            needed for         
                                                                                            lifecycle cost     
                                                                                            estimating.        

  Inability to control                                                                    The line of        
  systems requirements.                                                                     business with a    
                                                                                            need establishes   
                                                                                            initial            
                                                                                            requirements with  
                                                                                            support from cross 
                                                                                            functional teams.  
                                                                                            CM procedures,     
                                                                                            when complete,     
                                                                                            should ensure      
                                                                                            corporate          
                                                                                            management         
                                                                                            approval for all   
                                                                                            critical technical 
                                                                                            baseline changes.  

  **Acquisition and                                                                                            
  Procurement Process**                                                                                        

  Inability to                                                                            The new            
  obligate/spend                                                                            procurement        
  appropriated funding in                                                                   process            
  a timely manner.                                                                          facilitates        
                                                                                            quicker contract   
                                                                                            award, and         
                                                                                            ACQUIRE, the new   
                                                                                            procurement        
                                                                                            planning and       
                                                                                            tracking system,   
                                                                                            when implemented,  
                                                                                            should also aid in 
                                                                                            eliminating the    
                                                                                            problem.           

  Long, bureaucratic and                                                                  The process        
  difficult procurement                                                                     emphasizes "common 
  and acquisition cycle.                                                                    sense" decision    
                                                                                            making,            
                                                                                            flexibility,       
                                                                                            business judgment, 
                                                                                            and a team concept 
                                                                                            for managing       
                                                                                            procurements. FAST 
                                                                                            and the Learning   
                                                                                            System must be     
                                                                                            completely         
                                                                                            populated.         

  Inability to field                                                                      Changes to the     
  systems in timely                                                                         procurement system 
  fashion.                                                                                  address shortening 
                                                                                            that portion of    
                                                                                            the process.       
                                                                                            Although the       
                                                                                            continuous need    
                                                                                            analysis and       
                                                                                            rigorous           
                                                                                            investment         
                                                                                            analysis may       
                                                                                            actually lengthen  
                                                                                            the early phases   
                                                                                            of the lifecycle,  
                                                                                            they should        
                                                                                            ultimately         
                                                                                            accelerate the     
                                                                                            process and        
                                                                                            eliminate many of  
                                                                                            the problems with  
                                                                                            fielding. Solution 
                                                                                            Implementation     
                                                                                            guidance is        
                                                                                            incomplete. There  
                                                                                            is no mechanism    
                                                                                            implemented to     
                                                                                            measure success.   

  Inability to field                                                                      The process is     
  current technology                                                                        designed to foster 
  systems.                                                                                  COTS/NDI           
                                                                                            solutions,         
                                                                                            evolutionary       
                                                                                            product            
                                                                                            improvement, and   
                                                                                            faster technology  
                                                                                            insertion. The     
                                                                                            effective          
                                                                                            lifecycle of       
                                                                                            systems has been   
                                                                                            changed from 20    
                                                                                            years to less than 
                                                                                            10.                

  **Human Resources**                                                                                          

  Workforce does not have                                                                 The skill mix of   
  appropriate                                                                               the acquisition    
  competencies.                                                                             management         
                                                                                            workforce is still 
                                                                                            evolving. Related  
                                                                                            Learning System    
                                                                                            infrastructure     
                                                                                            activities,        
                                                                                            including the      
                                                                                            identification and 
                                                                                            assessment of core 
                                                                                            competencies, are  
                                                                                            not implemented.   

  Senior management            NA                                                                              AMS is not the appropriate vehicle for
  turnover.                                                                                                    this issue.

  Lack of incentives to                                                                   Primarily          
  reinforce desired                                                                         addressed in       
  behavior.                                                                                 personal reform,   
                                                                                            the concept of     
                                                                                            gainsharing (later 
                                                                                            termed             
                                                                                            goalsharing) was   
                                                                                            included in AMS,   
                                                                                            but policy has not 
                                                                                            been implemented.  

  **Financial Resources**                                                                                      

  Ineffective budget                                                                      Although AMS       
  process.                                                                                  planning           
                                                                                            activities have    
                                                                                            had a marked       
                                                                                            impact on budget   
                                                                                            formulation, the   
                                                                                            budget execution   
                                                                                            process is not     
                                                                                            documented and the 
                                                                                            current            
                                                                                            implementation of  
                                                                                            the Resource       
                                                                                            Management Council 
                                                                                            is not consistent  
                                                                                            with AMS baseline  
                                                                                            management policy. 

  **KEY**                                             Insufficient, no                                        
                                                       resolution                                              

                                                      Adequate, limited                                       
                                                       resolution                                              

                                                      Thorough, full                                          
                                                       resolution                                              

                          NA                                                   Not                             
                                                                               applicable                      
  -----------------------------------------------------------------------------------------------------------------------------------------------------

## Implementation Progress Against Milestones

Through a documentation review, FAA schedule and milestone commitments
were categorized as either "strategic" or "tactical." "Strategic
milestones" are those FAA has announced publicly, either in the initial
release of the AMS policy on April 1, 1996, or in subsequent testimony
to Congress. "Tactical milestones" are internal FAA milestones which
support either the strategic milestones or important related efforts in
acquisition reform. Most tactical milestones are being tracked by the
Acquisition Reform Implementation Team (ARIT). Milestone status and the
risks from slippage are listed in Table 3-3.

The FAA is making fair progress in achieving the tactical milestones
(see Figure 3-1). Most progress has been made in the procurement area,
with the slowest movement in the implementation of metrics and
establishment of the new Learning System.

Strategic milestone achievement shows less progress (see Figure 3-2).
The two most important strategic milestones (a 50 percent reduction in
the time to field new systems, and a 20 percent reduction in acquisition
cost) are not being tracked. There is a major concern that no meaningful
strategy, plan, or metrics for achieving these exists. In addition,
there appears to be no connection between these two strategic goals and
the tactical or program-level milestones. Therefore, there is no
assurance that achievement of every tactical milestone will result in 50
percent schedule and 20 percent cost reductions overall.

A major, focused, rigorous plan, beyond the AMS process improvements
already being put into place, is recommended. A common set of measures
for the IPTs is needed, as are mechanisms to record, report and monitor
the results.

## Measuring Implementation Progress Through Documentation

Table 3-4 provides another approach to quantifying the status of AMS
implementation. It is based on an inventory of existing versus required
AMS-related documentation and indicates which elements of policy and
direction (core AMS document, related FAA orders, and FAST components)
were available as of April, 1997. As the table reflects, most of the
acquisition policy and direction has been documented and disseminated in
FAST, but it will not be fully complete until some disposition is
reached on the 39 acquisition-related FAA orders. FAST is also being
populated with supporting guidance and tools, but a smaller percentage
of this effort has been completed.

Table 3-3. Strategic Milestone Status and Risks

+-----------------------------+--------+------------------------------+
| **Milestones**              | **Sc   | **Risks from Slippage**      |
|                             | hedule |                              |
|                             | St     |                              |
|                             | atus** |                              |
+=============================+========+==============================+
| **Acquisition Management    |        |                              |
| System Implementation**     |        |                              |
+-----------------------------+--------+------------------------------+
| Convert all FAA             | s      | Minor impact. Most programs  |
| acquisitions to new system, |        | can determine their own      |
| placing each in proper      |        | status.                      |
| place in new process        |        |                              |
+-----------------------------+--------+------------------------------+
| Complete detail in roles    | ¤      | Moderate risk of confusion   |
| and responsibilities for    |        | as to which policy is in     |
| AMS                         |        | force.                       |
+-----------------------------+--------+------------------------------+
| Complete initial online     | l      | None. Work completed.        |
| acquisition tools (FAST)    |        |                              |
+-----------------------------+--------+------------------------------+
| **Integrated Product        |        |                              |
| Development System**        |        |                              |
+-----------------------------+--------+------------------------------+
| Empower 15 product teams    | s      | Moderate impact. Undermines  |
|                             |        | confidence in leadership     |
|                             |        | support for IPDS.            |
+-----------------------------+--------+------------------------------+
| Empower all product teams   | s      | Moderate impact. Undermines  |
| (about 40 total)            |        | confidence in leadership     |
|                             |        | support for IPDS.            |
+-----------------------------+--------+------------------------------+
| Mature and implement the    | l      | None. ASD-200 already        |
| program evaluation function |        | functioning as acquisition   |
| at the agency level         |        | program evaluator.           |
+-----------------------------+--------+------------------------------+
| Achieve a minimum goal of   | s      | Major risks. Loss of         |
| 50% reduction in time to    |        | credibility with Congress,   |
| acquire and field new       |        | Office of Management and     |
| capabilities                |        | Budget, etc. if no measured  |
|                             |        | time reductions are          |
|                             |        | achieved.                    |
+-----------------------------+--------+------------------------------+
| Achieve a minimum goal of   | s      | Major risks. Loss of         |
| 20% reduction in            |        | credibility with Congress,   |
| acquisition cost            |        | Office of Management and     |
|                             |        | Budget, etc. if no measured  |
|                             |        | cost reductions are          |
|                             |        | achieved.                    |
+-----------------------------+--------+------------------------------+
| Develop metrics system to   | s      | Major risks. Need credible   |
| measure AMS performance     |        | metrics very soon to drive   |
|                             |        | desired FAA behavior and     |
|                             |        | ensure results.              |
+-----------------------------+--------+------------------------------+
| **Acquisition Workforce     |        |                              |
| Learning System**           |        |                              |
+-----------------------------+--------+------------------------------+
| Complete analysis of        | s      | Teams not fully empowered    |
| current workforce           |        | nor staffed with correct     |
| competencies                |        | skill mix.                   |
+-----------------------------+--------+------------------------------+
| Develop acquisition leader  | l      | None. Work initiated.        |
| learning program            |        |                              |
+-----------------------------+--------+------------------------------+
| Ensure all 15 empowered     | s      | Moderate impact. Sub-par     |
| teams have required         |        | performance if IPTs are not  |
| competencies                |        | staffed with personnel       |
|                             |        | having required              |
|                             |        | competencies. Team           |
|                             |        | maturation process will be   |
|                             |        | hindered.                    |
+-----------------------------+--------+------------------------------+
| Set up learning support     | s      | Difficult to evaluate        |
| center                      |        | impact.                      |
+-----------------------------+--------+------------------------------+
| Establish multi-path career | s      | Difficult to evaluate        |
| model                       |        | impact.                      |
+-----------------------------+--------+------------------------------+
| Achieve Level 4 of the      | ¤      | Minor impact. Loss of        |
| People Capability Maturity  |        | credibility of Learning      |
| Model learning system model |        | System.                      |
+-----------------------------+--------+------------------------------+
| Complete human and          | s      | Difficult to evaluate        |
| intellectual capital        |        | impact.                      |
| investment plan by January  |        |                              |
| 1997                        |        |                              |
+-----------------------------+--------+------------------------------+
| Identify core leadership    | l      | None. Work initiated.        |
| competencies, set standards |        |                              |
| of excellence for leaders,  |        |                              |
| and prepare lifecycle       |        |                              |
| acquisition executives to   |        |                              |
| lead the workforce through  |        |                              |
| a joint venture with        |        |                              |
| industry and a major        |        |                              |
| university                  |        |                              |
+-----------------------------+--------+------------------------------+
| **Procurement System**      |        |                              |
+-----------------------------+--------+------------------------------+
| Complete training for all   | l      | None. Work completed.        |
| procurement personnel       |        |                              |
+-----------------------------+--------+------------------------------+
| **KEY: Strategic            |        |                              |
| Milestones**Those reported |        |                              |
| to Congress in the AMS      |        |                              |
| policy document of April 1, |        |                              |
| 1996                        |        |                              |
|                             |        |                              |
| l Milestone completed       |        |                              |
|                             |        |                              |
| ¤ Milestone likely to be    |        |                              |
| late                        |        |                              |
|                             |        |                              |
| s Milestone already         |        |                              |
| significantly late          |        |                              |
+-----------------------------+--------+------------------------------+

> Table 3-4. Status of AMS Policy Elements

+------------------------------------------------+---------------------+
| **Element**                                    | **Status**          |
+================================================+=====================+
| **Policy/Direction**                           | **88% Complete**    |
+------------------------------------------------+---------------------+
| Core Document                                  |                    |
+------------------------------------------------+---------------------+
| Related FAA Orders                             |                    |
+------------------------------------------------+---------------------+
| Mission Need Statement \*                      |                    |
+------------------------------------------------+---------------------+
| Investment Analysis Report \*                  |                    |
+------------------------------------------------+---------------------+
| Requirements Document \*                       |                    |
+------------------------------------------------+---------------------+
| Acquisition Program Baseline \*                |                    |
+------------------------------------------------+---------------------+
| Acquisition Strategy Paper (ASP) \*            |                    |
+------------------------------------------------+---------------------+
| Integrated Program Plan (IPP) \*               |                    |
+------------------------------------------------+---------------------+
| **Guidance**                                   | **43% Complete**    |
+------------------------------------------------+---------------------+
| JRC Guidance                                   |                    |
+------------------------------------------------+---------------------+
| Resource Management Council Guidance           |                    |
+------------------------------------------------+---------------------+
| Mission Analysis Guidelines                    |                    |
+------------------------------------------------+---------------------+
| Investment Analysis Guidelines                 |                    |
+------------------------------------------------+---------------------+
| Air Traffic Requirements Process Guidelines    |                    |
+------------------------------------------------+---------------------+
| Solution Implementation Guidelines             |                    |
+------------------------------------------------+---------------------+
| In-Service Management Phase/Service Life       |                    |
| Extension Phase Guidelines                     |                     |
+------------------------------------------------+---------------------+
| **Tools**                                      | **25% Complete**    |
+------------------------------------------------+---------------------+
| Procurement Toolbox                            |                    |
+------------------------------------------------+---------------------+
| Learning System                                |                    |
+------------------------------------------------+---------------------+
| Metrics                                        |                    |
+------------------------------------------------+---------------------+
| Lessons Learned                                |                    |
+------------------------------------------------+---------------------+
| Key:  Complete                                |                     |
|                                                |                     |
|  Not Available                                |                     |
+------------------------------------------------+---------------------+
| \* *Includes document description and          |                     |
| template.*                                     |                     |
+------------------------------------------------+---------------------+

# Chapter 4FAA and Industry Experiences {#chapter-4faa-and-industry-experiences .Chapter}

The evaluation team interviewed approximately 200 people to determine
the extent of AMS policy implementation throughout the FAA and its
impact on internal and external customers. FAA representatives in the
Southwest, Northwest Mountain, and Western Pacific Regions, the
Aeronautical and Technical Centers, and FAA headquarters were
interviewed, as were representatives from large, small, and minority
businesses.

The following is a distillation of the data into the most commonly heard
opinions. This data is primarily subjective; however, the success of the
AMS depends greatly on the perceptions, confidence, and understanding of
those who use it daily.

# FAA Perceptions and Responses

The team gathered information concerning implementation of acquisition
reform, its effectiveness, and how it has impacted people throughout the
agency. Interviewees were asked to describe their experiences with the
AMS and how acquisition reform had affected their organizations and
co-workers. The interviews were held under agreement of non-attribution.
Participants included executives, management, and practitioners. Table
4-1 lists the questions all were asked.

> Table 4-1. Workforce Core Questions

+-----------------------------------------------------------------------+
| 1.  What has been your involvement with acquisition reform?           |
|                                                                       |
| 2.  From your perspective, what are the primary goals and objectives  |
|     of FAA's new Acquisition Management System? So far, is the agency |
|     on track to meet them?                                            |
+-----------------------------------------------------------------------+
| 3.  In general, in a few words, how has AMS affected your work?       |
+-----------------------------------------------------------------------+
| 4.  Specifically, what is different now as a result of implementing   |
|     AMS? Please give as many examples as you can.                     |
+-----------------------------------------------------------------------+
| 5.  What specifically has your organization done (or needs to do) to  |
|     implement AMS?                                                    |
+-----------------------------------------------------------------------+
| 6.  What works best about AMS? What is not working? What is the       |
|     impact of that not working?                                       |
+-----------------------------------------------------------------------+
| 7.  What steps or actions should be taken to improve AMS?             |
+-----------------------------------------------------------------------+
| 8.  What have you observed to be the impact of AMS on people?         |
+-----------------------------------------------------------------------+
| 9.  In evaluating the effectiveness of AMS, what should the team look |
|     for?                                                              |
+-----------------------------------------------------------------------+
| 10. Is there anything else you want to say about AMS?                 |
+-----------------------------------------------------------------------+

## Progress

-   Across the agency, there is widespread knowledge of AMS and
    acquisition reform, its goals, and its philosophy.

-   AMS is considered a good beginning with potential for greater
    efficiency.

-   Practitioners feel AMS presents opportunities for greater
    flexibility, better judgment, and less bureaucracy. The sense is
    that AMS gives permission to do many things which were available
    under the old system, but were discouraged as risky or unusual.

-   The movement toward empowered teams is viewed as a strength of
    acquisition reform.

```{=html}
<!-- -->
```
-   Colocation of contracting personnel with IPTs was seen as very
    effective.

## Findings

-   There were five clear, recurring positive statements about AMS:

-    acquisitions are cheaper, faster, and more efficient;

-    the acquisition process is more flexible;

-    decision making has improved;

-    IPTs are more effective; and

-    the culture is beginning to change.

-   When asked what can be done to improve AMS, four recurring themes
    were mentioned:

-    don't go backwards;

-    achieve better integration across the agency (bring all lines of
    business into AMS);

-    deal with the organizational culture issues; and

-   --- improve the learning system.

-   The personnel involved in creating AMS were much more positive and
    were more likely to take advantage of the flexibility.

-   Tension is perceived around boundary issues within IPDS: "The
    horizontal notion of empowered teams bumps up against vertical
    notion of functional responsibility."

-   There is uncertainty, confusion, and lack of uniformity in applying
    AMS.

-   Procedures to implement the policy are not well understood, and
    guidance is slow to "trickle down."

-   There is concern, primarily from management, that teams are
    self-monitoring with insufficient accountability, and more checks
    and balances are needed.

-   Training and the Learning System are perceived to be unavailable and
    behind schedule.

-   Little emphasis has been given to results-oriented performance
    measures (metrics).

```{=html}
<!-- -->
```
-   Most feel that it is too soon to predict success or failure of AMS.

## Barriers

-   There is the perception that FAA culture actively opposes change.

-   Many believe that the lack of progress in personnel reform is
    slowing acquisition reform implementation. There is strong feeling
    that without personnel reform there will be no culture change, and
    the agency will slide back into old habits and processes.

-   Senior executives do not seem to actively encourage change. No sense
    of urgency or commitment is personally communicated to the
    workforce.

-   FAA culture is seen to hinder the sharing of lessons learned. No one
    wants to publish mistakes.

```{=html}
<!-- -->
```
-   There is some undercurrent of cynicism. Much has been promised, so
    much is expected.

## Discussion

### Flexibility

Nearly all the people interviewed felt that the AMS was much more
flexible than the previous system, but there was disagreement on whether
this was good or bad. One interviewee remarked, "AMS is so flexible that
it's too complex." Those pleased by the flexibility said that AMS gives
them the opportunity to consider options; one characterized AMS as an
easier process with a lighter workload. Some felt that the Contracting
Officer was the main beneficiary of this flexibility, which affected the
speed of the procurement process. Factors including more freedom, a move
away from bureaucracy, and streamlined processes were also seen as
positive results. People who found the AMS a step backward reported that
the documentation is so vague that it causes frustration and tension and
requires extra time to research and consider options. Another respondent
warned that flexibility is a problem because without formats or
standards, people tend to revert to the old way. One executive offered
this analogy: "If there are people imprisoned for a long time and
suddenly a hole is blown in the wall, there are going to be three
responses. Some people will rush out of the hole yelling 'free at last,
free at last!'; some folks will cautiously step through the hole, wander
around a little, and consider whether there is anything of interest
outside the hole; and the remaining folks will call in a contractor to
fix the hole."

### Culture 

Interviewees frequently expressed a belief that if the cultural concerns
of FAA employees are not dealt with, the institutionalization of AMS
cannot fully succeed. The term "culture" as used by the respondents is
not easy to define, for it is associated with everything from
organizational stovepipes and poor personnel management to arrogance and
resistance to change. The authoritative management style and back room
decision making characterized as "negotiating with stakeholders
crisis-to-crisis" were commonly mentioned examples of a dysfunctional
organization.

FAA interviewees universally agreed that communication must be improved
and increased on all levels. They suggested that the agency share
lessons learned, information from headquarters, and real data on
successes and failures to help all organizations make the best use of
AMS. One respondent put it this way: "Some people are threatened by
errors and don't want to disclose them; lessons learned input is
sanitized so it's not really helpful."

The need for major cultural changes was frequently raised. One
headquarters executive said that, "If we truly implemented the AMS (like
industry) we would have to have major structural (organizational)
changes." Another executive stated: "We have a culture that accommodates
to change, not one that innovates."

Participants felt that ARA needs to focus on change management, and line
managers need to "manage people and put Human Resource Management
reforms into effect." Another executive, referring to poor management
practices and ingrained prejudices, voiced the opinion that real
cultural change will only come about if the agency can take ineffective
and recalcitrant managers, "line 'em up, and shoot 'em." Blue Ribbon
Panel members suggested that, to alter the culture, ARA needed at least
200 new people to form a critical mass for change.

Despite some negative comments, one headquarters executive stated that
under AMS and IPDS "decision making is more data-driven" with "more
conscious decisions at all levels," while another headquarters executive
felt that "AMS forces people to think about what they are doing and make
better decisions."

### Empowerment and Accountability

Empowerment and responsibility were often mentioned in interviews. Major
concerns included roles and responsibilities and decision making
authority. Varying opinions on the current degree of empowerment were
expressed: a William J. Hughes Technical Center interviewee felt that
"people are more empowered," while a headquarters executive believed
that "empowerment is not working." Many said that "we have empowerment
in name only," and "empowerment exists in theory only; there are still
turf wars at the top." There is a view that "some people still don't
want or understand empowerment."

Empowerment and accountability are inseparable. For some, freedom is an
opportunity to thrive, but, as one interviewee stated, "People still
fear liability and accountability." Agency personnel do feel that
responsibility has been given "back to people" and "back to the IPT/CO."
However, many expressed that "the FAR \[Federal Acquisition Regulation\]
protected them" and "AMS took away their crutch."

A headquarters policy developer said, "The people are frustrated, not
knowing what to do or where to get information, and afraid of the
consequences of their decisions." An IPT lead said, "People are afraid
that if decisions go wrong, they will be fired. Trust is needed for
empowerment. Mistakes mean death in this organization. There is no clear
guidance from above that it's okay to fail."

There is tension around boundary issues within IPDS: "The horizontal
notion of empowered teams bumps up against the vertical notion of
functional responsibility." Turf issues mentioned in interviews included
deployment readiness, second level software maintenance, and the fight
over operational and maintenance budgets. One respondent said, "We've
just papered over the jurisdictional issues." Another stated that
"Executives are the least empowered people in the FAA because they seek
consensus among peers."

There was considerable discussion about the lack of line of business
(vertical) support for decisions reached on teams (horizontal). Others
felt boundaries were being acknowledged and handled well, saying "AMS
gets organizations outside of ARA included," "we have a better
relationship with ASU and AGC now," there is "more trust at lower
levels," and "the ATS and ARA working relationship is improved."

### Metrics and Measures

In the metrics area, little emphasis has been placed on results-oriented
performance measures. A commonly expressed opinion was that "It's too
soon to predict success or failure of AMS," and that it will be several
years before a major acquisition makes it through all phases of AMS.
Despite this, other metrics could be collected now to provide feedback
on particular aspects of the AMS. Interviewees were fairly uniform in
offering suggestions of metrics to collect and agreeing that more
emphasis should be put on metrics collection and analysis to use the
data effectively. The "faster, cheaper, higher quality" indicators were
suggested most frequently. Specific examples of recommended measurements
include: customer satisfaction; interviews with unsuccessful bidders;
the number of procurements per person, dates, dollars obligated/person;
stress on people; Contracting Officers' complaints; cost for similar
contracts; types of contract vehicles; FAST hits; contract
administration costs; trends of cases that went to Office of Dispute
Resolution (ODR); leadership support; and the amount of paper.
Regardless of the indicators chosen for evaluation, one respondent from
the Technical Center summed it up by saying, "If we don't start
measuring now, we'll never know."

# Industry Perceptions and Responses

Fourteen individual companies and a consortium of companies were
interviewed to determine how the AMS was affecting FAA contractors and
vendors. The questions asked were somewhat different than those asked of
FAA personnel and were designed to indicate the company's level of
experience with AMS and its history with FAA. Table 4-2 lists the core
questions for industry.

> Table 4-2. Industry Core Questions

+-----------------------------------------------------------------------+
| 1.  How familiar are you with the new FAA Acquisition Management      |
|     System?                                                           |
+-----------------------------------------------------------------------+
| 2.  What specific experience have you had with the new process?       |
+-----------------------------------------------------------------------+
| 3.  Would you consider your experience positive, negative or about    |
|     the same as before?                                               |
+-----------------------------------------------------------------------+
| 4.  Do you have (with the FAA): Completed contracts? Continuing       |
|     contracts finalized before April 1996? Contracts awarded since    |
|     April 1996? Proposal efforts for an ongoing procurement?          |
|     Resources monitoring future FAA procurements?                     |
+-----------------------------------------------------------------------+
| 5.  From your company's viewpoint, what has been the impact of        |
|     acquisition reform on: Knowledge of procurements the agency is    |
|     pursuing? Are you using the Internet to gather information?       |
|     Proposal development and the bidding process? Costs associated    |
|     with marketing, bidding, teaming, and proposal development? Risks |
|     associated with bidding? Feedback from the FAA? Dispute           |
|     resolution?                                                       |
+-----------------------------------------------------------------------+
| 6.  Has there been a change in the relationship or communication      |
|     between your personnel and the FAA contracting officer? The       |
|     IPT/program personnel?                                            |
+-----------------------------------------------------------------------+
| 7.  Has acquisition reform affected existing contracts (e.g.          |
|     requirements stability, schedule stability)? If yes, please give  |
|     some examples.                                                    |
+-----------------------------------------------------------------------+
| 8.  Have there been appreciable differences in the administration and |
|     execution of contracts awarded under the AMS? If yes, please give |
|     some examples.                                                    |
+-----------------------------------------------------------------------+
| 9.  Are there any other comments, suggestions, improvements you would |
|     like to share?                                                    |
+-----------------------------------------------------------------------+

Overall, industry response was very positive. Only one company expressed
serious doubts about AMS, and that was primarily about its possible
misuse as a means of justifying sole source awards to favorite
contractors. Small, disadvantaged businesses also expressed concerns
about the impact on their sector.

## Progress

-   The interviews showed widespread industry acceptance of AMS and
    acquisition reform, its goals, and its philosophy as a better way to
    do business.

-   The new bidding process has saved industry an estimated 30 to 40
    percent over previous bid and proposal costs.

-   Companies generally favor using the Internet, particularly to
    determine what Screening Information Requests (SIRs) are in process
    and what updates have been made to the procurement policy.

-   Openness of process and access to information have cut procurement
    and product cost by reducing guess work in responding to
    requirements.

-   Flexibility and innovation in contracting make it easier to award or
    extend contracts.

```{=html}
<!-- -->
```
-   It is easier for contractors with good performance to get follow-on
    work.

## Findings

-   Industry respondents are primarily concerned with consistency of
    policy implementation across programs. They appreciate the new rules
    but desire predictability in the application of contracting policy.

-   AMS has had a negative impact on minority/disadvantaged businesses.

-    Business is being bundled into large, existing contracts.

-    There is no longer a need for 8(a) as a "quick turnaround"
    process.

-   Coordination of SIRs to reduce duplication would help companies
    focus their resources and could provide cost savings to the FAA as
    well.

-   Misuse of sole source is a potential problem that industry is
    watching closely.

-   The dispute resolution process is not clear, and some companies are
    wary of it.

```{=html}
<!-- -->
```
-   With release of SIRs on the Internet, it is harder to plan work
    because there is no longer advance notice available (as in the
    *Commerce Business Daily* ), and short turnarounds put additional
    pressure on small business marketing staffs.

## Barrier

-   Maintaining public and industry trust in the fairness of
    acquisitions may be difficult given the new flexibility and the
    reduction in process.

## Discussion

### Impact of AMS on Small Economically Disadvantaged Businesses 

Small economically disadvantaged businesses (SEDBs) interviewed
perceived that the new AMS offers them less opportunity to compete for
FAA contracts. Respondents viewed the new AMS as more subjective, with
the potential for more sole source contracts despite the appearance of
an unbiased process. As examples, they cited SIRs repeatedly advertised
for short periods of time and subsequently canceled, recompeted, placed
on indefinite hold, or issued as sole source. To them, these actions
signaled a lessening of FAA's commitment toward inclusion of small and
disadvantaged businesses. Initial protest actions are underway by
disadvantaged companies, and the number of protests in this area is
likely to increase.

Additionally, it was perceived that, since FAA is now exempt from the
Small Business Administration legislation, there is little incentive for
the agency to seek and award contracts to SEDBs. Respondents generally
observed that more requirements are being rolled up into larger,
existing vehicles like the Department of Transportation's Information
Technology Omnibus Procurement (ITOP). On two occasions, an SEDB
contractor was shut out of a recompete only to have its staff hired away
by the larger firm.

Internet advertising of SIRs was generally seen as positive. However,
two interviewees observed more difficulty in planning work because
advance notice and forecasting information are no longer available in
the *Commerce Business Daily*. (To address this, the FAA Small Business
Office recently created an Internet Web page that projects business
opportunities FAA-wide.) The shortened response time to SIRs can make it
difficult to respond within the allotted time frame. One interviewee
noted that selected companies receive additional or advance information
regarding the original SIRs, including amendments and informational
letters, while others do not.

# Chapter 5---Lifecycle Acquisition Management  {#chapter-5lifecycle-acquisition-management .Chapter}

Lifecycle Management encompasses the processes from identification of a
mission need through the implementation, operation, and final
disposition of a solution. The JRC was created to make corporate-level
decisions on resource allocation and budgeting, while the IPDS was
designed to push program and technical decisions to the lowest possible
level. The set of processes and tools established to support these
decisions includes mission analysis, investment analysis, decision
making, solution implementation, in-service decisions, in-service
management, and service life extension.

There have been few new programs initiated since the establishment of
AMS, and no programs have gone through the entire process. Little data
is available to assess how well the system is working to date. This
initial evaluation provides a qualitative assessment of implementation
progress, barriers to success, and potential future problems.

# Evaluation Results

## Progress

-   A revised core policy has been completed and awaits signature by the
    Administrator.

-   Approximately 50 percent of the generic processes and guidance that
    support the core policy of AMS have been completed, approved, and
    entered into FAST.

-   Detailed guidelines for conducting mission analysis and investment
    analysis have been created, approved and published for use.

-   A configuration management process for AMS has been developed.

-   The JRC has been established and meets regularly to make
    corporate-level decisions and optimize resource allocation.

-   The IPDS philosophy and mechanics are being implemented. IPTs are
    taking greater responsibility for project-related decisions.

-   Mission analysis staffs have been established within several of the
    FAA Lines of Business. The Mission Analysis Steering Group
    coordinates performance across all lines of business.

-   The Office of System Architecture and Investment Analysis (ASD) now
    provides a central focus for development of the NAS Architecture and
    for investment analysis. An Architecture group and an investment
    analysis staff have been put in place.

-   Several multi-organization investment analysis teams are working
    collaboratively to provide a basis for major strategic investment
    decisions.

```{=html}
<!-- -->
```
-   The System Engineering Operational Analysis Team (SEOAT) established
    to support the JRC conducts affordability assessments for proposed
    new programs and develops prioritized F&E/R,E&D budget proposals.

## Findings 

-   The revised AMS document has been delayed seven months and awaits
    the Administrator's signature. The hiatus has caused confusion as to
    whether the "old" policy or the "new" policy is governing. The
    policies are inconsistent in key areas including the development and
    coordination of requirements documents and acquisition program
    baselines.

-   The solution implementation guidelines for system/software programs
    need to be completed. Although the operational framework has been
    put in place, much of the FAST content is still under development.

-   Establishing a Resource Management Council (RMC) as a second
    corporate-level decisionmaking body is inconsistent with the AMS
    principle of unified agency-wide planning, programming, and
    budgeting within a long-range strategic framework.

-   FAA Associate organizations have been slow to accept and implement
    IPDS.

-   There is no consistent acquisition program measurement across the
    agency.

```{=html}
<!-- -->
```
-   JRC-approved acquisition program baselines do not exist for most
    programs, complicating the completion of realistic affordability
    assessments and rendering IPT accountability for meeting commitments
    more difficult.

## Barriers

-   The traditional roles and responsibilities of the separate
    Operations and F&E/R,E&D budget processes, the different accounting
    and estimating techniques applied to each, and the establishment of
    the RMC outside of AMS make the consolidation of these activities
    daunting.

-   Prioritize programs, functions and processes and reallocate
    resources accordingly, eliminating low-priorities. Centralize
    processes where efficiency or effectiveness gains are possible.

-   Establishment of realistic baselines is difficult because funding
    requirements exceeding the available resources, and out-year funding
    remains uncertain.

-   Limited acquisition personnel resources have been overextended by
    the creation of new product teams, new process teams, new
    organizations, and new initiatives.

```{=html}
<!-- -->
```
-   A natural tension exists between empowerment of IPTs and the need
    for centralized and standardized processes where products are highly
    interdependent and safety is critical.

## Recommendations 

-   Obtain the Administrator\'s approval of the revised AMS core policy.

-   Complete the solution implementation guidance for systems/software
    as soon as possible, and expedite the creation of remaining
    processes, guidance, and tools.

-   Merge the JRC and RMC into a single Assistant and Associate
    Administrator-level body to optimize investment decisions and
    resource allocation across all FAA appropriations. An expanded SEOAT
    could support the merged council.

-   Establish and enforce a baseline management policy and procedures,
    and expedite the establishment of baselines for legacy programs.

# Discussion

## Core Policy 

The AMS core policy document, signed on April 1, 1996, provides the
necessary policy for the implementation of acquisition reform for the
Agency. During the past year, this document has been updated based upon
experience, providing clarification and additional detail for the
initial policy. It is supported by a series of guidance documents.
Although completed in December, 1996, the updated AMS policy has not
been signed. Delay in approval and dissemination, accompanied by
inconsistencies between policy versions is causing confusion as to which
is the governing policy document**.**

## Joint Resources Council 

JRC guidance was issued in October, 1996, and expanded in January, 1997.
Interviews with program offices revealed some confusion about the
details of JRC activities and requirements. Administrative meetings of
the JRC have provided improvements in the effectiveness of the JRCs,
scheduling guidance, and guidance for "paper JRCs" under specific
circumstances.

During the first year, 22 formal and 4 "paper" JRC meetings were
conducted. Table 5-1 summarizes the JRC activity.

Table 5-1. JRC Decisions

  ------------------------- --------------- --------------- -----------------
  **Decision**              **Approved**    **Deferred**    **Disapproved**

  Mission Need              2               0               1

  Investment                6               1               

  Re-Baseline               5               2               

  Administrative            8                               
  ------------------------- --------------- --------------- -----------------

Scheduling difficulties plagued the first year of JRC meetings; over 73
percent of those scheduled through February, 1997, were canceled or
postponed. Steps have been taken to correct these problems. Meetings are
now scheduled at the request of the Mission Need Analysis Group or
Investment Analysis Team, ensuring that pertinent information is
available for the decision. When necessary, a designated FAA executive
may attend in place of a JRC member.

The JRC has strengthened its role in the initial year by assuming
responsibility for guidance, review, and approval of recommendations for
F&E/R,E&D budget submissions. This is being made consistent with the NAS
Architecture through the budget process and through the JRC's assumption
of the approval role for the Architecture.

The establishment of the RMC with approval authority over the operations
budget and with essentially the same membership as the JRC, is a
deviation from the intent of the AMS. A major principle of the AMS is to
unify the elements of lifecycle acquisition management into an efficient
and effective system. The establishment of two separate bodies confuses
the roles and responsibilities of both. Visibility into the operations
budget by the JRC is limited, making realistic affordability assessments
impossible. With the AMS defining a lifecycle process and IPTs
responsible for lifecycle management, a means for decisions on lifecycle
funding of all types of appropriations is needed.

## Metrics and Performance Measurement 

The agency has made progress in defining and implementing metrics for
procurement activities. Monthly status reporting to ARA-1 of cost,
schedule, and technical progress continues to be refined. However,
considerable work remains to enable meaningful measurement of other
aspects of AMS. Major stumbling blocks include the inherent difficulties
in defining and measuring established agency strategic goals as well as
multiple uncoordinated agency efforts to develop and implement metrics
and performance measurement systems.

The agency appears to have at least three separate, simultaneous,
marginally coordinated metrics efforts. With no agency focus on
developing and implementing a single metrics system to measure
achievement of acquisition reform goals and objectives, there is no way
to determine the success of AMS. The necessity of measurement for AMS
evaluation should not obscure the additional needs for consistent
metrics definitions and reporting processes and procedures; cost
accounting methods that allow a clear understanding of full lifecycle
acquisition costs; and automated management information systems to
collect, process, and communicate the relevant data to FAA decision
makers.

## Program Baselines

The agency has committed to establishing cost, schedule, performance,
and benefits baselines for acquisition programs to support:

-   The IPTs in their efforts to meet their commitments to the FAA
    Acquisition Executive and the JRC;

-   The Acquisition Executive and the JRC in their efforts to accomplish
    "more timely and cost-effective acquisitions of equipment and
    materials" in compliance with Public Law 104-50, Sec. 348;

-   The Administrator's - efforts to determine if and when acquisition
    programs should be terminated in compliance with Public Law 104-264,
    Title II, Sec. 252; and

```{=html}
<!-- -->
```
-   Other government entities (e.g., Office of Management and Budget,
    General Accounting Office, and congressional committees) in their
    efforts to gain insight into the value, status, and prognosis for
    FAA acquisition programs.

Although the AMS provides for JRC approval of the cost, schedule,
performance, and benefits baselines documented in the Acquisition
Program Baseline (APB) at the investment decision, only a few
acquisition programs are new enough to have gone through this process.
The agency has also undertaken the task of establishing baselines for
all F&E/R,E&D acquisition programs begun prior to AMS.

Development of a comprehensive agency plan for establishing and managing
acquisition program baselines has been complicated by unresolved issues
regarding the applicability of "standard" baselines to some types of
programs and the potential value of implementing baselines for
relatively "mature" acquisitions.

Baselining has also been difficult because there are greater funding
needs than available resources. Thus, it has not been possible to date
to devise an acceptable way to baseline programs to achieve goals
consistent with out-year affordability constraints. Moreover, there is
considerable uncertainty with the expected level of out-year funding
itself; FAA's planning forecast has shrunk by a third, from nearly \$3
billion per year to \$1.9 billion per year in just three years, as a
consequence of budget cuts. The current APB status of 145 F&E/R,E&D
funded programs is summarized in Table 5-2.

Table 5-2. Summary Baseline Inventory

  ---------------- ---------- ---------- ---------- ------------- ---------- ------------ ------------
  **CIP Program    **APB      **APB      **Total    **Remaining   **Mature   **Programs   **Total
  Area**           Approved   Approved   Approved   Programs      Programs   to be        Number of
                   Prior to   by JRC**   APBs**     Requiring     (75 %      Reviewed for Programs**
                   AMS**                            APBs**        Spent)**   APB Level**  

  Automation       4          2          6          11            5          0            **22**
  Programs                                                                                

  Communications   0          0          0          10            4          3            **17**
  Programs                                                                                

  Facilities\      3          2          5          0             0          31           **36**
  Programs                                                                                

  Mission Support  0          0          0          1             0          36           **37**
  Programs                                                                                

  Navigation and   1          0          1          3             9          1            **14**
  Landing Programs                                                                        

  Surveillance     3          0          3          2             5          0            **10**
  Programs                                                                                

  Weather\         2          2          4          1             4          0            **9**
  Programs                                                                                

  **Total          **13**     **6**      **19**     **28**        **27**     **71**       **145**
  Programs**                                                                              
  ---------------- ---------- ---------- ---------- ------------- ---------- ------------ ------------

Six programs have APBs approved by the JRC. In addition, 13 programs
have baselines approved prior to AMS implementation, 27 programs are
relatively mature (over 75 percent of funds expended) and may only
require limited baselining at this time, and 70 programs require
development of "non-standard" baseline parameters. These "non-standard"
programs include support services such as System Engineering and
Technical Assistance (SETA) and MITRE's Center for Advanced Aviation
System Development (CAASD), various regional projects, test equipment
modernization, and some facility refurbishment programs. Twenty-eight
major programs require JRC approval of baselines. This constitutes a
significant high-priority workload for the JRC during the next year.

Other concerns regarding the agency's current acquisition baselining
effort include issues of consistency in baseline definitions,
measurement techniques, and reporting practices across all management
levels from the IPT through the JRC. Acquisition program baselining is
an important watch area for the FAA.

## Integrated Product Development System/Lead the Fleet Programs

AMS incorporates the IPDS, which was initiated prior to acquisition
reform. IPDS provides for the cross-discipline communication and
collaboration critical to institutionalizing lifecycle acquisition
management. It also implements the empowerment of IPTs required for
programmatic decisions to be made at the lowest possible level.

The IPDS is based upon a concept of "teams leading teams." There is
still significant work to be done to put this concept into place. The
agency has made significant strides, but this is a cultural change which
will take more than the two years which have elapsed since its
inception. Progress in the past six months has increased with the
Integrated Product Leadership Team (IPLT) taking a more visible role.

Empowerment of the individual IPTs has been accomplished, as expected,
to varying degrees depending on the individuals comprising the teams.
The Lead the Fleet concept was intended to provide "move-out" prototype
models and to demonstrate how IPTs could execute more smartly if they
were empowered and staffed with personnel possessing the right set of
skills. As these IPTs carved out new techniques and methods, they would
be emulated by the rest of the workforce. Unfortunately, the
establishment of Lead the Fleet programs has been of little value to
either the programs themselves or to overall AMS implementation speed.
Enhanced staffing never occurred, nor did these programs receive any
specialized assistance. Several Lead the Fleet product team leaders felt
that they were "rewarded" with more work and briefing responsibilities
and gained no benefit from the designation.

Empowerment was to be measured by the number of IPT Program Plans
developed and approved. The intent was to have 15 of these (including
Lead the Fleet programs) approved within the first year. So far, only
nine have been approved, but three more are currently scheduled for IPLT
review. Initial approvals appear to have been held up by organizational
issues with lifecycle management responsibilities delegated to IPTs.
These issues have been resolved, and the approval process for program
plans may now move faster.

Ironically, the success of the individual IPTs could become a harbinger
of future overall National Airspace System (NAS) failure or of
additional costs to correct problems with system inter-connectivity. The
empowerment of the IPTs must be carefully balanced against the need for
standardized reporting systems, centralized data availability, a strong
NAS Architecture, and connected management control systems. The initial
review indicates a number of inconsistent efforts and designs. How will
the system architecture be maintained and used to ensure system
interoperability? Will the processes allow for the system engineering to
overcome the empowerment of the individual IPTs striving for faster,
lower cost acquisitions? A careful review of the processes and
organizations being put into place should be conducted to determine
which processes, data bases, systems, and management controls should be
standardized or centralized and to what extent.

## Human Resource Concerns

Limited human resources are available to accomplish acquisitions and to
implement acquisition reform. As new product teams and IPTs are created
to meet new acquisition needs, and as new organizations are formed for
new or high-priority functions, the resources come from the same pool of
talent. There are concerns that resources are spread too thin to
accomplish all projects and programs effectively. There appears to be
insufficient management flexibility to obtain the necessary new
resources, increase the competency and effectiveness of current
resources, and reduce the number of poor performers who have not
responded to the needs. The most pressing concern is the lack of the
numbers and skill base to adequately cover new demands.

With limitations on the overall number of employee positions, movement
of personnel is a zero sum game, even though it may help to maximize the
overall organizational effectiveness by putting the "right people" in
the "right positions." Without personnel reform, which has moved at a
significantly slower pace than acquisition reform, the agency will
encounter great difficulty in obtaining and optimally allocating
sufficiently qualified personnel. Personnel reform needs to be sped up,
and personnel management is an important area to watch and measure in
the coming year.

# Chapter 6Procurement {#chapter-6procurement .Chapter}

In all of AMS, the most radical changes to the traditional way of doing
business are found in the procurement process. The removal of the
requirements contained in the FAR and other regulatory documents has had
a profound impact on both regional and headquarters procurement
activities. It is also procurement that gives the earliest indications
of how AMS is performing.

The evaluation team conducted over 100 personal and group interviews of
procurement and program personnel in the Southwestern, Western Pacific,
Northwest Mountain, Great Lakes, and Southern Regions, the William J.
Hughes Technical Center, the Mike Monroney Aeronautical Center, FAA
Headquarters, the Office of Dispute Resolution, and the Office for Small
and Socially and Economically Disadvantaged Business. The team also
collected and analyzed data from existing automated systems, secondary
compiled reports, and existing records.

# Evaluation Results

## Progress

-   One year after the FAA announced AMS, new processes with automated
    guidance and tools are available to all FAA procurement employees.

-   Communication networks, semi-annual meetings, and procurement tool
    refinements support the workforce.

-   A comprehensive system of procurement performance measures has been
    identified, and collection has been initiated.

-   A communication network with representatives from regional, center,
    and headquarters procurement communities regularly addresses
    important AMS procurement issues and problems.

```{=html}
<!-- -->
```
-   Automated data and interviews with customers indicate that
    procurement actions are more timely, and increased credit card usage
    indicates a measurable cost saving.

## Findings

-   The time from SIR to contract award for contracts over \$100,000 has
    been reduced from a minimum time of 90 days before AMS to an average
    of less than 60 days.

-   Customers and contracting officers perceive the system to be more
    flexible.

-   Measures for procurement savings are unknown; however savings from
    the increased use of credit cards are estimated at \$1.4 million.

-   Competition remains the preferred method of contracting.

-   The percentage of total contracts awarded to all small and
    economically disadvantaged businesses has decreased by 72 percent
    from pre-AMS levels. The percentage of total awards to all small
    businesses is up 15 percent over that period.

-   Automated procurement tools are in place, but some, including
    lessons learned, are not used by most procurement offices.

-   The process for resolution of protests and contract disputes is not
    formally defined.

```{=html}
<!-- -->
```
-   Poor communication, lack of formal process, unstable funding, and
    lack of permanent staffing all contribute to protest and contract
    dispute problems in the Office of Dispute Resolution (ODR).

## Barriers

-   The current automated procurement system does not automatically
    identify management and performance data and metrics.

-   The Learning System has not inventoried procurement competencies or
    indicated skill needs, which results in a perceived need for formal
    procurement training.

-   Some individuals are slow to move from proven ways even when given
    new options and flexibility. Availability of new tools and processes
    does not guarantee their immediate use.

```{=html}
<!-- -->
```
-   Socio-economic goals are independent from business goals and require
    special emphasis.

## Recommendations 

-   Fully fund and staff the Office of Dispute Resolution and complete
    the required processes and procedures.

-   Determine the requirements for and articulate the benefit of small
    economically disadvantaged business contracts and implement guidance
    accordingly.

-   Develop an integrated metrics program and educate the workforce on
    its use and value.

# Discussion

## Procurement Guidance and Processes

The AMS implementation plan for changes in procurement policies and
practices included:

-   Developing automated tools for good acquisition decisions;

-   Building skills and confidence of the procurement workforce to best
    use the flexibility of the new regulatory environment; and

```{=html}
<!-- -->
```
-   Establishing a comprehensive system of performance measurements.

### Communication

Procurement processes and guidance developed by ASU-100 are available to
the FAA procurement community through the Procurement Toolbox in FAST
and by e-mail.

A network of New Process Coordinators (representatives from each region,
center, and FAA headquarters procurement offices) communicates through
monthly teleconferences and e-mail to address issues important to the
procurement community.

### Training

Initial orientation training was provided to all in the FAA procurement
community and included legal and program personnel. A total of 850
individuals attended the briefing, which introduced the philosophy of
AMS but did not provide a guide for applying the new flexibility. As new
guidance was developed and became available in the electronic toolbox,
the New Process Coordinators took on the responsibility of disseminating
the new information to the procurement offices. Thirty-five of the
individuals interviewed expressed concerns about training and the
Learning System (14 wanted more formal AMS training, and seven wanted
more specific procurement skill training).

### Tracking and Metrics

The Procurement Performance Measurement Working Group developed the FAA
Performance Measurement Plan to establish the framework for tracking
procurement system reform and for measuring its effectiveness. In a plan
presented on August 30, 1996, the group identified ten objectives for
AMS procurement success.

Attempts to collect data for the objectives have been 60 percent
successful, and the data is being validated. The balance of procurement
measures is not being collected. Table 6-1 describes the status of the
metrics.

## Process Flexibility

Interview data revealed that practitioners are divided as to the
effectiveness of the AMS flexibility. Approximately an equal number of
individuals like the freedom to implement creative solutions to problems
as are concerned about the lack of standardization; the impact on the
legal staff; and concerns over litigation, accountability, and personal
error.![](media/image10.emf){width="3.670138888888889in"
height="2.716666666666667in"}

### Timeliness

Timeliness is one of the areas where the new procurement policy seems
effective. The time from SIR release to award has decreased for
contracts over \$100,000. As a measure of timeliness, the Office of
Acquisition monitors the results of contract awards over \$100,000 at
FAA Headquarters from the issuance of the Screening Information Request
(SIR) to actual contract award. The minimum time to award a contract
exceeding \$100,000 using the FAR restrictions is 90 days. The data in
Figure 6-1 was collected by ASU-100 quarterly since April 1, 1996, and
shows a significant reduction in the average procurement time.

Table 6-1. Procurement Metrics Objectives, Measures, and Status

+---------------------------+------------------------------+----------+
| **Objective**             | **Measure**                  | **       |
|                           |                              | Status** |
+===========================+==============================+==========+
| 1.  To reduce by 50% the  | Time from receipt of         | On hand  |
|     time it takes to      | procurement request in       |          |
|     acquire and field new | contracting office to SIR    |          |
|     products              |                              |          |
+---------------------------+------------------------------+----------+
|                           | Time from SIR to Contract    | On hand  |
|                           | Award                        |          |
+---------------------------+------------------------------+----------+
|                           | Number of credit card        | On hand  |
|                           | transactions versus number   |          |
|                           | of purchase orders           |          |
+---------------------------+------------------------------+----------+
| 2.  To improve the        | Total procurement            | Not      |
|     productivity of the   | actions/number of full time  | co       |
|     procurement system    | employees (DELETED)          | llecting |
|     workforce             |                              |          |
+---------------------------+------------------------------+----------+
| 3.  To reduce by 20% the  | Total procurement dollars    | Not      |
|     cost to acquire       | (DELETED)                    | co       |
|     products and services |                              | llecting |
+---------------------------+------------------------------+----------+
| 4.  To establish          | Number of new contracts      | On hand  |
|     competition for       | completed/total number of    |          |
|     products and services | new contracts                |          |
|     among two or more     |                              |          |
|     sources as the        |                              |          |
|     preferred method of   |                              |          |
|     source selection      |                              |          |
+---------------------------+------------------------------+----------+
|                           | Dollars obligated for        | On hand  |
|                           | completed new                |          |
|                           | contracts/total new contract |          |
|                           | dollars                      |          |
+---------------------------+------------------------------+----------+
| 5.  To increase the       | Number of commercial         | Not      |
|     procurement of        | procurement actions/total    | co       |
|     commercial and        | number of procurement        | llecting |
|     non-developmental     | actions                      |          |
|     items                 |                              |          |
+---------------------------+------------------------------+----------+
|                           | Dollars obligated for        | Not      |
|                           | commercial procurement       | co       |
|                           | actions/total procurement    | llecting |
|                           | dollars (DELETED)            |          |
+---------------------------+------------------------------+----------+
|                           | Dollars spent on credit card | On hand  |
|                           | transactions/total           |          |
|                           | procurement dollars          |          |
+---------------------------+------------------------------+----------+
| 6.  To enable tailoring   | Percentage of contracts      | Not      |
|     to meet the goals of  | using oral proposal          | co       |
|     each requirement      |                              | llecting |
+---------------------------+------------------------------+----------+
| 7.  To improve internal   | User Satisfaction (PMAT)     | 2        |
|     user satisfaction     |                              | offices  |
|                           |                              | only     |
+---------------------------+------------------------------+----------+
| 8.  To strive to provide  | Number of contracts awarded  | On hand  |
|     small business with   | to small business/total      |          |
|     attainable and        | number of contracts          |          |
|     reasonable            |                              |          |
|     opportunities to      |                              |          |
|     participate as        |                              |          |
|     c                     |                              |          |
| ontractors/subcontractors |                              |          |
+---------------------------+------------------------------+----------+
|                           | Dollars obligated for small  | On hand  |
|                           | business contracts/total     |          |
|                           | contract dollars             |          |
+---------------------------+------------------------------+----------+
|                           | Dollars sub-contracted by    | On hand  |
|                           | large business to small      |          |
|                           | business/total large         |          |
|                           | business contract dollars    |          |
+---------------------------+------------------------------+----------+
| 9.  To reduce contractor  | Industry satisfaction and    | Not      |
|     bid and proposal      | cost savings (voluntary      | co       |
|     costs by using the    | survey in SIR)               | llecting |
|     new acquisition       |                              |          |
|     management system     |                              |          |
+---------------------------+------------------------------+----------+
| 10. To ensure that        | Number of protests           | On hand  |
|     offerors are treated  |                              |          |
|     fairly and disputes   |                              |          |
|     are resolved quickly  |                              |          |
|     and inexpensively     |                              |          |
+---------------------------+------------------------------+----------+
|                           | Number of protests/number of | On hand  |
|                           | new awards                   |          |
+---------------------------+------------------------------+----------+
|                           | Average time to render a     | Not      |
|                           | protest decision             | co       |
|                           |                              | llecting |
+---------------------------+------------------------------+----------+
|                           | Procedural fairness          | Not      |
|                           | (voluntary survey)           | co       |
|                           |                              | llecting |
+---------------------------+------------------------------+----------+

Interview responses from both procurement officials and user groups have
expressed optimism that, due to increased flexibility and relief from
previous time thresholds imposed by the FAR, significant time savings
have been achieved. Each region, each center, and ASU-300 discussed
success stories whereby they have been able to plan, solicit, and award
a contract in a much shorter time frame under AMS than under FAR.
Examples of these successes from ASU-300 include:

-   The Security Equipment Installation and Integration Services
    contract took just 4 weeks from the time of SIR issuance to the
    contract award, and

```{=html}
<!-- -->
```
-   The CTX 5000SP Explosives Detection System took just 2 weeks from
    the time of the SIR to the award of the contract.

### Competition

The AMS emphasizes and encourages competition as the preferred
procurement method to be utilized whenever practical and reasonable. It
defines competition as providing two or more sources with an opportunity
to express an interest in satisfying the agency's requirement.
Interviews with both procurement employees and user groups cite positive
developments brought about by procurement reform.

Procurement reform has also relieved the procurement employees and user
groups from cumbersome documentation requirements. AMS requires that
sufficient documentation be developed to explain and justify the
procurement action and support the business decisions made. Again,
through employee interviews, this administrative relief has described as
a positive change.

As shown in Table 6-2, data from the Contract Information System
indicates that the agency is increasing the percentage of both
competitive and single source awards. However, this increase is at the
expense of disadvantaged businesses. The impact of this finding is
discussed in the section reviewing small business and minority business
procurements.

> Table 6-2 Competitive vs. Single Source Awards

  ------------- ------------- ----------- ------------ ------------- ---------- ----------- ------------ -- ------------
                **Pre-AMS                              **AMS                                                
                4/95-3/96**                            4/96-3/97**                                          

  **Type of     **Number of   **Percent   **Value of                 **Number   **Percent   **Value of      **Change in
  Award**       Awards**      of Total    Awards**                   of         of Total    Awards**        Percentage
                              Awards**                               Awards**   Awards**                    of Total
                                                                                                            Awards**

  Single Source 123           18%         \$90.87 M                  162        24%         \$107.73 M      +33%

  SEDB/8A       173           26%         \$84.09 M                  89         14%         \$33.98 M       -44%

  Competitive   377           56%         \$283.74 M                 416        62%         \$212.55 M      +11%

  **Total       **673**                   **\$458.73                 **677**                **\$354.26      
  Awards**                                M**                                               M**             
  ------------- ------------- ----------- ------------ ------------- ---------- ----------- ------------ -- ------------

### Credit Cards

AMS authorizes the use of credit cards (also known as purchase cards)
consistent with prudent business practice. Though the cards are not an
innovation of the AMS, a strong emphasis has been placed on their use.
The Senior Executive Working Group, which includes representatives from
all major civilian government agencies, has determined that the savings
by using a credit card over that of using the standard purchase order
(PO) is \$53.77 per transaction.

Since the establishment of AMS, the number of authorized cardholders has
grown 19 percent. Credit card transactions during the same period
increased by 39 percent. Obligations by credit card increased by 74
percent. This larger percentage increase in obligations can be explained
in part by the elimination of many restrictive procurement thresholds in
the new acquisition document as well as the delegation of purchasing
authority to the lowest possible level.

Conversely, the number of simplified purchase transactions has dropped
by more than 50 percent since AMS was implemented. Additionally, there
was a 68 percent decline in the amount of obligations by PO for the same
period. The figures in Table 6-3 were provided by the Standard Form 281
report consolidated by ASU-100 for the FAA and reported to DOT.

Table 6-3. Summary of Credit Card Data

+----------------+-------------------+-------------------+------------+
| **Credit Card  | **Pre-AMS**       | **AMS**           | **Change** |
| Purchases**    |                   |                   |            |
|                | **(               | **(               |            |
|                | 4/1/95-3/30/96)** | 4/1/96-3/30/97)** |            |
+----------------+-------------------+-------------------+------------+
| Number of      | 5591              | 6675              | +19%       |
| Cardholders    |                   |                   |            |
+----------------+-------------------+-------------------+------------+
| Number of      | 125,399           | 174,876           | +39%       |
| Transactions   |                   |                   |            |
+----------------+-------------------+-------------------+------------+
| Total Amount   | \$39.20 M         | \$68.52 M         | +75%       |
| of             |                   |                   |            |
| Transactions   |                   |                   |            |
+----------------+-------------------+-------------------+------------+

Using the \$53.77 savings per transaction, and assuming that most of the
decrease in simplified purchases is the result of increased credit card
usage, the agency shows an additional administrative savings of \$1.4
million by making purchases on credit cards instead of using other
simplified purchase methods.

### Socio-Economic and Other Policies and Programs

AMS states that the FAA will implement and aggressively strive to
provide small businesses and SEDBs attainable and reasonable
opportunities to participate as prime contractors and subcontractors for
required products and services. It furthermore tasks the FAA's Small
Business Utilization staff with the responsibility for:

-   Overseeing the agency's policy and program on the utilization of
    small and SEDB firms;

-   Establishing mechanisms for monitoring and evaluating the
    effectiveness of the small business program; and

```{=html}
<!-- -->
```
-   Ensuring FAA wide implementation and accomplishment of small
    business program objectives.

During the interview process of this evaluation, all procurement offices
contacted expressed the opinion that their commitment to contracting
with small business and SEDB firms remained high, but that awards to
SEDB firms had declined slightly.

Members of the agency's Small Business Utilization staff (ARA-5)
interviewed raised concerns with the contracting totals for SEDB firms.
Data collected directly from the procurement offices through the Major
Procurement Preference Goal (MPPG) report supports the concern.

The MPPG report dated September 30, 1996, shows that the percentage of
awards to small businesses increased since AMS began. Furthermore, the
AMS has established a new reporting group of entitled very small
businesses which received an additional 3 percent of awards. AMS appears
to be achieving its goal of providing reasonable opportunities for small
businesses to contract for products and services.

As shown in Table 6-4, the percentages of total awards which went to
SEDB firms during the same time frames were significantly reduced. One
explanation for the decrease in SBA 8(a) awards is that the AMS no
longer requires that the agency contract directly with SBA. AMS does
however, encourage competitive use of 8(a) firms which would be recorded
as SEDB firms for reporting purposes. As can be seen by the MPPG report,
the percentage of awards to all types of disadvantaged businesses fell
by 72 percent.

Table 6-4. Comparison of Small Business Obligations

+-----------+------------+----------+------------+----------+--------+
| **Small   | *          | **P      | **AMS**    | **AMS**  | **     |
| Business  | *Pre-AMS** | re-AMS** |            |          | Change |
| Ut        |            |          | **(4/1/96- | **(4     | in     |
| ilization | **(4/1/95- | **(4     | 9/30/96)** | /1/96-9/ | perc   |
| Direct    | 9/30/95)** | /1/95-9/ |            | 30/96)** | entage |
| Pr        |            | 30/95)** | **% of     |          | of     |
| ocurement | **% of     |          | total      | **\$ in  | Total  |
| Ob        | total      | **\$ in  | obl        | Mi       | O      |
| ligations | obl        | Mi       | igations** | llions** | bligat |
| (Excludes | igations** | llions** |            |          | ions** |
| Mods.)**  |            |          |            |          |        |
+-----------+------------+----------+------------+----------+--------+
| **Total   | **100%**   | **\$534  | **100%**   | **       |        |
| Obli      |            | M**      |            | \$267M** |        |
| gations** |            |          |            |          |        |
+-----------+------------+----------+------------+----------+--------+
| **Other** | **28%**    | **\$150  | **45%**    | **\$120  | **     |
|           |            | M**      |            | M**      | +61%** |
+-----------+------------+----------+------------+----------+--------+
| **Small** | **40%**    | **\$214  | **46%**    | **\$123  | **     |
|           |            | M**      |            | M**      | +15%** |
+-----------+------------+----------+------------+----------+--------+
| **All     | **32%**    | **\$170  | **9%**     | **\$24   | **     |
| disadv    |            | M**      |            | M**      | -72%** |
| antaged** |            |          |            |          |        |
+-----------+------------+----------+------------+----------+--------+
| -   8(a)  | 12%        | \$63 M   | 2%         | \$6 M    | -83%   |
|           |            |          |            |          |        |
|  Concerns |            |          |            |          |        |
+-----------+------------+----------+------------+----------+--------+
| -   SEDBs | 16%        | \$87 M   | 4%         | \$10 M   | -75%   |
+-----------+------------+----------+------------+----------+--------+
| -   Women | 4%         | \$20 M   | 3%         | \$8 M    | -25%   |
|     Owned |            |          |            |          |        |
+-----------+------------+----------+------------+----------+--------+

In interviews conducted by the evaluation team, industry representatives
stated their perception that the AMS provides fewer opportunities for
SEDB companies. They believe that the agency has lessened its commitment
to inclusion of SEDBs in contracting. It appears that additional effort
is required for the agency to meet the AMS goal of providing reasonable
contracting opportunities to SEDB firms.

On August 19, 1996, the Administrator, issued a small business policy
statement that directed the agency to implement a Mentor-Protégé
program. The program will encourage prime contractors to mentor small
businesses and SEDB firms. Guidance was issued in March, 1997, and no
documentation is yet available for review. It is recommended that the
success of the Mentor-Protégé program be reviewed by future evaluation
efforts.

## Dispute Resolution

The AMS designed a dispute resolution process with flexibility and
timeliness in mind. This goal has not been realized. AMS states that
protests and contract disputes shall be resolved at the agency level
through the FAA dispute resolution system, and judicial review shall
apply only to final agency decisions. The FAA created the Office of
Dispute Resolution (ODR) as an independent internal organization which
reports to the FAA Chief Counsel.

The ODR provides recommendations to the Administrator (or his designee),
who makes a final agency decision concerning the issues at hand. Because
the AMS is not a regulatory document, AMS Section 3.9 is considered
policy. While Section 3.9 does not currently have the force and effect
of law, it does define the process which must be followed by offerors,
potential offerors and contractors in order to exhaust their
administrative remedies before a final agency decision may be obtained.
Only final agency decisions or "orders" are appealable under 49 U.S.C.
46110. At the present time, the policy in Section 3.9 is made binding to
offerors, potential offerors, and contractors through SIRs and contract
clauses. The number of disputes has nearly tripled from pre-AMS figures.

During interviews, contracting officers and legal counsels expressed
concerns regarding the processes and procedures currently being utilized
under the dispute resolution system. Specific policy decisions
addressing concerns as to timeliness of filings, generation and use of
qualified vendor lists, and timeliness of decision issuance in total
were raised. Both procurement officials and user groups have stated that
protests and disputes under AMS are expensive and untimely. It should be
noted that there have been comments received from offerors who like the
new system and feel that their experience with the ODR has been
excellent.

The current process is resource constrained. The ODR does not have an
approved budget and is not permanently or fully staffed. These
deficiencies compound the frustrations, concerns and complaints from
those in the contacting arena. Delays in release and exchange of
information and unavailability of a disputes resolution officer,
contracting officers and legal counsels contributes to delay the
process. Another bottleneck of the process is the availability of the
Administrator, given his schedule. These constraints and delays have
resulted in approximately one third of all disputes presently still
awaiting decisions.

The process remains a challenge for everyone. The ODR expects rulemaking
to be in effect by the end of Fiscal Year 1997. Once rulemaking is in
effect, the Dispute Resolution Process will be regulatory in nature and
will have the force and effect of law. With formal rules and a staffed
and funded office, current restraints and limitations upon the process
will be eliminated and the perception that the system takes too long
should vanish.

# Chapter 7Conclusions and Recommendations {#chapter-7conclusions-and-recommendations .Chapter}

As discussed in Chapter 1, it is difficult to evaluate the effectiveness
of much of the AMS because of the latency involved in most large
procurements and the evolving nature of the processes. This section
provides preliminary information on effectiveness based on early numbers
from the procurement organizations and their databases, analysis of
records and memoranda, and anecdotal data collected through interviews.
Table 7-1 summarizes the evaluation team's assessment of AMS
effectiveness in responding to the problems that prompted acquisition
reform. Table 7-2 shows a similar summary organized by change area and
AMS component. The remainder of this chapter elaborates the evaluation
results and is organized roughly according to the change areas described
in chapter 2.

> Table 7-1. Summary of AMS Effectiveness Against Problems

+-----------+-------------------------------------------+-------------+
| **        |                                           | **Effe      |
| Problem** |                                           | ctiveness** |
+-----------+-------------------------------------------+-------------+
| **M       |                                           |             |
| anagement |                                           |             |
| Control** |                                           |             |
+-----------+-------------------------------------------+-------------+
| Widely    |                                           | ⇑           |
| diffused  |                                           |             |
| respo     |                                           |             |
| nsibility |                                           |             |
| and       |                                           |             |
| accoun    |                                           |             |
| tability. |                                           |             |
+-----------+-------------------------------------------+-------------+
| Poor      |                                           | ⇑           |
| coo       |                                           |             |
| rdination |                                           |             |
| (i        |                                           |             |
| nternal/e |                                           |             |
| xternal). |                                           |             |
+-----------+-------------------------------------------+-------------+
| Poor      |                                           | 6           |
| contract  |                                           |             |
| man       |                                           |             |
| agement/o |                                           |             |
| versight. |                                           |             |
+-----------+-------------------------------------------+-------------+
| **P       |                                           |             |
| lanning** |                                           |             |
+-----------+-------------------------------------------+-------------+
| Efficient |                                           | 6           |
| and       |                                           |             |
| effective |                                           |             |
| planning  |                                           |             |
| p         |                                           |             |
| rocesses. |                                           |             |
+-----------+-------------------------------------------+-------------+
| Unde      |                                           | 6           |
| rstanding |                                           |             |
| customer  |                                           |             |
| needs.    |                                           |             |
+-----------+-------------------------------------------+-------------+
| Mission   |                                           | ⇑           |
| focus,    |                                           |             |
| req       |                                           |             |
| uirements |                                           |             |
| analysis  |                                           |             |
| and       |                                           |             |
| control.  |                                           |             |
+-----------+-------------------------------------------+-------------+
| Ability   |                                           | 6           |
| to        |                                           |             |
| recognize |                                           |             |
| technical |                                           |             |
| co        |                                           |             |
| mplexity. |                                           |             |
+-----------+-------------------------------------------+-------------+
| Ability   |                                           | 6           |
| to        |                                           |             |
| provide   |                                           |             |
| good      |                                           |             |
| e         |                                           |             |
| stimates. |                                           |             |
+-----------+-------------------------------------------+-------------+
| Ability   |                                           | 6           |
| to        |                                           |             |
| control   |                                           |             |
| systems   |                                           |             |
| requ      |                                           |             |
| irements. |                                           |             |
+-----------+-------------------------------------------+-------------+
| **Ac      |                                           |             |
| quisition |                                           |             |
| and       |                                           |             |
| Pr        |                                           |             |
| ocurement |                                           |             |
| Process** |                                           |             |
+-----------+-------------------------------------------+-------------+
| Ability   |                                           | ⇑           |
| to        |                                           |             |
| oblig     |                                           |             |
| ate/spend |                                           |             |
| app       |                                           |             |
| ropriated |                                           |             |
| funding   |                                           |             |
| in timely |                                           |             |
| manner.   |                                           |             |
+-----------+-------------------------------------------+-------------+
| Simpl     |                                           | ⇑           |
| ification |                                           |             |
| of        |                                           |             |
| pr        |                                           |             |
| ocurement |                                           |             |
| and       |                                           |             |
| ac        |                                           |             |
| quisition |                                           |             |
| cycle.    |                                           |             |
+-----------+-------------------------------------------+-------------+
| Ability   |                                           | 6           |
| to field  |                                           |             |
| systems   |                                           |             |
| in timely |                                           |             |
| fashion.  |                                           |             |
+-----------+-------------------------------------------+-------------+
| Ability   |                                           | ⇑           |
| to field  |                                           |             |
| current   |                                           |             |
| t         |                                           |             |
| echnology |                                           |             |
| systems.  |                                           |             |
+-----------+-------------------------------------------+-------------+
| **Human   |                                           |             |
| Re        |                                           |             |
| sources** |                                           |             |
+-----------+-------------------------------------------+-------------+
| Competent |                                           | 6           |
| w         |                                           |             |
| orkforce. |                                           |             |
+-----------+-------------------------------------------+-------------+
| Senior    |                                           | X           |
| m         |                                           |             |
| anagement |                                           |             |
| turnover. |                                           |             |
+-----------+-------------------------------------------+-------------+
| I         |                                           | X           |
| ncentives |                                           |             |
| to        |                                           |             |
| reinforce |                                           |             |
| desired   |                                           |             |
| behavior. |                                           |             |
+-----------+-------------------------------------------+-------------+
| **        |                                           |             |
| Financial |                                           |             |
| Re        |                                           |             |
| sources** |                                           |             |
+-----------+-------------------------------------------+-------------+
| Effective |                                           | ⇑           |
| budget    |                                           |             |
| process.  |                                           |             |
+-----------+-------------------------------------------+-------------+
| *         | ⇑ More effective                          |             |
| **KEY:*** |                                           |             |
|           | ⇓ Less effective                          |             |
|           |                                           |             |
|           | ⇔ No change                               |             |
|           |                                           |             |
|           | 6 Too early to tell                       |             |
|           |                                           |             |
|           | X Not applicable                          |             |
+-----------+-------------------------------------------+-------------+

> [Table 7-2. Evaluation Team Assessment of AMS
> Effectiveness]{.underline}

+--------------+------------------------------+-----------------------+
| **AMS Area** |                              | **Effectiveness**     |
+==============+==============================+=======================+
| *            |                              |                       |
| *Acquisition |                              |                       |
| Lifecycle    |                              |                       |
| Management** |                              |                       |
+--------------+------------------------------+-----------------------+
| Core Policy  |                              | ⇑                     |
+--------------+------------------------------+-----------------------+
| Joint        |                              | ⇑                     |
| Resources    |                              |                       |
| Council      |                              |                       |
+--------------+------------------------------+-----------------------+
| Integrated   |                              | ⇑                     |
| Product      |                              |                       |
| Development  |                              |                       |
| System       |                              |                       |
+--------------+------------------------------+-----------------------+
| Metrics and  |                              | ⇔                     |
| Performance  |                              |                       |
| Measurement  |                              |                       |
+--------------+------------------------------+-----------------------+
| Implementing |                              | 6                     |
| Guidance     |                              |                       |
+--------------+------------------------------+-----------------------+
| Program      |                              | 6                     |
| Baselines    |                              |                       |
+--------------+------------------------------+-----------------------+
| Long-Range   |                              | 6                     |
| Planning     |                              |                       |
+--------------+------------------------------+-----------------------+
| Mission      |                              | 6                     |
| Analysis     |                              |                       |
+--------------+------------------------------+-----------------------+
| Investment   |                              | 6                     |
| Analysis     |                              |                       |
+--------------+------------------------------+-----------------------+
| Baseline     |                              | 6                     |
| Management   |                              |                       |
+--------------+------------------------------+-----------------------+
| Solution     |                              | ⇔                     |
| Im           |                              |                       |
| plementation |                              |                       |
+--------------+------------------------------+-----------------------+
| **P          |                              |                       |
| rocurement** |                              |                       |
+--------------+------------------------------+-----------------------+
| Procurement  |                              | ⇑                     |
| Guidance and |                              |                       |
| Processes    |                              |                       |
+--------------+------------------------------+-----------------------+
| Process      |                              | ⇑                     |
| Flexibility  |                              |                       |
+--------------+------------------------------+-----------------------+
| Dispute      |                              | ⇓                     |
| Resolution   |                              |                       |
+--------------+------------------------------+-----------------------+
| ***KEY:***   | ⇑ More effective             |                       |
|              |                              |                       |
|              | ⇓ Less effective             |                       |
|              |                              |                       |
|              | ⇔ No change                  |                       |
|              |                              |                       |
|              | 6 Too early to tell          |                       |
+--------------+------------------------------+-----------------------+

# Management Control

AMS can enhance management control. The creation of the JRC and IPTs
forms a framework for balancing authority, responsibility, and
accountability. Implementation of control, however, will be difficult
without the consistent management and monitoring of Acquisition Program
Baselines for all FAA capital programs.

# Planning

The new AMS, coupled with the development of the NAS Architecture,
creates an analytical, needs-driven planning infrastructure that should
lead to low risk acquisitions of the right systems. Much depends upon
the capability of the Mission Analysis and Investment Analysis
processes. FAA needs to invest the time and resources for these
processes to mature so that they produce high quality recommendations.

# Acquisition and Procurement Process

The institutional mechanisms are in place to allow rapid procurement and
fielding of new systems. Success will be achieved if the IPTs can
maintain early and continuing involvement of operational organizations
in ensuring new systems meet their needs. IPTs need to act in innovative
ways to take advantage of their new flexibility.

# Financial Resources

The NAS Architecture development and the Capital Investment Plan (CIP)
re-engineering process can improve internal FAA financial planning and
resource control. Nonetheless, external forces (OMB, Congress) have much
greater roles in financial reform. The recommendations to Congress of
the newly-created National Civil Aviation Review Commission (NCARC) will
be crucial in determining how well FAA is able to achieve stable,
predictable future funding.

# Conclusions

An underlying theme that surfaced throughout this evaluation has been
the role of FAA cultural transformation in the success of acquisition
reform. To achieve meaningful reform, FAA needs to have an acquisition
workforce that is creative, bold, and imaginative; a workforce
functioning as an empowered organization, taking advantage of the new
freedoms derived from AMS. At the same time, it must enhance its
attention to disciplined management control. Financial resources are
dwindling and FAA must do more with less.

Since cultural transformation is the central challenge, FAA management
needs to create a sense of urgency around the adoption of AMS and IPDS
principles. They must also bring home the reality of the fiscal
situation. Now that the AMS implementation mechanisms are beginning to
function, the focus must be on harnessing the workforce to a new way of
doing business. Innovation and risk-taking must be encouraged and
rewarded.

# Recommendations

-   Establish a focused, rigorous plan for the IPTs to reduce the time
    to field new systems by 50 percent and reduce acquisition costs by
    20 percent. Establish an appropriate, effective reporting system to
    monitor progress.

-   Obtain the Administrator\'s approval of the revised AMS core policy.

-   Develop an integrated metrics program and educate the workforce on
    its use and value.

-   Establish and enforce a baseline management policy and procedures,
    and expedite the establishment of baselines for legacy programs.

-   Complete the solution implementation guidance for systems/software
    as soon as possible and expedite the remaining processes, guidance,
    and tools.

-   Prioritize programs, functions and processes and reallocate
    resources accordingly, eliminating low-priorities. Centralize
    processes where efficiency or effectiveness gains are possible.

-   Merge the JRC and RMC into a single Assistant and Associate
    Administrator-level body to optimize investment decisions and
    resource allocation across all FAA appropriations. An expanded SEOAT
    could support the merged council.

-   Fully fund and staff the ODR and complete the required processes and
    procedures.

```{=html}
<!-- -->
```
-   Determine the requirements for, and articulate the benefit of SEDB
    contracts and implement guidance accordingly.

# AppendixList of Acronyms {#appendixlist-of-acronyms .Chapter}

ABZ Office of Business Management

AMS Acquisition Management System

APB Acquisition Program Baseline

ARIT Acquisition Reform Implementation Team

ARSR-4 Air Route Surveillance Radar 4

ASD Office of System Architecture and Investment Analysis

ASOS Automated Surface Observing System

ASU Office of Acquisitions

CAASD Center for Advanced Aviation System Design

CIP Capital Investment Plan

CIS Contract Information System

CM Configuration Management

CO Contracting Officer

COTS Commercial Off the Shelf

F&E Facilities and Equipment (budget)

FAA Federal Aviation Administration

FAR Federal Acquisition Regulations

FAST FAA Acquisition System Toolbox

GAO General Accounting Office

GSBCA General Services Board of Appeal

HICIP Human and Intellectual Capital Investment Plan

HRM Human Resource Management

IPDS Integrated Product Development System

IPLT Integrated Product Leadership Team

IPT Integrated Product Team

ITOP Information Technology Omnibus Procurement

JRC Joint Resources Council

Mode S Modes Select Radar

MPPG Major Procurement Preference Goal

NAS National Airspace System

NCARC National Civil Aviation Review Commission

NDI Non-Developmental Item

ODR Office of Dispute Resolution

OMB Office of Management and Budget

OPS Operational (budget)

PMAT Procurement Management Assessment Tool

PT Project/Program Team

R,E&D Research, Engineering and Development (budget)

RMC Resource Management Council

SBA Small Business Administration

SEDB Small Economically Disadvantaged Business

SEOAT System Engineering Operational Analysis Team

SETA System Engineering and Technical Assistance

SIR Screening Information Request

U..S.C. United States Code