Air Traffic Control: FAA Needs to Ensure Better Coordination When Approving Air Traffic Control Systems
Published: 2004-11-17T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Federal Aviation Administration's (FAA) process for ensuring that air traffic control (ATC) systems will operate safely in the national airspace system is an integral part of the agency's multibillion-dollar ATC modernization and safety effort. GAO was asked to review (1) FAA's process for approving ATC systems for safe use in the national airspace system; (2) challenges FAA has faced approving ATC systems and how these challenges affected the cost, schedule, and performance estimates of the systems; and (3) actions FAA has taken to improve its process for approving ATC systems.
section_title: What GAO Found
paragraphs: FAA has separate processes for approving ground systems and certifying aircraft equipment for safe use in the national airspace system. FAA's process for approving ground systems, such as radar systems, is done in accordance with policies and procedures in FAA's Acquisition Management System. Approving ground systems, which are usually developed, owned, and operated by FAA, typically involves FAA's Air Traffic Organization determining whether a vendor is in compliance with contract requirements, followed by a rigorous test-and-evaluation process to ensure that the new system will operate safely in the national airspace system. The process for certifying aircraft equipment, which is usually developed by private companies, is done in accordance with Federal Aviation Regulations, with FAA serving as the regulator. If a system has both ground components and aircraft equipment components, then the system must go through both processes before it is approved for safe use in the national airspace system. FAA has faced challenges approving systems for safe use in the national airspace system that contributed to cost growth, delays, and performance shortfalls in deploying these systems. We identified three specific challenges through the review of 5 ATC systems and our past work. These challenges are the need to (1) involve appropriate stakeholders, such as users and technical experts, throughout the approval process; (2) ensure that the FAA offices that have responsibility for approving ground systems and certifying aircraft equipment effectively coordinate their efforts for integrated systems; and (3) accurately estimate the amount of time needed to meet complex technical requirements at the beginning of the design and development phase. FAA has taken some actions to address two of the three challenges we identified. However, FAA has not taken action to fully involve all stakeholders, such as air traffic controllers and technical experts, throughout the approval process. FAA officials believe that the agency's new Safety Management System will help ensure that the ground system approval and aircraft certification processes are better coordinated. FAA stated that coordination would improve because, as part of the new Safety Management System, the agency plans to realign its organizational structure to create a formal link between the Air Traffic Organization and the Office of Regulation and Certification. FAA expects full implementation of this system to take 3 to 5 years. We are reserving judgment on whether this change will fully address the challenge because of the early state of this effort and FAA's long-standing problems with internal coordination when approving ATC systems. As such, we believe that FAA should, in the interim, develop specific plans that describe how both internal and external coordination will occur on a system-specific basis.

section_title: Letter
paragraphs: The Federal Aviation Administration’s (FAA) process for ensuring that air  traffic control systems will operate safely in the national airspace system is  an integral part of FAA’s multibillion-dollar air traffic control modernization  and safety effort. New air traffic control systems cannot be used in the  national airspace system until FAA has determined that the systems will  operate safely. Over the years, FAA has approved about 45,000 pieces of air  traffic control equipment for safe use in the national airspace system. Some  in the aviation industry and government contend that FAA’s approval  process for air traffic control systems is too lengthy and, therefore,  contributes to cost growth, schedule delays, and performance problems  that have plagued many of the systems that FAA has been trying to develop  for years. In addition, some in the aviation industry have raised concerns  about whether FAA’s approval process has kept pace with changes in  technology. For example, more of today’s new air traffic control systems  are integrated—that is, involving both ground systems and equipment used  exclusively in aircraft (aircraft equipment) that must work together—than  in the past.
In response to your request, we examined   FAA’s process for approving air traffic control systems for safe use in the   challenges FAA faces in approving air traffic control systems and how  these challenges have affected the cost, schedule, and performance of  the systems; and   actions FAA has taken to improve its process for approving air traffic  control systems.
In this report, we use the word “approval” to describe the process of  ensuring the safety of an air traffic control system when it has both a  ground system and aircraft equipment. We also use the word “approval” to  describe the process of ensuring the safety of ground systems exclusively.  We use the word “certification” to describe the process of ensuring the  safety of aircraft equipment for safe use in the national airspace system.
To identify FAA’s process for approving air traffic control systems for safe  use in the national airspace system, we reviewed FAA documents that  describe the agency’s process for approving such systems and equipment  and RTCA’s 1999 and 2001 reports that also address this process. To  determine the challenges FAA has faced in approving air traffic control  systems and how these challenges affected the cost, schedule, and  performance of the systems, we (1) conducted case illustrations on 5 of  FAA’s 25 air traffic control systems currently receiving funding that were  approved or in the process of being approved for safe use in the national  airspace system and (2) reviewed reports prepared by GAO and the  Department of Transportation’s Inspector General. The 5 air traffic control  systems are   Airport Surface Detection Equipment - Model X (ASDE-X),  Controller-Pilot Data Link Communications (CPDLC),  Local Area Augmentation System (LAAS),  Standard Terminal Automation Replacement System (STARS), and  Wide Area Augmentation System (WAAS).
We selected these 5 systems because collectively they accounted for about  46 percent of FAA’s air traffic control modernization costs in fiscal year  2002 and 3 of the 5 systems are integrated—that is, they require the  approval of the ground systems as well as certification of aircraft  equipment before they can be used in the national airspace system. In  addition, we interviewed, among others, officials from FAA program  offices; RTCA; aviation industry groups; manufacturers of aircraft  equipment; ground system developers, including Honeywell, Raytheon, and  Sensis Corporation; industry experts; Wide Area Augmentation System  Integrity Performance Panel and Local Area Augmentation System  Integrity Panel members; and unions representing air traffic controllers  and maintenance technicians. We also reviewed reports on air traffic  control systems prepared by GAO, the Department of Transportation’s  Inspector General, RTCA, and the Commission on the Future of the U.S.  Aerospace Industry (Aerospace Commission). To identify what actions  FAA has taken to improve its processes for approving air traffic control  systems, we interviewed representatives from FAA, RTCA, the Aerospace  Commission, and aviation industry groups. See appendix I for additional  information on our objectives, scope, and methodology. We conducted our  review from October 2003 through September 2004 in accordance with  generally accepted government auditing standards.
subsections: section_title: Background
paragraphs: Several offices within FAA’s Air Traffic Organization and Office of  Regulation and Certification have responsibility for approving ground  systems and certifying aircraft equipment, as shown in figure 1.
Before the creation of the Air Traffic Organization in November 2003, FAA’s  Research and Acquisitions (acquisitions office) and Air Traffic Services  were the primary offices responsible for approving ground systems for safe  use in the national airspace system. The 5 systems that we reviewed began  the approval process under that structure. Currently, these offices,  although renamed, form the core of the Air Traffic Organization. The  responsibilities of Air Traffic Services are now distributed among several  offices, including System Operations Services and Terminal Services. The  responsibilities of Research and Acquisitions are distributed among several  offices, including Technical Operations Services and En Route and Oceanic  Services. In addition, the Air Traffic Organization includes Safety Services,  which is its focal point for safety, quality assurance, and quality control and  is the primary interface with FAA’s Office of Regulation and Certification.
FAA’s Office of Regulation and Certification has responsibility for certifying  and regulating aircraft and its equipment. The following 3 offices within the  Office of Regulation and Certification are involved in the certification of  aircraft equipment:  Aircraft Certification Service (aircraft certification office) is responsible  for administering safety standards for aircraft and aircraft equipment  that are manufactured in the United States.
Flight Standards Service is responsible for granting operational  approval to air carriers that plan to use equipment on their aircraft.
Air Traffic Safety Oversight Service is responsible for monitoring the  safety of air traffic operations through the establishment, approval, and  acceptance of safety standards and the monitoring of safety  performance and trends. It will also improve coordination between the  Office of Regulation and Certification and the Air Traffic Organization.
In addition to the internal FAA stakeholders, the approval of air traffic  control (ATC) systems can also involve a number of other external  stakeholders. FAA generally makes the decision about which other  stakeholders will be involved in approving ATC systems for safe use in the  national airspace system. For example, stakeholders involved in approving  ATC systems may include  manufacturers of aircraft equipment;   users, such as controllers and maintenance technicians.
FAA also regularly requests RTCA, a private, not-for-profit corporation, to  develop consensus-based performance standards for the aircraft  equipment component of ATC systems. RTCA functions as a federal  advisory committee that provides recommendations used by FAA as the  basis for policy, program, and regulatory decisions and by the private  sector as the basis for development, investment, and other business  decisions.
In this report, we focus on the approval of the 5 ATC systems described in  table 1 and further discussed in appendixes II through VI.
section_title: FAA Has Separate Processes for Approving Ground Systems and Certifying Aircraft Equipment
paragraphs: FAA has separate processes for approving ground systems and certifying  aircraft equipment for safe use in the national airspace system. FAA’s  process for approving ground systems, such as radar systems, is done in  accordance with policies and procedures in FAA’s Acquisition Management  System. This process involves a determination by FAA’s Air Traffic  Organization regarding whether a vendor is in compliance with contract  requirements and/or FAA operational requirements, followed by a rigorous  test-and-evaluation process to ensure that the new system will operate  safely in the national airspace system. In contrast, the process for certifying  aircraft equipment, which is usually developed by private companies, is  done in accordance with Federal Aviation Regulations, with FAA serving as  the regulator. If an ATC system has both a ground system and aircraft  equipment, as was the case for 3 of the 5 systems we reviewed, then the  system must go through both processes before it is approved for safe use in  the national airspace system.
subsections: section_title: Ground System Approval Process
paragraphs: The approval of a ground system focuses on safety and is done in  accordance with FAA contract documents and policies and procedures that  are part of the agency’s Acquisition Management System. Most ground  systems that provide air traffic services and air navigation services are  developed, owned, and operated by FAA. Prior to November 2003, FAA’s  Research and Acquisitions and Air Traffic Service offices were responsible  for the approval of ground systems. Currently, FAA’s Air Traffic  Organization has primary responsibility for the approval of ground  systems. FAA’s ground system approval process includes the following six  phases—concept of operations, requirements setting, design and  development, test and evaluation, operational readiness, commissioning— and involves various stakeholders, which are also noted below.
Concept of operations: The ground system approval process begins with  the concept of operations phase. If the system being developed has both  a ground system and aircraft equipment, FAA’s Office of Regulation and  Certification, Air Traffic Services Office, and Acquisitions Office may work together to develop the concept of operations. During this phase,  FAA generally identifies and defines a service or capability to meet a  particular need in the national airspace system and may involve other  stakeholders, such as air traffic controllers. FAA also defines the roles  and responsibilities of key participants, such as controllers and  maintenance technicians, and the key elements of the required  capability. The concept of operations phase is not a static process. As  FAA obtains more information about the system it develops, the  concept is revised to reflect the new information even though the next  phase of the process may have already begun. Potential stakeholders in  this phase include FAA’s Office of Regulation and Certification, FAA’s  Air Traffic Organization, aircraft manufacturers, aviation industry  associations, airlines, air traffic controllers, maintenance technicians,  manufacturers of aircraft equipment, ground system developers, and  representatives of general aviation.
Requirements setting: During the requirements-setting phase, FAA  establishes a minimum set of requirements, including safety objectives,  and specifies how well the new system must perform its intended  functions. For example, it was during this phase that FAA established  WAAS’ and LAAS’ integrity requirement—which is that the system  cannot fail to warn pilots of misleading information that could  potentially create hazardous situations more than once in 10 million  approaches. After analyzing the initial requirements and comparing the  cost, benefits, schedule, and risk of various solutions, FAA sets final  requirements and presents them to the Joint Resources Council as part  of the investment plan. After the council has approved the requirements  for the new system, FAA will issue a request for proposals, evaluate the  offers received, and select a contractor to design a system based on the  requirements set by FAA. Potential stakeholders in this phase include  FAA’s Office of Regulation and Certification, FAA’s Air Traffic  Organization, aircraft manufacturers, aviation industry associations,  airlines, air traffic controllers, maintenance technicians, manufacturers  of aircraft equipment, ground system developers, and representatives of  general aviation.
Design and development: The design and development of ground  systems is generally completed by a contractor and monitored by FAA.  During this phase, the contractor conducts preliminary and critical  design reviews, which include plans for how it will conduct the testing  phase. FAA must approve these plans before the contractor can proceed  to the next phase. Potential stakeholders in this phase include FAA,  ground system developers, air traffic controllers, and maintenance  technicians.
Test and evaluation: After FAA has approved the design and  development of the system, it is ready to be tested and evaluated. The  testing and evaluation of ground systems typically includes three major  tests: development tests, operational tests, and an independent  operational test and evaluation. Development testing is performed by  the contractor to verify compliance with contractual requirements and  is overseen by FAA. Operational testing is performed by FAA and is  designed to demonstrate that a new system is operationally effective  and suitable for use in the national airspace system. An independent  operational test and evaluation is a full system-level evaluation  conducted by FAA in an operational environment to confirm the  operational readiness of a system to be part of the national airspace  system. Potential stakeholders in this phase include FAA, ground system  developers, air traffic controllers, and maintenance technicians.
Operational readiness: During the operational readiness phase, FAA  personnel are trained to operate and maintain the new system, usually in  conjunction with its predecessor system. Following operational  readiness approval, the system is ready to be commissioned. Potential  stakeholders in this phase include FAA, ground system developers, air  traffic controllers, and maintenance technicians.
Commissioning: The commissioning phase ensures that the new  ground system as installed meets the intended mission and operational  requirements and is fully supported by the national airspace system  infrastructure. Potential stakeholders in this phase include FAA, ground  system developers, air traffic controllers, and maintenance technicians.
section_title: Aircraft Equipment Certification Process
paragraphs: In contrast to the ground system approval process, certification of aircraft  equipment is done in accordance with procedures outlined in the Federal  Aviation Regulations, Title 14, Code of Federal Regulations, Part 21. Under  Title 49, Section 44704, of the U.S. Code, FAA has the authority to issue type  certificates, supplemental type certificates, and production certificates,  among others, for aircraft and equipment that will be used in the national  airspace system. Unlike the approval of ground systems, which FAA  accomplishes with the help of a contractor, FAA is the regulator of aircraft  equipment and is not typically involved in the development of the  equipment. An applicant, such as a manufacturer of aircraft equipment,  generally brings fully developed aircraft equipment to FAA for certification.  The aircraft equipment certification process includes the following five  phases—concept of operations, requirements setting, design and  production approval, installation approval, and operational approval—and  involves several stakeholders, which are also noted below:   Concept of operations: Like the ground system approval process, the  aircraft equipment certification process generally begins with the  concept of operations phase, when the aircraft equipment is part of an  ATC system. If the aircraft equipment certification process is not  associated with the approval of a new ground system, then the  certification process may begin with an idea for better equipment.  During this phase, FAA, sometimes with the help of industry, identifies  and defines a service or capability to meet a particular need in the  national airspace system. Potential stakeholders in this phase include  FAA’s Office of Regulation and Certification, FAA’s Air Traffic  Organization, aircraft manufacturers, aviation industry associations,  airlines, air traffic controllers, maintenance technicians, manufacturers  of aircraft equipment, ground system developers, and representatives of  general aviation.
Requirements setting: Once FAA has identified the need for a new  system with aircraft equipment, FAA determines the requirements for the aircraft equipment. In some cases, the requirements for aircraft  equipment may already exist in the Federal Aviation Regulations. In  other cases, FAA may ask RTCA to develop the requirements, including  safety requirements, which are referred to as minimum operating  performance standards. RTCA typically takes 1 to 5 years to develop  the standards because of the need to reach consensus between FAA  and the industry and the increasing complexity of systems being  developed today. According to a RTCA official, the time required to  develop recommended standards is a function of many variables,  including urgency of the situation and the commitment and availability  of government and industry volunteers to collaboratively develop the  standards. For example, in the case of WAAS, RTCA began setting  performance standards in 1994, completed the original version of the  standards in January 1996, and completed the most recent version of  WAAS performance standards in November 2001. Potential  stakeholders in this phase include FAA’s Office of Regulation and  Certification, FAA’s Air Traffic Organization, aircraft manufacturers,  aviation industry associations, airlines, air traffic controllers,  maintenance technicians, manufacturers of aircraft equipment, ground  system developers, and representatives of general aviation.
Design and production approval: The requirements/performance  standards, most often developed by RTCA, typically form the basis for a  technical standard order, which FAA uses to grant design and  production approval for most new aircraft equipment developed in  support of national airspace system modernization efforts. Technical  standard orders are FAA’s requirements for materials, parts, processes,  and appliances used on civil aircraft. Most aircraft manufacturers want  technical standard orders because they make installation approval  simpler and less costly and allow for operation in any type of aircraft.  Technical standard orders are issued for items ranging from safety belts  to navigation equipment. If the applicant successfully completes the  design and production approval phase, FAA provides the applicant with  a technical standard order authorization letter, which states that the  applicant has met a specific technical standard order and the product is  now ready for the installation approval phase. Potential stakeholders in  this phase include FAA’s Aircraft Certification Service, manufacturers of  aircraft equipment, and aircraft manufacturers.
Installation approval: After receiving a technical standard order  authorization for new aircraft equipment, the initial applicant must  receive installation approval from FAA before the aircraft equipment  may be used in the national airspace system. To receive installation  approval, the applicant submits a certification plan and test plan to one  of FAA’s aircraft certification offices for review and approval. In  addition, the applicant conducts ground and flight tests under FAA’s  supervision to ensure that the new equipment operates properly upon  installation. Once the tests are completed to FAA’s satisfaction, FAA  issues a supplemental type certificate, which is evidence of FAA’s  approval to modify an aircraft from its original design. Potential  stakeholders in this phase include FAA’s Aircraft Certification Service,  manufacturers of aircraft equipment, and aircraft manufacturers.
Operational approval: Finally, for the aircraft equipment to become  certified for use in the national airspace system by air carrier operators,  operational approval is also needed from FAA. To obtain operational  approval, the applicant must successfully demonstrate, among other  things, that the pilots are properly trained to use the aircraft equipment  and that maintenance personnel are properly trained to maintain the  equipment. Potential stakeholders in this phase include FAA’s Flight  Standards Service, airlines, and representatives of general aviation.
section_title: FAA Faced Challenges in Approving Several ATC Systems
paragraphs: FAA faced challenges in approving systems for safe use in the national  airspace system that contributed to cost growth, delays, and performance  shortfalls in deploying these systems. We identified three specific challenges through the review of 5 ATC systems and our past work. These  challenges are the need to  involve appropriate stakeholders, such as users and technical experts,  throughout the approval process;   ensure that the FAA offices that have responsibility for approving  ground systems and certifying aircraft equipment effectively coordinate  their efforts for integrated systems; and  accurately estimate the amount of time needed to meet complex  technical requirements at the beginning of the design and development  phase.
Although most of the challenges we found relate to the ground system  approval process, RTCA and the Aerospace Commission have identified  challenges with FAA’s aircraft equipment certification process. For  example, RTCA found that there was a need for better internal FAA  communication and coordination, including the establishment of an  organizational focal point to provide coordinated responses to all matters  related to ground systems and aircraft equipment. In addition, the  Aerospace Commission found that FAA’s regulatory process needs to be  streamlined to enable the timely development of regulations needed to  address new technologies.
subsections: section_title: FAA Did Not Always Adequately Involve Appropriate Stakeholders, Such as Users and Technical Experts, Throughout Its Approval Process
paragraphs: FAA failed to adequately involve appropriate stakeholders, such as air  traffic controllers and maintenance technicians, for 3 of the 5 systems we  reviewed. For example, FAA did not adequately involve controllers and  maintenance technicians throughout the approval process of STARS, which  will replace controller workstations with new color displays, processors,  and computer software. Although controllers and technicians were  involved in developing requirements for STARS in 1994 prior to the 1996  contract award to Raytheon, the original approved acquisition plan  provided for only limited human factors evaluation by controllers and  technicians during STARS’ design and development because the aggressive  development schedule limited the amount of time available to involve  them. Consequently, FAA and Raytheon had to restructure the contract to  address controllers’ concerns that were identified later, such as the  inconsistency of visual warning alarms and color codes with the new  system. According to FAA officials, not involving controllers and  maintenance technicians in the design phase caused the agency to revise its  strategy for acquiring and approving STARS, which contributed to STARS’  overall cost growth of $500 million and added 3 years to the schedule.
FAA also did not always sufficiently involve technical experts early in its  approval process for 2 additional systems that we reviewed. For example,  FAA did not obtain technical expertise on how to resolve the integrity  requirement of WAAS, a navigation system for aviation that augments the  Global Positioning System (GPS), until late in the design and development  phase. FAA acknowledges that the agency’s in-house technical expertise  was not sufficient to address the technical challenges of WAAS. Initially,  FAA and the contractor believed they could meet the WAAS integrity  requirement to alert the pilot in a timely manner when the system should  not be used. However, although WAAS was being developed by an  integrated product team that included representatives from several FAA  offices, the team did not function effectively in resolving issues related to  meeting an important functional requirement to alert the pilot in a timely  manner when the system should not be used because of a possible error.  According to FAA officials, the reason coordination did not occur was that  the two offices had competing priorities that were not associated with  WAAS’ development. Consequently, in 2000, FAA convened the WAAS  Integrity Performance Panel to help it meet the integrity requirement. The  WAAS Integrity Panel worked for about 2-1/2 years before it came up with a  solution to the integrity requirement. In addition, in August 2000, the  agency established an Independent Review Board, which is independent of  the panel and included experts in satellite navigation and safety  certification, to oversee the panel and evaluate the soundness of its efforts.
According to a member of the WAAS Integrity Panel, if FAA had involved  these technical groups immediately after the contract was awarded to  Raytheon in 1996, these groups could have started devising a solution in  1996, rather than in 2000. This lack of technical expertise contributed to a  6-year delay in WAAS’ commissioning and a $1.5 billion increase in its  development costs from the 1994 baseline.
FAA also did not fully engage technical experts early in the approval  process of LAAS, a precision approach and landing system that will  augment GPS. According to FAA officials, meeting the LAAS integrity  requirement to alert the pilot in a timely manner when the system should  not be used is perhaps the most difficult part of approving this system for  safe use in the national airspace system. According to the Department of  Transportation’s Inspector General, although FAA had a LAAS Integrity  Panel in place since 1996 to assist with its research and development  activities, the panel was not formally tasked with resolving LAAS’ integrity  issues. According to one satellite navigation expert and the Department of  Transportation’s Inspector General, focusing the LAAS Integrity Panel on  resolving the integrity requirement early in the approval process may have  enabled FAA to develop a quicker solution. In 2003, FAA focused the  LAAS Integrity Panel on developing a solution to meet the integrity  requirement. However, FAA and another satellite expert maintain that the  technical complexity of this problem is the main reason that LAAS is not  commissioned. According to FAA officials, the need to validate integrity  requirements and further software development has resulted in FAA  placing LAAS in its research and development program and suspending  funding for fiscal year 2005.
In contrast, FAA faced fewer schedule and cost problems in approving  ASDE-X for use in the national airspace system. This was, in part, because  FAA included stakeholders early and throughout the approval process and  because program managers had strong technical expertise. The ASDE-X  program office brought in stakeholders, including maintenance technicians  and air traffic controllers, during the concept of operations phase and  continued to involve them during requirements setting, design and  development, and test and evaluation. FAA also brought ASDE-X  stakeholders together at technical meetings to provide input on ASDE-X  design and development, which allowed the ASDE-X program office to  design a system that met requirements and incorporated stakeholders’  needs. By obtaining the input of controllers and technicians at the  beginning of the approval process, FAA was able to ensure that ASDE-X  requirements were set at appropriate levels and not overspecified or  underspecified. Some stakeholders commented that the program  managers’ strong technical expertise was one reason that ASDE-X’s  requirements were set appropriately. As a result, this system was initially  commissioned only 5 months behind schedule and its cost increased  moderately from $424 million to $510 million.
section_title: FAA Did Not Always Effectively Coordinate Its Certification and Approval Processes
paragraphs: FAA did not always effectively coordinate its certification and approval  processes for CPDLC, WAAS, and LAAS. Coordination between FAA’s  offices responsible for approval of ground systems and certification of  aircraft equipment is becoming increasingly important given that more and  more ATC systems have both ground systems and aircraft equipment.  However, we found that coordination was not effective on CPDLC Build  1A, which allows pilots and controllers to transmit digital data messages  directly between FAA ground automation systems and suitably equipped  aircraft. In the interest of meeting the original cost and schedule  estimates, FAA awarded the contract before it had a full understanding of  system requirements. Requirements that specify how the ground system  and aircraft equipment would operate together were not yet completed  prior to award of the Build 1A contract. Consequently, changes needed to  be made after the contract was awarded. New hardware requirements,  software requirements, and other system requirement changes were added,  which increased CPDLC’s costs by $41 million, almost 61 percent of the  total cost increases associated with CPDLC.
The lack of effective coordination among FAA offices responsible for  approving WAAS also contributed to delays and increased costs in  commissioning WAAS. Although WAAS was being developed by an  integrated product team that included representatives from various FAA  offices, the team did not function effectively in resolving issues related to  meeting an important functional requirement to alert the pilot in a timely  manner when the system should not be used because of a possible error.  According to FAA officials, the reason coordination was not effective was  because the two offices had competing priorities that were not associated  with development of WAAS. Consequently, it was not until September 1999,  when the aircraft certification office became fully involved, that FAA  recognized that its solution to meet WAAS’ integrity requirement was not  sufficient and that it did not have the technical expertise needed to develop  a solution. This lack of coordination contributed to a 6-year delay in WAAS’  commissioning and a $1.5 billion increase in its development costs.
LAAS is another example of how FAA did not effectively coordinate its  efforts. For example, FAA’s Office of Regulation and Certification  completed the design and production approval of LAAS aircraft equipment  without effectively coordinating with the offices responsible for acquisition  to determine the consequences of certifying aircraft equipment before  approval of the associated ground system. According to an FAA official,  once the Office of Regulation and Certification has given design and  production approval to the LAAS aircraft equipment, it is not possible to  make a change to the requirements for the aircraft equipment so that they  are better integrated with the associated LAAS ground system.  Consequently, LAAS ground system developers may have to make more  costly and time-consuming changes to the ground system than would have  been necessary if the Office of Regulation and Certification and  acquisitions offices had coordinated their efforts.
section_title: FAA Did Not Always Prepare Accurate Estimates of the Amount of Time Needed to Meet Complex Technical Requirements
paragraphs: We have reported in the past that when FAA attempts to combine different  phases of system development in an effort to more quickly implement the  systems to meet milestones, it repeatedly experiences major performance  shortfalls and rework, which leads to schedule delays and cost increases.  We found that WAAS, STARS, and LAAS all experienced delays and cost  increases in part because FAA did not prepare accurate estimates of the  amount of time needed to meet complex technical requirements, leading to  an accelerated schedule that sometimes failed to include activities such as  human factors evaluations and technical expert consultations. For  example, in 1994, in response to the concerns of government and aviation  groups, FAA accelerated implementation of WAAS milestones from 2000 to  1997. FAA planned to develop, test, and deploy WAAS within 28 months, an  unrealistic goal given that software development alone was expected to  take 24 to 28 months. It was not until July 2003, over 6 years later, that FAA  was able to commission WAAS for initial operating capability. The  accelerated schedule contributed to the 6-year delay in the commissioning  of the system because the schedule itself was unrealistic and additional  design work needed to be completed. During that time, the cost to develop  the system increased about $1.5 billion, and the system has yet to meet its  original performance goal of providing pilots with the ability to navigate  down to 200 feet during their approach to the runway.
FAA also accelerated the schedule for STARS in 1995. FAA’s approach to  commissioning STARS was oriented to rapid deployment to meet critical  needs for new equipment. To meet these needs, FAA compressed its  original development and testing schedule from 32 to 25 months.  Consequently, this acceleration in schedule left only limited time for human  factors evaluations and, according to FAA officials, contributed to STARS’  overall cost growth of $500 million and added 3 years to the first  deployment because the agency had to revise its strategy for acquiring and  approving STARS.
Although FAA had not developed a solution for meeting the integrity  requirement, FAA also accelerated the LAAS schedule in 1999 by setting  system milestones before completely designing the system. FAA originally  planned to deploy LAAS in 2002 but has since moved it to fiscal year 2009  because the system’s software development is not complete and a solution  for meeting LAAS’ integrity requirements has yet to be developed.
section_title: RTCA and the Aerospace Commission Found Challenges with FAA’s Process for Approving Ground Systems and Certifying Aircraft Equipment
paragraphs: RTCA and the Aerospace Commission also identified challenges with FAA’s  process for approving ground systems and certifying aircraft equipment. In  1998, at the request of the FAA Administrator, RTCA reviewed FAA’s  certification/approval process to determine if it could be made more  responsive to the changing state of aviation, including its more integrated  technologies. RTCA found that FAA’s ground system approval process and  aircraft equipment certification process took too long and cost too much,  and RTCA made several recommendations to improve the processes. For  example, in 2001, RTCA recommended that FAA implement a coordinated  approval process that, among other things, would ensure that all  stakeholders, including those outside FAA’s program offices, participate in  all phases of the approval process. Specifically, similar to our finding that  the FAA offices that had responsibility for approving ground systems and  certifying aircraft equipment did not always effectively coordinate their  efforts, RTCA found that there was a need for better internal FAA  communication and coordination, including the establishment of an  organizational focal point to provide coordinated responses to all matters  related to ground systems and aircraft equipment. RTCA also found that  there was a need for an earlier and better exchange of information between  FAA and those involved in the approval and certification processes from  outside FAA, such as manufacturers of aircraft equipment.
In 2000, Congress asked the Commission on the Future of the U.S.  Aerospace Industry to study the health of the aerospace industry and  identify actions that the United States needs to take to ensure the industry’s  health. As part of this study, the Aerospace Commission reviewed FAA’s  certification process for aircraft equipment and made recommendations.  The Aerospace Commission found that FAA’s certification of new aircraft  technologies has become uncertain in terms of time and cost and  recommended that FAA’s regulatory process be streamlined to enable the  timely development of regulations needed to address new technologies.  According to the Aerospace Commission, instead of focusing on rules and  regulations that dictate the design and approval of equipment, FAA should  focus on certifying that manufacturing organizations have safety built into  their processes for designing, testing, and ensuring the performance of an  overall system. The commission believed that such an approach would  allow FAA personnel to better keep up with technological progress by  becoming less design-specific and more safety-focused.
section_title: FAA Has Taken Action to Improve Its Process for Approving ATC Systems
paragraphs: FAA has taken action to address two of the three management challenges  that we identified. However, FAA has not taken action to ensure that all  stakeholders, such as air traffic controllers, maintenance technicians,  technical experts, and industry representatives, are involved throughout  the ground system approval process. FAA has also taken some action to  address recommendations made by RTCA and the Aerospace Commission.  Examples of some of the actions FAA has taken that address the  management challenges that we found as well as RTCA and Aerospace  Commission recommendations are discussed below:  Coordinating FAA’s acquisitions offices and Office of Regulation and  Certification efforts for approving systems with ground and aircraft  components: FAA officials believe that the agency’s new Safety  Management System, which is designed to formalize the agency’s safety  process, will also improve coordination among FAA internal  stakeholders once it is implemented. FAA stated that coordination  would improve because as part of the new Safety Management System  the agency plans to realign its organizational structure to create a formal  link between the Air Traffic Organization and the Office of Regulation  and Certification. Within the Office of Regulation and Certification,  there is the newly created Air Traffic Safety Oversight Service, which  oversees the safety operations of the Air Traffic Organization and  collaborates with the Air Traffic Organization’s Safety Services. In  addition, according to FAA officials, both ground systems and aircraft  equipment will be more consistently assessed for their effect on safety  as safety terminology is standardized. FAA expects full implementation  to take 3 to 5 years. We are reserving judgment on whether this change  will fully address the challenge because of the early state of this effort  and because FAA’s problems with internal coordination when approving  ATC systems are long-standing. In addition, because FAA has  historically faced internal and external coordination challenges in  approving ATC systems for safe use in the national airspace, we believe  that as FAA moves forward with the agency’s new Safety Management  System, it should, in the interim, develop plans that describe how both  internal and external coordination will occur on a system-specific basis.  In addition, plans to include external stakeholders are particularly  important since the Safety Management System is not intended to  address this challenge.
Estimating the amount of time needed to meet complex technical  requirements: During the development of WAAS and STARS, FAA  adopted an incremental approach to developing and testing these  systems to get them back on track, which is referred to as the “build a  little, test a little” or spiral development approach. For example, to get  WAAS back on track, FAA decided to take a more incremental approach  to implementing the new navigation system—focusing more on the  successful completion of research and development before starting  system approval. In particular, FAA allowed time for collecting and  evaluating data on key system performance requirements like the WAAS  integrity requirement before moving forward. FAA officials  acknowledged that the manner in which FAA decided to implement  WAAS development before implementing this incremental approach  was a high-risk approach and was a primary issue underlying the  system’s problems. Some aviation stakeholders believe this approach is  advantageous because, although it can increase costs initially, money  can be saved in the long run because the approach may help to avoid  mistakes that are very costly to fix once a system has been developed.  This approach also helps to ensure that the necessary building blocks of  a system are tested along the way through the early and ongoing  involvement of key stakeholders, those who will use and maintain the  system. These stakeholders are key to identifying critical omissions and  issues that could prevent a system from operating as intended.
As previously discussed, RTCA and the Aerospace Commission reviewed  FAA’s approval process and made a number of recommendations to  improve it. FAA has taken some action to address these recommendations.  For example:  In response to RTCA’s recommendation to implement a process in  which the regulators and applicants come to an early and clear  agreement on their respective roles, responsibilities, expectations,  schedules, and standards to be used in certification projects, FAA issued  The FAA and Industry Guide to Avionics Approval in 2001, which is  intended to help FAA reduce the time and cost for the certification of  aircraft equipment. This guide describes how to plan, manage, and  document an effective, efficient aircraft equipment certification process  and how to develop a working relationship between FAA and the  applicant. In addition, as part of the 1999 FAA and Industry Guide to  Product Certification, FAA encourages the manufacturers of aircraft  equipment to develop a Partnership for Safety Plan that defines roles  and responsibilities, describes how the certification process will be  conducted, and identifies the milestones for completing the  certification. A WAAS aircraft equipment manufacturer said that the  certification of the WAAS aircraft equipment it developed went  smoothly, primarily because of this up-front agreement with FAA.  Although FAA’s actions address the aircraft equipment certification  process, it does not have a similar process for its ground system  approval process.
In response to RTCA’s recommendation to establish an organizational  focal point to provide one-stop service to users, industry, and other  governments in all matters related to advanced ground electronics and  aircraft equipment, FAA has completed a Web site that provides a broad  range of information on the certification process for aircraft equipment.  However, there is still no focal point to which industry can address  questions about the approval process and be assured of getting a fully  coordinated FAA answer.
In response to the Aerospace Commission’s recommendation to  streamline its aircraft equipment certification process to ensure timely  development of regulations needed to address new technologies and to  focus on certifying that manufacturing organizations have built safety  into their processes for designing, testing, and ensuring the performance  of an overall system, FAA proposed creating an Organizational  Designation Authorization program in January 2004. The program would  expand the approval functions of FAA organizational designees,  standardize these functions to increase efficiency, and expand eligibility  for organizational designees.
section_title: Conclusions
paragraphs: FAA did not always include stakeholders throughout the process for  approving ATC systems for safe use in the national airspace system.  Including stakeholders is particularly important because the new ATC  systems are more integrated today than in the past and thus require more  coordination among all the stakeholders, particularly FAA’s Office of  Regulation and Certification and the recently created Air Traffic  Organization, but also between FAA and other stakeholders, such as  technical experts, controllers, and maintenance technicians. When  decisions regarding integrated ATC systems are made in isolation, they may  contribute to the ineffective use of resources and time. We found that 3 of  the 5 ATC systems we reviewed experienced cost growth and schedule  delays, in part, because FAA did not always involve all necessary  stakeholders, such as controllers and technical experts, throughout the  approval process. In 2001, RTCA recommended that FAA implement a  coordinated approval process that, among other things, would ensure that  all stakeholders, including those outside FAA’s program offices, participate  in all phases of the approval process. We agree with RTCA’s  recommendation, which FAA has not fully implemented, and believe that  fully implementing it would help address some of the challenges we found  with FAA’s approval and certification processes.
In addition, although FAA’s new Safety Management System and the  planned alignment between FAA’s Air Traffic Organization and Office of  Regulation and Certification have the potential to improve FAA’s internal  coordination, FAA has just begun implementing these initiatives with full  implementation 3 to 5 years away. FAA also has historically faced internal  coordination challenges in approving ATC systems for safe use in the  national airspace system as we found for each of the 3 integrated systems  that we reviewed. We believe that the implementation of the Safety  Management System, coupled with the new formal link between FAA’s Air  Traffic Organization and Office of Regulation and Certification, will give  FAA the opportunity to improve its internal coordination among its offices  that are responsible for ground system approval and aircraft equipment  certification. However, the system will not be implemented until 3 to 5  years. Therefore, because of FAA’s history of internal and external  coordination challenges, such as the lack of effective coordination between  FAA offices responsible for approving WAAS, which contributed to WAAS’  cost increase of about $1.5 billion and schedule delays of 6 years, we  believe that specific plans for improving coordination both internally and  externally on a system-specific basis are needed now.
section_title: Recommendation for Executive Action
paragraphs: To ensure that key stakeholders, such as air traffic controllers,  maintenance technicians, and technical experts, outside FAA’s acquisitions  offices and Office of Regulation and Certification, are involved early and  throughout FAA’s ground system approval process and to ensure better  internal coordination between FAA’s offices responsible for approving  ground systems and certifying aircraft equipment, we recommend that the  Secretary of Transportation direct the Administrator of FAA to develop  ATC system-specific plans early in the approval process that specify how  and when the approving and certifying offices within FAA and other  stakeholders, including controllers, maintenance technicians, technical  experts, and industry representatives, will meet to ensure coordination.
section_title: Agency Comments
paragraphs: We provided a draft of this report to the Secretary of Transportation for  review and comment. FAA generally agreed with our findings and  recommendation and provided technical corrections, which we  incorporated as appropriate. FAA also commented that it has started to  take actions to improve its coordination efforts for integrated ATC systems.
We are sending copies of this report to interested congressional  committees, the Secretary of Transportation, and the FAA Administrator.  We will also make copies available to others on request. In addition, the  report will be available at no charge on the GAO Web site at  http://www.gao.gov. Should you or your staff have questions on matters  discussed in this report, please contact me on (202) 512-2834 or at  siggerudk@gao.gov. GAO contacts and key contributors to this report are  listed in appendix VII.
section_title: Objectives, Scope, and Methodology
paragraphs: To complete our first objective, to describe FAA’s process for approving air  traffic control (ATC) systems for safe use in the national airspace system,  we obtained and analyzed documents from the Federal Aviation  Administration (FAA) and RTCA’s 1999 report that discussed FAA’s  process for certifying aircraft equipment and approving ground systems.  We also interviewed FAA officials, contractors, industry experts, and  unions representing air traffic controllers and maintenance technicians  that are involved in approving ATC systems.
To complete our second objective, to describe the challenges FAA has  faced approving ATC systems and how those challenges affected the cost,  schedule, and performance estimates of the systems, we conducted case  illustrations on 5 of FAA’s 25 air traffic control systems that are currently  receiving funding:   Airport Surface Detection Equipment - Model X (ASDE-X),   Controller-Pilot Data Link Communications (CPDLC),  Local Area Augmentation System (LAAS),  Standard Terminal Automation Replacement System (STARS), and   Wide Area Augmentation System (WAAS).
We selected these 5 systems because collectively they accounted for about  46 percent of FAA’s ATC modernization costs in fiscal year 2002 and 3 of the  5 systems are integrated—that is, they require the approval of the ground  systems as well as aircraft equipment. To select the 5 case illustration  systems, we used FAA’s capital investment project data file. We met with  knowledgeable FAA officials to discuss issues related to the accuracy and  completeness of the data file, which was deemed adequate for the purpose  of our work. We also met with knowledgeable FAA officials to determine  the number of ATC systems from the data file that needed to be approved  before entry into the national airspace system. For each of the case  illustrations, we reviewed FAA documents, including acquisition program  baseline reports, Joint Resource Council decisions, and briefing  documents. We also reviewed GAO and Department of Transportation’s  Inspector General reports and testimonies. In addition, we interviewed  officials from FAA program offices; RTCA; the General Aviation  Manufacturers Association; the Air Transport Association; the Aircraft  Owners and Pilots Association; NavCanada; Transport Canada; the MITRE  Corporation; Boeing; Garmin; Rockwell Collins; contractors, including  Honeywell, Raytheon, and the Sensis Corporation; industry experts; the  WAAS Integrity Performance Panel; the LAAS Integrity Panel members; and  unions representing air traffic controllers and maintenance technicians.
To compete our third objective, to describe actions FAA has taken to  improve its processes for approving ATC systems, we interviewed  representatives from FAA; RTCA; the Commission on the Future of the U.S.  Aerospace Industry; aviation industry groups, including the General  Aviation Manufacturers Association, the Air Transport Association, and the  Aircraft Owners and Pilots Association; manufacturers of aircraft  equipment, including Garmin and Rockwell Collins; Boeing; and  contractors, including Honeywell, Raytheon, and the Sensis Corporation;  industry experts; and unions representing air traffic controllers and  maintenance technicians.
We conducted our review in Washington, D.C., from October 2003 through  September 2004 in accordance with generally accepted government  auditing standards.
section_title: Airport Surface Detection Equipment - Model X Case Illustration
subsections: section_title: Background
paragraphs: ASDE-X is an airport surface surveillance system that air traffic controllers  use to track aircraft and vehicle surface movements. (See fig. 2.) ASDE-X  uses a combination of surface movement primary radar and  multilateration sensors to display aircraft position and vehicle position on  an ATC tower display. According to FAA, the integration of these sensors  provides accurate, up-to-date, and reliable data for improving airport safety  in all weather conditions. ASDE-X was developed to prevent accidents  resulting from runway incursions, which have increased since 1993. The  number of reported runway incursions rose from 186 in 1993 to 383 in 2001.  According to FAA, because air traffic in the United States is expected to  double by 2010, runway incursions may pose a significant safety threat to  U.S. aviation.
FAA expects that ASDE-X will increase the level of safety at airports and  provide air traffic controllers with detailed information about aircraft  locations and movement at night and in bad weather due to the (1)  association of flight plan information with aircraft position on controller  displays; (2) continuous surveillance coverage of the airport from arrival  through departure; (3) elimination of blind spots and coverage gaps; and  (4) availability of surveillance data with an accuracy and update rate  suitable for, among other things, awareness in all weather conditions.
section_title: Status
paragraphs: In October 2003, FAA commissioned ASDE-X at Mitchell International  Airport in Milwaukee, Wisconsin, for use in the national airspace system.  ASDE-X came in close to its original schedule and cost baselines. The  ASDE-X system was approximately 5 months over its original schedule  baseline, but maintained its original performance baselines. In June 2002,  FAA approved $80.9 million in additional funding to add ASDE-X at 7  additional sites. (See table 2.) FAA is currently scheduled to deploy ASDE- X at 25 U.S. airports over the next 4 years and to update existing surface  detection systems (i.e., ASDE-3) at 9 other facilities. FAA plans to introduce  an upgraded ASDE-X system at T.F. Green Airport in Providence, Rhode  Island, with deployment tentatively slated for the 4th quarter of 2004. FAA  is also investigating whether to add ASDE-X at 25 airports that use ASDE-3  and Airport Movement Area Safety Systems.
section_title: FAA Faced Fewer Challenges in Approving ASDE-X
paragraphs: Of the five systems we reviewed, FAA faced fewer schedule and cost  challenges in approving ASDE-X for safe use in the national airspace  system. This is partly because FAA included stakeholders early and  throughout the approval process and because of the strong technical  expertise of its managers. The ASDE-X program office brought in  stakeholders, including maintenance technicians and air traffic controllers,  beginning with the concept of operations phase and continued their  stakeholder involvement through the requirements-setting, design-and- development, and test-and-evaluation phases and then continued  involvement throughout the deployment phase. For example, FAA obtained  the input of controllers and technicians at the beginning of the approval  process, which helped to ensure that ASDE-X requirements were set at  appropriate levels and not overspecified or underspecified. Stakeholders  pointed toward the strong technical expertise of the program’s managers as  a reason for the appropriate specification of ASDE-X’s requirements. In  addition, FAA brought ASDE-X stakeholders together at technical meetings  to provide input on ASDE-X design and development, which allowed the  ASDE-X program office to design a system that met requirements and  incorporated stakeholders’ needs.
However, FAA did experience some challenges in approving ASDE-X. In  response to Congress’ desire to deploy the system quickly, FAA attempted  to accelerate ASDE-X’s approval. However, FAA experienced problems in  accelerating the approval when it awarded the contract before all  requirements had been finalized.
Table 3 shows the major phases and time frames associated with the ASDE- X approval process.
section_title: Controller-Pilot Data Link Communications Case Illustration
subsections: section_title: Background
paragraphs: CPDLC will allow pilots and controllers to transmit digital data messages  directly between FAA ground automation computers and suitably equipped  aircraft. (See fig. 3.) CPDLC is a new way for controllers and pilots to  communicate that is analogous to e-mail. The pilot can read the message  displayed on a screen in the cockpit and respond to the message with the  push of a key. In the future, this will alleviate frequency congestion  problems and increase controller efficiency. One of the most important  aspects of this technology is its intended reduction of operational errors  from misunderstood instructions and readback errors. The initial phase  (Build 1) consisted of four services: initial contact, altimeter setting,  transfer of communication, and predefined instructions via menu text. The  CPDLC program will ultimately develop additional capabilities in an  incremental manner through further development stages. Originally, Build  1 was to be followed by Build 1A, which was designed to increase the  CPDLC message set and include assignment of speeds, headings, and  altitudes as well as a route clearance function.
section_title: Status
paragraphs: CPDLC was commissioned for initial daily use by controllers at Miami on  October 7, 2002. This completed the stage called Build 1, which included  four services. American Airlines is the CPDLC launch airline with about 25  aircraft operating in the Miami Center airspace. Further deployment of  CPDLC has been deferred until about 2009 after the Joint Resources  Council did not approve the program in April 2003. The council made this  decision because it believed that the benefits of CPDLC did not outweigh  the costs. A number of factors contributed to this decision. First, FAA had  concerns about how quickly aircraft would install the new airborne  equipment. Second, the approved program baseline was no longer valid as  Build 1A investment costs had increased from $114.5 million to $181.7  million, while the number of locations decreased from 20 to 8 as shown in  table 4. Third, CPDLC would add $83 million to the operations account.
For fiscal year 2005, program officials requested $3 million for CPDLC.  According to FAA, this amount would be suitable for shutdown of CPDLC  at Miami, closeout of Build 1, and alternatives analysis for a follow-on  program. The contractor, ARINC, had been providing messaging service for  Miami at no cost. However, the contract for this free service expired on  June 30, 2004.
section_title: Challenges in Approving CPDLC
paragraphs: Lack of full coordination between FAA’s aircraft certification and  acquisition offices, in which there would have been a full understanding of  all requirements, compromised the schedule and cost of CPDLC. FAA’s  acquisitions office, in the interest of meeting the original cost and schedule  estimates, awarded the contract before FAA had a full understanding of  system requirements, including those of FAA’s aircraft certification office.  Requirements that specified in detail how the air and ground equipment  would operate together were not yet completed prior to award of the Build  1A contract. The addition of CPDLC hardware and software requirements  increased costs by $26 million, 39 percent of CPDLC’s Build 1A  development cost growth. In addition, other system requirement changes  after contract award increased CPDLC’s baseline development cost  estimate by another $15 million. In total, these requirement additions  increased costs by $41 million, almost 61 percent of the total cost increases  associated with CPDLC Build 1A. (See tables 5, 6, and 7 for timelines of  CPDLC’s ground system approval and aircraft equipment certification.)
section_title: Local Area Augmentation System Case Illustration
subsections: section_title: Background
paragraphs: LAAS is a precision approach and landing system that will augment the  Global Positioning System (GPS) to broadcast highly accurate information  to aircraft on the final phases of a flight. LAAS is being developed  specifically to provide augmentation to GPS satellites to support Category  I, II, and III precision approach and landing capability to aircraft operating  within a 20- to 30-mile radius of an airport. LAAS approaches are to be  designed to avoid obstacles, restricted airspace, noise-sensitive areas, or  congested airspace. In addition, a single LAAS ground station is to be  capable of providing precision approach capability to multiple runways.  LAAS has both ground and air components. LAAS ground components  include four or more GPS reference receivers, which monitor and track  GPS signals; very high frequency transmitters for broadcasting the LAAS  signal to aircraft; and ground station equipment, which generates precision  approach data and is housed at or near an airport. (See fig. 4.) LAAS users  will have to purchase aircraft equipment to take advantage of the system’s  benefits.
section_title: Status
paragraphs: FAA’s fiscal year 2005 budget request eliminated funding for LAAS, which is  being moved from the acquisition program into a research and  development effort. LAAS was slated for a 2006 rollout, but the target has  now been deferred until at least 2009. FAA officials said they will  reconsider national deployment when more research results are  completed.
Before FAA decided to suspend funding for LAAS in fiscal year 2005, the  LAAS program office was negotiating with Honeywell to develop a plan for  determining how to meet the integrity requirements for the LAAS Category  I system. According to FAA officials, the LAAS program office will use the  $18 million remaining in fiscal year 2004 to continue the LAAS Integrity  Panel for developing the LAAS Category I system, to validate LAAS  Category II/III requirements, and to solve radio frequency interference  issues. The $18 million will last through 2005, and FAA’s goal is to meet  LAAS integrity requirement by September 2005. Because of the budget cuts  in fiscal year 2005, the LAAS program office will not be developing a  Category II/III prototype.
As shown in table 8, the LAAS Category I system was initially expected to  be operational in 2002. However, FAA was unable to meet the milestone,  primarily due to development and integrity requirement issues. According  to FAA officials, the research needed to validate the integrity requirement  of LAAS Category I is scheduled to be completed by September 2005. If  funds are fully restored in fiscal year 2005, FAA officials said that a LAAS  Category I system can be developed and deployed by fiscal year 2009.
section_title: FAA Faced Challenges in Approving LAAS
paragraphs: FAA faced a number of challenges in approving LAAS for safe use in the  national airspace system, including (1) its inability to meet LAAS’ integrity  requirement, (2) not always communicating with the contractor about what  was required to satisfy LAAS ground system requirements, and (3)  accelerating the LAAS schedule by setting milestones before designing the  system.
According to Honeywell officials, meeting the integrity requirement has  been perhaps the most difficult part of approving LAAS for safe use in the  national airspace system. Under FAA’s integrity requirement for LAAS, the  system must alert the pilot with timely warnings when it should not be  used. However, FAA has not been able to develop a solution to meet this  requirement because it has not been able to prove that the system is safe  during solar storms. According to FAA officials, one of the reasons that  FAA has not been able to develop a solution to meet this requirement is that  a solar storm’s effect on the ionosphere has not been modeled. The  modeling is scheduled for completion in September 2004, and it will be  used to design a monitor for ionosphere anomalies that could be developed  and deployed by fiscal year 2009.
FAA also did not always communicate with the contractor about what was  required to satisfy LAAS ground system requirements. Initially, FAA was in  a partnership with industry, including Honeywell and others, to develop a  LAAS Category I precision approach and landing system, which has a 200- foot ceiling height and one-half mile visibility. FAA partnered with industry  to develop LAAS because FAA would have to pay industry only if industry  achieved preset milestones, such as an analysis of the LAAS system  integrity requirement. However, the partnership was not able to develop a  system that FAA believed would operate safely in the national airspace  system. Consequently, FAA decided to acquire LAAS on its own. In April  2003, FAA awarded a contract to Honeywell to develop a LAAS Category I  precision approach and landing system. At the time the contract was  awarded, FAA believed that 80 percent of the LAAS was developed and met  its ground system requirements based on a review of documents. However,  5 months later, after further review, FAA discovered that only about 20  percent of development was complete. Nevertheless, Honeywell believes it  met 80 percent of the LAAS requirements. Both parties attribute the  disagreement to lack of communication about what was needed to satisfy  the LAAS ground system requirements. In fiscal year 2005, FAA decided to  suspend funding and placed LAAS into its research and development  program due to a lack of software development and the inability of the  system to meet the integrity requirement. According to FAA officials, the  research needed to validate the integrity requirement of LAAS Category I is  scheduled to be completed by September 2005. If funds are fully restored in  fiscal year 2005, FAA believes that a LAAS Category I system can be  developed and deployed by fiscal year 2009.
FAA also experienced challenges in approving LAAS because it accelerated  the schedule in 1998 to meet system milestones before completely  designing the system and developing a solution for meeting the LAAS  integrity requirement. FAA originally planned to deploy LAAS in 2002 but  had to subsequently delay deployment to 2006 because of additional  development work, evolving requirements, and unresolved issues regarding  how the system would be approved. Lack of a solution for verifying that its  integrity requirement had been met and incomplete software development  were significant approval issues facing the LAAS program.
Table 9 shows the major phases and time frames for approving the LAAS  ground system.
section_title: Certification of LAAS Aircraft Equipment Has Been Affected by Delays in Ground System Approval
paragraphs: LAAS aircraft equipment received design and production approval in  August 2004. It still awaits installation approval. (See table 10.) Because  LAAS’ aircraft and ground components are linked, certification of LAAS  aircraft equipment has been affected by delays occurring during ground  system approval. For example, according to aviation industry officials,  requirement additions on LAAS’ ground system led to requirement  additions on LAAS’ aircraft equipment. According to aviation industry  officials, the addition of requirements to the ground system increased the  cost and time to develop aircraft equipment, which changed the calculation  for industry about whether developing LAAS aircraft equipment was a  worthwhile investment and discourages future investment in aircraft  equipment that will modernize the national airspace system.
section_title: FAA’s Aircraft Certification Office Needs to Coordinate Better with Acquisitions Offices
paragraphs: FAA’s aircraft certification office completed the design and production  approval of LAAS aircraft equipment without coordinating with the offices  responsible for acquisition to determine the consequences of certifying  aircraft equipment before approval of the associated ground system.  According to an FAA official, once the aircraft certification office has given  design and production approval to the LAAS aircraft equipment, it is not  possible to make a change to the requirements for the aircraft equipment so  that they are better integrated with the associated LAAS ground system.  Consequently, LAAS ground system developers may have to make more  costly and time-consuming changes to the ground system than would have  been necessary if the aircraft certification and acquisitions offices had  coordinated their efforts.
section_title: Standard Terminal Automation Replacement System Case Illustration
subsections: section_title: Background
paragraphs: STARS is a joint Department of Transportation, FAA, and Department of  Defense (DOD) program established under 31 U.S.C. 1535, the Economy  Act, as amended, to replace aging FAA and DOD legacy terminal  automation systems with state-of-the-art terminal ATC systems. The joint  program is intended to avoid duplication of development and logistic costs  while providing easier transition of controllers between the civil and  military sectors. Civil and military air traffic controllers across the nation  are using STARS to direct aircraft near major airports. FAA’s goal for  STARS is to provide an open, expandable terminal automation platform  that can accommodate future air traffic growth and allow for the  introduction of new hardware- and software-based tools to promote safety,  maximize operational efficiency, and improve controllers’ productivity.  FAA believes that STARS will facilitate efforts to optimally configure the  terminal airspace around the country, exchange digital information  between pilots and controllers, and introduce new position and  surveillance capabilities for pilots. (See fig. 5.)
section_title: Status
paragraphs: In June 2003, FAA first commissioned STARS for use at the Philadelphia  International Airport in Pennsylvania. Currently, STARS is fully operational  at 25 FAA terminal radar control facilities and 17 DOD facilities. Under the  Air Traffic Organization’s new business model of breaking large and  complex programs into smaller phases to control cost and schedule,  STARS is a candidate for further deployment to about 120 FAA terminal  radar control facilities. As shown in table 11, in April 2004, FAA changed  STARS’ cost and schedule estimates for the third time and now estimates  that it will cost $1.46 billion to deploy STARS at the 50 most important  terminal radar control facilities that provide air traffic control services to  20 of the nation’s top 35 airports. The original baseline in February 1996  was $940 million for 172 systems. The April 2004 estimate is an increase of  about $500 million for 122 fewer systems (i.e., over 70 percent less) than  originally planned.
section_title: FAA Faced Challenges in Approving STARS
paragraphs: FAA faced challenges in approving STARS. Although controllers and  technicians were involved in developing requirements for STARS prior to  the 1996 contract award to Raytheon, the original approved acquisition  plan provided only limited human factors evaluation from controllers and  technicians during STARS’ design and development phase. The acquisition  approach was to employ a commercial off-the-shelf system with limited  modifications, and the competition was limited to companies with already  operational ATC systems. In 1997, FAA controllers, who were accustomed  to using the older equipment, began to voice concerns about computer- human interface issues that could hamper their ability to monitor air  traffic. For example, the controllers noted that many features of the old  equipment could be operated with knobs, allowing controllers to focus on  the screen. By contrast, the STARS commercial system was menu-driven  and required the controllers to make several keystrokes and use a  trackball, diverting their attention from the screen. The maintenance  technicians also identified differences between STARS and its backup  system that made monitoring the system less efficient. For example, the  visual warning alarms and color codes identifying problems were not  consistent between the two systems. In 1997, FAA, the National Air Traffic  Controllers Association, the Professional Airways System Specialists, and  Raytheon formed a team to deal with these computer-human interface  issues. The team identified 98 air traffic and 52 airway facilities computer- human interface enhancements to address these issues.
FAA and Raytheon restructured the contract to address the technicians’  and controllers’ concerns. According to FAA, not involving controllers and  maintenance technicians caused FAA to revise its strategy for approving  STARS, which FAA estimates added $500 million and 3 years to the  schedule. The original STARS cost estimate of $940 million included limited  human factors evaluations and the use of a basic commercial off-the-shelf  configuration. This acquisition strategy was replaced by an incremental  development strategy that incorporated up front the majority of human  factors considerations and additional functionality that were not included  in the original cost estimate. This new acquisition strategy added years to  the development schedule and significantly increased the system’s  requirements specifications. These additional requirements resulted in  both cost and schedule growth. FAA’s own guidance showed that limiting  human factors evaluations will result in higher costs and schedule delays.  Initially, it is more expensive (in terms of time and funding) to deal with  human factors considerations than to ignore them. However, an initial  human factors investment pays high dividends, in terms of costs and  schedule, in later stages of acquisition when changes are more costly and  difficult to make.
FAA also experienced challenges in approving STARS, partly, because of  aggressive scheduling. FAA’s approach to approving STARS was oriented to  rapid deployment to meet critical needs. To meet these needs, FAA  compressed its original development and testing schedule from 32 months  to 25 months. This acceleration in schedule left only limited time for human  factors evaluations and not enough time for involvement of controllers and  maintenance technicians.
Table 12 shows the major phases and time frames associated with the  STARS approval process.
section_title: Wide Area Augmentation System Case Illustration
subsections: section_title: Background
paragraphs: WAAS is a GPS-based navigation and landing system. According to FAA,  WAAS is to improve safety by providing precision guidance to aircraft in all  phases of flight at thousands of airports and landing strips, including  runways, where there is no ground-based landing capability. To use WAAS  for navigation, an aircraft must be equipped with a certified WAAS receiver  that is able to process the information carried by GPS and WAAS  geostationary satellite signals. Pilots are able to use this information to  determine their aircrafts’ time and speed, and latitude, longitude, and  altitude positions. WAAS currently consists of a network of 25 ground  reference stations, 2 leased geostationary satellites, 2 master stations, and  4 uplink (ground earth) stations. The ground reference stations are  strategically positioned across the United States to collect GPS satellite  data. (See fig. 6.) WAAS is designed to improve the accuracy, integrity, and  availability of information coming from GPS satellites and to correct signal  errors caused by solar storms, among other things.
FAA expects that WAAS will improve the national airspace system by (1)  increasing runway capability; (2) reducing separation standards that allow  increased capacity in a given airspace without increased risk; (3) providing  more direct en route flight paths; (4) providing new precision approach  services; (5) reducing the amount of and simplifying equipment on board  aircraft; (6) saving the government money due to the elimination of  maintenance costs associated with older, more expensive ground-based  navigation aids; and (7) providing vertical guidance in all phases of flight to  improve safety.
section_title: Status
paragraphs: In July 2003, FAA commissioned WAAS to provide initial operating  capability for 95 percent of the United States. In July 2003, the first of the  LPV approaches were provided whereby pilots could safely descend to a  250-foot decision height. As of August 2004, there were about 20 LPV  landing procedures published for WAAS. With over 4,000 runways needing  them, much work still needs to be done to fully utilize the WAAS capability.  FAA expects to have WAAS available in the rest of the country, with the  exceptions of a few parts of Alaska, by the end of 2008 when it completes  the addition of 13 ground reference stations and 2 leased geostationary  satellites. WAAS is not scheduled to achieve full (Category I) operating  capability, the final phase of WAAS when pilots will be able to use it to  navigate as low as 200 feet above the runway, until the 2013-2019 time  frame.
As shown in table 13, FAA changed WAAS’ cost and schedule estimates for  the third time in May 2004. According to FAA, the reasons for the May 2004  rebaselining were that the system was not able to achieve full Category 1  capability and because of FAA internal and congressional budget cuts.  Under the May 2004 baseline, FAA estimates that WAAS development costs  will be about $2.0 billion, which is $1.5 billion higher than the 1994  estimated development costs. Also, FAA has not yet met some of its  original performance goals, such as providing pilots with the ability to  navigate as low as 200 feet above the runway. According to FAA, WAAS  cannot easily achieve Category I as a single frequency system because the  error sources caused by solar storms are difficult to correct without the use  of a second civil aviation frequency in space, which is the responsibility of  the Department of Defense. FAA, realizing the difficulty and risk associated  with developing a single frequency Category I system, decided to wait and  leverage the benefits of the White House policy to include the second civil  frequency on the GPS satellite network. According to FAA, budget cuts and  the decision to wait until the second civil frequency is placed on the GPS  constellation have caused it to extend the timeline for reaching WAAS’ full  Category I operating capability to between 2013 and 2019.
section_title: FAA Faced Challenges in Approving WAAS
paragraphs: FAA faced challenges in approving WAAS ground and satellite components  for use in the national airspace system, partly because of FAA’s accelerated  scheduling, lack of effective coordination between its aircraft certification  office and acquisitions office, and technical challenges which resulted in a  delay meeting the integrity requirement. FAA’s challenges in approving  WAAS began in 1994 when FAA accelerated the implementation of  milestones, including moving up the commissioning of WAAS by 3 years.  FAA originally planned to commission WAAS in 2000; however, at the  urging of government and aviation industry groups in the 1990s, it decided  to change WAAS’ commissioning date to 1997. FAA tried to develop, test,  and deploy WAAS within 28 months, despite the fact that software  development alone was expected to take 24 to 28 months. FAA also set  system milestones before completing the research and development  required to prove the system’s capability. Although FAA attempted to  accelerate the implementation of WAAS, it wasn’t until July 2003, 6 years  later, that it was able to commission WAAS with initial operating capability.
Lack of full involvement between FAA’s aircraft certification members and  the rest of the integrated product team contributed to delays in approving  WAAS. For example, although an integrated product team, which included  representatives from aircraft certification and acquisition offices, was  developing WAAS, it was not until September 1999, when the aircraft  certification office became fully involved, that FAA recognized (1) the  difficulty of meeting the integrity requirement—that WAAS must alert the  pilot in a timely manner when the system should not be used—and (2) it did  not have the technical expertise needed. According to FAA officials, the  reason coordination did not occur was because the two offices had  competing priorities, such as the day-to-day aircraft equipment  certification activities not associated with the development of a new ATC  system. This situation may have developed because FAA’s aircraft  certification organization is more accustomed to being involved after a  project is developed, rather than actively participating throughout project  development.
The need to meet WAAS’ integrity requirement also hampered FAA’s ability  to approve WAAS for safe use in the national airspace system. In December  1999, FAA found that WAAS did not meet the agency’s integrity requirement  for precision approaches, and FAA recognized that it did not have the  technical expertise required to resolve the issue. Therefore, in 2000, FAA  established a team of satellite navigation experts, which was referred to as  the WAAS Integrity Performance Panel and included representatives from  the MITRE Corporation, Stanford University, Ohio University, and the Jet  Propulsion Laboratory. Developing a solution to prove that the WAAS  design met the integrity requirement added about 2 years and 4 months to  the approval process and contributed to WAAS’ cost growth. All of these  challenges contributed to a 6-year delay in WAAS’ commissioning and a  $1.5 billion increase in its estimated total development costs through 2028,  exclusive of operating and maintaining geostationary satellites, which were  not part of WAAS’ original 1994 baseline. Table 14 shows the major phases  and time frames associated with approving WAAS’ ground system.
section_title: FAA Did Not Experience Major Challenges in Certifying the Aircraft Equipment of WAAS
paragraphs: In contrast to the challenges that it encountered during the approval of the  WAAS ground system, FAA did not encounter major challenges with the  certification of WAAS aircraft equipment, primarily because FAA had an  up-front approval agreement with one of the first applicants, United Parcel  Service Aviation Technology, through the creation and approval of a safety  plan and a project-specific certification plan. Table 15 shows the major  phases and time frames associated with certifying the aircraft equipment of  WAAS. Currently, WAAS GPS receivers have been certified and are  available for use.
section_title: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contacts
section_title: Staff Acknowledgments
paragraphs: In addition to the individuals named above, other key contributors to this  report were Geraldine Beard, Gerald Dillingham, Seth Dykes, David  Hooper, Kevin Jackson, Gregg Justice III, Donna Leiss, and Kieran  McCarthy.