Air Traffic Control: FAA Needs to Ensure Better Coordination When Approving Air Traffic Control Systems
Released: 2004-11-17T00:00:00

Letter

Background

Several offices within FAA’s Air Traffic Organization and Office of Regulation and Certification have responsibility for approving ground systems and certifying aircraft equipment, as shown in figure 1.

Before the creation of the Air Traffic Organization in November 2003, FAA’s Research and Acquisitions (acquisitions office) and Air Traffic Services were the primary offices responsible for approving ground systems for safe use in the national airspace system. The 5 systems that we reviewed began the approval process under that structure. Currently, these offices, although renamed, form the core of the Air Traffic Organization. The responsibilities of Air Traffic Services are now distributed among several offices, including System Operations Services and Terminal Services. The responsibilities of Research and Acquisitions are distributed among several offices, including Technical Operations Services and En Route and Oceanic Services. In addition, the Air Traffic Organization includes Safety Services, which is its focal point for safety, quality assurance, and quality control and is the primary interface with FAA’s Office of Regulation and Certification.

FAA’s Office of Regulation and Certification has responsibility for certifying and regulating aircraft and its equipment. The following 3 offices within the Office of Regulation and Certification are involved in the certification of aircraft equipment: Aircraft Certification Service (aircraft certification office) is responsible for administering safety standards for aircraft and aircraft equipment that are manufactured in the United States.

Flight Standards Service is responsible for granting operational approval to air carriers that plan to use equipment on their aircraft.

Air Traffic Safety Oversight Service is responsible for monitoring the safety of air traffic operations through the establishment, approval, and acceptance of safety standards and the monitoring of safety performance and trends. It will also improve coordination between the Office of Regulation and Certification and the Air Traffic Organization.

In addition to the internal FAA stakeholders, the approval of air traffic control (ATC) systems can also involve a number of other external stakeholders. FAA generally makes the decision about which other stakeholders will be involved in approving ATC systems for safe use in the national airspace system. For example, stakeholders involved in approving ATC systems may include manufacturers of aircraft equipment; users, such as controllers and maintenance technicians.

FAA also regularly requests RTCA, a private, not-for-profit corporation, to develop consensus-based performance standards for the aircraft equipment component of ATC systems. RTCA functions as a federal advisory committee that provides recommendations used by FAA as the basis for policy, program, and regulatory decisions and by the private sector as the basis for development, investment, and other business decisions.

In this report, we focus on the approval of the 5 ATC systems described in table 1 and further discussed in appendixes II through VI.

FAA Has Separate Processes for Approving Ground Systems and Certifying Aircraft Equipment

FAA has separate processes for approving ground systems and certifying aircraft equipment for safe use in the national airspace system. FAA’s process for approving ground systems, such as radar systems, is done in accordance with policies and procedures in FAA’s Acquisition Management System. This process involves a determination by FAA’s Air Traffic Organization regarding whether a vendor is in compliance with contract requirements and/or FAA operational requirements, followed by a rigorous test-and-evaluation process to ensure that the new system will operate safely in the national airspace system. In contrast, the process for certifying aircraft equipment, which is usually developed by private companies, is done in accordance with Federal Aviation Regulations, with FAA serving as the regulator. If an ATC system has both a ground system and aircraft equipment, as was the case for 3 of the 5 systems we reviewed, then the system must go through both processes before it is approved for safe use in the national airspace system.

Ground System Approval Process

The approval of a ground system focuses on safety and is done in accordance with FAA contract documents and policies and procedures that are part of the agency’s Acquisition Management System. Most ground systems that provide air traffic services and air navigation services are developed, owned, and operated by FAA. Prior to November 2003, FAA’s Research and Acquisitions and Air Traffic Service offices were responsible for the approval of ground systems. Currently, FAA’s Air Traffic Organization has primary responsibility for the approval of ground systems. FAA’s ground system approval process includes the following six phases—concept of operations, requirements setting, design and development, test and evaluation, operational readiness, commissioning— and involves various stakeholders, which are also noted below.

Concept of operations: The ground system approval process begins with the concept of operations phase. If the system being developed has both a ground system and aircraft equipment, FAA’s Office of Regulation and Certification, Air Traffic Services Office, and Acquisitions Office may work together to develop the concept of operations. During this phase, FAA generally identifies and defines a service or capability to meet a particular need in the national airspace system and may involve other stakeholders, such as air traffic controllers. FAA also defines the roles and responsibilities of key participants, such as controllers and maintenance technicians, and the key elements of the required capability. The concept of operations phase is not a static process. As FAA obtains more information about the system it develops, the concept is revised to reflect the new information even though the next phase of the process may have already begun. Potential stakeholders in this phase include FAA’s Office of Regulation and Certification, FAA’s Air Traffic Organization, aircraft manufacturers, aviation industry associations, airlines, air traffic controllers, maintenance technicians, manufacturers of aircraft equipment, ground system developers, and representatives of general aviation.

Requirements setting: During the requirements-setting phase, FAA establishes a minimum set of requirements, including safety objectives, and specifies how well the new system must perform its intended functions. For example, it was during this phase that FAA established WAAS’ and LAAS’ integrity requirement—which is that the system cannot fail to warn pilots of misleading information that could potentially create hazardous situations more than once in 10 million approaches. After analyzing the initial requirements and comparing the cost, benefits, schedule, and risk of various solutions, FAA sets final requirements and presents them to the Joint Resources Council as part of the investment plan. After the council has approved the requirements for the new system, FAA will issue a request for proposals, evaluate the offers received, and select a contractor to design a system based on the requirements set by FAA. Potential stakeholders in this phase include FAA’s Office of Regulation and Certification, FAA’s Air Traffic Organization, aircraft manufacturers, aviation industry associations, airlines, air traffic controllers, maintenance technicians, manufacturers of aircraft equipment, ground system developers, and representatives of general aviation.

Design and development: The design and development of ground systems is generally completed by a contractor and monitored by FAA. During this phase, the contractor conducts preliminary and critical design reviews, which include plans for how it will conduct the testing phase. FAA must approve these plans before the contractor can proceed to the next phase. Potential stakeholders in this phase include FAA, ground system developers, air traffic controllers, and maintenance technicians.

Test and evaluation: After FAA has approved the design and development of the system, it is ready to be tested and evaluated. The testing and evaluation of ground systems typically includes three major tests: development tests, operational tests, and an independent operational test and evaluation. Development testing is performed by the contractor to verify compliance with contractual requirements and is overseen by FAA. Operational testing is performed by FAA and is designed to demonstrate that a new system is operationally effective and suitable for use in the national airspace system. An independent operational test and evaluation is a full system-level evaluation conducted by FAA in an operational environment to confirm the operational readiness of a system to be part of the national airspace system. Potential stakeholders in this phase include FAA, ground system developers, air traffic controllers, and maintenance technicians.

Operational readiness: During the operational readiness phase, FAA personnel are trained to operate and maintain the new system, usually in conjunction with its predecessor system. Following operational readiness approval, the system is ready to be commissioned. Potential stakeholders in this phase include FAA, ground system developers, air traffic controllers, and maintenance technicians.

Commissioning: The commissioning phase ensures that the new ground system as installed meets the intended mission and operational requirements and is fully supported by the national airspace system infrastructure. Potential stakeholders in this phase include FAA, ground system developers, air traffic controllers, and maintenance technicians.

Aircraft Equipment Certification Process

In contrast to the ground system approval process, certification of aircraft equipment is done in accordance with procedures outlined in the Federal Aviation Regulations, Title 14, Code of Federal Regulations, Part 21. Under Title 49, Section 44704, of the U.S. Code, FAA has the authority to issue type certificates, supplemental type certificates, and production certificates, among others, for aircraft and equipment that will be used in the national airspace system. Unlike the approval of ground systems, which FAA accomplishes with the help of a contractor, FAA is the regulator of aircraft equipment and is not typically involved in the development of the equipment. An applicant, such as a manufacturer of aircraft equipment, generally brings fully developed aircraft equipment to FAA for certification. The aircraft equipment certification process includes the following five phases—concept of operations, requirements setting, design and production approval, installation approval, and operational approval—and involves several stakeholders, which are also noted below: Concept of operations: Like the ground system approval process, the aircraft equipment certification process generally begins with the concept of operations phase, when the aircraft equipment is part of an ATC system. If the aircraft equipment certification process is not associated with the approval of a new ground system, then the certification process may begin with an idea for better equipment. During this phase, FAA, sometimes with the help of industry, identifies and defines a service or capability to meet a particular need in the national airspace system. Potential stakeholders in this phase include FAA’s Office of Regulation and Certification, FAA’s Air Traffic Organization, aircraft manufacturers, aviation industry associations, airlines, air traffic controllers, maintenance technicians, manufacturers of aircraft equipment, ground system developers, and representatives of general aviation.

Requirements setting: Once FAA has identified the need for a new system with aircraft equipment, FAA determines the requirements for the aircraft equipment. In some cases, the requirements for aircraft equipment may already exist in the Federal Aviation Regulations. In other cases, FAA may ask RTCA to develop the requirements, including safety requirements, which are referred to as minimum operating performance standards. RTCA typically takes 1 to 5 years to develop the standards because of the need to reach consensus between FAA and the industry and the increasing complexity of systems being developed today. According to a RTCA official, the time required to develop recommended standards is a function of many variables, including urgency of the situation and the commitment and availability of government and industry volunteers to collaboratively develop the standards. For example, in the case of WAAS, RTCA began setting performance standards in 1994, completed the original version of the standards in January 1996, and completed the most recent version of WAAS performance standards in November 2001. Potential stakeholders in this phase include FAA’s Office of Regulation and Certification, FAA’s Air Traffic Organization, aircraft manufacturers, aviation industry associations, airlines, air traffic controllers, maintenance technicians, manufacturers of aircraft equipment, ground system developers, and representatives of general aviation.

Design and production approval: The requirements/performance standards, most often developed by RTCA, typically form the basis for a technical standard order, which FAA uses to grant design and production approval for most new aircraft equipment developed in support of national airspace system modernization efforts. Technical standard orders are FAA’s requirements for materials, parts, processes, and appliances used on civil aircraft. Most aircraft manufacturers want technical standard orders because they make installation approval simpler and less costly and allow for operation in any type of aircraft. Technical standard orders are issued for items ranging from safety belts to navigation equipment. If the applicant successfully completes the design and production approval phase, FAA provides the applicant with a technical standard order authorization letter, which states that the applicant has met a specific technical standard order and the product is now ready for the installation approval phase. Potential stakeholders in this phase include FAA’s Aircraft Certification Service, manufacturers of aircraft equipment, and aircraft manufacturers.

Installation approval: After receiving a technical standard order authorization for new aircraft equipment, the initial applicant must receive installation approval from FAA before the aircraft equipment may be used in the national airspace system. To receive installation approval, the applicant submits a certification plan and test plan to one of FAA’s aircraft certification offices for review and approval. In addition, the applicant conducts ground and flight tests under FAA’s supervision to ensure that the new equipment operates properly upon installation. Once the tests are completed to FAA’s satisfaction, FAA issues a supplemental type certificate, which is evidence of FAA’s approval to modify an aircraft from its original design. Potential stakeholders in this phase include FAA’s Aircraft Certification Service, manufacturers of aircraft equipment, and aircraft manufacturers.

Operational approval: Finally, for the aircraft equipment to become certified for use in the national airspace system by air carrier operators, operational approval is also needed from FAA. To obtain operational approval, the applicant must successfully demonstrate, among other things, that the pilots are properly trained to use the aircraft equipment and that maintenance personnel are properly trained to maintain the equipment. Potential stakeholders in this phase include FAA’s Flight Standards Service, airlines, and representatives of general aviation.

FAA Faced Challenges in Approving Several ATC Systems

FAA faced challenges in approving systems for safe use in the national airspace system that contributed to cost growth, delays, and performance shortfalls in deploying these systems. We identified three specific challenges through the review of 5 ATC systems and our past work. These challenges are the need to involve appropriate stakeholders, such as users and technical experts, throughout the approval process; ensure that the FAA offices that have responsibility for approving ground systems and certifying aircraft equipment effectively coordinate their efforts for integrated systems; and accurately estimate the amount of time needed to meet complex technical requirements at the beginning of the design and development phase.

Although most of the challenges we found relate to the ground system approval process, RTCA and the Aerospace Commission have identified challenges with FAA’s aircraft equipment certification process. For example, RTCA found that there was a need for better internal FAA communication and coordination, including the establishment of an organizational focal point to provide coordinated responses to all matters related to ground systems and aircraft equipment. In addition, the Aerospace Commission found that FAA’s regulatory process needs to be streamlined to enable the timely development of regulations needed to address new technologies.

FAA Did Not Always Adequately Involve Appropriate Stakeholders, Such as Users and Technical Experts, Throughout Its Approval Process

FAA failed to adequately involve appropriate stakeholders, such as air traffic controllers and maintenance technicians, for 3 of the 5 systems we reviewed. For example, FAA did not adequately involve controllers and maintenance technicians throughout the approval process of STARS, which will replace controller workstations with new color displays, processors, and computer software. Although controllers and technicians were involved in developing requirements for STARS in 1994 prior to the 1996 contract award to Raytheon, the original approved acquisition plan provided for only limited human factors evaluation by controllers and technicians during STARS’ design and development because the aggressive development schedule limited the amount of time available to involve them. Consequently, FAA and Raytheon had to restructure the contract to address controllers’ concerns that were identified later, such as the inconsistency of visual warning alarms and color codes with the new system. According to FAA officials, not involving controllers and maintenance technicians in the design phase caused the agency to revise its strategy for acquiring and approving STARS, which contributed to STARS’ overall cost growth of $500 million and added 3 years to the schedule.

FAA also did not always sufficiently involve technical experts early in its approval process for 2 additional systems that we reviewed. For example, FAA did not obtain technical expertise on how to resolve the integrity requirement of WAAS, a navigation system for aviation that augments the Global Positioning System (GPS), until late in the design and development phase. FAA acknowledges that the agency’s in-house technical expertise was not sufficient to address the technical challenges of WAAS. Initially, FAA and the contractor believed they could meet the WAAS integrity requirement to alert the pilot in a timely manner when the system should not be used. However, although WAAS was being developed by an integrated product team that included representatives from several FAA offices, the team did not function effectively in resolving issues related to meeting an important functional requirement to alert the pilot in a timely manner when the system should not be used because of a possible error. According to FAA officials, the reason coordination did not occur was that the two offices had competing priorities that were not associated with WAAS’ development. Consequently, in 2000, FAA convened the WAAS Integrity Performance Panel to help it meet the integrity requirement. The WAAS Integrity Panel worked for about 2-1/2 years before it came up with a solution to the integrity requirement. In addition, in August 2000, the agency established an Independent Review Board, which is independent of the panel and included experts in satellite navigation and safety certification, to oversee the panel and evaluate the soundness of its efforts.

According to a member of the WAAS Integrity Panel, if FAA had involved these technical groups immediately after the contract was awarded to Raytheon in 1996, these groups could have started devising a solution in 1996, rather than in 2000. This lack of technical expertise contributed to a 6-year delay in WAAS’ commissioning and a $1.5 billion increase in its development costs from the 1994 baseline.

FAA also did not fully engage technical experts early in the approval process of LAAS, a precision approach and landing system that will augment GPS. According to FAA officials, meeting the LAAS integrity requirement to alert the pilot in a timely manner when the system should not be used is perhaps the most difficult part of approving this system for safe use in the national airspace system. According to the Department of Transportation’s Inspector General, although FAA had a LAAS Integrity Panel in place since 1996 to assist with its research and development activities, the panel was not formally tasked with resolving LAAS’ integrity issues. According to one satellite navigation expert and the Department of Transportation’s Inspector General, focusing the LAAS Integrity Panel on resolving the integrity requirement early in the approval process may have enabled FAA to develop a quicker solution. In 2003, FAA focused the LAAS Integrity Panel on developing a solution to meet the integrity requirement. However, FAA and another satellite expert maintain that the technical complexity of this problem is the main reason that LAAS is not commissioned. According to FAA officials, the need to validate integrity requirements and further software development has resulted in FAA placing LAAS in its research and development program and suspending funding for fiscal year 2005.

In contrast, FAA faced fewer schedule and cost problems in approving ASDE-X for use in the national airspace system. This was, in part, because FAA included stakeholders early and throughout the approval process and because program managers had strong technical expertise. The ASDE-X program office brought in stakeholders, including maintenance technicians and air traffic controllers, during the concept of operations phase and continued to involve them during requirements setting, design and development, and test and evaluation. FAA also brought ASDE-X stakeholders together at technical meetings to provide input on ASDE-X design and development, which allowed the ASDE-X program office to design a system that met requirements and incorporated stakeholders’ needs. By obtaining the input of controllers and technicians at the beginning of the approval process, FAA was able to ensure that ASDE-X requirements were set at appropriate levels and not overspecified or underspecified. Some stakeholders commented that the program managers’ strong technical expertise was one reason that ASDE-X’s requirements were set appropriately. As a result, this system was initially commissioned only 5 months behind schedule and its cost increased moderately from $424 million to $510 million.

FAA Did Not Always Effectively Coordinate Its Certification and Approval Processes

FAA did not always effectively coordinate its certification and approval processes for CPDLC, WAAS, and LAAS. Coordination between FAA’s offices responsible for approval of ground systems and certification of aircraft equipment is becoming increasingly important given that more and more ATC systems have both ground systems and aircraft equipment. However, we found that coordination was not effective on CPDLC Build 1A, which allows pilots and controllers to transmit digital data messages directly between FAA ground automation systems and suitably equipped aircraft. In the interest of meeting the original cost and schedule estimates, FAA awarded the contract before it had a full understanding of system requirements. Requirements that specify how the ground system and aircraft equipment would operate together were not yet completed prior to award of the Build 1A contract. Consequently, changes needed to be made after the contract was awarded. New hardware requirements, software requirements, and other system requirement changes were added, which increased CPDLC’s costs by $41 million, almost 61 percent of the total cost increases associated with CPDLC.

The lack of effective coordination among FAA offices responsible for approving WAAS also contributed to delays and increased costs in commissioning WAAS. Although WAAS was being developed by an integrated product team that included representatives from various FAA offices, the team did not function effectively in resolving issues related to meeting an important functional requirement to alert the pilot in a timely manner when the system should not be used because of a possible error. According to FAA officials, the reason coordination was not effective was because the two offices had competing priorities that were not associated with development of WAAS. Consequently, it was not until September 1999, when the aircraft certification office became fully involved, that FAA recognized that its solution to meet WAAS’ integrity requirement was not sufficient and that it did not have the technical expertise needed to develop a solution. This lack of coordination contributed to a 6-year delay in WAAS’ commissioning and a $1.5 billion increase in its development costs.

LAAS is another example of how FAA did not effectively coordinate its efforts. For example, FAA’s Office of Regulation and Certification completed the design and production approval of LAAS aircraft equipment without effectively coordinating with the offices responsible for acquisition to determine the consequences of certifying aircraft equipment before approval of the associated ground system. According to an FAA official, once the Office of Regulation and Certification has given design and production approval to the LAAS aircraft equipment, it is not possible to make a change to the requirements for the aircraft equipment so that they are better integrated with the associated LAAS ground system. Consequently, LAAS ground system developers may have to make more costly and time-consuming changes to the ground system than would have been necessary if the Office of Regulation and Certification and acquisitions offices had coordinated their efforts.

FAA Did Not Always Prepare Accurate Estimates of the Amount of Time Needed to Meet Complex Technical Requirements

We have reported in the past that when FAA attempts to combine different phases of system development in an effort to more quickly implement the systems to meet milestones, it repeatedly experiences major performance shortfalls and rework, which leads to schedule delays and cost increases. We found that WAAS, STARS, and LAAS all experienced delays and cost increases in part because FAA did not prepare accurate estimates of the amount of time needed to meet complex technical requirements, leading to an accelerated schedule that sometimes failed to include activities such as human factors evaluations and technical expert consultations. For example, in 1994, in response to the concerns of government and aviation groups, FAA accelerated implementation of WAAS milestones from 2000 to 1997. FAA planned to develop, test, and deploy WAAS within 28 months, an unrealistic goal given that software development alone was expected to take 24 to 28 months. It was not until July 2003, over 6 years later, that FAA was able to commission WAAS for initial operating capability. The accelerated schedule contributed to the 6-year delay in the commissioning of the system because the schedule itself was unrealistic and additional design work needed to be completed. During that time, the cost to develop the system increased about $1.5 billion, and the system has yet to meet its original performance goal of providing pilots with the ability to navigate down to 200 feet during their approach to the runway.

FAA also accelerated the schedule for STARS in 1995. FAA’s approach to commissioning STARS was oriented to rapid deployment to meet critical needs for new equipment. To meet these needs, FAA compressed its original development and testing schedule from 32 to 25 months. Consequently, this acceleration in schedule left only limited time for human factors evaluations and, according to FAA officials, contributed to STARS’ overall cost growth of $500 million and added 3 years to the first deployment because the agency had to revise its strategy for acquiring and approving STARS.

Although FAA had not developed a solution for meeting the integrity requirement, FAA also accelerated the LAAS schedule in 1999 by setting system milestones before completely designing the system. FAA originally planned to deploy LAAS in 2002 but has since moved it to fiscal year 2009 because the system’s software development is not complete and a solution for meeting LAAS’ integrity requirements has yet to be developed.

RTCA and the Aerospace Commission Found Challenges with FAA’s Process for Approving Ground Systems and Certifying Aircraft Equipment

RTCA and the Aerospace Commission also identified challenges with FAA’s process for approving ground systems and certifying aircraft equipment. In 1998, at the request of the FAA Administrator, RTCA reviewed FAA’s certification/approval process to determine if it could be made more responsive to the changing state of aviation, including its more integrated technologies. RTCA found that FAA’s ground system approval process and aircraft equipment certification process took too long and cost too much, and RTCA made several recommendations to improve the processes. For example, in 2001, RTCA recommended that FAA implement a coordinated approval process that, among other things, would ensure that all stakeholders, including those outside FAA’s program offices, participate in all phases of the approval process. Specifically, similar to our finding that the FAA offices that had responsibility for approving ground systems and certifying aircraft equipment did not always effectively coordinate their efforts, RTCA found that there was a need for better internal FAA communication and coordination, including the establishment of an organizational focal point to provide coordinated responses to all matters related to ground systems and aircraft equipment. RTCA also found that there was a need for an earlier and better exchange of information between FAA and those involved in the approval and certification processes from outside FAA, such as manufacturers of aircraft equipment.

In 2000, Congress asked the Commission on the Future of the U.S. Aerospace Industry to study the health of the aerospace industry and identify actions that the United States needs to take to ensure the industry’s health. As part of this study, the Aerospace Commission reviewed FAA’s certification process for aircraft equipment and made recommendations. The Aerospace Commission found that FAA’s certification of new aircraft technologies has become uncertain in terms of time and cost and recommended that FAA’s regulatory process be streamlined to enable the timely development of regulations needed to address new technologies. According to the Aerospace Commission, instead of focusing on rules and regulations that dictate the design and approval of equipment, FAA should focus on certifying that manufacturing organizations have safety built into their processes for designing, testing, and ensuring the performance of an overall system. The commission believed that such an approach would allow FAA personnel to better keep up with technological progress by becoming less design-specific and more safety-focused.

FAA Has Taken Action to Improve Its Process for Approving ATC Systems

FAA has taken action to address two of the three management challenges that we identified. However, FAA has not taken action to ensure that all stakeholders, such as air traffic controllers, maintenance technicians, technical experts, and industry representatives, are involved throughout the ground system approval process. FAA has also taken some action to address recommendations made by RTCA and the Aerospace Commission. Examples of some of the actions FAA has taken that address the management challenges that we found as well as RTCA and Aerospace Commission recommendations are discussed below: Coordinating FAA’s acquisitions offices and Office of Regulation and Certification efforts for approving systems with ground and aircraft components: FAA officials believe that the agency’s new Safety Management System, which is designed to formalize the agency’s safety process, will also improve coordination among FAA internal stakeholders once it is implemented. FAA stated that coordination would improve because as part of the new Safety Management System the agency plans to realign its organizational structure to create a formal link between the Air Traffic Organization and the Office of Regulation and Certification. Within the Office of Regulation and Certification, there is the newly created Air Traffic Safety Oversight Service, which oversees the safety operations of the Air Traffic Organization and collaborates with the Air Traffic Organization’s Safety Services. In addition, according to FAA officials, both ground systems and aircraft equipment will be more consistently assessed for their effect on safety as safety terminology is standardized. FAA expects full implementation to take 3 to 5 years. We are reserving judgment on whether this change will fully address the challenge because of the early state of this effort and because FAA’s problems with internal coordination when approving ATC systems are long-standing. In addition, because FAA has historically faced internal and external coordination challenges in approving ATC systems for safe use in the national airspace, we believe that as FAA moves forward with the agency’s new Safety Management System, it should, in the interim, develop plans that describe how both internal and external coordination will occur on a system-specific basis. In addition, plans to include external stakeholders are particularly important since the Safety Management System is not intended to address this challenge.

Estimating the amount of time needed to meet complex technical requirements: During the development of WAAS and STARS, FAA adopted an incremental approach to developing and testing these systems to get them back on track, which is referred to as the “build a little, test a little” or spiral development approach. For example, to get WAAS back on track, FAA decided to take a more incremental approach to implementing the new navigation system—focusing more on the successful completion of research and development before starting system approval. In particular, FAA allowed time for collecting and evaluating data on key system performance requirements like the WAAS integrity requirement before moving forward. FAA officials acknowledged that the manner in which FAA decided to implement WAAS development before implementing this incremental approach was a high-risk approach and was a primary issue underlying the system’s problems. Some aviation stakeholders believe this approach is advantageous because, although it can increase costs initially, money can be saved in the long run because the approach may help to avoid mistakes that are very costly to fix once a system has been developed. This approach also helps to ensure that the necessary building blocks of a system are tested along the way through the early and ongoing involvement of key stakeholders, those who will use and maintain the system. These stakeholders are key to identifying critical omissions and issues that could prevent a system from operating as intended.

As previously discussed, RTCA and the Aerospace Commission reviewed FAA’s approval process and made a number of recommendations to improve it. FAA has taken some action to address these recommendations. For example: In response to RTCA’s recommendation to implement a process in which the regulators and applicants come to an early and clear agreement on their respective roles, responsibilities, expectations, schedules, and standards to be used in certification projects, FAA issued The FAA and Industry Guide to Avionics Approval in 2001, which is intended to help FAA reduce the time and cost for the certification of aircraft equipment. This guide describes how to plan, manage, and document an effective, efficient aircraft equipment certification process and how to develop a working relationship between FAA and the applicant. In addition, as part of the 1999 FAA and Industry Guide to Product Certification, FAA encourages the manufacturers of aircraft equipment to develop a Partnership for Safety Plan that defines roles and responsibilities, describes how the certification process will be conducted, and identifies the milestones for completing the certification. A WAAS aircraft equipment manufacturer said that the certification of the WAAS aircraft equipment it developed went smoothly, primarily because of this up-front agreement with FAA. Although FAA’s actions address the aircraft equipment certification process, it does not have a similar process for its ground system approval process.

In response to RTCA’s recommendation to establish an organizational focal point to provide one-stop service to users, industry, and other governments in all matters related to advanced ground electronics and aircraft equipment, FAA has completed a Web site that provides a broad range of information on the certification process for aircraft equipment. However, there is still no focal point to which industry can address questions about the approval process and be assured of getting a fully coordinated FAA answer.

In response to the Aerospace Commission’s recommendation to streamline its aircraft equipment certification process to ensure timely development of regulations needed to address new technologies and to focus on certifying that manufacturing organizations have built safety into their processes for designing, testing, and ensuring the performance of an overall system, FAA proposed creating an Organizational Designation Authorization program in January 2004. The program would expand the approval functions of FAA organizational designees, standardize these functions to increase efficiency, and expand eligibility for organizational designees.

Conclusions

FAA did not always include stakeholders throughout the process for approving ATC systems for safe use in the national airspace system. Including stakeholders is particularly important because the new ATC systems are more integrated today than in the past and thus require more coordination among all the stakeholders, particularly FAA’s Office of Regulation and Certification and the recently created Air Traffic Organization, but also between FAA and other stakeholders, such as technical experts, controllers, and maintenance technicians. When decisions regarding integrated ATC systems are made in isolation, they may contribute to the ineffective use of resources and time. We found that 3 of the 5 ATC systems we reviewed experienced cost growth and schedule delays, in part, because FAA did not always involve all necessary stakeholders, such as controllers and technical experts, throughout the approval process. In 2001, RTCA recommended that FAA implement a coordinated approval process that, among other things, would ensure that all stakeholders, including those outside FAA’s program offices, participate in all phases of the approval process. We agree with RTCA’s recommendation, which FAA has not fully implemented, and believe that fully implementing it would help address some of the challenges we found with FAA’s approval and certification processes.

In addition, although FAA’s new Safety Management System and the planned alignment between FAA’s Air Traffic Organization and Office of Regulation and Certification have the potential to improve FAA’s internal coordination, FAA has just begun implementing these initiatives with full implementation 3 to 5 years away. FAA also has historically faced internal coordination challenges in approving ATC systems for safe use in the national airspace system as we found for each of the 3 integrated systems that we reviewed. We believe that the implementation of the Safety Management System, coupled with the new formal link between FAA’s Air Traffic Organization and Office of Regulation and Certification, will give FAA the opportunity to improve its internal coordination among its offices that are responsible for ground system approval and aircraft equipment certification. However, the system will not be implemented until 3 to 5 years. Therefore, because of FAA’s history of internal and external coordination challenges, such as the lack of effective coordination between FAA offices responsible for approving WAAS, which contributed to WAAS’ cost increase of about $1.5 billion and schedule delays of 6 years, we believe that specific plans for improving coordination both internally and externally on a system-specific basis are needed now.

Recommendation for Executive Action

To ensure that key stakeholders, such as air traffic controllers, maintenance technicians, and technical experts, outside FAA’s acquisitions offices and Office of Regulation and Certification, are involved early and throughout FAA’s ground system approval process and to ensure better internal coordination between FAA’s offices responsible for approving ground systems and certifying aircraft equipment, we recommend that the Secretary of Transportation direct the Administrator of FAA to develop ATC system-specific plans early in the approval process that specify how and when the approving and certifying offices within FAA and other stakeholders, including controllers, maintenance technicians, technical experts, and industry representatives, will meet to ensure coordination.

Agency Comments

We provided a draft of this report to the Secretary of Transportation for review and comment. FAA generally agreed with our findings and recommendation and provided technical corrections, which we incorporated as appropriate. FAA also commented that it has started to take actions to improve its coordination efforts for integrated ATC systems.

We are sending copies of this report to interested congressional committees, the Secretary of Transportation, and the FAA Administrator. We will also make copies available to others on request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov. Should you or your staff have questions on matters discussed in this report, please contact me on (202) 512-2834 or at siggerudk@gao.gov. GAO contacts and key contributors to this report are listed in appendix VII.

Objectives, Scope, and Methodology

To complete our first objective, to describe FAA’s process for approving air traffic control (ATC) systems for safe use in the national airspace system, we obtained and analyzed documents from the Federal Aviation Administration (FAA) and RTCA’s 1999 report that discussed FAA’s process for certifying aircraft equipment and approving ground systems. We also interviewed FAA officials, contractors, industry experts, and unions representing air traffic controllers and maintenance technicians that are involved in approving ATC systems.

To complete our second objective, to describe the challenges FAA has faced approving ATC systems and how those challenges affected the cost, schedule, and performance estimates of the systems, we conducted case illustrations on 5 of FAA’s 25 air traffic control systems that are currently receiving funding: Airport Surface Detection Equipment - Model X (ASDE-X), Controller-Pilot Data Link Communications (CPDLC), Local Area Augmentation System (LAAS), Standard Terminal Automation Replacement System (STARS), and Wide Area Augmentation System (WAAS).

We selected these 5 systems because collectively they accounted for about 46 percent of FAA’s ATC modernization costs in fiscal year 2002 and 3 of the 5 systems are integrated—that is, they require the approval of the ground systems as well as aircraft equipment. To select the 5 case illustration systems, we used FAA’s capital investment project data file. We met with knowledgeable FAA officials to discuss issues related to the accuracy and completeness of the data file, which was deemed adequate for the purpose of our work. We also met with knowledgeable FAA officials to determine the number of ATC systems from the data file that needed to be approved before entry into the national airspace system. For each of the case illustrations, we reviewed FAA documents, including acquisition program baseline reports, Joint Resource Council decisions, and briefing documents. We also reviewed GAO and Department of Transportation’s Inspector General reports and testimonies. In addition, we interviewed officials from FAA program offices; RTCA; the General Aviation Manufacturers Association; the Air Transport Association; the Aircraft Owners and Pilots Association; NavCanada; Transport Canada; the MITRE Corporation; Boeing; Garmin; Rockwell Collins; contractors, including Honeywell, Raytheon, and the Sensis Corporation; industry experts; the WAAS Integrity Performance Panel; the LAAS Integrity Panel members; and unions representing air traffic controllers and maintenance technicians.

To compete our third objective, to describe actions FAA has taken to improve its processes for approving ATC systems, we interviewed representatives from FAA; RTCA; the Commission on the Future of the U.S. Aerospace Industry; aviation industry groups, including the General Aviation Manufacturers Association, the Air Transport Association, and the Aircraft Owners and Pilots Association; manufacturers of aircraft equipment, including Garmin and Rockwell Collins; Boeing; and contractors, including Honeywell, Raytheon, and the Sensis Corporation; industry experts; and unions representing air traffic controllers and maintenance technicians.

We conducted our review in Washington, D.C., from October 2003 through September 2004 in accordance with generally accepted government auditing standards.

Airport Surface Detection Equipment - Model X Case Illustration

Background

ASDE-X is an airport surface surveillance system that air traffic controllers use to track aircraft and vehicle surface movements. (See fig. 2.) ASDE-X uses a combination of surface movement primary radar and multilateration sensors to display aircraft position and vehicle position on an ATC tower display. According to FAA, the integration of these sensors provides accurate, up-to-date, and reliable data for improving airport safety in all weather conditions. ASDE-X was developed to prevent accidents resulting from runway incursions, which have increased since 1993. The number of reported runway incursions rose from 186 in 1993 to 383 in 2001. According to FAA, because air traffic in the United States is expected to double by 2010, runway incursions may pose a significant safety threat to U.S. aviation.

FAA expects that ASDE-X will increase the level of safety at airports and provide air traffic controllers with detailed information about aircraft locations and movement at night and in bad weather due to the (1) association of flight plan information with aircraft position on controller displays; (2) continuous surveillance coverage of the airport from arrival through departure; (3) elimination of blind spots and coverage gaps; and (4) availability of surveillance data with an accuracy and update rate suitable for, among other things, awareness in all weather conditions.

Status

In October 2003, FAA commissioned ASDE-X at Mitchell International Airport in Milwaukee, Wisconsin, for use in the national airspace system. ASDE-X came in close to its original schedule and cost baselines. The ASDE-X system was approximately 5 months over its original schedule baseline, but maintained its original performance baselines. In June 2002, FAA approved $80.9 million in additional funding to add ASDE-X at 7 additional sites. (See table 2.) FAA is currently scheduled to deploy ASDE- X at 25 U.S. airports over the next 4 years and to update existing surface detection systems (i.e., ASDE-3) at 9 other facilities. FAA plans to introduce an upgraded ASDE-X system at T.F. Green Airport in Providence, Rhode Island, with deployment tentatively slated for the 4th quarter of 2004. FAA is also investigating whether to add ASDE-X at 25 airports that use ASDE-3 and Airport Movement Area Safety Systems.

FAA Faced Fewer Challenges in Approving ASDE-X

Of the five systems we reviewed, FAA faced fewer schedule and cost challenges in approving ASDE-X for safe use in the national airspace system. This is partly because FAA included stakeholders early and throughout the approval process and because of the strong technical expertise of its managers. The ASDE-X program office brought in stakeholders, including maintenance technicians and air traffic controllers, beginning with the concept of operations phase and continued their stakeholder involvement through the requirements-setting, design-and- development, and test-and-evaluation phases and then continued involvement throughout the deployment phase. For example, FAA obtained the input of controllers and technicians at the beginning of the approval process, which helped to ensure that ASDE-X requirements were set at appropriate levels and not overspecified or underspecified. Stakeholders pointed toward the strong technical expertise of the program’s managers as a reason for the appropriate specification of ASDE-X’s requirements. In addition, FAA brought ASDE-X stakeholders together at technical meetings to provide input on ASDE-X design and development, which allowed the ASDE-X program office to design a system that met requirements and incorporated stakeholders’ needs.

However, FAA did experience some challenges in approving ASDE-X. In response to Congress’ desire to deploy the system quickly, FAA attempted to accelerate ASDE-X’s approval. However, FAA experienced problems in accelerating the approval when it awarded the contract before all requirements had been finalized.

Table 3 shows the major phases and time frames associated with the ASDE- X approval process.

Controller-Pilot Data Link Communications Case Illustration

Background

CPDLC will allow pilots and controllers to transmit digital data messages directly between FAA ground automation computers and suitably equipped aircraft. (See fig. 3.) CPDLC is a new way for controllers and pilots to communicate that is analogous to e-mail. The pilot can read the message displayed on a screen in the cockpit and respond to the message with the push of a key. In the future, this will alleviate frequency congestion problems and increase controller efficiency. One of the most important aspects of this technology is its intended reduction of operational errors from misunderstood instructions and readback errors. The initial phase (Build 1) consisted of four services: initial contact, altimeter setting, transfer of communication, and predefined instructions via menu text. The CPDLC program will ultimately develop additional capabilities in an incremental manner through further development stages. Originally, Build 1 was to be followed by Build 1A, which was designed to increase the CPDLC message set and include assignment of speeds, headings, and altitudes as well as a route clearance function.

Status

CPDLC was commissioned for initial daily use by controllers at Miami on October 7, 2002. This completed the stage called Build 1, which included four services. American Airlines is the CPDLC launch airline with about 25 aircraft operating in the Miami Center airspace. Further deployment of CPDLC has been deferred until about 2009 after the Joint Resources Council did not approve the program in April 2003. The council made this decision because it believed that the benefits of CPDLC did not outweigh the costs. A number of factors contributed to this decision. First, FAA had concerns about how quickly aircraft would install the new airborne equipment. Second, the approved program baseline was no longer valid as Build 1A investment costs had increased from $114.5 million to $181.7 million, while the number of locations decreased from 20 to 8 as shown in table 4. Third, CPDLC would add $83 million to the operations account.

For fiscal year 2005, program officials requested $3 million for CPDLC. According to FAA, this amount would be suitable for shutdown of CPDLC at Miami, closeout of Build 1, and alternatives analysis for a follow-on program. The contractor, ARINC, had been providing messaging service for Miami at no cost. However, the contract for this free service expired on June 30, 2004.

Challenges in Approving CPDLC

Lack of full coordination between FAA’s aircraft certification and acquisition offices, in which there would have been a full understanding of all requirements, compromised the schedule and cost of CPDLC. FAA’s acquisitions office, in the interest of meeting the original cost and schedule estimates, awarded the contract before FAA had a full understanding of system requirements, including those of FAA’s aircraft certification office. Requirements that specified in detail how the air and ground equipment would operate together were not yet completed prior to award of the Build 1A contract. The addition of CPDLC hardware and software requirements increased costs by $26 million, 39 percent of CPDLC’s Build 1A development cost growth. In addition, other system requirement changes after contract award increased CPDLC’s baseline development cost estimate by another $15 million. In total, these requirement additions increased costs by $41 million, almost 61 percent of the total cost increases associated with CPDLC Build 1A. (See tables 5, 6, and 7 for timelines of CPDLC’s ground system approval and aircraft equipment certification.)

Local Area Augmentation System Case Illustration

Background

LAAS is a precision approach and landing system that will augment the Global Positioning System (GPS) to broadcast highly accurate information to aircraft on the final phases of a flight. LAAS is being developed specifically to provide augmentation to GPS satellites to support Category I, II, and III precision approach and landing capability to aircraft operating within a 20- to 30-mile radius of an airport. LAAS approaches are to be designed to avoid obstacles, restricted airspace, noise-sensitive areas, or congested airspace. In addition, a single LAAS ground station is to be capable of providing precision approach capability to multiple runways. LAAS has both ground and air components. LAAS ground components include four or more GPS reference receivers, which monitor and track GPS signals; very high frequency transmitters for broadcasting the LAAS signal to aircraft; and ground station equipment, which generates precision approach data and is housed at or near an airport. (See fig. 4.) LAAS users will have to purchase aircraft equipment to take advantage of the system’s benefits.

Status

FAA’s fiscal year 2005 budget request eliminated funding for LAAS, which is being moved from the acquisition program into a research and development effort. LAAS was slated for a 2006 rollout, but the target has now been deferred until at least 2009. FAA officials said they will reconsider national deployment when more research results are completed.

Before FAA decided to suspend funding for LAAS in fiscal year 2005, the LAAS program office was negotiating with Honeywell to develop a plan for determining how to meet the integrity requirements for the LAAS Category I system. According to FAA officials, the LAAS program office will use the $18 million remaining in fiscal year 2004 to continue the LAAS Integrity Panel for developing the LAAS Category I system, to validate LAAS Category II/III requirements, and to solve radio frequency interference issues. The $18 million will last through 2005, and FAA’s goal is to meet LAAS integrity requirement by September 2005. Because of the budget cuts in fiscal year 2005, the LAAS program office will not be developing a Category II/III prototype.

As shown in table 8, the LAAS Category I system was initially expected to be operational in 2002. However, FAA was unable to meet the milestone, primarily due to development and integrity requirement issues. According to FAA officials, the research needed to validate the integrity requirement of LAAS Category I is scheduled to be completed by September 2005. If funds are fully restored in fiscal year 2005, FAA officials said that a LAAS Category I system can be developed and deployed by fiscal year 2009.

FAA Faced Challenges in Approving LAAS

FAA faced a number of challenges in approving LAAS for safe use in the national airspace system, including (1) its inability to meet LAAS’ integrity requirement, (2) not always communicating with the contractor about what was required to satisfy LAAS ground system requirements, and (3) accelerating the LAAS schedule by setting milestones before designing the system.

According to Honeywell officials, meeting the integrity requirement has been perhaps the most difficult part of approving LAAS for safe use in the national airspace system. Under FAA’s integrity requirement for LAAS, the system must alert the pilot with timely warnings when it should not be used. However, FAA has not been able to develop a solution to meet this requirement because it has not been able to prove that the system is safe during solar storms. According to FAA officials, one of the reasons that FAA has not been able to develop a solution to meet this requirement is that a solar storm’s effect on the ionosphere has not been modeled. The modeling is scheduled for completion in September 2004, and it will be used to design a monitor for ionosphere anomalies that could be developed and deployed by fiscal year 2009.

FAA also did not always communicate with the contractor about what was required to satisfy LAAS ground system requirements. Initially, FAA was in a partnership with industry, including Honeywell and others, to develop a LAAS Category I precision approach and landing system, which has a 200- foot ceiling height and one-half mile visibility. FAA partnered with industry to develop LAAS because FAA would have to pay industry only if industry achieved preset milestones, such as an analysis of the LAAS system integrity requirement. However, the partnership was not able to develop a system that FAA believed would operate safely in the national airspace system. Consequently, FAA decided to acquire LAAS on its own. In April 2003, FAA awarded a contract to Honeywell to develop a LAAS Category I precision approach and landing system. At the time the contract was awarded, FAA believed that 80 percent of the LAAS was developed and met its ground system requirements based on a review of documents. However, 5 months later, after further review, FAA discovered that only about 20 percent of development was complete. Nevertheless, Honeywell believes it met 80 percent of the LAAS requirements. Both parties attribute the disagreement to lack of communication about what was needed to satisfy the LAAS ground system requirements. In fiscal year 2005, FAA decided to suspend funding and placed LAAS into its research and development program due to a lack of software development and the inability of the system to meet the integrity requirement. According to FAA officials, the research needed to validate the integrity requirement of LAAS Category I is scheduled to be completed by September 2005. If funds are fully restored in fiscal year 2005, FAA believes that a LAAS Category I system can be developed and deployed by fiscal year 2009.

FAA also experienced challenges in approving LAAS because it accelerated the schedule in 1998 to meet system milestones before completely designing the system and developing a solution for meeting the LAAS integrity requirement. FAA originally planned to deploy LAAS in 2002 but had to subsequently delay deployment to 2006 because of additional development work, evolving requirements, and unresolved issues regarding how the system would be approved. Lack of a solution for verifying that its integrity requirement had been met and incomplete software development were significant approval issues facing the LAAS program.

Table 9 shows the major phases and time frames for approving the LAAS ground system.

Certification of LAAS Aircraft Equipment Has Been Affected by Delays in Ground System Approval

LAAS aircraft equipment received design and production approval in August 2004. It still awaits installation approval. (See table 10.) Because LAAS’ aircraft and ground components are linked, certification of LAAS aircraft equipment has been affected by delays occurring during ground system approval. For example, according to aviation industry officials, requirement additions on LAAS’ ground system led to requirement additions on LAAS’ aircraft equipment. According to aviation industry officials, the addition of requirements to the ground system increased the cost and time to develop aircraft equipment, which changed the calculation for industry about whether developing LAAS aircraft equipment was a worthwhile investment and discourages future investment in aircraft equipment that will modernize the national airspace system.

FAA’s Aircraft Certification Office Needs to Coordinate Better with Acquisitions Offices

FAA’s aircraft certification office completed the design and production approval of LAAS aircraft equipment without coordinating with the offices responsible for acquisition to determine the consequences of certifying aircraft equipment before approval of the associated ground system. According to an FAA official, once the aircraft certification office has given design and production approval to the LAAS aircraft equipment, it is not possible to make a change to the requirements for the aircraft equipment so that they are better integrated with the associated LAAS ground system. Consequently, LAAS ground system developers may have to make more costly and time-consuming changes to the ground system than would have been necessary if the aircraft certification and acquisitions offices had coordinated their efforts.

Standard Terminal Automation Replacement System Case Illustration

Background

STARS is a joint Department of Transportation, FAA, and Department of Defense (DOD) program established under 31 U.S.C. 1535, the Economy Act, as amended, to replace aging FAA and DOD legacy terminal automation systems with state-of-the-art terminal ATC systems. The joint program is intended to avoid duplication of development and logistic costs while providing easier transition of controllers between the civil and military sectors. Civil and military air traffic controllers across the nation are using STARS to direct aircraft near major airports. FAA’s goal for STARS is to provide an open, expandable terminal automation platform that can accommodate future air traffic growth and allow for the introduction of new hardware- and software-based tools to promote safety, maximize operational efficiency, and improve controllers’ productivity. FAA believes that STARS will facilitate efforts to optimally configure the terminal airspace around the country, exchange digital information between pilots and controllers, and introduce new position and surveillance capabilities for pilots. (See fig. 5.)

Status

In June 2003, FAA first commissioned STARS for use at the Philadelphia International Airport in Pennsylvania. Currently, STARS is fully operational at 25 FAA terminal radar control facilities and 17 DOD facilities. Under the Air Traffic Organization’s new business model of breaking large and complex programs into smaller phases to control cost and schedule, STARS is a candidate for further deployment to about 120 FAA terminal radar control facilities. As shown in table 11, in April 2004, FAA changed STARS’ cost and schedule estimates for the third time and now estimates that it will cost $1.46 billion to deploy STARS at the 50 most important terminal radar control facilities that provide air traffic control services to 20 of the nation’s top 35 airports. The original baseline in February 1996 was $940 million for 172 systems. The April 2004 estimate is an increase of about $500 million for 122 fewer systems (i.e., over 70 percent less) than originally planned.

FAA Faced Challenges in Approving STARS

FAA faced challenges in approving STARS. Although controllers and technicians were involved in developing requirements for STARS prior to the 1996 contract award to Raytheon, the original approved acquisition plan provided only limited human factors evaluation from controllers and technicians during STARS’ design and development phase. The acquisition approach was to employ a commercial off-the-shelf system with limited modifications, and the competition was limited to companies with already operational ATC systems. In 1997, FAA controllers, who were accustomed to using the older equipment, began to voice concerns about computer- human interface issues that could hamper their ability to monitor air traffic. For example, the controllers noted that many features of the old equipment could be operated with knobs, allowing controllers to focus on the screen. By contrast, the STARS commercial system was menu-driven and required the controllers to make several keystrokes and use a trackball, diverting their attention from the screen. The maintenance technicians also identified differences between STARS and its backup system that made monitoring the system less efficient. For example, the visual warning alarms and color codes identifying problems were not consistent between the two systems. In 1997, FAA, the National Air Traffic Controllers Association, the Professional Airways System Specialists, and Raytheon formed a team to deal with these computer-human interface issues. The team identified 98 air traffic and 52 airway facilities computer- human interface enhancements to address these issues.

FAA and Raytheon restructured the contract to address the technicians’ and controllers’ concerns. According to FAA, not involving controllers and maintenance technicians caused FAA to revise its strategy for approving STARS, which FAA estimates added $500 million and 3 years to the schedule. The original STARS cost estimate of $940 million included limited human factors evaluations and the use of a basic commercial off-the-shelf configuration. This acquisition strategy was replaced by an incremental development strategy that incorporated up front the majority of human factors considerations and additional functionality that were not included in the original cost estimate. This new acquisition strategy added years to the development schedule and significantly increased the system’s requirements specifications. These additional requirements resulted in both cost and schedule growth. FAA’s own guidance showed that limiting human factors evaluations will result in higher costs and schedule delays. Initially, it is more expensive (in terms of time and funding) to deal with human factors considerations than to ignore them. However, an initial human factors investment pays high dividends, in terms of costs and schedule, in later stages of acquisition when changes are more costly and difficult to make.

FAA also experienced challenges in approving STARS, partly, because of aggressive scheduling. FAA’s approach to approving STARS was oriented to rapid deployment to meet critical needs. To meet these needs, FAA compressed its original development and testing schedule from 32 months to 25 months. This acceleration in schedule left only limited time for human factors evaluations and not enough time for involvement of controllers and maintenance technicians.

Table 12 shows the major phases and time frames associated with the STARS approval process.

Wide Area Augmentation System Case Illustration

Background

WAAS is a GPS-based navigation and landing system. According to FAA, WAAS is to improve safety by providing precision guidance to aircraft in all phases of flight at thousands of airports and landing strips, including runways, where there is no ground-based landing capability. To use WAAS for navigation, an aircraft must be equipped with a certified WAAS receiver that is able to process the information carried by GPS and WAAS geostationary satellite signals. Pilots are able to use this information to determine their aircrafts’ time and speed, and latitude, longitude, and altitude positions. WAAS currently consists of a network of 25 ground reference stations, 2 leased geostationary satellites, 2 master stations, and 4 uplink (ground earth) stations. The ground reference stations are strategically positioned across the United States to collect GPS satellite data. (See fig. 6.) WAAS is designed to improve the accuracy, integrity, and availability of information coming from GPS satellites and to correct signal errors caused by solar storms, among other things.

FAA expects that WAAS will improve the national airspace system by (1) increasing runway capability; (2) reducing separation standards that allow increased capacity in a given airspace without increased risk; (3) providing more direct en route flight paths; (4) providing new precision approach services; (5) reducing the amount of and simplifying equipment on board aircraft; (6) saving the government money due to the elimination of maintenance costs associated with older, more expensive ground-based navigation aids; and (7) providing vertical guidance in all phases of flight to improve safety.

Status

In July 2003, FAA commissioned WAAS to provide initial operating capability for 95 percent of the United States. In July 2003, the first of the LPV approaches were provided whereby pilots could safely descend to a 250-foot decision height. As of August 2004, there were about 20 LPV landing procedures published for WAAS. With over 4,000 runways needing them, much work still needs to be done to fully utilize the WAAS capability. FAA expects to have WAAS available in the rest of the country, with the exceptions of a few parts of Alaska, by the end of 2008 when it completes the addition of 13 ground reference stations and 2 leased geostationary satellites. WAAS is not scheduled to achieve full (Category I) operating capability, the final phase of WAAS when pilots will be able to use it to navigate as low as 200 feet above the runway, until the 2013-2019 time frame.

As shown in table 13, FAA changed WAAS’ cost and schedule estimates for the third time in May 2004. According to FAA, the reasons for the May 2004 rebaselining were that the system was not able to achieve full Category 1 capability and because of FAA internal and congressional budget cuts. Under the May 2004 baseline, FAA estimates that WAAS development costs will be about $2.0 billion, which is $1.5 billion higher than the 1994 estimated development costs. Also, FAA has not yet met some of its original performance goals, such as providing pilots with the ability to navigate as low as 200 feet above the runway. According to FAA, WAAS cannot easily achieve Category I as a single frequency system because the error sources caused by solar storms are difficult to correct without the use of a second civil aviation frequency in space, which is the responsibility of the Department of Defense. FAA, realizing the difficulty and risk associated with developing a single frequency Category I system, decided to wait and leverage the benefits of the White House policy to include the second civil frequency on the GPS satellite network. According to FAA, budget cuts and the decision to wait until the second civil frequency is placed on the GPS constellation have caused it to extend the timeline for reaching WAAS’ full Category I operating capability to between 2013 and 2019.

FAA Faced Challenges in Approving WAAS

FAA faced challenges in approving WAAS ground and satellite components for use in the national airspace system, partly because of FAA’s accelerated scheduling, lack of effective coordination between its aircraft certification office and acquisitions office, and technical challenges which resulted in a delay meeting the integrity requirement. FAA’s challenges in approving WAAS began in 1994 when FAA accelerated the implementation of milestones, including moving up the commissioning of WAAS by 3 years. FAA originally planned to commission WAAS in 2000; however, at the urging of government and aviation industry groups in the 1990s, it decided to change WAAS’ commissioning date to 1997. FAA tried to develop, test, and deploy WAAS within 28 months, despite the fact that software development alone was expected to take 24 to 28 months. FAA also set system milestones before completing the research and development required to prove the system’s capability. Although FAA attempted to accelerate the implementation of WAAS, it wasn’t until July 2003, 6 years later, that it was able to commission WAAS with initial operating capability.

Lack of full involvement between FAA’s aircraft certification members and the rest of the integrated product team contributed to delays in approving WAAS. For example, although an integrated product team, which included representatives from aircraft certification and acquisition offices, was developing WAAS, it was not until September 1999, when the aircraft certification office became fully involved, that FAA recognized (1) the difficulty of meeting the integrity requirement—that WAAS must alert the pilot in a timely manner when the system should not be used—and (2) it did not have the technical expertise needed. According to FAA officials, the reason coordination did not occur was because the two offices had competing priorities, such as the day-to-day aircraft equipment certification activities not associated with the development of a new ATC system. This situation may have developed because FAA’s aircraft certification organization is more accustomed to being involved after a project is developed, rather than actively participating throughout project development.

The need to meet WAAS’ integrity requirement also hampered FAA’s ability to approve WAAS for safe use in the national airspace system. In December 1999, FAA found that WAAS did not meet the agency’s integrity requirement for precision approaches, and FAA recognized that it did not have the technical expertise required to resolve the issue. Therefore, in 2000, FAA established a team of satellite navigation experts, which was referred to as the WAAS Integrity Performance Panel and included representatives from the MITRE Corporation, Stanford University, Ohio University, and the Jet Propulsion Laboratory. Developing a solution to prove that the WAAS design met the integrity requirement added about 2 years and 4 months to the approval process and contributed to WAAS’ cost growth. All of these challenges contributed to a 6-year delay in WAAS’ commissioning and a $1.5 billion increase in its estimated total development costs through 2028, exclusive of operating and maintaining geostationary satellites, which were not part of WAAS’ original 1994 baseline. Table 14 shows the major phases and time frames associated with approving WAAS’ ground system.

FAA Did Not Experience Major Challenges in Certifying the Aircraft Equipment of WAAS

In contrast to the challenges that it encountered during the approval of the WAAS ground system, FAA did not encounter major challenges with the certification of WAAS aircraft equipment, primarily because FAA had an up-front approval agreement with one of the first applicants, United Parcel Service Aviation Technology, through the creation and approval of a safety plan and a project-specific certification plan. Table 15 shows the major phases and time frames associated with certifying the aircraft equipment of WAAS. Currently, WAAS GPS receivers have been certified and are available for use.

GAO Contacts and Staff Acknowledgments

GAO Contacts

Staff Acknowledgments

In addition to the individuals named above, other key contributors to this report were Geraldine Beard, Gerald Dillingham, Seth Dykes, David Hooper, Kevin Jackson, Gregg Justice III, Donna Leiss, and Kieran McCarthy.