National Airspace System: Status of FAA's Standard Terminal Automation Replacement System
Released: 2002-09-17T00:00:00

Letter

Background

STARS will replace controller workstations with new color displays, processors, and computer software at FAA and DOD terminal air traffic control facilities. (See fig. 1.) The total number of facilities scheduled to receive STARS has fluctuated between 70 and 190 because some of the facilities have received interim systems and may not get full STARS. FAA is designing STARS to provide a platform that allows easy and rapid incorporation of new hardware- and software-based tools to help improve controllers’ productivity and make the nation’s airspace safer and managed more efficiently.

For each acquisition project that the agency undertakes, FAA officially estimates, or develops baselines for, the project’s life-cycle costs, schedule, benefits, and performance in a formal document called the acquisition program baseline. This baseline, which is approved by the Joint Resources Council, FAA’s acquisition decision-making body, is used to monitor a project’s progress in these four areas. The initial acquisition plan for STARS was approved in March 1996; and in September 1996, FAA signed a contract with Raytheon Corporation to acquire STARS. The initial strategy for STARS included two phases: (1) initial system capability, which was to provide the same functions as the equipment in use at the time and (2) final system capability, which was to implement new functions to help controllers move traffic more safely and efficiently.

FAA’s acquisition policy requires that projects follow a structured and disciplined test and evaluation process appropriate to the product or facility being tested. Typically, this process includes system testing and field familiarization testing. System testing usually includes development and operational, production, and site acceptance testing. Field familiarization testing includes system and software testing in an operational environment to verify operational readiness. Raytheon and FAA have already conducted a series of tests of the STARS software and plan to continue such testing. As problems arise during these tests, they are documented using program trouble reports (PTR) and are classified from type 1, the most severe, to type 4, the least severe. FAA’s policy defines each type. The policy states that type-1 PTRs prevent the accomplishment of an operational or mission-essential capability and could jeopardize safety and security. Type-2 PTRs adversely affect but does not preclude the performance of an operational or mission-essential capability and a workaround solution is not available. Type-3 PTRs adversely affect but does not preclude the performance of an operational or mission-essential capability and a workaround solution is available. Type-4 PTRs prevent or adversely affect the accomplishment of a nonessential capability and can be handled procedurally. FAA’s contract with Raytheon calls for the contractor to correct all type-1 and type-2 PTRs and, as directed by the government, to correct type-3 and type-4 PTRs. The timing of the corrective action depends, in part, on the severity of the PTR and on its relevance to upcoming activities.

From the project’s inception until 2001, a multidisciplinary team oversaw STARS and was responsible for carrying out the acquisition strategy for implementing the project. In November 2000, FAA began formulating a new organization that would be responsible for all terminal modernization activities. This new organization, the Terminal Business Service, was intended to move the agency from a project-driven to a point-of-service approach, which would address performance issues at each facility in an integrated fashion. This new organization is now responsible for the STARS program along with other projects for terminal facilities.

Major STARS Project Changes Make Comparisons of Costs and Schedules Difficult

The current STARS program is not the program that FAA contracted for in 1996. When FAA awarded the contract in September 1996, it estimated that STARS would cost $940 million and be implemented at 172 facilities by 2005. This estimate was based on acquiring STARS through a commercial off-the-shelf technology with limited development, since a version of this technology was already in use in other countries. In 1997, when FAA first introduced STARS, FAA controllers, who were accustomed to using the older equipment, began to voice concerns about computer-human interface issues that could hamper their ability to monitor air traffic. For example, the controllers noted that many features of the old equipment could be operated with knobs, allowing controllers to focus on the screen. By contrast, STARS was menu-driven and required the controllers to make several keystrokes and use a trackball, diverting their attention from the screen. The maintenance technicians also identified differences between STARS and its backup system that made it difficult to monitor the system. For example, the visual warning alarms and the color codes identifying problems were not consistent between the two systems.

Addressing these and other issues required extensive software development because the commercial version of STARS that Raytheon delivered to FAA very tightly coupled the software for the information that would be displayed on the screen and the software that would calculate aircraft position. Because of this coupling, it was difficult for Raytheon to implement the new or modified display requirements that FAA had identified. Accordingly, FAA directed Raytheon to separate the display software from the air traffic control software so that Raytheon could more efficiently implement future display- and air traffic control-related changes to each type of software.

To help ensure that STARS meets all of these and other requirements, FAA is developing multiple versions of STARS software, each with specific features, and plans to integrate them into a single version, which will be deployed nationwide. (See fig. 2.) This incremental approach, according to FAA, gives air traffic controllers early experience with the software as it is being developed rather than introducing an entirely new system at the end, as was the case with the commercially available version. For example, FAA has developed a version known as early display configuration, which would replace the controllers’ current displays and monitoring equipment but would use the existing computer and processing software.

Figure 2 shows FAA’s new strategy for developing STARS software incrementally. In the early display configurations, FAA separated the display software from the original commercial version and installed and tested the display software, together with some of the original software, at El Paso and Syracuse. In the initial system configuration, FAA took the original software and added some air traffic control software and tested this software at Eglin Air Force Base. After each type of software was tested, FAA began combining the two types to run together in a version called full STARS 2. Subsequent versions of full STARS incorporate additional functions. Figure 3 provides the schedule for when each version of STARS became or is scheduled to become operational at the first facility.

FAA Has Changed Its Official Estimates Twice

Since 1996, FAA acquisition executives have approved two changes to the cost and schedule estimates for STARS. These changes are presented in table 1. The October 1999 change was approved to give Raytheon enough time to add and modify the display software in order to resolve computer- human interface issues. The March 2002 change was approved after FAA decided to deploy STARS to facilities where frequent equipment failures caused delays; to new facilities; and to facilities where a digital radar, needed to operate STARS, is available. Under this strategy, FAA is also assessing how to deploy STARS to remaining facilities in a cost-effective manner. Facilities that previously received new hardware and software so that they could continue to operate while waiting for STARS would get new technology but may not get the full STARS system.

FAA Officials Said They Plan to Follow Agency Policy in Testing STARS and Addressing Software Problems but Acknowledge that Challenges Remain

FAA responded to the DOT IG’s concerns about the agency’s plans for deploying STARS at Philadelphia by stating that FAA plans to follow its policy for testing STARS and addressing critical software problems. However, FAA officials, controllers and maintenance technicians all have concerns about whether required training can be completed by the November 17, 2002, deployment date.

In June 2002, the DOT IG questioned whether FAA’s commitment to deploy STARS in Philadelphia before testing it first in Memphis, as planned, would allow the agency to test the system adequately and address critical software problems that might be identified before deployment. While the Memphis terminal facility has fewer and less complex air traffic control operations than more congested facilities, such as the one in Philadelphia, FAA changed its plans because meeting the commitment to deploy STARS in Philadelphia would not allow enough time to test STARS first in Memphis. FAA testified in September 2001 that it would deploy STARS to Philadelphia to coincide with the opening of a new terminal, scheduled for November 17, 2002. FAA officials said they view the achievement of the November 17, 2002, deployment as important to the agency’s credibility and that they believe they will learn more from testing STARS in Philadelphia, which is more representative of terminal facilities, than they would have learned in Memphis.

According to FAA, its plans for deploying STARS in Philadelphia are consistent with its testing policy, which calls for independent operational testing of a system after it has been deployed in one location. Under the current plan, FAA will use STARS to control live traffic at Philadelphia beginning on November 17, 2002—a step signifying initial operating capability—but the current air traffic control system will remain available as a backup. In accordance with its policy, the agency will then conduct independent testing after a “period of use,” scheduled from the day after initial operations through December 2002. At that point, as the policy directs, the agency will declare the system ready for operational use and will complete the switch to the new system. At that time, now scheduled for February 2003, the new system will be formally commissioned and the current system decommissioned.

To address critical STARS software problems identified prior to deploying STARS, FAA is attempting to resolve the most critical problems (type-1 and type-2 PTRs) before November 17, 2002. According to FAA’s definition, type-1 problems are those that, if not corrected, might prevent the accomplishment of an operational or mission-essential capability or might jeopardize safety, while type-2 problems adversely affect but does not prevent the accomplishment of an operational or mission-critical capability. FAA’s data showed that as of August 30, 2002, there were 5 type-1 PTRs and 68 type-2 PTRs, against the system being deployed in Philadelphia, that still need to be resolved. FAA officials stated that they have assigned these problems to the contractor and plan to validate the contractor’s fixes. Validation is important because, in some instances, the fixes have not performed as intended. In addition, FAA has identified at least 12 type-3 PTRs and other issues, such as completing required training, that need to be resolved prior to deployment in Philadelphia. FAA is also meeting biweekly with Raytheon to monitor the contractor’s progress in implementing and testing fixes for PTRs. In addition, FAA has installed STARS hardware and an earlier version of STARS software at Philadelphia so that users can become familiar with the system. On September 19, 2002, FAA plans to begin testing the most recent STARS software in Philadelphia.

While FAA maintains that its plans for testing STARS and addressing critical software problems are adequate to address the DOT IG’s concerns, the agency is less certain that it will be able to complete the certification training required for maintenance technicians at the Philadelphia terminal before the new version of STARS begins operation in November. The union representing maintenance technicians expressed concern because FAA has not yet finalized the content and schedule of the training for controllers and maintenance technicians on the software that will be deployed in Philadelphia. Under a new training agreement between the union and FAA, on-site certification training—rather than training at FAA’s central facility in Oklahoma City—is required for all employees before a new system begins operation. Union officials expressed concern that without a finalized training schedule, its members will not have enough time to receive training for certification before the November deployment. FAA officials acknowledged that having enough time for training is an issue. Union and FAA officials are working to solve these concerns prior to deployment. Moreover, according to FAA officials, FAA is meeting with maintenance technicians and controllers to discuss issues related to training, as well as maintenance and testing.

Changes in the Schedule for Deploying STARS Led FAA to Implement Interim Projects

Because FAA was not able to deploy STARS according to its original schedule, under which some terminals would have received the new equipment by 1998, FAA implemented several interim projects. Under these projects, FAA replaced failing equipment with new software, radar displays, and other hardware so that the terminals could continue operating while STARS was delayed. Under one project, Common Automated Radar Terminal System (Common ARTS), FAA procured common software for the automated equipment at some of its largest terminal facilities and about 130 smaller facilities. Common ARTS provides functions similar to those being designed for STARS, such as the ability to support simultaneous multiple radar displays and adapt to site changes. FAA also purchased 294 ARTS color displays, which replaced aging radar displays at six terminals with those that are high-resolution. The cost for Common ARTS and the ARTS color displays attributable to STARS delays was around $90.5 million.

Agency Comments

We provided a draft of this report to DOT. We met with DOT officials, including the Director, Terminal Business Service, FAA. These officials generally agreed with the facts and made technical and clarifying comments, which we have incorporated into this report as appropriate.

Scope and Methodology

FAA initially began the Common ARTS project because of delays in a program that preceded STARS. Under the initial phase of this project, developed by Lockheed Martin Corporation, equipment was delivered to 131 small- to medium-sized facilities beginning in 1997 and to 5 large facilities in 1998 and 1999. However, FAA later purchased equipment for five additional facilities, which was installed in 2001 and 2002. documentation on the prioritization of trouble reports and agency policy and guidance on critical trouble reports and test and evaluation requirements. To determine the impact of changes in the schedule for deploying STARS, we reviewed FAA documentation on the interim projects and the associated costs and also reviewed IG and GAO products on the impact of delays on implementing STARS. We did not independently verify the data we received from FAA.

We performed our work in August 2002 in accordance with generally accepted government auditing standards.

We are sending copies of this report to interested Members of Congress, the Secretary of Transportation, and the Administrator, FAA. We will also make copies available to others upon request. In addition, this report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-3650. I can also be reached by E-mail at dillinghamg@gao.gov. Key contributors to this report are listed in appendix I.

GAO Contacts and Staff Acknowledgments

GAO Contacts

Staff Acknowledgments

In addition to those individuals listed above, Nabajyoti Barkakati, Geraldine Beard, Elizabeth Eisenstadt, Tammi Nguyen, Madhav Panwar, and Glenda Wright made key contributions to this report.

GAO’s Mission

The General Accounting Office, the investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony

The fastest and easiest way to obtain copies of GAO documents at no cost is through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full- text files of current reports and testimony and an expanding archive of older products. The Web site features a search engine to help you locate documents using key words and phrases. You can print these documents in their entirety, including charts and other graphics.

Each day, GAO issues a list of newly released reports, testimony, and correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site daily. The list contains links to the full-text document files. To have GAO e-mail this list to you every afternoon, go to www.gao.gov and select “Subscribe to daily E-mail alert for newly released products” under the GAO Reports heading.

Order by Mail or Phone

To Report Fraud, Waste, and Abuse in Federal Programs

Public Affairs