National Airspace System: Status of FAA's Standard Terminal Automation Replacement System
Published: 2002-09-17T00:00:00

Fast Facts:

Highlights:
section_title: What GAO Found
paragraphs: Since September 1996, the Federal Aviation Administration (FAA) has been developing the Standard Terminal Automation Replacement System (STARS) project to replace the outdated computer equipment that air traffic controllers currently use in some facilities to control air traffic within 5 to 50 nautical miles of an airport. Comparing the currently projected cost and deployment schedule for STARS with the original cost and schedule is difficult because the program presently bears little resemblance to the program envisioned in 1996. FAA has officially changed the cost, schedule, and requirements for STARS twice. In October 1999, FAA estimated the cost for its new approach at $1.4 billion, with a schedule to begin deploying STARS in 2002 at 188 facilities and complete installation at all facilities by 2008. The second change occurred in March 2002, when FAA lowered its estimate from $1.4 billion to $1.33 billion, reduced the number of facilities receiving STARS from 188 to 74, and changed the date to complete installation at all facilities from 2008 to 2005. FAA responded to the Department of Transportation Inspector General's concerns about the agency's plans for deploying STARS in Philadelphia by stating that it plans to follow its policy for testing STARS and addressing critical software problems. Because the FAA changed the date for deploying STARS at the first facility from 1998 to 2002, it had to implement interim systems to allow it to continue to meet demands for air traffic services. Specifically, it had to replace radar displays and other hardware that were difficult to maintain and had limited capacity to accommodate software that would allow FAA to add new features. FAA documents show the cost to implement these interim solutions when STARS was delayed was $90.5 million.

section_title: Letter
paragraphs: Since September 1996, the Federal Aviation Administration (FAA) has been  developing the Standard Terminal Automation Replacement System  (STARS) project to replace the outdated computer equipment that air  traffic controllers currently use in some facilities to control air traffic  within 5 to 50 nautical miles of an airport. Under the STARS project,  controllers at both FAA and the Department of Defense (DOD) locations  will receive new hardware and software that produce color displays of  aircraft position and flight information.  In the future, FAA will be able to  upgrade the software to provide air traffic control tools, such as those that  will allow better spacing of aircraft as they descend into airports. The  STARS project is complex, costly, and software intensive. Since 1997, GAO  and the Department of Transportation’s (DOT) Office of the Inspector  General (IG) have reported on the status of STARS. Recently, the DOT IG  raised concerns about FAA’s plans for testing STARS and resolving  problems identified during testing before deploying STARS later this year at  the Philadelphia air traffic control facility. As agreed with your offices, this  report addresses the following questions:  How do the currently projected cost and deployment schedule for  STARS compare with the original cost and schedule?
How often has FAA changed its approved estimates?
How has FAA responded to the DOT IG’s concerns about the agency’s  plans for deploying STARS in Philadelphia?
What has been the impact of changes in the schedule for deploying  STARS?
You also asked how our analysis of the cost and schedule for STARS  compares with FAA’s and the DOT IG’s analyses. We did not perform an  independent analysis of the cost and schedule of STARS, in part because of  time constraints and in part because FAA is reassessing its plans for  modernizing the equipment in terminal facilities. Specifically, FAA is  exploring how it can reduce its future STARS costs by using portions of  equipment developed for STARS with other automated equipment that it  recently deployed to around 140 facilities. FAA officials indicated that they  hope to have information in early spring 2003 for making a decision about  the future direction of the STARS project.
Our work covers FAA’s efforts to develop and deploy STARS; we did not  review similar efforts by DOD. To conduct this work, we reviewed FAA  biweekly status reports, deployment schedules, and test results; and we  interviewed officials from FAA’s program and independent testing offices,  the MITRE Corporation, the DOT IG, and union officials representing the  air traffic controller and maintenance technician workforces. We did not  independently verify cost and performance data that we received from FAA  and did not adjust the costs that FAA and others have reported to reflect  the effects of inflation. We performed our work in August 2002 in  accordance with generally accepted government auditing standards.
subsections: section_title: Background
paragraphs: STARS will replace controller workstations with new color displays,  processors, and computer software at FAA and DOD terminal air traffic  control facilities. (See fig. 1.)  The total number of facilities scheduled to  receive STARS has fluctuated between 70 and 190 because some of the  facilities have received interim systems and may not get full STARS. FAA is  designing STARS to provide a platform that allows easy and rapid  incorporation of new hardware- and software-based tools to help improve  controllers’ productivity and make the nation’s airspace safer and managed  more efficiently.
For each acquisition project that the agency undertakes, FAA officially  estimates, or develops baselines for, the project’s life-cycle costs, schedule,  benefits, and performance in a formal document called the acquisition  program baseline. This baseline, which is approved by the Joint Resources  Council, FAA’s acquisition decision-making body, is used to monitor a  project’s progress in these four areas. The initial acquisition plan for STARS  was approved in March 1996; and in September 1996, FAA signed a contract  with Raytheon Corporation to acquire STARS. The initial strategy for  STARS included two phases:  (1) initial system capability, which was to  provide the same functions as the equipment in use at the time and (2) final  system capability, which was to implement new functions to help  controllers move traffic more safely and efficiently.
FAA’s acquisition policy requires that projects follow a structured and  disciplined test and evaluation process appropriate to the product or  facility being tested. Typically, this process includes system testing and  field familiarization testing. System testing usually includes development  and operational, production, and site acceptance testing. Field  familiarization testing includes system and software testing in an  operational environment to verify operational readiness. Raytheon and  FAA have already conducted a series of tests of the STARS software and  plan to continue such testing. As problems arise during these tests, they are  documented using program trouble reports (PTR) and are classified from  type 1, the most severe, to type 4, the least severe. FAA’s policy defines each  type. The policy states that type-1 PTRs prevent the accomplishment of an  operational or mission-essential capability and could jeopardize safety and  security. Type-2 PTRs adversely affect but does not preclude the  performance of an operational or mission-essential capability and a  workaround solution is not available. Type-3 PTRs adversely affect but  does not preclude the performance of an operational or mission-essential  capability and a workaround solution is available. Type-4 PTRs prevent or  adversely affect the accomplishment of a nonessential capability and can  be handled procedurally. FAA’s contract with Raytheon calls for the  contractor to correct all type-1 and type-2 PTRs and, as directed by the  government, to correct type-3 and type-4 PTRs. The timing of the corrective  action depends, in part, on the severity of the PTR and on its relevance to  upcoming activities.
From the project’s inception until 2001, a multidisciplinary team oversaw  STARS and was responsible for carrying out the acquisition strategy for  implementing the project.  In November 2000, FAA began formulating a  new organization that would be responsible for all terminal modernization  activities. This new organization, the Terminal Business Service, was  intended to move the agency from a project-driven to a point-of-service  approach, which would address performance issues at each facility in an  integrated fashion. This new organization is now responsible for the STARS  program along with other projects for terminal facilities.
section_title: Major STARS Project Changes Make Comparisons of Costs and Schedules Difficult
paragraphs: The current STARS program is not the program that FAA contracted for in  1996. When FAA awarded the contract in September 1996, it estimated that  STARS would cost $940 million and be implemented at 172 facilities by  2005. This estimate was based on acquiring STARS through a commercial  off-the-shelf technology with limited development, since a version of this  technology was already in use in other countries. In 1997, when FAA first  introduced STARS, FAA controllers, who were accustomed to using the  older equipment, began to voice concerns about computer-human interface  issues that could hamper their ability to monitor air traffic. For example,  the controllers noted that many features of the old equipment could be  operated with knobs, allowing controllers to focus on the screen. By  contrast, STARS was menu-driven and required the controllers to make  several keystrokes and use a trackball, diverting their attention from the  screen. The maintenance technicians also identified differences between  STARS and its backup system that made it difficult to monitor the system.  For example, the visual warning alarms and the color codes identifying  problems were not consistent between the two systems.
Addressing these and other issues required extensive software  development because the commercial version of STARS that Raytheon  delivered to FAA very tightly coupled the software for the information that  would be displayed on the screen and the software that would calculate  aircraft position. Because of this coupling, it was difficult for Raytheon to  implement the new or modified display requirements that FAA had  identified. Accordingly, FAA directed Raytheon to separate the display  software from the air traffic control software so that Raytheon could more  efficiently implement future display- and air traffic control-related changes  to each type of software.
To help ensure that STARS meets all of these and other requirements, FAA  is developing multiple versions of STARS software, each with specific  features, and plans to integrate them into a single version, which will be  deployed nationwide. (See fig. 2.)  This incremental approach, according to  FAA, gives air traffic controllers early experience with the software as it is  being developed rather than introducing an entirely new system at the end,  as was the case with the commercially available version. For example, FAA  has developed a version known as early display configuration, which would  replace the controllers’ current displays and monitoring equipment but  would use the existing computer and processing software.
Figure 2 shows FAA’s new strategy for developing STARS software  incrementally. In the early display configurations, FAA separated the  display software from the original commercial version and installed and  tested the display software, together with some of the original software, at  El Paso and Syracuse. In the initial system configuration, FAA took the  original software and added some air traffic control software and tested  this software at Eglin Air Force Base. After each type of software was  tested, FAA began combining the two types to run together in a version  called full STARS 2.  Subsequent versions of full STARS incorporate  additional functions. Figure 3 provides the schedule for when each version  of STARS became or is scheduled to become operational at the first  facility.
section_title: FAA Has Changed Its Official Estimates Twice
paragraphs: Since 1996, FAA acquisition executives have approved two changes to the  cost and schedule estimates for STARS. These changes are presented in  table 1. The October 1999 change was approved to give Raytheon enough  time to add and modify the display software in order to resolve computer- human interface issues. The March 2002 change was approved after FAA  decided to deploy STARS to facilities where frequent equipment failures  caused delays; to new facilities; and to facilities where a digital radar,  needed to operate STARS, is available. Under this strategy, FAA is also  assessing how to deploy STARS to remaining facilities in a cost-effective  manner. Facilities that previously received new hardware and software so  that they could continue to operate while waiting for STARS would get new  technology but may not get the full STARS system.
section_title: FAA Officials Said They Plan to Follow Agency Policy in Testing STARS and Addressing Software Problems but Acknowledge that Challenges Remain
paragraphs: FAA responded to the DOT IG’s concerns about the agency’s plans for  deploying STARS at Philadelphia by stating that FAA plans to follow its  policy for testing STARS and addressing critical software problems.  However, FAA officials, controllers and maintenance technicians all have  concerns about whether required training can be completed by the  November 17, 2002, deployment date.
In June 2002, the DOT IG questioned whether FAA’s commitment to deploy  STARS in Philadelphia before testing it first in Memphis, as planned, would  allow the agency to test the system adequately and address critical  software problems that might be identified before deployment. While the  Memphis terminal facility has fewer and less complex air traffic control  operations than more congested facilities, such as the one in Philadelphia,  FAA changed its plans because meeting the commitment to deploy STARS  in Philadelphia would not allow enough time to test STARS first in  Memphis. FAA testified in September 2001 that it would deploy STARS to  Philadelphia to coincide with the opening of a new terminal, scheduled for  November 17, 2002. FAA officials said they view the achievement of the  November 17, 2002, deployment as important to the agency’s credibility and  that they believe they will learn more from testing STARS in Philadelphia,  which is more representative of terminal facilities, than they would have  learned in Memphis.
According to FAA, its plans for deploying STARS in Philadelphia are  consistent with its testing policy, which calls for independent operational  testing of a system after it has been deployed in one location. Under the  current plan, FAA will use STARS to control live traffic at Philadelphia  beginning on November 17, 2002—a step signifying initial operating  capability—but the current air traffic control system will remain available  as a backup. In accordance with its policy, the agency will then conduct  independent testing after a “period of use,” scheduled from the day after  initial operations through December 2002. At that point, as the policy  directs, the agency will declare the system ready for operational use and  will complete the switch to the new system. At that time, now scheduled  for February 2003, the new system will be formally commissioned and the  current system decommissioned.
To address critical STARS software problems identified prior to deploying  STARS, FAA is attempting to resolve the most critical problems (type-1 and  type-2 PTRs) before November 17, 2002. According to FAA’s definition,  type-1 problems are those that, if not corrected, might prevent the  accomplishment of an operational or mission-essential capability or might  jeopardize safety, while type-2 problems adversely affect but does not  prevent the accomplishment of an operational or mission-critical  capability. FAA’s data showed that as of August 30, 2002, there were 5 type-1 PTRs and 68 type-2 PTRs, against the system being deployed in  Philadelphia, that still need to be resolved. FAA officials stated that they  have assigned these problems to the contractor and plan to validate the  contractor’s fixes. Validation is important because, in some instances, the  fixes have not performed as intended. In addition, FAA has identified at  least 12 type-3 PTRs and other issues, such as completing required training,  that need to be resolved prior to deployment in Philadelphia. FAA is also  meeting biweekly with Raytheon to monitor the contractor’s progress in  implementing and testing fixes for PTRs. In addition, FAA has installed  STARS hardware and an earlier version of STARS software at Philadelphia  so that users can become familiar with the system. On September 19, 2002,  FAA plans to begin testing the most recent STARS software in Philadelphia.
While FAA maintains that its plans for testing STARS and addressing  critical software problems are adequate to address the DOT IG’s concerns,  the agency is less certain that it will be able to complete the certification  training required for maintenance technicians at the Philadelphia terminal  before the new version of STARS begins operation in November. The union  representing maintenance technicians expressed concern because FAA has  not yet finalized the content and schedule of the training for controllers and  maintenance technicians on the software that will be deployed in  Philadelphia. Under a new training agreement between the union and FAA,  on-site certification training—rather than training at FAA’s central facility  in Oklahoma City—is required for all employees before a new system  begins operation. Union officials expressed concern that without a  finalized training schedule, its members will not have enough time to  receive training for certification before the November deployment. FAA  officials acknowledged that having enough time for training is an issue.  Union and FAA officials are working to solve these concerns prior to  deployment. Moreover, according to FAA officials, FAA is meeting with  maintenance technicians and controllers to discuss issues related to  training, as well as maintenance and testing.
section_title: Changes in the Schedule for Deploying STARS Led FAA to Implement Interim Projects
paragraphs: Because FAA was not able to deploy STARS according to its original  schedule, under which some terminals would have received the new  equipment by 1998, FAA implemented several interim projects. Under these  projects, FAA replaced failing equipment with new software, radar  displays, and other hardware so that the terminals could continue  operating while STARS was delayed. Under one project, Common  Automated Radar Terminal System (Common ARTS), FAA procured  common software for the automated equipment at some of its largest  terminal facilities and about 130 smaller facilities.  Common ARTS  provides functions similar to those being designed for STARS, such as the  ability to support simultaneous multiple radar displays and adapt to site  changes. FAA also purchased 294 ARTS color displays, which replaced  aging radar displays at six terminals with those that are high-resolution.   The cost for Common ARTS and the ARTS color displays attributable to  STARS delays was around $90.5 million.
section_title: Agency Comments
paragraphs: We provided a draft of this report to DOT. We met with DOT officials,  including the Director, Terminal Business Service, FAA. These officials  generally agreed with the facts and made technical and clarifying  comments, which we have incorporated into this report as appropriate.
section_title: Scope and Methodology
paragraphs: FAA initially began the Common ARTS project because of delays in a program that  preceded STARS. Under the initial phase of this project, developed by Lockheed Martin  Corporation, equipment was delivered to 131 small- to medium-sized facilities beginning in  1997 and to 5 large facilities in 1998 and 1999. However, FAA later purchased equipment for  five additional facilities, which was installed in 2001 and 2002. documentation on the prioritization of trouble reports and agency policy  and guidance on critical trouble reports and test and evaluation  requirements. To determine the impact of changes in the schedule for  deploying STARS, we reviewed FAA documentation on the interim projects  and the associated costs and also reviewed IG and GAO products on the  impact of delays on implementing STARS. We did not independently verify  the data we received from FAA.
We performed our work in August 2002 in accordance with generally  accepted government auditing standards.
We are sending copies of this report to interested Members of Congress,  the Secretary of Transportation, and the Administrator, FAA. We will also  make copies available to others upon request. In addition, this report will  be available at no charge on the GAO Web site at http://www.gao.gov.
If you or your staff have any questions about this report, please contact me  at (202) 512-3650. I can also be reached by E-mail at dillinghamg@gao.gov.  Key contributors to this report are listed in appendix I.
section_title: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contacts
section_title: Staff Acknowledgments
paragraphs: In addition to those individuals listed above, Nabajyoti Barkakati,  Geraldine Beard,  Elizabeth Eisenstadt, Tammi Nguyen, Madhav Panwar,  and Glenda Wright made key contributions to this report.
section_title: GAO’s Mission
paragraphs: The General Accounting Office, the investigative arm of Congress, exists to  support Congress in meeting its constitutional responsibilities and to help improve  the performance and accountability of the federal government for the American  people. GAO examines the use of public funds; evaluates federal programs and  policies; and provides analyses, recommendations, and other assistance to help  Congress make informed oversight, policy, and funding decisions. GAO’s  commitment to good government is reflected in its core values of accountability,  integrity, and reliability.
section_title: Obtaining Copies of GAO Reports and Testimony
paragraphs: The fastest and easiest way to obtain copies of GAO documents at no cost is  through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full- text files of current reports and testimony and an expanding archive of older  products. The Web site features a search engine to help you locate documents  using key words and phrases. You can print these documents in their entirety,  including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and  correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site  daily. The list contains links to the full-text document files. To have GAO e-mail this  list to you every afternoon, go to www.gao.gov and select “Subscribe to daily E-mail alert for newly released products” under the GAO Reports heading.
subsections: section_title: Order by Mail or Phone
section_title: To Report Fraud, Waste, and Abuse in Federal Programs
section_title: Public Affairs