Air Traffic Control: System Management Capabilities Improved, but More Can Be Done to Institutionalize Improvements
Released: 2004-09-20T00:00:00

Letter

Introduction

The primary mission of the Federal Aviation Administration (FAA) is to provide a safe, secure, and efficient global aerospace system that contributes to national security and the promotion of U.S. aerospace safety. FAA’s ability to fulfill this mission depends on the adequacy and reliability of the nation’s air traffic control (ATC) systems—a vast network of computer hardware, software, and communications equipment.

To accommodate forecasted growth in air traffic and to relieve the problems of aging ATC systems, FAA embarked on an ambitious ATC modernization program in 1981. FAA now estimates that it will spend about $51 billion to replace and modernize ATC systems through 2007. Our work over the years has chronicled many FAA problems in meeting ATC projects’ cost, schedule, and performance goals. As a result of these issues as well as the tremendous cost, complexity, and mission criticality of the modernization program, we designated the program as a high-risk information technology initiative in 1995, and it has remained on our high- risk list since that time.

Overview of ATC

Automated information processing and display, communication, navigation, surveillance, and weather resources permit air traffic controllers to view key information—such as aircraft location, aircraft flight plans, and prevailing weather conditions—and to communicate with pilots. These resources reside at, or are associated with, several ATC facilities—ATC towers, terminal radar approach control facilities, air route traffic control centers (en route centers), flight service stations, and the ATC System Command Center. Figure 2 shows a visual summary of ATC over the continental United States and oceans.

FAA’s ATC Modernization Is a High-Risk Initiative

Faced with growing air traffic and aging equipment, in 1981, FAA initiated an ambitious effort to modernize its ATC system. This effort involves the acquisition of new surveillance, data processing, navigation, and communications equipment, in addition to new facilities and support equipment. Initially, FAA estimated that its ATC modernization effort would cost $12 billion and could be completed over 10 years. Now, 2 decades and $35 billion later, FAA expects to need another $16 billion through 2007 to complete key projects, for a total cost of $51 billion.

Over the past 2 decades, many of the projects that make up the modernization program have experienced substantial cost overruns, schedule delays, and significant performance shortfalls. Our work over the years has documented many of these shortfalls. As a result of these problems, as well as the tremendous cost, complexity, and mission criticality of the modernization program, we designated the program as a high-risk information technology initiative in 1995, and it has remained on our high-risk list since that time.

Our work since the mid-1990s has pinpointed root causes of the modernization program’s problems, including (1) immature software acquisition capabilities, (2) lack of a complete and enforced system architecture, (3) inadequate cost estimating and cost accounting practices, (4) an ineffective investment management process, and (5) an organizational culture that impaired the acquisition process. We have made over 30 recommendations to address these issues, and FAA has made substantial progress in addressing them. Nonetheless, in our most recent high-risk report, we noted that more remains to be done—and with FAA still expecting to spend billions on new ATC systems, these actions are as critical as ever.

Prior Report Noted Weaknesses in FAA’s Software Acquisition Capabilities

In March 1997, we reported that FAA’s processes for acquiring software, the most costly and complex component of its ATC systems, were ad hoc, sometimes chaotic, and not repeatable across projects. We also reported that the agency lacked an effective management structure for ensuring software process improvement. As a result, the agency was at great risk of not delivering promised software capabilities on time and within budget. We recommended that FAA establish a Chief Information Officer organizational structure, as prescribed in the Clinger-Cohen Act, and assign responsibility for software acquisition process improvement to this organization. We also recommended several actions intended to help FAA improve its software acquisition capabilities by institutionalizing mature processes. These included developing a comprehensive plan for process improvement, allocating adequate resources to ensure that improvement efforts were implemented, and requiring that projects achieve a minimum level of maturity before being approved.

FAA has implemented most of our recommendations. The agency established a Chief Information Officer position that reports directly to the administrator and gave this position responsibility for process improvement. The Chief Information Officer’s process improvement office developed a strategy and led the way in developing an integrated framework for improving maturity in system acquisition, development, and engineering processes. Some of the business organizations within FAA, including the organizations responsible for ATC acquisitions and operations, adopted the framework and provided resources to process improvement efforts.

FAA did not, however, implement our recommendation to require that projects achieve a minimum level of maturity before being approved. Officials reported that rather than establish arbitrary thresholds for maturity, FAA intended to evaluate process areas that were most critical or at greatest risk for each project during acquisition management reviews.

FAA Is Changing Its Approach to ATC Management

Recent legislation and an executive order have led to major changes in the way that FAA manages its ATC mission. In April 2000, the Wendell H. Ford Aviation Investment and Reform Act for the 21st Century (Air-21) established the position of Chief Operating Officer for the ATC system. In December 2000, executive order 13180 instructed FAA to establish a performance-based organization known as the Air Traffic Organization and to have the Chief Operating Officer lead this organization under the authority of the FAA administrator. This order, amended in June 2002, called for the Air Traffic Organization to enhance the FAA’s primary mission of ensuring the safety, security, and efficiency of the National Airspace System and further improve the delivery of air traffic services to the American public by reorganizing air traffic services and related offices into a performance-based, results-oriented organization. The order noted that as a performance-based organization, the Air Traffic Organization would be able to take better advantage of the unique procurement and personnel authorities currently used by FAA, as well as of the additional management reforms enacted by Congress under Air-21. In addition, the Air Traffic Organization is responsible for developing methods to accelerate ATC modernization, improving aviation safety related to ATC, and establishing strong incentives to agency managers for achieving results.

In leading the new Air Traffic Organization, the Chief Operating Officer’s responsibilities include establishing and maintaining organizational and individual goals, a 5-year strategic plan including ATC system mission and objectives, and a framework agreement with the Administrator to establish the new organization’s relationships with other FAA organizations. In August 2003, the first Chief Operating Officer joined the agency and initiated a reorganization combining the separate ATC-related organizations and offices into the Air Traffic Organization.

The Capability Maturity Model Integration Provides a Means of Assessing an Organization’s Ability to Manage Software and System Acquisition and Development

An essential aspect of FAA’s ATC modernization program is the quality of the software and systems involved, which is heavily influenced by the quality and maturity of the processes used to acquire, develop, manage, and maintain them. Carnegie Mellon University’s Software Engineering Institute (SEI), recognized for its expertise in software and system processes, has developed the Capability Maturity Model Integration (CMMI) and a CMMI appraisal methodology to evaluate, improve, and manage system and software development and engineering processes. The CMMI model and appraisal methodology provide a logical framework for measuring and improving key processes needed for achieving high-quality software and systems. The model can help an organization set process improvement objectives and priorities and improve processes; the model can also provide guidance for ensuring stable, capable, and mature processes. According to SEI, organizations that implement such process improvements can achieve better project cost and schedule performance and higher quality products.

In brief, the CMMI model identifies 25 process areas—clusters of related practices that, when performed collectively, satisfy a set of goals that are considered important for making significant improvements in that area. Table 1 describes these process areas.

The CMMI model provides two alternative ways to view these process areas. One way, called continuous representation, focuses on improving capabilities in individual process areas. The second way, called staged representation, groups process areas together and focuses on achieving increased maturity levels by improving the group of process areas.

The CMMI appraisal methodology calls for assessing process areas by determining whether the key practices are implemented and whether the overarching goals are satisfied. Under continuous representation, successful implementation of these practices and satisfaction of these goals result in the achievement of successive capability levels in a selected process area. CMMI capability levels range from 0 to 5, with level 0 meaning that the process is either not performed or partially performed; level 1 meaning that the basic process is performed; level 2 meaning that the process is managed; level 3 meaning that the processes is defined throughout the organization; level 4 meaning that the process is quantitatively managed; and level 5 meaning that the process is optimized. Figure 3 provides details on CMMI capability levels.

Objectives, Scope, and Methodology

The Chairman, House Committee on Government Reform, and the Chairman of that Committee’s Subcommittee on Technology, Information Policy, Intergovernmental Relations and the Census requested that we evaluate FAA’s software and system development processes used to manage its ATC modernization. Our objectives were (1) to evaluate FAA’s capabilities for developing and acquiring software and systems on its ATC modernization program and (2) to assess the actions FAA has under way to improve these capabilities.

To evaluate FAA’s capabilities for developing and acquiring software and systems, we applied the CMMI model (continuous representation) and its related appraisal methodology to four FAA projects. Our appraisers were all SEI-trained software and information systems specialists. In addition, we employed SEI-trained consultants as advisors on our first evaluation to ensure proper application of the model and appraisal methodology.

In consultation with FAA officials, we selected four FAA projects with high impact, visibility, and cost, which represented different air traffic domains and reflected different stages of life cycle development. The projects included the Voice Switching and Control System (VSCS), the Integrated Terminal Weather System (ITWS), the En Route Automation Modernization (ERAM) project, and the Airport Surface Detection Equipment–Model X (ASDE-X). The four projects are described in table 2.

In conjunction with FAA’s process improvement organization, we identified relevant CMMI process areas for each appraisal. In addition, because system deployment is an important aspect of FAA systems management that is not included in CMMI, we used the deployment, transition, and disposal process area from FAA’s integrated Capability Maturity Model, version 2. For consistency, we merged FAA’s criteria with SEI’s framework and added the standard goals and practices needed to achieve capability level 2.

In selected cases, we did not review a certain process area because it was not relevant to the current stage of a project’s life cycle. For example, we did not evaluate supplier agreement management or deployment on VSCS because the system is currently in operation, and these process areas are no longer applicable to this system. Table 3 displays the CMMI process areas that we reviewed for each project.

For each process area reviewed, we evaluated project-specific documentation and interviewed project officials to determine whether key practices were implemented and goals were achieved. In accordance with CMMI guidance, we characterized practices as fully implemented, largely implemented, partially implemented, and not implemented, and characterized goals as satisfied or unsatisfied. After combining the practices and goals, the team determined if successive capability levels were achieved. According to the CMMI appraisal method, practices must be largely or fully implemented in order for a goal to be satisfied. Further, all goals must be satisfied in order to achieve a capability level. In order to achieve advanced capability levels, all preceding capability levels must be achieved. For example, a prerequisite for level 2 is the achievement of level 1. As agreed with FAA process improvement officials, we evaluated the projects through capability level 2. Consistent with the CMMI appraisal methodology, we validated our findings by sharing preliminary observations with the project team so that they were able to provide additional documentation or information as warranted.

To assess the actions FAA has under way to improve its system and software acquisition and development processes, we evaluated process improvement strategies and plans. We also evaluated the progress the agency has made in expanding its process improvement initiative, both through the maturity of the model and the acceptance of the model by project teams. We also interviewed officials from the offices of the Chief Information Officer and the Chief Operating Officer to determine the effect current changes in the ATC organization could have on the process improvement initiatives.

The Department of Transportation and FAA provided oral comments on a draft of this report. These comments are presented in chapter 17. We performed our work from September 2003 through July 2004 in accordance with generally accepted government auditing standards.

FAA Is Performing Most Project Planning Practices, but It Is Not Yet Fully Managing the Process

The purpose of project planning is to establish and maintain plans that define the project activities. This process area involves developing and maintaining a plan, interacting with stakeholders, and obtaining commitment to the plan.

As figure 4 shows, three of the four FAA projects satisfied all criteria for the “performing” capability level (level 1) in this process area. The fourth project would have achieved level 1 if it had performed one more practice (see the overview in table 4 for details). None of the four projects satisfied all criteria for the “managing” capability level (level 2). While all four projects had differing weaknesses that contributed to this result, common weaknesses across most of the projects occurred in the areas of monitoring and controlling the project planning process and in ensuring quality assurance of the process. As a result of these weaknesses, FAA is exposed to increased risks that projects will not meet cost, schedule, or performance goals and that projects will not meet mission needs.

Looked at another way, of the 96 practices we evaluated in this process area, FAA projects had 88 practices that were fully or largely implemented and 8 practices that were partially or not implemented.

Additional details on each project’s appraisal results at successive capability levels are provided in tables 5 through 12. Specifically, tables 5 and 6 provide results for VSCS; tables 7 and 8 provide results for ERAM; tables 9 and 10 provide results for ITWS; and tables 11 and 12 provide results for ASDE-X.

FAA Is Performing Most Project Monitoring and Control Practices, but It Is Not Yet Fully Managing the Process

The purpose of project monitoring and control is to provide an understanding of the project’s progress so that appropriate corrective actions can be taken when the project’s performance deviates significantly from the plan. Key activities include monitoring activities, communicating status, taking corrective action, and determining progress.

As shown in figure 5, three of the four FAA projects satisfied all criteria for the “performing” capability level (level 1) in this process area. The fourth project would have achieved level 1 if it had performed one more practice (see the overview in table 13 for details). None of the four projects satisfied all criteria for the “managing” capability level (level 2). While the projects had differing weaknesses that contributed to this result, a common weakness across most of the projects occurred in the area of ensuring quality assurance of the process. As a result of this weakness, FAA is exposed to increased risks that projects will not meet cost, schedule, or performance goals and that projects will not meet mission needs.

Looked at another way, of the 80 practices we evaluated in this process area, FAA projects had 74 practices that were fully or largely implemented and 6 practices that were partially or not implemented.

Additional details on each project’s appraisal results at successive capability levels are provided in tables 14 through 21. Specifically, tables 14 and 15 provide results for VSCS; tables 16 and 17 provide results for ERAM; tables 18 and 19 provide results for ITWS; and tables 20 and 21 provide results for ASDE-X.

FAA Is Performing Most Risk Management Practices, but It Is Not Yet Fully Managing the Process

The purpose of risk management is to identify potential problems before they occur, so that risk-handling activities may be planned and invoked as needed across the life of the product or project to mitigate adverse impacts on achieving objectives. Effective risk management includes early and aggressive identification of risks through the involvement of relevant stakeholders. Early and aggressive detection of risk is important, because it is typically easier, less costly, and less disruptive to make changes and correct work efforts during the earlier phases of the project.

As shown in figure 6, three of the four FAA projects satisfied all criteria for the “performing” capability level (level 1) in this process area. The fourth project would have achieved level 1 if it had performed one more practice (see the overview in table 22 for details). Two of the four FAA projects also satisfied all criteria for the “managed” capability level (level 2) in this process area. While the other projects had differing weaknesses that contributed to this result, common weaknesses across some of the projects occurred in the area of monitoring and controlling the risk management process and in ensuring quality assurance of the process. As a result of these weaknesses, FAA faces increased likelihood that project risks will not be identified and addressed in a timely manner—thereby increasing the likelihood that projects will not meet cost, schedule, or performance goals.

Looked at another way, of the 68 practices we evaluated in this key process area, FAA projects had 59 practices that were fully or largely implemented and 9 practices that were partially or not implemented.

Additional details on each project’s appraisal results at successive capability levels are provided in tables 23 through 30. Specifically, tables 23 and 24 provide results for VSCS; tables 25 and 26 provide results for ERAM; tables 27 and 28 provide results for ITWS; and tables 29 and 30 provide results for ASDE-X.

FAA Is Performing Requirements Development Practices, but It Is Not Yet Fully Managing the Process

The purpose of requirements development is to produce and analyze customer, product, and product-component needs. This process area addresses the needs of relevant stakeholders, including those pertinent to various product life-cycle phases. It also addresses constraints caused by the selection of design solutions. The development of requirements includes elicitation, analysis, validation, and communication of customer and stakeholder needs and expectations.

As shown in figure 7, all four FAA projects satisfied all criteria for the “performing” capability level (level 1) in this process area. None of the four projects satisfied all criteria for the “managing” capability level (level 2). While all four projects had differing weaknesses that contributed to this result, common weaknesses across multiple projects occurred in the areas of training people and in ensuring quality assurance of the requirements development process, as shown in the overview in table 31. As a result of these weaknesses, FAA is exposed to increased risks that projects will not fulfill mission and user needs.

Looked at another way, of the 84 practices we evaluated in this key process area, FAA projects had 77 practices that were fully or largely implemented and 7 practices that were partially or not implemented.

Additional details on each project’s appraisal results at successive capability levels are provided in tables 32 through 39. Specifically, tables 32 and 33 provide results for VSCS; tables 34 and 35 provide results for ERAM; tables 36 and 37 provide results for ITWS; and tables 38 and 39 provide results for ASDE-X.

FAA Is Performing Requirements Management Practices, but It Is Not Yet Fully Managing the Process

The purpose of requirements management is to manage the project’s product components and to identify inconsistencies between requirements and the project’s plans and work products. This process area includes managing all technical and nontechnical requirements and any changes to these requirements as they evolve.

As shown in figure 8, all four FAA projects satisfied all criteria for the “performing” capability level (level 1) in this process area, but none satisfied all criteria for achieving a “managed” capability level (level 2). While the projects had differing weaknesses that contributed to this result, a common weakness across most of the projects occurred in the area of ensuring quality assurance of the requirements management process, as shown in the overview in table 40. As a result of these weaknesses, FAA is exposed to increased risks that projects will not fulfill mission and user needs.

Looked at another way, of the 60 practices we evaluated in this key process area, FAA projects had 54 practices that were fully or largely implemented and 6 practices that were partially or not implemented.

Additional details on each project’s appraisal results at successive capability levels are provided in tables 41 through 48. Specifically, tables 41 and 42 provide results for VSCS; tables 43 and 44 provide results for ERAM; tables 45 and 46 provide results for ITWS; and tables 47 and 48 provide results for ASDE-X.

FAA Is Performing Most Technical Solution Practices, but It Is Not Yet Fully Managing the Process

The purpose of the technical solution process area is to design, develop, and implement products, product components, and product-related life- cycle processes to meet requirements. This process involves evaluating and selecting solutions that potentially satisfy an appropriate set of allocated requirements, developing detailed designs, and implementing the design.

As shown in figure 9, three FAA projects satisfied all criteria for the “performing” capability level (level 1) in this process area. The fourth project would have achieved level 1 if it had performed two more practices (see the overview in table 49 for details). None of the four projects satisfied all criteria for the “managing” capability level (level 2). While all four projects had differing weaknesses that contributed to this result, common weaknesses across most of the projects occurred in the area of ensuring quality assurance of the technical solution process. As a result of this weakness, FAA is exposed to increased risks that projects will not meet mission needs.

Looked at another way, of the 72 practices we evaluated in this key process area, FAA projects had 62 practices that were fully or largely implemented and 10 practices that were partially or not implemented.

Additional details on each project’s appraisal results at successive capability levels are provided in tables 50 through 57. Specifically, tables 50 and 51 provide results for VSCS; tables 52 and 53 provide results for ERAM; tables 54 and 55 provide results for ITWS; and tables 56 and 57 provide results for ASDE-X.

FAA Is Performing Product Integration Practices, but It Is Not Yet Fully Managing the Process

The purpose of the product integration process is to assemble the product components, ensure that the integrated product functions properly, and deliver the product. A critical aspect of this process is managing the internal and external interfaces of the products and product components, in one stage or in incremental stages.

For this process area, we did not perform an appraisal for the ERAM project, because it was at a stage in which product integration was not applicable. As shown in figure 10, the three remaining projects satisfied all criteria for the “performing” capability level (level 1) in this process area. None of the projects satisfied all criteria for the “managing” capability level (level 2). While the projects had differing weaknesses that contributed to this result, common weaknesses across most of the projects occurred in the areas of monitoring and controlling the product integration process and ensuring quality assurance of the process, as shown in the overview in table 58. As a result of this weakness, FAA is exposed to increased risk that product components will not be compatible, resulting in projects that will not meet cost, schedule, or performance goals.

Looked at another way, of the 54 practices we evaluated in this process area, FAA projects had 49 practices that were fully or largely implemented and 5 practices that were partially or not implemented.

Additional details on each project’s appraisal results at successive capability levels are provided in tables 59 through 64. Specifically, tables 59 and 60 provide results for VSCS; tables 61 and 62 provide results for ITWS; and tables 63 and 64 provide results for ASDE-X.

FAA Is Not Performing Key Verification Practices or Fully Managing the Process

The purpose of verification is to ensure that selected work products meet their specified requirements. This process area involves preparing for and performing tests and identifying corrective actions. Verification of work products substantially increases the likelihood that the product will meet the customer, product, and product-component requirements.

As shown in figure 11, only one of four FAA projects satisfied all criteria for the “performing” capability level (level 1) in this process area. As shown in the overview in table 65, key weaknesses in preparing and conducting peer reviews prevented the other three projects from achieving level 1. None of the four projects satisfied all criteria for the “managing” capability level (level 2). While all four projects had differing weaknesses that contributed to this result, common weaknesses across most of the projects occurred in the areas of monitoring and controlling the verification process and in ensuring quality assurance of the process. As a result of these weaknesses, FAA is exposed to increased risk that the product will not meet the user and mission requirements, increasing the likelihood that projects that will not meet cost, schedule, or performance goals.

Looked at another way, of the 68 practices we evaluated in this process area, FAA projects had 51 practices that were fully or largely implemented and 17 practices that were partially or not implemented.

Additional details on each project’s appraisal results at successive capability levels are provided in tables 66 through 73. Specifically, tables 66 and 67 provide results for VSCS; tables 68 and 69 provide results for ERAM; tables 70 and 71 provide results for ITWS; and tables 72 and 73 provide results for ASDE-X.

FAA Is Performing Validation Practices, but It Is Not Yet Fully Managing the Process

The purpose of validation is to demonstrate that a product or product component fulfills its intended use when placed in its intended environment. Validation activities are vital to ensuring that the products are suitable for use in their intended operating environment.

As shown in figure 12, all four FAA projects satisfied all criteria for the “performing” capability level (level 1) in this process area. None of the four projects satisfied all criteria for the “managing” capability level (level 2). While all four projects had differing weaknesses that contributed to this result, common weaknesses across most of the projects occurred in the areas of monitoring and controlling the validation process and in ensuring quality assurance of the process, as shown in the overview in table 74. As a result of these weaknesses, FAA is exposed to increased risk that the project will not fulfill its intended use, thereby increasing the likelihood that the projects will not meet cost, schedule, or performance goals.

Looked at another way, of the 56 practices we evaluated in this process area, FAA projects had 47 practices that were fully or largely implemented and 9 practices that were partially or not implemented.

Additional details on each project’s appraisal results at successive capability levels are provided in tables 75 through 82. Specifically, tables 75 and 76 provide results for VSCS; tables 77 and 78 provide results for ERAM; tables 79 and 80 provide results for ITWS; and tables 81 and 82 provide results for ASDE-X.

FAA Is Performing Most Configuration Management Practices, but It Is Not Yet Fully Managing the Process

The purpose of configuration management is to establish and maintain the integrity of work products. This process area includes both the functional processes used to establish and track work product changes and the technical systems used to manage these changes. Through configuration management, accurate status and data are provided to developers, end users, and customers.

As shown in figure 13, three of the four FAA projects satisfied all criteria for the “performing” capability level (level 1) in this process area. The fourth project would have achieved level 1 if it had performed two more practices (see the overview in table 83 for details). Only one of the four projects satisfied all criteria for the “managing” capability level (level 2). While all four projects had differing weaknesses that contributed to this result, common weaknesses across some of the projects occurred in the areas of monitoring and controlling the process and in ensuring the quality assurance of the configuration management process, as shown in the overview in table 83. As a result of these weaknesses, FAA is exposed to increased risk that the project teams will not effectively manage their work products, resulting in projects that do not meet cost, schedule, or performance goals.

Looked at another way, of the 68 practices we evaluated in this process area, FAA projects had 60 practices that were fully or largely implemented and 8 practices that were partially or not implemented.

Additional details on each project’s appraisal results at successive capability levels are provided in tables 84 through 91. Specifically, tables 84 and 85 provide results for VSCS; tables 86 and 87 provide results for ERAM; tables 88 and 89 provide results for ITWS; and tables 90 and 91 provide results for ASDE-X.

FAA Is Not Performing Key Process and Product Quality Assurance Practices or Managing the Process

The purpose of process and product quality assurance is to provide staff and management with objective insights into processes and associated work products. This process area includes the objective evaluation of project processes and products against approved descriptions and standards. Through process and product quality assurance, the project is able to identify and document noncompliance issues and provide appropriate feedback to project members.

As shown in figure 14, only one of the four FAA projects satisfied all criteria for the “performing” capability level (level 1) in this process area. Weaknesses in the objective evaluation of designated performed processes, work products, and services against the applicable process descriptions, standards, and procedures prevented the projects from achieving level 1. None of the four projects satisfied all criteria for the “managing” capability level (level 2). Table 92 provides an overview of our appraisal results. As shown in the table, while the four projects had differing weaknesses that contributed to this result, common weaknesses across multiple projects occurred in the areas of establishing a plan, providing resources, training people, providing configuration management, identifying stakeholders, monitoring and controlling the process, ensuring quality assurance, and reviewing the status of the quality assurance process with higher level managers. As a result of these weaknesses, FAA is exposed to increased risk that the projects will not effectively implement key management processes, resulting in projects that will not meet cost, schedule, or performance goals, and that will not meet mission needs.

Looked at another way, of the 56 practices we evaluated in this process area, FAA projects had 33 practices that were fully or largely implemented and 23 practices that were partially or not implemented.

Additional details on each project’s appraisal results at successive capability levels are provided in tables 93 through 100. Specifically, tables 93 and 94 provide results for VSCS; tables 95 and 96 provide results for ERAM; tables 97 and 98 provide results for ITWS; and tables 99 and 100 provide results for ASDE-X.

FAA Is Not Performing Most Measurement and Analysis Practices or Managing the Process

The purpose of measurement and analysis is to develop and sustain a measurement capability that is used to support management information needs. This process area includes the specification of measures, data collection and storage, analysis techniques, and the reporting of these values. This process allows users to objectively plan and estimate project activities and identify and resolve potential issues.

As shown in figure 15, none of the four FAA projects satisfied all criteria for the “performing” capability level (level 1) in this process area. Weaknesses in managing and storing measurement data, measurement specifications, and analysis results kept the projects from achieving level 1. Further, none of the four projects satisfied all criteria for the “managing” capability level (level 2). As shown in the overview in table 101, while the four projects had differing weaknesses that contributed to this result, common weaknesses across multiple projects occurred in the areas of establishing an organizational policy, establishing a plan, providing resources, assigning responsibility, training people, configuration management, identifying stakeholders, monitoring and controlling the process, ensuring quality assurance, and reviewing status with higher level management of the measurement and analysis process. As a result of these weaknesses, FAA is exposed to increased risk that the projects will not have adequate estimates of work metrics or a sufficient view into actual performance. This increases the likelihood that projects will not meet cost, schedule, or performance goals, and that projects will not meet mission needs.

Looked at another way, of the 72 practices we evaluated in this process area, FAA projects had 30 practices that were fully or largely implemented and 42 practices that were partially or not implemented.

Additional details on each project’s appraisal results at successive capability levels are provided in tables 102 through 109. Specifically, tables 102 and 103 provide results for VSCS; tables 104 and 105 provide results for ERAM; tables 106 and 107 provide results for ITWS; and tables 108 and 109 provide results for ASDE-X.

FAA Is Performing Supplier Agreement Management Practices, but It Is Not Yet Fully Managing the Process

The purpose of supplier agreement management is to manage the acquisition of products. This process area involves determining the type of acquisition that will be used for the products acquired, selecting suppliers, establishing, maintaining, and executing agreements, accepting delivery of acquired products, and transitioning acquired products to the project, among other items.

For this process area, we did not perform an appraisal for the VSCS or ITWS projects, because these projects were at stages in which supplier agreement management was not applicable. As shown in figure 16, both of the remaining FAA projects satisfied all criteria for the “performing” capability level (level 1) in this process area. One of the two projects satisfied all criteria for the “managing” capability level (level 2). In not consistently managing this process, FAA is exposed to increased risk that projects will not be performed in accordance with contractual requirements, resulting in projects that will not meet cost, schedule, or performance goals, and systems that will not meet mission needs.

Looked at another way, of the 34 practices we evaluated in this process area, FAA projects had 33 practices that were fully or largely implemented and 1 practice that was partially implemented. Table 110 provides an overview of the appraisal results.

Additional details on each project’s appraisal results at successive capability levels are provided in tables 111 through 114. Specifically, tables 111 and 112 provide results for ERAM, and tables 113 and 114 provide results for ASDE-X.

FAA Is Performing Deployment, Transition, and Disposal Practices, but It Is Not Yet Fully Managing the Process

The purpose of the deployment, transition, and disposal process area is to place a product or service into an operational environment, transfer it to the customer and to the support organization, and deactivate and dispose of the replaced product or dispense with the service. This process area includes the design and coordination of plans and procedures for placement of a product or service into an operational or support environment and bringing it into operational use. It ensures that an effective support capability is in place to manage, maintain, and modify the supplied product or service. It further ensures the successful transfer of the product or service to the customer/stakeholder and the deactivation and disposition of the replaced capability.

For this process area, we did not perform an appraisal for the VSCS or ERAM projects, because these projects were at stages in which deployment was not applicable. As shown in figure 17, both of the remaining FAA projects satisfied all criteria for the “performing” capability level (level 1) in this process area. Neither satisfied all criteria for the “managing” capability level (level 2). As shown in the overview in table 115, while the projects had differing weaknesses that contributed to this result, a common weakness across projects occurred in the area of monitoring and controlling the deployment process. As a result of this weakness, FAA is exposed to increased risk that the projects will not be delivered on time, resulting in projects that will not meet cost, schedule, or performance goals.

Looked at another way, of the 32 practices we evaluated in this process area, FAA projects had 28 practices that were fully or largely implemented and 4 practices that were partially implemented.

Additional details on each project’s appraisal results at successive capability levels are provided in tables 116 through 119. Specifically, tables 116 and 117 provide results for ITWS, and tables 118 and 119 provide results for ASDE-X.

FAA’s Process Improvement Initiative Has Matured, but It Is Not Yet Institutionalized

Since our 1997 report, the Federal Aviation Administration’s (FAA) process improvement initiative has grown tremendously in rigor and scope. In our earlier appraisal, we found that FAA’s performance of key processes was ad hoc and sometimes chaotic, whereas current results show that FAA projects are performing most key practices. However, these process improvement activities are not required throughout the air traffic organizations, and the recurring weaknesses we identified in our project- specific evaluations are due in part to the choices these projects were given in deciding whether to and how to adopt process improvement initiatives. Further, because of a recent reorganization, the new Air Traffic Organization’s commitment to this process improvement initiative is not certain. As a result, FAA is not consistent in its adoption and management of process improvement efforts, so that individual projects’ costs, schedules, and performance remain at risk. Without agencywide adoption of process improvement initiatives, the agency cannot increase the maturity of its organizational capabilities.

FAA’s Process Improvement Initiative Has Matured

Over the past several years, FAA has made considerable progress in improving its processes for acquiring and developing software and systems. Acting on our prior recommendations, in 1999, FAA established a centralized process improvement office that reports directly to the Chief Information Officer. This office led the government in an effort to integrate various standards and models into a single maturity model, called the integrated Capability Maturity Model (iCMM). In fact, FAA’s iCMM served as a demonstration for the Software Engineering Institute’s effort to integrate various models into its own Capability Maturity Model Integration (CMMI).

The Chief Information Officer’s process improvement office also developed and sponsored iCMM-related training, and by late 2003, it had trained over 7,000 participants. The training offered ranges from overviews on how to use the model to more focused courses in such specific process areas as quality assurance, configuration management, and project management. The office also guides FAA organizations in using the model and leads appraisal teams in evaluating the process maturity of the projects and organizations that adopted the model.

In addition to the Chief Information Officer–sponsored process improvement efforts, several of FAA’s business areas, including the business areas with responsibility for air traffic control (ATC) system acquisitions and operations, endorsed and set goals for process improvement activities using the iCMM. As a result, there has been a continuing growth over the years in the number of individual projects and umbrella organizations that adopted process improvement and the iCMM model. Specifically, the number of projects and organizations (which account for multiple projects) undergoing iCMM appraisals grew from 1 project in 1997, to 28 projects and 3 organizations by 2000, to 39 projects and 11 organizations by 2003.

These projects and organizations have demonstrated improvements in process maturity. Under the iCMM model, in addition to achieving capability levels in individual process areas, entities can achieve successive maturity levels by demonstrating capabilities in a core set of process areas. FAA process improvement officials reported that by 2000, 10 projects and one organization had achieved iCMM maturity level 2. To date, 14 projects and three organizations have achieved iCMM maturity level 2, and one project and two organizations have achieved iCMM maturity level 3. Additionally, 13 projects and four organizations achieved capability levels 2 or 3 in one or more process areas.

Moreover, in internal surveys, the programs and organizations pursuing process improvement have consistently reported enhanced productivity, higher quality, increased ability to predict schedules and resources, higher morale, and better communication and teamwork. These findings are reiterated by the Software Engineering Institute in its recent study of the benefits of using the CMMI model for process improvement. According to that study, organizations that implement such process improvements can achieve better project cost and schedule performance and higher quality products. Specifically, of the 12 cases that the Software Engineering Institute assessed, there were nine examples of cost related benefits, including reductions in the cost to find and fix a defect, and in overall cost savings; eight cases of schedule related benefits, including decreased time needed to complete tasks and increased predictability in meeting schedules; five cases of measurable improvements in quality, mostly related to reducing defects over time; three cases of improvements in customer satisfaction; and three cases showing positive return on investment from their CMMI- based process improvements.

FAA Has Not Yet Institutionalized Process Improvement in Its ATC Organizations

Leading organizations have found that in order to achieve advanced system management capabilities and to gain the benefits of more mature processes, an organization needs to institutionalize process improvement. Specifically, to be effective, an organization needs senior-level endorsement of its process improvement initiatives and consistency in the adoption and management of process improvement efforts.

In recent years, FAA’s ATC-related organizations have encouraged process improvement through the iCMM model. Specifically, FAA’s acquisition policy calls for continuous process improvement and endorses the use of the iCMM model. Also, the former air traffic organizations set annual goals for improving maturity using the iCMM model in selected projects and process areas. For example, in 1997, the former ATC acquisition organization set a goal of having 11 selected projects achieve iCMM maturity level 2 by 1999 and maturity level 3 by 2001. While the projects did not meet the 1999 goal, several projects achieved level 2 in 2000, and most made improvements in selected process areas.

However, FAA did not institutionalize the use of the iCMM model throughout the organization and, as a result, individual projects’ use and application of the model has been voluntary. Individual project teams could determine whether or not they would implement the model and which process areas to work on. In addition, project teams could decide when, if ever, to seek an appraisal of their progress in implementing the model.

Because of this voluntary approach, to date less than half of the projects listed in FAA’s system architecture have sought appraisals in at least one process area. Specifically, of the 48 systems listed in FAA’s system architecture, only 18 have sought appraisals. Some of the mission critical systems that have not sought appraisals include an advanced radar system and air traffic information processing system.

Another result of this voluntary approach is that individual projects are making uneven progress in core areas. For example, the four projects that we appraised ranged from capability levels 0 to 2 in the risk management process area: in other words, projects varied from performing only part of the basic process, to performing the basic process, to actively managing the process. As another example, all four of the projects we appraised captured some metrics on their performance. However, these metrics varied greatly from project to project in depth, scope, and usefulness. Individual weaknesses in key processes could lead to systems that do not meet the users’ needs, exceed estimated costs, or take longer than expected to complete.

While FAA encouraged process improvement in the past, the agency’s current commitment to process improvement in its new Air Traffic Organization is not certain. FAA recently moved its air traffic–related organizations into a single, performance-based organization, the Air Traffic Organization, under the direction of a Chief Operating Officer. The Chief Operating Officer is currently reevaluating all policies and processes, and plans to issue new acquisition guidance in coming months. As a result, the Air Traffic Organization does not currently have a policy that requires organizations and project teams to implement process improvement initiatives such as the iCMM. It also does not have a detailed plan— including goals, metrics, and milestones—for implementing these initiatives throughout the organization, nor does it have a mechanism for enforcing compliance with any requirements—such as taking a project’s capability levels into consideration before approving new investments. Further, because the Air Traffic Organization’s commitment to the iCMM is not yet certain, FAA’s centralized process improvement organization is unable to define a strategy for improving and overseeing process improvement efforts in the Air Traffic Organization.

Unless the Chief Operating Officer demonstrates a strong commitment to process improvement and establishes a consistent, institutionalized approach to implementing, enforcing, and evaluating this process improvement, FAA risks taking a major step backwards in its capabilities for acquiring ATC systems and software. That is, FAA may not be able to ensure that critical projects will continue to make progress in improving systems acquisition and development capabilities, and the agency is not likely to proceed to the more advanced capability levels which focus on organizationwide management of processes. Further, FAA may miss out on the benefits that process improvement models offer, such as better managed projects and improved product quality. Should this occur, FAA will continue to be vulnerable to project management problems including cost overruns, schedule delays, and performance shortfalls.

Conclusions and Recommendations

Conclusions

The Federal Aviation Administration (FAA) has made considerable progress in implementing processes for managing software acquisitions. Key projects are performing most of the practices needed to reach a basic level of capability in process areas including risk management, project planning, project monitoring and control, and configuration management. However, recurring weaknesses in the areas of verification, quality assurance, and measurement and analysis prevented the projects from achieving a basic level of performance in these areas and from effectively managing these and other process areas. These weaknesses could lead to systems that do not meet the users’ needs, exceed estimated costs, or take longer than expected to complete. Further, because of the recurring weaknesses in measurement and analysis, senior executives may not receive the project status information they need to make sound decisions on major project investments.

FAA’s process improvement initiative has matured in recent years, but more can be done to institutionalize improvement efforts. The Chief Information Officer’s centralized process improvement organization has developed an integrated Capability Maturity Model (iCMM) and demonstrated improvements in those using the model, but to date the agency has not ensured that projects and organizational units consistently adopt such process improvements. Specifically, the agency lacks a detailed plan—including goals, metrics, and milestones—for implementing these initiatives throughout the new Air Traffic Organization, and a mechanism for enforcing compliance with any requirements—such as taking a project’s capability level into consideration before approving new investments. With the recent move of FAA’s air traffic control–related organizations into a performance-based organization, the agency has an opportunity to reiterate the value of process improvement and to achieve the benefits of more mature processes. In the coming months, it will be critical for this new organization to demonstrate its commitment to process improvement through its policies, plans, goals, oversight, and enforcement mechanisms. Without such endorsement, the progress that FAA has made in recent years could dissipate.

Recommendations for Executive Action

Given the importance of software-intensive systems to FAA’s air traffic control modernization program, we recommend that the Secretary of Transportation direct the FAA Administrator to ensure that the following five actions take place: The four projects that we appraised should take action to fully implement the practices that we identified as not implemented or partially implemented.

The new Air Traffic Organization should establish a policy requiring organizations and project teams to implement iCMM or equivalent process improvement initiatives and a plan for implementing iCMM or equivalent process improvement initiatives throughout the organization. This plan should specify a core set of process areas for all projects, clear criteria for when appraisals are warranted, and measurable goals and time frames.

The Chief Information Officer’s process improvement office, in consultation with the Air Traffic Organization, should develop a strategy for overseeing all air traffic projects’ progress to successive levels of maturity; this strategy should specify measurable goals and time frames.

To enforce process improvement initiatives, FAA investment decision makers should take a project’s capability level in core process areas into consideration before approving new investments in the project.

Agency Comments

In its oral comments on a draft of this report, Department of Transportation and FAA officials generally concurred with our recommendations, and they indicated that FAA is pleased with the significant progress that it has achieved in improving the processes used to acquire software and systems. Further, these officials noted that FAA has already started implementing changes to address issues identified in the report. They said that progress is evident in both the improved scores, compared with our prior study, and also in the way FAA functions on a day-to-day basis. For example, these officials explained that FAA is now working better as a team because the organization is using cross-organizational teams that effectively share knowledge and best practices for systems acquisition and management. FAA officials also noted that the constructive exchange of information with us was very helpful to them in achieving progress, and they emphasized their desire to maintain a dialog with us to facilitate continued progress. Agency officials also provided technical corrections, which we have incorporated into this report as appropriate.