Air Traffic Control: System Management Capabilities Improved, but More Can Be Done to Institutionalize Improvements
Published: 2004-08-20T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Since 1981, the Federal Aviation Administration (FAA) has been working to modernize its aging air traffic control (ATC) system. Individual projects have suffered cost increases, schedule delays, and performance shortfalls of large proportions, leading GAO to designate the program a high-risk information technology initiative in 1995. Because the program remains a high risk initiative, GAO was requested to assess FAA's progress in several information technology management areas. This report, one in a series responding to that request, has two objectives: (1) to evaluate FAA's capabilities for developing and acquiring software and systems on its ATC modernization program and (2) to assess the actions FAA has under way to improve these capabilities.
section_title: What GAO Found
paragraphs: FAA has made progress in improving its capabilities for acquiring software-intensive systems, but some areas still need improvement. GAO had previously reported in 1997 that FAA's processes for acquiring software were ad hoc and sometimes chaotic. Focusing on four mission critical air traffic projects, GAO's current review assessed system and software management practices in numerous key areas such as project planning, risk management, and requirements development. GAO found that these projects were generally performing most of the desired practices: of the 900 individual practices evaluated, 83 percent were largely or fully implemented. The projects were generally strong in several areas such as project planning, requirements management, and identifying technical solutions. However, there were recurring weaknesses in the areas of measurement and analysis, quality assurance, and verification. These weaknesses hinder FAA from consistently and effectively managing its mission critical systems and increase the risk of cost overruns, schedule delays, and performance shortfalls. To improve its software and system management capabilities, FAA has undertaken a rigorous process improvement initiative. In response to earlier GAO recommendations, in 1999, FAA established a centralized process improvement office, which has worked to help FAA organizations and projects to improve processes through the use of a standard model, the integrated Capability Maturity Model. This model, which is a broad model that integrates multiple maturity models, is used to assess the maturity of FAA's software and systems capabilities. The projects that have adopted the model have demonstrated growth in the maturity of their processes, and more and more projects have adopted the model. However, the agency does not require the use of this process improvement method. To date, less than half of FAA's major ATC projects have used this method, and the recurring weaknesses we identified in our project-specific evaluations are due in part to the choices these projects were given in deciding whether to and how to adopt this process improvement initiative. Further, as a result of reorganizing its ATC organizations to a performance-based organization, FAA is reconsidering prior policies, and it is not yet clear that process improvement will continue to be a priority. Without a strong senior-level commitment to process improvement and a consistent, institutionalized approach to implementing and evaluating it, FAA cannot ensure that key projects will continue to improve systems acquisition and development capabilities. As a result, FAA will continue to risk the project management problems--including cost overruns, schedule delays, and performance shortfalls--that have plagued past acquisitions.

section_title: Letter
paragraphs: The accompanying report is one in a series of reports responding to your  request that we review the information technology management processes  that the Federal Aviation Administration (FAA) uses to manage its air  traffic control modernization. This report discusses our assessment of the  department’s capabilities to develop and acquire software and systems, as  well as actions FAA has under way to improve these capabilities. We are  making recommendations to the Secretary of Transportation to assist the  agency in improving these capabilities.
As agreed with your staffs, unless you publicly announce its contents  earlier, we plan no further distribution of this report until 30 days from the  date on the report. At that time, we will send copies of the report to the  Secretary of Transportation, the Administrator of the Federal Aviation  Administration, and other congressional committees. Copies will also be  made available at our Web site at www.gao.gov. If you have any questions  or wish to discuss the issues in this report, please contact David Powner at  (202) 512-9286 or by e-mail at PownerD@gao.gov, or Keith Rhodes at (202)  512-6412 or by e-mail at RhodesK@gao.gov. Additional GAO contact and  staff acknowledgements are listed in appendix I.
subsections: section_title: Introduction
paragraphs: The primary mission of the Federal Aviation Administration (FAA) is to  provide a safe, secure, and efficient global aerospace system that  contributes to national security and the promotion of U.S. aerospace safety.  FAA’s ability to fulfill this mission depends on the adequacy and reliability  of the nation’s air traffic control (ATC) systems—a vast network of  computer hardware, software, and communications equipment.
To accommodate forecasted growth in air traffic and to relieve the  problems of aging ATC systems, FAA embarked on an ambitious ATC  modernization program in 1981. FAA now estimates that it will spend about  $51 billion to replace and modernize ATC systems through 2007. Our work  over the years has chronicled many FAA problems in meeting ATC projects’  cost, schedule, and performance goals. As a result of these issues as well  as the tremendous cost, complexity, and mission criticality of the  modernization program, we designated the program as a high-risk  information technology initiative in 1995, and it has remained on our high- risk list since that time.
subsections: section_title: Overview of ATC
paragraphs: Automated information processing and display, communication,  navigation, surveillance, and weather resources permit air traffic  controllers to view key information—such as aircraft location, aircraft  flight plans, and prevailing weather conditions—and to communicate with  pilots. These resources reside at, or are associated with, several ATC  facilities—ATC towers, terminal radar approach control facilities, air route  traffic control centers (en route centers), flight service stations, and the  ATC System Command Center. Figure 2 shows a visual summary of ATC  over the continental United States and oceans.
section_title: FAA’s ATC Modernization Is a High-Risk Initiative
paragraphs: Faced with growing air traffic and aging equipment, in 1981, FAA initiated  an ambitious effort to modernize its ATC system. This effort involves the  acquisition of new surveillance, data processing, navigation, and  communications equipment, in addition to new facilities and support  equipment. Initially, FAA estimated that its ATC modernization effort would  cost $12 billion and could be completed over 10 years. Now, 2 decades and  $35 billion later, FAA expects to need another $16 billion through 2007 to  complete key projects, for a total cost of $51 billion.
Over the past 2 decades, many of the projects that make up the  modernization program have experienced substantial cost overruns,  schedule delays, and significant performance shortfalls. Our work over the  years has documented many of these shortfalls. As a result of these  problems, as well as the tremendous cost, complexity, and mission  criticality of the modernization program, we designated the program as a  high-risk information technology initiative in 1995, and it has remained on  our high-risk list since that time.
Our work since the mid-1990s has pinpointed root causes of the  modernization program’s problems, including (1) immature software  acquisition capabilities, (2) lack of a complete and enforced system  architecture, (3) inadequate cost estimating and cost accounting practices,  (4) an ineffective investment management process, and (5) an  organizational culture that impaired the acquisition process. We have made  over 30 recommendations to address these issues, and FAA has made  substantial progress in addressing them. Nonetheless, in our most recent  high-risk report, we noted that more remains to be done—and with FAA  still expecting to spend billions on new ATC systems, these actions are as  critical as ever.
section_title: Prior Report Noted Weaknesses in FAA’s Software Acquisition Capabilities
paragraphs: In March 1997, we reported that FAA’s processes for acquiring software, the  most costly and complex component of its ATC systems, were ad hoc,  sometimes chaotic, and not repeatable across projects. We also reported  that the agency lacked an effective management structure for ensuring  software process improvement. As a result, the agency was at great risk of  not delivering promised software capabilities on time and within budget.  We recommended that FAA establish a Chief Information Officer  organizational structure, as prescribed in the Clinger-Cohen Act, and assign  responsibility for software acquisition process improvement to this  organization. We also recommended several actions intended to help FAA  improve its software acquisition capabilities by institutionalizing mature  processes. These included developing a comprehensive plan for process  improvement, allocating adequate resources to ensure that improvement  efforts were implemented, and requiring that projects achieve a minimum  level of maturity before being approved.
FAA has implemented most of our recommendations. The agency  established a Chief Information Officer position that reports directly to the  administrator and gave this position responsibility for process  improvement. The Chief Information Officer’s process improvement office  developed a strategy and led the way in developing an integrated  framework for improving maturity in system acquisition, development, and  engineering processes. Some of the business organizations within FAA,  including the organizations responsible for ATC acquisitions and  operations, adopted the framework and provided resources to process  improvement efforts.
FAA did not, however, implement our recommendation to require that  projects achieve a minimum level of maturity before being approved.  Officials reported that rather than establish arbitrary thresholds for  maturity, FAA intended to evaluate process areas that were most critical or  at greatest risk for each project during acquisition management reviews.
section_title: FAA Is Changing Its Approach to ATC Management
paragraphs: Recent legislation and an executive order have led to major changes in the  way that FAA manages its ATC mission. In April 2000, the Wendell H. Ford  Aviation Investment and Reform Act for the 21st Century (Air-21)  established the position of Chief Operating Officer for the ATC system. In  December 2000, executive order 13180 instructed FAA to establish a  performance-based organization known as the Air Traffic Organization and  to have the Chief Operating Officer lead this organization under the  authority of the FAA administrator. This order, amended in June 2002,  called for the Air Traffic Organization to enhance the FAA’s primary mission  of ensuring the safety, security, and efficiency of the National Airspace  System and further improve the delivery of air traffic services to the  American public by reorganizing air traffic services and related offices into  a performance-based, results-oriented organization. The order noted that  as a performance-based organization, the Air Traffic Organization would be  able to take better advantage of the unique procurement and personnel  authorities currently used by FAA, as well as of the additional management  reforms enacted by Congress under Air-21. In addition, the Air Traffic  Organization is responsible for developing methods to accelerate ATC  modernization, improving aviation safety related to ATC, and establishing  strong incentives to agency managers for achieving results.
In leading the new Air Traffic Organization, the Chief Operating Officer’s  responsibilities include establishing and maintaining organizational and  individual goals, a 5-year strategic plan including ATC system mission and  objectives, and a framework agreement with the Administrator to establish  the new organization’s relationships with other FAA organizations. In  August 2003, the first Chief Operating Officer joined the agency and  initiated a reorganization combining the separate ATC-related  organizations and offices into the Air Traffic Organization.
section_title: The Capability Maturity Model Integration Provides a Means of Assessing an Organization’s Ability to Manage Software and System Acquisition and Development
paragraphs: An essential aspect of FAA’s ATC modernization program is the quality of  the software and systems involved, which is heavily influenced by the  quality and maturity of the processes used to acquire, develop, manage, and  maintain them. Carnegie Mellon University’s Software Engineering  Institute (SEI), recognized for its expertise in software and system  processes, has developed the Capability Maturity Model Integration  (CMMI) and a CMMI appraisal methodology to evaluate, improve, and  manage system and software development and engineering processes. The  CMMI model and appraisal methodology provide a logical framework for  measuring and improving key processes needed for achieving high-quality  software and systems. The model can help an organization set process  improvement objectives and priorities and improve processes; the model  can also provide guidance for ensuring stable, capable, and mature  processes. According to SEI, organizations that implement such process  improvements can achieve better project cost and schedule performance  and higher quality products.
In brief, the CMMI model identifies 25 process areas—clusters of related  practices that, when performed collectively, satisfy a set of goals that are  considered important for making significant improvements in that area.  Table 1 describes these process areas.
The CMMI model provides two alternative ways to view these process  areas. One way, called continuous representation, focuses on improving  capabilities in individual process areas. The second way, called staged  representation, groups process areas together and focuses on achieving  increased maturity levels by improving the group of process areas.
The CMMI appraisal methodology calls for assessing process areas by  determining whether the key practices are implemented and whether the  overarching goals are satisfied. Under continuous representation,  successful implementation of these practices and satisfaction of these  goals result in the achievement of successive capability levels in a selected  process area. CMMI capability levels range from 0 to 5, with level 0  meaning that the process is either not performed or partially performed;  level 1 meaning that the basic process is performed; level 2 meaning that  the process is managed; level 3 meaning that the processes is defined  throughout the organization; level 4 meaning that the process is  quantitatively managed; and level 5 meaning that the process is optimized.  Figure 3 provides details on CMMI capability levels.
section_title: Objectives, Scope, and Methodology
paragraphs: The Chairman, House Committee on Government Reform, and the  Chairman of that Committee’s Subcommittee on Technology, Information  Policy, Intergovernmental Relations and the Census requested that we  evaluate FAA’s software and system development processes used to  manage its ATC modernization. Our objectives were (1) to evaluate FAA’s  capabilities for developing and acquiring software and systems on its ATC  modernization program and (2) to assess the actions FAA has under way to  improve these capabilities.
To evaluate FAA’s capabilities for developing and acquiring software and  systems, we applied the CMMI model (continuous representation) and its  related appraisal methodology to four FAA projects. Our appraisers were  all SEI-trained software and information systems specialists. In addition,  we employed SEI-trained consultants as advisors on our first evaluation to  ensure proper application of the model and appraisal methodology.
In consultation with FAA officials, we selected four FAA projects with high  impact, visibility, and cost, which represented different air traffic domains  and reflected different stages of life cycle development. The projects  included the Voice Switching and Control System (VSCS), the Integrated  Terminal Weather System (ITWS), the En Route Automation Modernization  (ERAM) project, and the Airport Surface Detection Equipment–Model X  (ASDE-X). The four projects are described in table 2.
In conjunction with FAA’s process improvement organization, we identified  relevant CMMI process areas for each appraisal. In addition, because  system deployment is an important aspect of FAA systems management  that is not included in CMMI, we used the deployment, transition, and  disposal process area from FAA’s integrated Capability Maturity Model,  version 2. For consistency, we merged FAA’s criteria with SEI’s framework  and added the standard goals and practices needed to achieve capability  level 2.
In selected cases, we did not review a certain process area because it was  not relevant to the current stage of a project’s life cycle. For example, we  did not evaluate supplier agreement management or deployment on VSCS  because the system is currently in operation, and these process areas are  no longer applicable to this system. Table 3 displays the CMMI process  areas that we reviewed for each project.
For each process area reviewed, we evaluated project-specific  documentation and interviewed project officials to determine whether key  practices were implemented and goals were achieved. In accordance with  CMMI guidance, we characterized practices as fully implemented, largely  implemented, partially implemented, and not implemented, and  characterized goals as satisfied or unsatisfied. After combining the  practices and goals, the team determined if successive capability levels  were achieved. According to the CMMI appraisal method, practices must  be largely or fully implemented in order for a goal to be satisfied. Further,  all goals must be satisfied in order to achieve a capability level. In order to  achieve advanced capability levels, all preceding capability levels must be  achieved. For example, a prerequisite for level 2 is the achievement of level  1. As agreed with FAA process improvement officials, we evaluated the  projects through capability level 2. Consistent with the CMMI appraisal  methodology, we validated our findings by sharing preliminary  observations with the project team so that they were able to provide  additional documentation or information as warranted.
To assess the actions FAA has under way to improve its system and  software acquisition and development processes, we evaluated process  improvement strategies and plans. We also evaluated the progress the  agency has made in expanding its process improvement initiative, both  through the maturity of the model and the acceptance of the model by  project teams. We also interviewed officials from the offices of the Chief  Information Officer and the Chief Operating Officer to determine the effect  current changes in the ATC organization could have on the process  improvement initiatives.
The Department of Transportation and FAA provided oral comments on a  draft of this report. These comments are presented in chapter 17. We  performed our work from September 2003 through July 2004 in accordance  with generally accepted government auditing standards.
section_title: FAA Is Performing Most Project Planning Practices, but It Is Not Yet Fully Managing the Process
paragraphs: The purpose of project planning is to establish and maintain plans that  define the project activities. This process area involves developing and  maintaining a plan, interacting with stakeholders, and obtaining  commitment to the plan.
As figure 4 shows, three of the four FAA projects satisfied all criteria for the  “performing” capability level (level 1) in this process area. The fourth  project would have achieved level 1 if it had performed one more practice  (see the overview in table 4 for details). None of the four projects satisfied  all criteria for the “managing” capability level (level 2). While all four  projects had differing weaknesses that contributed to this result, common  weaknesses across most of the projects occurred in the areas of monitoring  and controlling the project planning process and in ensuring quality  assurance of the process. As a result of these weaknesses, FAA is exposed  to increased risks that projects will not meet cost, schedule, or  performance goals and that projects will not meet mission needs.
Looked at another way, of the 96 practices we evaluated in this process  area, FAA projects had 88 practices that were fully or largely implemented  and 8 practices that were partially or not implemented.
Additional details on each project’s appraisal results at successive  capability levels are provided in tables 5 through 12. Specifically, tables 5  and 6 provide results for VSCS; tables 7 and 8 provide results for ERAM;  tables 9 and 10 provide results for ITWS; and tables 11 and 12 provide  results for ASDE-X.
section_title: FAA Is Performing Most Project Monitoring and Control Practices, but It Is Not Yet Fully Managing the Process
paragraphs: The purpose of project monitoring and control is to provide an  understanding of the project’s progress so that appropriate corrective  actions can be taken when the project’s performance deviates significantly  from the plan. Key activities include monitoring activities, communicating  status, taking corrective action, and determining progress.
As shown in figure 5, three of the four FAA projects satisfied all criteria for  the “performing” capability level (level 1) in this process area. The fourth  project would have achieved level 1 if it had performed one more practice  (see the overview in table 13 for details). None of the four projects satisfied  all criteria for the “managing” capability level (level 2). While the projects  had differing weaknesses that contributed to this result, a common  weakness across most of the projects occurred in the area of ensuring  quality assurance of the process. As a result of this weakness, FAA is  exposed to increased risks that projects will not meet cost, schedule, or  performance goals and that projects will not meet mission needs.
Looked at another way, of the 80 practices we evaluated in this process  area, FAA projects had 74 practices that were fully or largely implemented  and 6 practices that were partially or not implemented.
Additional details on each project’s appraisal results at successive  capability levels are provided in tables 14 through 21. Specifically, tables 14  and 15 provide results for VSCS; tables 16 and 17 provide results for ERAM;  tables 18 and 19 provide results for ITWS; and tables 20 and 21 provide  results for ASDE-X.
section_title: FAA Is Performing Most Risk Management Practices, but It Is Not Yet Fully Managing the Process
paragraphs: The purpose of risk management is to identify potential problems before  they occur, so that risk-handling activities may be planned and invoked as  needed across the life of the product or project to mitigate adverse impacts  on achieving objectives. Effective risk management includes early and  aggressive identification of risks through the involvement of relevant  stakeholders. Early and aggressive detection of risk is important, because it  is typically easier, less costly, and less disruptive to make changes and  correct work efforts during the earlier phases of the project.
As shown in figure 6, three of the four FAA projects satisfied all criteria for  the “performing” capability level (level 1) in this process area. The fourth  project would have achieved level 1 if it had performed one more practice  (see the overview in table 22 for details). Two of the four FAA projects also  satisfied all criteria for the “managed” capability level (level 2) in this  process area. While the other projects had differing weaknesses that  contributed to this result, common weaknesses across some of the projects  occurred in the area of monitoring and controlling the risk management  process and in ensuring quality assurance of the process. As a result of  these weaknesses, FAA faces increased likelihood that project risks will  not be identified and addressed in a timely manner—thereby increasing the  likelihood that projects will not meet cost, schedule, or performance goals.
Looked at another way, of the 68 practices we evaluated in this key process  area, FAA projects had 59 practices that were fully or largely implemented  and 9 practices that were partially or not implemented.
Additional details on each project’s appraisal results at successive  capability levels are provided in tables 23 through 30. Specifically, tables 23  and 24 provide results for VSCS; tables 25 and 26 provide results for ERAM;  tables 27 and 28 provide results for ITWS; and tables 29 and 30 provide  results for ASDE-X.
section_title: FAA Is Performing Requirements Development Practices, but It Is Not Yet Fully Managing the Process
paragraphs: The purpose of requirements development is to produce and analyze  customer, product, and product-component needs. This process area  addresses the needs of relevant stakeholders, including those pertinent to  various product life-cycle phases. It also addresses constraints caused by  the selection of design solutions. The development of requirements  includes elicitation, analysis, validation, and communication of customer  and stakeholder needs and expectations.
As shown in figure 7, all four FAA projects satisfied all criteria for the  “performing” capability level (level 1) in this process area. None of the four  projects satisfied all criteria for the “managing” capability level (level 2).  While all four projects had differing weaknesses that contributed to this  result, common weaknesses across multiple projects occurred in the areas  of training people and in ensuring quality assurance of the requirements  development process, as shown in the overview in table 31. As a result of  these weaknesses, FAA is exposed to increased risks that projects will not  fulfill mission and user needs.
Looked at another way, of the 84 practices we evaluated in this key process  area, FAA projects had 77 practices that were fully or largely implemented  and 7 practices that were partially or not implemented.
Additional details on each project’s appraisal results at successive  capability levels are provided in tables 32 through 39. Specifically, tables 32  and 33 provide results for VSCS; tables 34 and 35 provide results for ERAM;  tables 36 and 37 provide results for ITWS; and tables 38 and 39 provide  results for ASDE-X.
section_title: FAA Is Performing Requirements Management Practices, but It Is Not Yet Fully Managing the Process
paragraphs: The purpose of requirements management is to manage the project’s  product components and to identify inconsistencies between requirements  and the project’s plans and work products. This process area includes  managing all technical and nontechnical requirements and any changes to  these requirements as they evolve.
As shown in figure 8, all four FAA projects satisfied all criteria for the  “performing” capability level (level 1) in this process area, but none  satisfied all criteria for achieving a “managed” capability level (level 2).  While the projects had differing weaknesses that contributed to this result,  a common weakness across most of the projects occurred in the area of  ensuring quality assurance of the requirements management process, as  shown in the overview in table 40. As a result of these weaknesses, FAA is  exposed to increased risks that projects will not fulfill mission and user  needs.
Looked at another way, of the 60 practices we evaluated in this key process  area, FAA projects had 54 practices that were fully or largely implemented  and 6 practices that were partially or not implemented.
Additional details on each project’s appraisal results at successive  capability levels are provided in tables 41 through 48. Specifically, tables 41  and 42 provide results for VSCS; tables 43 and 44 provide results for ERAM;  tables 45 and 46 provide results for ITWS; and tables 47 and 48 provide  results for ASDE-X.
section_title: FAA Is Performing Most Technical Solution Practices, but It Is Not Yet Fully Managing the Process
paragraphs: The purpose of the technical solution process area is to design, develop,  and implement products, product components, and product-related life- cycle processes to meet requirements. This process involves evaluating and  selecting solutions that potentially satisfy an appropriate set of allocated  requirements, developing detailed designs, and implementing the design.
As shown in figure 9, three FAA projects satisfied all criteria for the  “performing” capability level (level 1) in this process area. The fourth  project would have achieved level 1 if it had performed two more practices  (see the overview in table 49 for details). None of the four projects satisfied  all criteria for the “managing” capability level (level 2). While all four  projects had differing weaknesses that contributed to this result, common  weaknesses across most of the projects occurred in the area of ensuring  quality assurance of the technical solution process. As a result of this  weakness, FAA is exposed to increased risks that projects will not meet  mission needs.
Looked at another way, of the 72 practices we evaluated in this key process  area, FAA projects had 62 practices that were fully or largely implemented  and 10 practices that were partially or not implemented.
Additional details on each project’s appraisal results at successive  capability levels are provided in tables 50 through 57. Specifically, tables 50  and 51 provide results for VSCS; tables 52 and 53 provide results for ERAM;  tables 54 and 55 provide results for ITWS; and tables 56 and 57 provide  results for ASDE-X.
section_title: FAA Is Performing Product Integration Practices, but It Is Not Yet Fully Managing the Process
paragraphs: The purpose of the product integration process is to assemble the product  components, ensure that the integrated product functions properly, and  deliver the product. A critical aspect of this process is managing the  internal and external interfaces of the products and product components,  in one stage or in incremental stages.
For this process area, we did not perform an appraisal for the ERAM  project, because it was at a stage in which product integration was not  applicable. As shown in figure 10, the three remaining projects satisfied all  criteria for the “performing” capability level (level 1) in this process area.  None of the projects satisfied all criteria for the “managing” capability level  (level 2). While the projects had differing weaknesses that contributed to  this result, common weaknesses across most of the projects occurred in  the areas of monitoring and controlling the product integration process and  ensuring quality assurance of the process, as shown in the overview in  table 58. As a result of this weakness, FAA is exposed to increased risk that  product components will not be compatible, resulting in projects that will  not meet cost, schedule, or performance goals.
Looked at another way, of the 54 practices we evaluated in this process  area, FAA projects had 49 practices that were fully or largely implemented  and 5 practices that were partially or not implemented.
Additional details on each project’s appraisal results at successive  capability levels are provided in tables 59 through 64. Specifically, tables 59  and 60 provide results for VSCS; tables 61 and 62 provide results for ITWS;  and tables 63 and 64 provide results for ASDE-X.
section_title: FAA Is Not Performing Key Verification Practices or Fully Managing the Process
paragraphs: The purpose of verification is to ensure that selected work products meet  their specified requirements. This process area involves preparing for and  performing tests and identifying corrective actions. Verification of work  products substantially increases the likelihood that the product will meet  the customer, product, and product-component requirements.
As shown in figure 11, only one of four FAA projects satisfied all criteria for  the “performing” capability level (level 1) in this process area. As shown in  the overview in table 65, key weaknesses in preparing and conducting peer  reviews prevented the other three projects from achieving level 1. None of  the four projects satisfied all criteria for the “managing” capability level  (level 2). While all four projects had differing weaknesses that contributed  to this result, common weaknesses across most of the projects occurred in  the areas of monitoring and controlling the verification process and in  ensuring quality assurance of the process. As a result of these weaknesses,  FAA is exposed to increased risk that the product will not meet the user  and mission requirements, increasing the likelihood that projects that will  not meet cost, schedule, or performance goals.
Looked at another way, of the 68 practices we evaluated in this process  area, FAA projects had 51 practices that were fully or largely implemented  and 17 practices that were partially or not implemented.
Additional details on each project’s appraisal results at successive  capability levels are provided in tables 66 through 73. Specifically, tables 66  and 67 provide results for VSCS; tables 68 and 69 provide results for ERAM;  tables 70 and 71 provide results for ITWS; and tables 72 and 73 provide  results for ASDE-X.
section_title: FAA Is Performing Validation Practices, but It Is Not Yet Fully Managing the Process
paragraphs: The purpose of validation is to demonstrate that a product or product  component fulfills its intended use when placed in its intended  environment. Validation activities are vital to ensuring that the products are  suitable for use in their intended operating environment.
As shown in figure 12, all four FAA projects satisfied all criteria for the  “performing” capability level (level 1) in this process area. None of the four  projects satisfied all criteria for the “managing” capability level (level 2).  While all four projects had differing weaknesses that contributed to this  result, common weaknesses across most of the projects occurred in the  areas of monitoring and controlling the validation process and in ensuring  quality assurance of the process, as shown in the overview in table 74. As a  result of these weaknesses, FAA is exposed to increased risk that the  project will not fulfill its intended use, thereby increasing the likelihood  that the projects will not meet cost, schedule, or performance goals.
Looked at another way, of the 56 practices we evaluated in this process  area, FAA projects had 47 practices that were fully or largely implemented  and 9 practices that were partially or not implemented.
Additional details on each project’s appraisal results at successive  capability levels are provided in tables 75 through 82. Specifically, tables 75  and 76 provide results for VSCS; tables 77 and 78 provide results for ERAM;  tables 79 and 80 provide results for ITWS; and tables 81 and 82 provide  results for ASDE-X.
section_title: FAA Is Performing Most Configuration Management Practices, but It Is Not Yet Fully Managing the Process
paragraphs: The purpose of configuration management is to establish and maintain the  integrity of work products. This process area includes both the functional  processes used to establish and track work product changes and the  technical systems used to manage these changes. Through configuration  management, accurate status and data are provided to developers, end  users, and customers.
As shown in figure 13, three of the four FAA projects satisfied all criteria  for the “performing” capability level (level 1) in this process area. The  fourth project would have achieved level 1 if it had performed two more  practices (see the overview in table 83 for details). Only one of the four  projects satisfied all criteria for the “managing” capability level (level 2).  While all four projects had differing weaknesses that contributed to this  result, common weaknesses across some of the projects occurred in the  areas of monitoring and controlling the process and in ensuring the quality  assurance of the configuration management process, as shown in the  overview in table 83. As a result of these weaknesses, FAA is exposed to  increased risk that the project teams will not effectively manage their work  products, resulting in projects that do not meet cost, schedule, or  performance goals.
Looked at another way, of the 68 practices we evaluated in this process  area, FAA projects had 60 practices that were fully or largely implemented  and 8 practices that were partially or not implemented.
Additional details on each project’s appraisal results at successive  capability levels are provided in tables 84 through 91. Specifically, tables 84  and 85 provide results for VSCS; tables 86 and 87 provide results for ERAM;  tables 88 and 89 provide results for ITWS; and tables 90 and 91 provide  results for ASDE-X.
section_title: FAA Is Not Performing Key Process and Product Quality Assurance Practices or Managing the Process
paragraphs: The purpose of process and product quality assurance is to provide staff  and management with objective insights into processes and associated  work products. This process area includes the objective evaluation of  project processes and products against approved descriptions and  standards. Through process and product quality assurance, the project is  able to identify and document noncompliance issues and provide  appropriate feedback to project members.
As shown in figure 14, only one of the four FAA projects satisfied all criteria  for the “performing” capability level (level 1) in this process area.  Weaknesses in the objective evaluation of designated performed processes,  work products, and services against the applicable process descriptions,  standards, and procedures prevented the projects from achieving level 1.  None of the four projects satisfied all criteria for the “managing” capability  level (level 2). Table 92 provides an overview of our appraisal results. As  shown in the table, while the four projects had differing weaknesses that  contributed to this result, common weaknesses across multiple projects  occurred in the areas of establishing a plan, providing resources, training  people, providing configuration management, identifying stakeholders,  monitoring and controlling the process, ensuring quality assurance, and  reviewing the status of the quality assurance process with higher level  managers. As a result of these weaknesses, FAA is exposed to increased  risk that the projects will not effectively implement key management  processes, resulting in projects that will not meet cost, schedule, or  performance goals, and that will not meet mission needs.
Looked at another way, of the 56 practices we evaluated in this process  area, FAA projects had 33 practices that were fully or largely implemented  and 23 practices that were partially or not implemented.
Additional details on each project’s appraisal results at successive  capability levels are provided in tables 93 through 100. Specifically, tables  93 and 94 provide results for VSCS; tables 95 and 96 provide results for  ERAM; tables 97 and 98 provide results for ITWS; and tables 99 and 100  provide results for ASDE-X.
section_title: FAA Is Not Performing Most Measurement and Analysis Practices or Managing the Process
paragraphs: The purpose of measurement and analysis is to develop and sustain a  measurement capability that is used to support management information  needs. This process area includes the specification of measures, data  collection and storage, analysis techniques, and the reporting of these  values. This process allows users to objectively plan and estimate project  activities and identify and resolve potential issues.
As shown in figure 15, none of the four FAA projects satisfied all criteria for  the “performing” capability level (level 1) in this process area. Weaknesses  in managing and storing measurement data, measurement specifications,  and analysis results kept the projects from achieving level 1. Further, none  of the four projects satisfied all criteria for the “managing” capability level  (level 2). As shown in the overview in table 101, while the four projects had  differing weaknesses that contributed to this result, common weaknesses  across multiple projects occurred in the areas of establishing an  organizational policy, establishing a plan, providing resources, assigning  responsibility, training people, configuration management, identifying  stakeholders, monitoring and controlling the process, ensuring quality  assurance, and reviewing status with higher level management of the  measurement and analysis process. As a result of these weaknesses, FAA is  exposed to increased risk that the projects will not have adequate  estimates of work metrics or a sufficient view into actual performance.  This increases the likelihood that projects will not meet cost, schedule, or  performance goals, and that projects will not meet mission needs.
Looked at another way, of the 72 practices we evaluated in this process  area, FAA projects had 30 practices that were fully or largely implemented  and 42 practices that were partially or not implemented.
Additional details on each project’s appraisal results at successive  capability levels are provided in tables 102 through 109. Specifically, tables  102 and 103 provide results for VSCS; tables 104 and 105 provide results for  ERAM; tables 106 and 107 provide results for ITWS; and tables 108 and 109  provide results for ASDE-X.
section_title: FAA Is Performing Supplier Agreement Management Practices, but It Is Not Yet Fully Managing the Process
paragraphs: The purpose of supplier agreement management is to manage the  acquisition of products. This process area involves determining the type of  acquisition that will be used for the products acquired, selecting suppliers,  establishing, maintaining, and executing agreements, accepting delivery of  acquired products, and transitioning acquired products to the project,  among other items.
For this process area, we did not perform an appraisal for the VSCS or  ITWS projects, because these projects were at stages in which supplier  agreement management was not applicable. As shown in figure 16, both of  the remaining FAA projects satisfied all criteria for the “performing”  capability level (level 1) in this process area. One of the two projects  satisfied all criteria for the “managing” capability level (level 2). In not  consistently managing this process, FAA is exposed to increased risk that  projects will not be performed in accordance with contractual  requirements, resulting in projects that will not meet cost, schedule, or  performance goals, and systems that will not meet mission needs.
Looked at another way, of the 34 practices we evaluated in this process  area, FAA projects had 33 practices that were fully or largely implemented  and 1 practice that was partially implemented. Table 110 provides an  overview of the appraisal results.
Additional details on each project’s appraisal results at successive  capability levels are provided in tables 111 through 114. Specifically, tables  111 and 112 provide results for ERAM, and tables 113 and 114 provide  results for ASDE-X.
section_title: FAA Is Performing Deployment, Transition, and Disposal Practices, but It Is Not Yet Fully Managing the Process
paragraphs: The purpose of the deployment, transition, and disposal process area is to  place a product or service into an operational environment, transfer it to  the customer and to the support organization, and deactivate and dispose  of the replaced product or dispense with the service. This process area  includes the design and coordination of plans and procedures for  placement of a product or service into an operational or support  environment and bringing it into operational use. It ensures that an  effective support capability is in place to manage, maintain, and modify the  supplied product or service. It further ensures the successful transfer of the  product or service to the customer/stakeholder and the deactivation and  disposition of the replaced capability.
For this process area, we did not perform an appraisal for the VSCS or  ERAM projects, because these projects were at stages in which deployment  was not applicable. As shown in figure 17, both of the remaining FAA  projects satisfied all criteria for the “performing” capability level (level 1) in  this process area. Neither satisfied all criteria for the “managing” capability  level (level 2). As shown in the overview in table 115, while the projects had  differing weaknesses that contributed to this result, a common weakness  across projects occurred in the area of monitoring and controlling the  deployment process. As a result of this weakness, FAA is exposed to  increased risk that the projects will not be delivered on time, resulting in  projects that will not meet cost, schedule, or performance goals.
Looked at another way, of the 32 practices we evaluated in this process  area, FAA projects had 28 practices that were fully or largely implemented  and 4 practices that were partially implemented.
Additional details on each project’s appraisal results at successive  capability levels are provided in tables 116 through 119. Specifically, tables  116 and 117 provide results for ITWS, and tables 118 and 119 provide  results for ASDE-X.
section_title: FAA’s Process Improvement Initiative Has Matured, but It Is Not Yet Institutionalized
paragraphs: Since our 1997 report, the Federal Aviation Administration’s (FAA) process  improvement initiative has grown tremendously in rigor and scope. In our  earlier appraisal, we found that FAA’s performance of key processes was ad  hoc and sometimes chaotic, whereas current results show that FAA  projects are performing most key practices. However, these process  improvement activities are not required throughout the air traffic  organizations, and the recurring weaknesses we identified in our project- specific evaluations are due in part to the choices these projects were given  in deciding whether to and how to adopt process improvement initiatives.  Further, because of a recent reorganization, the new Air Traffic  Organization’s commitment to this process improvement initiative is not  certain. As a result, FAA is not consistent in its adoption and management  of process improvement efforts, so that individual projects’ costs,  schedules, and performance remain at risk. Without agencywide adoption  of process improvement initiatives, the agency cannot increase the  maturity of its organizational capabilities.
subsections: section_title: FAA’s Process Improvement Initiative Has Matured
paragraphs: Over the past several years, FAA has made considerable progress in  improving its processes for acquiring and developing software and  systems. Acting on our prior recommendations, in 1999, FAA established a  centralized process improvement office that reports directly to the Chief  Information Officer. This office led the government in an effort to integrate  various standards and models into a single maturity model, called the  integrated Capability Maturity Model (iCMM). In fact, FAA’s iCMM served  as a demonstration for the Software Engineering Institute’s effort to  integrate various models into its own Capability Maturity Model Integration  (CMMI).
The Chief Information Officer’s process improvement office also developed  and sponsored iCMM-related training, and by late 2003, it had trained over  7,000 participants. The training offered ranges from overviews on how to  use the model to more focused courses in such specific process areas as  quality assurance, configuration management, and project management.  The office also guides FAA organizations in using the model and leads  appraisal teams in evaluating the process maturity of the projects and  organizations that adopted the model.
In addition to the Chief Information Officer–sponsored process  improvement efforts, several of FAA’s business areas, including the  business areas with responsibility for air traffic control (ATC) system  acquisitions and operations, endorsed and set goals for process  improvement activities using the iCMM. As a result, there has been a  continuing growth over the years in the number of individual projects and  umbrella organizations that adopted process improvement and the iCMM  model. Specifically, the number of projects and organizations (which  account for multiple projects) undergoing iCMM appraisals grew from 1  project in 1997, to 28 projects and 3 organizations by 2000, to 39 projects  and 11 organizations by 2003.
These projects and organizations have demonstrated improvements in  process maturity. Under the iCMM model, in addition to achieving  capability levels in individual process areas, entities can achieve successive  maturity levels by demonstrating capabilities in a core set of process  areas. FAA process improvement officials reported that by 2000, 10  projects and one organization had achieved iCMM maturity level 2. To date,  14 projects and three organizations have achieved iCMM maturity level 2,  and one project and two organizations have achieved iCMM maturity level  3. Additionally, 13 projects and four organizations achieved capability  levels 2 or 3 in one or more process areas.
Moreover, in internal surveys, the programs and organizations pursuing  process improvement have consistently reported enhanced productivity,  higher quality, increased ability to predict schedules and resources, higher  morale, and better communication and teamwork. These findings are  reiterated by the Software Engineering Institute in its recent study of the  benefits of using the CMMI model for process improvement. According to  that study, organizations that implement such process improvements can  achieve better project cost and schedule performance and higher quality  products. Specifically, of the 12 cases that the Software Engineering  Institute assessed, there were   nine examples of cost related benefits, including reductions in the cost  to find and fix a defect, and in overall cost savings;  eight cases of schedule related benefits, including decreased time  needed to complete tasks and increased predictability in meeting  schedules;  five cases of measurable improvements in quality, mostly related to  reducing defects over time;  three cases of improvements in customer satisfaction; and  three cases showing positive return on investment from their CMMI- based process improvements.
section_title: FAA Has Not Yet Institutionalized Process Improvement in Its ATC Organizations
paragraphs: Leading organizations have found that in order to achieve advanced system  management capabilities and to gain the benefits of more mature  processes, an organization needs to institutionalize process improvement.  Specifically, to be effective, an organization needs senior-level  endorsement of its process improvement initiatives and consistency in the  adoption and management of process improvement efforts.
In recent years, FAA’s ATC-related organizations have encouraged process  improvement through the iCMM model. Specifically, FAA’s acquisition  policy calls for continuous process improvement and endorses the use of  the iCMM model. Also, the former air traffic organizations set annual goals  for improving maturity using the iCMM model in selected projects and  process areas. For example, in 1997, the former ATC acquisition  organization set a goal of having 11 selected projects achieve iCMM  maturity level 2 by 1999 and maturity level 3 by 2001. While the projects did  not meet the 1999 goal, several projects achieved level 2 in 2000, and most  made improvements in selected process areas.
However, FAA did not institutionalize the use of the iCMM model  throughout the organization and, as a result, individual projects’ use and  application of the model has been voluntary. Individual project teams could  determine whether or not they would implement the model and which  process areas to work on. In addition, project teams could decide when, if  ever, to seek an appraisal of their progress in implementing the model.
Because of this voluntary approach, to date less than half of the projects  listed in FAA’s system architecture have sought appraisals in at least one  process area. Specifically, of the 48 systems listed in FAA’s system  architecture, only 18 have sought appraisals. Some of the mission critical  systems that have not sought appraisals include an advanced radar system  and air traffic information processing system.
Another result of this voluntary approach is that individual projects are  making uneven progress in core areas. For example, the four projects that  we appraised ranged from capability levels 0 to 2 in the risk management  process area: in other words, projects varied from performing only part of  the basic process, to performing the basic process, to actively managing the  process. As another example, all four of the projects we appraised captured  some metrics on their performance. However, these metrics varied greatly  from project to project in depth, scope, and usefulness. Individual  weaknesses in key processes could lead to systems that do not meet the  users’ needs, exceed estimated costs, or take longer than expected to  complete.
While FAA encouraged process improvement in the past, the agency’s  current commitment to process improvement in its new Air Traffic  Organization is not certain. FAA recently moved its air traffic–related  organizations into a single, performance-based organization, the Air Traffic  Organization, under the direction of a Chief Operating Officer. The Chief  Operating Officer is currently reevaluating all policies and processes, and  plans to issue new acquisition guidance in coming months. As a result, the  Air Traffic Organization does not currently have a policy that requires  organizations and project teams to implement process improvement  initiatives such as the iCMM. It also does not have a detailed plan—  including goals, metrics, and milestones—for implementing these  initiatives throughout the organization, nor does it have a mechanism for  enforcing compliance with any requirements—such as taking a project’s  capability levels into consideration before approving new investments.  Further, because the Air Traffic Organization’s commitment to the iCMM is  not yet certain, FAA’s centralized process improvement organization is  unable to define a strategy for improving and overseeing process  improvement efforts in the Air Traffic Organization.
Unless the Chief Operating Officer demonstrates a strong commitment to  process improvement and establishes a consistent, institutionalized  approach to implementing, enforcing, and evaluating this process  improvement, FAA risks taking a major step backwards in its capabilities  for acquiring ATC systems and software. That is, FAA may not be able to  ensure that critical projects will continue to make progress in improving  systems acquisition and development capabilities, and the agency is not  likely to proceed to the more advanced capability levels which focus on  organizationwide management of processes. Further, FAA may miss out on  the benefits that process improvement models offer, such as better  managed projects and improved product quality. Should this occur, FAA  will continue to be vulnerable to project management problems including  cost overruns, schedule delays, and performance shortfalls.
section_title: Conclusions and Recommendations
subsections: section_title: Conclusions
paragraphs: The Federal Aviation Administration (FAA) has made considerable  progress in implementing processes for managing software acquisitions.  Key projects are performing most of the practices needed to reach a basic  level of capability in process areas including risk management, project  planning, project monitoring and control, and configuration management.  However, recurring weaknesses in the areas of verification, quality  assurance, and measurement and analysis prevented the projects from  achieving a basic level of performance in these areas and from effectively  managing these and other process areas. These weaknesses could lead to  systems that do not meet the users’ needs, exceed estimated costs, or take  longer than expected to complete. Further, because of the recurring  weaknesses in measurement and analysis, senior executives may not  receive the project status information they need to make sound decisions  on major project investments.
FAA’s process improvement initiative has matured in recent years, but  more can be done to institutionalize improvement efforts. The Chief  Information Officer’s centralized process improvement organization has  developed an integrated Capability Maturity Model (iCMM) and  demonstrated improvements in those using the model, but to date the  agency has not ensured that projects and organizational units consistently  adopt such process improvements. Specifically, the agency lacks a detailed  plan—including goals, metrics, and milestones—for implementing these  initiatives throughout the new Air Traffic Organization, and a mechanism  for enforcing compliance with any requirements—such as taking a project’s  capability level into consideration before approving new investments. With  the recent move of FAA’s air traffic control–related organizations into a  performance-based organization, the agency has an opportunity to reiterate  the value of process improvement and to achieve the benefits of more  mature processes. In the coming months, it will be critical for this new  organization to demonstrate its commitment to process improvement  through its policies, plans, goals, oversight, and enforcement mechanisms.  Without such endorsement, the progress that FAA has made in recent years  could dissipate.
section_title: Recommendations for Executive Action
paragraphs: Given the importance of software-intensive systems to FAA’s air traffic  control modernization program, we recommend that the Secretary of  Transportation direct the FAA Administrator to ensure that the following  five actions take place:   The four projects that we appraised should take action to fully  implement the practices that we identified as not implemented or  partially implemented.
The new Air Traffic Organization should establish   a policy requiring organizations and project teams to implement  iCMM or equivalent process improvement initiatives and  a plan for implementing iCMM or equivalent process improvement  initiatives throughout the organization. This plan should specify a  core set of process areas for all projects, clear criteria for when  appraisals are warranted, and measurable goals and time frames.
The Chief Information Officer’s process improvement office, in  consultation with the Air Traffic Organization, should develop a strategy  for overseeing all air traffic projects’ progress to successive levels of  maturity; this strategy should specify measurable goals and time frames.
To enforce process improvement initiatives, FAA investment decision  makers should take a project’s capability level in core process areas into  consideration before approving new investments in the project.
section_title: Agency Comments
paragraphs: In its oral comments on a draft of this report, Department of Transportation  and FAA officials generally concurred with our recommendations, and they  indicated that FAA is pleased with the significant progress that it has  achieved in improving the processes used to acquire software and systems.  Further, these officials noted that FAA has already started implementing  changes to address issues identified in the report. They said that progress is  evident in both the improved scores, compared with our prior study, and  also in the way FAA functions on a day-to-day basis. For example, these  officials explained that FAA is now working better as a team because the  organization is using cross-organizational teams that effectively share  knowledge and best practices for systems acquisition and management.  FAA officials also noted that the constructive exchange of information with  us was very helpful to them in achieving progress, and they emphasized  their desire to maintain a dialog with us to facilitate continued progress.  Agency officials also provided technical corrections, which we have  incorporated into this report as appropriate.