Federal Aviation Administration: Stronger Architecture Program Needed to Guide Systems Modernization Efforts
Released: 2005-05-31T00:00:00

Letter

Background

FAA’s Mission and Organizational Structure

FAA’s primary mission is to provide a safe, secure, and efficient global airspace system that promotes airspace safety in the United States and contributes to national security. The agency’s roles include regulating civil aviation, developing and operating a system of air traffic control and navigation for civil and military aircraft, and researching and developing the NAS, which consists of more than 19,000 airports, 750 air traffic control facilities, and about 45,000 pieces of equipment.

FAA’s mission performance depends on the adequacy and reliability of the nation’s air traffic control system. The air traffic control system, the primary component of the NAS, is a vast network of computer hardware, software, and communications equipment. This system consists of automated information processing and display, communication, navigation, surveillance, and weather resources that permit air traffic controllers to view key information—such as aircraft location, aircraft flight plans, and prevailing weather conditions—and to communicate with pilots. These resources reside at, or are associated with, several air traffic control facilities—towers, terminal radar approach control facilities, air route traffic control centers (en route centers), flight service stations, and the System Command Center. Figure 1 shows a visual summary of the air traffic control system over the continental United States and oceans.

FAA’s mission performance also depends on the skills and expertise of its work force, composed of over 50,000 staff who provide aviation services— including air traffic control; maintenance of air traffic control equipment; and certification of aircraft, airline operations, and pilots. In fiscal year 2005, FAA’s budget authority to support its mission was approximately $14 billion. According to FAA officials, approximately 95 percent of the agency’s total spending is in support of the NAS. Further, FAA estimates that it will spend $7.6 billion over the next two years to complete key modernization projects.

As figure 2 illustrates, FAA has twelve staff offices to accomplish its mission—including the Office of International Aviation and the Office of Information Services/Chief Information Officer—and four lines of business—Air Traffic Organization, Commercial Space Transportation, Airports, and Regulation and Certification. Tables 1 and 2 provide additional information about the responsibilities of these offices and lines of business.

An Enterprise Architecture Is Critical to Successful Systems Modernization

Effective use of enterprise architectures, or modernization blueprints, is a trademark of successful public and private organizations. For more than a decade, we have promoted the use of architectures to guide and constrain systems modernization, recognizing them as a crucial means to a challenging goal: agency operational structures that are optimally defined in both business and technological environments. The Congress, the Office of Management and Budget (OMB), and the federal Chief Information Officer (CIO) Council have also recognized the importance of an architecture-centric approach to modernization. The Clinger-Cohen Act of 1996 mandates that an agency’s CIO develops, maintains, and facilitates the implementation of an IT architecture. Further, the E-Government Act of 2002 requires OMB to oversee the development of enterprise architectures within and across agencies.

Enterprise Architecture: A Brief Description

Generally speaking, an enterprise architecture connects an organization’s strategic plan with program and system solution implementations by providing the fundamental business and technology details needed to guide and constrain investments in a consistent, coordinated, and integrated fashion. As such, it should provide a clear and comprehensive picture of an entity, whether it is an organization (e.g., federal agency) or a functional or mission area that cuts across more than one organization (e.g., air traffic control). This picture consists of snapshots of both the enterprise’s current or “As Is” environment and its target or “To Be” environment, as well as a capital investment road map for transitioning from the current to the target environment. These snapshots further consist of “views,” which are basically one or more architecture products that provide conceptual or logical representations of the enterprise.

The suite of products and their content that form a given entity’s enterprise architecture are largely governed by the framework used to develop the architecture. Since the 1980s, various frameworks have emerged and been applied. For example, John Zachman developed a structure or “framework” for defining and capturing an architecture. This framework provides for six windows from which to view the enterprise, which Zachman terms “perspectives” on how a given entity operates: those of (1) the strategic planner, (2) the system user, (3) the system designer, (4) the system developer, (5) the subcontractor, and (6) the system itself. Zachman also proposed six abstractions or models associated with each of these perspectives: these models cover (1) how the entity operates, (2) what the entity uses to operate, (3) where the entity operates, (4) who operates the entity, (5) when entity operations occur, and (6) why the entity operates.

In September 1999, the federal CIO Council published the Federal Enterprise Architecture Framework (FEAF), which is intended to provide federal agencies with a common construct for their respective architectures, to facilitate the coordination of common business processes, technology insertion, information flows, and system investments among federal agencies. FEAF describes an approach, including models and definitions, for developing and documenting architecture descriptions for multiorganizational functional segments of the federal government. Similar to most frameworks, FEAF’s proposed models describe an entity’s business, the data necessary to conduct the business, applications to manage the data, and technology to support the applications.

More recently, OMB established the Federal Enterprise Architecture (FEA) Program Management Office to develop a federated enterprise architecture according to a collection of five “reference models, and a security and privacy profile overlaying the five models.”

The Performance Reference Model is intended to describe a set of performance measures for the major IT initiatives and their contribution to program performance. Version 1.0 of the model was released in September 2003.

The Business Reference Model is intended to describe the federal government’s businesses, independent of the agencies that perform them. It serves as the foundation for the FEA. Version 2.0 of the model was released in June 2003.

The Service Component Reference Model is intended to identify and classify IT service (i.e., application) components that support federal agencies and promote the reuse of components across agencies. Version 1.0 of the model was released in June 2003.

The Data Reference Model is intended to describe, at an aggregate level, the types of data and information that support program and business line operations and the relationships among these types. Version 1.0 of the model was released in September 2004.

The Technical Reference Model is intended to describe the standards, specifications, and technologies that collectively support the secure delivery, exchange, and construction of service components. Version 1.1 of the model was released in August 2003.

The Security and Privacy Profile is intended to provide guidance on designing and deploying measures that ensure the protection of information resources. OMB has released Version 1.0 of the profile.

Although these various enterprise architecture frameworks differ in their nomenclatures and modeling approaches, they consistently provide for defining an enterprise’s operations in both (1) logical terms, such as interrelated business processes and business rules, information needs and flows, and work locations and users and (2) technical terms, such as hardware, software, data, communications, and security attributes and performance standards. The frameworks also provide for defining these perspectives for both the enterprise’s current or “As Is” environment and its target or “To Be” environment, as well as a transition plan for moving from the “As Is” to the “To Be” environment.

The importance of developing, implementing, and maintaining an enterprise architecture is a basic tenet of both organizational transformation and IT management. Managed properly, an enterprise architecture can clarify and help to optimize the interdependencies and relationships among an organization’s business operations and the underlying IT infrastructure and applications that support these operations. Employed in concert with other important management controls, such as portfolio-based capital planning and investment control practices, architectures can greatly increase the chances that an organization’s operational and IT environments will be configured to optimize its mission performance. Our experience with federal agencies has shown that making IT investments without defining these investments in the context of an architecture often results in systems that are duplicative, not well integrated, and unnecessarily costly to maintain and interface.

Our Prior Work Has Emphasized the Need for FAA to Establish Architecture Management Capabilities

In November 2003, we reported the results of our governmentwide survey of agencies’ progress—including FAA’s—in establishing key enterprise architecture management capabilities as described in Version 1.1 of our architecture management maturity framework. This framework associates specific architecture management capabilities with five hierarchical stages of management maturity, starting with creating enterprise architecture awareness and followed by building the enterprise architecture management foundation, developing the enterprise architecture, completing the enterprise architecture, and leveraging the enterprise architecture to manage change. Table 3 provides a more detailed description of the stages of Version 1.1 of the framework.

Based on information provided by FAA, we reported that the agency had not established an architecture management foundation; as a result, we rated the agency to be at stage 1 of our framework. Specifically, we reported that it had not (1) allocated adequate resources and (2) established a framework, methodology, and automated tools to build the enterprise architecture. According to our framework, effective architecture management is generally not achieved until an enterprise has a completed and approved architecture that is being effectively maintained and is being used to leverage organizational change and support investment decision making. An enterprise with these characteristics would need to have satisfied all of the stage 2 and 3 core elements and most of the stage 4 and 5 elements.

Our Prior Work Has Also Emphasized the Need for FAA to Institutionalize Other Key IT Management Controls

In August 2004, we reported that FAA had established most—about 80 percent—of the basic practices needed to manage its mission-critical investments, including many of the foundational practices for selecting and controlling IT investments. However, we reported that weaknesses still existed in the process. For example, FAA had not involved its senior IT investment board in regular reviews of investments that had completed development and become operational, and had not implemented standard practices for managing its mission-support and administrative investments. Because of these weaknesses, we concluded that agency executives could not be assured that they were selecting and managing the mix of investments that best met the agency’s needs and priorities. Accordingly, we made several recommendations, including that the agency develop and implement a plan aimed at addressing the weaknesses identified in our report. FAA generally concurred with our conclusion and recommendations.

In addition, in August 2004, we reported that FAA had made progress in improving its capabilities for acquiring software-intensive systems, but that there were still areas that needed improvement. Specifically, we reported that it had recurring weaknesses in the areas of measurement and analysis, quality assurance, and verification. We concluded that these weaknesses prevented FAA from consistently and effectively managing its mission-critical systems and increased the risk of cost overruns, schedule delays, and performance shortfalls. We made several recommendations, including that FAA address these specific weaknesses and institutionalize its process improvement initiatives by establishing a policy and plans for implementing and overseeing process improvement initiatives. FAA generally concurred with our conclusion and recommendations.

Our Prior Work Has Identified Problems with the Air Traffic Control Modernization Program

FAA has a long and well-documented history of problems with its air traffic control modernization program, including cost overruns, schedule delays, and performance shortfalls. We first identified this program as an area at high risk in 1995 because of the modernization’s size, complexity, cost, and problem-plagued past. Over the past decade, we have continued to report on these problems. The program remains on our high-risk list today.

In March 1999, we testified that FAA had had some success in deploying new modernization systems over the past two decades, but that the agency had not delivered most of its major air traffic control systems in accordance with its cost, schedule, and performance goals, due largely to its failure to implement established guidelines for acquiring new systems. Specifically, we testified that the agency had not fully implemented an effective process for monitoring the cost, schedule, benefits, performance, and risk of its key projects throughout their life cycles. We also noted that FAA lacked an evaluation process for assessing outcomes after projects had been developed, in order to help improve the selection and monitoring of future projects. Moreover, we testified that the agency’s problems in modernizing its systems resulted from several root causes, including the agency’s attempt to undertake this modernization without the benefit of a complete NAS architecture to guide its efforts. We concluded that the agency would continue to experience problems in deploying new systems until it had fully implemented solutions that addressed these root causes of its modernization problems and strengthened controls over its modernization investments.

In February and October 2003, we testified that FAA had taken steps to improve the management of its air traffic control modernization, but that systemic management issues, including inadequate management controls and human capital issues, were contributing to the continued cost overruns, schedule delays, and performance shortfalls that major air traffic projects have consistently experienced. We stated that to overcome these problems, FAA would need to, among other things, improve its software capabilities by requiring that all systems achieve a minimum level of progress before they would be funded, and improve its cost estimating and cost accounting practices by incorporating actual costs from related system development efforts in its processes for estimating the costs of new projects. We testified that until these issues had been resolved, resources would not be spent cost-effectively, and improvements in capacity and efficiency would be delayed.

FAA’s Enterprise Architecture Program: A Brief Description

According to FAA, its enterprise architecture initiative is intended to influence the agency’s ongoing initiatives in E-Government, data management, information systems security, capital planning, investment analysis, and air traffic control and navigation and is to benefit the agency by aligning business processes with IT processes; improving flight safety; reducing the development and maintenance costs of systems; decreasing airline delays; guiding IT investments; and improving the security, interoperability, and data usage of these systems. FAA officials told us that the agency plans by April 2006 to have a comprehensive version of its enterprise architecture to guide and constrain the agency’s investment decisions.

The Assistant Administrator for Information Services, who is the agency’s CIO, has been assigned responsibility for developing and maintaining the agency’s enterprise architecture. The CIO has designated a program director to oversee this effort. Two project offices are responsible for developing the NAS and non-NAS segments of the enterprise architecture, respectively, in coordination with the program director. Brief descriptions of the NAS and non-NAS architecture projects are provided below.

NAS Architecture Project

According to FAA, the NAS architecture is intended to be the agency’s comprehensive plan for improving NAS operations through the year 2015 and is to address how FAA will replace aging equipment and introduce new systems, capabilities, and procedures. The NAS architecture, which FAA reports is being developed in collaboration with the aviation community, is intended to achieve several objectives. For example, it is to (1) ensure that the NAS can handle future growth in aviation without disrupting critical aviation services, (2) improve flight safety and the use of airspace, (3) decrease airline delays, and (4) improve systems integration and investment planning.

The agency is developing the NAS architecture in a series of incremental versions. It released the first version of the NAS architecture in September 1995. In 1999, FAA released Version 4.0 of this architecture, which, according to the agency, was the first version to include a 15 to 20-year view (a “To Be” view) and support budget forecasts. According to FAA, the current version of the NAS architecture (Version 5.0) shows how the agency intends to achieve the target system described by 2015.

The chief operating officer (COO) for the Air Traffic Organization is responsible for developing and implementing the NAS segment of the architecture. The COO has tasked the Operations Planning/Systems Engineering group within FAA’s Air Traffic Organization with the day-to-day activities involved in this effort. This group is headed by the Vice President for Operations Planning, who reports directly to the COO. The COO has also designated a chief architect, who reports to the Director of Systems Engineering, to develop and maintain the NAS architecture and to provide technical leadership and guidance, as necessary, to support investment decision making. The Operations Planning/Systems Engineering group receives input from several FAA organizations, but primarily from business units within the Air Traffic Organization.

Non-NAS Architecture Project

According to FAA, the non-NAS architecture will cover the agency’s administrative services and mission support activities—the process areas, data, systems, and technology that support such functions as budget and finance, as well as all of the other governmental air transportation missions and functions that are unique to the agency (e.g., certification of aircraft). FAA initiated a project to develop the non-NAS architecture in March 2002 and, according to FAA, the agency plans to have, by January 2005, an initial baseline architecture that will describe the “As Is” and “To Be” environments. According to FAA, it plans to incrementally build on this baseline and have a version of the non-NAS architecture by April 2006 that will also include a sequencing plan.

According to FAA, the Information Management Division within the Office of Information Services/CIO is responsible for developing and maintaining the non-NAS architecture. FAA has designated a chief architect, who reports to the program director, to oversee the day-to-day program activities for developing and maintaining the non-NAS architecture. To develop the non-NAS architecture, this division will receive input from the agency’s twelve staff offices and four lines of business.

FAA Has Yet to Establish Key Architecture Development, Maintenance, and Implementation Processes

FAA recognizes the need for and has begun to develop an enterprise architecture; however, it has yet to establish key architecture management capabilities that it will need to effectively develop, maintain, and implement the architecture. As previously stated, the agency has set up two separate project offices and tasked each with developing one of the two architecture segments (NAS and non-NAS) that together are to compose FAA’s enterprise architecture. The agency also reports that it has allocated adequate resources to these project offices and that chief architects have been assigned to head the architecture projects. However, FAA has not established other key architecture management capabilities, such as designating a committee or group representing the enterprise to direct, oversee, or approve the architecture effort; having an approved policy for developing, maintaining, and implementing the architecture; and fully developing architecture products that meet contemporary guidance and describe both the “As Is” and “To Be” environments and a sequencing plan for transitioning between the two.

According to FAA officials, attention to and oversight of the enterprise architecture program have been limited in the past, and the agency has not documented its architecture management policies, procedures, and processes; but this is changing. For example, by the end of this fiscal year, FAA plans to issue a policy governing its enterprise architecture efforts and to establish a steering committee to guide and direct the program. By April 2005, the agency also plans to approve an architecture project management plan for the non-NAS architecture. In addition, it plans to have a framework for developing the NAS architecture by September 2005.

Based on our experience in reviewing other agencies, not having an effective enterprise architecture program is attributable to, among other things, limited senior management understanding and commitment and cultural resistance to having and using an architecture. The result is an inability to implement modernized systems in a way that minimizes overlap and duplication and maximizes integration and mission support.

FAA Has Yet to Implement Key Best Practices for Managing Its NAS Architecture Project

As we first reported in 1997, it is critical that FAA have and use a comprehensive NAS architecture to guide and constrain its air traffic control system investment decisions. To effectively develop, maintain, and implement this architecture, FAA will need to employ rigorous and disciplined architecture management practices. Such practices form the basis of our architecture management maturity framework; the five maturity stages of our Version 1.1 framework are described in table 3. Some of these key practices or core elements associated with each of the stages are summarized below. For additional information on these key practices or core elements, see the framework.

For stage 2, our framework specifies nine key practices or core elements that are necessary to provide the management foundation for successfully launching and sustaining an architecture effort. Examples of stage 2 core elements are described below.

Establish a committee or group, representing the enterprise, that is responsible for directing, overseeing, or approving the enterprise architecture. This committee should include executive-level representatives from each line of business, and these representatives should have the authority to commit resources and enforce decisions within their respective organizational units. By establishing this enterprisewide responsibility and accountability, the agency demonstrates its commitment to building the management foundation and obtaining buy-in from across the organization.

Appoint a chief architect. The chief architect should be responsible and accountable for the enterprise architecture, supported by the architecture program office, and overseen by the architecture steering committee. The chief architect, in collaboration with the CIO, the architecture steering committee, and the organizational head is instrumental in obtaining organizational buy-in for the enterprise architecture, including support from the business units, as well as in securing resources to support architecture management functions such as risk management, configuration management, quality assurance, and security management.

Use a framework, methodology, and automated tool to develop the enterprise architecture. These elements are important because they provide the means for developing the architecture in a consistent and efficient manner. The framework provides a formal structure for representing the enterprise architecture, while the methodology is the common set of procedures that the enterprise is to follow in developing the architecture products. The automated tool serves as a repository where architectural products are captured, stored, and maintained.

Develop an architecture program management plan. This plan specifies how and when the architecture is to be developed. It includes a detailed work breakdown structure, resource estimates (e.g., funding, staffing, and training), performance measures, and management controls for developing and maintaining the architecture. The plan demonstrates the organization’s commitment to managing architecture development and maintenance as a formal program.

Our framework similarly identifies key architecture management practices associated with later stages of architecture management maturity. For example, at stage 3—the stage at which organizations focus on architecture development activities—organizations need to satisfy six core elements. Examples of these core elements are discussed below.

Issue a written and approved organization policy for development of the enterprise architecture. The policy defines the scope of the architecture, including the requirement for a description of the baseline and target architectures, as well as an investment road map or sequencing plan specifying the move between the two. This policy is an important means for ensuring enterprisewide commitment to developing an enterprise architecture and for clearly assigning responsibility for doing so.

Ensure that enterprise architecture products are under configuration management. This involves ensuring that changes to products are identified, tracked, monitored, documented, reported, and audited. Configuration management maintains the integrity and consistency of products, which is key to enabling effective integration among related products and for ensuring alignment between architecture artifacts.

At stage 4, during which organizations focus on architecture completion activities, organizations need to satisfy eight core elements. Examples of these core elements are described below.

Ensure that enterprise architecture products and management processes undergo independent verification and validation. This core element involves having an independent third party—such as an internal audit function or a contractor that is not involved with any of the architecture development activities—verify and validate that the products were developed in accordance with architecture processes and product standards. Doing so provides organizations with needed assurance of the quality of the architecture.

Ensure that business, performance, information/data, application/service, and technology descriptions address security. An organization should explicitly and consistently address security in its business, performance, information/data, application/service, and technology architecture products. Because security permeates every aspect of an organization’s operations, the nature and substance of institutionalized security requirements, controls, and standards should be captured in the enterprise architecture products.

At stage 5, during which the focus is on architecture maintenance and implementation activities, organizations need to satisfy eight core elements. Examples of these core elements are described below.

Make the enterprise architecture an integral component of the IT investment management process. Because the road map defines the IT systems that an organization plans to invest in as it transitions from the “As Is” to the “To Be” environment, the enterprise architecture is a critical frame of reference for making IT investment decisions. Using the architecture when making such decisions is important because organizations should approve only those investments that move the organization toward the “To Be” environment, as specified in the road map.

Measure and report return on enterprise architecture investment. Like any investment, the enterprise architecture should produce a return on investment (i.e., a set of benefits), and this return should be measured and reported in relation to costs. Measuring return on investment is important in order to ensure that expected benefits from the architecture are realized and to share this information with executive decision makers, who can then take corrective action to address deviations from expectations.

Table 4 summarizes our framework’s five stages and all of the associated core elements for each.

For its NAS architecture project, FAA is currently at stage 1 of our maturity framework. The NAS project office has satisfied three of the core elements associated with “building the enterprise architecture management foundation”—stage 2 of our framework-—and three of the elements associated with “developing enterprise architecture products”—stage 3 of our framework. It has not satisfied other stage 2 and 3 core elements or any core elements associated with stages 4 and 5. According to the framework, effective architecture management is generally not achieved until an enterprise has a completed and approved architecture that is being effectively maintained and is being used to leverage organizational change and support investment decision making; having these characteristics is equivalent to having satisfied all of the stage 3 core elements and many of the stage 4 and 5 elements.

For the stage 2 core elements, FAA reports that it has allocated adequate resources for developing a NAS architecture. Further, it has established a project office that is responsible for architecture development and maintenance and has assigned a chief architect to the project. However, the agency has not satisfied other core elements for stage 2, such as assigning responsibility for directing, overseeing, or approving the architecture to a committee or group representing the enterprise. Without such an entity to lead and be accountable for the architectural effort, there is increased risk that the architecture will not represent a corporate decision-making tool and will not be viewed and endorsed as an agencywide asset.

With respect to stage 3, according to the CIO, FAA plans to build on the current version of the NAS architecture (Version 5.0) to ensure that architecture products are developed that meet contemporary guidance and standards. According to FAA officials, including the CIO and the chief scientist for the NAS project office, the current NAS architecture does not conform to contemporary architecture guidance or standards—including OMB’s FEA reference models and GAO’s enterprise architecture management maturity framework—because it predates them and has not been updated to comply with them. However, the CIO stated that future versions of the architecture will conform to this guidance. Among other things, this guidance calls for products that describe the “As Is” and “To Be” business, performance, information/data, applications/services, technology, and security environments as well as a sequencing plan for transitioning from the “As Is” to the “To Be” states. Other stage 3 core elements nevertheless have not been met, such as having a written and approved architecture development policy. Further, none of the stage 4 and 5 core elements have been met, although the CIO stated that FAA has recently begun to take steps associated with meeting some of these core elements. The detailed results of our assessment of the NAS project office’s progress in implementing the core elements associated with the five maturity stages are provided in appendix II.

In addition, FAA’s senior enterprise architecture officials, including the program director, stated that attention to and oversight of the enterprise architecture program have been limited in the past and that the agency has not documented its architecture management policies, procedures, and processes. These officials stated that the agency recognizes the need to establish an effective NAS architecture project and that it intends to do so.

To this end, FAA currently plans to have, by September 2005, a framework for developing the architecture and an approved enterprise architecture policy requiring the development, maintenance, and implementation of an enterprise architecture. The CIO also stated that the agency plans to update its NAS architecture to reflect current architecture standards and guidance.

Our research of successful organizations and our experience in reviewing other agencies’ enterprise architecture efforts show that not having these controls is, among other things, a function of limited senior management understanding of and commitment to an enterprise architecture and cultural resistance to having and using one. Until such barriers are addressed and effective architecture management structures and processes are established, it is unlikely that an agency will be able to produce and maintain a complete and enforceable architecture and thus implement modernized systems in a way that minimizes overlap and duplication and maximizes integration and mission support. Given the size and complexity of FAA’s air traffic control systems and their importance to FAA’s ability to achieve its mission, it is critical that FAA develop a well-defined architecture that can be used to guide and constrain system investment decisions.

FAA Has Yet to Implement Key Best Practices for Managing Its Non-NAS Architecture Project

Similar to its NAS architecture effort, FAA’s attempt to develop, maintain, and implement its non-NAS architecture needs to be grounded in the kind of rigorous and disciplined management practices embodied in Version 1.1 of our architecture management maturity framework. (Tables 3 and 4 provide a description of the framework’s five maturity stages and the key practices or core elements associated with each stage.)

For its non-NAS architecture project, FAA is currently at stage 1 of our maturity framework. The non-NAS project office has satisfied three of the core elements associated with “building the enterprise architecture management foundation”—stage 2 of our framework—and four of the core elements associated with stages 3 and 5. According to the framework, effective architecture management is generally not achieved until an enterprise has a completed and approved architecture that is being effectively maintained and is being used to leverage organizational change and support investment decision making; having these characteristics is equivalent to having satisfied all of the stage 2 and 3 core elements and many of the stage 4 and 5 elements.

For stage 2 core elements, FAA reports that it has allocated adequate resources, and it has established a project office and assigned a chief architect. However, the agency has not satisfied several of the stage 2 core elements that are critical to effective architecture management. For example, the agency has not established a committee or group representing the enterprise to guide, direct, or approve the architecture. Having such a corporate entity is critical to overcoming cultural resistance to using an enterprise architecture. As previously stated, the absence of such an entity increases the risk that the architecture will not represent a corporate decision-making tool and will not be viewed and endorsed as an agencywide asset.

Concerning stage 3, FAA has not satisfied three of the six core elements. For example, although the agency is developing architecture products, it does not have a written and approved policy for architecture development. Without such a policy, which, for example, identifies the major players in the development process and provides for architecture guidance, direction, and approval, FAA will be challenged in overcoming cultural resistance to using an enterprise architecture and achieving agencywide commitment and support for an architecture.

The agency has not implemented any of the stage 4 core elements, but it has implemented one core element—architecture products are periodically updated—associated with stage 5 of our framework. For example, FAA has not (1) documented and approved a policy for architecture implementation, (2) implemented an independent verification and validation function that covers architecture products and architecture management processes, and (3) made the architecture an integral component of its IT investment management process. The detailed results of our assessment of the non-NAS project office’s progress in implementing the core elements associated with the five maturity stages are provided in appendix III.

According to FAA’s senior enterprise architecture officials, including the chief architect, the attention to and oversight of the enterprise architecture program have been limited in the past, and the agency has not documented its architecture management policies, procedures, and processes. FAA officials, including the CIO and the chief architect for the non-NAS project, agreed with our assessment of the project office’s current architecture management capabilities. These officials stated that the agency recognizes the need to establish an effective non-NAS architecture project, and it intends to do so. For example, the agency’s strategic plan includes the goal of having an approved enterprise architecture policy requiring the development, maintenance, and implementation of an enterprise architecture by September 2005, and the agency intends to establish a steering committee. In addition, the chief architect stated that FAA plans to have an approved architecture project management plan by April 2005, and a comprehensive version of the non-NAS architecture by April 2006.

As previously stated, our research and our experience show that not having these controls is, among, other things, attributable to limited senior management understanding of and commitment to an enterprise architecture and cultural resistance to having and using one. Until such barriers are addressed and effective architecture management structures and processes are established, it is unlikely that any agency will be able to develop and maintain a complete and enforceable architecture and thus implement modernized systems in a way that minimizes overlap and duplication and maximizes integration and mission support.

Conclusions

Having a well-defined and enforced enterprise architecture is critical to FAA’s ability to effectively and efficiently modernize its NAS and non-NAS systems. To accomplish this, it is important for FAA to establish effective management practices for developing, maintaining, and implementing an architecture. Currently, FAA does not have these practices in place. Establishing them begins with agency top management commitment and support for having and using an architecture to guide and constrain investment decision making.

Recommendations for Executive Action

To ensure that FAA has the necessary agencywide context within which to make informed decisions about its air traffic control system and other systems modernization efforts, we recommend that the Secretary of the Department of Transportation direct the FAA Administrator to ensure that the following four actions take place.

Demonstrate institutional commitment to and support for developing and using an enterprise architecture by issuing a written and approved enterprise architecture policy.

Ensure that the CIO, in collaboration with the COO, implements, for the NAS architecture project, the best practices involved in stages 2 through 5 of our enterprise architecture management maturity framework.

Ensure that the CIO focuses first on developing and implementing a NAS architecture.

Ensure that the CIO implements, for the non-NAS architecture project, the best practices involved in stages 2 through 5 of our enterprise architecture management maturity framework.

Agency Comments

In commenting on a draft of this report, the Department of Transportation’s Director of Audit Relations stated via e-mail that FAA is continuing its NAS architecture efforts. The Director also provided technical comments, which we have incorporated as appropriate in the report. The Director’s comments did not state whether the department agreed or disagreed with the report’s conclusions and recommendations.

As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to interested congressional committees, the Director of OMB, the Secretary of the Department of Transportation, the FAA Administrator, FAA’s CIO, and FAA’s COO. We will also make copies available to others upon request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions on matters discussed in this report, please contact Randolph C. Hite at (202) 512-3439 or hiter@gao.gov, or David A. Powner at (202) 512-9286 or pownerd@gao.gov. Major contributors to this report are acknowledged in appendix IV.

Objective, Scope, and Methodology

Our objective was to determine whether the Federal Aviation Administration (FAA) has established effective processes for managing the development and implementation of an enterprise architecture.

To address our objective, we used our enterprise architecture management maturity framework, Version 1.1, which organizes architecture management best practices into five stages of management maturity. Specifically, we compared our framework to the ongoing efforts of FAA’s two project offices to develop the National Airspace System (NAS) and non-NAS segments of the architecture. For example, for the NAS architecture, we reviewed program documentation, such as the acquisition management system policy, the Joint Resources Council’s investment management guidance for NAS investments, and FAA’s NAS architecture development process. We reviewed, for the non-NAS architecture, program documentation, such as the methodology FAA is using to develop this architecture, a Systems Research and Applications report on the agency’s efforts to implement management processes and controls over its architecture development activities, and the Department of Transportation’s Enterprise Architecture Subcommittee and Architecture Review Board charters. We then compared these documents with the elements in our framework.

To augment our documentation reviews of FAA’s architecture management efforts, we interviewed various officials, including the chief information officer, the program director, the chief architects for the NAS and non-NAS architectures, and the chief scientist for the NAS architecture, to determine, among other things, the agency’s plans to develop an enterprise architecture. Specifically, we inquired about (1) the agency’s plans for developing an enterprise architecture, including the key milestones and deliverables for completing the two segments of the architecture, (2) the content of the NAS and non-NAS architecture segments (i.e., architecture products that have been developed to date), and (3) the strategy to be used to align the NAS and non-NAS architectures.

We conducted our work at FAA headquarters in Washington, D.C. We performed our work from June 2004 to March 2005, in accordance with generally accepted government auditing standards.

Assessment of Architecture Management Efforts for the National Airspace System

Agency is aware of EA.

The Federal Aviation Administration (FAA) strategic plan includes the goal of having an approved EA policy requiring the development, maintenance, and implementation of an EA by September 2005.

Adequate resources exist (funding, people, tools, and technology).

According to the chief scientist and the chief architect for the NAS architecture, the agency has adequate project funding. FAA reports that fiscal year 2005 funding for the National Airspace System (NAS) architecture is about $2.6 million. In addition, the agency reports that it has skilled staff, including contractor support, for its NAS architecture project. Furthermore, FAA is using automated tools and technology, such as Rational Rose by Rational Software Corporation/IBM Software Group, CORE by Vitech Corporation, and Dynamic Object Oriented Requirements System by Telelogic.

Committee or group representing the enterprise is responsible for directing, overseeing, or approving the EA.

FAA has not assigned responsibility for directing, overseeing, or approving a NAS architecture to a group or committee representing the enterprise.

Program office responsible for EA development and maintenance exists.

In 1997, FAA established a project office that is responsible for developing and maintaining a NAS architecture.

Chief architect exists.

In February 2004, FAA designated the chief architect for the NAS architecture project.

EA is being developed using a framework, methodology, and automated tool.

According to the chief scientist, the NAS architecture is being developed using a framework that focuses on strategically supporting FAA’s investment management process. The chief scientist stated that, unlike other architecture frameworks, this framework is not fully developed or documented. Further, FAA has yet to provide us with any documentation on this framework and on how it is being implemented to support the agency’s investment management process. According to the chief information officer (CIO), the agency plans to select an architecture framework by September 2005. FAA does not have a methodology that defines the standards, steps, tools, techniques, and measures that it is following to develop, maintain, and validate a NAS architecture. As stated above, FAA is using automated tools to build a NAS architecture.

EA plans call for describing both the “As Is” and the “To Be” environments of the enterprise, as well as a sequencing plan for transitioning from the “As Is” to the “To Be.”

FAA has yet to develop architecture project management plans.

EA plans call for describing both the “As Is” and the “To Be” environments in terms of business, performance, information/data, application/service, and technology.

FAA has yet to develop architecture project management plans.

EA plans call for business, performance, information/data, application/service, and technology descriptions to address security.

FAA has yet to develop architecture project management plans.

EA plans call for developing metrics for measuring EA progress, quality, compliance, and return on investment.

FAA has yet to develop architecture project management plans and metrics for measuring NAS architecture progress, quality, compliance, and return on investment.

Written/approved organization policy exists for EA development.

FAA has yet to develop a written/approved policy for developing a NAS architecture. However, FAA’s strategic plan includes the goal of developing an approved enterprise architecture policy by September 2005.

EA products are under configuration management.

FAA has yet to establish a configuration management process.

EA products describe or will describe both the “As Is” and the “To Be” environments of the enterprise, as well as a sequencing plan for transitioning from the “As Is” to the “To Be.”

According to the chief information officer (CIO), future versions of the NAS architecture will conform to contemporary guidance. Such guidance describes, among other things, products that describe the “As Is” and “To Be” environments and a sequencing plan.

Both the “As Is” and the “To Be” environments are described or will be described in terms of business, performance, information/data, application/service, and technology.

According to the CIO, future versions of the NAS architecture will conform to contemporary guidance. Such guidance describes, among other things, products that describe the “As Is” and “To Be” environments in terms of business, performance, information/data, application/service, and technology.

Business, performance, information/data, application/service, and technology descriptions address or will address security.

According to the CIO, future versions of the NAS architecture will conform to contemporary guidance. Such guidance includes, among other things, business, performance, information/data, application/service, and technology descriptions that address security in both the “As Is” and “To Be” environments.

Progress against EA plans is measured and reported.

FAA has yet to develop architecture project management plans and metrics; therefore, progress against plans is not measured and reported.

Written/approved organization policy exists for EA maintenance.

FAA has yet to develop a written/approved policy for maintaining a NAS architecture. However, FAA’s strategic plan includes the goal of developing an approved enterprise architecture policy by September 2005.

EA products and management processes undergo independent verification and validation.

FAA has yet to establish an independent verification and validation process.

EA products describe both the “As Is” and the “To Be” environments of the enterprise, as well as a sequencing plan for transitioning from the” As Is” to the “To Be.”

FAA has yet to develop NAS architecture products that describe both the “As Is” and the “To Be” environments and a sequencing plan.

Both the “As Is” and the “To Be” environments are described in terms of business, performance, information/data, application/service, and technology.

FAA has yet to develop NAS architecture products that describe both the “As Is” and the “To Be” environments in terms of business, performance, information/data, application/service, and technology.

Business, performance, information/data, application/service, and technology descriptions address security.

FAA has yet to develop business, performance, information/data, application/service, and technology descriptions that address security in both the “As Is” and “To Be” environments.

Organization CIO has approved current version of EA.

FAA has yet to develop a version of the NAS architecture for the CIO to approve that conforms to contemporary guidance and standards.

Committee or group representing the enterprise or the investment review board has approved current version of EA.

FAA has yet to develop a version of the NAS architecture that conforms to contemporary guidance and standards for a committee or investment review board to approve.

Quality of EA products is measured and reported.

FAA has yet to develop NAS architecture product metrics; therefore, product quality is not measured and reported.

Written/approved organization policy exists for IT investment compliance with EA.

FAA has yet to develop a written/approved policy requiring IT investments to comply with a NAS architecture. However, FAA’s strategic plan includes the goal of developing an approved enterprise architecture policy by September 2005.

Process exists to formally manage EA change.

FAA has yet to establish a formal process for managing changes to a NAS architecture.

EA is integral component of IT investment management process.

According to the CIO, FAA has recently begun to consider architecture compliance as part of its Joint Resources Council process and the CIO’s approval of Exhibit 300 budget exhibits for NAS investments, and he anticipates that over the next couple of years the NAS architecture will become integral to the investment process.

EA products are periodically updated.

FAA has yet to complete development of NAS architecture products.

IT investments comply with EA.

According to the CIO, FAA has recently begun to consider investment compliance with the architecture, and the CIO expects this compliance determination to expand and evolve over the next couple of years.

Organization head has approved current version of EA.

FAA has yet to complete development of a NAS architecture for the Administrator to approve.

Return on EA investment is measured and reported.

FAA has yet to develop metrics and processes for measuring NAS architecture benefits; therefore, return on investment is not measured and reported.

Compliance with EA is measured and reported.

FAA has yet to develop metrics for measuring compliance with the NAS architecture; therefore, compliance with an architecture is not measured and reported.

Assessment of Architecture Management Efforts for the Non-National Airspace System

Agency is aware of EA.

The Federal Aviation Administration (FAA) strategic plan includes the goal of having an approved EA policy requiring the development, maintenance, and implementation of an EA by September 2005.

Adequate resources exist (funding, people, tools, and technology).

According to the chief architect, the agency has adequate project funding. FAA reports that fiscal year 2005 funding for the non-National Airspace System (NAS) architecture is $1.5 million. In addition, the agency reports that it has skilled staff (two government employees, six full-time contractors, and additional contractor staff as needed) working to develop its non- NAS architecture. FAA is also using automated tools, such as Rational Rose by Rational Software Corporation/IBM Software Group, Microsoft Visio and an Oracle portal server.

Committee or group representing the enterprise is responsible for directing, overseeing, or approving the EA.

FAA has not assigned responsibility for directing, overseeing, or approving the non-NAS architecture to any group or committee. According to the chief architect, FAA plans to assign responsibility for directing the non-NAS architecture to its Information Technology Executive Board by April 2005.

Program office responsible for EA development and maintenance exists.

In January 2003, FAA established a project office that is responsible for developing and maintaining the non- NAS architecture.

Chief architect exists.

In January 2003, FAA designated a chief architect for the non-NAS architecture.

EA is being developed using a framework, methodology, and automated tool.

According to the chief architect, FAA is using the Federal Enterprise Architecture Framework and the Office of Management and Budget’s Federal Enterprise Architecture reference models to develop the non-NAS architecture. FAA also has a methodology for developing the architecture, but the methodology does not define the standards, steps, tools, techniques, and measures that it is following to develop, maintain, and validate the non-NAS architecture. However, according to the chief architect, FAA will update its methodology to describe management activities by March 2005. As stated above, FAA is using automated tools to build the non-NAS architecture.

EA plans call for describing both the “As Is” and the “To Be” environments of the enterprise, as well as a sequencing plan for transitioning from the “As Is” to the “To Be.”

FAA has yet to develop architecture project management plans. However, the chief architect stated that the agency intends to have an approved plan by April 2005 and the plan will call for describing both the “As Is” and the “To Be” environments of the enterprise, as well as a sequencing plan.

EA plans call for describing both the “As Is” and the “To Be” environments in terms of business, performance, information/data, application/service, and technology.

FAA has yet to develop architecture project management plans. However, the chief architect stated that the agency intends to have an approved plan by April 2005 and the plan will call for describing both the “As Is” and the “To Be” environments in terms of business, performance, information/data, application/service, and technology.

EA plans call for business, performance, information/data, application/service, and technology descriptions to address security.

FAA has yet to develop architecture project management plans. However, the chief architect stated that the agency intends to have an approved plan by April 2005 and the plan will call for the business, performance, information/data, application/service, and technology descriptions to address security for both the “As Is” and “To Be” environments.

EA plans call for developing metrics for measuring EA progress, quality, compliance, and return on investment.

FAA has yet to develop architecture project management plans and metrics. According to the chief architect, the agency intends to have an approved plan by April 2005 and the plan will include metrics for measuring non-NAS architecture progress, quality, compliance, and return on investment.

Written/approved organization policy exists for EA development.

FAA has yet to develop a written/approved policy for developing the non-NAS architecture. However, FAA’s strategic plan includes the goal of developing an approved enterprise architecture policy by September 2005.

EA products are under configuration management.

FAA has yet to develop non-NAS architecture products and a configuration management process has not been established. However, according to the chief architect, FAA plans to update its methodology to address how changes to all architecture products will be documented by March 2005.

EA products describe or will describe both the “As Is” and the “To Be” environments of the enterprise, as well as a sequencing plan for transitioning from the “As Is” to the “To Be.”

The chief architect stated that the agency intends to have an approved plan by April 2005 and that the plan will call for describing both the “As Is” and the “To Be” environments of the enterprise, as well as a sequencing plan. The chief architect also stated that the agency will have a comprehensive non-NAS architecture by April 2006.

Both the “As Is” and the “To Be” environments are described or will be described in terms of business, performance, information/data, application/service, and technology.

The chief architect stated that the agency intends to have an approved plan by April 2005 and that the plan will call for describing both the “As Is” and the “To Be” environments in terms of business, performance, information/data, application/service, and technology. The chief architect also stated that the agency will have a comprehensive non-NAS architecture by April 2006.

Business, performance, information/data, application/service, and technology descriptions address or will address security.

The chief architect stated that the agency intends to have an approved plan by April 2005 and that the plan will call for the business, performance, information/data, application/service, and technology descriptions to address security for both the “As Is” and “To Be” environments. The chief architect also stated that the agency will have a comprehensive non-NAS architecture by April 2006.

Progress against EA plans is measured and reported.

FAA has yet to develop architecture project management plans and metrics; therefore, progress against plans is not measured and reported. However, according to the chief architect, the agency intends to have an approved plan by April 2005 and progress against the plan will be measured and reported.

Written/approved organization policy exists for EA maintenance.

FAA has yet to develop a written/approved policy for maintaining the non-NAS architecture. However, FAA’s strategic plan includes the goal of developing an approved enterprise architecture policy by September 2005.

EA products and management processes undergo independent verification and validation.

FAA has yet to establish an independent verification and validation process. However, according to the chief architect, the non-NAS architecture products and architecture management processes will undergo independent verification and validation by December 2005.

EA products describe both the “As Is” and the “To Be” environments of the enterprise, as well as a sequencing plan for transitioning from the” As Is” to the “To Be.”

The current non-NAS architecture products do not yet fully describe both the “As Is” and the “To Be” environments of the enterprise, or a sequencing plan. However, according to the chief architect, FAA will have a comprehensive non-NAS architecture that describes both the “As Is” and the “To Be” environments of the enterprise, as well as the sequencing plan by April 2006.

Both the “As Is” and the “To Be” environments are described in terms of business, performance, information/data, application/service, and technology.

The current non-NAS architecture products do not yet fully describe both the “As Is” and the “To Be” environments in terms of business, performance, information/data, application/service, and technology. However, according to the chief architect, FAA will have a comprehensive non-NAS architecture that describes these terms by April 2006.

Business, performance, information/data, application/service, and technology descriptions address security.

According to the chief architect, the non-NAS architecture does not yet contain complete business, performance, information/data, application/service, and technology descriptions that address security for both the “As Is” and “To Be” environments. However, FAA will have a comprehensive non-NAS architecture comprised of these terms by April 2006.

Organization chief information officer (CIO) has approved current version of EA.

FAA has yet to develop a non-NAS architecture for the CIO to approve. According to the chief architect, the first comprehensive version of the non-NAS architecture is scheduled for release in April 2006.

Committee or group representing the enterprise or the investment review board has approved current version of EA.

FAA has yet to develop a non-NAS architecture for the committee or investment review board to approve. According to the chief architect, the first comprehensive version of the non-NAS architecture is scheduled for release in April 2006. In addition, FAA has yet to establish a committee or investment review board that will be responsible for approving the non-NAS architecture.

Quality of EA products is measured and reported.

FAA has yet to develop metrics and assess the quality of the non-NAS architecture products that it is currently developing; therefore, product quality is not measured and reported. However, according to the chief architect, the agency intends to have an approved plan by April 2005 and the plan will include metrics for measuring the quality of the non-NAS architecture products.

Written/approved organization policy exists for IT investment compliance with EA.

FAA has yet to develop a written/approved policy requiring that IT investments comply with the architecture. However, FAA’s strategic plan includes the goal of developing an approved enterprise architecture policy by September 2005.

Process exists to formally manage EA change.

FAA has yet to establish a formal process for managing changes to the non-NAS architecture. However, according to the chief architect, the agency intends to have an approved architecture project management plan by April 2005 and the plan will include a formal process for managing architecture changes.

EA is integral component of IT investment management process.

FAA has yet to complete development of a non-NAS architecture, and it is not an integral component of the IT investment management process.

EA products are periodically updated.

FAA updates the non-NAS architecture products annually to reflect the agency’s investment decisions.

IT investments comply with EA.

FAA has yet to complete development of a non-NAS architecture; therefore, IT investments are not evaluated for compliance with the architecture. However, the first version of the non-NAS architecture is scheduled for release in April 2006.

Organization head has approved current version of EA.

FAA has yet to complete development of a non-NAS architecture for the Administrator to approve. However, the first version of the non-NAS architecture is scheduled for release in April 2006.

Return on EA investment is measured and reported.

FAA has yet to develop metrics and processes for measuring non-NAS architecture benefits; therefore, return on investment is not measured and reported. However, the first version of the non-NAS architecture is scheduled for release in April 2006.

Compliance with EA is measured and reported.

FAA has yet to develop metrics for measuring compliance with the non-NAS architecture. However, the first version of the non-NAS architecture is scheduled for release in April 2006.

According to the chief architect, these core elements will be addressed in the enterprise architecture policy that FAA plans to issue by September 2005.

GAO Staff Acknowledgments

Staff Acknowledgments

Staff who made key contributions to this report were Kristina Badali, Joanne Fiorino, Michael Holland, Anh Le, William Wadsworth, and Angela Watson.