Information Technology: Architecture Needed to Guide NASA's Financial Management Modernization
Published: 2003-11-21T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The National Aeronautics and Space Administration (NASA) is in the process of modernizing its financial management operations and supporting information technology systems. This modernization, known as the Integrated Financial Management Program (IFMP), is intended to provide NASA with an agencywide, integrated approach to performing critical business functions, such as contract management--an area that GAO first designated as high risk in 1990 and continues to do so today. GAO was requested to review various aspects of IFMP, and this report is one in a series on the program. The objective of this review was to determine whether NASA has been acquiring and implementing IFMP in the context of an enterprise architecture.
section_title: What GAO Found
paragraphs: To date, NASA has acquired and implemented significant components of IFMP without an enterprise architecture to guide and constrain the program. An enterprise architecture is an organizational blueprint that defines--in both business and technology terms--how an organization operates today and how it intends to operate in the future; it also provides a plan for transitioning to this future state. Using an enterprise architecture to guide and constrain systems modernization programs is a federal requirement and a recognized best practice of successful public and private organizations. In addition, GAO's research has shown that attempting major modernization programs such as IFMP without a well-defined enterprise architecture risks, among other things, building systems that are duplicative, are not interoperable, and do not effectively and efficiently support mission operations and performance. During the course of GAO's work, NASA recognized the need for an enterprise architecture and has taken steps to develop one. For example, it has established an architecture program office, designated a chief architect, and selected an architecture framework to use. In addition, after GAO completed its audit work, NASA released an initial version of an enterprise architecture, which the chief technology officer stated was not yet complete and would be improved upon in future versions. However, the agency has yet to establish other key architecture management capabilities, such as designating an accountable corporate entity to lead the architecture effort, having an approved policy for developing and maintaining the architecture, and implementing an independent verification and validation function to provide needed assurance that architecture products and architecture management processes are effective. Moreover, the architecture products used to date to manage NASA's investment in IFMP did not provide sufficient context (depth and scope of agencywide operational and technical requirements) to effectively guide and constrain the program. The chief technology officer agreed that NASA needs an effective enterprise architecture program and stated that efforts are under way to establish one. GAO's experience in reviewing other agencies has shown that not having an effective enterprise architecture program can be attributed to, among other things, an absence of senior management understanding and support, as well as cultural resistance. NASA's current approach to acquiring and implementing IFMP outside the context of an architecture unnecessarily increases the risk that the program's system components will not effectively and efficiently support agencywide operations. The result will be costly system rework. It is critical for NASA to discontinue this approach and adopt the best practice of managing its IFMP system investments within the context of a well-defined enterprise architecture.

section_title: Letter
paragraphs: To improve its ability to manage its contractors, which is an area that we  designated as high risk in 1990, and continue to do so today, the National  Aeronautics and Space Administration (NASA) began its third attempt at  modernizing its financial management systems in April 2000. This  modernization effort, known as the Integrated Financial Management  Program (IFMP), is expected to produce an integrated, NASA-wide  business systems environment by acquiring and incrementally  implementing commercial hardware and software components. Our  research of successful public- and private-sector organizations shows that  attempting a modernization program, like IFMP, without having and using a  well-defined modernization blueprint, commonly called an enterprise  architecture, results in operations and systems that are duplicative, are not  well integrated, are unnecessarily costly to maintain and interface, and do  not effectively optimize mission performance.
In April 2003, we issued the first in a series of reports on the program, in  which we concluded that NASA’s approach to acquiring and implementing  IFMP components had and would continue to introduce risk and increase  the chances that the agency would fall short of meeting its program goal.  Because of the importance of IFMP to overall mission performance, you  asked us to continue our review. Specifically, you requested that we  determine whether (1) NASA has been acquiring and implementing IFMP in  the context of an enterprise architecture, (2) the core financial module as  implemented in June 2003 would satisfy NASA’s key external reporting  requirements, and (3) NASA’s life-cycle cost estimate, program schedule,  and funding reserves for IFMP were reasonable.
We are responding to the second two issues in separate reports, as well as  summarizing our findings on all three areas in an additional report. This  report addresses the first issue—whether NASA had and was using an  enterprise architecture to acquire and implement IFMP. To accomplish this,  we compared the architecture documents that NASA provided us, and  represented as being used to manage IFMP, against published guidance  governing the content of a well-defined architecture. We also compared  NASA’s architecture development, maintenance, and implementation  practices against our enterprise architecture management maturity  framework. Details on our objective, scope, and methodology are in  appendix I.
subsections: section_title: Background
paragraphs: NASA’s mission encompasses human exploration and development of  space, the advancement and communication of scientific knowledge, and  research and development of aeronautics and space technologies. Its  activities span a broad range of complex and technical endeavors—from  investigating the composition, evaluation, and resources of Mars; to  working with the agency’s international partners to complete and operate  the International Space Station; to providing satellite and aircraft  observations of Earth for scientific and weather forecasting purposes; to  developing new technologies designed to improve air safety. NASA’s  workforce comprises over 19,000 civil service employees, primarily located  at its headquarters and 10 major field centers, and more than 40,000  contractors and grantees, who collectively perform a wide range of roles  and responsibilities. Table 1 describes the roles and responsibilities of  NASA’s headquarters and field centers.
Transcending NASA’s organizational components are six strategic mission  enterprises or business areas, each with a unique set of strategic goals,  objectives, and implementation strategies focused on the requirements of  the agency’s customers. Each enterprise draws on the capabilities of  several centers so that each center contributes to multiple enterprises.  Table 2 summarizes NASA’s strategic enterprises and the contributing  centers.
To execute its mission responsibilities, NASA performs numerous  management functions, such as contract management, financial  management, and human capital management, relying heavily on  information technology (IT) to assist it in performing these functions. For  fiscal year 2003, the agency estimated that it would spend approximately  $2.3 billion on IT systems and services. Of this amount, NASA anticipated  spending $32.5 million on IT security and $11 million on enterprise  architecture.
subsections: section_title: NASA Continues to Face Challenges in Managing Large Programs
paragraphs: In January 2003, we reported that NASA faced challenges that threaten its  ability to effectively run its largest programs. We also reported that because  these challenges are rooted in NASA’s culture and long-standing ways of  doing business, the agency needed to make a major transformation. In  particular, we identified the following four performance and accountability  challenges facing the agency:  Strengthening strategic human capital management. NASA is  facing shortages in its workforce, which could likely worsen as the  workforce continues to age and the pipeline of talent shrinks. This  dilemma is more pronounced among areas crucial to NASA’s ability to  perform its mission, such as engineering, science, and IT. NASA is  addressing this challenge through strategic planning, through a new  workforce planning and analysis system, and by requesting additional  personnel flexibilities, among other initiatives.
Controlling International Space Station costs. Development costs  for this project have soared to the point where NASA has had to cut  back the program substantially, including reducing construction, the  number of crew members, and scientific research. These cutbacks have  raised concern among NASA’s international partners, who have a large  stake in the scientific research to be performed on the station. Although  NASA is instituting management and cost-estimating reforms, it still  needs to reach agreement with its partners on its planned cutbacks.
Reducing costs of space launches. The administration submitted an  amendment to NASA’s fiscal year 2003 budget request, which (1)  extends the life of the space shuttle and enhances its reliability, (2)  funds the development of a new vehicle for ferrying crew to and from  the space station, and (3) alters the time frame for a shuttle  replacement. Accomplishing these and other goals related to space  launches will be difficult and risky in light of the technology advances  that NASA would like to pursue and the high degree of communication  and coordination required among industry and government partners.
Improving contract management. NASA spends most of its funds on  acquisitions. Yet, for many years, the agency has been unable to  oversee contracts effectively, principally because it lacked accurate and  reliable information on contract spending and placed little emphasis on  end results, product performance, and cost control. NASA has  addressed many acquisition-related weaknesses and is beginning to  tackle one of its most formidable barriers to sound contract  management—the lack of a modern, integrated financial management  system. Considerable work remains to be done since NASA is only in the  early stages of designing and implementing this new system, and NASA  reported that it is already facing challenges in terms of cost,  interoperability, and security.
We also reported that NASA’s ability to collect, maintain, and report the full  cost of its projects and programs is weakened by diverse and often  incompatible and nonintegrated center-level accounting systems; uneven  and nonstandard cost-reporting capabilities; decentralized policies,  procedures, and practices that are unique to its field centers; nonstandard  data formats; and online financial information that is not readily available  to program managers. Thus, it is difficult to ensure that contracts are being  efficiently and effectively implemented and that budgets are executed as  planned. This lack of integration and standardization also impedes the  agency’s ability to provide data required for external reporting purposes.
Recognizing the need for change, NASA’s Administrator articulated a new  vision for the agency—one that is science-driven, not destination-driven. To  better enable NASA to fulfill this vision, the agency is taking on a major  transformation aimed at eliminating stovepipes, becoming more integrated  and results-oriented, and reducing risks while working more economically,  efficiently, and effectively.
section_title: NASA Has Initiated a Large, Complex Systems Modernization to Address Financial Management Concerns
paragraphs: A key transformation effort is IFMP, which is NASA’s third attempt in more  than 12 years to modernize its financial management processes and  systems. NASA spent about $180 million on its two prior failed efforts, and  NASA’s data indicate that the agency will spend approximately $983 million  through 2010 for its current effort, IFMP, which it began in April 2000.  IFMP is expected to produce an integrated, NASA-wide financial  management system by acquiring and incrementally implementing  commercial software packages and related hardware and software  components. The main objective of IFMP is to improve the financial,  physical, and human capital management processes throughout the agency.  According to NASA, once fully implemented, IFMP will reengineer NASA’s  business operations around industry “best practices” and use enabling  technology to provide necessary management information to support  implementation of the agency’s strategic plan. To meet this objective and  support these crosscutting activities, NASA has identified the following  business drivers for the program:  providing timely, consistent, and reliable information for management   improving NASA’s accountability and enabling full cost management;  achieving increased efficiencies and operating effectively;  exchanging information with customers and stakeholders in a timely  and reliable way; and  attracting and retaining a world-class workforce.
The IFMP system is to consist of nine modules supporting a range of  functionality (see table 3).
As structured, NASA is the IFMP system integrator and thus is responsible  for acquiring and integrating the multiple commercial components and  ensuring that they collectively perform in a manner that meets the defined  requirements. Table 4 describes the key IFMP program management  positions/entities and their respective responsibilities.
section_title: Early Problems with IFMP Have Been Reported
paragraphs: In April 2003, we reported that NASA was not following key best practices  for acquiring and implementing IFMP. For example, the agency had not  established an analytical capability to understand and proactively manage  the dependencies among IFMP commercial components. Further, in  implementing the core financial module component, NASA had deferred  addressing the needs of key systems users and had not properly developed  detailed system requirements. We concluded that the agency was at risk of  making a substantial investment in a system that would fall far short of its  stated goal of providing meaningful and reliable information to support  effective program management and congressional oversight.
To address its problems, we recommended that NASA (1) develop and  implement a short-term plan to identify and mitigate the risks currently  associated with relying on already deployed IFMP commercial components  and to expeditiously stabilize these components’ operational capability and  performance; (2) as part of the short-term plan, develop and properly  document requirements, reengineer acquisition management processes,  and fully engage stakeholders—including program managers, cost  estimators, and the Congress—in the development of user requirements;  and (3) develop a longer-term strategy for acquiring additional IFMP  components that includes implementing a methodology for analyzing  commercial system component dependencies. NASA concurred with the  need for a short-term plan but disagreed with most of our findings and  recommendations related to user needs and requirements and testing.  NASA also agreed with the importance of having an approach for acquiring  additional IFMP components, but stated that it already has an effective  strategy in place.
In May 2003, NASA’s OIG reported that the core financial module  software, which had been deployed at six NASA centers, had the capability  to implement full cost accounting. However, before this implementation  could take place, NASA needed to resolve several complex accounting and  costing issues. These issues involved how to allocate service and general  and administrative costs, civil service costs, and unassigned costs. Once  these accounting and costing issues were resolved, the OIG reported that  NASA would have to configure the IFMP software to reflect the changes.  The OIG recommended that NASA revise the IFMP plans to include  (1) time frames and milestones for completing steps to implement full cost  accounting, including addressing and resolving the cost issues identified  above; (2) identification of the personnel and other resources necessary to  perform the steps within the established time frames; and (3) senior  management approval and support of these additional procedures. IFMP  officials concurred with the recommendations and plan to have all phases  of full cost accounting implemented by October 1, 2003. (NASA reported  that full implementation of the core financial module at all centers was  completed in June 2003.)
section_title: An Enterprise Architecture Is Critical to an Organization’s Ability to Effectively Modernize Its Business Operations and Systems
paragraphs: Effective use of enterprise architectures, or modernization blueprints, is a  trademark of successful public and private organizations. For a decade, we  have promoted the use of architectures to guide and constrain systems  modernization, recognizing them as a crucial means to a challenging goal:  agency operational structures that are optimally defined in both business  and technological environments. The Congress, the Office of Management  and Budget (OMB), and the federal Chief Information Officer (CIO)  Council have also recognized the importance of an architecture-centric  approach to modernization. The Clinger-Cohen Act of 1996 mandates that  an agency’s CIO develop, maintain, and facilitate the implementation of  architectures as a means for managing the integration of business  processes and supporting systems. Further, OMB has issued guidance that,  among other things, requires system investments to be consistent with  these architectures.
An enterprise architecture provides a clear and comprehensive picture of  an entity, whether it is an organization (e.g., federal department or agency)  or a functional or mission area that cuts across more than one organization  (e.g., financial management). This picture consists of snapshots of both the  enterprise’s current or “As Is” operational and technological environment  and its target or “To Be” environment, as well as a capital investment road  map for transitioning from the current to the target environment. These  snapshots further consist of “views,” which are basically one or more  architecture products that provide conceptual or logical representations of  the enterprise.
The suite of products and their content that form a given entity’s enterprise  architecture are largely governed by the framework used to develop the  architecture. Since the 1980’s, various frameworks have emerged and been  applied. For example, John Zachman developed a structure or “framework”  for defining and capturing an architecture. This framework provides for  six windows from which to view the enterprise, which Zachman terms  “perspectives” on how a given entity operates: those of (1) the strategic  planner, (2) the system user, (3) the system designer, (4) the system  developer, (5) the subcontractor, and (6) the system itself. Zachman also  proposed six abstractions or models associated with each of these  perspectives: these models cover (1) how the entity operates, (2) what the  entity uses to operate, (3) where the entity operates, (4) who operates the  entity, (5) when entity operations occur, and (6) why the entity operates.
In September 1999, the federal CIO Council published the Federal  Enterprise Architecture Framework (FEAF), which is intended to provide  federal agencies with a common construct for their respective  architectures, thereby facilitating the coordination of common business  processes, technology insertion, information flows, and system  investments among federal agencies. FEAF describes an approach,  including models and definitions, for developing and documenting  architecture descriptions for multiorganizational functional segments of  the federal government. Similar to most frameworks, FEAF’s proposed  models describe an entity’s business, data necessary to conduct the  business, applications to manage the data, and technology to support the  applications.
More recently, OMB established the Federal Enterprise Architecture  Program Management Office to develop a federated enterprise architecture  according to a collection of five “reference models:”
The Business Reference Model is intended to describe the business  operations of the federal government independent of the agencies that  perform them, including defining the services provided to state and local  governments.
The Performance Reference Model is to provide a common set of  general performance outputs and measures for agencies to use to  achieve business goals and objectives.
The Data and Information Reference Model is to describe, at an  aggregate level, the types of data and information that support program  and business line operations, and the relationships among these types.
The Service Component Reference Model is to identify and classify IT  service (i.e., application) components that support federal agencies and  promote the reuse of components across agencies.
The Technical Reference Model is to describe how technology is  supporting the delivery of service components, including relevant  standards for implementing the technology.
These various enterprise architecture frameworks differ in their  nomenclatures and modeling approach. However, the frameworks  consistently provide for defining an enterprise’s operations in both  (1) logical terms, such as interrelated business processes and business  rules, information needs and flows, and work locations and users, and  (2) technical terms, such as hardware, software, data, communications,  and security attributes and performance standards. The frameworks also  provide for defining these perspectives for both the enterprise’s current or  “As Is” environment and its target or “To Be” environment, as well as a  transition plan for moving from the “As Is” to the “To Be” environment.
The importance of developing, implementing, and maintaining an  enterprise architecture is a basic tenet of both organizational  transformation and IT management. Managed properly, an enterprise  architecture can clarify and help optimize the interdependencies and  relationships among an organization’s business operations and the  underlying IT infrastructure and applications that support these  operations. Employed in concert with other important management  controls, such as portfolio-based capital planning and investment control  practices, architectures can greatly increase the chances that  organizations’ operational and IT environments will be configured to  optimize mission performance. Our experience with federal agencies has  shown that investing in IT without defining these investments in the  context of an architecture often results in systems that are duplicative, not  well integrated, and unnecessarily costly to maintain and interface.
section_title: IFMP Has Proceeded without an Enterprise Architecture, and NASA’s Ongoing Architecture Management Efforts Are Missing Key Elements
paragraphs: NASA has thus far acquired and deployed IFMP without a sufficiently  complete enterprise architecture to guide and constrain program  investment decisions. During the course of our review of IFMP, NASA took  steps to correct this situation by establishing key architecture management  capabilities and undertaking the development of an initial version of an  enterprise architecture that, according to the chief technology officer, will  provide some missing contextual information (operational and technical).  However, NASA has not established other key architecture management  capabilities, such as designating an accountable corporate entity to lead  the architecture effort, having an approved policy for developing and  maintaining the architecture, and implementing an independent  verification and validation function to provide needed assurance that  architecture products and architecture management processes are  effective. The chief technology officer agreed that NASA needs an effective  enterprise architecture program and stated that efforts are under way to  establish one. Based on our experience in reviewing other agencies, not  having an effective enterprise architecture program is attributable to,  among other things, limited senior management understanding and  commitment and cultural resistance to having and using an architecture.  The result is an inability to implement modernized systems in a way that  minimizes overlap and duplication and maximizes integration and mission  support.
subsections: section_title: The Architecture Products that NASA Has Used for IFMP Are Limited
paragraphs: As previously discussed, the various frameworks used to develop  architecture products consistently provide for describing a given enterprise  in both logical (e.g., business, performance, application, information) and  technical (e.g., hardware, software, data) terms, and for doing so for the  enterprise’s current or “As Is” environment and its target or “To Be”  environment; these frameworks also provide for defining a capital  investment sequencing plan to transition from the “As Is” to the “To Be”  environment. However, the frameworks do not prescribe the degree to  which the component parts should be described to be considered correct,  complete, understandable, and usable—essential attributes of any  architecture. This is because the depth and detail of the descriptive content  depends on what the architecture is to be used for (i.e., its intended  purpose).
NASA’s stated intention is to use an architecture as the basis for  agencywide business transformation and systems modernization, including  IFMP. This purpose necessitates that NASA’s architecture products provide  considerable depth and detail, as well as logical and rational structuring  and internal linkages. More specifically, it means that these architecture  products should contain sufficient scope and detail so that, for example,  (1) duplicative business operations and systems are eliminated;   (2) business operations are standardized and integrated, and supporting  systems are interoperable; (3) use of enterprisewide services is maximized;  and (4) related shared solutions are aligned, like OMB’s e-government  initiatives. Moreover, this scope and detail should be accomplished in a  way that (1) provides flexibility in adapting to changes in the enterprise’s  internal and external environments; (2) facilitates its usefulness and  comprehension by varying perspectives, users, or stakeholders; and   (3) provides for properly sequencing investments to recognize, for  example, the investments’ respective dependencies and relative business  value.
The architecture artifacts that NASA’s chief technology officer provided to  us and represented as those used to date in acquiring and implementing  IFMP do not contain sufficient context (depth and scope of agencywide  operational and technical requirements) to effectively guide and constrain  agencywide business transformation and systems modernization efforts.  More specifically, these artifacts do not satisfy the most basic  characteristics of architecture content, such as clearly distinguishing  between artifacts that represent the “As Is” and the “To Be” environments.  The agency’s chief technology officer agreed that these existing  architecture products do not clearly distinguish between the two  environments. Therefore, for purposes of our analyses, the chief  technology officer told us to treat the architecture products as descriptive  of the “To Be” environment, and to assume that any “As Is” content in these  products represented capability intended for reuse in the future  environment. This characterization is consistent with NASA contractual  documents associated with developing these architecture products. On the  basis of this characterization, we did not assess these artifacts for their “As  Is” content and accepted the chief technology officer’s acknowledgment  that this content was missing. Instead, we focused on the “To Be” content  and the transition plan. To assess the “To Be” architecture products, we  divided them into five architectural components similar to those in OMB’s  architecture reference models: the business, information/data,  services/applications, technical, and performance components; we added  security as a sixth component because of its recognized importance and  relevance to the other five. We then compared architecture products NASA  used to date for IFMP to relevant criteria governing the content of key  architectural elements for the transition plan and these six components of  the “To Be” architecture. Based on this comparison, we determined  whether the architecture products generally satisfied, did not satisfy, or  partially satisfied each architectural element.
Overall, we found that NASA’s “To Be” architecture products did not satisfy  18 of 35 (51 percent) key elements and partially satisfied the remaining 17  (49 percent), and its transition plan partially satisfied 1 (25 percent) of 4  elements and did not satisfy the remaining 3 (75 percent) (see fig. 1).
This means that the architecture products that NASA used to date in  acquiring and implementing IFMP have not provided an adequate context  in which to wisely invest in the program. In general, these products were  limited to descriptions of (1) technology characteristics, which is one of  many enterprise architecture elements, and (2) one of nine business  operations (finance and accounting). Our specific analysis of “To Be” and  transition plan products follows. “To Be” Products: A “To Be” architecture is intended to capture the vision  of future business operations and supporting technology. It should describe  the desired capabilities, structures (e.g., entities, activities, and roles), and  relationships among these structures at a specified point(s) in the future.  The “To Be” architecture should show, for example, future business  processes, information needs, and supporting infrastructure and be fiscally and technologically achievable. According to relevant guidance, the “To  Be” architecture should contain, among other things, a description of   (1) the future business operations that will be performed to support the  organization’s mission, including the entities or people that will perform  the functions, processes, and activities, and the locations where the  functions, processes, and activities will be performed; (2) the logical  database model that is to be used to guide the creation of the physical  databases where information will be stored; (3) the systems to be  developed or acquired to support the business operations; (4) the physical  infrastructure (e.g., hardware and software) that will be needed to support  the business systems; (5) the organizations that will be accountable for  implementing security and the tools to be used to secure and protect  systems and data; and (6) the metrics that will be used to evaluate the  effectiveness of mission operations and supporting system performance in  achieving mission goals and objectives. By including these elements, the  architecture would provide NASA with the necessary frame of reference  for engineering business processes and systems in a manner that supports  agencywide goals and objectives, such as ensuring that decision makers  routinely receive timely, accurate, and reliable information.
The “To Be” architecture products used to date in acquiring and  implementing IFMP provide minimal descriptive content. On the positive  side, they contain a description of one future business operation (i.e.,  finance and accounting). However, they do not describe other future  business operations (e.g., asset management and human resources). In  addition, they do not describe (1) finance and accounting in terms of the  entities or people who will perform the functions, processes, and activities  and the locations where the functions, processes, and activities will be  performed; (2) the logical database model to be used to create the physical  databases; (3) the actual systems to be developed or acquired to support  future business operations; (4) the physical infrastructure (e.g., hardware  and software) that will be needed to support the business systems; (5) the  organizations that will be accountable for implementing security and the  tools to be used to secure and protect systems and data; and (6) the metrics  that will be used to evaluate the effectiveness of mission operations and  supporting system performance in achieving mission goals and objectives.  Without this information, the organization will not have a common vision  and frame of reference for defining a transition plan to guide and constrain  the transformation of business operations and associated capital  investments and, thus, will be unable to effectively leverage technology to  orchestrate logical and systematic change and optimize enterprisewide  mission performance. Detailed results of our analysis are provided in  appendix II.
Transition Plan Products: According to relevant guidance and best  practices, the transition plan should provide a temporal road map for  moving from the “As Is” to the “To Be” environment. An important step in  developing a well-defined transition plan is a gap analysis—comparison of  the “As Is” and “To Be” architectures to identify differences. Other  important steps include analyzing technology opportunities and  marketplace trends, as well as assessing fiscal and budgetary realities and  institutional acquisition and development capabilities. With the use of such  analyses and assessments, options are explored and decisions are made  regarding which legacy systems to retain, modify, or retire and which new  systems to introduce on a tactical (temporary) basis or to pursue as  strategic solutions. Accordingly, transition plans identify legacy, migration,  and new systems and sequence them to show, for example, the phasing out  and termination of systems and capabilities and the timing of the  introduction of new systems and capabilities, and they do so in light of  resource constraints, such as budget, people, acquisition/development  process maturity, and associated time frames.
The transition plan artifacts that NASA relied on in acquiring and  implementing IFMP generally do not possess these attributes. Specifically,  they do not (1) provide a gap analysis identifying the needed changes to  current business processes and systems; (2) identify all of the systems that  will not become part of the “To Be” architecture, as well as the time frames  for phasing out these systems; (3) show a time-based strategy for replacing  legacy systems, including identifying intermediate (i.e., migration) systems  that may be temporarily needed; and (4) define the resources (e.g., funding  and staff) needed to transition to the target environment. The result is that  NASA has not had a meaningful and reliable basis for managing the  disposition of its systems or for sequencing the introduction of modernized  business operations and supporting systems. Detailed results of our  analysis appear in appendix II.
The chief technology officer agreed that the architecture products used to  date to acquire and implement IFMP do not provide sufficient scope and  content to constitute a well-defined enterprise architecture. Based on our  experience in reviewing other agencies, not having an effective enterprise  architecture program is attributable to, among other things, limited senior  management understanding and commitment and cultural resistance to  having and using an architecture.
Our experience with federal agencies has shown that attempting to define  and build major IT systems without first completing an enterprise  architecture often results in IT systems that are duplicative, are not well  integrated, are unnecessarily costly to maintain and interface, and do not  effectively optimize mission performance. In fact, NASA’s OIG recently  reported that the agency would need to resolve several accounting and  costing issues before the IFMP core financial module, which is to  implement NASA’s finance and accounting business process, would be able  to provide full cost-accounting capabilities as envisioned. To accomplish  this, the agency will have to reconfigure the IFMP software to reflect these  changes, resulting in system rework and additional associated costs that  could have been prevented.
Beyond this known rework, additional corrective action could be  necessary to address any misalignment between already implemented  IFMP system components and NASA’s just-released initial version of an  enterprise architecture. Specifically, the chief technology officer provided  us with an initial version of a NASA enterprise architecture on September  24, 2003, which was after we completed our audit work. According to this  official, although this initial version of the architecture is incomplete, it  does provide some of the missing contextual information (operational and  technical) that we had identified during our review. The official also stated  that future versions of the architecture are to be issued quarterly through  June 2004 and semiannually thereafter, and that plans are currently being  developed for assessing IFMP’s alignment with the architecture. The IFMP  deputy program manager affirmed these plans for assessing IFMP’s  alignment. In the likely event that any misalignment is found, NASA will be  faced with additional system rework demands.
section_title: NASA Does Not Have Key Capabilities in Place for Effectively Managing Its Recently Launched Enterprise Architecture Effort
paragraphs: As NASA proceeds with its enterprise architecture effort, it is critical that it  employs rigorous and disciplined management practices in doing so. Such  practices form the basis of our architecture management maturity  framework, which specifies by stages the key architecture management  controls that are embodied in federal guidance and best practices, provides  an explicit benchmark for gauging the effectiveness of architecture  management, and provides a road map for making improvements. Each of  the five stages is described below.  1. Creating enterprise architecture awareness. The organization does  not have plans to develop and use an architecture, or it has plans that  do not demonstrate an awareness of the value of having and using an  architecture. While stage 1 agencies may have initiated some  architecture activity, these agencies’ efforts are ad hoc and  unstructured, lack institutional leadership and direction, and do not  provide the management foundation necessary for successful  architecture development. 2. Building the enterprise architecture management foundation. The  organization recognizes that the architecture is a corporate asset by  vesting accountability for it in an executive body that represents the  entire enterprise. At this stage, an organization assigns architecture  management roles and responsibilities and establishes plans for  developing enterprise architecture products and for measuring  program progress and product quality; it also commits the resources  necessary for developing an architecture—people, processes, and  tools. 3. Developing the enterprise architecture. The organization focuses on  developing architecture products according to the selected framework,  methodology, tool, and established management plans. Roles and  responsibilities assigned in the previous stage are in place, and  resources are being applied to develop actual enterprise architecture  products. The scope of the architecture has been defined to encompass  the entire enterprise, whether organization-based or function-based. 4. Completing the enterprise architecture. The organization has  completed its enterprise architecture products, meaning that the  products have been approved by the architecture steering committee or  an investment review board, and by the CIO. Further, an independent  agent has assessed the quality (i.e., completeness and accuracy) of the  architecture products. Additionally, evolution of the approved products  is governed by a written architecture maintenance policy approved by  the head of the organization. 5. Leveraging the enterprise architecture to manage change. The  organization has secured senior leadership approval of the enterprise  architecture products and a written institutional policy stating that IT  investments must comply with the architecture, unless granted an  explicit compliance waiver. Further, decision makers are using the  architecture to identify and address ongoing and proposed IT  investments that are conflicting, overlapping, not strategically linked,  or redundant. Also, the organization tracks and measures architecture  benefits or return on investment, and adjustments are continuously  made to both the architecture management process and the enterprise  architecture products.
For stage 2, our framework specifies nine key practices or core elements  that are necessary to provide the management foundation for successfully  launching and sustaining an architecture effort. Examples of stage 2 core  elements are described below.
Establish a committee or group representing the enterprise that is  responsible for directing, overseeing, or approving the enterprise  architecture. This committee should include executive-level  representatives from each line of business, and these representatives  should have the authority to commit resources and enforce decisions  within their respective organizational units. By establishing this  enterprisewide responsibility and accountability, the agency  demonstrates its commitment to building the management foundation  and obtaining buy-in from across the organization.
Appoint a chief architect. The chief architect should be responsible and  accountable for the enterprise architecture, supported by the  architecture program office, and overseen by the architecture steering  committee. The chief architect, in collaboration with the CIO, the  architecture steering committee, and the organizational head, is  instrumental in obtaining organizational buy-in for the enterprise  architecture, including support from the business units, as well as in  securing resources to support architecture management functions, such  as risk management, configuration management, quality assurance, and  security management.
Use a framework, methodology, and automated tool to develop the  enterprise architecture. These elements are important because they  provide the means for developing the architecture in a consistent and  efficient manner. The framework provides a formal structure for  representing the enterprise architecture, while the methodology is the  common set of procedures that the enterprise is to follow in developing  the architecture products. The automated tool serves as a repository  where architectural products are captured, stored, and maintained.
Develop an architecture program management plan. This plan  specifies how and when the architecture is to be developed. It includes a  detailed work breakdown structure; resource estimates (e.g., funding,  staffing, and training); performance measures; and management  controls for developing and maintaining the architecture. The plan  demonstrates the organization’s commitment to managing architecture  development and maintenance as a formal program.
Allocate adequate resources. An organization needs to have the  resources (funding, people, tools, and technology) to establish and  effectively manage its architecture. This includes, among other things,  identifying and securing adequate funding to support architecture  activities, hiring and retaining the right people, and selecting and  acquiring the right tools and technology to support activities.
Our framework similarly identifies key architecture management practices  associated with later stages of architecture management maturity. For  example, at stage 3, the stage at which organizations focus on architecture  development activities, organizations need to satisfy six core elements.  Examples of these core elements are discussed below.
Issue a written and approved organization policy for development of  the enterprise architecture. The policy defines the scope of the  architecture, including the requirement for a description of the baseline  and target architecture, as well as an investment road map or  sequencing plan specifying the move between the two. This policy is an  important means for ensuring enterprisewide commitment to  developing an enterprise architecture and for clearly assigning  responsibility for doing so.
Ensure that enterprise architecture products are under configuration  management. This involves ensuring that changes to products are  identified, tracked, monitored, documented, reported, and audited.  Configuration management maintains the integrity and consistency of  products, which is key to enabling effective integration among related  products and for ensuring alignment between architecture artifacts.
At stage 4, during which organizations focus on architecture completion  activities, organizations need to satisfy eight core elements. Examples of  these core elements are described below.
Ensure that enterprise architecture products and management  processes undergo independent verification and validation. This core  element involves having an independent third party—such an as internal  audit function or contractor that is not involved with any of the  architecture development activities—verify and validate that the  products were developed in accordance with architecture processes  and product standards. Doing so provides organizations with needed  assurance of the quality of the architecture.
Ensure that business, performance, information/data,  application/service, and technology descriptions address security. An  organization should explicitly and consistently address security in its  business, performance, information/data, application/service, and  technology architecture products. Because security permeates every  aspect of an organization’s operations, the nature and substance of  institutionalized security requirements, controls, and standards should  be captured in the enterprise architecture products.
At stage 5, during which the focus is on architecture maintenance and  implementation activities, organizations need to satisfy eight core  elements. Examples of these core elements are described below.
Make the enterprise architecture an integral component of the IT  investment management process. Because the road map defines the IT  systems that an organization plans to invest in as it transitions from the  “As Is” to the “To Be” environment, the enterprise architecture is a  critical frame of reference for making IT investment decisions. Using the  architecture when making such decisions is important because  organizations should approve only those investments that move the  organization toward the “To Be” environment, as specified in the road  map.
Measure and report return on enterprise architecture investment. Like  any investment, the enterprise architecture should produce a return on  investment (i.e., a set of benefits), and this return should be measured  and reported in relation to costs. Measuring return on investment is  important to ensure that expected benefits from the architecture are  realized and to share this information with executive decision makers,  who can then take corrective action to address deviations from  expectations.
Table 5 summarizes our framework’s five stages and the associated core  elements for each stage.
The state of NASA’s implementation of key enterprise architecture  management practices, conditions, and structures currently places the  agency at stage 1 of our maturity framework. Specifically, it has satisfied all  but one of the core elements associated with building the architecture  management foundation—stage 2 of our framework—but only about 23  percent (5 of the 22) of the core elements associated with stages 3, 4, and 5.  According to our framework, effective architecture management is  generally not achieved until an enterprise has a completed and approved  architecture that is being effectively maintained and is being used to  leverage organizational change and support investment decision making;  having these characteristics is equivalent to having satisfied all stage 3 core  elements and many stage 4 and 5 elements.
Regarding stage 2 core elements, NASA has, for example, recently  established a program office, assigned a chief architect, and selected a  framework (Zachman) and automated tools (e.g., the Rational Rose by  Rational Software Corporation/IBM Software Group). However, the agency  has not satisfied a stage 2 core element that is critical to effective  architecture management. Specifically, a committee or group representing  the enterprise has not yet been established to guide, direct, or approve the  architecture. Instead, the CIO is guiding the architecture development  effort. Having such a corporate entity is critical to overcoming cultural  resistance to using an enterprise architecture. Without such an entity to  lead and be accountable for the architectural effort, there is increased risk  that the architecture will not represent a corporate decision-making tool  and will not be viewed and endorsed as an agencywide asset.
Concerning stage 3, NASA has not satisfied three of six core elements. For  example, the agency does not have a written and approved policy for  architecture development, which is a stage 3 core element. Without such a  policy that, for example, identifies the major players in the development  process and provides for architecture guidance, direction, and approval,  NASA will be challenged in overcoming cultural resistance to using an  enterprise architecture and achieving agencywide architecture  commitment and support.
The agency also has yet to implement numerous stage 4 and 5 core  elements. For example, NASA has not (1) documented and approved a  policy for architecture maintenance, (2) implemented an independent  verification and validation function that covers architecture products and  architecture management processes, and (3) made the architecture an  integral component of its IT investment management process. (The  detailed results of our assessment of NASA’s satisfaction of each of the  stages and associated core elements are provided in app. III.)
According to the chief technology officer, the agency recognizes the  importance of having rigorous and disciplined architecture management  controls and is in the process of establishing them. Our research of  successful organizations and experience in reviewing other agency  enterprise architecture efforts shows that not having these controls is,  among other things, a function of limited senior management  understanding of and commitment to an enterprise architecture and  cultural resistance to having and using one. Until such barriers are  addressed, and effective architecture management structures and  processes are established, it is unlikely that any agency will be able to  produce and maintain a complete and enforceable architecture or  implement modernized systems in a way that minimizes overlap and  duplication and maximizes integration and mission support.
section_title: Conclusions
paragraphs: NASA’s acquisition and implementation of six major IFMP system  components outside the context of an enterprise architecture was not a  prudent decision. Such a systems modernization approach unnecessarily  increases the risk that system components will not effectively and  efficiently support agencywide operations, which in turn leads to costly  system rework. It is critical for NASA to discontinue this approach and  adopt the best practice of managing its IFMP system investments within  the context of a well-defined enterprise architecture. In order to do so, it is  important for NASA to establish an effective means for developing and  implementing an architecture, which includes gaining top management  understanding and support to lead the way in overcoming any cultural  resistance. It is equally important that the agency ensure that the  architecture contains sufficient depth and scope, quickly determine  whether existing and planned IFMP component systems align with initial  and subsequent versions of the architecture, and limit further investment in  these systems until such determinations are made. To do less risks  introducing additional system rework to that already facing the agency on  already implemented system components.
section_title: Recommendations for Executive Action
paragraphs: To ensure that NASA has the necessary agencywide context within which  to make informed IFMP and other systems modernization decisions, we  recommend that the NASA Administrator demonstrate an institutional  commitment to developing and using an enterprise architecture by  establishing a NASA enterprise architecture policy and designating a NASA  architecture board, or comparable body, that is made up of agency  executives who are responsible and accountable for developing and  maintaining the architecture.
In carrying out its responsibility, we recommend that the Administrator  direct the architecture board, in collaboration with the CIO, to ensure that  the architecture content requirements identified in this report are satisfied  by first determining the extent to which NASA’s initial release of an  enterprise architecture satisfies these content requirements and then  developing and approving a plan for incorporating any content that is  missing.
We further recommend that the Administrator direct the IFMP Program  Executive Officer to appropriately limit acquisition and implementation  activities until the agency ensures that the program’s plans are aligned with  the initial and subsequent versions of the enterprise architecture. In  addition, we recommend that the Administrator direct the architecture  board, in collaboration with the CIO, to immediately map already  implemented IFMP components to the agency’s enterprise architecture and  report to the Program Executive Officer any instances of misalignment, the  associated risks, and proposed corrective actions. Moreover, we  recommend that the Administrator direct the Program Executive Officer to  develop corrective action plans, as appropriate, that include specific  milestones, cost estimates, and detailed actions to be taken to align the  program with the enterprise architecture.
To further assist NASA, we recommend that the Administrator direct the  board, in collaboration with the CIO, to ensure that the best practices  involved in stages 3 through 5 of our enterprise architecture management  maturity framework are implemented. More specifically, we recommend  that the board and the CIO (1) establish a written and approved policy for  architecture development, (2) place enterprise architecture products under  configuration management, and (3) ensure that progress against  architecture plans is measured and reported.
In completing the architecture, we recommend that the board and CIO  (1) establish a written and approved policy for architecture maintenance;  (2) ensure that enterprise architecture products and management  processes undergo independent verification and validation; (3) ensure that  architecture products describe the enterprise’s business and the data,  application, and technology that supports it; (4) ensure that enterprise  architecture products describe the “As Is” environment, the “To Be”  environment, and a sequencing plan; (5) ensure that business,  performance, data, application, and technology descriptions address  security; (6) ensure that the CIO approves the enterprise architecture;  (7) ensure that the steering committee and/or the investment review board  has approved the current version of the enterprise architecture; and  (8) measure and report on the quality of enterprise architecture products.
In implementing the architecture, we recommend that the board and CIO  (1) establish a written and approved policy for IT investment compliance  with the enterprise architecture, (2) ensure that the enterprise architecture  is an integral component of IT investment management processes,  (3) ensure that IT investments comply with the enterprise architecture,  (4) obtain Administrator approval of each enterprise architecture version,  (5) measure and report enterprise architecture return on investment, and  (6) measure and report on enterprise architecture compliance.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report signed by the Deputy  Administrator (reprinted in app. IV), NASA concurred with our  recommendations and described recently completed, ongoing, or planned  efforts to address them. For example, the agency stated that it has  developed a 3-year plan for refining the latest version of its architecture, as  well as a plan to guide the agency in using the architecture to achieve  NASA’s strategic goals. In addition, the agency stated that it has adopted  our architecture management maturity framework and is currently working  to satisfy the framework’s core elements, including establishing  architecture policies and a function for independently verifying and  validating architecture artifacts and management practices. Additionally, it  stated that it plans to continually validate IFMP against its architecture on a  quarterly basis.
NASA also stated that its CIO board, which is chaired by NASA’s CIO and  composed of the CIOs from the agency’s six major lines of business and its  ten field centers, serves as the NASA architecture board or steering  committee. While we support CIO representation on an architecture  steering committee, recognized best practices and our maturity framework  both advocate that architecture ownership and accountability be vested  with an enterprise’s business owners. Thus, we state in our framework that  the architecture steering committee should be composed of executive-level  representatives from each line of business and that these representatives  should have the authority to commit resources and enforce decisions for  their respective organizational units. Without such an entity to lead and be  accountable for the architectural effort, there is increased risk that the  architecture will be viewed solely as an IT tool and not represent a  corporate, business-driven decision-making tool and will not be viewed and  endorsed as an agencywide asset. Accordingly, it is important for NASA to  ensure that its architecture board’s membership includes business owner  representation.
As agreed with your offices, unless you announce its contents earlier, we  will not distribute this report further until 30 days from its date. At that  time, we will send copies to interested congressional committees as well as  to the NASA Administrator and the Director of OMB. We will make copies  available to others upon request. In addition, the report will be available at  no charge on the GAO Web site at http://www.gao.gov.
If you or your staff have any questions concerning this report, please  contact me at (202) 512-3439 or hiter@gao.gov. Key contributors to this  report are acknowledged in appendix V.
section_title: Objective, Scope and Methodology
paragraphs: To determine whether the National Aeronautics and Space Administration  (NASA) had and was using an enterprise architecture to guide and  constrain its investment in its Integrated Financial Management Program  (IFMP), we requested all NASA enterprise architecture artifacts and related  documentation that had been used to date to guide and constrain IFMP  and, based on what we were provided by NASA’s chief technology officer,  compared them to relevant guidance.
In doing so, we first segmented our analysis of artifacts and guidance into  the three primary component parts of any architecture: the “As Is”  architecture, the “To Be,” and the transition plan. We then further divided  the “As Is” and “To Be” architectures into five architectural components  similar to the Office of Management and Budget’s architecture reference  models: business, information/data, services/applications, technical, and  performance. We also added security as a sixth component because of its  recognized importance in the various architecture frameworks and  relevance to the other five architectural components. Because NASA had  not clearly distinguished between its “As Is” and “To Be” environments, the  chief technology officer told us to treat the architecture products provided  as the “To Be” environment and assume that any “As Is” content would be  intended for reuse in the future environment. As a result, we did not  analyze whether NASA’s architecture products satisfied relevant “As Is”  guidance; instead, we accepted the chief technology officer’s  acknowledgment that NASA did not have any “As Is” artifacts.
To augment our documentation reviews and analyses of architecture  products used to date in acquiring and implementing IFMP, we also  interviewed various officials, including the chief information officer and  chief technology officer, to determine, among other things, the agency’s  plans to develop an enterprise architecture. Specifically, we inquired as to  NASA’s basis for selecting already acquired IFMP commercial products and  its plans for selecting future IFMP modules, including whether the agency  had developed an enterprise architecture to guide and constrain its future  investment in IFMP.
We also requested information on ongoing efforts to develop the initial  version of NASA’s enterprise architecture, such as detailed program plans,  updated policies and procedures, and the architecture itself, but this  information was not provided until September 24, 2003, which was after we  had completed our audit work. As a result, our review did not include an  assessment of the initial version of NASA’s enterprise architecture, which  the chief technology officer stated addressed some, but not all, of the  limitations discussed in this report.
To determine whether NASA’s initial and subsequent versions of its  enterprise architecture were supported by effective management  structures and processes, we used our Enterprise Architecture  Management Maturity Framework, which describes the five stages of  management maturity, and determined the extent to which NASA has  adopted key elements of architecture management best practices  embodied in the framework. To make this determination, we reviewed  program documentation, such as program policies and procedures, an IBM  report on the agency’s efforts to implement management processes and  controls over its architecture development activities, and the architecture  products used to date in acquiring IFMP system components, and we  compared them to the elements in our framework. We did not  independently validate the cost and budget information provided by the  chief technology officer.
We conducted our work at NASA headquarters in Washington, D.C. We  performed our work from June to mid-September 2003 in accordance with  generally accepted government auditing standards.
section_title: Detailed Results of GAO’s Analyses of Architecture Products Used to Date by NASA to Acquire and Implement IFMP
paragraphs: Detailed Analysis of NASA’s “To Be” Architecture Products Element satisfied?
The available architecture products contain a high-level  description of the agency’s OneNASA vision, which focuses on  how technology will be managed to improve services (e.g.,  providing secure and highly interoperable information systems  in support of all NASA operations). It lists mission statements  for both the agency and the lines of business. The architecture  also lists business architecture drivers, which can be  considered business goals.
However, the available architecture products do not contain a  description of the strategic goals, objectives, missions, and  implementing strategies established to support NASA lines of  business. In addition, the architecture products do not explain  what the OneNASA vision encompasses since it appears to be  technology-centric, as opposed to business-centric (i.e., it  addresses business, information/data, services/applications,  and technology).
The available architecture products list business strategies,  such as implementing the Integrated Financial Management  Program (IFMP). However, they do not describe how these  strategies will be implemented.
The available architecture products contain a description of the  finance and accounting processes (i.e., the processes to be  supported by the IFMP core financial module). This description  also identifies the subprocesses within these processes and  includes detailed diagrams of process flows.
However, these products do not identify the organizational  units responsible for performing the finance and accounting  business processes nor the locations where they will be  performed. Moreover, the architecture products do not contain  a description of other business processes, such as asset  management, human resource management, and budget.
Element satisfied?
The available architecture products recognize the need for an  implementing strategy to streamline financial operations and  identify IFMP as that strategy. However, the products do not  describe specific opportunities for improving weaknesses in  the “As Is” financial systems or processes or how IFMP will be  implemented to achieve this unification/simplification.
The available architecture products contain a description of the  management reporting lines for the agency’s chief information  officer (CIO) organization as it relates to managing the  architecture products and standards. However, they do not  describe the roles and responsibilities of other organizations.  For example, these products do not have a model for roles and  an organization chart that shows the lines of communication  and reporting responsibilities for financial management  operations.
The available architecture products contain a description of  technical standards currently being used. However, these  products do not state whether these standards are still relevant  or will need to be updated to reflect changes to the current  environment. In addition, the architecture products do not  identify data standards upon which business rules can later be  developed.
X Element satisfied?
The available architecture products contain a description of an  enterprise application system that supports finance and  accounting (IFMP core financial module) functions. They also  identify legacy systems that interface with this application  system.
However, this description is limited to this one system.
Element satisfied?
The available architecture products list several architectural  principles for system development and acquisition (e.g.,  modular design, open system approach) and identify a strategy  (i.e., a hub-spoke configuration) for integrating legacy systems.  They also identify a minimum set of technical standards for  hardware and software. In addition, the architecture products  contain policies and guidance for implementing systems.
However, these products do not describe a common technical  approach. In addition, the products did not state whether the  existing policies and procedures are common, complete, and  sufficient to effectively implement the architecture. They do  recognize the need to revise existing policies and procedures.
The available architecture products recognize the need for a  TRM and contain a generic description of the TRM. These  products also note that technology services needed to support  the application portfolio should be defined and identify several  of these services.
However, according to NASA’s chief technology officer, the  TRM is incomplete and flawed. In addition, the list of  technology services identified is incomplete.
The available architecture products note that these standards  should be identified and documented. They also contain a list  of specific standards.
However, the architecture products do not state whether these  standards are for the current or future environment. In addition,  the architecture products do not identify the technical  standards to be implemented for specific enterprise services,  such as query processing.
Element satisfied?
The available architecture products contain a high-level  description (i.e., diagrams without supporting narrative) of the  IFMP core financial network, OneNASA network backbone,  and NASA’s Information System Services Utility (NISSU)  network architecture.
However, these networks do not encompass the entire  enterprise, but rather a subset of activities.
The available architecture products contain a list of policies  and procedures for implementing systems. However, the  products do not state whether these policies and procedures  are common, complete, and sufficient to effectively implement  the architecture.
The available architecture products contain a high-level  description of security goals and strategies and identify some  security requirements. They also note that an “Information  Assurance Trust Model” is needed and will be developed.
The architecture products do not contain a description of  security policies and procedures. They also do not identify  important security requirements (e.g., availability and access  control), nor do they link identified security requirements to  security services. Moreover, the architecture products do not  define the “Information Assurance Trust Model” or address  plans for its completion. Finally, regarding the strategies, they  do not identify and summarize the agency’s most significant  security risks.
According to NASA’s chief technology officer, a clear computer  security policy does not exist within the agency, and there is a  lack of understanding as to how such a policy could be  integrated into the network infrastructure.
The available architecture products contain definitions for some  security-related terms (e.g., authentication, confidentiality, and  intrusion detection); however, they do not define other key  terms listed (e.g., integrity, physical security, and encryption  services). In addition, the definitions for these terms are  inconsistent with the definitions shown in current standards  (e.g., firewalls).
Element satisfied?
The available architecture products contain high-level  descriptions of enterprise security services; however, in most  instances, these products describe the technology components  that will be implemented to provide the security service, and  not the security service. For example, the architecture products  classify “Audit Logs” as a security service; however, audit logs  are generally the function/component within an “Auditing  Service.”
The architecture products also do not link the security services  to security policies, procedures, goals, strategies, and  requirements.
The available architecture products contain a description of  various security standards, but it is unclear if these standards  are relevant to the “As Is” or “To Be” environment or both.
In addition, the architecture products do not contain a  traceability matrix that links goals, strategies, requirements,  and services to the security standards and security products  (e.g., SmartCard). They also do not clearly state whether the  list of security standards for enterprise services is complete.
Element satisfied?
The available architecture products contain a high-level  description of protection mechanisms (e.g., firewalls). However,  they do not describe the level of protection needed and the  types of services the protection mechanisms will provide to  protect IFMP applications that access information/data,  business services/applications, and the various networks.
In addition, the architecture products do not contain a  traceability matrix that links goals, strategies, requirements,  and services to the security standards, so it is unclear whether  this is the definitive list of protection mechanisms.
A CRUD (create, read, update, and/or delete) matrix shows the specific business functions and  applications that create, read, update, and/or delete specific data elements, which enables the  organization to develop applications.
Business and operational rules define specific constraints for the data, such as security needs (e.g.,  confidentiality and accessibility of data) and actions that should or should not occur, such as updating  or deleting data.
Although the framework that NASA is using identifies a logical database model as a work product, the  available architecture products do not include such a model, and there was contradictory evidence in  the architecture products that stated that NASA considered this model to be nonessential. As a result,  it is unclear whether the agency plans to produce a logical database model as part of its architecture  description.
Metadata is “data about data” that enables automation and consistent management and use of  information, such as rules and standards.
The framework that NASA is using does not identify the metadata model as a type of work product nor  does the agency’s action plan address the development of a metadata model for later inclusion in the  architecture description.
While the framework that NASA is using does identify the application portfolio as a type of work  product, the agency’s action plan does not address the development of this portfolio for later inclusion  in the architecture description.
The technical reference model (TRM) describes how technology is supporting the delivery of service  components, including relevant standards for implementing the technology. The TRM is a generally  accepted representation of the generic components of an information system. It allows designers,  developers, and users to agree on definitions, have a common understanding of the services to be  provided, and identify and resolve issues affecting such requirements as interoperability, portability,  reliability, scalability, and serviceability.
Examples of enterprise services include application services, such as Web services, and collaboration  services, such as instant messaging and video conferencing.
Technical standards are strict rules and protocols governing how a given enterprise service is to be  implemented.
subsections: section_title: Detailed Analysis of NASA’s Transition Plan Artifacts
paragraphs: Element satisfied?
The transition plan identifies only the core financial legacy  systems that have been or will be retired. It does not identify all  legacy systems or provide a schedule for terminating these  systems.
The transition plan provides a high-level description of a  training strategy and references a change management  strategy for IFMP. It also identifies a business driver for  improving human capital management within the organization.
However, the architecture does not (1) address training needs  for nonfinancial business operations, (2) contain training plans,  (3) identify changes to existing policies and procedures, or  (4) estimate resource needs. Moreover, this generic strategy  was developed without the benefit of a gap analysis.
The transition plan notes that there is a list of project  development initiatives, such as core financial and travel  manager, but it does not provide a complete list of systems to  be developed or acquired to achieve business needs (e.g.,  human resources and budget).
The transition plan contains a description of an EAI strategy for  IFMP applications. However, the transition plan does not state  whether this strategy would be applied to other agencywide  application systems.
Element satisfied?
Acquisition/business strategy is a plan or action for achieving a specific goal or result through  contracting for software products and services.
section_title: Assessment of NASA’s EA Management Efforts against GAO’s Architecture Management Maturity Framework
paragraphs: Satisfied?
Stage 1: EA awareness Agency is aware of EA.
In December 2002, the CIO issued a  memorandum stating the agency’s intent to  develop and use an EA.
Adequate resources exist (funding,  people, tools, and technology).
According to the chief technology officer (CTO),  the agency has adequate program funding. NASA  estimates that it will cost $750,000 to develop its  EA for fiscal years 2001 through 2003. Further, the  agency reports that it has skilled staff (government  employees and contractors) for its architecture  program. In addition, NASA is using automated  tools and technology, such as Rational Rose by  Rational Software Corporation/IBM Software  Group.
Committee or group representing the  enterprise is responsible for directing,  overseeing, or approving the EA.
NASA has not assigned responsibility for directing,  overseeing, or approving the EA to a committee or  group comprising representatives from across the  agency.
Program office responsible for EA  development and maintenance exists.
In December 2002, NASA established a program  office that is responsible for EA development and  maintenance.
Chief architect exists.
In January 2003, NASA designated the CTO as the  chief architect.
EA is being developed using a  framework, methodology, and automated  tool.
The EA is being developed using the Zachman  framework. According to the CTO, the agency is  also using a defined methodology to develop the  EA. In addition, NASA is using automated tools,  such as Rational Rose by Rational Software  Corporation/IBM Software Group, to build the EA.
According to the CTO, the plans for the EA  “As Is” and the “To Be” environments of  the enterprise, as well as a sequencing  plan for transitioning from the “As Is” to  the “To Be.” provide for describing both the “As Is” and the   “To Be” environments and a sequencing plan.
According to the CTO, the plans for the EA  “As Is” and the “To Be” environments in  terms of business, performance,  information/data, application/service, and  technology. provide for describing both the “As Is” and the   “To Be” environments in terms of business,  performance, information/data, application/service,  and technology.
EA plans call for business, performance,  information/data, application/service, and  technology descriptions to address  security.
According to the CTO, the plans for the EA  provide for addressing security for the “As Is” and  “To Be” environments.
EA plans call for developing metrics for  measuring EA progress, quality,  compliance, and return on investment.
According to the CTO, the plans for the EA  provide for developing metrics to measure  progress, quality, compliance, and return on  investment.
Satisfied?
Written/approved organization policy  exists for EA development.
According to the CTO, the agency is revising its  existing policy to require the development of an  EA. As written, the policy requires the CIO to  develop an information technology (IT)  architecture, which is one aspect of an EA.
EA products are under configuration  management.
According to the CTO, EA products are not  currently under configuration management.
EA products describe or will describe  both the “As Is” and the “To Be”  environments of the enterprise, as well as  a sequencing plan for transitioning from  the “As Is” to the “To Be.”
According to the CTO, the plans for the EA  products provide for describing the “As Is” and the  “To Be” environments, as well as a sequencing  plan.
Both the “As Is” and the “To Be”  environments are described or will be  described in terms of business,  performance, information/data,  application/service, and technology.
According to the CTO, the plans for the EA  provide for describing both the “As Is” and “To Be”  environments in terms of business, performance,  information/data, application/service, and  technology.
Business, performance, information/data,  application/service, and technology  descriptions address or will address  security.
According to the CTO, the plans for the EA  provide for the business, performance,  information/data, application/service, and  technology descriptions addressing security for the  “As Is” and “To Be” environments.
Progress against EA plans is measured  and reported.
According to the CTO, the agency is measuring  and reporting progress against EA plans; however,  NASA was unable to provide evidence of these  reports.
Written/approved organization policy  exists for EA maintenance.
There is no written/approved policy for EA  maintenance.
EA products and management processes  undergo independent verification and  validation.
According to the CTO, management processes are  independently verified and validated, but EA  products do not undergo independent verification  and validation. According to the CTO, EA products  will be subject to independent verification and  validation in the future.
According to the CTO, the plans for the EA  provide for describing both the “As Is” and the   “As Is” to the “To Be.” “To Be” environments of the enterprise, as well as  a sequencing plan for transitioning from the “As Is”  to the “To Be.” However, the initial version of  NASA’s EA was not provided to us in time to  determine if its products address this core element.  Therefore, our analysis is based on the products  that were used to date to guide and constrain  IFMP.
Satisfied?
Both the “As Is” and the   “To Be” environments are described in  terms of business, performance,  information/data, application/service, and  technology.
According to the CTO, the plans for the EA  provide for describing both the “As Is” and “To Be”  environments in terms of business, performance,  information/data, application/service, and  technology. However, the initial version of NASA’s  EA was not provided to us in time to determine if its  products address this core element. Therefore, our  analysis is based on the products that were used  to date to guide and constrain IFMP.
Business, performance, information/data,  application/service, and technology  descriptions address security.
According to the CTO, the plans for the EA  provide for the business, performance,  information/data, application/service, and  technology descriptions addressing security.  However, the initial version of NASA’s EA was not  provided to us in time to determine if its products  address this core element. Therefore, our analysis  is based on the products that were used to date to  guide and constrain IFMP.
Organization CIO has approved current  version of EA.
The CIO approved the initial version of the EA.  However, the initial version of NASA’s EA was not  provided to us in time to determine if its products  address this core element. Therefore, our analysis  is based on the products that were used to date to  guide and constrain IFMP.
Committee or group representing the  enterprise or the investment review board  has approved current version of EA.
According to the CTO, the plans for the EA do not  provide for approval by a committee or group  representing the enterprise or the investment  review board.
Quality of EA products is measured and  reported.
According to the CTO, the quality of EA products is  not measured and reported.
Written/approved organization policy  exists for IT investment compliance with  EA.
There is no written/approved policy requiring IT  investment compliance with the EA. The current  policy requires the CIO to ensure that new IT  investments are in alignment with technology  architectures, which are one aspect of an EA.  According to the CTO, this policy is being revised.
Process exists to formally manage EA  change.
According to the CTO, there is a process for  formally managing EA change.
EA is integral component of IT investment  management process.
Since the EA is currently being developed and has  not been used to date in acquiring and  implementing IFMP, it is not part of the investment  management process.
EA products are periodically updated.
According to NASA, it plans to update the EA  quarterly through June 2004, and semiannually  thereafter.
Satisfied?
IT investments comply with EA.
Since the EA is currently being developed and has  not been used to date in acquiring and  implementing IFMP, it is not part of the investment  management process.
Organization head has approved current  version of EA.
The organization head has not approved the  current version of the EA.
Return on EA investment is measured  and reported.
Metrics and processes for measuring EA benefits  have not been developed, and an initial version of  the EA has not been completed, thus precluding  return-on-investment measurement.
Compliance with EA is measured and  reported.
Metrics for measuring EA compliance have not  been developed, and an initial version of the EA  has not been completed, thus precluding  measuring and reporting on compliance.
section_title: Comments from the National Aeronautics and Space Administration
section_title: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Acknowledgments
paragraphs: Staff making key contributions to this report were Sophia Harrison, Anh Le,  Randolph Tekeley, William Wadsworth, and Lillian Whren.
section_title: GAO’s Mission
paragraphs: The General Accounting Office, the audit, evaluation and investigative arm of  Congress, exists to support Congress in meeting its constitutional responsibilities  and to help improve the performance and accountability of the federal government  for the American people. GAO examines the use of public funds; evaluates federal  programs and policies; and provides analyses, recommendations, and other  assistance to help Congress make informed oversight, policy, and funding  decisions. GAO’s commitment to good government is reflected in its core values of  accountability, integrity, and reliability.
section_title: Obtaining Copies of GAO Reports and Testimony
paragraphs: The fastest and easiest way to obtain copies of GAO documents at no cost is  through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full- text files of current reports and testimony and an expanding archive of older  products. The Web site features a search engine to help you locate documents  using key words and phrases. You can print these documents in their entirety,  including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and  correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site  daily. The list contains links to the full-text document files. To have GAO e-mail this  list to you every afternoon, go to www.gao.gov and select “Subscribe to   e-mail alerts” under the “Order GAO Products” heading.
subsections: section_title: Order by Mail or Phone
section_title: To Report Fraud, Waste, and Abuse in Federal Programs
section_title: Public Affairs