Enterprise Architecture: Leadership Remains Key to Establishing and Leveraging Architectures for Organizational Transformation
Published: 2006-08-14T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: A well-defined enterprise architecture is an essential tool for leveraging information technology (IT) to transform business and mission operations. GAO's experience has shown that attempting to modernize and evolve IT environments without an architecture to guide and constrain investments results in operations and systems that are duplicative, not well integrated, costly to maintain, and ineffective in supporting mission goals. In light of the importance of enterprise architectures, GAO developed a five stage architecture management maturity framework that defines what needs to be done to effectively manage an architecture program. Under GAO's framework, a fully mature architecture program is one that satisfies all elements of all stages of the framework. As agreed, GAO's objective was to determine the status of major federal department and agency enterprise architecture efforts.
section_title: What GAO Found
paragraphs: The state of the enterprise architecture programs at the 27 major federal departments and agencies is mixed, with several having very immature programs, several having more mature programs, and most being somewhere in between. Collectively, the majority of these architecture efforts can be viewed as a work-in-progress with much remaining to be accomplished before the federal government as a whole fully realizes their transformational value. More specifically, seven architecture programs have advanced beyond the initial stage of the GAO framework, meaning that they have fully satisfied all core elements associated with the framework's second stage (establishing the management foundation for developing, using, and maintaining the architecture). Of these seven, three have also fully satisfied all the core elements associated with the third stage (developing the architecture). None have fully satisfied all of the core elements associated with the fourth (completing the architecture) and fifth (leveraging the architecture for organizational change) stages. Nevertheless, most have fully satisfied a number of the core elements across the stages higher than the stage in which they have met all core elements, with all 27 collectively satisfying about 80, 78, 61, and 52 percent of the stage two through five core elements, respectively. Further, most have partially satisfied additional elements across all the stages, and seven need to fully satisfy five or fewer elements to achieve the fifth stage. The key to these departments and agencies building upon their current status, and ultimately realizing the benefits that they cited architectures providing, is sustained executive leadership, as virtually all the challenges that they reported can be addressed by such leadership. Examples of the challenges are organizational parochialism and cultural resistance, adequate resources (human capital and funding), and top management understanding; examples of benefits cited are better information sharing, consolidation, improved productivity, and reduced costs.

section_title: Letter
paragraphs: A well-defined enterprise architecture is an essential tool for leveraging  information technology (IT) in the transformation of business and mission  operations. Our experience with federal departments and agencies has  shown that attempting to modernize and evolve IT environments without  an enterprise architecture to guide and constrain investments often results  in operations and systems that are duplicative, not well integrated,  unnecessarily costly to maintain and interface, and ineffective in  supporting mission goals. Moreover, the development, implementation, and  maintenance of architectures are widely recognized as hallmarks of  successful public and private organizations, and their use is required by the  Clinger-Cohen Act and the Office of Management and Budget (OMB). In  light of the importance of these architectures, you requested that we  determine the current status of major federal department and agency  enterprise architecture efforts.
To accomplish our objective, we surveyed 27 major federal departments  and agencies using a questionnaire that was based on our maturity  framework for assessing and improving enterprise architecture  management, and we collected and reviewed documentation to verify  agency responses. We then analyzed the results to determine the extent to  which each of the 27 satisfied our maturity framework, and the challenges  and benefits that each department and agency sees. We also collected  information about, for example, department and agency architecture costs  and architecture framework and tool use and satisfaction, which is  summarized in appendixes I and II. Because our framework defines what  needs to be done to effectively manage an enterprise architecture program,  and not the details surrounding how it needs to be done, the scope of our  review did not include assessing the quality of enterprise architecture  products and activities and associated management structures and  processes that make up our framework. As such, scoring high on our  maturity scale should be viewed as an indicator of, and not a guarantee  that, a department or agency necessarily has a well-defined architecture  and that it is being effectively implemented. We conducted our work in  accordance with generally accepted government auditing standards.  Details of our objective, scope, and methodology are in appendix III.
subsections: section_title: Background
paragraphs: An enterprise architecture is a blueprint that describes the current and  desired state of an organization or functional area in both logical and  technical terms, as well as a plan for transitioning between the two states.  Enterprise architectures are a recognized tenet of organizational  transformation and IT management in public and private organizations.  Without an enterprise architecture, it is unlikely that an organization will be  able to transform business processes and modernize supporting systems to  minimize overlap and maximize interoperability. The concept of enterprise  architectures originated in the mid-1980s; various frameworks for defining  the content of these architectures have been published by government  agencies and OMB. Moreover, legislation and federal guidance requires  agencies to develop and use architectures. For more than a decade, we  have conducted work to improve agency architecture efforts. To this end,  we developed an enterprise architecture management maturity framework  that provides federal agencies with a common benchmarking tool for  assessing the management of their enterprise architecture efforts and  developing improvement plans.
subsections: section_title: Enterprise Architecture Description and Importance
paragraphs: An enterprise can be viewed as either a single organization or a functional  area that transcends more than one organization (e.g., financial  management, homeland security). An architecture can be viewed as the  structure (or structural description) of any activity. Thus, enterprise  architectures are basically systematically derived and captured  descriptions—in useful models, diagrams, and narrative.
More specifically, an architecture describes the enterprise in logical terms  (such as interrelated business processes and business rules, information  needs and flows, and work locations and users) as well as in technical  terms (such as hardware, software, data, communications, and security  attributes and performance standards). It provides these perspectives both  for the enterprise’s current or “as-is” environment and for its target or “to- be” environment, as well as a transition plan for moving from the “as-is” to  the “to-be” environment.
The importance of enterprise architectures is a basic tenet of both  organizational transformation and IT management, and their effective use  is a recognized hallmark of successful public and private organizations. For  over a decade, we have promoted the use of architectures, recognizing  them as a crucial means to a challenging end: optimized agency operations  and performance. The alternative, as our work has shown, is the  perpetuation of the kinds of operational environments that burden most  agencies today, where a lack of integration among business operations and  the IT resources supporting them leads to systems that are duplicative,  poorly integrated, and unnecessarily costly to maintain and interface.  Employed in concert with other important IT management controls (such  as portfolio-based capital planning and investment control practices),  architectures can greatly increase the chances that the organizations’  operational and IT environments will be configured so as to optimize  mission performance.
section_title: Brief History of Architecture Frameworks and Management Guidance
paragraphs: During the mid-1980s, John Zachman, widely recognized as a leader in the  field of enterprise architecture, identified the need to use a logical  construction blueprint (i.e., an architecture) for defining and controlling  the integration of systems and their components. Accordingly, Zachman  developed a structure or framework for defining and capturing an  architecture, which provides for six perspectives or “windows” from which  to view the enterprise. Zachman also proposed six abstractions or models  associated with each of these perspectives. Zachman’s framework  provides a way to identify and describe an entity’s existing and planned  component parts and the parts’ relationships before the entity begins the  costly and time-consuming efforts associated with developing or  transforming itself.
Since Zachman introduced his framework, a number of frameworks have  emerged within the federal government, beginning with the publication of  the National Institute of Standards and Technology (NIST) framework in  1989. Since that time, other federal entities have issued frameworks,  including the Department of Defense (DOD) and the Department of the  Treasury. In September 1999, the federal Chief Information Officers (CIO)  Council published the Federal Enterprise Architecture Framework  (FEAF), which was intended to provide federal agencies with a common  construct for their architectures, thereby facilitating the coordination of  common business processes, technology insertion, information flows, and  system investments among federal agencies. The FEAF described an  approach, including models and definitions, for developing and  documenting architecture descriptions for multi-organizational functional  segments of the federal government.
More recently, OMB established the Federal Enterprise Architecture  Program Management Office (FEAPMO) to develop a federal enterprise  architecture according to a collection of five reference models (see table  1). These models are intended to facilitate governmentwide improvement  through cross-agency analysis and the identification of duplicative  investments, gaps, and opportunities for collaboration, interoperability,  and integration within and across government agencies.
OMB has identified multiple purposes for the Federal Enterprise  Architecture, such as the following:   informing agency enterprise architectures and facilitating their  development by providing a common classification structure and  vocabulary;  providing a governmentwide framework that can increase agency  awareness of IT capabilities that other agencies have or plan to acquire,  so that they can explore opportunities for reuse;   helping OMB decision makers identify opportunities for collaboration  among agencies through the implementation of common, reusable, and  interoperable solutions; and  providing the Congress with information that it can use as it considers  the authorization and appropriation of funding for federal programs.
Although these post-Zachman frameworks differ in their nomenclatures  and modeling approaches, each consistently provides for defining an  enterprise’s operations in both logical and technical terms, provides for  defining these perspectives for the enterprise’s current and target  environments, and calls for a transition plan between the two.
Several laws and regulations address enterprise architecture. For example,  the Clinger-Cohen Act of 1996 directs the CIOs of major departments and  agencies to develop, maintain, and facilitate the implementation of  information technology architectures as a means of integrating agency  goals and business processes with information technology. Also, OMB  Circular A-130, which implements the Clinger-Cohen Act, requires that  agencies document and submit their initial enterprise architectures to OMB  and that agencies submit updates when significant changes to their  enterprise architectures occur. The circular also directs OMB to use various  reviews to evaluate the adequacy and efficiency of each agency’s  compliance with the circular.
section_title: A Decade of GAO Work Has Focused on Improving Agency Enterprise Architecture Efforts
paragraphs: We began reviewing federal agencies’ use of enterprise architectures in  1994, initially focusing on those agencies that were pursuing major systems  modernization programs that were high risk. These included the National  Weather Service systems modernization, the Federal Aviation  Administration (FAA) air traffic control modernization, and the Internal  Revenue Service tax systems modernization. Generally, we reported that  these agencies’ enterprise architectures were incomplete, and we made  recommendations that they develop and implement complete enterprise  architectures to guide their modernization efforts.
Since then, we have reviewed enterprise architecture management at other  federal agencies, including the Department of Education (Education), the  Customs Service, the Immigration and Naturalization Service, the  Centers for Medicare and Medicaid Services, FAA, and the Federal  Bureau of Investigation (FBI). We have also reviewed the use of  enterprise architectures for critical agency functional areas, such as the  integration and sharing of terrorist watch lists across key federal  departments and DOD financial management, logistics management,  combat identification, and business systems modernization. These  reviews continued to identify the absence of complete and enforced  enterprise architectures, which in turn has led to agency business  operations, systems, and data that are duplicative, incompatible, and not  integrated; these conditions have either prevented agencies from sharing  data or forced them to depend on expensive, custom-developed system  interfaces to do so. Accordingly, we made recommendations to improve the  respective architecture efforts. In some cases progress has been made,  such as at DOD and FBI. As a practical matter, however, considerable time  is needed to completely address the kind of substantive issues that we have  raised and to make progress in establishing more mature architecture  programs.
In 2002 and 2003, we also published reports on the status of enterprise  architectures governmentwide. The first report (February 2002) showed  that about 52 percent of federal agencies self-reported having at least the  management foundation that is needed to successfully develop, implement,  and maintain an enterprise architecture, and that about 48 percent of  agencies had not yet advanced to that basic stage of maturity. We attributed  this state of architecture management to four management challenges: (1)  overcoming limited executive understanding, (2) inadequate funding, (3)  insufficient number of skilled staff, and (4) organizational parochialism.  Additionally, we recognized OMB’s efforts to promote and oversee  agencies’ enterprise architecture efforts. Nevertheless, we determined that  OMB’s leadership and oversight could be improved by, for example, using a  more structured means of measuring agencies’ progress and by addressing  the above management challenges.
The second report (November 2003) showed the percentage of agencies  that had established at least a foundation for enterprise architecture  management was virtually unchanged. We attributed this to long-standing  enterprise architecture challenges that had yet to be addressed. In  particular, more agencies reported lack of agency executive understanding  of enterprise architecture and the scarcity of skilled architecture staff as  significant challenges. OMB generally agreed with our findings and the  need for additional agency assessments. Further, it stated that fully  implementing our recommendations would require sustained management  attention, and that it had begun by working with the CIO Council to  establish the Chief Architect Forum and to increase the information OMB  reports on enterprise architecture to Congress.
Since then, OMB has developed and implemented an enterprise  architecture assessment tool. According to OMB, the tool helps better  understand the current state of an agency’s architecture and assists  agencies in integrating architectures into their decision-making processes.  The latest version of the assessment tool (2.0) was released in December  2005 and includes three capability areas: (1) completion, (2) use, and (3)  results. Table 2 describes each of these areas.
The tool also includes criteria for scoring an agency’s architecture program  on a scale of 0 to 5. In early 2006, the major departments and agencies  were required by OMB to self assess their architecture programs using the  tool. OMB then used the self assessment to develop its own assessment.  These assessment results are to be used in determining the agency’s e- Government score within the President’s Management Agenda.
section_title: GAO’s Enterprise Architecture Management Maturity Framework (EAMMF)
paragraphs: In 2002, we developed version 1.0 of our Enterprise Architecture  Management Maturity Framework (EAMMF) to provide federal agencies  with a common benchmarking tool for planning and measuring their efforts  to improve enterprise architecture management, as well as to provide OMB  with a means for doing the same governmentwide. We issued an update of  the framework (version 1.1) in 2003. This framework is an extension of A  Practical Guide to Federal Enterprise Architecture, Version 1.0,  published by the CIO Council. Version 1.1 of the framework arranges 31  core elements (practices or conditions that are needed for effective  enterprise architecture management) into a matrix of five hierarchical  maturity stages and four critical success attributes that apply to each stage.  Within a given stage, each critical success attribute includes between one  and four core elements. Based on the implicit dependencies among the  core elements, the EAMMF associates each element with one of five  maturity stages (see fig. 1). The core elements can be further categorized  by four groups: architecture governance, content, use, and measurement.
subsections: section_title: EAMMF Stages
paragraphs: Stage 1: Creating EA awareness. At stage 1, either an organization does  not have plans to develop and use an architecture, or it has plans that do  not demonstrate an awareness of the value of having and using an  architecture. While stage 1 agencies may have initiated some enterprise  architecture activity, these agencies’ efforts are ad hoc and unstructured,  lack institutional leadership and direction, and do not provide the  management foundation necessary for successful enterprise architecture  development as defined in stage 2.
Stage 2: Building the EA management foundation. An organization at  stage 2 recognizes that the enterprise architecture is a corporate asset by  vesting accountability for it in an executive body that represents the entire  enterprise. At this stage, an organization assigns enterprise architecture  management roles and responsibilities and establishes plans for developing  enterprise architecture products and for measuring program progress and  product quality; it also commits the resources necessary for developing an  architecture—people, processes, and tools. Specifically, a stage 2  organization has designated a chief architect and established and staffed a  program office responsible for enterprise architecture development and  maintenance. Further, it has established a committee or group that has  responsibility for enterprise architecture governance (i.e., directing,  overseeing, and approving architecture development and maintenance).  This committee or group membership has enterprisewide representation.  At stage 2, the organization either has plans for developing or has started  developing at least some enterprise architecture products, and it has  developed an enterprisewide awareness of the value of enterprise  architecture and its intended use in managing its IT investments. The  organization has also selected a framework and a methodology that will be  the basis for developing the enterprise architecture products and has  selected a tool for automating these activities.
Stage 3: Developing the EA. An organization at stage 3 focuses on  developing architecture products according to the selected framework,  methodology, tool, and established management plans. Roles and  responsibilities assigned in the previous stage are in place, and resources  are being applied to develop actual enterprise architecture products. At  this stage, the scope of the architecture has been defined to encompass the  entire enterprise, whether organization-based or function-based. Although  the products may not be complete, they are intended to describe the  organization in terms of business, performance, information/data,  service/application, and technology (including security explicitly in each)  as provided for in the framework, methodology, tool, and management  plans. Further, the products are to describe the current (as-is) and future  (to-be) states and the plan for transitioning from the current to the future  state (the sequencing plan). As the products are developed and evolve, they  are subject to configuration management. Further, through the established  enterprise architecture management foundation, the organization is  tracking and measuring its progress against plans, identifying and  addressing variances, as appropriate, and then reporting on its progress.
Stage 4: Completing the EA. An organization at stage 4 has completed  its enterprise architecture products, meaning that the products have been  approved by the enterprise architecture steering committee (established in  stage 2) or an investment review board, and by the CIO. The completed  products collectively describe the enterprise in terms of business,  performance, information/data, service/application, and technology for  both its current and future operating states, and the products include a plan  for transitioning from the current to the future state. Further, an  independent agent has assessed the quality (i.e., completeness and  accuracy) of the enterprise architecture products. Additionally, evolution  of the approved products is governed by a written enterprise architecture  maintenance policy approved by the head of the organization.
Stage 5: Leveraging the EA to manage change. An organization at  stage 5 has secured senior leadership approval of the enterprise  architecture products and a written institutional policy stating that IT  investments must comply with the architecture, unless granted an explicit  compliance waiver. Further, decision makers are using the architecture to  identify and address ongoing and proposed IT investments that are  conflicting, overlapping, not strategically linked, or redundant. As a result,  stage 5 entities avoid unwarranted overlap across investments and ensure  maximum systems interoperability, which in turn ensures the selection and  funding of IT investments with manageable risks and returns. Also, at stage  5, the organization tracks and measures enterprise architecture benefits or  return on investment, and adjustments are continuously made to both the  enterprise architecture management process and the enterprise  architecture products.
section_title: EAMMF Attributes
paragraphs: Attribute 1: Demonstrates commitment. Because the enterprise  architecture is a corporate asset for systematically managing institutional  change, the support and sponsorship of the head of the enterprise are  essential to the success of the architecture effort. An approved enterprise  policy statement provides such support and sponsorship, promoting  institutional buy-in and encouraging resource commitment from  participating components. Equally important in demonstrating  commitment is vesting ownership of the architecture with an executive  body that collectively owns the enterprise.
Attribute 2: Provides capability to meet commitment. The success of  the enterprise architecture effort depends largely on the organization’s  capacity to develop, maintain, and implement the enterprise architecture.  Consistent with any large IT project, these capabilities include providing  adequate resources (i.e., people, processes, and technology), defining clear  roles and responsibilities, and defining and implementing organizational  structures and process management controls that promote accountability  and effective project execution.
Attribute 3: Demonstrates satisfaction of commitment. Satisfaction  of the organization’s commitment to develop, maintain, and implement an  enterprise architecture is demonstrated by the production of artifacts (e.g.,  the plans and products). Such artifacts demonstrate follow through—that  is, actual enterprise architecture production. Satisfaction of commitment is  further demonstrated by senior leadership approval of enterprise  architecture documents and artifacts; such approval communicates  institutional endorsement and ownership of the architecture and the  change that it is intended to drive.
Attribute 4: Verifies satisfaction of commitment. This attribute  focuses on measuring and disclosing the extent to which efforts to develop,  maintain, and implement the enterprise architecture have fulfilled stated  goals or commitments of the enterprise architecture. Measuring such  performance allows for tracking progress that has been made toward  stated goals, allows appropriate actions to be taken when performance  deviates significantly from goals, and creates incentives to influence both  institutional and individual behaviors.
section_title: EAMMF Groups
paragraphs: The framework’s 31 core elements can also be placed in one of four groups  of architecture related activities, processes, products, events, and  structures. The groups are architecture governance, content, use, and  measurement. These groups are generally consistent with the capability  area descriptions in the previously discussed OMB enterprise architecture  assessment tool. For example, OMB’s completion capability area addresses  ensuring that architecture products describe the agency in terms of  processes, services, data, technology, and performance and that the agency  has developed a transition strategy. Similarly, our content group includes  developing and completing these same enterprise architecture products. In  addition, OMB’s results capability area addresses performance  measurement as does our measurement group, and OMB’s use capability  area addresses many of the same elements in our governance and use  groups.
Table 3 lists the core elements according to EAMMF group.
section_title: Overall State of Enterprise Architecture Management Is a Work- in-Progress, Although a Few Agencies Have Largely Satisfied Our Framework
paragraphs: Most of the 27 major departments and agencies have not fully satisfied all  the core elements associated with stage 2 of our maturity framework. At  the same time, however, most have satisfied a number of core elements at  stages 3, 4, and 5. Specifically, although only seven have fully satisfied all  the stage 2 elements, the 27 have on average fully satisfied 80, 78, 61, and 52  percent of the stage 2, 3, 4, and 5 elements, respectively. Of the core  elements that have been fully satisfied, 77 percent of those related to  architecture governance have been fully satisfied, while 68, 52, and 47  percent of those related to architecture content, use, and measurement,  respectively, have been fully satisfied. Most of the 27 have also at least  partially satisfied a number of additional core elements across all the  stages. For example, all but 7 have at least partially satisfied all the  elements required to achieve stage 3 or higher. Collectively, this means  efforts are underway to mature the management of most agency enterprise  architecture programs, but overall these efforts are uneven and still a work- in-progress and they face numerous challenges that departments and  agencies identified. It also means that some architecture programs provide  examples from which less mature programs could learn and improve.
Without mature enterprise architecture programs, some departments and  agencies will not realize the many benefits that they attributed to  architectures, and they are at risk of investing in IT assets that are  duplicative, not well-integrated, and do not optimally support mission  operations.
subsections: section_title: The Degree to which Major Departments and Agencies Have Fully Satisfied Our Framework’s Core Elements Is Uneven and Their Collective Efforts Can Be Viewed as a Work-in- Progress
paragraphs: To qualify for a given stage of maturity under our architecture management  framework, a department or agency had to fully satisfy all of the core  elements at that stage. Using this criterion, three departments and agencies  are at stage 2, meaning that they demonstrated to us through verifiable  documentation that they have established the foundational commitments  and capabilities needed to manage the development of an architecture. In  addition, four are at stage 3, meaning that they similarly demonstrated that  their architecture development efforts reflect employment of the basic  control measures in our framework. Table 4 summarizes the maturity stage  of each architecture program that we assessed. Appendix IV provides the  detailed results of our assessment of each department and agency  architecture program against our maturity framework.
While using this criterion provides an important perspective on the state of  department and agency architecture programs, it can mask the fact that the  programs have met a number of core elements across higher stages of  maturity. When the percentage of core elements that have been fully  satisfied at each stage is considered, the state of the architecture efforts  generally shows both a larger number of more robust architecture  programs as well as more variability across the departments and agencies.  Specifically, 16 departments and agencies have fully satisfied more than 70  percent of the core elements. Examples include Commerce, which has  satisfied 87 percent of the core elements, including 75 percent of the stage  5 elements, even though it is at stage 1 because its enterprise architecture  approval board does not have enterprisewide representation (a stage 2 core  element). Similarly, SSA, which is also a stage 1 because the agency’s  enterprise architecture methodology does not describe the steps for  developing, maintaining, and validating the agency’s enterprise architecture  (a stage 2 core element), has at the same time satisfied 87 percent of all the  elements, including 63 percent of the stage 5 elements. In contrast, the  Army, which is also in stage 1, has satisfied but 3 percent of all framework  elements. Overall, 10 agency architecture programs fully satisfied more  than 75 percent of the core elements, 14 between 50 and 75 percent, and 4  fewer than 50 percent. These four included the three military departments.  Table 5 summarizes for each department and agency the percentage of core  elements fully satisfied in total and by maturity stage.
Notwithstanding the additional perspective that the percentage of core  elements fully satisfied across all stages provides, it is important to note  that the staged core elements in our framework represent a hierarchical or  systematic progression to establishing a well-managed architecture  program, meaning that core elements associated with lower framework  stages generally support the effective execution of higher maturity stage  core elements. For instance, if a program has developed its full suite of “as- is” and “to-be” architecture products, including a sequencing plan (stage 4  core elements), but the products are not under configuration management  (stage 3 core element), then the integrity and consistency of the products  will be not be assured. Our analysis showed that this was the case for a  number of architecture programs. For example, State has developed  certain “as-is” and “to-be” products for the Joint Enterprise Architecture,  which is being developed in collaboration with USAID, but an enterprise  architecture configuration management plan has not yet been finalized.
Further, not satisfying even a single core element can have a significant  impact on the effectiveness of an architecture program. For example, not  having adequate human capital with the requisite knowledge and skills  (stage 2 core element), not using a defined framework or methodology  (stage 2 core element), or not using an independent verification and  validation agent (stage 4 core element), could significantly limit the quality  and utility of an architecture. The DOD’s experience between 2001 and  2005 in developing its BEA is a case in point. During this time, we identified  the need for the department to have an enterprise architecture for its  business operations, and we made a series of recommendations grounded  in, among other things, our architecture management framework to ensure  that it was successful in doing so. In 2005, we reported that the  department had not implemented most of our recommendations. We  further reported that despite developing multiple versions of a wide range  of architecture products, and having invested hundreds of millions of  dollars and 4 years in doing so, the department did not have a well-defined  architecture and that what it had developed had limited utility. Among  other things, we attributed the poor state of its architecture products to  ineffective program governance, communications, program planning,  human capital, and configuration management, most of which are stage 2  and 3 foundational core elements. To the department’s credit, we recently  reported that it has since taken a number of actions to address these  fundamental weaknesses and our related recommendations and that it is  now producing architecture products that provide a basis upon which to  build.
The significance of not satisfying a single core element is also readily  apparent for elements associated with the framework’s content group. In  particular, the framework emphasizes the importance of planning for,  developing, and completing an architecture that includes the “as-is” and the  “to-be” environments as well as a plan for transitioning between the two. It  also recognizes that the “as-is” and “to-be” should address the business,  performance, information/data, application/service, technology, and  security aspects of the enterprise. To the extent these aspects are not  addressed in this way, the quality of the architecture and thus its utility will  suffer. In this regard, we found examples of departments and agencies that  were addressing some but not all of these aspects. For example, HUD has  yet to adequately incorporate security into its architecture. This is  significant because security is relevant to all the other aspects of its  architecture, such as information/data and applications/services. As  another example, NASA’s architecture does not include a plan for  transitioning from the “as-is” to the “to-be” environments. According to the  administration’s Chief Enterprise Architect, a transition plan has not yet  been developed because of insufficient time and staff.
Looking across all the departments and agencies at core elements that are  fully satisfied, not by stage of maturity, but by related groupings of core  elements, provides an additional perspective on the state of the federal  government’s architecture efforts. As noted earlier, these groupings of core  elements are architecture governance, content, use, and measurement.  Overall, departments and agencies on average have fully satisfied 77  percent of the governance-related elements. In particular, 93 and 96 percent  of the agencies have established an architecture program office and  appointed a chief architect, respectively. In addition, 93 percent have plans  that call for their respective architectures to describe the “as-is” and the  “to-be” environments, and for having a plan for transitioning between the  two (see fig. 2). In contrast, however, the core element associated with  having a committee or group with representation from across the  enterprise directing, overseeing, and approving the architecture was fully  satisfied by only 57 percent of the agencies. This core element is important  because the architecture is a corporate asset that needs to be  enterprisewide in scope and accepted by senior leadership if it is to be  leveraged for organizational change.
In contrast to governance, the extent of full satisfaction of those core  elements that are associated with what an architecture should contain  varies widely (see fig. 3). For example, the three content elements that  address prospectively what the architecture will contain, either in relation  to plans or some provision for including needed content, were fully  satisfied about 90 percent of the time. However, the core elements  addressing whether the products now contain such content were fully  satisfied much less frequently (between 54 and 68 percent of the time,  depending on the core element), and the core elements associated with  ensuring the quality of included content, such as employing configuration  management and undergoing independent verification and validation, were  also fully satisfied much less frequently (54 and 21 percent of the time,  respectively). The state of these core elements raises important questions  about the quality and utility of the department and agency architectures.
The degree of full satisfaction of those core elements associated with the  remaining two groups—use and measurement—is even lower (see figs. 4  and 5, respectively). For example, the architecture use-related core  elements were fully satisfied between 39 and 64 percent of the time, while  the measurement-related elements were satisfied between 14 and 71  percent. Of particular note is that only 39 percent of the departments and  agencies could demonstrate that IT investments comply with their  enterprise architectures, only 43 percent of the departments and agencies  could demonstrate that compliance with the enterprise architecture is  measured and reported, and only 14 percent were measuring and reporting  on their respective architecture program’s return on investment. As our  work and related best practices show, the value in having an architecture is  using it to affect change and produce results. Such results, as reported by  the departments and agencies include improved information sharing,  increased consolidation, enhanced productivity, and lower costs, all of  which contribute to improved agency performance. To realize these  benefits, however, IT investments need to comply with the architecture and  measurement of architecture activities, including accrual of expected  benefits, needs to occur.
section_title: Most Agencies Have at Least Partially Satisfied Most Framework Elements
paragraphs: In those instances where departments and agencies have not fully satisfied  certain core elements in our framework, most have at least partially  satisfied these elements. To illustrate, 4 agencies would improve to at  least stage 4 if the criterion for being a given stage was relaxed to only  partially satisfying a core element. Moreover, 11 of the remaining agencies  would advance by two stages under such a less demanding criterion, and  only 6 would not improve their stage of maturity under these  circumstances. A case in point is Commerce, which could move from stage  1 to stage 5 under these circumstances because it has fully satisfied all but  four core elements and these remaining four (one each at stages 2 and 4  and two at stage 5) are partially satisfied. Another case in point is the SSA,  which has fully satisfied all but four core elements (one at stage 2 and three  at stage 5) and has partially satisfied three of these remaining four. If the  criterion used allowed advancement to the next stage by only partially  satisfying core elements, the administration would be stage 4. (See fig. 6 for  a comparison of department and agency program maturity stages under the  two criteria.)
As mentioned earlier, departments and agencies can require considerable  time to completely address issues related to their respective enterprise  architecture programs. It is thus important to note that even though certain  core elements are partially satisfied, fully satisfying some of them may not  be accomplished quickly and easily. It is also important to note the  importance of fully, rather than partially, satisfying certain elements, such  as those that fall within the architecture content group. In this regard, 18,  18, and 21 percent of the departments and agencies partially satisfied the  following stage 4 content-related core elements, respectively: “EA products  describe ‘as-is’ environment, ‘to-be’ environment and sequencing plan”;  “Both ‘as-is’ and ‘to-be’ environments are described in terms of business,  performance, information/data, application/service, and technology”; and  “These descriptions fully address security.” Not fully satisfying these  elements can have important implications for the quality of an architecture,  and thus its usability and results.
section_title: Seven Departments or Agencies Need to Satisfy Five or Fewer Core Elements to Be at Stage 5
paragraphs: Seven departments or agencies would meet our criterion for stage 5 if each  was to fully satisfy one to five additional core elements (see table 6). For  example, Interior could achieve stage 5 by satisfying one additional  element: “EA products and management processes undergo independent  verification and validation.” In this regard, Interior officials have drafted a  statement of work intended to ensure that independent verification and  validation of enterprise architecture products and management processes  is performed. The other six departments and agencies are HUD and OPM,  which could achieve stage 5 by satisfying two additional elements;  Commerce, Labor, and SSA, which could achieve the same by satisfying  four additional elements; and Education which could be at stage 5 by  satisfying five additional elements. Of these seven, five have not fully  satisfied the independent verification and validation core element.
Notwithstanding the fact that five or fewer core elements need to be  satisfied by these agencies to be at stage 5, it is important to note that in  some cases the core elements not being satisfied are not only very  important, but also neither quickly nor easily satisfied. For example, one of  the two elements that HUD needs to satisfy is having its architecture  products address security. This is extremely important as security is an  integral aspect of the architecture’s performance, business,  information/data, application/service, and technical models, and needs to  be reflected thoroughly and consistently across each of them.
section_title: Departments and Agencies Report Numerous Challenges Facing Them in Developing and Using Enterprise Architectures
paragraphs: The challenges facing departments and agencies in developing and using  enterprise architectures are formidable. The challenge that most  departments and agencies cited as being experienced to the greatest extent  is the one that having and using an architecture is intended to overcome—  organizational parochialism and cultural resistance to adopting an  enterprisewide mode of operation in which organizational parts are sub- optimized in order to optimize the performance and results of the  enterprise as a whole. Specifically, 93 percent of the departments and  agencies reported that they encountered this challenge to a significant  (very great or great) or moderate extent. Other challenges reported to this  same extent were ensuring that the architecture program had adequate  funding (89 percent), obtaining staff skilled in the architecture discipline  (86 percent), and having the department or agency senior leaders  understand the importance and role of the enterprise architecture (82  percent).
As we have previously reported, sustained top management leadership is  the key to overcoming each of these challenges. In this regard, our  enterprise architecture management maturity framework provides for such  leadership and addressing these and other challenges through a number of  core elements. These elements contain mechanisms aimed at, for example,  establishing responsibility and accountability for the architecture with  senior leaders and ensuring that the necessary institutional commitments  are made to the architecture program, such as through issuance of  architecture policy and provision of adequate resources (both funding and  people). See table 7 for a listing of the reported challenges and the extent to  which they are being experienced.
section_title: Many Departments and Agencies Reported That They Have Already Realized Significant Architecture Benefits, While Most Expect to Do So in the Future
paragraphs: A large percentage of the departments and agencies reported that they have  already accrued numerous benefits from their respective architecture  programs (see table 8). For example, 70 percent said that have already  improved the alignment between their business operations and the IT that  supports these operations to a significant extent. Such alignment is  extremely important. According to our IT investment management  maturity framework, alignment between business needs and IT  investments is a critical process in building the foundation for an effective  approach to IT investment management. In addition, 64 percent responded  that they have also improved information/knowledge sharing to a  significant or moderate extent. Such sharing is also very important. In 2005,  for example, we added homeland security information sharing to our list of  high-risk areas because despite the importance of information to fighting  terrorism and maintaining the security of our nation, many aspects of  homeland security information sharing remain ineffective and  fragmented. Other examples of mission-effectiveness related benefits  reported as already being achieved to a significant or moderate extent by  roughly one-half of the departments and agencies included improved  agency management and change management and improved system and  application interoperability.
Beyond these benefits, departments and agencies also reported already  accruing, to a significant or moderate extent, a number of efficiency and  productivity benefits. For example, 56 percent reported that they have  increased the use of enterprise software licenses, which can permit cost  savings through economies of scale purchases; 56 percent report that they  have been able to consolidate their IT infrastructure environments, which  can reduce the costs of operating and maintaining duplicative capabilities;  41 percent reported that they have been able to reduce the number of  applications, which is a key to reducing expensive maintenance costs; and  37 percent report productivity improvements, which can free resources to  focus on other high priority matters.
Notwithstanding the number and extent of benefits that department and  agency responses show have already been realized, these same responses  also show even more benefits that they have yet to realize (see table 8). For  example, 30 percent reported that they have thus far achieved, to little or  no extent, better business and IT alignment. They similarly reported that  they have largely untapped many other effectiveness and efficiency  benefits, with between 36 and 70 percent saying these benefits have been  achieved to little or no extent, depending on benefit. Moreover, for all the  cited benefits, a far greater percentage of the departments and agencies (74  to 93 percent) reported that they expect to realize each of the benefits to a  significant or moderate extent sometime in the future. What this suggests is  that the real value in the federal government from developing and using  enterprise architecture remains largely unrealized potential.
Our architecture maturity framework recognizes that a key to realizing this  potential is effectively managing department and agency enterprise  architecture programs. However, knowing whether benefits and results are  in fact being achieved requires having associated measures and metrics. In  this regard, very few (21 percent) of the departments and agencies fully  satisfied our stage 5 core element, “Return on EA investment is measured  and reported.” Without satisfying this element, it is unlikely that the degree  to which expected benefits are accrued will be known.
section_title: Conclusions
paragraphs: If managed effectively, enterprise architectures can be a useful change  management and organizational transformation tool. The conditions for  effectively managing enterprise architecture programs are contained in our  architecture management maturity framework. While a few of the federal  government’s 27 major departments and agencies have fully satisfied all the  conditions needed to be at stage 2 or above in our framework, many have  fully satisfied a large percentage of the core elements across most of the  stages, particularly those elements related to architecture governance.  Nevertheless, most departments and agencies are not yet where they need  to be relative to architecture content, use, and measurement and thus the  federal government is not as well positioned as it should be to realize the  significant benefits that a well-managed architecture program can provide.  Moving beyond this status will require most departments and agencies to  overcome some significant obstacles and challenges. The key to doing so  continues to be sustained organizational leadership. Without such  organizational leadership, the benefits of enterprise architecture will not be  fully realized.
section_title: Recommendations for Executive Action
paragraphs: To assist the 27 major departments and agencies in addressing enterprise  architecture challenges, managing their architecture programs, and  realizing architecture benefits, we recommend that the Administrators of  the Environmental Protection Agency, General Services Administration,  National Aeronautics and Space Administration, Small Business  Administration, and U.S. Agency for International Development; the  Attorney General; the Commissioners of the Nuclear Regulatory  Commission and Social Security Administration; the Directors of the  National Science Foundation and the Office of Personnel Management; and  the Secretaries of the Departments of Agriculture, Commerce, Defense,  Education, Energy, Health and Human Services, Homeland Security,  Housing and Urban Development, Interior, Labor, State, Transportation,  Treasury, and Veterans Affairs ensure that their respective enterprise  architecture programs develop and implement plans for fully satisfying  each of the conditions in our enterprise architecture management maturity  framework.
section_title: Agency Comments and Our Evaluation
paragraphs: We received written or oral comments on a draft of this report from 25 of  the departments and agencies in our review. Of the 25 departments and  agencies, all but one department fully agreed with our recommendation.  Nineteen departments and agencies agreed and six partially agreed with  our findings. Areas of disagreement for these six centered on (1) the  adequacy of the documentation that they provided to demonstrate  satisfaction of certain core elements and (2) recognition of steps that they  reported taking to satisfy certain core elements after we concluded our  review. For the most part, these isolated areas of disagreement did not  result in any changes to our findings for two primary reasons. First, our  findings across the departments and agencies were based on consistently  applied evaluation criteria governing the adequacy of documentation, and  were not adjusted to accommodate any one particular department or  agency. Second, our findings represent the state of each architecture  program as of March 2006, and thus to be consistent do not reflect  activities that may have occurred after this time. Beyond these comments,  several agencies offered suggestions for improving our framework, which  we will consider prior to issuing the next version of the framework. The  departments’ and agencies’ respective comments and our responses, as  warranted, are as follows:  Agriculture’s Associate CIO provided e-mail comments stating that the  department will incorporate our recommendation into its enterprise  architecture program plan.
Commerce’s CIO stated in written comments that the department  concurred with our findings and will consider actions to address our  recommendation. Commerce’s written comments are reproduced in  appendix V.
DOD’s Director, Architecture and Interoperability, stated in written  comments that the department generally concurred with our  recommendation to the five DOD architecture programs included in our  review. However, the department stated that it did not concur with the  one aspect of the recommendation directed at the GIG architecture  concerning independent verification and validation (IV&V) because it  believes that its current internal verification and validation activities are  sufficient. We do not agree for two reasons. First, these internal  processes are not independently performed. As we have previously  reported, IV&V is a recognized hallmark of well managed programs,  including architecture programs, and to be effective, it must be  performed by an entity that is independent of the processes and  products that are being reviewed. Second, the scope of the internal  verification and validation activities only extends to a subset of the  architecture products and management processes.
The department also stated that it did not concur with one aspect of our  finding directed at BEA addressing security. According to DOD, because  GIG addresses security and the GIG states that it extends to all defense  mission areas, including the business mission area, the BEA in effect  addresses security. We do not fully agree. While we acknowledge that  GIG addresses security and states that it is to extend to all DOD mission  areas, including the business mission area, it does not describe how this  will be accomplished for BEA. Moreover, nowhere in the BEA is security  addressed, either through statement or reference, relative to the  architecture’s performance, business, information/data,  application/service, and technology products. DOD’s written comments,  along with our responses, are reproduced in appendix VI.
Education’s Assistant Secretary for Management and Acting CIO stated  in written comments that the department plans to address our findings.  Education’s written comments are reproduced in appendix VII.
Energy’s Acting Associate CIO for Information Technology Reform  stated in written comments that the department concurs with our  report. Energy’s written comments are reproduced in appendix VIII.
DHS’s Director, Departmental GAO/OIG Liaison Office, stated in written  comments that the department has taken, and plans to take, steps to  address our recommendation. DHS’s written comments, along with our  responses to its suggestions for improving our framework, are  reproduced in appendix IX. DHS also provided technical comments via  e-mail, which we have incorporated, as appropriate, in the report.
HUD’s CIO stated in written comments that the department generally  concurs with our findings and is developing a plan to address our  recommendation. The CIO also provided updated information about  activities that the department is taking to address security in its  architecture. HUD’s written comments are reproduced in appendix X.
Interior’s Assistant Secretary, Policy, Management and Budget, stated in  written comments that the department agrees with our findings and  recommendation and that it has recently taken action to address them.  Interior’s written comments are reproduced in appendix XI.
DOJ’s CIO stated in written comments that our findings accurately  reflect the state of the department’s enterprise architecture program and  the areas that it needs to address. The CIO added that our report will  help guide the department’s architecture program and provided  suggestions for improving our framework and its application. DOJ’s  written comments, along with our responses to its suggestions, are  reproduced in appendix XII.
Labor’s Deputy CIO provided e-mail comments stating that the  department concurs with our findings. The Deputy CIO also provided  technical comments that we have incorporated, as appropriate, in the  report.
State’s Assistant Secretary for Resource Management and Chief  Financial Officer provided written comments that summarize actions  that the department will take to fully satisfy certain core elements and  that suggest some degree of disagreement with our findings relative to  three other core elements. First, the department stated that its  architecture configuration management plan has been approved by both  the State and USAID CIOs. However, it provided no evidence to  demonstrate that this was the case as of March 2006 when we concluded  our review, and thus we did not change our finding relative to  architecture products being under configuration management. Second,  the department stated that its enterprise architecture has been approved  by State and USAID executive offices. However, it did not provide any  documentation showing such approval. Moreover, it did not identify  which executive offices it was referring to so as to allow a  determination of whether they were collectively representative of the  enterprise. As a result, we did not change our finding relative to whether  a committee or group representing the enterprise or an investment  review board has approved the current version of the architecture.  Third, the department stated that it provided us with IT investment  score sheets during our review that demonstrate that investment  compliance with the architecture is measured and reported. However,  no such score sheets were provided to us. Therefore, we did not change  our finding. The department’s written comments, along with more  detailed responses, are reproduced in appendix XIII.
Treasury’s Associate CIO for E-Government stated in written comments  that the department concurs with our findings and discussed steps being  taken to mature its enterprise architecture program. The Associate CIO  also stated that our findings confirm the department’s need to provide  executive leadership in developing its architecture program and to  codify the program into department policy. Treasury’s written comments  are reproduced in appendix XIV.
VA’s Deputy Secretary stated in written comments that the department  concurred with our recommendation and that it will provide a detailed  plan to implement our recommendation. VA’s written comments are  reproduced in appendix XV.
EPA’s Acting Assistant Administrator and CIO stated in written  comments that the agency generally agreed with our findings and that  our assessment is a valuable benchmarking exercise that will help  improve agency performance. The agency also provided comments on  our findings relative to five core elements. For one of these core  elements, the comments directed us to information previously provided  about the agency’s architecture committee that corrected our  understanding and resulted in us changing our finding about this core  element. With respect to the other four core elements concerning use of  an architecture methodology, measurement of progress against program  plans, integration of the architecture into investment decision making,  and management of architecture change, the comments also directed us  to information previously provided but this did not result in any changes  to our findings because evidence demonstrating full satisfaction of each  core element was not apparent. EPA’s written comments, along with  more detailed responses to each, are reproduced in appendix XVI.
GSA’s Administrator stated in written comments that the agency  concurs with our recommendation. The Administrator added that our  findings will be critical as the agency works towards further  implementing our framework’s core elements. GSA’s written comments  are reproduced in appendix XVII.
NASA’s Deputy Administrator stated in written comments that the  agency concurs with our recommendation. NASA’s written comments  are reproduced in appendix XVIII. NASA’s GAO Liaison also provided  technical comments via e-mail, which we have incorporated, as  appropriate, in the report.
NSF’s CIO provided e-mail comments stating that the agency will use the  information in our report, where applicable, for future planning and  investment in its architecture program. The CIO also provided technical  comments that we have incorporated, as appropriate, in the report.
NRC’s GAO liaison provided e-mail comments stating that the agency  substantially agrees with our findings and describing activities it has  recently taken to address them.
OPM’s CIO provided e-mail comments stating that the agency agrees  with our findings and describing actions it is taking to address them.
SBA’s GAO liaison provided e-mail comments in which the agency  disagreed with our findings on two core elements. First, and  notwithstanding agency officials’ statements that its architecture  program did not have adequate resources, the liaison did not agree with  our “partially satisfied” assessment for this core element because,  according to the liaison, the agency has limited discretionary funds and  competing, but unfunded, federal mandates to comply with that limit  discretionary funding for an agency of its size. While we acknowledge  SBA’s challenges, we would note that they are not unlike the resource  constraints and competing priority decisions that face most agencies,  and that while the reasons why an architecture program may not be  adequately resourced may be justified, the fact remains that any  assessment of the architecture program’s maturity, and thus its  likelihood of success, needs to recognize whether adequate resources  exist. Therefore, we did not change our finding on this core element.  Second, the liaison did not agree with our finding that the agency did not  have plans for developing metrics for measuring architecture progress,  quality, compliance, and return on investment. However, our review of  documentation provided by SBA and cited by the liaison showed that  while such plans address metric development for architecture progress,  quality, and compliance, they do not address architecture return on  investment. Therefore, we did not change our finding that this core  element was partially satisfied.
SSA’s Commissioner stated in written comments that the report is both  informative and useful, and that the agency agrees with our  recommendation and generally agrees with our findings. Nevertheless,  the agency disagreed with our findings on two core elements. First, the  agency stated that documentation provided to us showed that it has a  methodology for developing, maintaining, and validating its  architecture. We do not agree. In particular, our review of SSA provided  documentation showed that it did not adequately describe the steps to  be followed relative to development, maintenance, or validation.  Second, the agency stated that having the head of the agency approve  the current version of the architecture is satisfied in SSA’s case because  the Clinger-Cohen Act of 1996 vests its CIO with enterprise architecture  approval authority and the CIO has approved the architecture. We do  not agree. The core element in our framework concerning enterprise  architecture approval by the agency head is derived from federal  guidance and best practices upon which our framework is based. This  guidance and related practices, and thus our framework, recognize that  an enterprise architecture is a corporate asset that is to be owned and  implemented by senior management across the enterprise, and that a  key characteristic of a mature architecture program is having the  architecture approved by the department or agency head. Because the  Clinger-Cohen Act does not address approval of an enterprise  architecture, our framework’s core element for agency head approval of  an enterprise architecture is not inconsistent with, and is not  superseded by, that act. SSA’s written comments, along with more  detailed responses, are reproduced in appendix XIX.
USAID’s Acting Chief Financial Officer stated in written comments  stated that the agency will work with State to implement our  recommendation. USAID’s written comments are reproduced in  appendix XX.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies of this report to the  Administrators of the Environmental Protection Agency, General Services  Administration, National Aeronautics and Space Administration, Small  Business Administration, and U.S. Agency for International Development;  the Attorney General; the Commissioners of the Nuclear Regulatory  Commission and Social Security Administration; the Directors of the  National Science Foundation and the Office of Personnel Management; and  the Secretaries of the Departments of Agriculture, Commerce, Defense,  Education, Energy, Health and Human Services, Homeland Security,  Housing and Urban Development, Interior, Labor, State, Transportation,  Treasury, and Veterans Affairs. We will also make copies available to others  upon request. In addition, the report will be available at no charge on the  GAO Web site at http://www.gao.gov.
If you have any questions concerning this information, please contact me at  (202) 512-3439 or by e-mail at hiter@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. Key contributors to this report are listed in appendix  XXI.
section_title: Reported Enterprise Architecture Costs Vary, with Contractors and Personnel Accounting for Most Costs
paragraphs: Department- and agency-reported data show wide variability in their costs  to develop and maintain their enterprise architectures. Generally, the costs  could be allocated to several categories with the majority of costs  attributable to contractor support and agency personnel.
subsections: section_title: Architecture Development and Maintenance Costs Vary
paragraphs: As we have previously reported, the depth and detail of the architecture to  be developed and maintained is dictated by the scope and nature of the  enterprise and the extent of enterprise transformation and modernization  envisioned. Therefore, the architecture should be tailored to the individual  enterprise and that enterprise’s intended use of the architecture.  Accordingly, the level of resources that a given department or agency  invests in its architecture is likely to vary.
Departments and agencies reported that they have collectively invested a  total of $836 million to date on enterprise architecture development.  Across the 27 departments and agencies, these development costs ranged  from a low of $2 million by the Department of the Navy to a high of $433  million by the Department of Defense (DOD) on its Business Enterprise  Architecture (BEA). Department and agency estimates of the costs to  complete their planned architecture development efforts collectively total  about $328 million. The department and agencies combined estimates of  annual architecture maintenance costs is about $146 million. These  development and maintenance estimates, however, do not include the  Departments of the Army and Justice because neither provided these cost  estimates. Figures 7 through 9 depict the variability of cost data reported  by the departments and agencies.
section_title: Contractor Support Accounts for the Majority of Architecture Development Costs
paragraphs: All of the departments and agencies reported developing their architecture  in-house using contractor support. All but two of the departments and  agencies allocated their respective architecture development costs to the  following cost categories: contractor support, agency personnel, tools,  methodologies, training, and other. These 26 agencies accounted for about  $741 million of the $836 million total development costs cited above. The  vast majority (84 percent) of the $741 million were allocated to contractor  services ($621 million), followed next by agency personnel (13 percent or  $94 million). The remaining $26 million were allocated as follows: $12  million (2 percent) to architecture tools; $9 million (1 percent) to “other”  costs; $4 million (1 percent) to architecture methodologies; and $2 million  (less than 1 percent) to training. (See fig. 10.)
section_title: Architecture Development Activities Were Reported as Largest Component of Contractor-Related Costs
paragraphs: The departments and agencies allocated the reported $621 million in  contractor-related costs to the following five contractor cost categories:  architecture development, independent verification and validation, methodology, support services, and other. Of these categories,  architecture development activities accounted for the majority of costs— about $594 million (87 percent). The remaining $85 million was allocated as  follows: $51 million (7 percent) to support services, $13 million (2 percent)  to “other” costs, $11 million (2 percent) to independent verification and  validation, and $10 million (1 percent) to methodologies. (See fig. 11.)
section_title: Departments and Agencies Reported Experiences with Their Architecture Tools and Frameworks
paragraphs: Departments and agencies reported additional information related to the  implementation of their enterprise architectures. This information includes  architecture tools and frameworks.
subsections: section_title: Departments and Agencies Reported Using a Variety of Enterprise Architecture Tools with Varying Degrees of Satisfaction
paragraphs: As stated in our enterprise architecture management maturity framework,  an automated architecture tool serves as the repository of architecture  artifacts, which are the work products that are produced and used to  capture and convey architectural information. An agency’s choice of tool  should be based on a number of considerations, including agency needs  and the size and complexity of the architecture.
The departments and agencies reported that they use various automated  tools to develop and maintain their enterprise architectures, with 12  reporting that they use more than one tool. In descending order of  frequency, the architecture tools identified were System Architect (18  instances), Microsoft Visio (17), Metis (12), Rational Rose (8), and  Enterprise Architecture Management System (EAMS) (4). In addition, 21  departments and agencies reported using one or more other architecture  tools. Figure 12 shows the number of departments and agencies using each  architecture tool, including the other tools.
The departments and agencies also reported various levels of satisfaction  with the different enterprise architecture tools. Specifically, about 75  percent of those using Microsoft Visio were either very or somewhat  satisfied with the tool, as compared to about 67 percent of those using  Metis, about 63 percent of those using Rational Rose, about 59 percent of  those using System Architect, and 25 percent of those using EAMS. This  means that the percentage of departments and agencies that were  dissatisfied, either somewhat or very, with their respective tools ranged  from a high of 75 percent of those using EAMS, to a low of about 6 percent  of those using System Architect. No departments or agencies that used  Metis, Rational Rose, or Microsoft Visio reported any dissatisfaction. See  table 9 for a summary of department and agency reported satisfaction with  their respective tools.
section_title: Departments and Agencies Reported Using a Variety of Enterprise Architecture Frameworks with Varying Levels of Satisfaction
paragraphs: As we have previously stated, an enterprise architecture framework  provides a formal structure for representing the architecture’s content and  serves as the basis for the specific architecture products and artifacts that  the department or agency develops and maintains. As such, a framework  helps ensure the consistent representation of information from across the  organization and supports orderly capture and maintenance of architecture  content.
The departments and agencies reported using various frameworks to  develop and maintain their enterprise architectures. The most frequently  cited frameworks were the Federal Enterprise Architecture Program  Management Office (FEAPMO) Reference Models (25 departments and  agencies), the Federal Enterprise Architecture Framework (FEAF) (19  departments and agencies), and the Zachman Framework (17 departments  and agencies), with 24 reporting using more than one framework. Other,  less frequently reported frameworks were the Department of Defense  Architecture Framework (DODAF), the National Institute of Standards and  Technology (NIST) framework, and The Open Group Architecture  Framework (TOGAF). See figure 13 for a summary of the number of  departments and agencies that reported using each framework.
Departments and agencies also reported varying levels of satisfaction with  their respective architecture. Specifically, about 72 percent of those using  the FEAF indicated that they were either very or somewhat satisfied, and  about 67 and 61 percent of those using the Zachman framework and the  FEAPMO reference models, respectively, reported that they were similarly  satisfied. As table 10 shows, few of the agencies that responded to our  survey reported being dissatisfied with any of the frameworks.
section_title: Objective, Scope, and Methodology
paragraphs: Our objective was to determine the current status of federal department  and agency enterprise architecture efforts. To accomplish this objective,  we focused on 28 enterprise architecture programs relating to 27 major  departments and agencies. These 27 included the 24 departments and  agencies included in the Chief Financial Officers Act. In addition, we  included the three military services (the Departments of the Army, Air  Force, and Navy) at the request of Department of Defense (DOD) officials.  For the DOD, we also included both of its departmentwide enterprise  architecture programs—the Global Information Grid and the Business  Enterprise Architecture. The U.S. Agency for International Development  (USAID), which is developing a USAID enterprise architecture and working  with the Department of State (State) to develop a Joint Enterprise  Architecture, asked that we evaluate its efforts to develop the USAID  enterprise architecture. State officials asked that we evaluate their agency’s  enterprise architecture effort based the Joint Enterprise Architecture being  developed with USAID. We honored both of these requests.
Table 11 lists the 28 department and agency enterprise architecture  programs that formed the scope of our review.
To determine the status of each of these architecture programs, we  developed a data collection instrument based on our Enterprise  Architecture Management Maturity Framework (EAMMF), and related  guidance, such as OMB Circular A-130 and guidance published by the  federal Chief Information Officers (CIO) Council, and our past reports and  guidance on the management and content of enterprise architectures. We  pretested this instrument at one department and one agency. Based on the  results of the pretest, we modified our instrument as appropriate to ensure  that our areas of inquiry were complete and clear.
Next, we identified the Chief Architect or comparable official at each of the  27 departments and agencies, and met with them to discuss our scope and  methodology, share our data collection instrument, and discuss the type  and nature of supporting documentation needed to verify responses to our  instrument questions.
On the basis of department and agency provided documentation to support  their respective responses to our data collection instrument, we analyzed  the extent to which each satisfied the 31 core elements in our architecture  maturity framework. To guide our analysis, we defined detailed evaluation  criteria for determining whether a given core element was fully satisfied,  partially satisfied, or not satisfied. The criteria for the stage 2, 3, 4, and 5  core elements are contained in tables 12, 13, 14, and 15 respectively. To  fully satisfy a core element, sufficient documentation had to be provided to  permit us to verify that all aspects of the core element were met. To  partially satisfy a core element, sufficient documentation had to be  provided to permit us to verify that at least some aspects of the core  element were met. Core elements that were neither fully nor partially  satisfied were judged to be not satisfied.
Our evaluation included first analyzing the extent to which each  department and agency satisfied the core elements in our framework, and  then meeting with department and agency representatives to discuss core  elements that were not fully satisfied and why. As part of this interaction,  we sought, and in some cases were provided, additional supporting  documentation. We then considered this documentation in arriving at our  final determinations about the degree to which each department and  agency satisfied each core element in our framework. In applying our  evaluation criteria, we analyzed the results of our analysis across different  core elements to determine patterns and issues. Our analysis made use of  computer programs that were developed by an experienced staff; these  programs were independently verified.
Through our data collection instrument, we also solicited from each  department and agency information on enterprise architecture challenges  and benefits, including the extent to which they had been or were expected  to be experienced. In addition, we solicited information on architecture  costs, including costs to date and estimated costs to complete and maintain  each architecture. We also solicited other information, such as use of and  satisfaction with architecture tools and frameworks. We analyzed these  additional data to determine relevant patterns. We did not independently verify these data. The results presented in this report reflect the state of  department and agency architecture programs as of March 8, 2006.
We conducted our work in the Washington, D.C., metropolitan area, from  May 2005 to June 2006, in accordance with generally accepted government  auditing standards.
section_title: Detailed Assessments of Individual Departments and Agencies against Our EA Management Maturity Framework
subsections: section_title: Department of Agriculture
paragraphs: Table 16 shows USDA’s satisfaction of framework elements in version 1.1 of  GAO’s EAMMF. version 1.1 of GAO’s EAMMF.
section_title: Department of the Army
paragraphs: Table 18 shows Army’s satisfaction of framework elements in version 1.1 of  GAO’s EAMMF.
section_title: Department of Commerce
paragraphs: Table 19 shows Commerce’s satisfaction of framework elements in version  1.1 of GAO’s EAMMF.
section_title: Department of Defense – Business Enterprise Architecture
paragraphs: Table 20 shows the BEA’s satisfaction of framework elements in version 1.1  of GAO’s EAMMF.
section_title: Department of Defense – Global Information Grid
paragraphs: Table 21 shows the GIG’s satisfaction of framework elements in version 1.1  of GAO’s EAMMF.
section_title: Department of Education
paragraphs: Table 22 shows Education’s satisfaction of framework elements in version  1.1 of GAO’s EAMMF.
section_title: Department of Energy
paragraphs: Table 23 shows Energy’s satisfaction of framework elements in version 1.1  of GAO’s EAMMF.
section_title: Department of Health and Human Services
paragraphs: Table 24 shows HHS’s satisfaction of framework elements in version 1.1 of  GAO’s EAMMF.
section_title: Department of Homeland Security
paragraphs: Table 25 shows DHS’s satisfaction of framework elements in version 1.1 of  GAO’s EAMMF.
section_title: Department of Housing and Urban Development
paragraphs: Table 26 shows HUD’s satisfaction of framework elements in version 1.1 of  GAO’s EAMMF.
section_title: The Department of the Interior
paragraphs: Table 27 shows DOI’s satisfaction of framework elements in version 1.1 of  GAO’s EAMMF.
section_title: Department of Justice
paragraphs: Table 28 shows DOJ’s satisfaction of framework elements in version 1.1 of  GAO’s EAMMF.
section_title: Department of Labor
paragraphs: Table 29 shows Labor’s satisfaction of framework elements in version 1.1 of  GAO’s EAMMF.
section_title: Department of the Navy
paragraphs: Table 30 shows Navy’s satisfaction of framework elements in version 1.1 of  GAO’s EAMMF.
section_title: Department of State
paragraphs: Table 31 shows State’s satisfaction of framework elements in version 1.1 of  GAO’s EAMMF.
section_title: Department of Transportation
paragraphs: Table 32 shows Transportation’s satisfaction of framework elements in  version 1.1 of GAO’s EAMMF.
section_title: Department of the Treasury
paragraphs: Table 33 shows the Treasury’s satisfaction of framework elements in  version 1.1 of GAO’s EAMMF.
section_title: Department of Veterans Affairs
paragraphs: Table 34 shows VA’s satisfaction of framework elements in version 1.1 of  GAO’s EAMMF.
section_title: Environmental Protection Agency
paragraphs: Table 35 shows EPA’s satisfaction of framework elements in version 1.1 of  GAO’s EAMMF.
section_title: General Services Administration
paragraphs: Table 36 shows GSA’s satisfaction of framework elements in version 1.1 of  GAO’s EAMMF.
section_title: National Aeronautics and Space Administration
paragraphs: Table 37 shows NASA’s satisfaction of framework elements in version 1.1 of  GAO’s EAMMF.
section_title: National Science Foundation
paragraphs: Table 38 shows NSF’s satisfaction of framework elements in version 1.1 of  GAO’s EAMMF.
section_title: Nuclear Regulatory Commission
paragraphs: Table 39 shows NRC’s satisfaction of framework elements in version 1.1 of  GAO’s EAMMF.
section_title: Office of Personnel Management
paragraphs: Table 40 shows OPM’s satisfaction of framework elements in version 1.1 of  GAO’s EAMMF.
section_title: Small Business Administration
paragraphs: Table 41 shows SBA’s satisfaction of framework elements in version 1.1 of  GAO’s EAMMF.
section_title: Social Security Administration
paragraphs: Table 42 shows SSA’s satisfaction of framework elements in version 1.1 of  GAO’s EAMMF.
section_title: U.S. Agency for International Development
paragraphs: Table 43 shows USAID’s satisfaction of framework elements in version 1.1  of GAO’s EAMMF.
section_title: Comments from the Department of Commerce
section_title: Comments from the Department of Defense
subsections: section_title: GAO Comments
paragraphs: 1. We do not agree for two reasons. First, DOD’s internal processes for  reviewing and validating the Global Information Grid (GIG), while  important and valuable to ensuring architecture quality, are not  independently performed. As we have previously reported,  independent verification and validation is a recognized hallmark of  well-managed programs, including architecture programs. To be  effective, it should be performed by an entity that is independent of the  processes and products that are being reviewed to help ensure that it is  done in an unbiased manner and that is based on objective evidence.  Second, the scope of these internal review and validation efforts only  extends to a subset of GIG products and management processes.  According to our framework, independent verification and validation  should address both the architecture products and the processes used  to develop them.  2. While we acknowledge that GIG program plans provide for addressing  security, and our findings relative to the GIG reflect this, this is not the  case for DOD’s Business Enterprise Architecture (BEA). More  specifically, how security will be addressed in the BEA performance,  business, information/data, application/service, and technology  products is not addressed in the BEA either by explicit statement or  reference. This finding relative to the BEA is consistent with our recent  report on DOD’s Business System Modernization.
section_title: Comments from the Department of Education
section_title: Comments from the Department of Energy
section_title: Comments from the Department of Homeland Security
subsections: section_title: GAO Comments
paragraphs: 1. We acknowledge this recommendation and offer three comments in  response. First, we have taken a number of steps over the last 5 years to  coordinate our framework with OMB. For example, in 2002, we based  version 1.0 of our framework on the OMB-sponsored CIO Council  Practical Guide to Federal Enterprise Architecture, and we obtained  concurrence on the framework from the practical guide’s principal  authors. Further, we provided a draft of this version to OMB for  comment, and in our 2002 report in which we assessed federal  departments and agencies against this version, we recommended that  OMB use the framework to guide and assess agency architecture  efforts. In addition, in developing the second version of our framework  in 2003, we solicited comments from OMB as well as federal  departments and agencies. We also reiterated our recommendation to  OMB to use the framework in our 2003 report in which we assessed  federal departments and agencies against the second version of the  framework.
Second, we have discussed alignment of our framework and OMB’s  architecture assessment tool with OMB officials. For example, after  OMB developed the first version of its architecture assessment tool in  2004, we met with OMB officials to discuss our respective tools and  periodic agency assessments. We also discussed OMB’s plans for  issuing the next version of its assessment tool and how this next  version would align with our framework. At that time, we advocated  the development of comprehensive federal standards governing all  aspects of architecture development, maintenance, and use. In our  view, neither our framework nor OMB’s assessment tool provide such  comprehensive standards, and in the case of our framework, it is not  intended to provide such standards. Nevertheless, we plan to continue  to evolve, refine, and improve our framework, and will be issuing an  updated version that incorporates lessons learned from the results of  this review. In doing so, we will continue to solicit comments from  federal departments and agencies, including OMB.
Third, we believe that while our framework and OMB’s assessment tool  are not identical, they nevertheless consist of a common cadre of best  practices and characteristics, as well as other relevant criteria that,  taken together, are complementary and provide greater direction to,  and visibility into, agency architecture programs than either does alone.
section_title: Comments from the Department of Housing and Urban Development
section_title: Comments from the Department of the Interior
section_title: Comments from the Department of Justice
subsections: section_title: GAO Comments
paragraphs: 1. See DHS comment 1 in appendix IX. Also, while we do not have a basis  for commenting on the content of the department’s OMB evaluation  submission package because we did not receive it, we would note that  the information that we solicit to evaluate a department or agency  against our framework includes only information that should be readily  available as part of any well-managed architecture program. 2. We understand the principles of federated and segmented  architectures, but would emphasize that our framework is intentionally  neutral with respect to these and other architecture approaches (e.g.,  service-oriented). That is, the scope of the framework, by design, does  not extend to defining how various architecture approaches should  specifically be pursued, although we recognize that supplemental  guidance on this approach would be useful. Our framework was  created to organize fundamental (core) architecture management  practices and characteristics (elements) into a logical progression. As  such, it was intended to fill an architecture management void that  existed in 2001 and thereby provide the context for more detailed  standards and guidance in a variety of areas. It was not intended to be  the single source of all relevant architecture guidance. 3. We agree, and believe that this report, by clearly identifying those  departments and agencies that have fully satisfied each core element,  serves as the only readily available reference tool of which we are  aware for gaining such best practice insights.
section_title: Comments from the Department of State
subsections: section_title: GAO Comments
paragraphs: 1. We acknowledge the comment that both CIOs approved the  configuration management plan. However, the department did not  provide us with any documentation to support this statement.  2. We acknowledge the comment that the architecture has been approved  by State and USAID executive offices. However, the department did not  provide any documentation describing to which executive offices the  department is referring to allow a determination of whether they were  collectively representative of the enterprise. Moreover, as we state in  the report, the chief architect told us that a body representative of the  enterprise has not approved the current version of the architecture, and  according to documentation provided, the Joint Management Council is  to be responsible for approving the architecture.  3. We acknowledge that steps have been taken and are planned to treat  the enterprise architecture as an integral part of the investment  management process, as our report findings reflect. However, our point  with respect to this core element is whether the department’s  investment portfolio compliance with the architecture is being  measured and reported to senior leadership. In this regard, State did  not provide the score sheets referred to in its comments, nor did it  provide any other evidence that such reporting is occurring.
section_title: Comments from the Department of the Treasury
section_title: Comments from the Department Veterans Affairs
section_title: Comments from the Environmental Protection Agency
subsections: section_title: GAO Comments
paragraphs: 1. We agree and have modified our report to recognize evidence contained  in the documents.  2. We do not agree. The 2002 documents do not contain steps for  architecture maintenance. Further, evidence was not provided  demonstrating that the recently prepared methodology documents  were approved prior to the completion of our evaluation.  3. We do not agree. While we do not question whether EPA’s EA  Transition Strategy and Sequencing Plan illustrates how annual  progress in achieving the target architectural environment is measured  and reported, this is not the focus of this core element. Rather, this core  element addresses whether progress against the architecture program  management plan is tracked and reported. While we acknowledge  EPA’s comment that it tracks and reports such progress against plans  on a monthly basis, neither a program plan nor reports of progress  against this plan were provided as documentary evidence to support  this statement.  4. We do not agree. First, while EPA’s IT investment management process  provides for consideration of the enterprise architecture in investment  selection and control activities, no evidence was provided  demonstrating that the process has been implemented. Second, while  EPA provided a description of its architecture change management  process, no evidence was provided that this process has been approved  and implemented.
section_title: Comments from the General Services Administration
section_title: Comments from the National Aeronautics and Space Administration
section_title: Comments from the Social Security Administration
subsections: section_title: GAO Comments
paragraphs: 1. We do not agree. Neither the governance committee charter nor the  configuration management plan explicitly describe a methodology that  includes detailed steps to be followed for developing, maintaining, and  validating the architecture. Rather, these documents describe, for  example, the responsibilities of the architecture governance committee  and architecture configuration management procedures. 2. We do not agree. The core element in our framework concerning  enterprise architecture approval by the agency head is derived from  federal guidance and best practices upon which our framework is  based. This guidance and related practices, and thus our framework,  recognize that an enterprise architecture is a corporate asset that is to  be owned and implemented by senior management across the  enterprise, and that a key characteristic of a mature architecture  program is having the architecture approved by the department or  agency head. Because the Clinger-Cohen Act does not address approval  of an enterprise architecture, our framework’s core element for agency  head approval of an enterprise architecture is not inconsistent with,  and is not superseded by, that act.
section_title: Comments from the U.S. Agency for International Development
section_title: GAO Contact and Staff Acknowledgements
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the person named above, Edward Ballard, Naba Barkakati,  Mark Bird, Jeremy Canfield, Jamey Collins, Ed Derocher, Neil Doherty,  Mary J. Dorsey, Marianna J. Dunn, Joshua Eisenberg, Michael Holland,  Valerie Hopkins, James Houtz, Ashfaq Huda, Cathy Hurley, Cynthia  Jackson, Donna Wagner Jones, Ruby Jones, Stu Kaufman, Sandra Kerr,  George Kovachick, Neela Lakhmani, Anh Le, Stephanie Lee, Jayne  Litzinger, Teresa M. Neven, Freda Paintsil, Altony Rice, Keith Rhodes,  Teresa Smith, Mark Stefan, Dr. Rona Stillman, Amos Tevelow, and Jennifer  Vitalbo made key contributions to this report.