DOD Business Systems Modernization: Progress in Establishing Corporate Management Controls Needs to Be Replicated Within Military Departments
Released: 2008-05-15T00:00:00

Letter

Background

DOD is a massive and complex organization. The department reported that its fiscal year 2007 operations involved approximately $1.5 trillion in assets and $2.1 trillion in liabilities; more than 2.9 million military and civilian personnel; and $544 billion in net cost of operations. For fiscal year 2008, the department has received discretionary budget authority for about $546 billion and reports total obligations of about $492 billion to support ongoing operations and activities related to the Global War on Terrorism. Organizationally, the department includes the Office of the Secretary of Defense, the Chairman of the Joint Chiefs of Staff, the military departments, numerous defense agencies and field activities, and various unified combatant commands that are either responsible for specific geographic regions or specific functions. (See fig. 1 for a simplified depiction of DOD’s organizational structure.)

In support of its military operations, the department performs an assortment of interrelated and interdependent business functions, including logistics management, procurement, health care management, and financial management. As we have previously reported, the DOD systems environment that supports these business functions is overly complex and error prone, and is characterized by (1) little standardization across the department, (2) multiple systems performing the same tasks, (3) the same data stored in multiple systems, and (4) the need for data to be entered manually into multiple systems. Moreover, the department recently reported that this systems environment is comprised of approximately 3,000 separate business systems. For fiscal year 2007, Congress appropriated approximately $15.7 billion to DOD, and for fiscal year 2008, the department has requested about $15.9 billion in appropriated funds to operate, maintain, and modernize these business systems and associated IT infrastructure.

As we have previously reported, the department’s nonintegrated and duplicative systems impair its ability to combat fraud, waste, and abuse. In fact, DOD currently bears responsibility, in whole or in part, for 15 of our 27 high-risk areas. Eight of these areas are specific to the department, while it shares responsibility for seven other governmentwide high-risk areas. DOD’s business systems modernization is one of the high-risk areas, and it is an essential enabler to addressing many of the department’s other high-risk areas. For example, modernized business systems are integral to the department’s efforts to address its financial, supply chain, and information security management high-risk areas.

Enterprise Architecture and IT Investment Management Controls Are Critical to Achieving Successful Systems Modernization

Effective use of an enterprise architecture—a modernization blueprint—is a hallmark of successful public and private organizations. For more than a decade, we have promoted the use of architectures to guide and constrain systems modernization, recognizing them as a crucial means to this challenging goal: optimally defined operational and technological environments. Congress, the Office of Management and Budget (OMB), and the federal Chief Information Officer’s (CIO) Council also have recognized the importance of an architecture-centric approach to modernization. The Clinger-Cohen Act of 1996 mandates that an agency’s CIO develop, maintain, and facilitate the implementation of an information technology architecture. Further, the E-Government Act of 2002 requires OMB to oversee the development of enterprise architectures within and across agencies. In addition, we, OMB, and the CIO Council have issued guidance that emphasizes the need for system investments to be consistent with these architectures.

A corporate approach to IT investment management is characteristic of successful public and private organizations. Recognizing this, Congress enacted the Clinger-Cohen Act of 1996, which requires OMB to establish processes to analyze, track, and evaluate the risks and results of major capital investments in IT systems made by executive agencies. In response to the Clinger-Cohen Act and other statutes, OMB has developed policy and issued guidance for planning, budgeting, acquisition, and management of federal capital assets. We also have issued guidance in this area.

Enterprise Architecture: A Brief Description

An enterprise architecture provides a clear and comprehensive picture of an entity, whether it is an organization (e.g., a federal department) or a functional or mission area that cuts across more than one organization (e.g., financial management). This picture consists of snapshots of both the enterprise’s current (“As Is”) environment and its target (“To Be”) environment. These snapshots consist of “views,” which are one or more interdependent and interrelated architecture products (e.g., models, diagrams, matrixes, and text) that provide logical or technical representations of the enterprise. The architecture also includes a transition or sequencing plan, which is based on an analysis of the gaps between the “As Is” and “To Be” environments; this plan provides a temporal road map for moving between the two environments and incorporates such considerations as technology opportunities, marketplace trends, fiscal and budgetary constraints, institutional system development and acquisition capabilities, legacy and new system dependencies and life expectancies, and the projected value of competing investments.

The suite of products produced for a given entity’s enterprise architecture, including its structure and content, is largely governed by the framework used to develop the architecture. Since the 1980s, various architecture frameworks have been developed, such as John A. Zachman’s “A Framework for Information Systems Architecture” and the DOD Architecture Framework.

The importance of developing, implementing, and maintaining an enterprise architecture is a basic tenet of both organizational transformation and systems modernization. Managed properly, an enterprise architecture can clarify and help optimize the interdependencies and relationships among an organization’s business operations (and the underlying IT infrastructure and applications) that support these operations. Moreover, when an enterprise architecture is employed in concert with other important management controls, such as portfolio-based capital planning and investment control practices, architectures can greatly increase the chances that an organization’s operational and IT environments will be configured to optimize mission performance. Our experience with federal agencies has shown that investing in IT without defining these investments in the context of an architecture often results in systems that are duplicative, not well integrated, and unnecessarily costly to maintain and interface.

One approach to structuring an enterprise architecture is referred to as a federated enterprise architecture. Such a structure treats the architecture as a family of coherent but distinct member architectures that conform to an overarching architectural view and rule set. This approach recognizes that each member of the federation has unique goals and needs as well as common roles and responsibilities with the levels above and below it. Under a federated approach, member architectures are substantially autonomous, although they also inherit certain rules, policies, procedures, and services from higher-level architectures. As such, a federated architecture enables component organization autonomy while ensuring enterprisewide linkages and alignment where appropriate. Where commonality among components exists, there also are opportunities for identifying and leveraging shared services.

A service-oriented architecture (SOA) is an approach for sharing business capabilities across the enterprise by designing functions and applications as discrete, reusable, and business-oriented services. As such, service orientation permits sharing capabilities that may be under the control of different component organizations. As we have previously reported, such capabilities or services need to be, among other things, (1) self-contained, meaning that they do not depend on any other functions or applications to execute a discrete unit of work; (2) published and exposed as self- describing business capabilities that can be accessed and used; and (3) subscribed to via well-defined and standardized interfaces. A SOA approach is thus not only intended to reduce redundancy and increase integration, but also to provide the kind of flexibility needed to support a quicker response to changing and evolving business requirements and emerging conditions.

IT Investment Management: A Brief Description

IT investment management is a process for linking IT investment decisions to an organization’s strategic objectives and business plans that focuses on selecting, controlling, and evaluating investments in a manner that minimizes risks while maximizing the return of investment.

During the selection phase, the organization (1) identifies and analyzes each project’s risks and returns before committing significant funds to any project and (2) selects those IT projects that will best support its mission needs.

During the control phase, the organization ensures that, as projects develop and investment expenditures continue, they continue to meet mission needs at the expected levels of cost and risk. If the project is not meeting expectations or if problems arise, steps are quickly taken to address the deficiencies.

During the evaluation phase, actual versus expected results are compared once a project has been fully implemented. This is done to (1) assess the project’s impact on mission performance, (2) identify any changes or modifications to the project that may be needed, and (3) revise the investment management process based on lessons learned.

Consistent with this guidance, our IT Investment Management framework (ITIM) consists of five progressive stages of maturity for any given agency relative to selecting, controlling, and evaluating its investment management capabilities. (See fig. 2 for the five ITIM stages of maturity.) Stage 2 critical processes lay the foundation by establishing successful, predictable, and repeatable investment control processes at the project level. Stage 3 is where the agency moves from project-centric processes to portfolio-based processes and evaluates potential investments according to how well they support the agency’s missions, strategies, and goals. Organizations implementing these Stages 2 and 3 practices have in place selection, control, and evaluation processes that are consistent with the Clinger-Cohen Act. Stages 4 and 5 require the use of evaluation techniques to continuously improve both investment processes and portfolios in order to better achieve strategic outcomes.

The overriding purpose of the framework is to encourage investment selection, control, and evaluate processes that promote business value and mission performance, reduce risk, and increase accountability and transparency. We have used the framework in several of our evaluations, and a number of agencies have adopted it. With the exception of the first stage, each maturity stage is composed of “critical processes” that must be implemented and institutionalized in order for the organization to achieve that stage. Each ITIM critical process consists of “key practices”—to include organizational structures, policies, and procedures—that must be executed to implement the critical process. Our research shows that agency efforts to improve investment management capabilities should focus on implementing all lower stage practices before addressing higher stage practices.

DOD’s Institutional Approach to Business Systems Modernization

In 2005, the department reassigned responsibility for providing executive leadership for the direction, oversight, and execution of its business systems modernization efforts to several entities. These entities and their responsibilities include the Defense Business Systems Management Committee (DBSMC), which serves as the highest ranking investment review and decision-making body for business systems modernization activities; the Principal Staff Assistants, who serve as the certification authorities for business system modernizations in their respective core business missions; the Investment Review Boards (IRB), which are chaired by the certifying authorities and form the review and decision- making bodies for business system investments in their respective areas of responsibility; and the Business Transformation Agency (BTA), which is responsible for supporting the DBSMC and the IRBs, and for leading and coordinating business transformation efforts across the department. DOD’s component organizations, to varying degrees, have leveraged existing, and established new, business system governance bodies to support their respective investment precertification responsibilities.

Table 1 lists these entities and provides greater detail on their roles, responsibilities, and composition.

In 2005, DOD reported that it had adopted a “tiered accountability” approach to business transformation. Under this approach, responsibility and accountability for business architectures and systems investment management are assigned to different levels in the organization. For example, the BTA is responsible for developing the corporate BEA (i.e., the thin layer of corporate policies, capabilities, standards, rules), and the associated enterprise transition plan (ETP). The components are responsible for defining a component-level architecture and transition plans associated with their own tier of responsibility and for doing so in a manner that is aligned with (i.e., does not violate) the corporate BEA. Similarly, program managers are responsible for developing program-level architectures and plans and ensuring alignment with the architectures and transition plans above them. This concept is to allow for autonomy while also ensuring linkages and alignment from the program level through the component level to the enterprise level. Table 2 describes the four investment tiers and identifies the associated reviewing and approving entities.

Summary of Fiscal Year 2005 National Defense Authorization Act Requirements

Congress included six provisions in the fiscal year 2005 National Defense Authorization Act that are aimed at ensuring DOD’s development of a well-defined BEA and associated ETP, as well as the establishment and implementation of effective investment management structures and processes. The requirements are as follows: 1. Develop a BEA that includes an information infrastructure that, at a comply with all federal accounting, financial management, and reporting requirements; routinely produce timely, accurate, and reliable financial information for management purposes; integrate budget, accounting, and program information and systems; provide for the systematic measurement of performance, including the ability to produce timely, relevant, and reliable cost information; include policies, procedures, data standards, and system interface requirements that are to be applied uniformly throughout the department; and be consistent with OMB policies and procedures. 2. Develop an ETP for implementing the architecture that includes: an acquisition strategy for new systems needed to complete the a list and schedule of legacy business systems to be terminated; a list and strategy of modifications to legacy business systems; and time-phased milestones, performance metrics, and a statement of financial and non-financial resource needs. 3. Identify each business system proposed for funding in DOD’s fiscal year budget submissions and include: description of the certification made on each business system proposed for funding in that budget; funds, identified by appropriations, for current services and for business systems modernization; and the designated approval authority for each business system. 4. Delegate the responsibility for business systems to designated approval authorities within the Office of the Secretary of Defense. 5. Require each approval authority to establish investment review structures and processes, including a hierarchy of IRBs—each with appropriate representation from across the department. The review process must cover: review and approval of each business system by an IRB before funds are obligated; at least an annual review of every business system investment; use of threshold criteria to ensure an appropriate level of review and use of procedures for making architecture compliance certifications; use of procedures consistent with DOD guidance; and incorporation of common decision criteria. 6. Effective October 1, 2005, DOD may not obligate appropriated funds for a defense business system modernization with a total cost of more than $1 million unless the approval authority certifies that the business system modernization: complies with the BEA and is necessary to achieve a critical national security capability or address a critical requirement in an area such as safety or security; or is necessary to prevent a significant adverse effect on an essential project in consideration of alternative solutions, and the certification is approved by the DBSMC.

Summary of Recent GAO Reviews of DOD’s Business Systems Modernization and Business Transformation Efforts

In November 2005, May 2006, and May 2007, we reported that DOD had partially satisfied four of the six business system modernization requirements in the fiscal year 2005 National Defense Authorization Act relative to architecture development, transition plan development, budgetary disclosure, and investment review. In addition, we reported that it had fully satisfied the requirement concerning designated approval authorities and it was in the process of satisfying the last requirement for certification and approval of modernizations costing in excess of $1 million. As a result, each report concluded that the department had made important progress in defining and beginning to implement institutional management controls (i.e., processes, structures, and tools). However, each report also concluded that much remained to be accomplished relative to the act’s requirements and relevant guidance. Among other things, this included developing component architectures that are aligned with the corporate BEA and ensuring that investment review and approval processes are fully developed and institutionally implemented across all organizational levels.

Notwithstanding this progress on business systems modernization, we previously reported and more recently testified in February 2008 that two items remained to be done before DOD’s overall business transformation efforts, which include business systems modernization, would be on a sustainable path to success. First, DOD had yet to establish a strategic planning process that results in a comprehensive, integrated, and enterprisewide plan or set of plans that would guide transformation. Second, DOD had yet to designate a senior official who could provide full- time attention and oversight to the business transformation effort. Subsequently, the National Defense Authorization Act for Fiscal Year 2008 designated the Deputy Secretary of Defense as the department’s Chief Management Officer (CMO), created a Deputy CMO position, and designated the undersecretaries of each military department as CMOs for their respective departments. The act also required the Secretary of Defense, acting through the CMO, to develop a strategic management plan that, among other things, is to include a detailed description of performance goals and measures for improving and evaluating the overall efficiency and effectiveness of the business operations of the department. According to DOD, steps have been taken and are ongoing to address these provisions.

We also testified in February 2008 that DOD continues to take steps to comply with key business systems modernization legislative requirements, but that much remained to be accomplished before the full intent of this legislation would be achieved. In particular, we stated that DOD’s BEA, while addressing several issues previously reported by us, was still not sufficiently complete to effectively and efficiently guide and constrain business system investments across all levels of the department. Most notably, the BEA did not yet include well-defined architectures for DOD’s components, and DOD’s strategy for “federating” or extending its architecture to the military departments and defense agencies was still evolving and had yet to be implemented. In addition, the scope and content of the department’s ETP still did not address DOD’s complete portfolio of IT investments. We also testified that while the department had established and begun to implement legislatively mandated corporate investment review structures and processes, neither DOD nor the military departments had done so in a manner that was fully consistent with relevant guidance.

DOD Is Continuing to Improve Its Approach to Modernizing Business Systems

DOD continues to take steps to comply with the requirements of the act and to satisfy relevant systems modernization management guidance. In particular, on March 14, 2008, DOD released an update to its BEA (version 5.0) and ETP, and issued its annual report to Congress describing steps that have been taken and are planned relative to the act’s requirements, among other things. Collectively, these steps address several legislative provisions and best practices concerning the BEA, transition plan, budgetary disclosure, and investment review of systems costing in excess of $1 million. However, additional steps are needed to fully comply with the act and relevant guidance. Most notably, the department has yet to extend and evolve its corporate BEA to the department’s component organizations’ (military departments and defense agencies) architectures and fully define IT investment management policies and procedures at the corporate and component levels. BTA officials agree that additional steps are needed to fully implement the act’s requirements and our related recommendations. According to these officials, DOD leadership is committed to fully addressing these areas and efforts are planned and under way to do so.

DOD Continues to Improve Its Corporate BEA, but Component Architectures Remain a Challenge

Among other things, the act requires DOD to develop a BEA that would cover all defense business systems and the functions and activities supported by defense business systems and enable the entire department to (1) comply with all federal accounting, financial management, and reporting requirements, (2) routinely produce timely, accurate, and reliable financial information for management purposes, and (3) include policies, procedures, data standards, and system interface requirements that are to be applied throughout the department. As such, the act provides for an architecture that extends to all defense organizational components. In 2006, the department adopted an incremental and federated approach to developing such an architecture. Under this approach, the department committed to releasing new versions of its BEA every 6 months that would include a corporate BEA that was augmented by a coherent family of component architectures. As we have previously reported, such an approach is consistent with best practices and appropriate given the DOD’s scope and size.

In 2007, we reported that the then current version of the BEA (version 4.1) resolved several of the architecture gaps associated with the prior version and added content proposed by DOD stakeholders, but that gaps still remained. On March 14, 2008, DOD released BEA 5.0 which addresses some of these remaining gaps. For example, it improves the Financial Visibility business enterprise area by expanding the Standard Financial Information Structure data elements (i.e., types of data) associated with information exchanges among operational nodes (e.g., organizational units or system functions) to include data attributes (characteristics of data elements). In addition, the latest version introduces data standards for the Enterprise Funds Distribution initiative. Together, these additions bolster the department’s efforts to standardize financial data across DOD so that information is available to inform corporate decision making.

Version 5.0 of the BEA also addresses, to varying degrees, missing elements, inconsistencies, and usability issues that we previously identified. Examples of these improvements and remaining issues are summarized below.

The latest version includes performance metrics for the business capabilities within enterprise priority areas, including actual performance relative to performance targets that are to be met. For example, it states that 62 percent of DOD assets are now using the Department of the Treasury’s United States Standard General Ledger compliant formats, as compared to a target of 100 percent. Further, this version provides actual baseline performance for operational activities (e.g., “Manage Audit and Oversight of Contractor”). As we previously reported, performance models are an essential part of any architecture because having defined performance baselines to measure actual performance against provides the means for knowing whether the intended mission value to be delivered by each business process is actually being realized.

The latest version includes important “As Is” information (e.g., current capability problems and limitations that enterprise priorities are to address and their root causes) for all six business enterprise priorities. As we previously reported, such “As Is” content is essential for analyzing capability gaps that in turn inform the plan for transitioning from the “As Is” to the “To Be” environments.

The latest version includes 1,201 new business rules. As we previously reported, business rules are important because they explicitly translate business policies and procedures into specific, unambiguous rules that govern what can and cannot be done. As such, they facilitate consistent implementation of policies and procedures. Examples of new business rules are that (1) each request for commercial export of DOD technology must be processed within 30 days of request from the Department of State or the Department of Commerce and (2) DOD must first seek to acquire commercial items before developing military unique material. In addition to adding business rules, Version 5.0 reflects the deletion of 1,046 business rules that were no longer applicable and thus obsolete.

Notwithstanding these additions and deletions, BEA 5.0 still does not provide business rules for all business processes. For example, there are no business rules for the “Perform Acceptance Procedures for Other Goods and Services” business process under the Common Supplier Engagement enterprise priority area. Also, business rules are defined at inconsistent levels of detail. For example, the Travel Authorization business rule states that each travel authorization must be processed in accordance with the Allowance Law, however, it does not identify the specific conditions that must be met. In contrast, the Trial Balance Reporting business rule is more explicitly defined, specifically citing the conditions under which actions are to be taken. Without well-defined business rules, policies and procedures can be implemented inconsistently because they will be interpreted differently by different organizations.

The latest version includes updates on the information that flows among operational nodes (i.e., organizations, business operations, and system elements). Information flows are important because they define what information is needed and where and how the information moves to and from operational entities. In particular, Version 5.0 adds 240 new information exchanges (e.g., Accounts Payable) among business operations and 28 data exchanges (e.g., Acknowledge Inter-governmental Order) among system elements. However, it still does not provide information flows for all organizational units. For example, it does not identify information exchanges among the organizations that support the Human Resources Management enterprise priority area, and continues to lack information flows among DOD corporate and components organizations. Without such information exchanges, a common understanding of the semantic meaning of the information moving among these organizations does not exist. Moreover, Version 5.0 contains information exchanges (e.g., Accounts Payable Account) that are not attached or linked to any operational nodes. Further, this version’s information-related architecture products contain inconsistencies. For example, “Acceptance Results” is identified as a new information exchange in the integrated dictionary, but it is not in the operational information exchange product.

The latest version expands on the operational activities that are or will be performed at each location and by each organization. For example, it now identifies the Defense Logistics Agency as one of the organizations involved in the “Authorize Return or Disposal” activity. However, as was the case with BEA Version 4.1, not all operational activities are assigned to an organization. For example, the “Manage Capabilities Based Acquisition” activity is not assigned. In addition, BEA 5.0 still does not include the roles and responsibilities of organizations performing the same operational activity, which is important because not doing so can result in either duplicative organizational efforts or gaps in activity coverage. Moreover, BEA 5.0 still does not include the Foreign Military Sales operational activity, which affects multiple DOD business missions and organizations.

The latest version continues to lack important security architecture content. For example, while DOD officials told us that the Enterprise Information Environment Mission Area will provide infrastructure information assurance services (e.g., secure, reliable messaging) for business systems and applications, this information is not reflected in the latest version. Also, this version still does not describe relevant information assurance requirements contained in laws, regulations, and policies, or provide a reference to where these requirements are described. Such information is essential to adequately reflect security in the BEA, and thereby ensure that designs for business systems, applications, and services comply with applicable information assurance requirements.

Beyond the above discussed limitations, Version 5.0 also continues to represent only the thin layer of corporate architectural policies, capabilities, rules, and standards that apply DOD-wide (i.e., to all DOD federation members). This means that Version 5.0 appropriately focuses on addressing a limited set of enterprise-level (DOD-wide) priorities, and providing the overarching and common architectural context that the distinct and substantially autonomous member (i.e., component) architectures inherit. However, this also means that Version 5.0 does not provide the total federated family of DOD parent and subsidiary architectures for the business mission area that are needed to comply with the act.

To produce the federated BEA, the BTA released an update to its federation strategy in January 2008. (See fig. 3 for a simplified diagram of DOD’s federated BEA.) In April 2007, we reported on the prior version of this strategy, concluding that while it provided a foundation on which to build and align DOD’s parent BEA with its subsidiary architectures, it lacked sufficient details to permit effective and efficient execution. Accordingly, we made recommendations to improve the strategy.

The updated strategy, along with the associated global information grid (GIG) strategy, partially addresses our recommendations. For example, the strategies now provide high-level roles and responsibilities for federating the architecture and additional definition around the tasks needed to achieve alignment among DOD and component architectures. In particular, the strategy for the business mission area provides for conducting pilot programs across the components to demonstrate the technical feasibility of architecture federation. BTA and CIO officials described the strategy for federating DOD’s architectures as still evolving. They added that lessons learned from the pilots will be used to improve and update the strategies. They also noted that subsequent releases of the corporate BEA will reflect the evolving federation strategy by, for example, defining enforceable interfaces to ensure interoperability and information sharing.

To help assist the department in its BEA federation efforts, we have made a number of recommendations. While DOD agreed with these recommendations, it did not implement one related to its latest annual report. Specifically, we previously recommended that DOD include in its annual report, required under the National Defense Authorization Act for Fiscal Year 2005, the results of its BEA independent verification and validation (IV&V) contractor’s assessment of the completeness, consistency, understandability, and usability of the federated family of architectures. However, its latest annual report does not include this information. According to BTA officials, this is because the contractor’s report was not finalized in time to include the results. While we have yet to receive either the contractor’s statement of work or the results of the contractor’s assessments, BTA officials provided us with a report dated April 11, 2008, that summarizes selected IV&V contractor observations and recommendations relative to the Version 5.0’s ability to provide a foundation for BEA federation. Overall, the summary confirms our findings by stating that while the BEA provides a foundation for federation, much remains to be done before the department will have a complete family of architectures. In this regard, it provides several recommendations, such as having BTA track, measure, and report on the adoption of shared vocabularies and standards within the component architectures. However, the summary does not demonstrate that the IV&V contractor is being used to address the full scope of our recommendation. For example, the summary does not address the extent to which the department’s federated family of architectures, including the related transition plan(s), are complete, consistent, understandable, and useable.

The challenges that the department faces in federating its BEA, and the importance of disclosing to congressional defense committees the state of its federation efforts, are amplified by our recent report on the current state of the military departments’ enterprise architecture programs. Specifically, we reported in May 2008, that none of the three military departments could demonstrate through verifiable documentation that it had established all of the core foundational commitments and capabilities needed to effectively manage the development, maintenance, and implementation of an architecture, although in relative terms the state of the Air Force’s architecture efforts was well ahead of those of the Navy and Army. Examples of their architecture limitations are discussed below.

None of the military departments had fully defined its “As Is” and “To Be” architecture environments and associated transition plans. This is important because without a full understanding of architecture-based capability gaps, the departments would not have an adequate basis for defining and sequencing its ongoing and planned business system investments.

None of the military departments had fully addressed security as part of its respective “As Is” and “To Be” environments. This is important because security is relevant and essential to every aspect of an organization’s operations, and therefore the nature and substance of institutionalized security requirements, controls, and standards should be embedded throughout the architecture, and reflected in each system investment.

None of the military departments was using an IV&V agent to help ensure the quality of its architecture products. IV&V is a proven means for obtaining unbiased insight into such essential architecture qualities as completeness, understandability, usability, and consistency.

None of the military departments had established a committee or group with representation from across the enterprise to direct, oversee, and approve its architecture. This is significant because the architecture is a corporate asset that needs to be enterprisewide in scope and endorsed by senior leadership if it is to be leveraged for optimizing operational and technology change.

None of the military departments could demonstrate that its IT investments were actually in compliance with its architectures. This is relevant because the benefits from using an architecture, such as improved information sharing, increased consolidation, enhanced productivity, and lower costs, cannot be fully realized unless individual investments are actually in compliance with, among other things, architectural rules and standards.

To address these limitations, we have made recommendations aimed at improving the management and content of these architectures. DOD agreed with our recommendations. Until DOD has a well-defined family of architectures for its business mission area, it will not fully satisfy the requirements of the act and it will remain challenged in its ability to effectively manage its business system modernization efforts.

DOD Continues to Expand and Update Its Enterprise Transition Plan, but Important Elements and Component Plans Are Still Missing

Among other things, the act requires DOD to develop an ETP for implementing its BEA that includes listings of the legacy systems that will and will not be part of the target business systems environment and specific time-phased milestones and performance metrics for each business system investment.

In 2007, we reported that the then version of the ETP addressed several of the missing elements that we previously identified relative to the act’s requirements and relevant guidance. However, we also reported that the ETP was limited in several ways. On March 15, 2008, DOD released the latest version of its ETP, which provides required information on 102 programs (systems and initiatives) that are linked to key transformational objectives. For example, it includes specific time-phased milestones for about 90 business system programs and performance metrics for about 75 of these. Further, the latest version of the ETP discusses progress made on business system investments over the last 6 months, as well as descriptions of planned near-term activities (i.e., next 6 months).

The Defense Integrated Military Human Resources System program completed all interface designs required for system deployment to the Army and to defense agencies, and over half of the interface designs required for deployment to the Air Force. It also states that system interface testing and operational testing for the Army deployment will be completed in the next 6 months.

The Contractor Performance Assessment Reporting System was fully implemented following replacement of a proprietary software product with an open source product and rehosting of this product to a new facility. As a result, improvements in system performance, reliability, and security were attained.

This version also partially addresses issues that we identified in our prior report. Examples of improvements and remaining issues are summarized here.

The latest version contains the results of analyses of gaps between its “As Is” and “To Be” architectural environments, in which capability and performance shortfalls are described and investments (such as transformation initiatives and systems) that are to address these shortfalls are identified. It also discusses planned and ongoing gap analyses. For example, it relates the DOD Electronic Mall investment to the Common Supplier Engagement business enterprise priority area and describes how it will address business capability gaps by providing access to off-the-shelf finished goods and services from both commercial and government sources. It also describes how related performance shortfalls will be addressed through shorter logistics response time, improved visibility of sources of supplies, one-stop tracking of order status, and improved ability to shop for best price. As we stated, determining how business capability gaps between the baseline and target architecture are to be addressed for all priority areas is key to the department’s transition plan’s ability to support informed investment selection and control decisions.

The latest version provides a range of information for the 102 systems and initiatives identified, such as 3 years of budget information for 67 of these systems and initiatives. However, as we reported last year, the plan has yet to address our prior finding for including system and budget information for investments by 13 of DOD’s 15 agencies and for eight of its nine combatant commands. At that time, BTA officials stated that information for these defense agencies and combatant commands was excluded because the ETP focused on those business-related organizations having the majority of the tier 1 and 2 business investments, and the majority of the defense agencies and combatant commands do not have investments that meet this threshold criteria. However, not all DOD components have developed subordinate transition plans. For example, we recently reported that only one military department, the Air Force, had developed a transition plan and that this plan was limited because it did not include an analysis of the gap in capabilities between the military departments’ “As Is” and “To Be” environments. This means that, similar to DOD’s federated BEA, a complete family of DOD and component transition plans does not yet exist.

The latest version provides performance measures for both enterprise and component investments (i.e., programs), including key milestones (e.g., initial operating capability). However, it does not include other important information needed to understand the sequencing of these investments. In particular, the planned investments are not sequenced based on a range of important factors cited in federal guidance, such as technology opportunities, marketplace trends, fiscal and budgetary constraints, institutional system development and acquisition capabilities, new and legacy system dependencies and life expectancies, and the projected value of competing investments. While the ETP has begun to incorporate some top-down analysis based on gaps in the business enterprise priorities, the plan continues to be largely based on a bottom-up planning process in which ongoing programs were examined and categorized in the plan around business enterprise priorities. For example, many of these investments are dependent on Net-Centric Enterprise Services (NCES) for its core services, and as such the plans and milestones for each should reflect the incremental capability deployment of NCES. According to the BTA official responsible for the ETP, the investments were sequenced based on only fiscal year budgetary constraints. However, BTA officials said that they intend to depict investment dependencies in future versions of the ETP, especially program-to-program dependencies associated with adoption of a service-oriented architecture approach.

The latest version of the ETP also includes discussion of how the department plans to use enterprise application integration, including plans, methods, and tools for reusing applications that already exist while also adding new applications and databases. However, as we reported last year, this discussion lacks specifics on which investments will reuse which applications.

According to BTA officials, a number of actions are envisioned to address the above cited areas and further improve the ETP, such as adding the results of capability gap analyses for all business priority areas, including tier 1 and 2 programs for all components, and recognizing dependencies among investments. Until the ETP, or a federated family of such plans, either directly or by reference includes relevant information on the full inventory of investments across the department (and does so in a manner that reflects consideration of the range of variables associated with a well- defined transition plan, such as timing dependencies among investments and the department’s capability to manage them), it will not have a sufficient basis for informed investment decision making regarding disposition of the department’s existing inventory of systems or for sequencing the introduction of modernized systems. To help DOD in addressing its transition planning challenges, we have previously made recommendations that the department is in the process of addressing.

DOD’s Fiscal Year 2009 Budget Submission Includes Key Information on Business Systems

Among other things, the act requires DOD’s annual IT budget submission to include key information on each business system for which funding is being requested, such as the system’s designated approval authority and the appropriation type and amount of funds associated with development/modernization and current services (i.e., operation and maintenance).

The department’s fiscal year 2009 budget submission includes a range of information for the approximately 3,000 business system investments for which DOD is requesting funding. Of these, 273 involve modernization/development activities. For each of the 273, the information provided includes the system’s (1) name, (2) approval authority, and (3) appropriation type. The submission also identifies the amount of the fiscal year 2009 request that is for development/modernization versus operations/maintenance. For example, the Army’s General Fund Enterprise Business System, the amount of modernization funds related to “Other Procurement, Army” and “Research, Development, Testing and Evaluation, Army” are identified. For systems in excess of $1 million in modernization funding, the submission also cites its certification status (e.g., approved, approved with conditions, not applicable, and withdrawing) and the DBSMC approval date, where applicable.

DOD and Military Departments Have Partially Established Key Investment Management Structures, but Have Yet to Fully Define Related Policies and Procedures

The National Defense Authorization Act for Fiscal Year 2005 requires DOD to establish business system investment review structures, such as the previously mentioned DBSMC and five IRBs, and processes that are consistent with the investment management provisions of the Clinger- Cohen Act. As we have previously reported, organizations that have satisfied stages 2 and 3 of our ITIM framework have established the investment selection, control, and evaluation structures, and the related policies, procedures, and practices that are consistent with the investment management provisions of the Clinger-Cohen Act.

DOD and the Air Force have established the kind of investment management structures provided for in the act and our ITIM framework. However, the Navy has not. Moreover, neither DOD, the Air Force, nor the Navy have defined the full range of related investment management policies and procedures that our framework identifies as necessary to effectively manage investments as individual business system projects (stage 2) and as portfolios of projects (stage 3). Accordingly, we made recommendations to address the limitations that the department is addressing. Until all of DOD has in place these requisite investment management structures and supporting policies and procedures, the billions of dollars that the department and its components invest annually in business systems will remain at risk.

Investment Management Structures Have Been Partially Established

DOD has partially established the organizational structures that are associated with Stages 2 and 3 of our framework. Specifically, we reported in May 2007 that the department had established an enterprisewide investment board and four subordinate boards, and assigned them responsibility for business systems investment governance, including conducting investment certification and approval reviews and annual reviews as provided for in the act. The enterprisewide board—the DBSMC—is composed of senior executives, such as the Deputy Secretary of Defense and the ASD(NII)/CIO, as provided for in the act. Among other things, the DBSMC is responsible for establishing and implementing policies governing the organization’s investment process and approving lower-level investment board processes and procedures. The subordinate boards include four IRBs that are composed of senior officials representing their respective business areas, including representatives from the combatant commands, defense agencies, military departments, and Joint Chiefs of Staff. Among other things, the IRBs are responsible and accountable for overseeing and controlling certain business system investments, including ensuring compliance and consistency with the BEA. The department has also assigned responsibility to the USD(AT&L) for managing business system portfolio selection criteria.

However, as we reported last year, the department has yet to establish the fifth review board required pursuant to the act, the Enterprise Information Environment Mission Area IRB. According to ASD(NII)/CIO officials, this board has been operating under a draft concept of operations for about 2 years, but has not been chartered because of issues surrounding its authority across IT infrastructure-related investments. However, they stated that a policy is expected to be approved and issued by the end of May 2008 that will, among other things, establish a CIO Enterprise Guidance Board that will meet the act’s requirements for Enterprise Information Environment Mission Area IRB. Specifically, the policy is to provide the Enterprise Guidance Board with DOD-wide oversight of IT investments.

With respect to the military departments’ investment management structures, we reported in October 2007 that the Air Force had established the organizational structures associated with stages 2 and 3 of our framework. Specifically, it has instituted a business systems IRB, called the Senior Working Group, consisting of senior executives from the functional business units, including the Office of the Air Force CIO. This group has been assigned responsibility for business system investment governance, including conducting investment precertification and approval reviews and annual reviews, as required by the act. However, we also reported in October 2007 that the Navy had not established such investment management structures. Specifically, it did not have an enterprisewide IRB, composed of senior executives from its IT and business units, to define and implement a Navy-wide business system governance process. Without such structures, we concluded that the Navy’s ability to ensure that business system investment decisions are made consistently and reflect the needs of the organization is limited. Accordingly we made a recommendation to the Navy for establishing these management structures.

Investment Management Policies and Procedures Are Lacking at Both Corporate and Component Levels

Neither DOD nor the departments of the Air Force and the Navy have defined the full range of policies and procedures needed to effectively support project-level (stage 2) and portfolio-based (stage 3) investment management practices. While the department is in the process of developing a new methodology for managing its business system investments throughout their life cycles that it reports will address this lack of policies and procedures, this new methodology is still in draft, has not been approved, and we have yet to be provided a copy. Until these missing policies and procedures are defined, it is unlikely that the thousands of DOD business system investments will be managed in a consistent, repeatable, and effective manner.

To DOD’s credit, it has defined corporate policies and procedures relative to several key practices in our ITIM framework that are associated with project-level investment management (stage 2). However, it does not have the full range of project-level policies and procedures needed for effective investment management. Specifically, we reported in May 2007 that DOD had satisfied several policy- and procedure-related stage 2 practices, such as requiring that systems support ongoing and future business needs through alignment with the BEA, having procedures for identifying and collecting information about these systems to support DBSMC and IRB investment decision making, and assigning responsibility for ensuring that the information collected about projects meets the needs of DOD’s investment review structures and processes. However, we also reported that it had not, for example, developed policies and procedures outlining how the DBSMC/IRB investment review processes are to be coordinated with other decision-support processes used at DOD, such as the Joint Capabilities Integration and Development System; the Planning, Programming, Budgeting, and Execution process; and the Defense Acquisition System. Without clear linkage among these processes, inconsistent and uninformed decision making may result. Furthermore, without considering component and corporate budget constraints and opportunities, the IRBs risk making investment decisions that do not effectively consider the relative merits of various projects and systems when funding limitations exists.

Other important project-level, as well as portfolio-based, investment management policies and procedures that we reported as lacking include ones that (1) specify how the full range of cost, schedule, and benefit data accessible by the IRBs is to be used in making selection decisions; (2) ensure sufficient oversight and visibility into component-level (e.g., Air Force and Navy) investment management activities, including component reviews of systems in operations and maintenance; (3) define the criteria to be used for making portfolio selection decisions; (4) create the portfolio of business systems investments; and (5) provide for conducting postimplementation reviews of these investments. DOD agreed with our findings and described actions that it planned to take to address our recommendations, including developing a new life cycle management methodology for business systems. In addition, it stated that while its actions would improve the department’s corporate policies and procedures for business system investments, each component is responsible for developing and executing investment management policies and procedures needed to manage its business systems.

In this regard, the military departments also have not developed the full range of related investment management policies and procedures needed to execute the project and portfolio-level practices reflected in our ITIM framework. Specifically, we reported in October 2007 that the state of the Air Force and the Navy’s investment management policies and procedures were similar to that of DOD in that while several of our ITIM framework stage 2 practices were satisfied, others were not, and none of the stage 3 practices were satisfied. For example, both the Air Force and the Navy, to their credit, had developed procedures for identifying and collecting information about their business systems to support investment selection and control, and assigned responsibility for ensuring that the information collected during project identification meets the needs of the investment management process. However, neither the Air Force nor the Navy had fully documented policies and procedures for overseeing the management of business system investments and for developing and managing complete business systems investment portfolio(s). Among other things, they did not have policies and procedures that specify decision-making processes for program oversight and describe how corrective actions should be taken when projects deviate from their project management plans. Without such policies and procedures, we concluded that both are at risk of investing in systems that are duplicative, stovepiped, nonintegrated, and unnecessarily costly to manage, maintain, and operate. To address these areas, we made recommendations aimed at implementing our framework’s stage 2 and 3 practices, and DOD partially agreed with these recommendations.

DOD reports that it has begun to address our investment management findings and recommendations. Specifically, it has drafted and is piloting aspects of (e.g., an Enterprise Risk Assessment Methodology) a new lifecycle management methodology, called the Business Capability Lifecycle (BCL). The annual report states that these pilots have validated the BCL and that interim guidance for major business systems has been developed. However, the new methodology has yet to be approved. Further, BTA officials stated that plans for its finalization and full implementation have been placed on hold until the department has implemented the Chief Management Officer (CMO) provisions of the National Defense Authorization Act for Fiscal Year 2008.

Based on a draft of the BCL and descriptions of it contained in the annual report and briefed to us by BTA officials, this new lifecycle methodology could address some, but not all, of the policy and procedure gaps that we have recently reported. For example, the BCL is to consolidate DOD’s currently distinct and separate system requirements, acquisition, and architectural/investment oversight processes into a single governance process. However, while lack of integration among these separate processes is a limitation that reported with DOD’s business system investment management policies and procedures, this limitation also included lack of integration with DOD’s budgeting process. Unless this new lifecycle methodology incorporates DOD’s funding process, the risk of the respective processes producing inconsistent investment decisions remains.

The following are other examples of investment management policy and procedure limitations cited in our recent reports that the draft of the BCL methodology does not fully address.

The BCL does not apply to programs after they have completed development/modernization activities and are in an operations and maintenance mode, except for certain programs designated as “special interest.” As we recently reported, our ITIM framework provides for including both new system development/acquisition investments and operations and maintenance of existing system investments in the investment management process. According to the department, it plans to examine the applicability of the BCL methodology to systems in operations and maintenance.

The BCL does not address how the full range of cost, schedule, and benefit data is to be used by the IRBs when making their program certification decisions. Without documenting how such boards are to consider cost, schedule, and benefits factors when making these decisions, the department cannot ensure that the boards consistently and objectively select proposals that best meet the department’s needs and priorities.

The BCL does not provide for DOD-level oversight and visibility into component-level investment management activities, including component reviews of systems in operations and maintenance and smaller investments, commonly referred to as tier 4 investments. This is particularly important because, as DOD reports, only 353 of about 3,000 total business systems have completed the IRB certification process and have been approved by the DBSMC. This means that the vast majority of business systems have not come before the IRBs and DBSMC, and thus are reviewed and approved only within the component organizations. Without policies and procedures defining how the DBSMC and IRBs have visibility into and oversight of all business system investments, DOD risks components continuing to invest in systems that will fall short of expectations.

The BCL does not provide for portfolio-based business system investment management. Without defining how projects are to be managed as part of portfolios of related investments, the department will not be able to take advantage of the synergistic benefits to be found among the entire collection of investments, rather than just from the sum of individual investments. Further, adequately documenting both the policies and procedures that provide predictable, repeatable, and reliable investment selection and control and govern how an organization reduces investment risk of failure and provides the basis for having rigor, discipline, and respectability in how investments are selected and controlled across the entire organization. According to the department, as it implements both the CMO provisions of the National Defense Authorization Act for Fiscal Year 2008, and capability portfolio management, the IRB/DBSMC investment management approach is expected to become more portfolio oriented.

In finalizing the BCL, it will be important for DOD to address these gaps in its draft methodology. If it does not, the department will continue to risk selecting and controlling its business system investments in an inconsistent, incomplete, and ad hoc manner, which in turn will reduce the chances that these investments will optimally support mission needs in the most cost-effective manner.

DOD Continues to Certify and Approve Business Systems Cited in the Act

The act specifies two basic requirements that took effect October 1, 2005, relative to DOD’s use of funds for business system modernizations that involve more than $1 million in obligations in any given fiscal year. First, it requires that these modernizations be certified by a designated approval authority as meeting specific criteria. Second, it requires that the DBSMC approve each of these certifications. The act also states that failure to do so before the obligation of funds for any such modernization constitutes a violation of the Anti-deficiency Act.

As we have previously reported, the department has established an approach to meeting the act’s requirements that reflects its philosophy of “tiered accountability.” Under its approach, investment review begins within the military departments and defense agencies and advances through a hierarchy of review and decision-making authorities, depending on the size, nature, and significance of the investment. For those investments that meet the act’s dollar thresholds, this sequence of review and decision making includes component precertification, IRB certification, and DBMSC approval. For those investments that do not, investment decision-making authority remains with the component. This review and decision-making approach has two types of reviews for business systems: certification/approval reviews and annual reviews.

Certification/approval reviews. Certification/approval reviews apply to new modernization projects with total costs over $1 million. These reviews focus on program alignment with the BEA and must be completed before components obligate funds for programs. Tiers 1, 2, and 3 investments in development and modernization are certified at three levels—components precertify, the IRBs certify, and the DBSMC approves. At the component level, program managers prepare, enter, maintain, and update information about their investments in their respective data repositories. Examples of information are regulatory compliance reporting, architectural profile, and requirements for investment certification and annual reviews. According to the process, the component precertification authority is to validate that the system information is complete and accessible on the repository, review system compliance with the BEA, and verify the economic viability analysis. This information is then transferred to DOD’s IT Portfolio Repository. The precertification authority asserts the status and validity of the investment information by submitting a component precertification letter to the appropriate IRB for its review.

At the corporate level, the IRB reviews the pre-certification letter and related material, and if certification is decided, prepares a certification memorandum for the designated certification authority’s signature that documents the IRB’s decisions and any related conditions. The memorandum is forwarded to the DBSMC, which either approves or disapproves the IRB’s decisions and issues a memorandum containing its decisions. If the DBSMC disapproves a system investment, it is up to the component precertification authority to decide whether to resubmit the investment after it has resolved the relevant issues.

Annual reviews. The annual reviews apply to all business system investments and are intended to determine whether the investment is meeting its milestones and addressing its IRB certification conditions. Tiers 1, 2, 3, and 4 business system investments are annually reviewed at two levels—the component and the IRBs. At the component level, program managers update information on all tiers of system investments that are identified in their component’s data repository. For tiers 1 through 3 systems that are in development or being modernized, information is updated on cost, milestones, and risk variances and actions or issues related to certification conditions. The component precertification authority then verifies and submits the information for these business system investments for the IRB in an annual letter. The letter addresses system compliance with the BEA and ETP and includes investment cost, schedule, and performance information.

IRBs annually review tiers 1, 2, and 3 business system development or modernization investments. These reviews focus on program compliance with the BEA, program cost and performance milestones, and progress in meeting certification conditions. IRBs can advise the DBSMC to revoke a certification when the investment has significantly failed to achieve performance commitments (i.e., capabilities and costs). When this occurs, the component must address the IRB’s concerns and resubmit the investment for certification.

Since October 1, 2005 (the effective date of the relevant provision of the act), DOD has continued to certify and approve investments with annual obligations in excess of $1 million. For example, as of March 2007, DOD reported that the DBSMC had approved 285 system investments that had been previously certified by the IRBs. By September 30, 2007, DOD reported that the DBSMC had approved an additional 29 IRB-certified system investments, for a total of 314 approved systems. According to DOD: All 314 systems were certified and approved as meeting the first condition in the act—being in compliance with the BEA—and the 314 systems represent all of the modernization programs meeting the act’s threshold through fiscal year 2007. Collectively, these 314 involved $7.9 billion in modernization funding.

About 60 percent (187) of the 314 were reviewed and precertified within the military departments. More specifically, 69 were pre-certified within the Army, 58 within the Navy, and 60 within the Air Force. The remaining 127 were reviewed and precertified within 1 of 15 defense agencies, including 26 in the Military Health Service, 24 within the Defense Logistics Agency, and 20 in the BTA.

Since September 30, 2007, the IRBs have certified and the DBSMC has approved 39 additional system modernization investments. Moreover, available information from the military departments shows that 35 additional investments have been precertified. Specifically, the Air Force, Navy, and Army, report that 14, 19, and 2 investments, respectively, have been precertified. In addition, both the Air Force and Navy reported that they have reviewed and approved investments that are below the act’s thresholds, and thus do not require IRB certification or DBSMC approval. Specifically, the Air Force reports 46 of these systems have been reviewed and approved, while the Navy reports 4 additional systems reviewed and approved. We have yet to receive comparable information from the Army.

The basis for DOD’s continuing efforts to certify and approve business systems modernization investments as being compliant with the BEA are essentially each individual program’s assertion of compliance. These assertions in turn are largely based on DOD BEA compliance assessment guidance. At the request of the Senate Armed Services Committee, we have ongoing reviews of several major business systems investments that include determining the extent to which these investments have demonstrated compliance with the BEA.

Conclusions

Over the last year, DOD has continued to make important progress in defining and implementing key institutional modernization management controls, but much remains to be accomplished. In particular, the corporate BEA, while continuing to improve, is still missing important content, and it has yet to be federated through development of aligned subordinate architectures for each of the department’s component organizations. Further, while the department has developed a strategy for federating the BEA in this manner, this strategy is still evolving and has yet to be implemented. Compounding this situation are recurring limitations in the ETP, as well as the immaturity of the military service architecture programs, to include their own transition plans. In addition, neither the corporate nor the military departments’ approaches to business systems investment management have all the requisite structures and defined policies and procedures in place to be considered effective investment selection, control, and evaluation mechanisms. These architecture and investment management limitations continue to put billions of dollars spent each year on thousands of business system investments at risk.

Development of a well-defined federated architecture and accompanying transition plans for the business mission area, along with institutionalization of effective business system investment management policies and procedures across all levels of the department, are critically important to addressing the business system modernization high-risk area. Equally, if not more important is for the department to actually implement the architecture and investment management controls on each and every business system investment. While not a guarantee, having an architecture-centric approach to investment management, combined with following the other key system acquisition disciplines that are reflected in our existing recommendations to the department, can be viewed as a recipe for the business systems modernization program’s removal from our high-risk list.

Related to implementing our existing recommendations is the department’s need to keep congressional defense committees fully informed about its progress in federating the DOD corporate BEA, to include the maturity of component organization architecture efforts and the related transition plan(s). In its most recent annual report to congressional defense committees pursuant to the National Defense Authorization Act for Fiscal Year 2005, the department missed an opportunity to do this by not including the results of its IV&V contractor’s assessments of the completeness, consistency, understandability, and usability of the federated family of business mission area architectures, including associated transition plans, as we previously recommended.

Recommendations for Executive Action

Because we have existing recommendations to the Secretary of Defense that address the issues raised in this report and that the department has yet to fully implement, we are not making additional recommendations at this time.

Agency Comments

In comments on a draft of this report, signed by the Deputy Under Secretary of Defense (Business Transformation), the department stated that it appreciated our support in advancing its business transformation efforts. It also provided several technical comments that we have incorporated throughout the report, as appropriate.

We are sending copies of this report to interested congressional committees; the Director, Office of Management and Budget and the Secretary of Defense. Copies of this report will be made available to other interested parties upon request. This report will also be available at no charge on our Web site at http://www.gao.gov.

If you or your staffs have any questions on matters discussed in this report, please contact me at (202) 512-3439 or hiter@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix II.

Appendix I: Objectives, Scope, and Methodology

As agreed with defense congressional committees, our objective was to assess the actions by the Department of Defense (DOD) to comply with the requirements of section 2222 of Title 10, U.S. Code. To address this, we focused on five of the six requirements in section 2222, and related best practices contained in federal guidance, that we identified in our last annual report under the act as not being fully satisfied. Generally, these five requirements are (1) development of a business enterprise architecture (BEA), (2) development of a transition plan for implementing the BEA, (3) inclusion of business systems information in DOD’s budget submission, (4) establishment of business systems investment review processes and structures, and (5) approval of defense business systems investments with obligations in excess of $1 million. (See the background section of this report for additional information on the act’s requirements.) We did not include the sixth requirement because our 2006 annual report under the act shows that it had been satisfied. Our methodology relative to each of the five requirements is as follows: To determine whether the BEA addressed the requirements specified in the act, and related guidance, we analyzed version 5.0 of the BEA, which was released on March 14, 2008, relative to the act’s specific architectural requirements and related guidance that our last annual report under the act identified as not being met. We also reviewed version 5.0 to confirm whether statements made in DOD’s March 15, 2008, annual report about the BEA’s content were accurate. In addition, we reviewed DOD’s Business Mission Area Federation Strategy and Road Map Version 2.0 released in January 2008, comparing the strategy and any associated implementation plans with prior findings and recommendations relative to the content of the strategy. Further, we reviewed the Business Transformation Agency’s report of selected independent verification and validation (IV&V) contractor observations and recommendations relative to the Version 5.0’s ability to provide a foundation for BEA federation, and compared this to our prior finding and recommendation relative to the content of an IV&V review of the BEA. Finally, we reviewed and leveraged the applicable results contained in our recent reports on the military departments’ enterprise architecture programs, on the Air Force and Navy’s investment management processes, and our recent testimony on DOD’s Business Transformation.

To determine whether the enterprise transition plan (ETP) addressed the requirements specified in the act, we reviewed the updated version of the ETP, which was released on March 15, 2008, relative to the act’s specific transition plan requirements and related guidance that our last annual report under the act identified as not being met. We also reviewed the ETP to confirm that statements in DOD’s March 15, 2008, annual report about the content of the ETP were accurate.

To determine whether DOD’s fiscal year 2009 information technology budget submission was prepared in accordance with the criteria set forth in the act, we reviewed and analyzed the department report entitled “Report on Defense Business System Modernization FY 2005 National Defense Authorization Act, Section 332,” dated February 2008 and compared it to the specific requirements in the act.

To determine whether DOD has established investment review structures and processes, we focused on the act’s requirements that our last annual report under the act identified as not being met, obtaining documentation and interviewing cognizant DOD officials about efforts to establish the one IRB specified in the act that we previously reported had yet to be established. We also reviewed and leveraged our recent reports that assessed the department’s, Air Force’s, and Navy’s approaches to managing business system investments.

To determine whether the department was reviewing and approving business system investments exceeding $1 million, we reviewed DOD’s list of business system investments certified by the Investment Review Boards (IRB) and approved by the Defense Business Systems Management Committee (DBSMC). We then compared the detailed information provided with the summary information contained in the department’s March 15, 2008, report to the congressional defense committees to identify any anomalies. We also obtained documentation from the Air Force and the Navy to ascertain the specific actions that were taken (or planned to be taken) in order to perform the annual systems reviews as required pursuant to the act. We requested similar information from representatives of the Army, but did not receive it in time to include in this report.

We did not independently validate the reliability of the cost and budget figures provided by DOD because the specific amounts were not relevant to our findings. We conducted this performance audit at DOD headquarters in Arlington, Virginia, from March 2008 to May 2008, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: GAO Contact and Staff Acknowledgments

Acknowledgments

In addition to the contact person named above, key contributors to this report were Elena Epps, Michael Holland, Tonia Johnson (Assistant Director), Neelaxi Lakhmani, Rebecca LaPaze, Anh Le, and Freda Paintsil.