DOD Business Systems Modernization: Important Progress Made in Establishing Foundational Architecture Products and Investment Management Practices, but Much Work Remains
Released: 2005-11-23T00:00:00

Letter

Background

DOD is a massive and complex organization. In fiscal year 2004, the department reported that its operations involved $1.2 trillion in assets, $1.7 trillion in liabilities, over 3.3 million military and civilian personnel, and over $605 billion in net cost of operations. For fiscal year 2005, the department received appropriations of about $417 billion. The department comprises a wide range of organizations, including the military services and their respective major commands and functional activities, numerous defense agencies and field activities, and various combatant and joint operational commands, which are responsible for military operations for specific geographic regions or theaters of operations.

In support of its military operations, the department performs an assortment of interrelated and interdependent business functions, including logistics management, procurement, health care management, and financial management. Earlier this year, DOD reported that, in order to support these business functions, it relied on about 4,200 business systems, for which the department received approximately $13.3 billion in fiscal year 2005 for operations, maintenance, and modernization. For fiscal year 2006, DOD received approximately $15.5 billion to operate, maintain, and modernize its business systems. As we have previously reported, DOD’s systems environment is overly complex and error prone and is characterized by (1) little standardization across the department, (2) multiple systems performing the same tasks, (3) the same data stored in multiple systems, and (4) the need for manual data entry into multiple systems. In addition, our reports continue to show that the department’s nonintegrated and duplicative systems contribute to fraud, waste, and abuse. Of the 25 areas on GAO’s governmentwide high-risk list, 8 are DOD program areas, and the department shares responsibility for 6 other governmentwide high-risk areas. DOD’s business systems modernization is one of the high-risk areas.

Enterprise Architecture and Information Technology Investment Management Are Critical to Achieving Successful Systems Modernization

Effective use of an enterprise architecture, or a modernization blueprint, is a hallmark of successful public and private organizations. For more than a decade, we have promoted the use of architectures to guide and constrain systems modernization, recognizing them as a crucial means to a challenging goal: agency operational structures that are optimally defined in both the business and technological environments. Congress, the Office of Management and Budget (OMB), and the federal Chief Information Officer (CIO) Council have also recognized the importance of an architecture-centric approach to modernization, and OMB and the CIO Council, in collaboration with us, have issued enterprise architecture guidance. The Clinger-Cohen Act of 1996 mandates that an agency’s CIO develop, maintain, and facilitate the implementation of an IT architecture. Further, the E-Government Act of 2002 requires OMB to oversee the development of enterprise architectures within and across agencies. In addition, we and OMB have issued guidance that, among other things, emphasizes the need for system investments to be consistent with these architectures.

A corporate approach to IT investment management is also characteristic of successful public and private organizations. Recognizing this, Congress developed and enacted the Clinger-Cohen Act in 1996, which requires OMB to establish processes to analyze, track, and evaluate the risks and results of major capital investments in information systems made by executive agencies. In response to the Clinger-Cohen Act and other statutes, OMB developed policy for planning, budgeting, acquisition, and management of federal capital assets and issued guidance. We have also issued guidance in this area, which defines institutional structures, such as investment review boards, and associated processes, such as common investment criteria.

Enterprise Architecture: A Brief Description

An enterprise architecture provides a clear and comprehensive picture of an entity, whether it is an organization (e.g., a federal department) or a functional or mission area that cuts across more than one organization (e.g., financial management). This picture consists of snapshots of both the enterprise’s current or “As Is” environment and its target or “To Be” environment. These snapshots consist of “views,” which are one or more architecture products (e.g., models, diagrams, matrixes, and text) that provide logical or technical representations of the enterprise. The architecture also includes a transition or sequencing plan, which is based on an analysis of the gaps between the “As Is” and “To Be” environments; this plan provides a temporal roadmap for moving between the two environments that incorporates such considerations as technology opportunities, marketplace trends, fiscal and budgetary constraints, institutional system development and acquisition capabilities, new and legacy system dependencies and life expectancies, and the projected value of competing investments.

The suite of products produced for a given entity’s enterprise architecture, including their structure and content, are largely governed by the framework used to develop the architecture. Since the 1980s, various architecture frameworks have emerged and been applied. Appendix III provides a discussion of these various frameworks.

The importance of developing, implementing, and maintaining an enterprise architecture is a basic tenet of both organizational transformation and systems modernization. Managed properly, an enterprise architecture can clarify and help to optimize the interdependencies and relationships among an organization’s business operations and the underlying IT infrastructure and applications that support these operations. To support effective architecture management in the federal government, we have issued architecture management guidance, as has the federal CIO Council and OMB. This guidance recognizes that when an enterprise architecture is employed in concert with other important management controls, such as portfolio-based capital planning and investment control practices, architectures can greatly increase the chances that an organization’s operational and IT environments will be configured to optimize its mission performance. Our experience with federal agencies has shown that investing in IT without defining these investments in the context of an architecture often results in systems that are duplicative, not well integrated, and unnecessarily costly to maintain and interface.

IT Investment Management: A Brief Description

IT investment management is a process for linking IT investment decisions to an organization’s strategic objectives and business plans. Generally, it includes structures (including decision-making bodies known as Investment Review Boards), processes for developing information on investments (such as costs and benefits), and practices to inform management decisions (such as whether a given investment is aligned with an enterprise architecture). The federal approach to IT investment management is based on establishing systematic processes for selecting, controlling, and evaluating investments that provides a systematic way for agencies to minimize risks while maximizing the returns of investments.

During the selection phase, the organization (1) identifies and analyzes each project’s risks and returns before committing significant funds to any project and (2) selects those IT projects that will best support its mission needs.

During the control phase, the organization ensures that, as projects develop and investment expenditures continue, the project is continuing to meet mission needs at the expected levels of cost and risk. If the project is not meeting expectations or if problems have arisen, steps are quickly taken to address the deficiencies.

During the evaluation phase, actual versus expected results are compared once a project has been fully implemented. This is done to (1) assess the project’s impact on mission performance, (2) identify any changes or modifications to the project that may be needed, and (3) revise the investment management process based on lessons learned.

Consistent with our architecture management framework, our investment management framework recognizes the importance of an enterprise architecture as a critical frame of reference for organizations making IT investment decisions, stating that only investments that move the organization toward its target architecture, as defined by its sequencing plan, should be approved, unless a waiver is provided or a decision is made to modify the architecture. Moreover, this framework states that an organization’s policies and procedures should describe the relationship between its architecture and its investment decision-making authority.

Our experience has shown that mature and effective management of IT investments can vastly improve government performance and accountability, and can help to avoid wasteful IT spending and lost opportunities for improving delivery of services to the public.

DOD’s Business Systems Modernization Program History and Structure

The Business Management Modernization Program was established in July 2001 in order to improve the efficiency and effectiveness of DOD’s business operations through, among other things, the development and implementation of an architecture. When the program was initially established, the Secretary assigned oversight responsibility to the Under Secretary of Defense (Comptroller), in coordination with the Under Secretary of Defense for Acquisition, Technology, and Logistics and the Assistant Secretary of Defense (Networks and Information Integration)/Chief Information Officer.

In 2001, the Comptroller established several governance bodies and assigned them responsibilities associated with developing, maintaining, and implementing the architecture. Specifically, the Comptroller established (1) the Executive and Steering Committees—which were made up of senior leaders from across the department—to provide program guidance; (2) a program office to execute daily program activities necessary to develop, maintain, and implement the architecture; and (3) domain owners, who were responsible for achieving business transformation, implementing the architecture, developing and executing the transition plan, and performing portfolio management. In 2003, the Comptroller also established the Domain Owners Integration Team, which comprised various senior executives from each domain and the director of the program office. This team reported to the steering committee and was responsible for facilitating communication and coordination across the domains for program activities, including extending and evolving the architecture.

In 2005, the department revised the program’s governance structure. Program direction and oversight is now provided by the Deputy Secretary through the dual leadership of the Under Secretary of Defense for Acquisition, Technology, and Logistics and the Under Secretary of Defense (Comptroller). In addition, DOD has reassigned responsibility for providing executive leadership for the direction, oversight, and execution of its business transformation and systems modernization efforts to several entities. These entities include the Defense Business Systems Management Committee (DBSMC), which serves as the highest ranking governance body for business systems modernization activities; the Principal Staff Assistants, who serve as the certification authorities for business system investments in their respective core business missions; and the Investment Review Boards, which form the review and decision-making bodies for business system investments in their respective areas of responsibility. Table 2 lists these entities and their roles and responsibilities.

DOD has defined five departmentwide core business missions to be addressed through identification of corporate business needs and analysis of capability gaps. The core business missions transcend DOD’s various functional areas (e.g., planning, budgeting, information technology, procurement, and maintenance) and are intended to be the means through which end-to-end warfighter support is delivered. Responsibility for the core business missions is assigned to specific Principal Staff Assistants. Table 3 provides descriptions of the core business missions and associated responsible parties.

On October 7, 2005, DOD established the Business Transformation Agency (BTA) to advance DOD-wide business transformation efforts, particularly with regard to business systems modernization. The BTA reports directly to the vice chair of the DBMSC. Among other things, the BTA includes a Defense Business Systems Acquisition Executive who is to be responsible for centrally managing 28 DOD-wide business projects, programs, systems, and initiatives. In addition, the BTA is to be responsible for integrating and supporting the work of the Office of the Secretary of Defense Principal Staff Assistants, who include the approval authorities that chair the business system investment review boards. Until a permanent director is named, the Deputy Under Secretary of Defense for Business Transformation and the Deputy Under Secretary of Defense for Financial Management will jointly function as directors and will report to the vice chair of the DBSMC.

According to a program official, the department has spent approximately $440 million on the Business Management Modernization Program since it was established in 2001.

Recent Reviews Have Assessed DOD’s Efforts to Develop, Maintain, and Implement an Architecture

Since 2001, we have regularly reported on DOD’s efforts to develop an architecture and to establish and implement effective investment management structures and processes. Our reports have continued to identify problems and raise concerns about the department’s architecture program, the quality of the architecture and the transition plan, and the lack of an investment management structure and controls to implement the architecture. Our most recent reports, which were issued in the third and fourth quarters of fiscal year 2005, made the following points: DOD had not established effective structures and processes for managing the development of its architecture. For example, the department had yet to finalize, approve, and effectively implement its plan, procedures, and charter governing the configuration management process. In addition, DOD had yet to establish an independent quality assurance function that addressed process standards and program performance.

DOD had not developed a well-defined architecture. The products that it had produced did not provide sufficient content and utility to effectively guide and constrain ongoing and planned systems investments. For example, the latest versions of the architecture did not include products describing the “As Is” business and technology environments. Further, although these versions included products describing the “To Be” environment, the descriptions were inadequate because the descriptions (1) did not have a clearly defined purpose that linked to the goals and objectives of the architecture; (2) were missing important content, such as the actual systems to be developed or acquired to support future business operations and the physical infrastructure needed to support the business systems; and (3) contained products that were neither consistent nor integrated. In short, the “To Be” environment lacked the detail needed to provide DOD with a common vision for defining the transition plan and informing investment decision making.

DOD had not developed a plan for transitioning from the “As Is” to the “To Be” architectural environments. The transition plan is based on an analysis of the gaps between these two environments and serves as an enterprisewide IT capital investment plan and acquisition strategy.

DOD did not have an effective departmentwide management structure for controlling its business investments. Although the department had established organizations to oversee its business system investments, these organizations were unable to do so, because the components controlled budget authority and continued to make their own parochial investment decisions.

DOD had not established common investment criteria for system reviews, and as a result different organizations were using different criteria. DOD also had not conducted a comprehensive review of its ongoing business system investments.

DOD had not included all of the reported systems in its fiscal year 2005 IT budget request. It lacked accurate information on the costs and number of its business systems.

The Under Secretary of Defense (Comptroller) had not certified all systems investments with reported obligations exceeding $1 million, as required by the fiscal year 2003 National Defense Authorization Act. Obligations totaling about $243 million were made for systems modernizations in fiscal year 2004 that were not referred to the DOD Comptroller for the required review.

DOD Has Satisfied Requirements in Its Fiscal Year Authorization Act to Varying Degrees

Section 2222 of Title 10, United States Code, as added by section 332 of the defense authorization act for fiscal year 2005, cites six requirements that DOD is required to meet.Generally, these are as follows: 1. By September 30, 2005, develop a business enterprise architecture that meets certain requirements. 2. By September 30, 2005, develop a transition plan for implementing the architecture that meets certain requirements.

3.

Identify each business system proposed for funding in DOD’s fiscal year 2006 and subsequent budget submissions and identify funds for current services and business systems modernization. 4. Delegate the responsibility for business systems to designated approval authorities within the Office of the Secretary of Defense. 5. By March 15, 2005, require each approval authority to establish a business system investment review process. 6. Effective October 1, 2005, obligate funds for business system modernizations with a total cost exceeding $1 million only after the system is certified by the designated approval authority and the certification is approved by the DBSMC.

DOD has partially satisfied the four legislative provisions relating to architecture development, transition plan development, budgetary disclosure, and investment review; it has satisfied the provision concerning designated approval authorities; and it is in the process of satisfying the provision for systems costing in excess of $1 million. According to DOD, the requirements of each provision will be fully implemented under its incremental approach to developing the architecture and transition plan, and its tiered accountability approach to business system investment management. Until they are, the department’s business systems modernization program will continue to be a high-risk endeavor.

Latest Version of Enterprise Architecture Partially Satisfies Act and Provides a Foundation upon Which to Add Missing Scope and Content

The defense authorization act for fiscal year 2005 requires DOD to develop a business enterprise architecture by September 30, 2005. According to the act, the architecture must satisfy three major requirements:1.

It must include an information infrastructure that, at a minimum, would enable DOD to comply with all federal accounting, financial management, and routinely produce timely, accurate, and reliable financial information for integrate budget, accounting, and program information and systems; provide for the systematic measurement of performance, including the ability to produce timely, relevant, and reliable cost information. 2. The architecture must include policies, procedures, data standards, and system interface requirements that are to be applied uniformly throughout the department. 3. The architecture must be consistent with OMB policies and procedures.

On September 28, 2005, the Acting Deputy Secretary of Defense approved Version 3.0 of the business enterprise architecture. According to DOD, this version is intended to provide a blueprint to help ensure near-term delivery of the right capabilities, resources, and materiel to the warfighter. To do so, this version focused on six business enterprise priorities, which DOD states are short-term objectives to achieve immediate results. These priorities are Personnel Visibility, Acquisition Visibility, Common Supplier Engagement, Materiel Visibility, Real Property Accountability, and Financial Visibility. According to DOD, these priorities will evolve and expand in future versions of the architecture. Table 4 provides a brief description of each of the six business enterprise priorities.

In addition to focusing the scope of Version 3.0 of the architecture on these priorities, the extent to which each priority was to be addressed, according to DOD, was limited to answering four key questions: Who are our people, what are their skills, and where are they located?

Who are our industry partners, and what is the state of our relationship with them?

What assets are we providing to support the warfighter, and where are these assets deployed?

How are we investing our funds to best enable the warfighting mission?

To produce a version of the architecture according to the above scope, DOD created 12 of the 26 recommended products identified in the DOD Architecture Framework (DODAF)—the structural guide that the department has established for developing an architecture—including 7 products that the DODAF designates as essential. Table 5 shows the DODAF products included in the architecture. (See app. IV for a complete list of the DODAF products.)

Version 3.0 of DOD’s business enterprise architecture partially satisfies each of the three major requirements specified in the act.

With respect to the first requirement, regarding an information infrastructure, the act cites four requirements, each of which Version 3.0 partially addresses, as described below.

Comply with federal accounting, financial management, and reporting requirements.

Partial compliance is achieved based on the architecture’s inclusion of the Standard Financial Information Structure (SFIS), which includes a Standard Accounting Classification Structure (SACS) that can allow DOD to standardize financial data elements necessary to support budgeting, accounting, cost/performance management, and external reporting. The SFIS and SACS are based upon mandated requirements defined by external regulatory entities, such as the U.S. Treasury, OMB, the Federal Accounting Standards Advisory Board, and the Joint Financial Management Improvement Program. As a result, SFIS can enable compliance with these entities’ requirements if implemented properly. SFIS, while not complete, has been used to develop and incorporate business rules in the architecture for such areas as managerial cost accounting, general ledger, and federally owned property. Business rules are important because they explicitly translate important business policies and procedures into specific, unambiguous rules that govern what can and cannot be done.

However, the architecture does not provide for compliance with all federal accounting, financial, and reporting requirements. For example, it does not do the following: It does not contain the information needed to achieve compliance with the Department of the Treasury’s United States Standard General Ledger. In particular, the logical data model (OV-7) does not contain all the data elements or attributes that are needed to facilitate information sharing and reconciliation with the Treasury. The architecture also does not include a strategy for achieving compliance with the Treasury’s general ledger. For example, it does not state whether DOD will adopt the Treasury data model or simply map its data model to the one for the Treasury. Program officials agreed and stated that this limitation is being reviewed and may be addressed in Version 3.1 of the architecture.

It does not address the locations where specified activities are to occur and where the systems are to be located. Program officials agreed; however, they stated that the architecture is not intended to include this level of detail because it is capabilities-based rather than solutions-based and that this information will be contained either within the department’s Global Information Grid or individual system programs’ documentation. We disagree with the department’s position that information pertaining to locations is better captured in a solutions-based architecture rather than in the business enterprise architecture. The identification of operationally significant and strategic business locations, as well as the need for a business logistics model, is a generally accepted best practice for defining the business operations. This is because the cost and performance of implemented business operations and technology solutions are affected by where they are located, and thus need to be examined, assessed, and decided in an enterprise context, rather than in a piecemeal systems-specific fashion.

Routinely produce timely, accurate, and reliable financial information for management purposes.

Partial compliance is achieved in light of the financial information that is to be produced through (1) SFIS, which can support data accuracy, reliability, and integrity requirements for budgeting, financial accounting, cost and performance management, and external reporting across DOD, and (2) a “Manage Business Enterprise Reporting” system function, which is intended to support the reporting of financial management and program performance information, including agency financial statements.

However, as previously discussed, SFIS is not complete and has yet to be implemented. Moreover, accurate and reliable information depends, in part, on using standard definitions of key terms in the architecture. The architecture does not include definitions for all such terms. In particular, the department has yet to define all enterprise-level terms, meaning terms relating to information that needs to be aggregated to support DOD-wide reporting. For example, in Version 3.0 of the architecture, terms such as “balance forwarded” and “receipt balances” were not defined in the integrated dictionary, even though these terms were used in process descriptions. In the absence of these definitions, component organizations (military services, defense agencies, and field activities) could continue to use local terms and definitions. Such locally meaningful terms cannot be reliably and accurately aggregated to permit DOD-wide visibility, as defined by the department’s business enterprise priorities. This inability to aggregate information for reporting purposes has historically required the department to produce financial information through inefficient methods (e.g., data calls or data translations), which have proven neither accurate nor timely. Program officials agreed and stated that they are currently working to complete SFIS and that they would continue to incorporate and define terms as appropriate as the architecture is evolved.

Integrate budget, accounting, and program information and systems.

Partial compliance is accomplished through information and systems that are to be integrated using (1) an enterprise-level automated reporting system known as Business Enterprise Information Services (BEIS), which is intended to provide timely, accurate, and reliable business information across the department to support auditable financial statements and provide detailed financial information visibility for management in support of the warfighter, and to integrate budget, accounting, and program information that is widely dispersed among systems and organizations across the department; (2) a generic system entity called “Financial Management System Entity,” which is to roll up component-level systems, or potential systems, that support current or future interface requirements; (3) the “Manage Business Enterprise Reporting” system function, which is to aggregate and distribute information according to requirements; and (4) other architectural elements, such as definitions and standards of data exchanges to ensure that the data can be mutually understood, received, processed, and potentially aggregated and analyzed, as well as some terms used in the architecture.

However, the architecture does not include certain elements: It does not include a fully defined and yet to be implemented SFIS— that is, an SFIS that includes all data exchanges as well as the business rules that are to be automated by SFIS, BEIS, and user activities, and are to be supported by procedure manuals.

It does not include all systems needed to achieve integration, as evidenced by instances in which the architecture provides “placeholders” or generic references for yet to be defined future systems (e.g., Financial Management System Entity). Program officials agreed and stated that these systems would be added as solutions are defined to address identified capability gaps.

Systematic measurement of performance, including the ability to produce timely, relevant, and reliable cost information.

Partial compliance is achieved via identification of operational activities that are to be established to monitor the performance of the DOD business mission area and to develop performance plans that include performance levels, outcomes, and expected risks.

However, the architecture does not do the following: It does not provide for the systematic measurement of performance, because it has not established operationally acceptable performance thresholds for such measures as timeliness, accuracy, and reliability of financial information. These operative thresholds have significant influence on how business process activities are to be organized and controlled. Program officials agreed and stated that this issue is being addressed.

It does not describe the “As Is” business and technology environments needed to conduct the gap analysis that is to show the performance shortfalls to be addressed, and thus it does not provide the underlying basis for the transition plan. Program officials agreed that the architecture does not contain an “As Is” architecture description. They stated that they have nevertheless examined the “As Is” conditions in identifying the “To Be” solutions in the architecture. They also stated that they recognize that these “As Is” conditions are not in the architecture and they have yet to be provided to us, and that they need to link this information to the “To Be” architecture.

With respect to the act’s second requirement, that the architecture includes policies, procedures, data standards, and system interface requirements to be applied departmentwide, Version 3.0 partially complies. In particular, the architecture identifies federal guidance relevant to core business missions, such as the financial management and the human resources missions. In addition, the architecture identifies a specific policy entitled “Supply Chain Materiel Management Policy”—dated April 22, 2004—that is relevant to guiding and controlling the department’s core business mission and business processes for materiel and logistics. Moreover, the architecture identifies conceptual, operational, and automated business rules that can be used to govern the implementation of systems investments in accordance with policies. However, not all relevant policies are included in the architecture. For example, policies governing the development, maintenance, and implementation of the architecture are not included. Program officials agreed, and stated that the decision memorandums that were used to guide the development of Version 3.0 will be formalized as a departmental policy.

In addition, Version 3.0 of the architecture includes a logical data model (OV-7) that contains data entities, attributes, and their relationships and an enterprise Technical Standards Profile (TV-1) that comprises a list of data standards (e.g. Extensible Markup Language 1.0 data exchange standard); however, the architecture does not include a systems standards profile that would ensure data sharing and interoperability among departmentwide business systems. Version 3.0 also identifies some, but not all, system interface requirements. For example, the architecture has yet to identify interface requirements with DOD systems that provide infrastructure services, such as network routing. Program officials acknowledged that the architecture does not include a systems standards profile and all system interface requirements and stated that they will address this in future versions.

With respect to the act’s third requirement, that the architecture be consistent with OMB policies and procedures, Version 3.0 partially complies. According to OMB guidance, an enterprise architecture should describe the “As Is” and “To Be” environments and a transition plan. Further, this guidance requires the architecture to include, among other things, the following: Business processes. The work performed to support the agency’s mission, vision, and performance goals. Agencies must also document change agents, such as legislation or new technologies that will drive architecture changes.

Information flow and relationships. The information used by the agency in its business processes, including where it is used and its movement among locations. These information flows are intended to show what information is needed where and how the information is shared to support mission functions.

Technology infrastructure. The functional characteristics, capabilities, and interconnections of the hardware, software, and telecommunications.

Security architecture. The support provided to secure information, systems, and operations.

Version 3.0 of the architecture includes a “To Be” architecture and a transition plan; however, it does not include an “As Is” architecture, which is essential to performing a gap analysis to identify capability and system performance shortfalls that the transition plan is to address. As previously discussed, program officials agreed and stated that they plan to address this. In addition: Version 3.0 defines some of the business processes at a high level.

However, it does not include all business processes. For example, the architecture does not describe key aspects of the planning, programming, budgeting, and execution processes. In particular, the architecture does not yet define a clear planning process that balances requirements with resources and provides direction for execution.

It includes information flows and relationships.

It does not include a description of the technology infrastructure.

It does not include a security architecture.

Beyond the above described areas in which Version 3.0 of the business enterprise architecture does not fully satisfy the requirements in the fiscal year 2005 defense authorization act, Version 3.0 has other limitations. Specifically: The scope of Version 3.0 is not fully consistent with the scope of the enterprise transition plan. For example, we identified 21 systems in the architecture that are not included in the transition plan’s “Master List of Systems and Initiatives” that support the business enterprise priorities and should therefore be funded. Instead of being on this master list, 19 of these 21 systems are included in the transition plan as part of a master list of “Non-priority DOD programs.” Therefore, the systems identified as targeted solutions in the architecture are not being recognized in the transition plan as systems to be funded to provide the needed business capabilities. The remaining 2 of the 21 systems, “Industry System” and “Unstructured Data Sources,” are not identified at all in the transition plan. As a result, the transition plan does not yet explicitly recognize the need to transition to the capabilities implied by these two systems, or else these systems exceed the scope of the transition plan, the Overview and Summary Information product (AV-1), or both.

In addition, the AV-1 states that the scope of Version 3.0 is limited to the six DOD business enterprise priorities. In contrast, the list of “Non-priority DOD programs” in the transition plan is described as a listing of systems “that are not DOD Enterprise or Component Priority Programs” and thus would not be targeted solutions for the business enterprise priorities. As a result, the stated scope of the AV-1 is narrower than the implied scope of the transition plan.

The transition plan treats certain entities, such as the Financial Management System Entity, as system solutions in the Master List of Systems, whereas Version 3.0 treats these entities as contextual placeholders. This difference is not explained.

Finally, another system (the Expeditionary Combat Support System) is explicitly related to four business enterprise priorities (Financial Visibility, Acquisition Visibility, Materiel Visibility, and Common Supplier Engagement) in the Master List of Systems in the transition plan, but it is not included in the architecture.

Version 3.0 refers to “Recruit Candidate” as a needed business capability, but this capability is not reflected in the transition plan. This is important because needed capabilities in the architecture should be reflected in the transition plan to ensure that they are addressed. As another example, “Access Candidate” is referred to as a needed business capability in the transition plan, but it is defined as an existing operational activity in the architecture. If it is in fact an operational activity, this means that the department plans to invest resources to achieve a business capability to address a performance shortfall that does not exist. Program officials stated that these are errors and that they will be corrected.

Version 3.0 does not explicitly state the time frame covered for the “To Be” environment. Rather, it describes the time frame as being “near-term To Be,” but it does not clearly define what is meant by “near-term,” nor does it link this time frame to the milestones associated with the business enterprise priorities or the capabilities and systems in the transition plan. According to relevant guidance, the “To Be” architecture should be fiscally and technologically achievable, and therefore it should generally project 3 to 5 years into the future to accommodate rapid changes in technologies, changes in mission focus and priorities, and uncertainty in future resource availability. Program officials agreed and stated that they would use “near-term” consistently in future versions of the architecture and transition plan.

Version 3.0 does not represent a fully integrated set of architecture products, although we did find greater product integration than in prior versions of the architecture. Examples of instances in which product integration was not apparent follow.

First, the Operational Event-Trace Description product (OV-6c)— which depicts when activities are to occur within operational processes—includes a process entitled “Send Statements of Accountability or Transactions or Trial Balance to Treasury.” However, the Operational Activity Model (OV-5)—which shows the operational activities (or tasks) that are to occur and the input and output process flows among these activities— identifies no corresponding activity. Instead, the OV-5 has an activity entitled “Perform Treasury Operations,” which has four subactivities, none of which is linked to the above process. Program officials agreed that these were not linked; however, they stated that the “Perform Treasury Operations” activity and its subactivities are not intended to link with the above mentioned process. However, intended linkages are not clear because the architecture does not include a traceability matrix that shows the connection between the two architecture products (OV-6c and OV-5). Program officials have acknowledged the need for greater product integration.

Second, one identified event in the architecture—“triggers the supplier process that provides supplier inventory information to the DOD”—is depicted as two separate events at different levels in the process decomposition. In particular, there are different names for this event on the parent diagrams and the child diagrams, and different templates were used to prepare the diagrams. Program officials agreed that these names differed and stated that this would be addressed.

Third, certain business rules are not explicitly linked to the events included in the architecture description, such as “ENT Post Concurrent Months” and “ENT_Estimate_Receivable.” Program officials stated that the guidelines being used by the department require the business rules to be linked to process steps or decision gateway objects, not events. However, because an event is something that “happens” during the course of a business process, it affects the flow of the process and usually has a cause (trigger) or an effect (result). Therefore, best practices recognize the need to integrate or link the “triggers” that are reflected in the Operational Information Exchange Matrix (OV-3) to both the business rules shown in the Operational Rules Model (OV-6a) and the business events shown in the Operational Event-Trace Description (OV-6c). Program officials stated that they will consider revising their guidelines to link business rules to events.

Fourth, the interface diagram for the Financial Management System Entity (FMSE) does not include 4 of the 21 relevant interfaces identified in the AV-2 product, which is the integrated dictionary. Instead, these four interfaces are shown in other system interface diagrams, which are not linked to the FMSE diagram. Program officials stated that they will address this.

Fifth, the timelines reflected in the transition plan are difficult to map to the “To Be” description, according to DOD’s contractor responsible for verification and validation of the architecture and transition plan.

Sixth, the architecture is not adequately linked to the component architectures and transition plans, although such linkage is particularly important given the department’s newly adopted federated approach to developing and implementing the architecture. According to DOD, a federated architecture is composed of a set of coherent but distinct entity architectures. The members of the federation collaborate to develop an integrated enterprise architecture that conforms to the enterprise view and to the overarching rules of the federation. Program officials agreed and stated that greater levels of integration will be a key goal of future versions of the architecture.

Moreover, while Version 3.0 of the architecture is easier to navigate through than prior versions because of improved product integration, it is still difficult to navigate and use this version, making verification and validation of completeness and correctness unnecessarily time consuming. For example, to trace business rules to their associated events (e.g., the business rule entitled “ENT Post Concurrent Months” to the event “trial balance closing is complete”), we had to first locate and review the description of the business rule, then locate the descriptions of the events by manually searching through numerous process diagrams. This was necessary because the architecture does not include a systematic function that enables the user to list all business rules that are associated with events and all events that are associated with business rules. Such a function is an accepted verification and validation method recommended by industry experts.

DOD and its verification and validation contractor have also identified limitations in Version 3.0 of the architecture, which program officials told us would be addressed in future versions. For example, the architecture does not do the following: It does not explicitly link to the department’s primary non-business enterprise architecture (the Global Information Grid Architecture, which covers the warfighting mission area).

It does not adequately address “net-centricity,” a DOD term that refers to having a robust, globally interconnected network environment (including infrastructure, systems, processes, and people) in which data and services (e.g., security services) are shared “timely and seamlessly” among users, applications, and platforms. According to DOD, the architecture must be improved to better designate enterprise data sources, business services, and IT infrastructure services.

It does not accurately and completely address stakeholder comments and their change requests.

Program officials, including the Director of the Transformation Support Office, the Chief Architect, and the Enterprise Transition Plan Team Lead, stated that the department has taken an incremental approach to developing the business enterprise architecture and meeting the act’s requirements. Accordingly, the Special Assistant to the Deputy Under Secretary of Defense for Business Transformation and contractor officials said that Version 3.0 was appropriately scoped to provide for that content that could be produced in the time available to both lay the foundation for fully meeting the act’s requirements and provide a blueprint for delivering near-term capabilities and systems to meet near-term business enterprise priorities. Because of this, they stated that Version 3.0 fully satisfies the intent of the act.

We support DOD taking an incremental approach to developing the business enterprise architecture, recognizing that adopting such an approach is a best practice that we have advocated. In addition, we believe that Version 3.0 provides a foundation upon which to build a more complete architecture. However, we do not agree that Version 3.0 fully satisfies the requirements in the fiscal year 2005 defense authorization act. Further, the missing scope and content and related shortcomings described above mean that while this version is a reasonable baseline upon which to build, it is not yet a sufficient frame of reference for defining a common vision and the kind of comprehensive transition plan needed to effectively and efficiently guide and constrain system investment decision making.

Transition Plan Partially Satisfies the Act, but Improvements Are Needed

The defense authorization act for fiscal year 2005 requires that DOD develop, by September 30, 2005, a transition plan for implementing its business enterprise architecture, and that this plan meet three requirements. The requirements are that it include an acquisition strategy for new systems that are expected to be needed to complete the defense business enterprise architecture; listings of the legacy systems that will and will not be part of the target business systems environment, and a strategy for making modifications to those systems that will be included; and specific time-phased milestones, performance metrics, and a statement of financial and nonfinancial resource needs.

On September 28, 2005, the Acting Deputy Secretary of Defense approved the transition plan. This plan, as described below, partially satisfies the three requirements.

With respect to the first requirement, concerning an acquisition strategy, the plan does describe a high-level approach for transforming the department’s business operations and systems, and the approach is driven by a set of priorities and a targeted set of business capabilities that are to be provided through the implementation of key programs. In general, the plan includes information (e.g., the lead core business mission, budget information, and milestones) for the 39 transformational initiatives and the 60 business systems that are to be part of the “To Be” architectural environment, including an acquisition strategy for each system.

However, the plan is largely based on a bottom-up planning process in which ongoing programs were examined and categorized in the plan around business enterprise priorities and capabilities, including a determination as to which programs would be designated and managed as DOD-wide programs versus component programs. This bottom-up approach to developing the plan does not explicitly reflect transition planning key practices cited in federal guidance, such as consideration of technology opportunities, marketplace trends, fiscal and budgetary constraints, institutional system development and acquisition capabilities, and new and legacy system dependencies and life expectancies, and the projected value of competing investments. Moreover, it means that the plan is not based on a top-down capability gap analysis between the “As Is” and “To Be” architectures in which capability and performance shortfalls are described, and investments (such as transformation initiatives and systems) that are to address these shortfalls are clearly identified. For example, those programs and systems that need to be acquired, developed, or modified and by when to meet the department’s time frame to have a general ledger capability in fiscal year 2006 or 2007 are not clearly identified. According to DOD, this general ledger capability is to be addressed by systems and initiatives that are spread across various appendixes in the transition plan. However, the transition plan should clearly describe the collective investments, including the components and their respective systems, the specific strategies to be used, and the estimated timelines for completion, to address this capability shortfall. This is not yet the case because for example, the transition plan states that “each component is still identifying the optimal path to achieve the capability to post to a United States Standard General Ledger compliant DOD corporate ledger.”

With respect to the second requirement, about identifying legacy systems that will and will not be part of the “To Be” architectural environment, including modifications to these systems, the plan does show some of the legacy systems that are to be replaced by ongoing programs. For example, it identifies the Defense Cash Accountability System (DCAS) as a target system and listed several legacy systems that would be replaced by DCAS (e.g., the Cash Reconciliation System, the Financial Operations Support system, and the International Balance of Payments system). It also provides a list of legacy systems that will be modified to provide capabilities associated with the target architecture environment, such as the Standard Procurement System and the Navy Marine Corps Intranet.

However, the transition plan does not include a number of elements: It does not include a complete listing of the legacy systems that will not be part of the target architecture. For example, the plan identified 145 legacy systems that would be migrating to the target system Expeditionary Combat Support System (ECSS). However, DOD documentation shows that this system includes over 659 legacy logistics systems and other legacy management information systems. This means that the plan does not account for 514 systems related to the integration and migration of ECSS. Program officials agreed and stated that the 145 systems included account for 90 percent of the Air Force’s Installation and Logistics portfolio. They also said that the Air Force is currently assessing the remaining 514 systems to identify interfaces and to determine duplication, and will update the transition plan to reflect this assessment.

The plan does not include system and budget information for 13 of its 15 defense agencies and for 8 of its 9 combatant commands. Exclusion of the Defense Information Systems Agency is particularly limiting, given that this agency provides IT infrastructure services that business systems will need to use. This omission makes it unclear whether the new business systems will be able to reuse existing components, thereby leveraging established capabilities, or will be allowed to introduce duplicative capabilities. According to program officials, the transition plan excluded information for 13 of the defense agencies and for 8 of its combatant commands because it was focused on the largest business-focused organizations in DOD—those meeting Tier 1 and Tier 2 investment review board certification criteria. They noted that the majority of these organizations do not meet these threshold criteria and therefore were not included in the transition plan.

The plan does not include a complete listing of the legacy systems that will be part of the target architecture, nor explicit strategies for modifying those legacy systems identified in the plan’s system migration diagrams. For example, other DOD documentation shows that ECSS, the Defense Enterprise Accounting Management System, and the Defense Integrated Military Human Resources System (DIMHRS) must interface to provide needed business capabilities. However, the transition plan does not reflect this needed integration or the specific capabilities that will be provided by ECSS. According to the transition plan, these strategies are incorporated in the components’ architectures. However, as we stated in the previous section of this report, the components’ architectures have yet to be linked to the business enterprise architecture. Program officials stated that this issue will be addressed through the department’s tiered accountability approach.

With respect to the third requirement, concerning milestones, performance metrics, and resource needs, the plan includes key milestone dates for the 60 systems identified. For example, September 2006 was given as the milestone date for the Defense Travel System to achieve full operational capability, and performance metrics were cited for some systems; for example, for DIMHRS, the plan cites a metric of reducing manual workarounds for military pay by 90 percent. However, the plan does not show specific dates for terminating or migrating many legacy systems, such as the Cash Reconciliation System and the Financial Operations Support system, and it does not include milestone dates for some ongoing programs, such as the Navy Tactical Command Support System. Further, the plan does not include benefits or measures and metrics focused on mission outcomes for each system that can be linked to the plan’s strategic goals. In addition, although the plan does identify resource needs in terms of funding, these needs are a reflection of the funding needs contained in the fiscal year 2006 budget submission; this submission was approved before the programs included in the transition plan were reevaluated by the DBMSC as to their fit within the “To Be” architectural environment and the reasonableness of their respective plans. According to program officials, this means that the resource needs in the transition plan for some programs are not current.

Beyond the transition plan’s partial compliance with the three requirements in the act, as described above, the plan is also missing relevant context and is not consistent with the architecture in various ways. For example: The plan identifies 60 systems as target systems (e.g., DCAS), but the “To Be” architecture includes only 23 of these systems. Program officials agreed and stated that the other 37 systems are contained within component architectures and transition plans. However, as we previously stated, the component architectures have not been linked to Version 3.0.

The plan identifies 21 enterprise initiatives (e.g., SFIS, Defense Acquisition Management Information Retrieval, and Customer Relationship Management), but only 1 of these—Defense Acquisition Management Information Retrieval—is included in the architecture, and it is shown in the architecture as a system, not an initiative. It is important for the architecture to include these initiatives and their relationships to systems. Program officials agreed and stated that Defense Acquisition Management Information Retrieval will be appropriately reflected as a system in the next version of the plan.

The plan includes a list of 66 systems that are characterized as nonpriority DOD enterprise or component programs that will be part of the target architecture, but the target architecture does not identify all these systems. Further, some systems on the list, such as the Mechanization of Contract Administration Services (MOCAS), are systems that in the past were considered eligible for elimination. Program officials agreed and stated that some of these systems are component-level systems and therefore are reflected within the yet to be linked component architectures and transition plans. With regard to systems that, like MOCAS, are slated for termination, these officials stated that replacement systems for such legacy systems have not yet been identified. Until they are, the legacy systems will continue to be shown as target solutions.

The specific business capabilities to be provided by the system solutions for the six business enterprise priorities have not been completely defined in the plan. For example, the Materiel Visibility business enterprise priority requires additional capabilities related to the supply chain planning process, according to DOD, but these capabilities have yet to be defined in the plan. Program officials stated that these will be addressed in future versions of the architecture and transition plan.

According to program officials, including the Director of the Transformation Support Office, the Chief Architect, and the Enterprise Transition Plan Team Lead, the transition plan is evolving, and any limitations will be addressed in future iterations of the plan. The Special Assistant to the Deputy Under Secretary of Defense for Business Transformation and contractor officials stated that the department has taken an incremental approach to developing a transition plan and that the plan, as constrained by the scope of Version 3.0 of the architecture, satisfies the intent of the act’s requirements.

We support an incremental approach to developing the transition plan, which is a best practice that we have advocated. However, the plan does not fully comply with the act’s requirements. Moreover, it was not derived on the basis of a gap analysis between “As Is” and “To Be” architectures, and it is not of sufficient scope, content, and alignment to effectively and efficiently manage the disposition of the department’s existing inventory of systems or for sequencing the introduction of modernized business operations and supporting systems.

Fiscal Year 2006 IT Budget Submission Includes Some but Not All Information That the Act Specifies

The fiscal year 2005 defense authorization act specifies information that the department is to incorporate in its budget request for fiscal year 2006 and each fiscal year thereafter. Specifically, the act states that each budget request must include information on each defense business system for which funding is being requested; all funds, by appropriation, for each such business system, including funds by appropriation specifically for current services (Operation and Maintenance) and systems modernization; and the designated approval authority for each business system.

DOD’s fiscal year 2006 IT budget submission partially satisfies these three requirements. With regard to the first requirement, to identify each business system for which funding is requested, the fiscal year 2006 budget does not reflect all business systems. Specifically, when DOD submitted its fiscal year 2006 budget submission in February 2005, it did not yet have a comprehensive single inventory of its business systems. As we reported in May 2004, DOD was relying at that time on several separate, inconsistent, and unreconciled databases to establish an inventory of its business and national security systems. Accordingly, we recommended that the department establish a single database for its inventory of business systems. On July 13, 2004, the Assistant Secretary of Defense (Networks and Information Integration)/Chief Information Officer (ASD(NII)/CIO) directed establishment of the DOD Information Technology Portfolio Data Repository (DITPR), and on September 28, 2005, the Deputy Assistant Secretary of Defense (Deputy CIO), issued guidance to begin merging the DOD IT registry into DITPR. According to DOD, all business systems will be entered into DITPR by December 31, 2005. According to DOD, all systems will be entered into DITPR by September 30, 2006. However, the establishment and merger of these repositories had not been completed before the development and submission of the fiscal year 2006 IT budget.

With respect to the fiscal year 2007 and future IT budget submissions, DOD plans to use a separate database, entitled the Select and Native Programming Data Collection System–Information Technology to develop the department’s IT budget submissions. For these future submissions, it will be important for DOD to ensure that this system contains all business systems investments.

The extent to which any of these repositories include all business systems, and thus the extent to which the fiscal year 2006 and future budget submissions will as well, is also a function of whether DOD classifies a given system as a business system or a national security system. We previously reported that DOD reclassified 56 systems in its fiscal year 2005 budget request from business systems to national security systems. The net effect of the reclassification was a decrease of approximately $6 billion in the fiscal year 2005 budget request for business systems and related infrastructure. While some of the reclassifications appeared reasonable, we reported that others were questionable. According to DOD, it is currently reviewing the 56 systems, and it plans to complete these reviews by February 2006 to ensure they are properly classified in the fiscal year 2007 IT budget submission.

Further reclassifications are in the fiscal year 2006 budget submission. Specifically, 13 systems have been reclassified from business systems to national security systems in the fiscal year 2006 submission. In addition, 10 national security systems have been reclassified as business systems in the fiscal year 2006 submission. For example: The Air Force’s Aviation Resource Management System, with a fiscal year 2006 budget of $3.3 million, was reclassified from a business to a national security system. DOD included this system in the department’s original inventory of business systems in April 2003 and also reported it as a business system under the Logistics domain in the fiscal year 2005 IT budget request.

The TRICARE Management Agency’s Medical Readiness Decision Support System, with a fiscal year 2006 budget of $1.3 million, was reclassified from a national security system to a business system.

Identification of each business system is also complicated by the fact that DOD’s definition of a business system, as given in its budget submission, differs from the definition of a business system in the fiscal year 2005 defense authorization act. According to the act, a defense business system is “an information system, other than a national security system, operated by, for, or on behalf of the Department of Defense, including financial systems, mixed systems, financial data feeder systems, and information technology and information assurance infrastructure, used to support business activities.” In contrast, the definition that DOD used as the basis for its fiscal year 2006 IT budget request notes that IT infrastructure and information assurance funding supports both business systems and national security systems. As a result, DOD’s position is that shared IT infrastructure and information assurance funding cannot be classified as related to business systems or to national security systems.

With regard to the second requirement, to identify the type of funding (i.e., appropriation) being requested and whether the funding was for current services or modernization, the fiscal year 2006 budget submission identifies the type of funding (i.e., appropriation) being requested and whether the funding was for current services or modernization. However, a number of systems are assigned to a category designated “All Other.” It is not clear what is included in the budget submission under this category. In the fiscal year 2006 IT budget submission, this category totaled about $1.2 billion, and includes, for example, about $22.6 million for financial management. As we previously reported, the ASD(NII)/CIO and military services’ budget officials told us that the “All Other” category in the IT budget includes system projects that do not have to be identified by name because they fall below the $2 million reporting threshold for budgetary purposes. This budgetary threshold is not consistent with the $1 million threshold that the act requires for modernization review and approval, as discussed later in this report, and thus could affect DOD’s ability to identify all system investments that are subject to the requirements of the act. According to ASD(NII)/CIO officials, the fiscal year 2007 budget submission will identify all business systems for which planned spending is equal to or greater than $1 million.

With respect to the third requirement, to identify the designated approval authority for each system, the fiscal year 2006 IT budget submission does so for most systems. However, the approval authority was not identified for 57 business systems. For example, the Navy’s C2 On-the-Move Network Digital Over-the-Horizon Relay system and the Defense Commissary Agency’s Enterprise Business System had a designated approval authority of “Other.”

DOD officials told us that the department recognizes the need to improve the accuracy of its budget submission to provide better information to both DOD management and the Congress on the department’s business systems.

Full compliance with the act’s requirements relative to budgetary disclosure is an important enabler of informed DOD budgetary decision making and congressional oversight. Lacking such disclosure, whether due to incomplete system repositories or incorrect system classification, hinders the department’s efforts to improve its control and accountability over its business systems investments and constrains the Congress’s ability to effectively monitor and oversee the billions of dollars spent annually to maintain, operate, and modernize the department’s business systems environment.

Act’s Requirement for Delegating IT System Responsibilities and Accountabilities to Designated Approval Authorities Has Been Satisfied

The defense authorization act for fiscal year 2005 directs DOD to put in place a specifically defined structure that is responsible and accountable for controlling business system investments to ensure compliance and consistency with the business enterprise architecture. More specifically, the act directs the Secretary of Defense to delegate responsibility for review, approval, and oversight of the planning, design, acquisition, deployment, operation, maintenance, and modernization of defense business systems to designated approval authorities or “owners” of certain business missions. These are as follows: The Under Secretary of Defense for Acquisition, Technology, and Logistics is to be responsible and accountable for any defense business system the primary purpose of which is to support acquisition, logistics, or installations and environment activities.

The Under Secretary of Defense (Comptroller) is to be responsible and accountable for any defense business system the primary purpose of which is to support financial management activities or strategic planning and budgeting.

The Under Secretary of Defense for Personnel and Readiness is to be responsible and accountable for any defense business system the primary purpose of which is to support human resource management activities.

The Assistant Secretary of Defense for Networks and Information Integration/Chief Information Officer of the Department of Defense is to be responsible and accountable for any defense business system the primary purpose of which is to support information technology infrastructure or information assurance activities.

The Deputy Secretary of Defense or an Under Secretary of Defense, as designated by the Secretary of Defense is to be responsible for any defense business system to support any DOD activity not covered above.

DOD has satisfied this requirement under the act. On March 19, 2005, the Deputy Secretary of Defense issued a memorandum that delegated the authority in accordance with the criteria specified in the act, as described above.

Our research and evaluations, as reflected in the guidance that we have issued, show that clear assignment of senior executive investment management responsibilities and accountabilities is crucial to having an effective institutional approach to IT investment management.

Act’s Requirements for Certain IT Investment Review Structures and Processes Have Been Partially Satisfied

The defense authorization act for fiscal year 2005 also required DOD to establish investment review structures and processes, including a hierarchy of investment review boards, each with representation from across the department, and a standard set of investment review and decision-making criteria for these boards to use to ensure compliance and consistency with the business enterprise architecture. In this regard, the act cites three specific requirements. First, it requires the establishment of the DBSMC for overseeing DOD’s business systems modernization efforts, and it specifically identifies the DOD positions to chair and be members of this committee. Second, it requires each designated approval authority to establish by March 15, 2005, an investment review board for investments falling under that authority’s responsibility. Third, the act requires establishment of an investment review process that includes, among other things, the use of common decision criteria, threshold criteria to ensure appropriate levels of review and accountability, and at least annual reviews of every business system investment.

DOD has partially satisfied this requirement in the act. Among other things, it has done the following.

In February 2005, DOD chartered the DBSMC, identifying it as the highest ranking governance body responsible for overseeing business systems modernization efforts. The DBSMC is responsible for ensuring that DOD improves its management and oversight of the department’s business systems. Consistent with the act, the DBSMC is chaired by the Deputy Secretary of Defense, and its members include those positions specified in the act: namely, the designated approval authorities previously discussed, the secretaries of the military services, and the heads of the defense agencies. The vice-chair of the committee is the Under Secretary of Defense for Acquisition, Technology, and Logistics.

DOD established four investment review boards to improve the control and accountability over business system investments. The four are (1) Financial Management, (2) Human Resources Management, (3) Real Property and Installations Lifecycle Management, and (4) Weapon Systems Lifecycle Management and Materiel Supply and Services Management. Each is chaired by the appropriate approval and certification authority (see previous section) and has DOD-wide representation, including membership from the combatant commands, military services, defense agencies, and the Joint Chiefs of Staff.

On June 2, 2005, the Under Secretary of Defense for Acquisition, Technology, and Logistics issued guidance entitled the Investment Review Process Overview and Concept of Operations for Investment Review Boards. This guidance integrates the policies, specifies responsibilities, and identifies the processes to govern the establishment and operation of investment review boards. Among other things, the guidance provides for these boards to review all business system investments, at least annually, and certify defense business system modernizations costing over $1 million, as required by the act. The guidance also specifies the certification process, including criteria to be used.

On July 15, 2005, the Under Secretary of Defense for Acquisition, Technology, and Logistics issued supplemental guidance and criteria for the components (military services, defense agencies, and DOD field activities) to use in preparing their respective defense business system modernization submissions to the investment review boards.

Overall, DOD’s investment structures and processes employ a concept that it refers to as “tiered accountability.” According to the department, tiered accountability is intended to place more responsibility for the management and oversight of business systems investments with the military services and defense agencies’ leaderships. Accordingly, DOD’s guidance describe a process in which business systems investments must be certified by multiple levels of approval and certification authorities, including the component program manager, the component- level precertification authority, the investment review board certification authority, and the DBSMC. As part of this process, a certification package for each system investment must be submitted to the approval authority, and this package is to include basic system information (e.g., system description and funding), justification as to how the system addresses enterprise-level or component-specific requirements; and analysis demonstrating compliance with the business enterprise architecture. A standard system certification template has been developed for use by all components and decision authorities.

The act designates the ASD(NII)/CIO as one of five designated approval authorities for which an investment review board is to be established. According to the act and the Deputy Secretary’s March 19, 2005, memorandum, the ASD(NII)/CIO is responsible and accountable for any business system the primary purpose of which is to support IT infrastructure or information assurance activities. However, the ASD(NII)/CIO has not established an investment review board. According to DOD officials, a separate investment review board has not been established because the ASD(NII)/CIO does not consider the IT infrastructure, information assurance, and related activities that are under its purview to be business systems. They added that the ASD(NII)/CIO is represented on the other investment review boards and can thus oversee issues related to infrastructure and information assurance at those meetings.

DOD’s not having established this investment review board is one of the reasons that the department’s satisfaction of this requirement in the act is as yet only partial. In addition, a key aspect of the act and DOD’s tiered accountability approach is the effective implementation of the defined structures and processes. It is important that such implementation occurs in a continuous and consistent fashion across the department, as we have previously stated. If it does not, the result could be investment decisions that perpetuate the existence of overly complex, error-prone, nonintegrated system environments and limit introduction of corporate solutions to long-standing business problems.

DOD Is Taking Actions Intended to Satisfy the Act’s Requirement for Reviewing Projects over $1 Million

The defense authorization act for fiscal year 2005 specifies two basic requirements, effective October 1, 2005, for obligation of funds for business system investments costing more than $1 million. First, it requires that these investments be certified by a designated “approval authority” as meeting specific criteria. Second, it requires that the DBSMC approve each certification. The act also states that failure to do so before the obligation of funds constitutes a violation of the Anti-Deficiency Act.

The department has taken a number of actions to comply with these two requirements. As mentioned in the previous section, the department has established an investment review process, and this process requires, among other things, that any defense business system modernization costing more than $1 million obtain component precertification, investment review board approval, approval authority certification, and DBSMC approval. This process, as described in investment review board guidance (including DOD Business Systems Investment Review Proposal Submission Guideline), defines the information that programs are to submit to obtain certification for systems meeting certain thresholds, referred to as tiers. Further, the process states that the component’s precertification authority must certify that the system is not a duplicative effort and that it is compliant with the DOD business enterprise architecture before sending the system’s certification package forward to an investment review board.

The department has identified 210 business system modernizations that meet this $1 million threshold and thus need to be approved by the DBSMC. Of the 210, 166 were approved by the DBSMC before September 30, 2005. The remaining 44 have yet to be approved. This means that under the law, DOD cannot obligate fiscal year 2006 funds for these 44 systems until they receive DBSMC approval. It is important to note, however, that the department can continue to invest in these systems by using funds that are still available from previous fiscal years.

Just as with the identification of business systems in DOD’s IT budget submissions (discussed earlier), the extent to which DOD ultimately complies with the act with regard to obligations costing more than $1 million depends, in part, on the proper classification of systems as business versus national security. The following example illustrates this point.

In its fiscal year 2006 budget, the department is requesting about $167 million for the modernization of the Army’s Global Combat Support System. The system, as we previously reported, was reclassified as a national security system in the fiscal year 2005 budget, even though it was included in the department’s reported inventory of about 4,200 business systems and approved by the DOD Comptroller in January 2004. Also, the DBSMC approved this Army system in September 2005, even though the system remains listed in the fiscal year 2006 IT budget request as a national security system. In contrast, the department is requesting about $31 million for the modernization of the Air Force’s version of this system (Global Combat Support System-Air Force) in its fiscal year 2006 budget. However, this system is not listed as one of the 210 systems requiring DBSMC approval, even though the system was reclassified as a business system in the fiscal year 2006 budget.

Another issue that will affect the degree to which the department complies with the act is whether it relies on system certifications and approvals that preceded the act’s requirements. According to financial management investment review board officials, not all of the financial management systems were reviewed in accordance with the fiscal year 2005 act’s requirements. More specifically, four business systems that had already been reviewed in accordance with the criteria specified in the defense authorization act for fiscal year 2003 were granted DBSMC approval in August 2005 on the basis of this prior approval. Table 6 shows the specific systems, fiscal year 2006 modernization funding, and the date of the previous approval.

However, the act does not provide for DBSMC approval based upon the previous review of a system. The act is specific in terms of what constitutes DBSMC review and approval, and these criteria were not followed for the above four systems. According to financial management investment review board officials, the systems listed in table 6 will go through the current investment review process no later than February 2006.

The department’s actions to review and approve business systems investments can be viewed as work in process. According to DOD, it intends to perform the requisite reviews and approvals of all applicable systems before it obligates fiscal year 2006 funds. If it does, it will have complied with the act.

Conclusions

The defense authorization act for fiscal year 2005 contains provisions aimed at strengthening DOD’s institutional approach to investing in IT business systems. To varying degrees, the department has satisfied six specific requirements in the act, and thus has made important progress in establishing the kind of fundamental management structures and processes that are needed to correct the long-standing and pervasive IT management weaknesses that have led to our designation of DOD business systems modernization as a high-risk program. This progress provides a foundation upon which to build. However, much more remains to be accomplished to fully satisfy the act and address the department’s IT management weaknesses, particularly with regard to sufficiently developing the enterprise architecture and transition plan and ensuring that investment review and approval processes are institutionally implemented. The road map for fully addressing these areas is embedded in our prior recommendations to the department. Therefore, we are not making additional recommendations at this time.

Agency Comments and Our Evaluation

In its written comments on a draft of this report, signed by the Deputy Under Secretary of Defense (Business Transformation) and reprinted in appendix II, the department recognized that our analysis, recommendations, guidance, and educational activities have made us a constructive player in DOD’s business transformation efforts. While not commenting on most of the findings in the report, the department also stated that it disagreed with us on two points—the level of development of an “As Is” architecture and consistency within and between the business enterprise architecture and the transition plan.

With respect to the first point, DOD stated that the sheer size and scope of its business operations makes development of a comprehensive “As Is” architecture an ineffective use of time and resources. Instead, according to DOD, while it understood that there needs to be an “easily traceable direct link” between the results of examining its “As Is” conditions and the “To Be” solutions, it maintained that the results of this “As Is” examination are not required to be in the enterprise architecture itself. According to DOD, such “As Is” related work “is more properly aligned with business process review than architecture management.” Notwithstanding these comments, DOD also stated that it was committed to documenting the “As Is” and “To Be” relationship in an appropriate manner.

We agree that both the “As Is” and the “To Be” architectures need to be documented in an appropriate manner. To date, DOD has yet to document its “As Is” architecture in a manner consistent with best practices and federal guidance, and thus we stand by our previous recommendations concerning development of an “As Is” architecture, and we look forward to DOD fulfilling the commitment it made in its comments to address this void in its business enterprise architecture. In this regard, we also agree that developing what the department termed in its comments as a “comprehensive ‘As Is’ architecture” may not be an effective use of time and resources. Accordingly, our prior recommendations for an “As Is” architecture have neither presumed nor prescribed a specific level of comprehensiveness for this “As Is” description, beyond recognizing that it should be defined in accordance with widely accepted best practices and federal guidance. According to these practices and guidance, it should capture the current inventory of enterprise capabilities (in terms of business processes and performance measures) in sufficient scope and detail to permit meaningful analysis of capability gaps in the “To Be” architecture in those areas of the enterprise that are likely to change during the defined transition period. In addition, it should capture descriptions of the information/data, services/applications, and technology environments currently in use, so that transition planning activities can appropriately take into account and address such things as data redundancies, application duplication, shared services, and infrastructure capacity. Our prior recommendations were, however, clear that these “As Is” descriptions should be part of the enterprise architecture (as opposed to what DOD referred to as a business process review), because including such descriptions is a widely accepted best practice and a condition in federal guidance.

With respect to the second point, DOD stated that great effort was made to integrate the business enterprise architecture and the transition plan and that “virtually all” of our examples demonstrating a lack of integration within and between the business enterprise architecture and the transition plan “would be more accurately described as misunderstandings regarding the scope, purpose or intent of the information presented.” It also stated that it was committed to correcting any integration issues.

We agree that considerable effort was made to integrate architecture products and the architecture with the transition plan, and we acknowledge this in the report by stating that the integration of products in this version of the architecture was an improvement over prior versions. However, because our “misunderstandings” arise directly from ambiguity and inconsistencies in the architecture products and the transition plan that blur their intended meaning, this is clear evidence that a well-defined architecture is needed and that current levels of ambiguity and inconsistency limit the utility and effectiveness of the products as reference tools for guiding and constraining system investment decisions. We agree with DOD that addressing these limitations will create better transformation tools that will benefit all stakeholders, most importantly those within the department.

We are sending copies of this report to interested congressional committees; the Director, Office of Management and Budget; the Secretary of Defense; the Deputy Secretary of Defense; the Under Secretary of Defense for Acquisition, Technology, and Logistics; the Under Secretary of Defense (Comptroller); the Assistant Secretary of Defense (Networks and Information Integration)/Chief Information Officer; the Under Secretary of Defense (Personnel and Readiness); and the Director, Defense Finance and Accounting Service. This report will also be available at no charge on our Web site at http://www.gao.gov.

If you or your staff have any questions on matters discussed in this report, please contact me at (202) 512-3439 or hiter@gao.gov, or McCoy Williams at (202) 512-6906 or williamsM1@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix V.

Objective, Scope, and Methodology

Our objective was to assess the Department of Defense’s (DOD) efforts to comply with the requirements of the defense authorization act for fiscal year 2005. Consistent with the act and as agreed with congressional defense committees’ staffs, we evaluated DOD’s efforts relative to six provisions in the act: (1) development of an enterprise architecture that includes an information infrastructure enabling DOD to support specific capabilities, such as data standards and system interface requirements; (2) development of a transition plan for implementing the enterprise architecture that includes specific elements, such as the acquisition strategy for new systems; (3) inclusion of business system information in DOD’s fiscal year 2006 budget submission; (4) establishment of a business system investment approval and accountability structure; (5) establishment of a business system investment review process; and (6) approval of defense business system investments in excess of $1 million.

To determine whether the architecture addressed the requirements specified in the act, we reviewed Version 3.0 of the business enterprise architecture, which was approved on September 28, 2005. This review included analyzing relevant criteria to identify the important architecture scope and content and comparing Version 3.0 architecture products to determine whether they provided this scope and content. In reviewing the products, we specifically focused on principles, business processes, business rules, and standards (e.g., process and data) because relevant criteria recognize that these are fundamental elements of a well-defined and enforceable architecture. In addition, we focused on consistency and completeness among the architecture products and their content (e.g., operational activities and functions to systems), as well as between the architecture and the transition plan. To do this, we traced linkages between the different architecture products to determine if these linkages had been specifically identified to ensure ease of stakeholder navigation and understanding. We also reviewed the traceability matrix prepared by DOD that documented the mapping of the architecture products to the act and interviewed program officials to obtain an understanding of the methodology used to prepare and validate the information in this matrix. In addition, we interviewed key program officials, including the Special Assistant to Business Transformation, Deputy Under Secretary of Defense (Financial Management), the Director of the Transformation Support Office, the Chief Architect, and the Enterprise Transition Plan Team Lead, to discuss the development and maintenance of the architecture products.

To determine whether the transition plan addresses the requirements specified in the act, we reviewed the transition plan approved on September 28, 2005. This review included determining whether the transition plan included elements specified in the act, such as an acquisition strategy for new systems and a statement of financial and nonfinancial resource needs. We also reviewed the transition plan to ascertain the relationship between the plan and the architecture. We reviewed the traceability matrix prepared by DOD that documented the mapping of the transition plan elements to the act and interviewed program officials to obtain an understanding of the methodology used to prepare and validate the information in this matrix. In addition, we interviewed key program officials, including the Special Assistant to Business Transformation, the Deputy Under Secretary of Defense (Financial Management), the Director of the Transformation Support Office, the Enterprise Transition Plan Team Lead, and the Chief Architect, to discuss the development and maintenance of the plan.

To determine whether DOD’s fiscal year 2006 information technology (IT) budget submission was prepared in accordance with the criteria set forth in the act, we reviewed and analyzed DOD’s approximately $30 billion fiscal year 2006 IT budget request. As part of our analysis, we determined what portion of the IT budget request related to DOD business systems. In addition, we compared the fiscal year 2005 and 2006 IT budget requests to determine the systems that were reclassified from business to national security systems, as well as from national security to business systems. We analyzed the 23 system reclassifications by using information in the IT budget requests and the department’s business system inventory. We also followed up with DOD officials to ascertain the department’s efforts to address our concerns regarding the reclassification of the 56 systems discussed in our April 2005 report. We also reviewed and analyzed the fiscal year 2006 IT budget request to ascertain whether the specific types of funds being requested were explicitly identified and whether an approval authority was designated for each business system.

To determine whether DOD has put in place a specifically defined structure that is responsible and accountable for controlling business systems investments to ensure compliance and consistency with the business enterprise architecture, we reviewed applicable memorandums that had been issued by the department and interviewed cognizant departmental officials.

To determine whether DOD has established investment review structures and processes and issued a standard set of investment review and decision- making criteria, we reviewed applicable policies and procedures issued by the department. In this regard, we reviewed the charter for each of the investment review boards. We also met with representatives from the Financial Management and the Weapon Systems Lifecycle Management and Materiel Supply and Services Management investment review boards to obtain an understanding of the specific roles and responsibilities of the investment review boards. We also obtained an understanding of the tiered accountability approach being followed by the department to help improve its control over business system investments. We also reviewed the department’s May 17, 2005, document entitled “Investment Review Process Overview and Concept of Operations for Investment Review Boards.”

To determine whether the department had established a process for the review of business system modernizations in excess of $1 million, we determined whether the department had identified the business systems that were subject to the $1 million threshold. For the 210 systems that the department identified as subject to the criteria set forth in the act, we reviewed the department’s July 2005 guidance entitled “DOD Business Systems Investment Review Proposal Submission Guideline.” In addition, we met representatives from the Financial Management and Weapon Systems Lifecycle Management and Materiel Supply and Services Management investment review boards to obtain an understanding of how they used the guidance in the review of the systems that they are accountable for.

We did not independently validate the reliability of the cost and budget figures provided by DOD, because the specific amounts were not relevant to our findings.

We conducted our work at DOD headquarters offices in Arlington, Virginia, from August through November 2005 in accordance with U.S. generally accepted government auditing standards.

Comments from the Department of Defense

Summary of Several Architecture Frameworks

Various enterprise architecture frameworks are available for organizations to follow. Although these frameworks differ in their nomenclatures and modeling approaches, they consistently provide for defining an enterprise’s operations in both (1) logical terms, such as interrelated business processes and business rules, information needs and flows, and work locations and users, and (2) technical terms, such as hardware, software, data, communications, and security attributes and performance standards. The frameworks also provide for defining these perspectives for both the enterprise’s current, or “As Is,” environment and its target, or “To Be,” environment, as well as a transition plan for moving from the “As Is” to the “To Be” environment.

For example, John A. Zachman developed a structure or framework for defining and capturing an architecture. This framework provides for six windows from which to view the enterprise, which Zachman terms “perspectives” on how a given entity operates: those of (1) the strategic planner, (2) the system user, (3) the system designer, (4) the system developer, (5) the subcontractor, and (6) the system itself. Zachman also proposed six models that are associated with each of these perspectives; these models describe (1) how the entity operates, (2) what the entity uses to operate, (3) where the entity operates, (4) who operates the entity, (5) when entity operations occur, and (6) why the entity operates. Zachman’s framework provides a conceptual schema that can be used to identify and describe an entity’s existing and planned components and their relationships to one another before beginning the costly and time- consuming efforts associated with developing or transforming the entity.

Since Zachman introduced his framework, a number of other frameworks has been proposed. In August 2003, the department released Version 1.0 of the DOD Architecture Framework (DODAF). The DODAF defines the type and content of the architectural products, as well as the relationships among the products that are needed to produce a useful architecture. (See app. IV for a list of the products prescribed by the DODAF.) Briefly, the framework decomposes an architecture into three primary views: operational, systems, and technical standards (see fig. 1). According to DOD, the three interdependent views are needed to ensure that IT systems support operational needs, and that they are developed and implemented in an interoperable and cost-effective manner.

In September 1999, the federal Chief Information Officer (CIO) Council published the Federal Enterprise Architecture Framework (FEAF), which is intended to provide federal agencies with a common construct on which to base their respective architectures and to facilitate the coordination of common business processes, technology insertion, information flows, and system investments among federal agencies. FEAF describes an approach, including models and definitions, for developing and documenting architecture descriptions for multiorganizational functional segments of the federal government. Similar to most frameworks, FEAF’s proposed models describe an entity’s business, the data necessary to conduct the business, applications to manage the data, and technology to support the applications.

In addition, the Office of Management and Budget (OMB) established the Federal Enterprise Architecture (FEA) Program Management Office to develop a federated enterprise architecture in the context of five “reference models, and a security and privacy profile that overlays the five models.”

The Business Reference Model is intended to describe the federal government’s businesses, independent of the agencies that perform them. This model consists of four business areas: (1) services for citizens, (2) mode of delivery, (3) support delivery of services, and (4) management of government resources. It serves as the foundation for the FEA. OMB expects agencies to use this model, as part of their capital planning and investment control processes, to help identify opportunities to consolidate IT investments across the federal government. Version 2.0 of this model was released in June 2003.

The Performance Reference Model is intended to describe a set of performance measures for major IT initiatives and their contribution to program performance. According to OMB, this model will help agencies produce enhanced performance information; improve the alignment and better articulate the contribution of inputs, such as technology, to outputs and outcomes; and identify improvement opportunities that span traditional organizational boundaries. Version 1.0 of this model was released in September 2003.

The Service Component Reference Model is intended to identify and classify IT service (i.e., application) components that support federal agencies and promote the reuse of components across agencies. This model is intended to provide the foundation for the reuse of applications, application capabilities, components (defined as “a self- contained business process or service with predetermined functionality that may be exposed through a business or technology interface”), and business services. According to OMB, this model is a business-driven, functional framework that classifies service components with respect to how they support business or performance objectives. Version 1.0 of this model was released in June 2003.

The Data Reference Model is intended to describe, at an aggregate level, the types of data and information that support program and business line operations and the relationships among these types. This model is intended to help describe the types of interactions and information exchanges that occur across the federal government. Version 1.0 of this model was released in September 2004.

The Technical Reference Model is intended to describe the standards, specifications, and technologies that collectively support the secure delivery, exchange, and construction of service components. Version 1.1 of this model was released in August 2003.

The Security and Privacy Profile is intended to provide guidance on designing and deploying measures that ensure the protection of information resources. OMB has released Version 1.0 of the profile.

List of the DOD Architecture Framework Products

Executive-level summary information on the scope, purpose, and context of the architecture Architecture data repository with definitions of all terms used in all products High-Level Operational Concept Graphic High-level graphical/textual description of what the architecture is supposed to do, and how it is supposed to do it Graphic depiction of the operational nodes (or organizations) with needlines that indicate a need to exchange information Information exchanged between nodes and the relevant attributes of that exchange Command structure or relationships among human roles, organizations, or organization types that are the key players in an architecture Operations that are normally conducted in the course of achieving a mission or a business goal, such as capabilities, operational activities (or tasks), input and output flows between activities, and input and output flows to/from activities that are outside the scope of the architecture One of three products used to describe operational activity—identifies business rules that constrain operations One of three products used to describe operational activity—identifies business process responses to events One of three products used to describe operational activity—traces actions in a scenario or sequence of events Documentation of the system data requirements and structural business process rules of the operational view Identification of systems nodes, systems, and systems items and their interconnections, within and between nodes Specific communications links or communications networks and the details of their configurations through which systems interface Relationships among systems in a given architecture; can be designed to show relationships of interest (e.g., system-type interfaces, planned versus existing interfaces)

Mapping of relationships between the set of operational activities and the set of system functions applicable to that architecture Characteristics of the system data exchanged between systems Systems Performance Parameters Matrix Quantitative characteristics of systems and systems hardware/software items, their interfaces, and their functions Planned incremental steps toward migrating a suite of systems to a more efficient suite, or toward evolving a current system to a future implementation Emerging technologies and software/hardware products that are expected to be available in a given set of time frames and that will affect future development of the architecture One of three products used to describe system functionality—identifies constraints that are imposed on systems functionality due to some aspect of systems design or implementation One of three products used to describe system functionality—identifies responses of a system to events One of three products used to describe system functionality—lays out the sequence of system data exchanges that occur between systems (external and internal), system functions, or human role for a given scenario Physical implementation of the Logical Data Model entities (e.g., message formats, file structures, and physical schema)

GAO Contact and Staff Acknowledgments

GAO Contact

Acknowledgments

In addition to the contacts named above, key contributors to this report were Cynthia Jackson and Darby Smith, Assistant Directors, and Beatrice Alff, Barbara Collier, Francine DelVecchio, Neelaxi Lakhmani, Anh Le, Mai Nguyen, Tarunkant Mithani, Freda Paintsil, Randolph Tekeley, and William Wadsworth.