Department of Defense: Further Actions Needed to Institutionalize Key Business System Modernization Management Controls
Released: 2011-06-29T00:00:00

Letter

Background

DOD is a massive and complex organization entrusted with more taxpayer dollars than any other federal department or agency. Organizationally, the department includes the Office of the Secretary of Defense, the Joint Chiefs of Staff, the military departments, numerous defense agencies and field activities, and various unified combatant commands that are responsible for either specific geographic regions or specific functions. (See fig. 1 for a simplified depiction of DOD’s organizational structure.)

In support of its military operations, DOD performs an assortment of interrelated and interdependent business functions, such as logistics management, procurement, health care management, and financial management. As we have previously reported, the DOD systems environment that supports these business functions is overly complex and error prone, and is characterized by (1) little standardization across the department, (2) multiple systems performing the same tasks, (3) the same data stored in multiple systems, and (4) the need for data to be entered manually into multiple systems. The department recently requested about $17.3 billion for its business systems environment and IT infrastructure investments for fiscal year 2012. According to the department’s systems inventory, this environment is composed of 2,258 business systems and includes 335 financial management, 709 human resource management, 645 logistics, 243 real property and installation, and 281 weapon acquisition management systems.

DOD currently bears responsibility, in whole or in part, for 14 of the 30 programs across the federal government that we have designated as high risk because they are highly susceptible to fraud, waste, abuse, and mismanagement. Seven of these areas are specific to the department, and seven other high-risk areas are shared with other federal agencies. Collectively, these high-risk areas relate to DOD’s major business operations that are inextricably linked to the department’s ability to perform its overall mission and directly affect the readiness and capabilities of U.S. military forces and can affect the success of a mission. In particular, the department’s nonintegrated and duplicative systems impair its ability to combat fraud, waste, and abuse. As such, DOD’s business systems modernization is one of the high-risk areas and is an essential enabler in addressing many of the department’s other high-risk areas. For example, modernized business systems are integral to the department’s efforts to address its financial, supply chain, and information security management high-risk areas.

Enterprise Architecture and IT Investment Management Controls Are Critical to Achieving Successful Systems Modernization

Effective use of a well-defined enterprise architecture is a hallmark of successful organizations and a basic tenet of organizational transformation and systems modernization. Since the early 1990s, we have promoted federal department and agency enterprise architecture adoption as an essential means to achieving a desired end: having operational and technology environments that maximize institutional mission performance and outcomes. Congress, the Office of Management and Budget (OMB), and the federal Chief Information Officers (CIO) Council have also recognized the importance of an architecture-centric approach to modernization. The Clinger-Cohen Act of 1996, among other things, requires the CIOs of federal departments and agencies to develop, maintain, and facilitate architectures as a means of integrating business processes and agency goals with IT. Further, the E-Government Act of 2002 established the OMB Office of Electronic Government and assigned it, among other things, responsibility for overseeing the development of enterprise architectures within and across agencies. In addition, OMB, the CIO Council, and we have issued guidance that emphasizes the need for system investments to be consistent with these architectures. For example, in April 2003 and in August 2010, we published a framework that emphasizes the importance of having an enterprise architecture as a critical frame of reference for organizations when they are making IT investment decisions. Also, in December 2008, OMB issued guidance that addresses system investment compliance with agency architectures.

A corporate approach to IT investment management is another important characteristic of successful public and private organizations. Recognizing this, the Clinger-Cohen Act requires OMB to establish processes to analyze, track, and evaluate the risks and results of major capital investments in IT systems made by executive agencies. In response to the Clinger-Cohen Act and other statutes, OMB developed policy and issued guidance for planning, budgeting, acquisition, and management of federal capital assets. We have also issued guidance in this area that defines institutional structures (such as investment boards), processes for developing information on investments (such as cost/benefit), and practices to inform management decisions (such as whether a given investment is aligned with an enterprise architecture).

Enterprise Architecture: A Brief Description

An enterprise architecture provides a clear and comprehensive picture of an entity, whether it is an organization (e.g., a federal department or agency) or a functional or mission area that cuts across more than one organization (e.g., financial management). An architecture describes the enterprise in logical terms (such as interrelated business processes and business rules, information needs and flows, and work locations and users) as well as in technical terms (such as hardware, software, data, communications, security attributes, and performance standards). It provides these perspectives both for the enterprise’s current environment and for its target environment, and it provides a transition plan for moving from the current to the target environment. This transition plan provides a temporal road map for moving between the two environments and incorporates considerations such as technology opportunities, marketplace trends, fiscal and budgetary constraints, institutional system development and acquisition capabilities, legacy and new system dependencies and life expectancies, and the projected value of competing investments.

The suite of products adopted for a given entity’s enterprise architecture, including its structure and content, is largely governed by the framework used to develop the architecture. Since the 1980s, various architecture frameworks have been developed, such as John A. Zachman’s “A Framework for Information Systems Architecture,” and the DOD Architecture Framework.

The importance of developing, implementing, and maintaining an enterprise architecture is a basic tenet of both organizational transformation and systems modernization. Managed properly, an enterprise architecture can clarify and help optimize the interdependencies and relationships among an organization’s business operations and the underlying IT infrastructure and applications that support these operations. Moreover, when an enterprise architecture is employed in concert with other important management controls, such as portfolio-based capital planning and investment control practices, the architecture can greatly increase the chances that an organization’s operational and IT environments will be configured to optimize mission performance. The alternative, as our work has shown, is the perpetuation of the kinds of operational environments that burden many agencies today, where a lack of integration among business operations and the IT resources supporting them leads to systems that are duplicative, poorly integrated, and unnecessarily costly to maintain and interface.

In February 2002 and April 2003, we issued versions 1.0 and 1.1 of our Enterprise Architecture Management Maturity Framework; in August 2010, we issued a major revision (version 2.0). The framework provides a standard yet flexible benchmark against which to determine where the enterprise stands in its progress toward the ultimate goal: having a continuously improving enterprise architecture program that can serve as a featured decision support tool when considering and planning large- scale organizational restructuring or transformation initiatives. In addition, it also provides a basis for developing architecture management improvement plans, as well as for measuring, reporting, and overseeing progress in implementing these plans.

Several approaches to structuring enterprise architecture exist and can be applied to the extent that they are relevant and appropriate for a given enterprise. In general, these approaches provide for decomposing an enterprise into its logical parts and architecting each of the parts in relation to enterprisewide needs and the inherent relationships and dependencies that exist among the parts. As such, the approaches are fundamentally aligned and consistent with a number of basic enterprise architecture principles, such as incremental rather than monolithic architecture development and implementation, optimization of the whole rather than optimization of the component parts, and maximization of shared data and services across the component parts rather than duplication. Moreover, these approaches are not mutually exclusive and, in fact, can all be applied to some degree for a given enterprise, depending on the characteristics and circumstances of that enterprise. The approaches, which are briefly described here, are federated, segmented, and service-oriented.

Federated

Under a federated approach, the architecture consists of a family of coherent but distinct member architectures that conform to an overarching corporate or parent architecture. This approach recognizes that each federation member has unique goals and needs as well as common roles and responsibilities with the members above and below it. As such, member architectures (e.g., component, subordinate, or subsidiary architectures) are substantially autonomous, but they also inherit certain rules, policies, procedures, and services from the parent architectures. A federated architecture enables component organization autonomy while ensuring corporate or enterprisewide linkages and alignment where appropriate.

Segmented

A segmented approach to enterprise architecture development and use, like a federated approach, employs a “divide and conquer” methodology in which architecture segments are identified, prioritized, developed, and implemented. In general, segments can be viewed as logical aspects, or “slivers,” of the enterprise that can be architected and pursued as separate initiatives under the overall corporate architecture. As such, the segments serve as a bridge between the corporate frame of reference captured in the enterprise architecture and individual programs within portfolios of system investments. OMB has issued guidance related to segment architectures. As part of its guidance, agencies are to group segments into three categories: core mission areas (e.g., air transportation), business services (e.g., financial management), and enterprise services (e.g., records management).

Service-Oriented

A service-oriented approach to enterprise architecture is intended to identify and promote the shared use of common business capabilities across the enterprise. Under this approach, functions and applications are defined and designed as discrete and reusable capabilities or services that may be under the control of different organizational entities. As such, the capabilities or services need to be, among other things, (1) self-contained, meaning that they do not depend on any other functions or applications to execute a discrete unit of work; (2) published and exposed as self- describing business capabilities that can be accessed and used; and (3) subscribed to via well-defined and standardized interfaces. This approach is intended to reduce redundancy and increase integration, as well as provide the flexibility needed to support a quicker response to changing and evolving business requirements and emerging conditions.

IT Investment Management: A Brief Description

IT investment management is a process for linking investment decisions to an organization’s strategic objectives and business plans that focuses on selecting, controlling, and evaluating investments in a manner that minimizes risks while maximizing the return of investment.

During the selection phase, the organization (1) identifies and analyzes each project’s risks and returns before committing significant funds to any project and (2) selects those IT projects that will best support its mission needs.

During the control phase, the organization ensures that, as projects develop and investment expenditures continue, the projects meet mission needs at the expected levels of cost and risk. If the project is not meeting expectations, or if problems arise, steps are quickly taken to address the deficiencies.

During the evaluation phase, actual versus expected results are compared once a project has been fully implemented. This is done to (1) assess the project’s impact on mission performance, (2) identify any changes or modifications to the project that may be needed, and (3) revise the investment management process based on lessons learned.

Consistent with this guidance, our IT Investment Management (ITIM) framework consists of five progressive stages of maturity for any given agency relative to selecting, controlling, and evaluating its investment management capabilities. (See fig. 2 for the five ITIM stages of maturity.) The overriding purpose of the framework is to encourage investment selection and control and to evaluate processes that promote business value and mission performance, reduce risk, and increase accountability and transparency. We have used the framework in many of our evaluations, and a number of agencies have adopted it.

In our ITIM framework, with the exception of the first stage, each maturity stage is composed of “critical processes” that must be implemented and institutionalized in order for the organization to achieve that stage. Each ITIM critical process consists of “key practices” (organizational structures, policies, and procedures) that must be executed to implement the critical process. Our research shows that agency efforts to improve investment management capabilities should focus on implementing all lower-stage practices before addressing the higher-stage practices.

Stage 2 critical processes lay the foundation by establishing successful, predictable, and repeatable investment control processes at the project level. Stage 3 is where the agency moves from project-centric processes to portfolio-based processes and evaluates potential investments according to how well they support the agency’s missions, strategies, and goals. Organizations implementing these Stage 2 and 3 practices have in place selection, control, and evaluation processes that are consistent with the Clinger-Cohen Act. Stages 4 and 5 require the use of evaluation techniques to continuously improve both investment processes and portfolios in order to better achieve strategic outcomes.

Our research shows that agency efforts to improve investment management capabilities should focus on implementing all lower-stage practices before addressing the higher-stage practices and therefore our reviews tend to focus on Stage 2 and Stage 3 critical processes. Specifically, within Stage 2, there are five critical processes and nine associated key practices (known as organizational commitments) that call for policies and procedures associated with effective project-level management. These are shown in table 1.

Within Stage 3, there are four critical processes and five associated key practices (known as organizational commitments) that call for policies and procedures associated with effective portfolio-based investment management. These are shown in table 2.

DOD’s Institutional Approach to Business Systems Modernization

The NDAA for Fiscal Year 2008 designated the Deputy Secretary of Defense position as the Chief Management Officer for DOD and created a deputy position to assist the Chief Management Officer. The Chief Management Officer’s responsibilities include developing and maintaining a departmentwide strategic plan for business reform and establishing performance goals and measures for improving and evaluating overall economy, efficiency, and effectiveness, and monitoring and measuring the progress of the department. The Deputy Chief Management Officer’s responsibilities include recommending to the Chief Management Officer methodologies and measurement criteria to better synchronize, integrate, and coordinate the business operations to ensure alignment in support of the warfighting mission. The Business Transformation Agency supports the Deputy Chief Management Officer in leading and coordinating business transformation efforts across the department. This includes maintaining and updating the department’s enterprise architecture for its business mission area.

The Chief Management Officer and Deputy Chief Management Officer are to interact with several entities to guide the direction, oversight, and execution of DOD’s business transformation efforts, which include business systems modernization. These entities include the Defense Business Systems Management Committee, which serves as the highest- ranking investment review and decisionmaking body for business systems modernization activities and is chaired by the Deputy Secretary of Defense; the principal staff assistants, who serve as the certification authorities for business system modernizations in their respective core business missions; the investment review boards (IRB), which are chaired by the certifying authorities and form the review and decision- making bodies for business system investments in their respective areas of responsibility; and the Business Transformation Agency, which is responsible for supporting the IRBs and for leading and coordinating business transformation efforts across the department. In August 2010 and in January 2011, the Secretary of Defense announced the plans to disestablish the Business Transformation Agency and the Office of the Assistant Secretary of Defense for Networks and Information Integration/Department of Defense CIO (ASD(NII)/DOD CIO) (who is a member of the Defense Business Systems Management Committee), respectively. According to DOD officials, the mission of the Office of the Deputy Chief Management Officer duplicates many of the Business Transformation Agency functions. They added that rather than lead in the development of better business practices, the agency’s prime focus has changed to the management of several relatively small business systems and providing direct support to the Deputy Chief Management Officer on various policy issues. As a result, according to these officials, the narrower function does not justify continuing the Business Transformation Agency as a stand-alone defense agency.

The Secretary of Defense also directed that the remaining functions of the Business Transformation Agency be reviewed and transferred to other organizations in DOD, as appropriate and that the disestablishment of the agency should be no later than June 30, 2011. However, as of June 2011, these implementation decisions had yet to be made and both the Business Transformation Agency and the Office of the ASD (NII)/DOD CIO organizations were still in operation.

Table 3 lists governance entities and provides greater detail on their roles, responsibilities, and composition.

Overview of DOD’s Tiered Accountability for Business Systems Modernization

Since 2005, DOD has employed a “tiered accountability” approach to business systems modernization. Under this approach, responsibility and accountability for business architectures and systems investment management are assigned to different levels in the organization. For example, the Business Transformation Agency is responsible for developing the corporate business enterprise architecture (i.e., the thin layer of DOD-wide policies, capabilities, standards, and rules) and the associated enterprise transition plan. Each component is responsible for defining a component-level architecture and transition plan associated with its own tiers of responsibility and for doing so in a manner that is aligned with (i.e., does not violate) the corporate business enterprise architecture. Similarly, program managers are responsible for developing program-level architectures and plans and for ensuring alignment with the architectures and transition plans above them. This concept is to allow for autonomy while also ensuring linkages and alignment from the program level through the component level to the corporate level. Table 4 describes the four investment tiers and identifies the associated reviewing and approving entities.

Consistent with the tiered accountability approach, the Fiscal Year 2008 NDAA required the secretaries of the military departments to designate the department under secretaries as chief management officers with primary responsibility for business operations. Moreover, the Fiscal Year 2009 Duncan Hunter NDAA required the military departments to establish business transformation offices to assist their chief management officers in the development of comprehensive business transformation plans. In response, all of the military departments have designated their respective Under Secretaries as the chief management officers.

We reported in January 2011 that DOD and the military departments had made limited progress in developing business transformation plans that are supported by a strategic planning process and would enable them to align goals and planning efforts and to measure progress. Specifically, we reported that DOD had not set up internal mechanisms, such as procedures and milestones, by which it can reach consensus with the military departments and others on priorities, synchronize the development of plans with each other and the budget process, and guide efforts to monitor progress and take corrective action. Therefore, while the military departments were in varying stages of developing business transformation plans, it was unclear to what extent the business transformation priorities for the military departments will be aligned with the priorities identified in DOD’s Strategic Management Plan or how these business transformation priorities will influence the department’s budget requests. Accordingly, we made recommendations to enhance DOD’s ability to set strategic direction for its business transformation efforts, and better align and institutionalize its efforts to develop and implement plans and measure progress against established goals. DOD partially agreed with the recommendations.

DOD’s Approach to Certifying Business System Investments

DOD has a two-stage process to select investments, which it refers to as certification and is a key step in its IT investment process that DOD has aimed to model after GAO’s ITIM framework. The first stage involves selection of systems using the Joint Capabilities Integration and Development System, the Defense Acquisition System, and the Planning, Programming, Budgeting, and Execution management systems. At this level, proposals and alternatives are viewed and prioritized for system selection. The second stage of selection involves (1) certifying and approving Tiers 1 through 3 investments and (2) elevating certain component investments to an enterprisewide status. More recently, DOD developed Business Capability Lifecycle guidance, dated November 2010, intended to streamline business system acquisition, and investment management processes to better guide and constrain departmentwide systems modernizations.

For certification, the IRBs rely on documentation submitted by appropriate chief management officers for the military departments and precertification authorities for the defense agencies. This documentation includes but is not limited to, a memorandum asserting that an investment is compliant with the business enterprise architecture; an economic viability analysis, which addresses the investment’s cost and benefits or cost effectiveness; a business process reengineering determination, which identifies the investment’s business process weaknesses, gaps, and opportunities for process improvement; and a certification dashboard, which includes cost and schedule status information. DOD guidance also gives the boards broad authority in their certification reviews and actions, thus allowing each board to review and consider whatever investment- related information that it deems appropriate. Moreover, Business Transformation Agency and IRB officials told us that a board is not limited in the conditions it can place on a program. After an IRB review, the Defense Business Systems Management Committee will be notified of all certification decisions and may elect to approve or disapprove a certification. If a certification is approved, the committee will sign an approval memo that warrants the ability to obligate the related funding for the investment.

Under DOD’s approach, there are four types of certification actions: Certify: An IRB certifies the modernization as fully meeting criteria defined in the act and IRB investment review guidance, such as compliance with the business enterprise architecture and the extent to which the investment is consistent with component and department IT investment portfolios, which are asserted by the component Chief Management Officer.

Certify with conditions: An IRB certifies the modernization with the understanding that it will address specific investment review board- imposed conditions. For example, the Army’s General Fund Enterprise Business System was certified with a condition to develop a plan for complying with the data standards of DOD’s Item Unique Identifier Registry.

Recertify: An IRB certifies the obligation of additional modernization funds for a previously-certified modernization investment. For example, the Air Force’s Defense Enterprise Accounting and Management System was recertified in September 2010 for $26.7 million to be spent across fiscal years 2010 and 2011. This recertification was in addition to the approximately $22.7 million previously certified in December 2009. In addition, a program must request IRB recertification if the program plans to redistribute previously approved modernization funds among multiple fiscal years and this redistribution will result in the funding for any given fiscal year exceeding the previously approved amount by 10 percent or more.

Decertify: An IRB may decertify or reduce the amount of modernization funds available to an investment when (1) a component reduces funding for a modernization by more than 10 percent of the originally certified amount, (2) the period of certification for a modernization is shortened, or (3) the entire amount of funding is not to be obligated as previously certified. For example, the Special Operations Command’s Special Operations Resource Business Information System had about $4.59 million decertified because funding was reduced by more than 10 percent of the originally certified amount. An investment review board may also decertify a modernization after development has been terminated. For example, DOD reported that approximately $2.77 million in research, development, test and evaluation funding for the Army’s Wounded Warrior Accountability System was decertified because it was determined that the system provided a capability that could be better utilized under another system.

Summary of Fiscal Year 2005 NDAA Requirements

Congress included provisions in the Fiscal Year 2005 NDAA that are aimed at ensuring DOD’s development of a well-defined business enterprise architecture and associated enterprise transition plan, as well as the establishment and implementation of effective investment management structures and processes. According to the act, DOD is required to develop a business enterprise architecture and an enterprise transition plan for implementing the architecture, identify each business system proposed for funding in DOD’s fiscal year budget submissions, delegate the responsibility for business systems to designated approval authorities within the Office of the Secretary of Defense, require each approval authority to establish investment review structures and processes, and effective October 1, 2005, not obligate appropriated funds for a defense business system modernization with a total cost of more than $1 million unless the approval authority certifies that the business system modernization meets several conditions.

The Fiscal Year 2005 NDAA also requires that the Secretary of Defense annually submit to the congressional defense committees a report on the department’s compliance with the above provisions.

Recent GAO Reviews of DOD’s Business Systems Modernization

Between 2005 and 2008, we reported that DOD had taken steps to comply with key requirements of the Fiscal Year 2005 NDAA relative to architecture development, transition plan development, budgetary disclosure, and investment review, and to satisfy relevant systems modernization management guidance; however, each report also concluded that much remained to be accomplished relative to the act’s requirements and relevant guidance. We also reported that DOD had fully satisfied the requirement concerning designated approval authorities and continued to certify and approve modernizations costing more than $1 million. We concluded that the department had made progress in defining and beginning to implement institutional management controls (i.e., processes, structures, and tools). We reiterated existing recommendations to address each of the areas.

In May 2008, we reported that progress in establishing corporate management controls needed to be replicated within the military departments. For example, we reported that the military departments did not yet have mature enterprise architecture programs. We also reported that they had yet to fully establish key investment review structures and had yet to define related policies and procedures for effectively performing both project-level and portfolio-based investment management. Because we had previously made recommendations to DOD aimed at putting in place the management controls needed to fully comply with the act and related federal guidance, we did not make additional recommendations.

In May 2009, we reported that the pace of DOD’s efforts in defining and implementing key institutional modernization management controls had slowed compared with progress made in each of the last four years, leaving much to be accomplished to fully implement the act’s requirements and related guidance. For example: The corporate business enterprise architecture had yet to be extended (i.e., federated) to the entire family of business mission area architectures, including using an independent verification and validation (IV&V) agent to assess the components’ subsidiary architectures and federation efforts.

The fiscal year 2009 budget submission included some, but omitted other key information about business system investments, in part because of the lack of a reliable, comprehensive inventory of all defense business systems.

The business system information used to support the development of the transition plan and DOD’s budget requests, as well as certification and annual reviews, was of questionable reliability.

DOD and the military departments had yet to fully define key practices (i.e., policies and procedures) related to effectively performing both project-level (Stage 2) and portfolio-based (Stage 3) investment management as called for in the ITIM. For example, of the nine Stage 2 key practices and five Stage 3 key practices, DOD had defined four and one, respectively, while Air Force had defined three and one, respectively. Subsequent to our reports, Army reported that it had (as of May 2009) efforts planned and under way to develop effective investment management processes, but the efforts did not fully satisfy any key practices at the time.

Business system investments costing more than $1 million continue to be certified and approved, but these decisions were not always based on complete information.

Accordingly, we reiterated existing recommendations to address each of these areas and further recommended that DOD, among other things, improve the quality of investment-related information. DOD partially agreed with our recommendations and described actions being planned or under way to address them. DOD is currently in the process of addressing these recommendations.

In May 2010, we reported that the scope and completeness of key information provided in the report were limited. Specifically, the report omitted information on the number of business system investment certification actions taken during fiscal year 2009 and did not include performance measures, such as measures of progress against program cost, capability, and benefit commitments. Further, we concluded that certification and approval decisions may not be sufficiently justified because investments were certified and approved without conditions even though our prior reports had identified program weaknesses that were unresolved at the time of certification and approval. Accordingly, we recommended that DOD ensure that the scope and content of future annual reports to Congress on compliance with section 332 of the NDAA for fiscal year 2005 be expanded to include cost, capability, and benefits performance measures for each business system modernization investment and actual performance against these measures as well as all certification actions on its business system modernization investments that were taken in the previous year by the department. In addition, we recommended that DOD guidance be revised to include provisions that require investment review board certification submissions to disclose program weaknesses raised by us and the status of actions to address our recommendations to correct the weaknesses to ensure that investment review board certification actions are better informed and justified. DOD agreed with our recommendations.

DOD Continues to Strengthen Management of Its Business Systems Modernization, but Long-standing Challenges Remain

DOD continues to take steps to comply with the provisions of the Fiscal Year 2005 NDAA and to satisfy relevant system modernization management guidance. In particular, DOD released its fiscal year 2011 enterprise transition plan in December 2010, followed by an update to its business enterprise architecture (version 8.0) in March 2011, and its annual report to Congress in May 2011 describing steps that have been taken and are planned relative to the act’s requirements. Collectively, these steps address several statutory provisions and best practices concerning the business enterprise architecture, transition plan, budgetary disclosure, and investment review of systems costing in excess of $1 million. However, challenges that we identified in prior years still need to be addressed in order for the department to be in full compliance with guidance and NDAA requirements. Most notably, the department has yet to extend and evolve its business enterprise architecture to the military departments’ architectures and to fully define IT investment management policies and procedures at the DOD enterprise and component levels. DOD officials agreed that additional steps are needed. They said that the lack of progress is due in part to the uncertainty and pending decisions surrounding the roles and responsibilities of key organizations and senior leadership positions, such as the Business Transformation Agency and ASD(NII)/DOD CIO. However, until DOD fully implements these long- standing institutional modernization management controls required by the act, addressed in GAO recommendations, and otherwise embodied in relevant guidance, its business systems modernization will likely remain a high-risk program. As a result, it is important that the department act quickly to resolve pending decisions about roles and responsibilities.

Adopting a Federated Approach to Architecture Continues to Be a Challenge with Much Remaining to Be Accomplished at the Component Level

Among other things, the act requires DOD to develop a business enterprise architecture to cover all defense business systems and their related functions and activities. According to the act, the architecture should extend to all defense organizational components. In 2006, the department adopted an incremental and federated approach to developing such an architecture. Under this approach, the department releases new architecture versions every year that include a corporate business enterprise architecture that is to be augmented by a coherent family of component architectures. As we have previously reported, such an approach is consistent with best practices and appropriate given DOD’s scope and size.

On March 18, 2011, DOD released its business enterprise architecture version 8.0, which focuses on improving the department’s ability to manage business operations from an end-to-end perspective. This version continues to represent the thin layer of corporate architectural policies, capabilities, rules, and standards that apply DOD-wide (i.e., to all DOD federation members). This means that version 8.0 appropriately focuses on addressing a limited set of enterprise-level (DOD-wide) priorities and provides the overarching and common architectural context that the distinct and autonomous member (i.e., component) architectures inherit. Nevertheless, this also means that version 8.0 does not provide the total federated family of DOD parent and subsidiary architectures for the business mission area. Having such an architecture is dependent on each military department (Air Force, Army, and Navy) having the capability to manage its enterprise architecture and develop the necessary content.

To assist DOD in its architecture federation efforts, we have previously made a number of recommendations. Specifically, in May 2007, we recommended that the department include in its annual report, required under the act, the results of its IV&V contractor’s assessment of the completeness, consistency, understandability, and usability of the federated family of architectures. While DOD agreed with the recommendation, none of its annual reports since 2007, including its latest annual report (issued in May 2011), have included this information. According to Business Transformation Agency officials, IV&V activities have focused on the corporate business enterprise architecture and not on the entire federated family of architectures. Business Transformation Agency officials provided us with a report dated March 25, 2011, which summarizes selected IV&V contractor observations and recommendations relative to version 8.0’s ability to provide a foundation for business enterprise architecture federation. Overall, the summary confirms our findings by stating that, while there has been previous work to develop a business enterprise architecture federation plan, the execution of the federation has yet to be completed. According to Business Transformation Agency officials, the current requirement under the IV&V contract is to provide analysis of the business enterprise architecture. There are no plans for reviewing the military departments’ enterprise architectures.

The challenges that the department faces in federating its architecture and the importance of disclosing to congressional defense committees the state of its federation efforts are amplified by our prior report on the state of the military departments’ enterprise architecture programs. Specifically, we reported in May 2008 that none of the three military departments could demonstrate through verifiable documentation that it had established all of the core foundational commitments and capabilities needed to effectively manage the development, maintenance, and implementation of an architecture, as outlined in our Enterprise Architecture Management Maturity Framework version 1.1.

Since then, each of the military departments has adopted a federated approach to developing its respective architecture program. However, the extent to which each of their architecture programs has improved since 2008 varies. For example, while all three have or are in the process of establishing an executive committee with responsibility and accountability for the department’s enterprise architecture, none have fully developed an enterprise architecture methodology or have a well-defined business enterprise architecture and transition plan to guide and constrain business transformation initiatives. Table 5 provides a description of the military departments’ progress relative to those elements that we previously reported as not satisfied by one or more of the military departments. (These elements were common to both versions 1.1 and 2.0 of our Enterprise Architecture Maturity Management Framework.)

As described in the table, the military departments have made progress in managing their respective enterprise architecture programs since we last reported in 2008. However, each has yet to address key elements, including developing the architecture content, in order to advance to a level that can be considered fully mature. According to DOD officials, the lack of progress in addressing key management elements has been due to the uncertainty and pending decisions surrounding the roles and responsibilities of key organizations and senior leadership positions. Air Force officials attributed the state of its enterprise architecture program in part to the lack of resources. Army officials also stated that a major challenge has been the lack of resources (i.e., people and funding) due to the shift of resources to higher priority initiatives. Navy officials stated that there is not one overarching reason for the state of the Navy enterprise architecture program but rather a number of reasons, such as limited resources. Navy officials agreed that work remains to be done and stated that they will continue to address the missing elements as they move forward. Although all the military departments reported resources to be a challenge, as noted earlier, DOD requested about $17.3 billion for its business systems environment. Given the department’s prioritization of about $17 billion, the military department architecture programs have not received resources that they deemed sufficient to meet their needs. What this means is that DOD, as a whole, is not as well positioned as it should be to realize the significant benefits that a well-managed federation of architectures could afford its business systems modernization efforts. We have ongoing work looking at the status of each of the military departments’ enterprise architecture programs relative to all the applicable elements in our Enterprise Architecture Management Maturity Framework version 2.0.

Military Departments Continue to Develop Architecture Content, but Have yet to Develop Well-Defined Business Enterprise Architectures and Transition Plans

The Fiscal Year 2009 NDAA requires that each military department develop a well-defined enterprisewide business architecture and transition plan encompassing end-to-end business processes and that is capable of providing accurate and timely information in support of business decisions of the military department. However, while each of the departments has taken steps to develop architecture content, none has a well-defined business enterprise architecture and associated transition plan to guide and constrain its business transformation initiatives. Individual examples of progress made and challenges still facing each department are discussed next.

Department of the Air Force The Department of the Air Force is developing a corporate enterprise architecture and 12 subordinate “sub-enterprise” architectures, each of which is to be supported by subordinate domain architectures (i.e., acquisition), as appropriate. According to Air Force officials, work is still under way to identify the respective domains and determining which domains will support each sub-enterprise. According to these officials, the Air Force business enterprise architecture was until now being developed as a separate initiative but work is under way to integrate it with the Agile Combat Support sub-enterprise architecture.

In 2008, we reported that, as part of its Agile Combat Support sub- enterprise architecture, the department had developed enterprise architecture products that described some, but not all, elements of its current and target environments as well as a transition plan for its business area. Specifically, we reported that the Air Force had not defined the architecture products that described its logistics enterprise architecture for its current environment and its acquisition enterprise architecture and health services enterprise architectures for its target environments. We also reported that, while it had developed a sequencing plan of systems, it had not defined a gap analysis describing how the department would transition from the current to the target environment. Since then, the department has developed architectural artifacts that capture some of the missing business-related elements. For example, the Air Force has identified acquisition transformation goals, health services operational activities, and logistics systems and services, such as the Commodity Management Service. In addition, the department has identified and described business transformation initiatives (e.g., streamline civilian hiring process to meet DOD’s goal of 80 days by 2012). However, not all important business enterprise architecture contents have been described, including current functional capabilities and data objects necessary for current business operations.

Furthermore, although the department has identified some business- related elements, it does not have a well-defined business enterprise architecture and a transition plan to guide and constrain business transformation initiatives. In particular, it has yet to develop architectural products under its Air Force business enterprise architecture that would describe its current and target business environments in terms of business, information/data, application/service, technology, performance, and security. According to the Air Force, it is focused instead on capturing and using existing architecture artifacts that describe current and target architectures associated with priority areas for improvement. In addition, although the Air Force has outlined the business improvement priorities, it has yet to develop an enterprise transition plan for the business environment, including descriptions of gaps in terms of functional capabilities, performance shortfalls of business processes, and potential duplications of system functions.

The department also has yet to determine what, if any, of the business architectural artifacts under the Agile Combat Support sub-enterprise is to be leveraged for the development of the Air Force business enterprise architecture. According to Air Force officials, the Air Force business enterprise architecture and the Agile Combat Support sub-enterprise will use one common set of artifacts, and the department has identified the business architectural artifacts from the Agile Combat Support sub- enterprise to be merged with the Air Force business enterprise architecture. However, we have yet to be provided with a listing of these artifacts. In addition, according to these officials, discussion is still under way on how the department plans to federate the sub-enterprise architectures, including the Agile Combat Support, to the corporate enterprise architecture.

According to Air Force officials, in the future, the Air Force business enterprise architecture will focus on end-to-end processes with defined outputs and it will include an alignment of business systems to these end- to-end business processes. Also, according to the Air Force, it plans to use an architecture-based approach to align current Air Force capabilities and systems against specific business processes to identify duplication or gaps in process-based capabilities. According to the department, an implementation plan will be developed by the end of fiscal year 2011.

Without architectural descriptions of current and target business environments and an associated transition plan, as well as a clear picture of how to leverage prior content in development of the Air Force business enterprise architecture, the department is not well positioned to realize the significant benefits that a well-defined enterprisewide business architecture and transition plan can provide, including objective decision making regarding capability enhancements across end-to-end business processes and resources allocation across business system investments.

The Department of the Army has yet to develop or establish plans for developing a corporate enterprise architecture that identifies and describes rules, policies, procedures, and services to be federated across the Army. Instead, the department’s approach consists of developing three separate and distinct segment architectures (i.e., battle command, networks, and generating force), each of which is being developed by separate department organizational units and is supported by individual segment and solution architectures. According to department officials, the department’s current approach to developing its business enterprise architecture calls for this content to be developed as part of the generating force enterprise architecture effort.

In 2008, we reported that Army had not defined its current and target business environments and developed a transition plan. To its credit, the department has since made progress in developing its first version of the Army’s business enterprise architecture. Specifically, it has defined the scope of this architecture, which comprises traditional business functions such as acquisition, logistics, financial management, human capital management, and installation and environment. The scope of the Army’s business enterprise architecture is also extended to include training and sub-segments of the LandWarNet/Battle Command functions. This provides logical groupings of the key business activities the department performs, and can be used to identify subbordinate architectures that support the department’s generating force mission area. In addition, the architecture includes the 15 end-to-end business processes listed in the DOD business enterprise architecture and Army plans to use these processes to guide and constrain business process modeling efforts and identify business systems that are to be integrated. This is consistent with DOD’s approach of using end-to-end business processes to optimize business processes and the systems that support them. Further, the department has outlined its business transformation initiatives (e.g., civilian hiring reform) to accelerate business process improvement and cost savings.

However, Army does not have a well-defined business enterprise architecture and a transition plan to guide and constrain business transformation initiatives. In particular, although the first version provides systems mappings to several end-to-end business processes (e.g., Procure- to-Pay), it has yet to provide evidence of mappings of system functions to all the end-to-end business processes, such as Service Request-to- Resolution. According to officials, the mapping of system functions to process steps will take place when the department performs detailed business process mapping. Further, the first version does not describe how end-to-end business processes will be implemented, including principles and guidance for implementing technologies that would support planned business systems investments. Without this, there is an increased risk of incompatible implementation and/or not being able to leverage the most benefits from the technologies.

The Army has also made progress in developing aspects of a business transition plan. For example, it identifies business transformation initiatives, such as civilian hiring reform, and describes results or changes to business operations to be achieved by the business transformation initiatives. It also includes diagrams that depict the migration from legacy business systems to commercial off-the-shelf enterprise resource planning solutions. Such diagrams provide a life cycle view of enterprise systems resources, including describing how each system evolves over time. However, the plan still does not include important content such as gap analyses at the enterprise level and for each business function (e.g. acquisition, financial management) and timelines for addressing the gaps. A gap analysis is an assessment of the differences between the current and target architectures. For example, a performance gap analysis identifies performance measures (e.g., effectiveness) of a business process, highlights which performance measures are not being met in the current environment, and describes performance expectations for these measures in the target environment, thereby describing expected performance improvements of the business process. This performance gap analysis should also identify the business process activities or steps that need to be changed to achieve the future performance expectations. As such, these gap analyses are important to help identify changes or adjustments that are necessary at the enterprise level and within each business function to achieve desired business performance results and mission outcomes.

The Department of the Navy’s enterprise architecture is comprised of corporate architecture products, which include applicable laws, regulations, and policies as well as enterprisewide reference models and architecture content and subordinate architectures, which comprise nine segment architectures, with each addressing a distinct functional area such as logistics and command and control. These subordinate architectures are to be developed by communities of practice groups that have yet to be formally established. According to Navy officials, the current approach calls for the Navy business enterprise architecture content to be incorporated into the corporate enterprise architecture and applicable segment architectures (e.g., corporate management and support, force support, and logistics segment architectures), as appropriate.

In 2008, our analysis showed that Navy had developed enterprise architecture products that describe some, but not all, elements of its current and target environments as well as a transition plan for its business area. Specifically, we found that the Navy had not defined its current core business processes or provided a comprehensive picture of its current business problems that the department needs to address. We also found that it had yet to develop a well-defined target architecture, including the purpose and scope of all the functional areas (e.g., joint logistics and planning). In addition, we also found that, while the department had developed a transition plan as part of its architecture, the plan was not based on a gap analysis between the current and target environments. This is important to lay out a road map for optimizing mission performance and transforming business operations by systematically implementing changes to technologies and processes.

Since 2008, Navy has identified its business segments such as logistics and force support, representing some of its core business processes. In addition, according to Navy officials, it plans to establish communities of practice to oversee the development of segment reference architectures and a road map for developing these architectures. It has also added additional business-related architecture content. For example, the architecture includes the Navy’s Common Operational Activity List that specifies some business activities related to identifying and resolving accounting records discrepancies. Such operational activities provide a basis from which the department identifies the business activities or functions to be reused (and optimized) across the department and those activities or functions that remain unique to each business segment or domain.

Nonetheless, Navy does not have a well-defined enterprisewide business architecture and transition plan to guide and constrain business transformation initiatives. In particular, the department has not developed a business enterprise architecture that provides a clear and comprehensive picture of its current and target business environments. For example, the enterprise architecture does not describe current and target business capabilities, systems that are to be integrated to support DOD end-to-end business processes, and information exchange requirements among business activities. According to Navy officials, rather than capturing the department’s current and target business environments, the focus of Navy’s enterprise architecture is to develop artifacts that are “actionable” such as enterprise rules for assessing compliance of business systems. While such actionable artifacts provide value, they do not provide a comprehensive and systematic approach for transforming business operations. Further, there is no evidence that the department has documented current problems or defined the scope and purpose for all of the business-related segment reference architectures. Documenting current problems will enable the Navy to identify opportunities (e.g., new applications, new processes, or new management approaches) for improvement and to assess whether transformation efforts address these problems.

Moreover, although Navy has developed an enterprise transition plan, the plan is not well-defined and does not include an enterprise gap analysis that identifies the differences between the current and target business architectures, particularly the critical differences or shortfalls that affect the successful accomplishment of the department’s mission. According to officials, gaps and shortfalls in the current department programs can be identified through enterprise architecture compliance assessments, and enterprise architecture waivers are granted with specific expiration dates and conditions that are to be met to address the gaps and shortfalls. Nevertheless, the focus of these compliance assessments is on gaps and shortfalls of individual programs (e.g., limited system availability) and not on gaps and shortfalls from the perspective of end-to-end business processes (e.g., process gaps) that are needed to achieve the target environment. Moreover, a well-defined business enterprise architecture and transition plan is essential for establishing an implementation road map to address key gaps and shortfalls that can significantly impact business operations across the department and for evolving and developing business systems that optimize their mission value.

Fiscal Year 2012 Budget Submission Did Not Include Key Information on All Business Systems

Among other things, the Fiscal Year 2005 NDAA requires DOD’s annual IT budget submission to include key information on each business system for which funding is being requested, such as the system’s designated approval authority and the appropriation type and amount of funds associated with modernization (i.e., development) and current services (i.e., operations and maintenance).

The department’s fiscal year 2012 budget submission includes a range of information for 1,637 business system investments. Of these, 272 involve development and modernization. For each of the 272, the information provided includes the system’s name, approval authority, and appropriation type. The submission also identifies the amount of the fiscal year 2012 request that is for development and modernization versus operations and maintenance. For systems in excess of $1 million in modernization funding, the submission also cites its certification status (e.g., approved, approved with conditions, approved decertification close- out, and withdrawn) and the Defense Business Systems Management Committee approval date, where applicable.

However, similar to prior budget submissions, the fiscal year 2012 budget submission still does not reflect all business system investments. To prepare the submission, DOD relied on business system investment information (e.g., funds requested, mission area, and system description) that is entered by the components into DOD’s Select and Native Programming Data Input System—Information Technology (SNAP-IT). In accordance with DOD guidance and according to ASD(NII)/DOD CIO officials, the business systems listed in SNAP-IT should match the systems listed in the Defense Information Technology Portfolio Repository (DITPR)—the department’s authoritative business systems inventory. However, the DITPR data provided by DOD in March 2011 included 2,258 business systems. Therefore, SNAP-IT does not reflect about 620 business systems that are identified in DITPR.

We previously reported that the information between SNAP-IT and DITPR were not consistent and accordingly made a recommendation for DOD to develop and implement plans for reconciling and validating the completeness and reliability of information in its DITPR and SNAP-IT system data repositories, and to include information on the status of these efforts in the department’s fiscal year 2010 report in response to the act. DOD agreed with the need to reconcile information between the two repositories and stated that it had begun to take actions to address this. However, according to the Office of the ASD(NII)/DOD CIO, efforts to provide automated SNAP-IT and DITPR integration work were delayed due to increased SNAP-IT requirements in supporting the fiscal year 2012 budget submission and ongoing reorganization efforts within DOD. The department plans to restart the process of integrating the two systems beginning in the third quarter of fiscal year 2011. Until DOD has a reliable, comprehensive inventory of all defense business systems, it will not be able to ensure the completeness and reliability of the department’s IT budget submissions. Moreover, the lack of current and accurate information increases the risk of oversight decisions that are not prudent and justified.

DOD Has Continued to Establish Investment Management Processes but Has Yet to Fully Define and Implement Key Practices

Since our 2009 report, DOD has continued to establish investment management processes but has not fully defined all key practices. Further, with regard to certifying and overseeing investments—two key DOD IT management processes for selecting, managing, and monitoring investments—the department largely followed these processes for four department investments we reviewed, but key steps integral to these processes were not performed. Until DOD fully defines these key practices and performs integral key steps, it is unlikely that the department’s reported 2,258 business system investments—totaling $17.4 billion in fiscal year 2011—will be managed in an effective manner that maximizes mission performance while minimizing or eliminating system overlap and duplication.

DOD Has Continued to Establish Effective Investment Management Processes, but Has Yet to Fully Define Many Key Processes and Associated Policies and Procedures

Since we reported in 2009, DOD has continued to make progress in establishing the kind of investment management processes and associated key practices (i.e., policies and procedures) called for in the act and our ITIM framework as being integral to effective IT investment management. Specifically, since 2009, Air Force, Navy, and Army have implemented additional key practices associated with effectively managing investments as individual business system programs (Stage 2) and as portfolios of programs (Stage 3), while the DOD-level organizations (herein referred to as DOD enterprise) responsible for DOD-level processes have not. With regard to Stage 2 practices, Navy and Army implemented two key practices, and Air Force implemented one such practice. For Stage 3, Navy implemented one key practice. For those key practices that have yet to be fully defined, DOD enterprise and the military departments have in large part partially defined these practices. Nonetheless, even with this progress, DOD enterprise and the military departments have yet to fully define a majority of the Stage 2 and Stage 3 practices.

Table 6 provides a summary (by DOD enterprise and each military department) of the key Stage 2 practices implemented since 2009 along with those practices we reported in 2009 as having been implemented. The table also includes those practices yet to be implemented. As such, table 6 provides an overall snapshot of where DOD enterprise and the military departments stand with regard to building their investment management foundation, including the ability to manage investments as individual business system programs.

As shown in the table, since 2009: DOD enterprise has not implemented any additional key practices.

Air Force has implemented one key practice—documenting policies for meeting business needs. Specifically, in its IT Investment Review Guide, Air Force defines a process for ensuring that IT business system investments support the department’s ongoing and future business needs. This process includes having an IRB—consisting of senior executives from IT and functional business units, including the Office of the Air Force CIO—to regularly review all business systems, including those in operations and maintenance, to assess their alignment with business needs using factors such as how well investments support the Air Force’s mission and their strategic value and risk.

Navy has implemented two additional key practices—(1) instituting an enterprisewide IT investment board and (2) documenting policies for meeting business needs. Specifically, in March 2011, Navy established an Information Enterprise Governance Board—consisting of senior executives from IT and functional business units, including the Navy CIO—to serve as a business systems IRB. Among other things, the board is responsible for business system investment governance, including approving and annually reviewing business system investments. In addition, for meeting business needs, Navy’s Investment Review Guide dated October 2009 defines a process for conducting annual reviews of ongoing IT investments to ensure they support ongoing and future business needs. The process calls for the annual review of all business systems, including those in operations and maintenance, to demonstrate that they support ongoing and future business needs by, among other things, complying with applicable strategic business guidance such as DOD’s business enterprise architecture (BEA).

Army has implemented two key practices associated with capturing investment information. First, it has established policies and procedures for collecting information about the department’s investments. Specifically, Army’s investment review guide dated March 2010 defines procedures directing Army’s system owners to submit, update, and maintain IT projects and system information in a departmental data repository called the Army Portfolio Management Solution. Second, Army has assigned responsibility for investment information collection and accuracy. Specifically, Army’s investment review guide states that system owners are responsible for the accuracy of their data in the repository.

With regard to Stage 3 key practices, the following table provides a summary (by DOD enterprise and each military department) of the key practices implemented since 2009 and those practices that have yet to be implemented. Table 7 provides an overall snapshot of where DOD enterprise and the military departments stand in having the capability to build a complete investment portfolio.

As shown in the table, although DOD enterprise, Air Force, and Army have not implemented additional key practices, Navy has implemented one key practice associated with defining portfolio criteria—assigning responsibility to an individual or group to manage the development and modification of IT portfolio selection criteria. Specifically, Navy developed guidance that assigns Mission Area Leads and Functional Area Managers with responsibility for portfolio selection criteria.

With regard to the Stage 2 and Stage 3 key practices that have yet to be fully implemented, it is important to note that DOD and the military departments have partially defined these practices. For example, for selection, DOD established a process that calls for investments involving more than $1 million in obligations to be certified and approved before funds are to be expended. Specifically, the process calls for investments to be, among other things, compliant with DOD’s BEA and be economically justified. However, the process does not fully address all aspects of the selection key practice. For example, the process does not specify how the IRBs are to use the full range of cost, schedule, and benefit data in making selection (i.e., certification) decisions, as called for in our ITIM framework.

In addition, for oversight, DOD has established an oversight process that calls for, among other things, investments to be reviewed annually to assess how each is performing. As part of the process, the IRBs assess program performance relative to cost, schedule, and capability commitments. However, DOD’s oversight process does not provide sufficient visibility into the military department’s investment management activities, including its reviews of systems in operations and maintenance and smaller investments, commonly referred to as Tier 4 investments.

Nonetheless, DOD enterprise and the military departments have still not fully defined these Stage 2 and 3 key practices. Specifically, with regard to the nine Stage 2 practices, DOD enterprise, Air Force, and Navy, as shown in table 6, have yet to fully define five key practices (or 56 percent of the practices), and Army has yet to do so for seven (or 78 percent) of the practices.

With regard to the five Stage 3 practices, DOD enterprise, Air Force, and Navy, as shown in table 7, have yet to fully define four key practices (or 80 percent of the practices), and Army has yet to do so for any (100 percent) of the practices.

Officials from DOD enterprise and the military departments attributed the incomplete state of their IT investment management processes to the following: DOD enterprise officials said the condition of its processes, including the lack of progress since 2009, was due in part to current ongoing DOD efforts to reorganize the business systems governance organizations (e.g., the Business Transformation Agency) that are responsible for implementing IT investment management at the DOD enterprise level. As noted earlier, in August 2010, the Secretary of Defense announced the plans to disestablish the Business Transformation Agency and that the functions of the Business Transformation Agency, such as its IT investment management function, be reviewed and transferred to other organizations in DOD, as appropriate. The DOD officials stated that these implementation plans have yet to be finalized and have resulted in their investment management implementation efforts being delayed. These officials added that they are aware of the absence of documented project- level and portfolio-level management practices; they also said they are currently working on developing policies and procedures to address the missing processes and practices but were not able to provide us with milestones and a plan with defined steps for when the policies and procedures were to be completed.

Air Force and Navy officials said their investment management implementation efforts were taking longer than originally planned given their workload and other priorities assigned to them since initiating investment management efforts. They also acknowledged that their processes were missing certain documented project-level and portfolio- level management policies and procedures and said they were in the process of developing policies and procedures to address these missing processes and practices. In particular, Air Force officials stated that they planned to have their policies and procedures approved and finalized by October 2011.

Army officials said the state of their IT investment management process is due to the fact that the department focused on first establishing selected institutional capabilities—such as defining roles and responsibilities and establishing a department-level IRB—rather than attempting to do everything at once. The officials added that once its initial steps are completed, Army intends to then focus on implementing remaining Stage 2 and three key processes and practices. Specifically, these officials said that they are aware that Army lacks a military department-level IRB and added that until now they have been relying on functional area experts to review investments before they are sent to the DOD IRBs. They also acknowledged that Army is missing certain documented project-level and portfolio-level management policies and procedures. They further stated that the department is currently working on guidance to address these missing items with the goal of having it approved and finalized by August 2011.

As discussed in our ITIM framework and previous reports on DOD’s investment management of its business systems, adequately documenting both policies and associated procedures that govern how an organization manages its IT projects and investment portfolios is important because doing so provides the basis for rigor, discipline, and repeatability in how investments are selected and controlled across the entire organization. Until DOD fully defines missing policies and procedures, it is unlikely that the department’s reported 2,258 business systems will be managed in a consistent, repeatable, and effective manner that, among other things, maximizes mission performance while minimizing or eliminating system overlap and duplication. To this point, there is evidence showing that DOD is not managing its systems in this manner. For example, DOD reported that of its 79 major business and other IT investments, roughly a third are encountering cost, schedule, and performance shortfalls requiring immediate and sustained management attention. In addition, we have consistently reported for some time that DOD’s business system environment has been characterized by (1) little standardization, (2) multiple systems performing the same tasks, (3) the same data stored in multiple systems, and (4) manual data entry into multiple systems. Because DOD spends over $10 billion each year on its business systems and related IT infrastructure, the potential for identifying and avoiding the costs associated with duplicative functionality across its business system investments is significant.

DOD Has Largely Followed Its Certification and Oversight Processes for the Investments Under Review, but Validation and Other Key Steps Were Not Performed

As discussed, certification and oversight are key DOD processes for selecting, and overseeing IT investments. DOD guidance calls for investments to be certified and approved before funds are to be expended on modernization activities. More specifically, the guidance states that investments involving more than $1 million in obligations are to be certified by designated approval authorities and as part of that certification, authority officials are to attest that each investment is compliant with DOD’s BEA, including all relevant architecture products, such as products that specify the technical standards needed to promote interoperability among related systems or examine overlaps with other business systems; is economically justified, based on an economic viability analysis developed using disciplined and rigorous cost estimating practices; and has undergone sufficient business process reengineering analysis, including identifying and developing approaches to streamlining and improving involved processes.

As part of each of these three requirements, we have said it is important for designated approval authorities to validate the results of BEA and other assessments to ensure investment decision making is based on accurate and reliable information. More specifically, we previously reported that DOD had not been performing this step and made recommendations that they do so. DOD agreed with our findings and recommendations, and stated that it planned to assign validation responsibilities and issue guidance that describes the methodology for performing validation activities but were not able to provide a date for when this would be completed.

Once investments have been certified, DOD guidance calls for investments to be effectively overseen. This includes reviewing annually the progress of investments using predefined criteria and checkpoints, in meeting cost, schedule, risk, and benefit expectations.

Consistent with this, DOD guidance calls for IRBs to annually review certified investments and in doing so, to focus on program performance against cost, schedule, and performance baselines, and progress in meeting the certification conditions discussed. Our ITIM research and experience with federal agencies also shows that it is important for oversight and other decisionmaking authorities to validate performance information used to make decisions so that investment decision making is based on accurate and reliable information.

DOD largely followed the certification process for each of the four investments we reviewed, but did not perform validation and other key aspects of the process. Specifically: BEA compliance. DOD enterprise and the military departments took steps to assess BEA compliance of their respective systems. This included following DOD guidance (the appropriate version of DOD’s BEA Compliance Guidance) and using an automated tool (called Architecture Compliance and Requirements Traceability) to determine and report on the extent of each system’s architectural compliance. In addition, in each case, once the BEA assessment had been completed, the appropriate DOD enterprise and military department precertification authorities asserted (via memorandum of certification) that each system was compliant with DOD’s BEA.

For example, on the Project Management Resource Tool (PMRT) project, Air Force followed the BEA compliance guidance and used the Architecture Compliance and Requirements Traceability tool to develop a compliance report that mapped BEA activities to PMRT’s capabilities. In August 2010, Air Force’s Director of Business Transformation and Deputy Chief Management Officer stated in a supporting precertification memorandum that this information was used to assert that PMRT was compliant with DOD’s BEA.

In another example, on the Defense Travel System, DOD also assessed BEA compliance by using the BEA Compliance Guidance and the Architecture Compliance and Requirements Traceability tool to develop a report showing extent of compliance. In an October 2010, precertification memo, DOD’s CIO said this information was used to assert that the system was compliant with DOD’s BEA.

Although DOD took these steps to certify BEA compliance, it did not take other key steps. For example, DOD and component designated approval authorities did not validate the assessments and assertions. Specifically, the BEA compliance assessments performed on the investments under review were not validated by DOD certification and approval entities. Although this was not done, the systems were nevertheless certified as compliant. We reported on this weakness in 2008 and made recommendations to DOD to, among other things, explicitly assign responsibility for validating program BEA compliance assessments and issue guidance that describes the methodology for performing such validation activities. To date, DOD has yet to implement these recommendations. However, DOD officials told us the department has actions planned or underway to address these recommendations, although they were not able to provide milestones for when this would be accomplished.

In addition, our 2008 report showed that DOD BEA assessments did not include all relevant architecture products, such as products that specify the technical standards needed to promote interoperability among related systems or examine overlaps with other business systems. Despite the limited assessments, DOD nonetheless certified the investments as BEA compliant even though they did not adequately demonstrate such compliance. Accordingly, we have made recommendations to DOD to revise its BEA compliance guidance, among other things, to address these shortfalls. To date, DOD has yet to implement the recommendations. However, DOD officials said the department has actions planned or underway to address the recommendations but they were not able to provide a date for when this would be completed.

Economic viability analysis. For the investments under review, DOD enterprise and the military departments used DOD’s IT Investment Review Process Guidance (dated January 2009) that specifies how investment economic viability is to be analyzed and assessed. They also used a related automated tool designed to support the development of such analyses. Once the analyses had been performed, the precertification authorities for each of the systems asserted that the efforts had been reviewed, and showed the investments were economically justified to proceed with obligating funds.

For example, on the Logistics Modernization Program, Army used DOD’s guidance to conduct its economic viability analysis. The Army also used the economic viability tool to complete the analysis. In addition, in December 2010 memorandum, the Army’s Chief Management Officer asserted that the economic viability analysis had been completed, and showed the investment was justified to proceed with obligating funds.

In another example, on the Navy Enterprise Resource Planning system, Navy used the January 2009 DOD guidance to comply with the economic viability analysis requirement and used the economic viability tool to complete the analysis. Further, in a July 2010 memo, the Navy’s Chief Management Officer asserted that the investment’s economic viability analysis had been completed, and showed the investment was justified to obligate funds.

Although DOD enterprise and the military departments took these steps to justify the investments, they did not perform other key steps. Specifically, DOD enterprise and the military departments did not use important cost estimating practices critical to developing such analyses. For example, in developing its economic justification for its ERP system, Navy did not implement key aspects of earned value management or develop risk mitigation strategies to address this risk. We have previously reported on these weaknesses and made recommendations to address them. Although the recommendations are still open, DOD enterprise and military department officials have said they have actions planned and under way to address the recommendations. However, they were not able to provide a timetable for when the actions are to be completed.

Business process reengineering assessment. For the investments under review, DOD enterprise and the military departments used DOD’s Business Process Reengineering Guidance (dated April 2011) to assess whether the investments complied with the business process reengineering requirement. Consistent with the guidance, DOD enterprise and the military departments completed questionnaires (contained in the guidance) that aim to help DOD enterprise and the military departments identify and develop approaches to streamlining and improving existing business processes. Once these assessments had been completed, the DOD enterprise and military department precertification authorities asserted that business process reengineering assessments had been performed.

For example, on the PMRT project, Air Force used the DOD reengineering guidance to assess whether there were ways to streamline and improve existing business processes to be supported by the system investment. Air Force completed the assessment in July 2010. Air Force reported that as part of this assessment, it had representatives from the offices of the CIO and the Deputy Chief Management Officer review the completed assessment questionnaire and supporting documentation to determine whether the project team had followed the reengineering requirement. Subsequently, in August 2010, the Air Force’s Director of Business Transformation and the Deputy Chief Management Officer used this information to assert that sufficient business process reengineering had been conducted in order for the program to obligate investment funding.

While DOD enterprise and military department precertification authorities largely followed DOD’s guidance, they did not perform the key step of validating the results of these reengineering assessments to ensure they, among other things, accurately assessed process weaknesses and identified opportunities to streamline and improve affected processes. We have ongoing work on actions the Air Force and Army have taken to comply with statutory requirements regarding business process reengineering.

The reason DOD did not follow key aspects of the certification process— primarily not validating assessment results—is attributed in part to unclear roles and responsibilities. According to military department officials responsible for the investments we reviewed, validation activities did not occur because DOD policy and guidance does not explicitly require them to be performed and there is no guidance that specifies how assessments should be validated. According to DOD officials, the oversight and designated approval authorities did not validate the DOD enterprise- level assessments and assertions because DOD policy and guidance has not yet been revised to require these authorities to do so. Consequently, until the policy and guidance is updated and roles and responsibilities with regard to who is to perform validation functions are clearly defined, there is an increased risk that DOD will be making business system investment decisions based on information that is inaccurate and unreliable.

For the investments under review, DOD largely followed its oversight process but did not perform an important validation activity. Specifically, DOD’s oversight process (as specified in the January 2009 Investment Review Guide) calls for investments to be reviewed annually to assess how each is performing. As part of this process, the IRBs assess program performance relative to, among other things, cost, schedule, and capability commitments. The IRBs do this using updated information provided by the programs and screened by DOD enterprise and military department precertification authorities for completeness. These oversight reviews are important because an investment board should have visibility into each project’s performance and progress toward predefined cost, schedule, and benefit expectations as well as each project’s exposure to risk. Without such visibility and validated information, organizations risk making investment decisions that are inconsistent and are not fully grounded in reliable and accurate data.

Consistent with this direction, DOD conducted (or is planning to conduct) annual reviews for the investments we reviewed. For example, DOD conducted annual reviews for the Defense Travel System in December 2010 and the Navy Enterprise Resource Planning system in July 2010. In doing these reviews, DOD assessed investment performance using the cost, schedule, and performance information provided by the programs and screened by precertification authorities.

Although DOD largely followed its oversight process, it did not validate the cost, schedule, and performance information used for decision making. This finding is consistent with our previous report noting that DOD’s oversight process does not provide for sufficient visibility into the military department’s investment management activities, including its reviews of systems in operations and maintenance and smaller investments, commonly referred to as Tier 4 investments. Such visibility is important because DOD reports that only 100 of approximately 2,258 total business systems are annually reviewed by the IRBs. This means that the vast majority of business systems are overseen only within the military departments. Accordingly, we have made recommendations to address this area. DOD officials said they plan to address the recommendations but were unable to provide a schedule for when this work is to be completed.

In explaining why the information used by the IRBs is not validated, DOD officials cited the same reasons—outdated policy and guidance and unclear roles and responsibilities—as those provided for the lack of validation in the investment certification process. Consequently, until such roles and responsibilities are clarified, DOD faces increased risk that it will not effectively be able to oversee its extensive business systems investments.

DOD’s Annual Report Describes Certification Actions for Its Business System Investments

Among other things, the act requires DOD to submit an annual report to congressional committees on DOD’s compliance with requirements of the act, including a description of specific actions the department has taken on each business system modernization investment submitted for certification. The act further requires that such investments involving more than $1 million in obligations must be certified by a designated approval authority as meeting specific criteria, such as demonstrating compliance with DOD’s business enterprise architecture. Further, the act requires the Defense Business Systems Management Committee to approve each of these certifications.

In May 2010, we reported that the department’s annual report did not discuss certification actions for all systems on which certification actions had been taken, primarily excluding business system recertifications. We recommended the Deputy Secretary of Defense expand future DOD annual reports to Congress to include all certification actions that had been taken in the previous year by the department on its business system modernization investments. DOD agreed with our recommendation and stated that it would include recertifications in future reports. Since then, the department has addressed this recommendation by including all types of certification actions in its 2011 annual report, including recertification actions. As it has since 2005, DOD continues to certify and approve business system modernization investments in excess of $1 million.

DOD’s annual report identifies IRB certification actions associated with 137 business system investments that underwent the IRB certification and Defense Business Systems Management Committee approval process for fiscal year 2010 and cost approximately $1.3 billion. Specifically, the annual report accurately states that during fiscal year 2010, 52 unique business system modernizations were certified—35 with and 17 without conditions. For the 35 systems, 32 conditions were reported. Examples of conditions cited in the report are the need for business enterprise architecture compliance to improve interoperability and integration of cross-functional processes and improved program management functions. The report also identifies 93 recertifications and 28 decertifications. For example, the Navy’s Enterprise Resource Planning system had about $7.6 million recertified in early August and another $96.6 million recertified later in the month.

While DOD has made progress by reporting its certification actions, the basis for these certification actions and subsequent approvals is limited as discussed in the previous section.

Conclusions

A well-defined federated architecture and accompanying transition plans for the business mission area, along with well-defined investment management policies and procedures across all levels of the department, are critical to effectively addressing DOD’s business systems modernization high-risk area. Relatedly, it is important for the department to obtain independent assessments of the completeness, consistency, understandability, and usability of the federated family of business mission area architectures, including associated transition plans. Equally important is for the department to actually implement its architecture and investment management controls in the years ahead on each and every business system investment, and in doing so ensure that it has reliable information on each investment on which to base executive decision making.

DOD has continued to take steps in defining and implementing these key institutional modernization management controls, but challenges that we identified in prior years still need to be addressed. Specifically, while DOD continues to release updates to its corporate enterprise architecture, the architecture has yet to be federated through development of aligned subordinate architectures for each of the military departments. In this regard, each of the military departments has made progress in managing its respective architecture program, but there are still limitations in the scope and completeness, as well as the immaturity of the military department architecture programs, including the completeness of their own transition plans. In addition, while DOD continues to establish investment management processes, the DOD enterprise and the military departments’ approaches to business systems investment management still lacks the defined policies and procedures to be considered effective investment selection, control, and evaluation mechanisms. Finally, information used to support the development of DOD’s budget requests, as well as to inform certification decisions, is still of questionable reliability. Collectively, these long-standing limitations in the department’s institutional modernization management controls continue to put the billions of dollars spent annually on thousands of business system investments at risk. Our previous recommendations to the department have been aimed at accomplishing these and other important activities related to its business systems modernization. To the department’s credit, it has agreed with these recommendations and is committed to implementing them.

However, the state of progress of DOD and military department business system modernization efforts is due, in part, to uncertainty and pending decisions surrounding the roles and responsibilities of key organizations and senior leadership positions. Accordingly, it is essential that DOD resolve these matters expeditiously, as doing so is on the department’s critical path for fully establishing the full range of institutional management controls needed to address its business systems modernization high-risk area.

Recommendation for Executive Action

Because we have existing recommendations that address the institutional management control weaknesses discussed in this report, we are making no further recommendations in these areas.

To address the uncertainty and pending decisions surrounding the roles and responsibilities of key organizations, we recommend that the Secretary of Defense expeditiously complete the implementation of the announced transfer of functions of the Business Transformation Agency and the Office of the Assistant Secretary of Defense for Networks and Information Integration/Department of Defense CIO and provide specificity as to when and where these functions will be transferred.

Agency Comments and Our Evaluation

In written comments on a draft of this report, signed by the Deputy Chief Management Officer and reprinted in appendix II, the department agreed with our recommendation and stated that it expects to announce the implementation details concerning the transfer of functions of the Business Transformation Agency and the Office of the Assistant Secretary of Defense for Networks and Information Integration/Department of Defense CIO prior to June 30, 2011.

We support the department’s efforts to address our recommendation and reiterate the importance of following through in implementing the recommendation within the stated time frame.

We are sending copies of this report to interested congressional committees; the Director, Office of Management and Budget; and the Secretary of Defense. This report will also be available at no charge on our Web site at http://www.gao.gov.

If you or your staffs have any questions on matters discussed in this report, please contact me at (202) 512-6304 or melvinv@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III.

Appendix I: Objective, Scope, and Methodology

As agreed with congressional defense committees, our objective was to assess the actions by the Department of Defense (DOD) to comply with provisions of section 332 of the Ronald W. Reagan National Defense Authorization Act for Fiscal Year 2005. This included (1) developing a business enterprise architecture and a transition plan for implementing the architecture, (2) identifying systems information in its annual budget submission, (3) establishing a system investment approval and accountability structure along with an investment review process, and (4) certifying and approving any system modernizations costing in excess of $1 million. (See the background section of this report for additional information on the act’s requirements.) Our methodology relative to each of these four provisions is as follows: To address the architecture and transition plan provision, we focused on the progress the departments of the Air Force, Army, and Navy have made in developing their respective parts of the federated DOD business enterprise architecture. In doing so, we compared the baseline enterprise architecture program status information as presented in our 2008 report, with information on the current status of each military department’s enterprise architecture program. In doing so, we focused on those select elements that were either similar or slightly modified across versions 1.1 and 2.0 of our Enterprise Architecture Management Maturity Framework and that were either partially or not satisfied by one or more of the military departments. Specifically, we reviewed written responses and supporting documentation on steps completed, under way, or planned from each military department to identify examples of progress made in addressing those elements that we had previously identified as being not satisfied or partially satisfied. We also reviewed business architectural artifacts to determine the progress each department had made in developing their respective business architectural content since we last reported in 2008. We interviewed cognizant DOD officials to validate the responses and identify any discrepancies. Further, we reviewed the independent verification and validation contractor’s statement of work and other work products to determine whether they addressed the department’s federated family of corporate and subordinate architectures.

To determine whether DOD’s fiscal year 2012 IT budget submission was prepared in accordance with the criteria set forth in the act, we reviewed and analyzed the Report on Defense Business System Modernization FY 2005 National Defense Authorization Act, Section 332, dated March 2011 and compared it to the specific requirements in the act. We also compared information contained in the department’s system that is used to prepare its budget submission (SNAP-IT) with information in DOD’s Defense Information Technology Portfolio Repository (DITPR) system to determine if DOD’s fiscal year 2012 budget request included all business systems. We interviewed Assistant Secretary of Defense for Networks and Information Integration/Department of Defense Chief Information Officer officials to discuss the accuracy and comprehensiveness of information contained in the SNAP-IT system, the discrepancies in the information contained in the DITPR and SNAP-IT systems, and efforts under way or planned to address these discrepancies. We did not independently validate the reliability of the cost and budget figures provided by DOD because the specific amounts were not relevant to our findings.

To assess the establishment of DOD enterprise and component investment management structures and processes, we analyzed whether DOD and its military departments’ information technology investment management processes were compliant with federal guidance and the extent to which DOD and the military departments were following their investment management processes, including those at the DOD enterprise-level for approving and certifying investments. To perform the first task, we compared the status of DOD enterprise and military department (Air Force, Army, and Navy) investment management processes—as noted in our May 2009 report and other sources—with the current status of these organization’s processes. As part of this analysis, we focused on the definition of project-level (Stage 2) and portfolio-level (Stage 3) policies and procedures contained in our Information Technology Investment Management (ITIM) Framework that were identified in our previous work as not being established. Specifically, we analyzed written department responses and supporting documentation on steps completed, under way, or planned against ITIM key practices to identify where progress had been made in addressing such previously identified practices. Where there were variances (i.e., support did not show the department was meeting a key practice), we reviewed relevant documentation and interviewed appropriate DOD enterprise and military department officials to identify the causes and impacts.

With regard to our second task, we selected four DOD enterprise-level and military department-level investments that met the following criteria: the investment was (1) either a (Tier 1 or 2) major automated information system from key DOD functional areas (i.e., Weapon Systems Lifecycle Management; Materiel Supply and Services Management; and Human Resources Management) and (2) was at a life cycle phase—such as production and deployment and operations and maintenance—where there were extensive opportunities for system investment officials to demonstrate the organization was following ITIM key practices. In reviewing these investments, we focused on DOD enterprise and military department activities related to certification and oversight, which are a key part of selecting, managing, and overseeing IT investments as called for in our ITIM framework and DOD guidance. For certification, we reviewed DOD Investment Review Board guidance to understand the types of actions related to the certification of business system modernizations and, in doing so, focused on three certification requirements (e.g., ensuring that designated approval authorities assert that each investment is compliant with the business enterprise architecture). For each requirement, we reviewed supporting documentation from DOD enterprise and the military departments to determine whether there was a documented process for how the requirement was to be certified and to ascertain whether artifacts prepared as part of the process demonstrated that the certification process was being followed. We did the same for the oversight process. When there were variances between the criteria and what DOD enterprise and the military departments had done, we interviewed cognizant DOD enterprise-level and military department-level officials on the causes and impacts.

To determine whether the department was certifying and approving business system investments with annual obligations exceeding $1 million, we reviewed and analyzed all Defense Business Systems Management Committee certification approval memoranda as well as IRB certification memoranda issued prior to the Defense Business Systems Management Committee’s final approval decisions for fiscal year 2010 and compared the results to those certification actions described in the annual report to identify differences. We also reviewed DOD IRB guidance to understand the types of actions related to certification of business system modernizations. We interviewed officials from the Business Transformation Agency and IRBs to discuss any discrepancies.

We conducted this performance audit at DOD and military department offices in Arlington, Virginia, from January 2011 to June 2011, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objective.

Appendix II: Comments from the Department of Defense

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, key contributors to this report were Gerard Aflague, Mathew Bader, Carl Barden, Shaun Byrnes, Debra Conner, Elena Epps, Rebecca Eyler, Nancy Glover, Neelaxi Lakhmani (Assistant Director), Anh Le, Lori Martinez, Gary Mountjoy, Freda Paintsil, David Powner (Director), Christine San, Sylvia Shanks, Donald Sebers, Teresa Smith, Jennifer Stavros-Turner, and Adam Vodraska.