Defense Business Systems: Further Refinements Needed to Guide the Investment Management Process
Released: 2014-05-12T00:00:00

Letter

Background

DOD is one of the largest and most complex organizations in the world. In support of its military operations, the department performs an assortment of interrelated and interdependent business functions, such as logistics management, procurement, health care management, and financial management. Yet, we have previously reported that the DOD systems environment that supports these business functions is overly complex and error prone, and is characterized by (1) little standardization across the department, (2) multiple systems performing the same tasks, (3) the same data stored in multiple systems, and (4) the need for data to be entered manually into multiple systems.

For fiscal year 2015, the department requested about $10.038 billion for its business system investments. According to the department’s authoritative source for certification data, its environment is composed of approximately 2,329 business systems, including 228 for acquisition, 14 for defense security enterprise, 28 for enterprise IT infrastructure, 286 for financial management, 730 for human resources management, 293 for installations and environment, 702 for logistics and materiel readiness, and 8 for security cooperation. (See fig. 1.) Of these 2,329 business systems, 1,180 are covered by the act’s certification and approval requirements.

DOD currently bears responsibility, in whole or in part, for 15 of the 30 programs across the federal government that we have designated as high risk. Seven of these areas are specific to the department, and eight other high-risk areas are shared with other federal agencies. Collectively, these high-risk areas relate to DOD’s major business operations that are inextricably linked to the department’s ability to perform its overall mission and directly affect the readiness and capabilities of U.S. military forces and can affect the success of a mission. In particular, the department’s nonintegrated and duplicative systems impair its ability to combat fraud, waste, and abuse. Consequently, DOD’s business systems modernization is one of the department’s specific high-risk areas and is essential for addressing many of the department’s other high-risk areas. For example, modernized business systems are integral to the department’s efforts to address its financial, supply chain, and information security management high-risk areas.

Summary of NDAA Requirements

Congress included provisions in the NDAA, as amended, that are aimed at ensuring DOD’s development of a well-defined BEA and associated ETP, as well as the establishment and implementation of effective investment management structures and processes. The act requires DOD to, among other things, establish an investment approval and accountability structure along with an investment review process, not obligate funds for a defense business system program with a total cost in excess of $1 million unless the approval authority certifies that the business system program meets specified conditions, develop a BEA that covers all defense business systems, develop an ETP for implementing the architecture, and identify systems information in DOD’s annual budget submissions.

The act also requires that the Secretary of Defense annually submit to the congressional defense committees a report on the department’s compliance with these provisions. DOD submitted its annual report to Congress in March 2014, describing steps taken, under way, and planned to address the act’s requirements.

DOD’s Approach to Business Systems Modernization

As of April 2014, the department’s approach to modernizing its business systems environment, which is part of DOD’s overall effort to transform its business operations, included improving business systems investment management, reengineering the business processes supported by its defense business systems, and improving and using the BEA and associated ETP. These efforts are to be guided by DOD’s Chief Management Officer and Deputy Chief Management Officer (DCMO). Specifically, the Chief Management Officer’s responsibilities include developing and maintaining a department-wide strategic plan for business reform; establishing performance goals and measures for improving and evaluating overall economy, efficiency, and effectiveness; and monitoring and measuring the progress of the department. The DCMO’s responsibilities include recommending to the Chief Management Officer methodologies and measurement criteria to better synchronize, integrate, and coordinate the business operations to ensure alignment in support of the warfighting mission and developing and maintaining the department’s enterprise architecture for its business mission area.

DOD has assigned roles and responsibilities to various governance entities and positions related to business systems modernization. For example, the Deputy’s Management Action Group is a senior-level forum that meets several times a month to discuss department-wide management issues, including business-related topics. This group is to convene as the Defense Business Systems Management Committee when it reviews defense business system portfolios. Under this committee, the Defense Business Council (DBC) acts as a corporate- level investment review board (IRB) by overseeing the approach and guidance for selecting and controlling the investment portfolio and making recommendations on funds certifications. Table 1 describes selected roles and responsibilities and composition of key governance entities and positions related to business systems modernization.

In order to manage and oversee the department’s business operations and approximately 1,180 covered defense business systems, the Office of the DCMO developed its Integrated Business Framework. According to the Office of the DCMO, this framework is used to align the department’s strategic objectives—laid out in the National Security Strategy, Quadrennial Defense Review, and Strategic Management Plan—to its defense business system investments.

Using the overarching goals of the Strategic Management Plan, principle staff assistants develop six functional strategies that cover eight functional areas. These functional strategies define business outcomes, priorities, measures, and standards for a given functional area within DOD. The functional areas are

Acquisition,

Defense Security Enterprise,

Enterprise IT Infrastructure,

Financial Management,

Human Resources Management and Health Management,

Installations and Environment,

Logistics and Materiel Readiness, and

Security Cooperation.

The business objectives and compliance requirements laid out in each functional strategy are to be integrated into the BEA.

The precertification authorities in the Air Force, Navy, Army, and other departmental organizations use the functional strategies to guide the development of organizational execution plans, which are to summarize each component’s business strategy for each functional area. Each plan includes a description of how the component’s goals and objectives align with those in the functional strategies and the Strategic Management Plan. In addition, each organizational execution plan includes a portfolio of defense business system investments organized by functional area. The components submit each of these portfolios of systems to the DBC for certification on an annual basis.

According to DOD’s investment management guidance, the DBC reviews the organizational execution plans and associated portfolios based on four investment criteria—compliance, strategic alignment, utility, and cost—to determine whether or not to recommend the portfolio for certification of funding. The Vice Chairman of the Defense Business Systems Management Committee approves certification decisions and then documents those decisions in investment decision memoranda. These memoranda state whether an individual organizational execution plan has been certified, conditionally certified (i.e., obligation of funds has been certified and approved but may be subject to conditions that restrict the use of funds, a time line for obligation of funds, or mandatory changes to the portfolio of business systems), or not certified (i.e., certification is not approved due to misalignment with strategic direction, mission needs, or other deficiencies).

The Office of the DCMO Currently Manages Business Systems, but a Planned Reorganization May Change These Responsibilities

The Office of the DCMO manages the day-to-day aspects of department- level business system oversight. It is currently composed of five directorates with various responsibilities related to business systems modernization. Table 2 shows the organizational components of the Office of the DCMO and key responsibilities.

These responsibilities may shift as a result of planned organizational changes. More specifically, on December 4, 2013, the Secretary of Defense issued a memorandum that outlined the results of an organizational review completed by the Office of the Secretary of Defense. Among other things, the Secretary called for strengthening the office of the DOD Chief Information Officer (CIO) by moving the oversight of business systems from the DCMO to the CIO, and for strengthening the Office of the DCMO to better coordinate the department’s business affairs. According to the Director of Investment and Acquisition Management in the Office of the DCMO, final decisions about how this directive will be implemented have not yet been made. The memorandum calls for all of the changes to occur by January 2015.

Overview of DOD’s BEA

DOD’s BEA is intended to serve as a blueprint for the department’s business transformation efforts. In particular, the architecture is to guide and constrain implementation of interoperable defense business systems by, among other things, documenting the department’s business functions and activities; the information needed to execute its functions and activities; the business outcomes from using the BEA; and the list of business rules, laws, regulations, and policies associated with its business functions and activities. According to DOD, its architecture is being developed using an incremental approach, where each new version of the architecture addresses business mission area gaps or weaknesses based on priorities identified by the department. The department’s BEA focuses on documenting information associated with its end-to-end business process areas (e.g., hire-to-retire and procure-to-pay).

The department considers its current approach to developing the BEA both a “top down” and “bottom-up” approach. Specifically, according to DOD, the architecture focuses on developing content to support investment management and strategic decision making and oversight (“top down”) while also responding to department needs associated with supporting system implementation, system integration, and software development (“bottom up”).

The department’s approach to developing its BEA involves the development of a federated enterprise architecture, where member architectures (e.g., Air Force, Army, and Navy) conform to an overarching corporate or parent architecture and use a common vocabulary. This approach is to provide governance across all business systems, functions, and activities within the department and improve visibility across DOD’s efforts.

Prior GAO Reviews of DOD’s Business Systems Modernization

Between 2005 and 2008, we reported that DOD had taken steps to comply with key requirements of the NDAA relative to architecture development, transition plan development, budgetary disclosure, and investment review, and to satisfy relevant systems modernization management guidance. However, each report also concluded that much remained to be accomplished relative to the act’s requirements and relevant guidance. We made recommendations to address each of the areas, and DOD largely agreed with our recommendations.

However, in May 2009, we reported that the pace of DOD’s efforts in defining and implementing key institutional modernization management controls had slowed compared with progress made in each of the previous 4 years, leaving much to be accomplished to fully implement the act’s requirements and related guidance. In addition, between 2009 and 2012, we found that progress had been made, but long-standing challenges we had previously identified remained to be addressed. For example, the department’s budget submission for fiscal year 2013 did not include all systems because of the lack of a reliable, comprehensive inventory of all defense business systems. We concluded that DOD’s progress in addressing the act’s requirements, its vision for a federated architecture, and our related recommendations was limited, in part, by continued uncertainty surrounding the department’s governance mechanisms, such as roles and responsibilities of key organizations and senior leadership positions. Accordingly, we made recommendations to address the issues identified. DOD partially agreed with our recommendations.

In 2013, we reported that DOD had continued to make improvements, particularly by taking steps to establish a portfolio-based approach to reviewing and certifying its defense business systems. However, among other things, we also found that the BEA and ETP were still missing important content and BEA and business process reengineering certifications were not validated. We made additional recommendations in these areas, and the department partially agreed with our recommendations.

In addition, our most recent high-risk report noted that, while DOD’s capability and performance relative to business systems modernization had improved, significant challenges remained. For example, the department had not fully defined and established a family of management controls, such as corporate and component business architectures and business system investment management processes. These management controls are vital to ensuring that DOD can effectively and efficiently manage an undertaking with the size, complexity, and significance of its business systems modernization, and minimize the associated risks. Furthermore, we also recently reported that, in order to better identify and address potential duplication, DOD needed to develop supporting component architectures, align them with its corporate architecture to complete the federated BEA, and leverage its federated architecture to avoid investments that provide similar, but duplicative, functionality in support of common DOD activities.

DOD Has Addressed Most Provisions of the Act, but Needs to Continue Improving Its Investment Management Approach

DOD has made progress in addressing the provisions of the NDAA and recommendations that we have made in recent years. Specifically, the department issued its March 2014 Congressional Report on Defense Business Operations, which describes updates and next steps for its investment review process; improved the data used to manage its business systems certification and approval process; issued an updated ETP in December 2013; and issued its fiscal year 2015 funding request for defense business systems. However, the department still faces challenges in complying with key parts of the act and managing its business systems. For example, the department needs to further refine its approach for reviewing system certifications so that the reviews occur at an appropriate level based on factors such as cost and complexity. Without continued progress in improving its investment management approach, DOD will be challenged in its ability to manage the billions of dollars invested annually in modernizing its business system investments.

An Integrated Business Framework Guides DOD’s Business System Investments, but Further Refinements Are Needed to Improve the Investment Review Process

The act, as amended by the NDAA for Fiscal Year 2012, included significant changes to the requirements for investment review and certification of defense business systems. Specifically, it required DOD to establish a department-wide IRB, chaired by the DCMO, and an investment management process. The act also called for the use of threshold criteria to ensure an appropriate level of review within the department of, and accountability for, defense business system programs depending on scope, complexity, and cost. In addition, IT investment management best practices describe key practices for instituting an IRB, including (1) grouping investments into portfolios, (2) coordinating the investment management process with other internal management controls, and (3) ensuring that systems receive an appropriate level of review by developing tiered review boards.

Since we reported in May 2013, DOD has continued its efforts to establish the IRB and further define and implement its defense business system governance framework—called the Integrated Business Framework—as discussed earlier in this report. This framework is used to manage the department’s business operations and investments, and includes six portfolios that align to functional areas. Of the three key practices for establishing an IRB described above, DOD’s Integrated Business Framework fully addresses one practice, partially addresses one practice, and does not address one practice (see table 3).

Officials from the Office of the DCMO provided several reasons why two of the key practices for instituting an IRB were not fully addressed. With respect to aligning the Integrated Business Framework with the budget process, officials stated that specific plans take time and coordination to develop and implement. These officials noted that these efforts are affected by the ongoing shift of responsibilities for business systems from the Office of the DCMO to the DOD CIO. Notwithstanding the uncertainty associated with this transition, until the department takes further steps to align its business system investment review and budgeting processes, it risks allocating resources to business system investments that might be used more efficiently or effectively elsewhere across the department. In addition, aligning the business system investment management process with the budget process would improve the ability of components to guide their investments from a strategic perspective and reduce pressure on the DBC to approve the obligation of funds that have already been budgeted for a given system.

Regarding the development of multiple review boards, officials from the Office of the DCMO said that the DBC guidance does not call for developing IRBs at multiple levels within the department because it is important for the DBC to certify and approve all covered business systems, and not just those that are more costly. These officials added that some systems may be important for the business environment regardless of their relatively small dollar value. In addition, these officials stated that the Integrated Business Framework calls for reviewing systems as part of system portfolios, not as individual systems. We agree that the DBC should certify and approve all systems as called for by the act, that it may want to occasionally focus greater attention on smaller systems, and that the department should review systems as part of larger system portfolios.

However, by not using a tiered review board approach, the department may hamper the quality of the information submitted and the associated reviews. This is evidenced by the results of the Office of the DCMO’s validations of certifications discussed later in this report. Systems— including systems submitted as part of portfolios—that are larger in scope, complexity, cost, or risk may benefit from additional scrutiny by the DBC, while those that are less complex or lower cost could be reviewed at the component level. Our IT investment management guidance notes that enterprise-wide IT investment boards should be responsible for systems that have high cost, high risk, or significant scope or duration, while retaining responsibility and visibility into other system review activities. However, lower-level boards could be chartered within business units to oversee the investment management process for other systems that are smaller or less costly, risky, or complex. While officials from the Office of the DCMO stated that different systems are reviewed with different levels of scrutiny based on various factors, those factors are not documented in existing guidance. Until DOD ensures that investments are reviewed at an appropriate level based on defined criteria, the DBC is at an increased risk of failing to identify and address important issues associated with large-scale and costly systems.

DOD Has Made Improvements in Data Used to Support Its Certification and Approval of Defense Business Systems, but Reviews Supporting These Decisions Can Be Improved

The NDAA requires that, prior to obligating funds, DOD certify that investments meet certain conditions set out in the act. For fiscal year 2014, the department certified and approved most of its business systems investments. In addition, it has made significant improvements to the data used to manage information about these investments. However, while DOD continued to assert the compliance of its investments with the department’s BEA and business process reengineering, the quality of the reviews supporting these assertions can be improved.

DOD Certified and Approved Most Business System Investments for Fiscal Year 2014

According to DOD’s March 2014 Congressional Report on Defense Business Operations, the DBC reviewed certification requests totaling $6.996 billion for 1,180 defense business systems across eight functional areas. Of those requests, according to the March report, the council approved $6.379 billion for 1,173 defense business systems and did not approve $617 million associated with 40 business systems (see table 4).

These certifications included about $563 million in funds for fiscal year 2014 that the DBC conditionally certified (i.e., certified on the condition that additional steps were taken). According to DOD’s March 2014 report, the primary reason that systems were denied certification or conditionally certified was that they were lacking a problem statement (or “business need”). The following are examples of systems that were denied certification or were conditionally certified in fiscal year 2014:

The Defense Logistics Agency’s Standard Procurement System requested—but was denied—$3.6 million in development funds because a problem statement had not been submitted. As of March 2014, no problem statement had been submitted, and the item remained open.

The Defense Health Agency’s Armed Forces Health Longitudinal Technology Application requested—and was conditionally approved for—about $133.1 million for fiscal year 2014, with the condition that the component submit a problem statement. As of April 2014, the problem statement had been submitted and approved.

The Defense Logistics Agency’s Defense Retired and Annuitant Pay System 2 requested a total of $12.6 million, but was only approved for $2.2 million, with the condition that, among other things, the component submit an updated cost estimate. As of March 2014, the cost estimate had been submitted to the Office of the DCMO, but had not yet been approved.

DOD Has Made Important Improvements to the Data Used to Manage Its Business System Investments

Our IT investment management framework states that to make good IT investment decisions, an organization must be able to acquire pertinent information about each investment and store that information in a retrievable format, to be used in future investment decisions. As part of this process, an organization should identify its IT assets and create a comprehensive repository of investment information. The information in the repository should identify each IT investment and its associated components and should be accessible where it is of the most value to those making decisions about IT investments.

DOD’s ability to address the act’s certification and approval requirements depends in large part on its ability to capture and use data about business systems and the department’s business system environment. To that end, DOD maintains three key data sources or data repositories and requires in its investment management guidance that components include updated data within each source. Table 5 provides additional information about each data source.

Since our May 2013 report, DOD has demonstrated important improvements to the data it uses to manage its business systems. In May 2013, we reported that we were unable to verify the number of certifications that occurred during the fiscal year 2013 certification and approval process. At the time, DOD officials noted that certain data fields in the original data sources were incomplete, incorrect, or duplicative when certification requests were submitted. They also stated that extensive manual cleanup efforts had been ongoing since the beginning of the investment review process. Since then, the department has established and used its DITIP system, which it developed in part as a response to our recommendations associated with improving data about business system investments. DITIP contains information on certification decisions, including the amounts requested for certification and subsequently certified (or not certified). According to officials from the Office of the DCMO and DOD business system investment review guidance, fiscal year 2014 certification requests and approval decisions were all entered and maintained directly in this system. As a result, information about investment certification requests and approval decisions was consistently documented.

While this new approach has allowed DOD to improve certain information about its business system certification and approvals, other information needs continued improvement. Specifically, we have previously reported that information about investment certifications and approvals was not always reliable, and we have made recommendations aimed at improving the department’s data about its business systems. For example, according to officials from the Office of the DCMO, the numbers of business systems reflected by DITIP and DITPR are not always consistent. These officials stated that data issues persist in DITPR due to the amount of data stored by the systems as well as the tight time frames associated with implementing improvements. DOD intends to make further updates to DITPR, including an update in the fourth quarter of fiscal year 2014, which will include a data validation process intended to align and reconcile the data common to DITPR and SNAP-IT via DITIP.

BEA and Business Process Reengineering Compliance Continues to Be Asserted for the Majority of Systems

Under the act, funds may not be obligated for covered business systems unless the precertification authority asserts that, among other things, the system is in compliance with the BEA and has undertaken appropriate business process reengineering efforts. For fiscal year 2014 certification reviews, investments were to follow the Office of the DCMO’s March 2013 BEA compliance guidance and September 2012 guidance on business process reengineering. The BEA guidance describes an incremental approach to BEA compliance requirements, including key elements needed for asserting architecture compliance. The business process reengineering guidance calls for specific levels of business process reengineering for systems in different stages of development. For example, prior to investing in development and modernization funds for a defense business system, the components are required to submit a problem statement to document and validate the system’s business need. DOD’s guidance required submission of a form that includes questions in areas related to various aspects of business process reengineering, such as change management and target process improvements.

For fiscal year 2014, DOD officials reported that 89 percent (1,053 of 1,180 systems) of defense business systems asserted compliance with the BEA. Of the 44 systems that asserted non-compliance, 35 were within one component of DOD—the Defense Finance and Accounting Service. According to DOD officials, this component now has plans to comply with BEA requirements and plans to update its assertions.

DOD’s business system investment review guidance calls for the department to validate selected business system BEA assessments that support the component’s assertions of compliance. We have previously reported on the need to improve the quality of BEA compliance assertions, and in 2013 we recommended that DOD implement and use the BEA compliance assessments more effectively to support organizational transformation efforts by, among other things, establishing milestones by which selected validations of BEA compliance assertions are to be completed. DOD partially agreed with this recommendation.

DOD selected 15 defense business systems for review, as called for in its guidance. DCMO officials stated that of the 15 selected, all had asserted that they complied with version 10.0 of the BEA. However, DCMO officials found that 13 of the 15 selected had issues with their BEA compliance assertions. Specifically, DCMO officials stated that they found that 9 of the 13 had inconsistencies in the organizational execution plan or had significantly incomplete compliance assertions in DOD’s BEA Compliance System, and 4 of the 15 were assessed as having other opportunities for improvements. According to DOD officials, some of these concerns were addressed prior to system certification. They further stated that no systems were denied funding certification due to incomplete BEA compliance assertions because the Office of the DCMO worked with the components to address the issues to the satisfaction of the DBC prior to certification. Where additional action was needed, action items were created and DCMO is tracking the progress of these systems’ efforts to become compliant.

The Office of the DCMO’s validations of selected assessments are important for helping to ensure that BEA assessments are accurate. However, as demonstrated by these results, DOD needs to continue working to ensure the quality of BEA assessments, as we have previously recommended. Continued improvement in the BEA assessments will help ensure that programs are being defined and implemented in a way that facilitates interoperability and avoids duplication and overlap, which are both goals of the BEA and the related investment management approach.

According to DOD officials, in fiscal year 2014, nearly 100 percent of certified and approved defense business systems asserted that business process reengineering had been completed. Further, the Office of the DCMO also validated a sample of assessments associated with these systems. According to DOD’s March 2014 report, this validation effort showed that assertions were incomplete. Specifically, the DCMO selected 12 investments for validation, and the results of this effort were as follows:

Four systems received a “positive” rating, indicating that appropriate business process reengineering was conducted.

Four systems received a “positive with recommendations” rating, indicating that sufficient business process reengineering was conducted but more could be done to improve the effort. DOD recommended that the precertification authorities for these systems ensure that appropriate business process reengineering compliance is again assessed at the next milestone. For example, one system, the Learning Management System, received a recommendation to consider creating expected business outcomes, such as providing career roadmaps to guide financial management employees in planning and progressing in a DOD career. Further, another system, the Next Generation Resource Management System did not provide evidence of a change management plan, or documentation of the resolution of the root causes identified in its “to-be” process models.

Four systems received a “could not validate” rating, indicating that the business process reengineering assessment team was not able to determine that a sufficient business process reengineering assessment had been conducted. In these cases, DOD recommended that the precertification authority ensure business process reengineering assessment compliance is assessed at the next acquisition milestone. For example, according to DOD officials, the systems could not be validated because insufficient objective evidence had been provided to substantiate their business process reengineering assertions. According to the Office of the DCMO, three of these four systems were in operations and maintenance, and it is not often cost-effective to conduct business process reengineering compliance reviews on investments in operations and maintenance.

The need for improvement demonstrated by some of DCMO’s selected validations is consistent with our previous findings. Specifically, in May 2013 we reported on business process reengineering assertions of four case study systems. All four assertions generally followed DOD’s business process reengineering guidance; however, we also found that limited documentation of root cause analyses was provided for all four. We have previously made recommendations to improve DOD’s business process reengineering-related efforts, including recommendations aimed at the four systems we assessed in 2013.

The Office of the DCMO’s validations of selected systems’ business process reengineering assessments are important for helping to ensure that sufficient business process reengineering is conducted. However, without further validation of the business process reengineering assertions—as we have previously recommended—DOD is at risk of not being able to accurately determine whether appropriate business process reengineering has been undertaken, including ensuring that the business processes supported by defense business systems have been streamlined and have eliminated or reduced unique requirements to the maximum practical extent.

DCMO Has Yet to Demonstrate That Its BEA Is Producing Business Value or Resulting in Elimination of Duplicative Systems

The act requires DOD to develop a BEA that covers all defense business systems and their related functions and activities and includes key information. For example, the act requires the BEA to include specific information about data standards, policies and procedures, and performance measures. In addition, the department seeks to use the BEA to help guide, constrain, and enable interoperable business systems. Moreover, we have previously emphasized that the BEA can be an important tool for reducing potential overlap and duplication among DOD business systems.

As we reported in May 2013, the current version of the BEA—Version 10.0— includes information aimed at meeting the act’s requirements, such as data standards; business rules to help ensure compliance with the laws, regulations, and policies incorporated in the BEA; and performance measures associated with the department’s Strategic Management Plan. However, as of April 2014, this version was still missing other important content associated with achieving the department’s goal of using the BEA to guide, constrain, and enable interoperable business systems. For example, the BEA was missing information about business systems associated with each of the architecture’s business activities, and its business activities have not been defined at a level that allows for more effective identification of potential duplication and overlap. Adding such content is important for various reasons, including improving the department’s ability to use the architecture as a tool for managing its business systems investments by more effectively identifying areas of potential duplication and overlap.

In addition, DOD has not yet demonstrated that the BEA has produced business value for the department. According to the Director of Investment Acquisition and Management and the Chief Architect, previous versions of the BEA were designed to meet the requirements of the act, but were not clearly focused on achieving business outcomes. However, the act defines specific business outcomes that the BEA is intended to support, such as integrating budget, accounting, and program information and systems, and the department has defined additional BEA goals, such as using the BEA to guide, constrain, and enable interoperable business systems. These officials stated that in the future they plan to use the BEA as a tool to better support interoperability, data sharing, and reducing duplication. To help achieve this goal, according to officials from the Office of the DCMO, DOD has established a process for defining changes to the BEA that are intended to better support the department’s ability to achieve business outcomes, and they first used this process for defining updates to BEA 10.0. More specifically, the Office of the DCMO established a BEA configuration control board— composed of representatives from the military departments and other departmental offices—to make recommendations to the DBC on BEA requirements and related content changes. Such a process may help the department as it refines the architecture and seeks to achieve business outcomes.

Further, while DOD has established a tool that can assist in identifying potential duplication and overlap among business systems even with the current gaps in BEA content, the department has not demonstrated that it has used this information to reduce duplication and overlap. The April 2013 investment management guidance and a January 2013 memorandum issued by the DCMO established a single BEA compliance tool—the BEA Compliance System—to document system-level BEA compliance assertions and required all business systems submitted for certification and approval for fiscal year 2014 to use this tool.

Officials from the Office of the DCMO stated that the results of BEA compliance assessments conducted as part of the business system certification and review process are available to staff across DOD to review in making determinations about potential overlap and duplication. For example, information provided by the department showed that, as of February 2014, 120 systems performed activities associated with maintaining asset information, 110 systems performed activities associated with managing military health services, and 73 systems performed activities associated with receiving and accepting a purchase request. Officials from the Army and the Air Force stated that in fiscal year 2014 they have begun to identify systems using BEA data that were potentially duplicative. According to these officials, both the Army and the Air Force have taken initial steps to phase out potentially duplicative systems. However, the department has yet to demonstrate that it has terminated these or other potentially duplicative programs that were identified by using the BEA.

Officials from the Office of the DCMO offered several reasons why the BEA compliance process has yet to result in the elimination of potentially duplicative systems. For example, officials stated that it is difficult for the DBC to make such determinations because information presented to the council is high level and only points to potential instances of overlap and duplication, which would require additional research. Instead, these officials said, precertification authorities at the component level are in the best position to determine whether their systems are duplicative of other systems. However, these precertification authorities are not required to identify and assess potential overlap and duplication before asserting compliance with the BEA. In addition, the BEA continues to lack important content, such as additional information about business activities, that would further assist in identifying potential duplication and overlap among business systems. Continued improvements to DOD’s BEA—as we have previously recommended—and calling for more proactive identification of potential overlap and duplication using information available in existing tools can help the department use its BEA to identify and address potential overlap and duplication among its billions of dollars in annual defense business system investments.

Enterprise Transition Plan Content Has Been Improved and Is Consistent with Portfolio- Based Investment Management Approach

The act calls for the development of an ETP that implements the BEA and covers all defense business systems and includes a listing of the (1) new systems that are expected to be needed to complete the target defense business systems computing environment, along with each system’s performance measures, financial resource needs, and risks or challenges to integration into the BEA; (2) legacy systems that will be phased out of the defense business systems computing environment within 3 years, together with the schedule for terminating those legacy systems; and (3) existing systems that are part of the target defense business systems computing environment, as well as a strategy for making the modifications to those systems that will be needed to ensure that such systems comply with the defense BEA, including time-phased milestones, performance measures, and financial resource needs.

The department’s fiscal year 2014 ETP, which was issued in December 2013 and uses data as of October 1, 2013, includes information about 1,179 covered defense business systems, including 79 legacy systems, or those that will be terminated within 36 months. The ETP also includes links to related documentation, including the functional strategies, organizational execution plans, and investment decision memoranda.

DOD has taken steps to improve its ETP by including key content, but has not included other information, as described below.

Milestones: The plan includes acquisition milestone information for covered defense business systems (both legacy and core) through its links to other systems, such as the Defense Acquisition Management Information Retrieval system, DITIP, DITPR, and Office of Management and Budget exhibit 300s. Specifically, according to DOD’s April 2013 investment management guidance, system owners are required to include start and end dates for life-cycle phases of each system within DITPR.

Performance measures: The plan includes portfolio-level performance measures in the organizational execution plans. For example, the Air Force Human Resources Management organizational execution plan includes performance measures associated with improving the accuracy and timeliness of pay for airmen. However, these measures are not linked to specific business systems.

Financial resource needs: The plan includes information about fiscal year 2014 funding that was requested and ultimately approved under the department’s business system investment review process for each business system (both legacy and core).

Risks or challenges for BEA integration: The plan and related documentation do not discuss each system’s risks or challenges to integration into the BEA. Instead, the organizational execution plans include portfolio-level risks and challenges, with only some related to the BEA. For example, the Air Force Defense Security Enterprise organizational execution plan identifies a challenge that the BEA and business process reengineering process do not fit well with the initiatives in this specific portfolio, and that many BEA and business process reengineering compliance requirements are not applicable.

Termination dates: The ETP data also includes termination dates for all 79 of the covered legacy systems and a listing of both new and existing systems that will be part of the target defense business systems computing environment.

Defining when and how the ETP will include the missing information—as we have previously recommended—will help to ensure that DOD more fully complies with the requirements of the act and help to inform oversight of its business systems investments.

DOD’s Fiscal Year 2015 Budget Submission Includes Key Information on Business Systems

Another requirement of the NDAA for Fiscal Year 2005, as amended, is that DOD’s annual IT budget submission must include key information on each business system for which funding is being requested, such as the system’s precertification authority and designated senior official, the appropriation type and amount of funds associated with modernization and current services (i.e., operation and maintenance), and the associated Defense Business Systems Management Committee approval decisions.

The department’s fiscal year 2015 budget submission includes a range of information for business system investments requesting funding, such as the system’s (1) name, (2) precertification authority, (3) designated senior official, (4) approved funding for fiscal year 2014, and (5) requested funding for fiscal year 2015. The submission also identifies the amount of the fiscal year 2015 request that is for modernization versus current services.

Conclusions

DOD has made important progress developing a framework to manage the department’s business systems and more fully comply with the act, yet additional and sustained efforts are necessary to fully achieve the department’s modernization goals. Specifically:

The Office of the DCMO developed a portfolio-based investment management framework—the Integrated Business Framework. However, establishing specific plans for aligning the budgeting and investment management processes would help ensure that systems included in the budget also meet the department’s mission needs and requirements, and may result in less money being budgeted for systems that are duplicative or do not meet the needs of the department. In addition, allowing less complex, costly, and risky systems to be reviewed at the component level will give the DBC more time to review more complex, costly, and risky systems.

The department also released a new system that should help to address data reliability issues and certified most systems as meeting the BEA and business process reengineering requirements. Implementing our prior recommendations in this area will help DOD improve the certification process.

The Office of the DCMO has taken steps to improve its ETP by including most of the key content required by the act. Fully implementing our prior recommendations will help to further inform oversight of the department’s business systems investments.

The department formed a configuration control board for the BEA to help manage future changes, which should provide decision makers with better information on which to base their certification and approval decisions. However, a more proactive use of the BEA—by requiring precertification authorities to ensure that their systems do not duplicate existing systems—will save time, money, and resources by not investing in redundant systems.

Accordingly, further refinements to DOD’s business system investment management process and more proactive use of the BEA to inform decisions would assist the department in leveraging its limited resources to more efficiently and effectively manage its business systems.

Recommendations for Executive Action

We are recommending that the Secretary of Defense direct the appropriate DOD management entity to take the following three actions to help ensure that the department’s business systems modernization program is fully compliant with the act and is more effectively implemented:

Define by when and how the department plans to align its business system certification and approval process with its Planning, Programming, Budgeting, and Execution process.

Define criteria for reviewing defense business systems at an appropriate level in the department based on factors such as complexity, scope, cost, and risk, in support of the certification and approval process.

Develop guidance requiring military departments and other defense organizations to use existing BEA content to more proactively identify potential duplication and overlap.

Agency Comments and Our Evaluation

We received written comments on a draft of this report from DOD, reprinted in appendix II. In its comments, the department concurred with two recommendations and partially concurred with one recommendation.

DOD concurred with our first recommendation, to define by when and how the department plans to align its business system certification and approval process with its Planning, Programming, Budgeting, and Execution process, and noted that further alignment of these processes will continue in fiscal year 2015.

The department partially concurred with our second recommendation, to define criteria for reviewing defense business systems at an appropriate level in the department, in support of the certification and approval process. In particular, DOD stated that it will continue to mature a process that incorporates a variety of factors, including business process complexity, risk, and portfolio costs. However, the department also stated that our recommendation implies that it has not ensured that reviews occur at an appropriate level. DOD noted that pre-certification authorities are required to review the entire portfolio of investments before they are presented to the DBC. The department added that these reviews enable decision making, to include resourcing decisions at an organizational level, which allows the DBC to focus on enterprise-level business issues facing DOD. DOD also stated that, for fiscal year 2015, pre-certification authorities will report the results of their review to the DBC, rather than simply presenting their portfolio for certification.

We acknowledge that DOD’s guidance calls for reviews of investment portfolios at multiple organizational levels, and our recommendation is not intended to imply otherwise. Rather, its intent is to help ensure that DOD establishes guidance and criteria for ensuring that detailed reviews of individual systems occur at the appropriate levels within the department. As described in this report, such an approach may involve establishing investment review boards at multiple organizational levels. Such a tiered review board approach would allow the DBC—acting at the executive- level IRB for the department—to focus its attention on more costly, complex, or risky systems, while delegating detailed reviews of other systems to lower organizational levels. By not using such an approach, the department may not be able to ensure that the quality of the information submitted for review is sufficient, and the associated investment reviews may not be adequate for identifying and addressing important issues associated with large-scale and costly systems. This is evidenced by the results of the Office of the DCMO’s validations of BEA and business process reengineering certifications discussed in this report. While DOD’s planned actions reflect continued improvement in the department’s oversight of defense business systems, these actions do not fully address the recommendation. Accordingly, we believe that full implementation of our recommendation is needed.

Finally, the department concurred with our recommendation to develop guidance requiring military departments and other defense organizations to use existing BEA content to more proactively identify potential duplication and overlap. DOD added that the department plans to improve its investment review process in fiscal year 2015 with a method that uses BEA data to identify candidate systems for further review and, if appropriate, retirement.

We are sending copies of this report to the appropriate congressional committees; the Director, Office of Management and Budget; the Secretary of Defense; and other interested parties. This report also is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff members have any questions on matters discussed in this report, please contact me at (202) 512-4456 or chac@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III.

Appendix I: Objective, Scope, and Methodology

Our objective was to assess the actions taken by the Department of Defense (DOD) to comply with section 332 of the National Defense Authorization Act for Fiscal Year 2005 as amended. The specific elements we assessed were (1) establishing a system investment approval and accountability structure along with an investment review process, (2) certifying and approving any business system program costing in excess of $1 million, (3) developing a business enterprise architecture (BEA) to cover all defense business systems, (4) developing a transition plan for implementing the architecture, and (5) identifying systems information in its annual budget submission.

To address the act’s provision for establishing an investment approval and accountability review structure, along with an investment review process, we reviewed the department’s current policies, procedures, and guidance relative to the act’s requirements and criteria documented in our information technology (IT) investment management framework. Specifically, to determine whether the foundation for DOD’s new investment management process had been fully established, we reviewed and analyzed the department’s investment management guidance— published in April 2013—and the six functional strategies prepared for the fiscal year 2014 certification process. We also reviewed DOD documentation on future plans, including a draft version of its updated investment management guidance and the March 2014 Congressional Report on Defense Business Operations. We reviewed and analyzed DOD’s investment management guidance to determine whether criteria and procedures had been fully defined for making portfolio-based investment decisions and the extent to which they aligned to our IT investment management framework. Further we reviewed prior GAO reports, DOD business system investment management guidance, and DOD budget process policy to determine how the budget process aligned with the business system investment review process. Finally, we interviewed cognizant officials in the Office of the Deputy Chief Management Officer (DCMO) to gain a better understanding of the investment management process, its outputs, and planned improvements for fiscal year 2015 defense business system certifications.

To determine whether the department was certifying and approving business system programs costing in excess of $1 million in accordance with the act’s provisions, we reviewed DOD’s March 2014 Congressional Report on Defense Business Operations, and all functional strategies and organizational execution plans, as well as their precertification request memoranda, which were submitted as part of the fiscal year 2014 investment review process. We also reviewed investment decision memoranda that documented the decisions made by the Defense Business Council.

To determine whether DOD had made progress in improving the data used to manage its business system investments, we reviewed data from the department’s three authoritative data sources—the Defense Information Technology Investment Portal (DITIP), the Defense Information Technology Portfolio Repository (DITPR), and the Select and Native Programming Data Input System for Information Technology (SNAP-IT). We compared our results with the information reported in DOD’s annual report to Congress to identify any discrepancies and discussed these with cognizant officials from the Office of the DCMO. Because DOD uses one system—DITIP—as the authoritative source for information about defense business system certification and approvals, we determined that the data were sufficiently reliable for our purposes.

As part of our evaluation under the certification requirement, we also identified the processes associated with asserting BEA and business process reengineering compliance for covered defense business systems and the documentation used to support those assertions. To accomplish this, we reviewed DOD guidance and related documentation that applied to the fiscal year 2014 investment portfolio reviews and relevant updates to that guidance issued during our audit. Further, we reviewed action items, the status of those actions, selected business process reengineering and BEA assertions, and data from DOD’s BEA compliance assertion system— the BEA Compliance System— to understand the extent to which defense business systems asserted compliance with BEA requirements. In addition, we interviewed officials from the Office of the DCMO about steps completed, under way, or planned relative to BEA and business process reengineering compliance and validation activities. We also reviewed DOD’s annual report to Congress to determine if business process reengineering outcomes had been reported.

To address the provision associated with developing the BEA, we reviewed our previous findings associated with version 10.0 of the BEA, released in February 2013, relative to the act’s architectural requirements. We also identified recommendations from previous GAO reports related to the act and measuring results and outcomes, and gathered documentation to support the current status of those recommendations. Specifically, we reviewed documentation related to steps completed, under way, or planned to address the act’s requirements and previously reported weaknesses, including slides presented to the Defense Business Council regarding BEA requirements criteria and updates, and the March 2014 Congressional Report on Defense Business Operations. In addition, we reviewed documentation related to efforts to achieve business value from the BEA, including meeting minutes and charters from the BEA configuration control board and sub-groups under that forum. To determine whether it is used to identify and reduce potential overlap and duplication, we reviewed data in DOD’s BEA Compliance System to determine how systems are mapping to the department’s business functions, and reviewed DOD’s guidance on BEA compliance. Finally, we interviewed officials from the Office of the DCMO about planned updates and improvements to the BEA’s content, and reasons that it has not yet been used to identify duplication and overlap.

To address the provision associated with developing a transition plan, we analyzed DOD’s enterprise transition plan, released in December 2013, and associated guidance in DOD’s April 2013 investment management guidance, relative to the act’s transition plan requirements and related findings documented in previous GAO reports. Specifically, we reviewed enterprise transition plan documentation, including functional strategies, organizational execution plans, Office of Management and Budget exhibit 300s, and data from the DITIP and DITPR systems, to identify whether that documentation met the requirements of the act. We also interviewed officials from the Office of the DCMO to determine the steps completed, under way, or planned to address the act’s requirements and previously reported weaknesses.

To assess the system information in DOD’s fiscal year 2015 information technology budget submission, we analyzed information contained in the department’s fiscal year 2015 budget request, including the department’s Section 332 Report and information contained in the systems that are used to prepare its budget submission—SNAP-IT, DITPR, and DITIP— against the act’s requirements in this area. We also interviewed officials from the Office of the DCMO and the Office of the Chief Information Officer to discuss the accuracy and comprehensiveness of information contained in the systems, and reasons for any discrepancies in the information they contained.

We conducted this performance audit from October 2013 to May 2014, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objective.

Appendix II: Comments from the Department of Defense

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact above, individuals making contributions to this report include Michael Holland (Assistant Director), Camille Chaires, Debra Conner, Kate Feild, Angel Ip, Anh Le, Lee McCracken, and Christine San.